2024-03-19T01:39:58.112 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T01:39:58.113 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359 --verbose --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/orig.config.yaml 2024-03-19T01:39:58.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T01:39:58.266 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359 branch: wip-batrick-testing-20240318.181317 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609359' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExK3MeTqY5yegk6BPu1UkymIULSfF7y5Y9GoblRO2xb0mDBqNcTgXalfPHOOnIecfH+PaBRfUf0C6WhuSd2nlU= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP76y+R/X891su6OJ6zDW3SuJGUFRxKzshM3Fq0BBiNykMchTCBKJlLTkePomavQiunKAn85o8KIA+9LoynM3Hk= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/1YuAsd4LGvbQGnzXfYEHIBR/xRsWmSfb2mOPTy3yIwEoyaGqIdEsSSMZubRw2zeS0sWMB6E8Cb0OJ/JxIGUU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T01:39:58.267 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T01:39:58.267 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T01:39:58.268 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T01:39:58.269 INFO:teuthology.task.internal:Checking packages... 2024-03-19T01:39:58.292 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T01:39:58.293 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T01:39:58.293 INFO:teuthology.packaging:ref: None 2024-03-19T01:39:58.293 INFO:teuthology.packaging:tag: None 2024-03-19T01:39:58.293 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T01:39:58.293 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:39:58.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T01:39:58.545 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T01:39:58.547 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T01:39:58.558 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T01:39:58.558 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T01:39:58.577 INFO:teuthology.task.internal:Saving configuration 2024-03-19T01:39:58.597 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T01:39:58.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:39:58.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.551268', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExK3MeTqY5yegk6BPu1UkymIULSfF7y5Y9GoblRO2xb0mDBqNcTgXalfPHOOnIecfH+PaBRfUf0C6WhuSd2nlU='} 2024-03-19T01:39:58.655 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.549751', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP76y+R/X891su6OJ6zDW3SuJGUFRxKzshM3Fq0BBiNykMchTCBKJlLTkePomavQiunKAn85o8KIA+9LoynM3Hk='} 2024-03-19T01:39:58.676 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.552466', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/1YuAsd4LGvbQGnzXfYEHIBR/xRsWmSfb2mOPTy3yIwEoyaGqIdEsSSMZubRw2zeS0sWMB6E8Cb0OJ/JxIGUU='} 2024-03-19T01:39:58.676 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T01:39:58.688 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-19T01:39:58.688 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-19T01:39:58.688 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-19T01:39:58.688 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T01:39:58.800 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f64b65f9160>, signals=[15]) 2024-03-19T01:39:58.801 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T01:39:58.811 INFO:teuthology.task.internal:Opening connections... 2024-03-19T01:39:58.812 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:39:58.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:39:58.904 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-03-19T01:39:58.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:39:58.992 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-19T01:39:58.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:39:59.081 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T01:39:59.090 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-19T01:39:59.108 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-19T01:39:59.109 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-19T01:39:59.166 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T01:39:59.169 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-19T01:39:59.169 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T01:39:59.169 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T01:39:59.170 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T01:39:59.170 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-19T01:39:59.191 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-03-19T01:39:59.209 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-03-19T01:39:59.209 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T01:39:59.271 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-03-19T01:39:59.272 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T01:39:59.272 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T01:39:59.272 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T01:39:59.272 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-03-19T01:39:59.291 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-19T01:39:59.308 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-19T01:39:59.308 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-19T01:39:59.365 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T01:39:59.366 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T01:39:59.367 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-19T01:39:59.391 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T01:39:59.403 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T01:39:59.424 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T01:39:59.425 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:39:59.427 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:39:59.430 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-19T01:39:59.445 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T01:39:59.456 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T01:39:59.456 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:39:59.484 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:39:59.490 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-19T01:39:59.596 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T01:39:59.681 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T01:39:59.700 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T01:39:59.700 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T01:39:59.701 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:39:59.701 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:39:59.702 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T01:39:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:39:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:39:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:39:59.720 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-19T01:39:59.721 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-19T01:39:59.721 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-03-19T01:39:59.777 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-03-19T01:39:59.777 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-03-19T01:39:59.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-03-19T01:39:59.778 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-19T01:39:59.778 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-19T01:39:59.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-19T01:39:59.781 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-361.el9.x86_64 2024-03-19T01:39:59.781 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-361.el9.x86_64 2024-03-19T01:39:59.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-03-19T01:40:02.007 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T01:40:02.032 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-19T01:40:02.470 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 6.3 MB/s | 6.7 MB 00:01 2024-03-19T01:40:03.374 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-19T01:40:03.415 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-19T01:40:03.854 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-19T01:44:29.749 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:25 2024-03-19T01:44:30.107 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:25 2024-03-19T01:44:31.021 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:27 2024-03-19T01:44:31.800 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-19T01:44:36.050 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-19T01:44:37.801 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-19T01:44:38.863 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-19T01:44:43.844 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-19T01:44:45.975 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T01:44:45.976 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T01:44:46.066 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-19T01:44:46.078 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T01:44:46.078 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-19T01:44:46.079 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T01:44:46.079 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-19T01:44:46.079 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T01:44:46.081 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-19T01:44:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T01:44:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 76 M 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-19T01:44:46.082 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-19T01:44:46.787 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-19T01:46:56.216 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T01:46:56.216 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T01:46:56.216 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T01:46:56.652 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 349 B/s | 44 kB 02:10 2024-03-19T01:47:08.368 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 107 kB/s | 15 MB 02:22 2024-03-19T01:47:20.083 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.2 MB/s | 27 MB 00:23 2024-03-19T01:47:23.741 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 217 kB/s | 33 MB 02:37 2024-03-19T01:47:23.742 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T01:47:23.742 INFO:teuthology.orchestra.run.smithi116.stdout:Total 493 kB/s | 76 MB 02:37 2024-03-19T01:47:24.401 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-19T01:47:24.786 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-19T01:47:24.786 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-19T01:47:25.775 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-19T01:47:25.776 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-19T01:47:28.241 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-19T01:47:28.903 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:47:28.982 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:47:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:47:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:47:36.589 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:47:36.685 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:47:41.389 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:48:10.517 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:48:10.517 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T01:48:10.518 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:48:10.561 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:48:10.903 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:48:10.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:48:10.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:48:10.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:48:11.268 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-19T01:48:11.601 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:48:11.694 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.694 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:48:11.694 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T01:48:11.695 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:48:11.695 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:48:11.695 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-19T01:48:11.695 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:48:11.695 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:48:11.774 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:48:11.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:48:11.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-19T01:48:11.808 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-19T01:48:11.808 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-19T01:48:11.808 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-19T01:48:11.808 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T01:48:11.809 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:48:11.810 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T01:48:11.811 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T01:48:11.811 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-430.el9.x86_64 2024-03-19T01:48:11.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T01:48:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-19T01:48:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T01:48:13.862 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-19T01:48:13.873 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-19T01:48:13.888 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.LO8MoQr9in 2024-03-19T01:48:13.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LO8MoQr9in 2024-03-19T01:48:13.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.LO8MoQr9in 2024-03-19T01:48:14.116 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.LO8MoQr9in is 6KB 2024-03-19T01:48:14.136 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.LO8MoQr9in 2024-03-19T01:48:14.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T01:48:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T01:48:14.215 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T01:48:14.215 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T01:48:14.215 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T01:48:14.216 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T01:48:14.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-19T01:48:44.377 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-19T01:48:44.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T01:48:44.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:49:02.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T01:49:03.615 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 71 kB/s | 19 MB 04:32 2024-03-19T01:49:07.558 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-19T01:49:09.966 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-19T01:49:11.651 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-19T01:49:11.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T01:49:11.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:49:14.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T01:49:16.558 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-19T01:49:17.577 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:27 2024-03-19T01:49:18.598 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T01:49:18.599 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T01:49:18.695 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T01:49:18.708 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T01:49:18.709 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:49:18.709 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-19T01:49:18.709 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-19T01:49:18.709 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-03-19T01:49:18.709 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:49:18.710 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 76 M 2024-03-19T01:49:18.710 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 123 M 2024-03-19T01:49:18.710 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-19T01:49:18.843 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-03-19T01:49:19.235 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 31 MB/s | 15 MB 00:00 2024-03-19T01:49:19.329 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-19T01:49:19.635 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-19T01:49:19.952 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-19T01:49:19.952 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-19T01:49:19.952 INFO:teuthology.orchestra.run.smithi162.stdout:Total 61 MB/s | 76 MB 00:01 2024-03-19T01:49:20.621 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-19T01:49:20.754 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-19T01:49:21.022 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-19T01:49:21.022 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-19T01:49:22.011 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-19T01:49:22.012 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-19T01:49:24.460 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-19T01:49:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:49:25.235 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:49:25.638 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-19T01:49:26.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T01:49:26.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:49:27.727 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T01:49:27.727 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T01:49:27.819 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:49:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 76 M 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 123 M 2024-03-19T01:49:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-19T01:49:27.980 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:49:28.118 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:49:28.222 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 422 kB/s | 44 kB 00:00 2024-03-19T01:49:28.606 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 30 MB/s | 15 MB 00:00 2024-03-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-19T01:49:29.306 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-19T01:49:29.306 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-19T01:49:29.306 INFO:teuthology.orchestra.run.smithi154.stdout:Total 51 MB/s | 76 MB 00:01 2024-03-19T01:49:29.972 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-19T01:49:30.342 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-19T01:49:30.342 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-19T01:49:31.329 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-19T01:49:31.329 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-19T01:49:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:49:32.878 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:49:33.806 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-19T01:49:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:49:34.515 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:49:37.245 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:49:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:49:37.601 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:49:42.086 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:49:42.195 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:49:45.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T01:49:46.920 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:00.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T01:50:00.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:50:00.717 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-19T01:50:01.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T01:50:01.316 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-19T01:50:01.316 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-19T01:50:01.336 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-430.el9.x86_64 2024-03-19T01:50:01.337 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T01:50:01.337 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:50:01.337 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T01:50:02.338 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-19T01:50:02.338 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:50:02.405 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-19T01:50:02.448 DEBUG:teuthology.parallel:result is None 2024-03-19T01:50:07.110 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:07.110 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T01:50:07.110 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:50:07.154 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:07.496 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:07.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:50:07.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:50:07.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:50:07.909 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-19T01:50:08.227 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:50:08.324 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:08.324 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:50:08.324 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T01:50:08.324 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:50:08.324 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:50:08.324 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-19T01:50:08.324 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:50:08.325 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:50:08.403 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:08.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:50:08.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T01:50:08.447 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:50:08.448 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T01:50:08.449 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T01:50:08.449 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T01:50:08.449 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-03-19T01:50:08.449 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:50:08.449 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T01:50:08.450 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T01:50:08.450 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-430.el9.x86_64 2024-03-19T01:50:08.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T01:50:09.094 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-03-19T01:50:10.491 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T01:50:10.530 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-03-19T01:50:10.532 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-19T01:50:10.547 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.AKj7uUg7FE 2024-03-19T01:50:10.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AKj7uUg7FE 2024-03-19T01:50:10.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.AKj7uUg7FE 2024-03-19T01:50:10.757 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.AKj7uUg7FE is 6KB 2024-03-19T01:50:10.805 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.AKj7uUg7FE 2024-03-19T01:50:10.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T01:50:10.884 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T01:50:10.884 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T01:50:10.885 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T01:50:10.885 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T01:50:10.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T01:50:11.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-03-19T01:50:16.497 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:16.498 INFO:teuthology.orchestra.run.smithi154.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T01:50:16.498 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:50:16.541 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:16.877 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:16.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T01:50:16.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T01:50:16.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.296 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:50:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-19T01:50:17.602 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:50:17.705 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:50:17.706 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T01:50:17.706 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T01:50:17.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T01:50:17.706 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-19T01:50:17.706 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T01:50:17.706 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T01:50:17.785 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T01:50:17.785 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T01:50:17.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-03-19T01:50:17.819 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-03-19T01:50:17.819 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 68.el9 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8264254 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-03-19T01:50:17.820 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T01:50:17.821 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T01:50:17.822 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T01:50:17.822 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-430.el9.x86_64 2024-03-19T01:50:17.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T01:50:18.431 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-03-19T01:50:19.762 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T01:50:19.800 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-03-19T01:50:19.802 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-03-19T01:50:19.817 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.8VYFjYyhJi 2024-03-19T01:50:19.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8VYFjYyhJi 2024-03-19T01:50:19.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.8VYFjYyhJi 2024-03-19T01:50:20.028 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.8VYFjYyhJi is 6KB 2024-03-19T01:50:20.077 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.8VYFjYyhJi 2024-03-19T01:50:20.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T01:50:20.156 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T01:50:20.157 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T01:50:20.157 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T01:50:20.157 INFO:teuthology.orchestra.run.smithi154.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T01:50:20.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T01:50:20.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-03-19T01:50:41.033 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-19T01:50:41.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:50:41.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:50:50.330 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-19T01:50:50.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T01:50:50.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:50:59.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-19T01:51:08.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:51:08.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:51:11.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-19T01:51:23.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:51:23.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:51:24.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-03-19T01:51:33.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T01:51:33.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:51:41.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-19T01:51:52.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-03-19T01:51:56.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:51:56.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:51:59.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-19T01:52:04.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T01:52:04.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:52:07.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-03-19T01:52:17.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:52:17.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:52:18.338 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-19T01:52:18.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T01:52:18.715 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-19T01:52:18.715 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-19T01:52:18.731 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-430.el9.x86_64 2024-03-19T01:52:18.731 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T01:52:18.731 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:52:18.731 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T01:52:19.732 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-19T01:52:19.733 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:52:19.807 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-03-19T01:52:19.848 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:22.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T01:52:22.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T01:52:22.788 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-03-19T01:52:23.157 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T01:52:23.157 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-19T01:52:23.158 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-03-19T01:52:23.173 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-430.el9.x86_64 2024-03-19T01:52:23.173 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T01:52:23.174 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T01:52:23.174 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T01:52:24.175 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-19T01:52:24.175 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T01:52:24.247 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-03-19T01:52:24.282 DEBUG:teuthology.parallel:result is None 2024-03-19T01:52:24.282 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T01:52:24.289 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T01:52:24.290 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:52:24.293 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:52:24.295 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T01:52:24.313 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T01:52:24.319 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T01:52:24.336 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T01:52:24.337 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:52:24.351 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:52:24.353 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T01:52:24.386 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T01:52:24.393 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T01:52:24.393 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:52:24.411 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:52:24.421 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T01:52:24.446 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.453 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.456 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.460 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.466 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.470 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T01:52:24.472 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T01:52:24.479 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T01:52:24.479 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:52:24.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:52:24.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T01:52:24.543 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T01:52:24.553 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T01:52:24.554 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:52:24.574 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:52:24.580 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T01:52:24.600 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:24.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:24.718 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:24.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:24.843 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:52:24.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:52:24.911 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:24.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:25.018 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:25.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:25.107 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T01:52:25.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:52:25.177 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:25.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T01:52:25.284 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:25.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T01:52:25.372 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:52:25.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T01:52:25.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-19T01:52:25.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-03-19T01:52:25.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-19T01:52:25.495 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:52:25.500 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:52:25.526 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:52:26.010 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T01:52:26.018 INFO:teuthology.task.internal:Starting timer... 2024-03-19T01:52:26.018 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T01:52:26.093 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T01:52:26.102 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-19T01:52:26.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-19T01:52:26.168 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-19T01:52:26.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-03-19T01:52:26.221 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-03-19T01:52:26.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-19T01:52:26.276 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-19T01:52:26.278 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T01:52:26.279 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-19T01:52:26.303 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-19T01:52:26.304 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-03-19T01:52:26.338 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-03-19T01:52:26.338 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-19T01:52:26.359 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-19T01:52:26.360 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-03-19T01:52:26.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:52:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:52:26.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:52:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:52:26.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:52:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:52:26.454 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T01:52:26.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-19T01:52:26.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-03-19T01:52:26.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-19T01:52:26.569 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T01:52:26.580 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T01:52:26.597 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T01:52:26.598 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi154.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T02:03:16.259 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-03-19T02:03:16.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T02:03:16.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:03:16.400 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-19T02:03:16.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T02:03:16.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T02:03:16.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:03:16.586 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-03-19T02:03:16.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-19T02:03:16.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T02:03:16.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:03:16.756 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-19T02:03:16.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T02:03:16.843 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T02:03:16.853 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T02:03:16.854 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:03:16.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:03:16.856 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:03:16.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:03:16.859 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:03:16.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:03:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:03:16.892 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:03:16.904 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:03:16.910 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:03:16.916 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:03:16.933 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:03:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-19T02:03:16.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-03-19T02:03:16.948 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-19T02:03:16.956 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-03-19T02:03:16.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-19T02:03:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:03:16.973 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:03:16.978 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-19T02:03:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:03:16.990 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:03:16.994 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:03:17.011 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:03:17.033 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.036 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.045 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.049 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.062 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.066 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:03:17.067 INFO:teuthology.run_tasks:Running task install... 2024-03-19T02:03:17.075 DEBUG:teuthology.task.install:project ceph 2024-03-19T02:03:17.075 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T02:03:17.075 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T02:03:17.075 INFO:teuthology.task.install:Using flavor: default 2024-03-19T02:03:17.081 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T02:03:17.082 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T02:03:17.082 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:03:17.082 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:03:17.084 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:03:17.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:03:17.085 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:03:17.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:03:17.225 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:03:17.225 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:03:17.231 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:03:17.232 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:03:17.302 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:03:17.302 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:03:17.510 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:03:17.511 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:03:17.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:03:17.540 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:03:17.540 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:03:17.585 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:03:17.585 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:03:17.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:03:17.600 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:03:17.600 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:03:17.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:03:17.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:03:17.615 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:03:17.615 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:03:17.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:03:17.630 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:03:17.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:03:17.688 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:03:17.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:03:17.753 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-03-19T02:03:17.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-03-19T02:03:17.765 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:03:17.780 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:03:17.830 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-19T02:03:17.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-19T02:03:17.847 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-19T02:03:17.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-19T02:03:18.022 INFO:teuthology.orchestra.run.smithi154.stdout:74 files removed 2024-03-19T02:03:18.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:03:18.068 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-19T02:03:18.090 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-19T02:03:18.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:03:18.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:03:19.070 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 124 kB/s | 83 kB 00:00 2024-03-19T02:03:19.073 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 126 kB/s | 83 kB 00:00 2024-03-19T02:03:19.096 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 112 kB/s | 83 kB 00:00 2024-03-19T02:03:19.665 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-19T02:03:19.705 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 18 kB/s | 10 kB 00:00 2024-03-19T02:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-19T02:03:20.132 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-03-19T02:03:20.271 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-19T02:03:20.618 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-19T02:03:20.899 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T02:03:21.298 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 1.2 kB/s | 1.9 kB 00:01 2024-03-19T02:03:21.959 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T02:03:22.110 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-19T02:03:22.331 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-19T02:03:23.600 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-19T02:03:25.124 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-19T02:03:26.675 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-19T02:03:30.058 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 19 MB 00:03 2024-03-19T02:03:36.808 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.1 MB 00:03 2024-03-19T02:03:38.418 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-19T02:03:40.205 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-19T02:03:42.075 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-19T02:03:45.127 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-19T02:03:46.100 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-19T02:03:47.509 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:03:47.510 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:03:47.524 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:03:47.525 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:03:47.526 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:03:47.527 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:03:47.529 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:03:47.532 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:03:47.533 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:03:47.559 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-19T02:03:47.614 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-19T02:03:47.637 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-19T02:03:47.637 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-03-19T02:03:47.637 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-19T02:03:47.637 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-03-19T02:03:47.637 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:03:47.638 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:03:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:03:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:03:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:03:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:03:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:03:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:03:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:03:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:03:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:03:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:03:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:03:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:Install 150 Packages 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 24 Packages 2024-03-19T02:03:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:03:47.652 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 224 M 2024-03-19T02:03:47.652 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-19T02:03:48.595 INFO:teuthology.orchestra.run.smithi154.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-03-19T02:03:48.962 INFO:teuthology.orchestra.run.smithi154.stdout:(2/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 9.8 MB/s | 5.4 MB 00:00 2024-03-19T02:03:48.996 INFO:teuthology.orchestra.run.smithi154.stdout:(3/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 2.2 MB/s | 891 kB 00:00 2024-03-19T02:03:49.021 INFO:teuthology.orchestra.run.smithi154.stdout:(4/174): ceph-immutable-object-cache-19.0.0-220 2.4 MB/s | 145 kB 00:00 2024-03-19T02:03:49.146 INFO:teuthology.orchestra.run.smithi154.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 12 MB/s | 1.5 MB 00:00 2024-03-19T02:03:49.463 INFO:teuthology.orchestra.run.smithi154.stdout:(6/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 4.7 MB 00:00 2024-03-19T02:03:49.539 INFO:teuthology.orchestra.run.smithi154.stdout:(7/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 4.3 MB/s | 2.4 MB 00:00 2024-03-19T02:03:50.414 INFO:teuthology.orchestra.run.smithi154.stdout:(8/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 18 MB/s | 17 MB 00:00 2024-03-19T02:03:50.490 INFO:teuthology.orchestra.run.smithi154.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 336 kB/s | 25 kB 00:00 2024-03-19T02:03:50.769 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 21 MB 00:03 2024-03-19T02:03:52.631 INFO:teuthology.orchestra.run.smithi154.stdout:(10/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 3.2 MB/s | 9.8 MB 00:03 2024-03-19T02:03:52.698 INFO:teuthology.orchestra.run.smithi154.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.3 MB/s | 221 kB 00:00 2024-03-19T02:03:53.173 INFO:teuthology.orchestra.run.smithi154.stdout:(12/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 18 MB/s | 48 MB 00:02 2024-03-19T02:03:53.198 INFO:teuthology.orchestra.run.smithi154.stdout:(13/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 54 kB/s | 27 kB 00:00 2024-03-19T02:03:53.265 INFO:teuthology.orchestra.run.smithi154.stdout:(14/174): libcephfs-devel-19.0.0-2206.gb6ae2491 346 kB/s | 32 kB 00:00 2024-03-19T02:03:53.365 INFO:teuthology.orchestra.run.smithi154.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.6 MB/s | 163 kB 00:00 2024-03-19T02:03:53.457 INFO:teuthology.orchestra.run.smithi154.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.3 MB/s | 127 kB 00:00 2024-03-19T02:03:53.491 INFO:teuthology.orchestra.run.smithi154.stdout:(17/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 2.5 MB/s | 746 kB 00:00 2024-03-19T02:03:53.575 INFO:teuthology.orchestra.run.smithi154.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 4.2 MB/s | 501 kB 00:00 2024-03-19T02:03:53.675 INFO:teuthology.orchestra.run.smithi154.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 452 kB/s | 45 kB 00:00 2024-03-19T02:03:53.867 INFO:teuthology.orchestra.run.smithi154.stdout:(20/174): ceph-common-19.0.0-2206.gb6ae2491.el9 3.8 MB/s | 21 MB 00:05 2024-03-19T02:03:53.892 INFO:teuthology.orchestra.run.smithi154.stdout:(21/174): python3-ceph-common-19.0.0-2206.gb6ae 605 kB/s | 131 kB 00:00 2024-03-19T02:03:53.993 INFO:teuthology.orchestra.run.smithi154.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.1 MB/s | 321 kB 00:00 2024-03-19T02:03:54.060 INFO:teuthology.orchestra.run.smithi154.stdout:(23/174): python3-cephfs-19.0.0-2206.gb6ae2491. 843 kB/s | 162 kB 00:00 2024-03-19T02:03:54.093 INFO:teuthology.orchestra.run.smithi154.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.9 MB/s | 298 kB 00:00 2024-03-19T02:03:54.177 INFO:teuthology.orchestra.run.smithi154.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 849 kB/s | 99 kB 00:00 2024-03-19T02:03:54.202 INFO:teuthology.orchestra.run.smithi154.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 786 kB/s | 85 kB 00:00 2024-03-19T02:03:54.294 INFO:teuthology.orchestra.run.smithi154.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-19T02:03:54.395 INFO:teuthology.orchestra.run.smithi154.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 339 kB/s | 34 kB 00:00 2024-03-19T02:03:54.495 INFO:teuthology.orchestra.run.smithi154.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.4 MB/s | 139 kB 00:00 2024-03-19T02:03:54.703 INFO:teuthology.orchestra.run.smithi154.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 8.6 MB/s | 1.8 MB 00:00 2024-03-19T02:03:55.120 INFO:teuthology.orchestra.run.smithi154.stdout:(31/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 3.2 MB/s | 5.1 MB 00:01 2024-03-19T02:03:55.204 INFO:teuthology.orchestra.run.smithi154.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-19T02:03:55.283 INFO:teuthology.orchestra.run.smithi154.stdout:(33/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 2.8 MB/s | 3.1 MB 00:01 2024-03-19T02:03:55.313 INFO:teuthology.orchestra.run.smithi154.stdout:(34/174): ceph-mgr-modules-core-19.0.0-2206.gb6 1.3 MB/s | 248 kB 00:00 2024-03-19T02:03:55.338 INFO:teuthology.orchestra.run.smithi154.stdout:(35/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 369 kB/s | 49 kB 00:00 2024-03-19T02:03:55.388 INFO:teuthology.orchestra.run.smithi154.stdout:(36/174): ceph-prometheus-alerts-19.0.0-2206.gb 157 kB/s | 16 kB 00:00 2024-03-19T02:03:55.447 INFO:teuthology.orchestra.run.smithi154.stdout:(37/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 2.4 MB/s | 264 kB 00:00 2024-03-19T02:03:55.480 INFO:teuthology.orchestra.run.smithi154.stdout:(38/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 5.2 MB/s | 749 kB 00:00 2024-03-19T02:03:55.581 INFO:teuthology.orchestra.run.smithi154.stdout:(39/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 95 kB/s | 18 kB 00:00 2024-03-19T02:03:55.594 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-19T02:03:55.873 INFO:teuthology.orchestra.run.smithi154.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-19T02:03:55.898 INFO:teuthology.orchestra.run.smithi154.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 335 kB/s | 106 kB 00:00 2024-03-19T02:03:55.973 INFO:teuthology.orchestra.run.smithi154.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 307 kB 00:00 2024-03-19T02:03:55.998 INFO:teuthology.orchestra.run.smithi154.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 316 kB/s | 32 kB 00:00 2024-03-19T02:03:56.265 INFO:teuthology.orchestra.run.smithi154.stdout:(44/174): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 920 kB 00:00 2024-03-19T02:03:56.340 INFO:teuthology.orchestra.run.smithi154.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 50 kB/s | 17 kB 00:00 2024-03-19T02:03:56.524 INFO:teuthology.orchestra.run.smithi154.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 39 kB/s | 10 kB 00:00 2024-03-19T02:03:56.608 INFO:teuthology.orchestra.run.smithi154.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.7 MB/s | 3.0 MB 00:00 2024-03-19T02:03:56.633 INFO:teuthology.orchestra.run.smithi154.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 545 kB/s | 159 kB 00:00 2024-03-19T02:03:56.658 INFO:teuthology.orchestra.run.smithi154.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 331 kB/s | 44 kB 00:00 2024-03-19T02:03:56.742 INFO:teuthology.orchestra.run.smithi154.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-19T02:03:56.767 INFO:teuthology.orchestra.run.smithi154.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 429 kB/s | 46 kB 00:00 2024-03-19T02:03:56.834 INFO:teuthology.orchestra.run.smithi154.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 3.2 MB/s | 578 kB 00:00 2024-03-19T02:03:56.868 INFO:teuthology.orchestra.run.smithi154.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-19T02:03:56.893 INFO:teuthology.orchestra.run.smithi154.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-19T02:03:56.918 INFO:teuthology.orchestra.run.smithi154.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-03-19T02:03:56.944 INFO:teuthology.orchestra.run.smithi154.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-03-19T02:03:57.502 INFO:teuthology.orchestra.run.smithi154.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 1.4 MB/s | 811 kB 00:00 2024-03-19T02:03:57.628 INFO:teuthology.orchestra.run.smithi154.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.1 MB/s | 1.5 MB 00:00 2024-03-19T02:03:57.770 INFO:teuthology.orchestra.run.smithi154.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 502 kB/s | 134 kB 00:00 2024-03-19T02:03:57.836 INFO:teuthology.orchestra.run.smithi154.stdout:(60/174): openblas-openmp-0.3.21-2.el9.x86_64.r 5.0 MB/s | 4.7 MB 00:00 2024-03-19T02:03:58.026 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:03:58.027 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:03:58.042 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:03:58.043 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:03:58.044 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:03:58.045 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:03:58.048 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:03:58.051 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:03:58.052 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:03:58.133 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-19T02:03:58.136 INFO:teuthology.orchestra.run.smithi154.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 260 kB/s | 78 kB 00:00 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:03:58.149 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:03:58.150 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:03:58.151 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:03:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:03:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:03:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:03:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:03:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:03:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:03:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:03:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:03:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:03:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T02:03:58.162 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-19T02:03:58.163 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 24 Packages 2024-03-19T02:03:58.163 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:03:58.163 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 224 M 2024-03-19T02:03:58.163 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-19T02:03:58.237 INFO:teuthology.orchestra.run.smithi154.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 2.2 MB/s | 1.0 MB 00:00 2024-03-19T02:03:58.482 INFO:teuthology.orchestra.run.smithi154.stdout:(63/174): postgresql-server-13.11-1.el9.x86_64. 6.7 MB/s | 5.7 MB 00:00 2024-03-19T02:03:58.513 INFO:teuthology.orchestra.run.smithi154.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 89 kB/s | 24 kB 00:00 2024-03-19T02:03:58.755 INFO:teuthology.orchestra.run.smithi154.stdout:(65/174): python3-babel-2.9.1-2.el9.noarch.rpm 9.4 MB/s | 5.8 MB 00:00 2024-03-19T02:03:58.780 INFO:teuthology.orchestra.run.smithi154.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 61 kB/s | 18 kB 00:00 2024-03-19T02:03:58.821 INFO:teuthology.orchestra.run.smithi154.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 471 kB/s | 144 kB 00:00 2024-03-19T02:03:58.847 INFO:teuthology.orchestra.run.smithi154.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 345 kB/s | 32 kB 00:00 2024-03-19T02:03:58.931 INFO:teuthology.orchestra.run.smithi154.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-19T02:03:58.954 INFO:teuthology.orchestra.run.smithi116.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-19T02:03:59.022 INFO:teuthology.orchestra.run.smithi154.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 996 kB/s | 174 kB 00:00 2024-03-19T02:03:59.181 INFO:teuthology.orchestra.run.smithi154.stdout:(71/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-19T02:03:59.206 INFO:teuthology.orchestra.run.smithi154.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 251 kB/s | 69 kB 00:00 2024-03-19T02:03:59.223 INFO:teuthology.orchestra.run.smithi154.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 206 kB/s | 41 kB 00:00 2024-03-19T02:03:59.248 INFO:teuthology.orchestra.run.smithi154.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-19T02:03:59.274 INFO:teuthology.orchestra.run.smithi154.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T02:03:59.299 INFO:teuthology.orchestra.run.smithi154.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-19T02:03:59.316 INFO:teuthology.orchestra.run.smithi154.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 708 kB/s | 47 kB 00:00 2024-03-19T02:03:59.333 INFO:teuthology.orchestra.run.smithi154.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 728 kB/s | 43 kB 00:00 2024-03-19T02:03:59.350 INFO:teuthology.orchestra.run.smithi154.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-19T02:03:59.437 INFO:teuthology.orchestra.run.smithi116.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.8 MB/s | 891 kB 00:00 2024-03-19T02:03:59.542 INFO:teuthology.orchestra.run.smithi154.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 197 kB/s | 41 kB 00:00 2024-03-19T02:03:59.562 INFO:teuthology.orchestra.run.smithi116.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 1.1 MB/s | 145 kB 00:00 2024-03-19T02:03:59.567 INFO:teuthology.orchestra.run.smithi154.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-19T02:03:59.717 INFO:teuthology.orchestra.run.smithi154.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 323 kB/s | 56 kB 00:00 2024-03-19T02:03:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:(4/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 4.7 MB/s | 5.4 MB 00:01 2024-03-19T02:03:59.776 INFO:teuthology.orchestra.run.smithi154.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 301 kB/s | 63 kB 00:00 2024-03-19T02:03:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:(5/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.6 MB/s | 2.4 MB 00:00 2024-03-19T02:03:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:(6/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-19T02:03:59.985 INFO:teuthology.orchestra.run.smithi154.stdout:(84/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-03-19T02:04:00.001 INFO:teuthology.orchestra.run.smithi154.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 156 kB/s | 44 kB 00:00 2024-03-19T02:04:00.027 INFO:teuthology.orchestra.run.smithi154.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 315 kB/s | 79 kB 00:00 2024-03-19T02:04:00.052 INFO:teuthology.orchestra.run.smithi154.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-19T02:04:00.077 INFO:teuthology.orchestra.run.smithi154.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-19T02:04:00.119 INFO:teuthology.orchestra.run.smithi154.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-19T02:04:00.144 INFO:teuthology.orchestra.run.smithi154.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-19T02:04:00.161 INFO:teuthology.orchestra.run.smithi154.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-19T02:04:00.330 INFO:teuthology.orchestra.run.smithi116.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-19T02:04:00.753 INFO:teuthology.orchestra.run.smithi154.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 330 kB/s | 195 kB 00:00 2024-03-19T02:04:00.778 INFO:teuthology.orchestra.run.smithi154.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 284 kB/s | 187 kB 00:00 2024-03-19T02:04:00.920 INFO:teuthology.orchestra.run.smithi154.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 152 kB/s | 21 kB 00:00 2024-03-19T02:04:00.954 INFO:teuthology.orchestra.run.smithi154.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 623 kB/s | 125 kB 00:00 2024-03-19T02:04:01.037 INFO:teuthology.orchestra.run.smithi154.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 218 kB/s | 25 kB 00:00 2024-03-19T02:04:01.338 INFO:teuthology.orchestra.run.smithi154.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 818 kB/s | 245 kB 00:00 2024-03-19T02:04:01.455 INFO:teuthology.orchestra.run.smithi154.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-19T02:04:01.547 INFO:teuthology.orchestra.run.smithi116.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 8.1 MB/s | 9.8 MB 00:01 2024-03-19T02:04:01.705 INFO:teuthology.orchestra.run.smithi154.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.5 MB/s | 4.1 MB 00:00 2024-03-19T02:04:01.739 INFO:teuthology.orchestra.run.smithi116.stdout:(9/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 6.6 MB/s | 21 MB 00:03 2024-03-19T02:04:01.764 INFO:teuthology.orchestra.run.smithi116.stdout:(10/174): ceph-selinux-19.0.0-2206.gb6ae2491.el 116 kB/s | 25 kB 00:00 2024-03-19T02:04:01.822 INFO:teuthology.orchestra.run.smithi154.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 299 kB/s | 35 kB 00:00 2024-03-19T02:04:01.831 INFO:teuthology.orchestra.run.smithi116.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.2 MB/s | 221 kB 00:00 2024-03-19T02:04:01.907 INFO:teuthology.orchestra.run.smithi116.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 361 kB/s | 27 kB 00:00 2024-03-19T02:04:01.930 INFO:teuthology.orchestra.run.smithi154.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-19T02:04:02.007 INFO:teuthology.orchestra.run.smithi116.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 318 kB/s | 32 kB 00:00 2024-03-19T02:04:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 864 kB/s | 94 kB 00:00 2024-03-19T02:04:02.189 INFO:teuthology.orchestra.run.smithi154.stdout:(103/174): libgfortran-11.4.1-3.el9.x86_64.rpm 396 kB/s | 803 kB 00:02 2024-03-19T02:04:02.207 INFO:teuthology.orchestra.run.smithi116.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 3.6 MB/s | 746 kB 00:00 2024-03-19T02:04:02.306 INFO:teuthology.orchestra.run.smithi154.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-19T02:04:02.324 INFO:teuthology.orchestra.run.smithi116.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.4 MB/s | 163 kB 00:00 2024-03-19T02:04:02.456 INFO:teuthology.orchestra.run.smithi154.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.6 MB/s | 548 kB 00:00 2024-03-19T02:04:02.491 INFO:teuthology.orchestra.run.smithi116.stdout:(16/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 6.6 MB/s | 17 MB 00:02 2024-03-19T02:04:02.498 INFO:teuthology.orchestra.run.smithi154.stdout:(106/174): qatlib-23.11.0-1.el9.x86_64.rpm 207 kB/s | 216 kB 00:01 2024-03-19T02:04:02.516 INFO:teuthology.orchestra.run.smithi116.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 661 kB/s | 127 kB 00:00 2024-03-19T02:04:02.532 INFO:teuthology.orchestra.run.smithi154.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-03-19T02:04:02.591 INFO:teuthology.orchestra.run.smithi154.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-03-19T02:04:02.608 INFO:teuthology.orchestra.run.smithi116.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 4.2 MB/s | 501 kB 00:00 2024-03-19T02:04:02.675 INFO:teuthology.orchestra.run.smithi154.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-03-19T02:04:02.716 INFO:teuthology.orchestra.run.smithi116.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 417 kB/s | 45 kB 00:00 2024-03-19T02:04:02.717 INFO:teuthology.orchestra.run.smithi154.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-03-19T02:04:02.742 INFO:teuthology.orchestra.run.smithi154.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-19T02:04:02.784 INFO:teuthology.orchestra.run.smithi154.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-19T02:04:02.809 INFO:teuthology.orchestra.run.smithi154.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-19T02:04:02.833 INFO:teuthology.orchestra.run.smithi116.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.1 MB/s | 131 kB 00:00 2024-03-19T02:04:02.859 INFO:teuthology.orchestra.run.smithi154.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-19T02:04:02.893 INFO:teuthology.orchestra.run.smithi154.stdout:(115/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-03-19T02:04:02.910 INFO:teuthology.orchestra.run.smithi154.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 867 kB/s | 43 kB 00:00 2024-03-19T02:04:02.943 INFO:teuthology.orchestra.run.smithi154.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 838 kB 00:00 2024-03-19T02:04:02.960 INFO:teuthology.orchestra.run.smithi154.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-19T02:04:02.967 INFO:teuthology.orchestra.run.smithi116.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.2 MB/s | 162 kB 00:00 2024-03-19T02:04:02.977 INFO:teuthology.orchestra.run.smithi154.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 383 kB/s | 32 kB 00:00 2024-03-19T02:04:03.002 INFO:teuthology.orchestra.run.smithi154.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-19T02:04:03.028 INFO:teuthology.orchestra.run.smithi154.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-19T02:04:03.045 INFO:teuthology.orchestra.run.smithi154.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 840 kB/s | 35 kB 00:00 2024-03-19T02:04:03.078 INFO:teuthology.orchestra.run.smithi154.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-19T02:04:03.092 INFO:teuthology.orchestra.run.smithi116.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 2.5 MB/s | 321 kB 00:00 2024-03-19T02:04:03.104 INFO:teuthology.orchestra.run.smithi154.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.3 MB/s | 303 kB 00:00 2024-03-19T02:04:03.129 INFO:teuthology.orchestra.run.smithi154.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-19T02:04:03.146 INFO:teuthology.orchestra.run.smithi154.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-19T02:04:03.163 INFO:teuthology.orchestra.run.smithi154.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 529 kB/s | 18 kB 00:00 2024-03-19T02:04:03.188 INFO:teuthology.orchestra.run.smithi154.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 555 kB/s | 23 kB 00:00 2024-03-19T02:04:03.205 INFO:teuthology.orchestra.run.smithi154.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 463 kB/s | 19 kB 00:00 2024-03-19T02:04:03.209 INFO:teuthology.orchestra.run.smithi116.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.5 MB/s | 298 kB 00:00 2024-03-19T02:04:03.255 INFO:teuthology.orchestra.run.smithi154.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-03-19T02:04:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-19T02:04:03.289 INFO:teuthology.orchestra.run.smithi154.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-19T02:04:03.301 INFO:teuthology.orchestra.run.smithi116.stdout:(24/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 6.6 MB/s | 5.1 MB 00:00 2024-03-19T02:04:03.327 INFO:teuthology.orchestra.run.smithi116.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 849 kB/s | 99 kB 00:00 2024-03-19T02:04:03.331 INFO:teuthology.orchestra.run.smithi154.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-03-19T02:04:03.356 INFO:teuthology.orchestra.run.smithi154.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 941 kB/s | 79 kB 00:00 2024-03-19T02:04:03.382 INFO:teuthology.orchestra.run.smithi154.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2024-03-19T02:04:03.402 INFO:teuthology.orchestra.run.smithi116.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 853 kB/s | 85 kB 00:00 2024-03-19T02:04:03.407 INFO:teuthology.orchestra.run.smithi154.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 324 kB/s | 16 kB 00:00 2024-03-19T02:04:03.433 INFO:teuthology.orchestra.run.smithi154.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-19T02:04:03.458 INFO:teuthology.orchestra.run.smithi154.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-03-19T02:04:03.483 INFO:teuthology.orchestra.run.smithi154.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-03-19T02:04:03.502 INFO:teuthology.orchestra.run.smithi116.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-19T02:04:03.586 INFO:teuthology.orchestra.run.smithi116.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 407 kB/s | 34 kB 00:00 2024-03-19T02:04:03.600 INFO:teuthology.orchestra.run.smithi154.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-03-19T02:04:03.678 INFO:teuthology.orchestra.run.smithi116.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.5 MB/s | 139 kB 00:00 2024-03-19T02:04:03.692 INFO:teuthology.orchestra.run.smithi154.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-03-19T02:04:03.717 INFO:teuthology.orchestra.run.smithi154.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-03-19T02:04:03.743 INFO:teuthology.orchestra.run.smithi154.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 663 kB/s | 188 kB 00:00 2024-03-19T02:04:03.768 INFO:teuthology.orchestra.run.smithi154.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-19T02:04:03.785 INFO:teuthology.orchestra.run.smithi154.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-19T02:04:03.818 INFO:teuthology.orchestra.run.smithi154.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-19T02:04:03.844 INFO:teuthology.orchestra.run.smithi154.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-03-19T02:04:03.861 INFO:teuthology.orchestra.run.smithi154.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-19T02:04:03.870 INFO:teuthology.orchestra.run.smithi116.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 5.7 MB/s | 3.1 MB 00:00 2024-03-19T02:04:03.886 INFO:teuthology.orchestra.run.smithi154.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-03-19T02:04:03.953 INFO:teuthology.orchestra.run.smithi154.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-19T02:04:03.998 INFO:teuthology.orchestra.run.smithi154.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-19T02:04:04.028 INFO:teuthology.orchestra.run.smithi116.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 5.1 MB/s | 1.8 MB 00:00 2024-03-19T02:04:04.129 INFO:teuthology.orchestra.run.smithi116.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.4 MB/s | 248 kB 00:00 2024-03-19T02:04:04.206 INFO:teuthology.orchestra.run.smithi116.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 657 kB/s | 49 kB 00:00 2024-03-19T02:04:04.304 INFO:teuthology.orchestra.run.smithi116.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 165 kB/s | 16 kB 00:00 2024-03-19T02:04:04.520 INFO:teuthology.orchestra.run.smithi154.stdout:(152/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-19T02:04:04.795 INFO:teuthology.orchestra.run.smithi154.stdout:(153/174): librados2-19.0.0-2206.gb6ae2491.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-19T02:04:04.896 INFO:teuthology.orchestra.run.smithi116.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 446 kB/s | 264 kB 00:00 2024-03-19T02:04:05.005 INFO:teuthology.orchestra.run.smithi116.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-03-19T02:04:05.055 INFO:teuthology.orchestra.run.smithi116.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 4.6 MB/s | 749 kB 00:00 2024-03-19T02:04:05.062 INFO:teuthology.orchestra.run.smithi154.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 158 kB/s | 42 kB 00:00 2024-03-19T02:04:05.080 INFO:teuthology.orchestra.run.smithi116.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 243 kB/s | 18 kB 00:00 2024-03-19T02:04:05.247 INFO:teuthology.orchestra.run.smithi116.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-19T02:04:05.281 INFO:teuthology.orchestra.run.smithi116.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.2 MB/s | 106 kB 00:00 2024-03-19T02:04:05.314 INFO:teuthology.orchestra.run.smithi116.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 920 kB 00:00 2024-03-19T02:04:05.321 INFO:teuthology.orchestra.run.smithi154.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 962 kB/s | 248 kB 00:00 2024-03-19T02:04:05.339 INFO:teuthology.orchestra.run.smithi116.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-19T02:04:05.365 INFO:teuthology.orchestra.run.smithi116.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-19T02:04:05.438 INFO:teuthology.orchestra.run.smithi154.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 613 kB/s | 71 kB 00:00 2024-03-19T02:04:05.440 INFO:teuthology.orchestra.run.smithi116.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-19T02:04:05.465 INFO:teuthology.orchestra.run.smithi116.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2024-03-19T02:04:05.499 INFO:teuthology.orchestra.run.smithi116.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-19T02:04:05.524 INFO:teuthology.orchestra.run.smithi116.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-19T02:04:05.538 INFO:teuthology.orchestra.run.smithi154.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 365 kB/s | 36 kB 00:00 2024-03-19T02:04:05.549 INFO:teuthology.orchestra.run.smithi116.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-19T02:04:05.574 INFO:teuthology.orchestra.run.smithi116.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-19T02:04:05.600 INFO:teuthology.orchestra.run.smithi116.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-19T02:04:05.625 INFO:teuthology.orchestra.run.smithi116.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 931 kB/s | 46 kB 00:00 2024-03-19T02:04:05.651 INFO:teuthology.orchestra.run.smithi116.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-19T02:04:05.676 INFO:teuthology.orchestra.run.smithi116.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-19T02:04:05.701 INFO:teuthology.orchestra.run.smithi116.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-19T02:04:05.713 INFO:teuthology.orchestra.run.smithi154.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.1 MB/s | 192 kB 00:00 2024-03-19T02:04:05.718 INFO:teuthology.orchestra.run.smithi116.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-03-19T02:04:05.805 INFO:teuthology.orchestra.run.smithi154.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 1.3 MB/s | 2.3 MB 00:01 2024-03-19T02:04:05.830 INFO:teuthology.orchestra.run.smithi154.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-19T02:04:05.906 INFO:teuthology.orchestra.run.smithi154.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-19T02:04:05.931 INFO:teuthology.orchestra.run.smithi154.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-19T02:04:06.006 INFO:teuthology.orchestra.run.smithi154.stdout:(163/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.2 MB/s | 1.8 MB 00:01 2024-03-19T02:04:06.034 INFO:teuthology.orchestra.run.smithi154.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-03-19T02:04:06.059 INFO:teuthology.orchestra.run.smithi154.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-19T02:04:06.109 INFO:teuthology.orchestra.run.smithi154.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 623 kB/s | 64 kB 00:00 2024-03-19T02:04:06.134 INFO:teuthology.orchestra.run.smithi154.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-19T02:04:06.169 INFO:teuthology.orchestra.run.smithi116.stdout:(56/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 11 MB/s | 48 MB 00:04 2024-03-19T02:04:06.244 INFO:teuthology.orchestra.run.smithi116.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.7 MB/s | 4.7 MB 00:00 2024-03-19T02:04:06.286 INFO:teuthology.orchestra.run.smithi116.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.7 MB/s | 1.5 MB 00:00 2024-03-19T02:04:06.319 INFO:teuthology.orchestra.run.smithi116.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-19T02:04:06.335 INFO:teuthology.orchestra.run.smithi154.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 1.2 MB 00:00 2024-03-19T02:04:06.344 INFO:teuthology.orchestra.run.smithi116.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-19T02:04:06.436 INFO:teuthology.orchestra.run.smithi116.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-03-19T02:04:06.469 INFO:teuthology.orchestra.run.smithi154.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-03-19T02:04:06.528 INFO:teuthology.orchestra.run.smithi116.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-19T02:04:06.545 INFO:teuthology.orchestra.run.smithi116.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 391 kB/s | 78 kB 00:00 2024-03-19T02:04:06.577 INFO:teuthology.orchestra.run.smithi154.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-19T02:04:06.637 INFO:teuthology.orchestra.run.smithi116.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-19T02:04:06.654 INFO:teuthology.orchestra.run.smithi116.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 198 kB/s | 24 kB 00:00 2024-03-19T02:04:06.671 INFO:teuthology.orchestra.run.smithi116.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-19T02:04:06.694 INFO:teuthology.orchestra.run.smithi154.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-19T02:04:06.696 INFO:teuthology.orchestra.run.smithi116.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-19T02:04:06.713 INFO:teuthology.orchestra.run.smithi116.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 765 kB/s | 32 kB 00:00 2024-03-19T02:04:06.795 INFO:teuthology.orchestra.run.smithi154.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 743 kB/s | 74 kB 00:00 2024-03-19T02:04:06.822 INFO:teuthology.orchestra.run.smithi116.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-19T02:04:06.847 INFO:teuthology.orchestra.run.smithi116.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-19T02:04:06.872 INFO:teuthology.orchestra.run.smithi116.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-19T02:04:06.889 INFO:teuthology.orchestra.run.smithi116.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-19T02:04:06.906 INFO:teuthology.orchestra.run.smithi116.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-19T02:04:06.931 INFO:teuthology.orchestra.run.smithi116.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-19T02:04:06.948 INFO:teuthology.orchestra.run.smithi116.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-19T02:04:06.973 INFO:teuthology.orchestra.run.smithi116.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 5.0 MB/s | 211 kB 00:00 2024-03-19T02:04:06.998 INFO:teuthology.orchestra.run.smithi116.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 712 kB/s | 47 kB 00:00 2024-03-19T02:04:07.024 INFO:teuthology.orchestra.run.smithi116.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-19T02:04:07.091 INFO:teuthology.orchestra.run.smithi116.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 421 kB/s | 39 kB 00:00 2024-03-19T02:04:07.103 INFO:teuthology.orchestra.run.smithi154.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.4 MB/s | 2.3 MB 00:00 2024-03-19T02:04:07.166 INFO:teuthology.orchestra.run.smithi116.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 289 kB/s | 41 kB 00:00 2024-03-19T02:04:07.236 INFO:teuthology.orchestra.run.smithi154.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.9 MB/s | 2.2 MB 00:01 2024-03-19T02:04:07.241 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:04:07.241 INFO:teuthology.orchestra.run.smithi154.stdout:Total 11 MB/s | 224 MB 00:19 2024-03-19T02:04:07.303 INFO:teuthology.orchestra.run.smithi116.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-19T02:04:07.467 INFO:teuthology.orchestra.run.smithi116.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-19T02:04:07.492 INFO:teuthology.orchestra.run.smithi116.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 174 kB/s | 56 kB 00:00 2024-03-19T02:04:07.526 INFO:teuthology.orchestra.run.smithi116.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 284 kB/s | 63 kB 00:00 2024-03-19T02:04:07.551 INFO:teuthology.orchestra.run.smithi116.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 753 kB/s | 44 kB 00:00 2024-03-19T02:04:07.576 INFO:teuthology.orchestra.run.smithi116.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 941 kB/s | 79 kB 00:00 2024-03-19T02:04:07.660 INFO:teuthology.orchestra.run.smithi116.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 241 kB 00:00 2024-03-19T02:04:07.693 INFO:teuthology.orchestra.run.smithi116.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 8.1 MB/s | 1.1 MB 00:00 2024-03-19T02:04:07.718 INFO:teuthology.orchestra.run.smithi116.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 725 kB/s | 103 kB 00:00 2024-03-19T02:04:07.720 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-19T02:04:07.760 INFO:teuthology.orchestra.run.smithi116.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-19T02:04:07.786 INFO:teuthology.orchestra.run.smithi116.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-19T02:04:07.969 INFO:teuthology.orchestra.run.smithi116.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 748 kB/s | 187 kB 00:00 2024-03-19T02:04:07.999 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-19T02:04:07.999 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-19T02:04:08.028 INFO:teuthology.orchestra.run.smithi116.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-19T02:04:08.053 INFO:teuthology.orchestra.run.smithi116.stdout:(94/174): libquadmath-11.4.1-3.el9.x86_64.rpm 731 kB/s | 195 kB 00:00 2024-03-19T02:04:08.129 INFO:teuthology.orchestra.run.smithi116.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-19T02:04:08.320 INFO:teuthology.orchestra.run.smithi116.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 74 kB/s | 21 kB 00:00 2024-03-19T02:04:08.345 INFO:teuthology.orchestra.run.smithi116.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 87 kB/s | 25 kB 00:00 2024-03-19T02:04:08.746 INFO:teuthology.orchestra.run.smithi116.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 577 kB/s | 245 kB 00:00 2024-03-19T02:04:08.771 INFO:teuthology.orchestra.run.smithi116.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 583 kB/s | 248 kB 00:00 2024-03-19T02:04:08.879 INFO:teuthology.orchestra.run.smithi116.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 322 kB/s | 35 kB 00:00 2024-03-19T02:04:08.930 INFO:teuthology.orchestra.run.smithi116.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-19T02:04:08.989 INFO:teuthology.orchestra.run.smithi116.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 409 kB/s | 44 kB 00:00 2024-03-19T02:04:09.039 INFO:teuthology.orchestra.run.smithi116.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 862 kB/s | 94 kB 00:00 2024-03-19T02:04:09.064 INFO:teuthology.orchestra.run.smithi116.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-03-19T02:04:09.289 INFO:teuthology.orchestra.run.smithi116.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.6 MB/s | 4.1 MB 00:01 2024-03-19T02:04:09.465 INFO:teuthology.orchestra.run.smithi116.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-03-19T02:04:09.573 INFO:teuthology.orchestra.run.smithi116.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-19T02:04:09.665 INFO:teuthology.orchestra.run.smithi116.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 823 kB/s | 309 kB 00:00 2024-03-19T02:04:09.690 INFO:teuthology.orchestra.run.smithi116.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-19T02:04:09.741 INFO:teuthology.orchestra.run.smithi116.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-19T02:04:09.841 INFO:teuthology.orchestra.run.smithi116.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-19T02:04:09.858 INFO:teuthology.orchestra.run.smithi116.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-03-19T02:04:09.925 INFO:teuthology.orchestra.run.smithi116.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-19T02:04:10.008 INFO:teuthology.orchestra.run.smithi116.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-19T02:04:10.011 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-19T02:04:10.012 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-19T02:04:10.075 INFO:teuthology.orchestra.run.smithi116.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-03-19T02:04:10.101 INFO:teuthology.orchestra.run.smithi116.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-19T02:04:10.151 INFO:teuthology.orchestra.run.smithi116.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-19T02:04:10.176 INFO:teuthology.orchestra.run.smithi116.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-19T02:04:10.201 INFO:teuthology.orchestra.run.smithi116.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-19T02:04:10.243 INFO:teuthology.orchestra.run.smithi116.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-19T02:04:10.269 INFO:teuthology.orchestra.run.smithi116.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-19T02:04:10.294 INFO:teuthology.orchestra.run.smithi116.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-19T02:04:10.319 INFO:teuthology.orchestra.run.smithi116.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-03-19T02:04:10.344 INFO:teuthology.orchestra.run.smithi116.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-19T02:04:10.395 INFO:teuthology.orchestra.run.smithi116.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-19T02:04:10.420 INFO:teuthology.orchestra.run.smithi116.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-19T02:04:10.479 INFO:teuthology.orchestra.run.smithi116.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-19T02:04:10.496 INFO:teuthology.orchestra.run.smithi116.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-19T02:04:10.512 INFO:teuthology.orchestra.run.smithi116.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-19T02:04:10.537 INFO:teuthology.orchestra.run.smithi116.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-03-19T02:04:10.571 INFO:teuthology.orchestra.run.smithi116.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-19T02:04:10.596 INFO:teuthology.orchestra.run.smithi116.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 791 kB/s | 46 kB 00:00 2024-03-19T02:04:10.639 INFO:teuthology.orchestra.run.smithi116.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-03-19T02:04:10.655 INFO:teuthology.orchestra.run.smithi116.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-03-19T02:04:10.680 INFO:teuthology.orchestra.run.smithi116.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 722 kB/s | 79 kB 00:00 2024-03-19T02:04:10.714 INFO:teuthology.orchestra.run.smithi116.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-19T02:04:10.731 INFO:teuthology.orchestra.run.smithi116.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-19T02:04:10.756 INFO:teuthology.orchestra.run.smithi116.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-19T02:04:10.773 INFO:teuthology.orchestra.run.smithi116.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-03-19T02:04:10.821 INFO:teuthology.orchestra.run.smithi116.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 342 kB/s | 31 kB 00:00 2024-03-19T02:04:10.841 INFO:teuthology.orchestra.run.smithi116.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-19T02:04:10.866 INFO:teuthology.orchestra.run.smithi116.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-19T02:04:10.883 INFO:teuthology.orchestra.run.smithi116.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 587 kB/s | 36 kB 00:00 2024-03-19T02:04:10.899 INFO:teuthology.orchestra.run.smithi116.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2024-03-19T02:04:10.942 INFO:teuthology.orchestra.run.smithi116.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-19T02:04:10.976 INFO:teuthology.orchestra.run.smithi116.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 971 kB/s | 90 kB 00:00 2024-03-19T02:04:11.001 INFO:teuthology.orchestra.run.smithi116.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-03-19T02:04:11.018 INFO:teuthology.orchestra.run.smithi116.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-19T02:04:11.043 INFO:teuthology.orchestra.run.smithi116.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-19T02:04:11.135 INFO:teuthology.orchestra.run.smithi116.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-19T02:04:11.185 INFO:teuthology.orchestra.run.smithi116.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-19T02:04:11.402 INFO:teuthology.orchestra.run.smithi116.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 8.8 MB/s | 3.4 MB 00:00 2024-03-19T02:04:11.611 INFO:teuthology.orchestra.run.smithi116.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-03-19T02:04:11.661 INFO:teuthology.orchestra.run.smithi116.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 844 kB/s | 42 kB 00:00 2024-03-19T02:04:11.803 INFO:teuthology.orchestra.run.smithi116.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-19T02:04:11.853 INFO:teuthology.orchestra.run.smithi116.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-19T02:04:11.903 INFO:teuthology.orchestra.run.smithi116.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.2 MB/s | 2.3 MB 00:00 2024-03-19T02:04:11.929 INFO:teuthology.orchestra.run.smithi116.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 483 kB/s | 36 kB 00:00 2024-03-19T02:04:12.020 INFO:teuthology.orchestra.run.smithi116.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-19T02:04:12.046 INFO:teuthology.orchestra.run.smithi116.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.3 MB/s | 192 kB 00:00 2024-03-19T02:04:12.071 INFO:teuthology.orchestra.run.smithi116.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-19T02:04:12.121 INFO:teuthology.orchestra.run.smithi116.stdout:(162/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.5 MB/s | 1.8 MB 00:00 2024-03-19T02:04:12.146 INFO:teuthology.orchestra.run.smithi116.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-19T02:04:12.172 INFO:teuthology.orchestra.run.smithi116.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-19T02:04:12.205 INFO:teuthology.orchestra.run.smithi116.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-19T02:04:12.231 INFO:teuthology.orchestra.run.smithi116.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 491 kB/s | 29 kB 00:00 2024-03-19T02:04:12.323 INFO:teuthology.orchestra.run.smithi116.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 675 kB/s | 136 kB 00:00 2024-03-19T02:04:12.698 INFO:teuthology.orchestra.run.smithi116.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 2.5 MB/s | 1.2 MB 00:00 2024-03-19T02:04:12.873 INFO:teuthology.orchestra.run.smithi116.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 2.6 MB/s | 465 kB 00:00 2024-03-19T02:04:12.915 INFO:teuthology.orchestra.run.smithi116.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.2 MB/s | 2.2 MB 00:00 2024-03-19T02:04:12.940 INFO:teuthology.orchestra.run.smithi116.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-19T02:04:13.074 INFO:teuthology.orchestra.run.smithi116.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 2.3 MB 00:00 2024-03-19T02:04:13.207 INFO:teuthology.orchestra.run.smithi116.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 279 kB/s | 74 kB 00:00 2024-03-19T02:04:13.307 INFO:teuthology.orchestra.run.smithi116.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.7 MB/s | 1.1 MB 00:00 2024-03-19T02:04:13.315 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:04:13.315 INFO:teuthology.orchestra.run.smithi116.stdout:Total 15 MB/s | 224 MB 00:15 2024-03-19T02:04:13.767 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-19T02:04:14.039 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-19T02:04:14.039 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-19T02:04:16.034 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-19T02:04:16.035 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-19T02:04:16.930 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-19T02:04:17.016 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:04:17.042 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:04:17.131 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:04:17.195 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:04:17.382 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:04:17.436 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:04:17.476 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:04:17.502 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:04:17.559 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:04:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:04:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:04:18.066 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:04:18.103 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:04:18.195 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:04:18.228 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:04:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:04:18.467 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:04:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:04:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:04:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:04:18.940 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:04:18.972 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:04:19.003 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:04:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:04:19.323 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:04:19.384 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:04:19.464 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:19.543 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:19.741 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:19.857 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:04:19.951 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:04:19.981 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:04:20.009 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:04:20.053 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:04:20.126 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:04:20.180 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:04:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:04:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:04:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:04:20.361 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:04:20.394 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:04:20.421 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:04:20.490 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:04:20.526 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:04:20.569 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:04:20.598 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:04:20.669 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:04:20.721 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:04:20.796 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:04:20.930 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:04:20.983 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:04:21.507 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:04:21.588 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:04:21.681 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:04:21.731 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:04:21.739 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:04:21.780 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:04:21.820 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:04:21.858 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:04:21.888 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:04:22.144 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:04:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:04:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:04:22.442 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:04:22.609 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:04:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:04:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:04:22.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:04:22.812 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:04:22.849 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:04:22.896 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:04:22.932 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:04:22.975 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:04:23.018 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:04:23.114 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-19T02:04:23.154 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:04:23.205 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:04:23.209 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:04:23.235 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:04:23.236 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:04:23.265 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:04:23.274 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:04:23.277 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:04:23.315 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:04:23.372 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:04:23.373 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:04:23.402 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:04:23.410 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:04:23.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:04:23.452 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:04:23.468 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:04:23.493 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:04:23.503 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:04:23.581 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:04:23.711 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:04:23.812 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:04:23.833 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:04:23.851 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:04:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:04:23.901 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:04:23.956 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:04:24.009 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:04:24.038 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:04:24.138 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:04:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:04:24.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:04:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:04:24.213 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:04:24.325 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:04:24.362 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:04:24.388 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:04:24.393 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:04:24.424 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:04:24.440 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:04:24.455 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:04:24.468 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:04:24.487 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:04:24.526 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:04:24.558 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:04:24.571 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:04:24.588 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:04:24.603 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:04:24.665 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:04:24.669 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:04:24.698 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:04:24.716 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:04:24.738 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:04:24.739 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:04:24.754 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:04:24.781 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:04:24.815 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:04:24.906 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:04:24.962 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:04:24.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:04:25.005 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:04:25.034 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:04:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:04:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:04:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:04:25.216 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:04:25.232 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:04:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:04:25.345 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:04:25.405 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:04:25.499 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:25.755 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:04:25.863 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:04:25.963 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:04:25.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:04:26.013 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:04:26.021 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:04:26.057 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:04:26.060 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:04:26.093 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:04:26.131 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:04:26.142 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:04:26.193 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:04:26.231 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:04:26.232 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:04:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:04:26.296 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:04:26.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:04:26.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:04:26.409 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:04:26.436 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:04:26.457 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:04:26.484 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:04:26.505 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:04:26.541 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:04:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:04:26.584 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:04:26.613 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:04:26.682 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:04:26.736 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:04:26.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:04:26.941 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:04:26.996 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:04:27.218 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:04:27.323 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:04:27.518 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:04:27.601 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:04:27.679 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:04:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:04:27.743 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:04:27.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:04:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:04:27.855 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:04:27.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:04:28.179 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:04:28.414 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:04:28.529 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:04:28.573 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:04:28.600 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:04:28.699 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:04:28.707 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:04:28.732 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:04:28.752 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:04:28.763 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:04:28.791 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:04:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:04:28.971 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:04:29.095 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:04:29.104 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:04:29.129 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:04:29.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:04:29.178 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:04:29.205 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:04:29.213 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:04:29.233 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:04:29.256 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:04:29.279 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:04:29.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:04:29.430 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:04:29.478 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:04:29.508 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:04:29.550 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:04:29.643 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:04:29.683 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:04:29.724 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:04:29.739 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:04:29.749 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:04:29.765 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:04:29.773 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:04:29.819 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:04:29.842 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:04:29.855 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:04:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:04:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:04:30.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:04:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:04:30.237 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:04:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:04:30.469 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:04:30.598 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:04:30.638 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:04:30.674 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:04:30.705 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:04:30.733 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:04:30.767 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:04:30.806 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:04:30.838 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:04:30.868 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:04:30.943 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:04:30.978 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:04:31.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:04:31.053 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:04:31.087 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:04:31.176 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:04:31.266 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:04:31.310 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:04:31.341 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:04:31.388 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:04:31.484 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:04:31.534 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:04:32.082 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:32.121 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:32.255 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:04:32.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:04:32.332 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:04:32.379 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:04:32.459 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:04:32.491 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:04:32.668 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:32.675 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:04:32.697 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:04:32.708 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:04:32.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:04:32.794 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:04:33.422 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:04:33.527 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:04:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:04:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:04:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:04:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:04:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:04:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:04:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:04:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:04:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:04:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:04:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:04:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:04:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:04:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:38.260 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:38.803 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:04:38.838 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:04:38.933 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:04:38.934 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:04:38.934 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:04:38.934 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:38.998 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-03-19T02:04:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-03-19T02:04:44.352 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:44.513 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:04:44.536 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:04:44.536 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:44.536 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:04:44.536 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:46.016 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:04:46.042 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:46.267 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:04:46.290 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:46.734 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:04:46.757 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:04:46.757 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:46.758 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:04:46.758 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:04:46.758 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:04:46.758 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:04:47.286 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:04:47.334 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:04:47.367 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:04:47.427 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:04:47.449 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:04:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:04:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:04:48.684 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:04:48.709 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:04:48.810 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:04:48.972 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:04:49.022 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:04:49.042 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:04:49.043 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:49.043 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:04:49.043 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:04:49.043 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:04:49.043 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:49.103 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:04:49.114 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:04:49.162 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:04:50.029 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:04:50.055 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:04:50.077 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:04:50.078 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:50.078 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:04:50.078 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:04:50.078 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:04:50.078 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:50.122 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:04:50.144 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:04:50.145 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:50.145 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:04:50.145 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:50.190 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:04:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:04:50.212 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:50.213 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:04:50.213 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:04:50.213 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:04:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:50.516 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:04:50.538 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:04:50.539 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:50.539 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:04:50.539 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:04:50.539 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:04:50.539 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:04:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-19T02:04:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:51.077 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:04:51.092 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:04:51.092 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:51.092 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:04:51.092 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:52.572 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:04:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:04:52.844 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:04:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:04:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:04:52.845 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:04:52.845 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:53.281 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:04:53.303 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:04:53.304 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:53.304 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:04:53.304 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:04:53.304 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:04:53.304 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:04:53.850 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:04:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:04:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:04:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:04:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:04:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:04:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:04:55.003 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:04:55.053 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:04:55.083 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:04:55.118 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:04:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:04:55.255 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:04:55.312 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:04:55.358 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:04:55.446 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:04:55.492 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:04:55.520 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:04:55.548 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:04:55.549 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:04:55.572 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:55.600 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:04:55.617 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:04:55.628 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:04:55.651 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:04:55.682 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:04:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:04:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:04:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:04:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:04:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:04:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:04:56.494 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:04:56.536 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:04:56.554 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:04:56.556 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:04:56.594 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:04:56.616 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:56.661 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:04:56.682 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:04:56.682 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:56.682 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:04:56.682 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:04:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:04:56.759 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:56.759 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:04:56.759 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:04:56.759 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:04:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:57.062 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:04:57.084 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:04:57.084 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:04:57.084 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:04:57.084 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:04:57.085 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:04:57.085 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:04:57.398 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:04:57.472 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:04:57.513 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:04:57.612 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:04:57.642 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:04:57.667 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:04:57.693 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:04:57.719 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:04:57.719 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:04:57.791 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:04:57.820 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:04:57.820 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:04:57.905 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:04:57.932 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:04:57.964 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:04:57.989 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:04:58.015 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:04:58.048 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:04:58.087 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:04:58.119 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:04:58.144 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:04:58.169 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:04:58.195 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:04:58.220 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:04:58.227 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:04:58.227 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:04:58.406 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:04:58.406 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:04:58.506 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:04:58.530 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:04:58.563 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:04:58.590 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:04:58.623 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:04:58.689 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:05:01.550 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:05:01.591 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:05:01.622 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:05:01.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:05:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:05:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:05:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:05:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:05:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:05:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:05:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:05:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:05:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:05:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:05:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:05:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:05:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:05:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:05:03.238 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:05:03.622 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:05:03.623 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:05:03.624 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:05:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:05:03.645 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:05:03.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:05:03.647 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:05:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:05:03.649 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:05:03.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:05:03.651 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:05:03.652 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:05:03.653 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:05:03.654 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:05:03.655 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:05:03.657 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:05:03.658 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:05:03.899 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:05:03.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:05:04.010 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:05:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:05:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:05:04.156 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:05:04.182 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:05:04.208 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:05:04.208 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:05:04.279 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:05:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:05:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:05:04.401 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:05:04.428 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:05:04.461 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:05:04.486 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:05:04.512 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:05:04.545 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:05:04.584 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:05:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:05:04.641 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:05:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:05:04.692 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:05:04.717 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:05:04.723 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:05:04.723 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:05:04.903 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:05:04.903 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:05:04.986 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:05:05.011 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:05:05.036 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:05:05.069 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:05:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:05:05.170 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:05:06.708 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:05:06.709 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.710 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.711 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.712 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:05:06.713 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.714 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:06.715 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:05:06.716 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:05:06.717 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:05:06.718 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:05:06.719 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:05:06.720 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:05:06.721 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:05:06.722 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:05:06.723 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:05:06.723 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-19T02:05:06.723 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-19T02:05:07.204 DEBUG:teuthology.parallel:result is None 2024-03-19T02:05:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:05:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:05:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:05:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:05:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:05:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:05:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:05:09.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:05:09.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:05:09.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:05:09.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:05:09.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:05:09.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:05:09.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:05:09.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:05:09.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:05:09.899 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:05:09.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:05:09.901 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:05:09.902 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:05:09.903 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:05:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:05:12.786 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:05:12.787 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.788 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.789 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.790 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:05:12.791 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:05:12.792 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:05:12.793 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:05:12.794 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:05:12.795 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:05:12.796 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:05:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:05:12.798 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:05:12.799 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:05:12.800 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:05:12.801 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:05:12.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:05:12.801 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-19T02:05:13.276 DEBUG:teuthology.parallel:result is None 2024-03-19T02:07:50.612 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:27 2024-03-19T02:08:00.877 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-19T02:08:06.310 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-19T02:08:08.086 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-19T02:08:09.779 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-19T02:08:14.654 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-19T02:08:17.058 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:08:17.059 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:08:17.073 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:08:17.074 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:08:17.075 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:08:17.076 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:08:17.078 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:08:17.082 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:08:17.082 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:08:17.165 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-19T02:08:17.190 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:08:17.191 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:08:17.192 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:08:17.193 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:08:17.194 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:08:17.195 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:08:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:08:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:08:17.198 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:08:17.199 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:08:17.200 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:08:17.201 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:08:17.202 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:08:17.203 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:08:17.204 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout:Install 150 Packages 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 24 Packages 2024-03-19T02:08:17.205 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:08:17.206 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 224 M 2024-03-19T02:08:17.206 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-19T02:08:17.696 INFO:teuthology.orchestra.run.smithi162.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-19T02:08:18.313 INFO:teuthology.orchestra.run.smithi162.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.4 MB/s | 891 kB 00:00 2024-03-19T02:08:18.472 INFO:teuthology.orchestra.run.smithi162.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 915 kB/s | 145 kB 00:00 2024-03-19T02:08:18.830 INFO:teuthology.orchestra.run.smithi162.stdout:(4/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 15 MB/s | 21 MB 00:01 2024-03-19T02:08:18.930 INFO:teuthology.orchestra.run.smithi162.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 1.5 MB 00:00 2024-03-19T02:08:19.222 INFO:teuthology.orchestra.run.smithi162.stdout:(6/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-19T02:08:19.306 INFO:teuthology.orchestra.run.smithi162.stdout:(7/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 2.8 MB/s | 2.4 MB 00:00 2024-03-19T02:08:19.889 INFO:teuthology.orchestra.run.smithi162.stdout:(8/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-03-19T02:08:20.013 INFO:teuthology.orchestra.run.smithi162.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 204 kB/s | 25 kB 00:00 2024-03-19T02:08:20.332 INFO:teuthology.orchestra.run.smithi162.stdout:(10/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 15 MB/s | 17 MB 00:01 2024-03-19T02:08:20.390 INFO:teuthology.orchestra.run.smithi162.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.7 MB/s | 221 kB 00:00 2024-03-19T02:08:20.440 INFO:teuthology.orchestra.run.smithi162.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 544 kB/s | 27 kB 00:00 2024-03-19T02:08:20.482 INFO:teuthology.orchestra.run.smithi162.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 761 kB/s | 32 kB 00:00 2024-03-19T02:08:20.566 INFO:teuthology.orchestra.run.smithi162.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 8.8 MB/s | 746 kB 00:00 2024-03-19T02:08:20.616 INFO:teuthology.orchestra.run.smithi162.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 3.2 MB/s | 163 kB 00:00 2024-03-19T02:08:20.674 INFO:teuthology.orchestra.run.smithi162.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 2.1 MB/s | 127 kB 00:00 2024-03-19T02:08:20.758 INFO:teuthology.orchestra.run.smithi162.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 5.9 MB/s | 501 kB 00:00 2024-03-19T02:08:20.867 INFO:teuthology.orchestra.run.smithi162.stdout:(18/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 6.3 MB/s | 9.8 MB 00:01 2024-03-19T02:08:20.950 INFO:teuthology.orchestra.run.smithi162.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 544 kB/s | 45 kB 00:00 2024-03-19T02:08:21.034 INFO:teuthology.orchestra.run.smithi162.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.5 MB/s | 131 kB 00:00 2024-03-19T02:08:21.117 INFO:teuthology.orchestra.run.smithi162.stdout:(21/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 14 MB/s | 5.1 MB 00:00 2024-03-19T02:08:21.143 INFO:teuthology.orchestra.run.smithi162.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.5 MB/s | 162 kB 00:00 2024-03-19T02:08:21.168 INFO:teuthology.orchestra.run.smithi162.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 6.2 MB/s | 321 kB 00:00 2024-03-19T02:08:21.210 INFO:teuthology.orchestra.run.smithi162.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 4.3 MB/s | 298 kB 00:00 2024-03-19T02:08:21.235 INFO:teuthology.orchestra.run.smithi162.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.4 MB/s | 99 kB 00:00 2024-03-19T02:08:21.321 INFO:teuthology.orchestra.run.smithi162.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 773 kB/s | 85 kB 00:00 2024-03-19T02:08:21.414 INFO:teuthology.orchestra.run.smithi162.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.9 MB/s | 170 kB 00:00 2024-03-19T02:08:21.470 INFO:teuthology.orchestra.run.smithi162.stdout:(28/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 13 MB/s | 3.1 MB 00:00 2024-03-19T02:08:21.495 INFO:teuthology.orchestra.run.smithi162.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 406 kB/s | 34 kB 00:00 2024-03-19T02:08:21.520 INFO:teuthology.orchestra.run.smithi162.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.7 MB/s | 139 kB 00:00 2024-03-19T02:08:21.721 INFO:teuthology.orchestra.run.smithi162.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 8.0 MB/s | 1.8 MB 00:00 2024-03-19T02:08:21.812 INFO:teuthology.orchestra.run.smithi162.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.7 MB/s | 248 kB 00:00 2024-03-19T02:08:21.896 INFO:teuthology.orchestra.run.smithi162.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 593 kB/s | 49 kB 00:00 2024-03-19T02:08:21.988 INFO:teuthology.orchestra.run.smithi162.stdout:(34/174): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-03-19T02:08:22.013 INFO:teuthology.orchestra.run.smithi162.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 141 kB/s | 16 kB 00:00 2024-03-19T02:08:22.039 INFO:teuthology.orchestra.run.smithi162.stdout:(36/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 5.1 MB/s | 264 kB 00:00 2024-03-19T02:08:22.081 INFO:teuthology.orchestra.run.smithi162.stdout:(37/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 437 kB/s | 18 kB 00:00 2024-03-19T02:08:22.114 INFO:teuthology.orchestra.run.smithi162.stdout:(38/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 7.3 MB/s | 749 kB 00:00 2024-03-19T02:08:22.265 INFO:teuthology.orchestra.run.smithi162.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-19T02:08:22.315 INFO:teuthology.orchestra.run.smithi162.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.2 MB/s | 920 kB 00:00 2024-03-19T02:08:22.340 INFO:teuthology.orchestra.run.smithi162.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-19T02:08:22.365 INFO:teuthology.orchestra.run.smithi162.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-19T02:08:22.382 INFO:teuthology.orchestra.run.smithi162.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 768 kB/s | 32 kB 00:00 2024-03-19T02:08:22.474 INFO:teuthology.orchestra.run.smithi162.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-19T02:08:22.491 INFO:teuthology.orchestra.run.smithi162.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-03-19T02:08:22.508 INFO:teuthology.orchestra.run.smithi162.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 300 kB/s | 10 kB 00:00 2024-03-19T02:08:22.533 INFO:teuthology.orchestra.run.smithi162.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-19T02:08:22.550 INFO:teuthology.orchestra.run.smithi162.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-19T02:08:22.576 INFO:teuthology.orchestra.run.smithi162.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-19T02:08:22.610 INFO:teuthology.orchestra.run.smithi162.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-19T02:08:22.626 INFO:teuthology.orchestra.run.smithi162.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-19T02:08:22.651 INFO:teuthology.orchestra.run.smithi162.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-19T02:08:22.686 INFO:teuthology.orchestra.run.smithi162.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-19T02:08:22.711 INFO:teuthology.orchestra.run.smithi162.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-19T02:08:22.728 INFO:teuthology.orchestra.run.smithi162.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-03-19T02:08:22.845 INFO:teuthology.orchestra.run.smithi162.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-19T02:08:22.878 INFO:teuthology.orchestra.run.smithi162.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-19T02:08:22.903 INFO:teuthology.orchestra.run.smithi162.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-19T02:08:22.920 INFO:teuthology.orchestra.run.smithi162.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-19T02:08:23.054 INFO:teuthology.orchestra.run.smithi162.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-19T02:08:23.088 INFO:teuthology.orchestra.run.smithi162.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-19T02:08:23.104 INFO:teuthology.orchestra.run.smithi162.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-19T02:08:23.230 INFO:teuthology.orchestra.run.smithi162.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-19T02:08:23.255 INFO:teuthology.orchestra.run.smithi162.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 163 kB/s | 24 kB 00:00 2024-03-19T02:08:23.280 INFO:teuthology.orchestra.run.smithi162.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-19T02:08:23.306 INFO:teuthology.orchestra.run.smithi162.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-19T02:08:23.331 INFO:teuthology.orchestra.run.smithi162.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-19T02:08:23.448 INFO:teuthology.orchestra.run.smithi162.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-19T02:08:23.473 INFO:teuthology.orchestra.run.smithi162.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-19T02:08:23.499 INFO:teuthology.orchestra.run.smithi162.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-19T02:08:23.524 INFO:teuthology.orchestra.run.smithi162.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-19T02:08:23.541 INFO:teuthology.orchestra.run.smithi162.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-19T02:08:23.566 INFO:teuthology.orchestra.run.smithi162.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-19T02:08:23.591 INFO:teuthology.orchestra.run.smithi162.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T02:08:23.616 INFO:teuthology.orchestra.run.smithi162.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-19T02:08:23.657 INFO:teuthology.orchestra.run.smithi162.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 732 kB/s | 47 kB 00:00 2024-03-19T02:08:23.675 INFO:teuthology.orchestra.run.smithi162.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 730 kB/s | 43 kB 00:00 2024-03-19T02:08:23.851 INFO:teuthology.orchestra.run.smithi162.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 221 kB/s | 39 kB 00:00 2024-03-19T02:08:24.068 INFO:teuthology.orchestra.run.smithi162.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-19T02:08:24.084 INFO:teuthology.orchestra.run.smithi162.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 177 kB/s | 41 kB 00:00 2024-03-19T02:08:24.110 INFO:teuthology.orchestra.run.smithi162.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-19T02:08:24.127 INFO:teuthology.orchestra.run.smithi162.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-19T02:08:24.144 INFO:teuthology.orchestra.run.smithi162.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-19T02:08:24.161 INFO:teuthology.orchestra.run.smithi162.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-19T02:08:24.186 INFO:teuthology.orchestra.run.smithi162.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-19T02:08:24.211 INFO:teuthology.orchestra.run.smithi162.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-19T02:08:24.254 INFO:teuthology.orchestra.run.smithi162.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-19T02:08:24.270 INFO:teuthology.orchestra.run.smithi162.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-19T02:08:24.296 INFO:teuthology.orchestra.run.smithi162.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-19T02:08:24.321 INFO:teuthology.orchestra.run.smithi162.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-19T02:08:27.638 INFO:teuthology.orchestra.run.smithi162.stdout:(91/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 6.2 MB/s | 48 MB 00:07 2024-03-19T02:10:36.439 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] jq-1.6-16.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/jq-1.6-16.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:10:36.439 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:10:36.439 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:10:37.052 INFO:teuthology.orchestra.run.smithi162.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:09 2024-03-19T02:10:37.161 INFO:teuthology.orchestra.run.smithi162.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.4 kB/s | 187 kB 02:12 2024-03-19T02:10:37.303 INFO:teuthology.orchestra.run.smithi162.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 499 kB/s | 125 kB 00:00 2024-03-19T02:10:37.520 INFO:teuthology.orchestra.run.smithi162.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.0 kB/s | 803 kB 02:13 2024-03-19T02:12:47.495 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] elfutils-libelf-devel-0.190-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.190-2.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:12:47.495 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] libuuid-devel-2.37.4-18.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libuuid-devel-2.37.4-18.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:12:47.495 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] openssl-devel-3.0.7-27.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openssl-devel-3.0.7-27.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:12:47.775 INFO:teuthology.orchestra.run.smithi162.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 199 B/s | 25 kB 02:10 2024-03-19T02:12:47.792 INFO:teuthology.orchestra.run.smithi162.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 168 B/s | 21 kB 02:10 2024-03-19T02:12:48.142 INFO:teuthology.orchestra.run.smithi162.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 669 kB/s | 245 kB 00:00 2024-03-19T02:12:48.168 INFO:teuthology.orchestra.run.smithi162.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 660 kB/s | 248 kB 00:00 2024-03-19T02:12:48.285 INFO:teuthology.orchestra.run.smithi162.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 299 kB/s | 35 kB 00:00 2024-03-19T02:12:48.343 INFO:teuthology.orchestra.run.smithi162.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-19T02:12:48.394 INFO:teuthology.orchestra.run.smithi162.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-19T02:12:48.461 INFO:teuthology.orchestra.run.smithi162.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 802 kB/s | 94 kB 00:00 2024-03-19T02:12:50.977 INFO:teuthology.orchestra.run.smithi162.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 32 kB/s | 4.1 MB 02:13 2024-03-19T02:12:51.269 INFO:teuthology.orchestra.run.smithi162.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-19T02:12:51.436 INFO:teuthology.orchestra.run.smithi162.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.8 MB/s | 309 kB 00:00 2024-03-19T02:12:51.561 INFO:teuthology.orchestra.run.smithi162.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-19T02:12:52.128 INFO:teuthology.orchestra.run.smithi162.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-19T02:12:52.237 INFO:teuthology.orchestra.run.smithi162.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-19T02:12:52.378 INFO:teuthology.orchestra.run.smithi162.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 343 kB/s | 49 kB 00:00 2024-03-19T02:12:52.520 INFO:teuthology.orchestra.run.smithi162.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 476 kB/s | 67 kB 00:00 2024-03-19T02:12:52.687 INFO:teuthology.orchestra.run.smithi162.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 907 kB/s | 151 kB 00:00 2024-03-19T02:12:52.929 INFO:teuthology.orchestra.run.smithi162.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-19T02:12:53.129 INFO:teuthology.orchestra.run.smithi162.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-19T02:12:53.263 INFO:teuthology.orchestra.run.smithi162.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 326 kB/s | 43 kB 00:00 2024-03-19T02:12:53.388 INFO:teuthology.orchestra.run.smithi162.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-19T02:12:53.505 INFO:teuthology.orchestra.run.smithi162.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-03-19T02:12:53.638 INFO:teuthology.orchestra.run.smithi162.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-19T02:12:53.855 INFO:teuthology.orchestra.run.smithi162.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 1.6 MB/s | 358 kB 00:00 2024-03-19T02:12:54.006 INFO:teuthology.orchestra.run.smithi162.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-03-19T02:12:54.114 INFO:teuthology.orchestra.run.smithi162.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 327 kB/s | 35 kB 00:00 2024-03-19T02:12:54.248 INFO:teuthology.orchestra.run.smithi162.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-19T02:12:54.523 INFO:teuthology.orchestra.run.smithi162.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-03-19T02:12:54.648 INFO:teuthology.orchestra.run.smithi162.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-19T02:12:54.749 INFO:teuthology.orchestra.run.smithi162.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-19T02:12:54.849 INFO:teuthology.orchestra.run.smithi162.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-03-19T02:12:54.949 INFO:teuthology.orchestra.run.smithi162.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-03-19T02:12:55.050 INFO:teuthology.orchestra.run.smithi162.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-19T02:12:55.141 INFO:teuthology.orchestra.run.smithi162.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-03-19T02:12:55.325 INFO:teuthology.orchestra.run.smithi162.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-03-19T02:12:55.458 INFO:teuthology.orchestra.run.smithi162.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 348 kB/s | 46 kB 00:00 2024-03-19T02:12:55.585 INFO:teuthology.orchestra.run.smithi162.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 631 kB/s | 79 kB 00:00 2024-03-19T02:12:55.767 INFO:teuthology.orchestra.run.smithi162.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 315 kB/s | 58 kB 00:00 2024-03-19T02:12:55.918 INFO:teuthology.orchestra.run.smithi162.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-19T02:12:56.026 INFO:teuthology.orchestra.run.smithi162.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-19T02:12:56.160 INFO:teuthology.orchestra.run.smithi162.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 676 kB/s | 90 kB 00:00 2024-03-19T02:12:56.277 INFO:teuthology.orchestra.run.smithi162.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 242 kB/s | 28 kB 00:00 2024-03-19T02:12:56.394 INFO:teuthology.orchestra.run.smithi162.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-03-19T02:12:56.527 INFO:teuthology.orchestra.run.smithi162.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-03-19T02:12:56.644 INFO:teuthology.orchestra.run.smithi162.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-03-19T02:12:56.769 INFO:teuthology.orchestra.run.smithi162.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-03-19T02:12:56.911 INFO:teuthology.orchestra.run.smithi162.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 360 kB/s | 51 kB 00:00 2024-03-19T02:12:57.045 INFO:teuthology.orchestra.run.smithi162.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-19T02:12:57.170 INFO:teuthology.orchestra.run.smithi162.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 718 kB/s | 90 kB 00:00 2024-03-19T02:12:57.354 INFO:teuthology.orchestra.run.smithi162.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2024-03-19T02:12:57.470 INFO:teuthology.orchestra.run.smithi162.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 172 kB/s | 20 kB 00:00 2024-03-19T02:12:57.621 INFO:teuthology.orchestra.run.smithi162.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-19T02:12:57.946 INFO:teuthology.orchestra.run.smithi162.stdout:(148/174): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-03-19T02:12:58.529 INFO:teuthology.orchestra.run.smithi162.stdout:(149/174): librados2-19.0.0-2206.gb6ae2491.el9. 5.8 MB/s | 3.4 MB 00:00 2024-03-19T02:12:58.729 INFO:teuthology.orchestra.run.smithi162.stdout:(150/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 16 MB/s | 3.1 MB 00:00 2024-03-19T02:12:58.855 INFO:teuthology.orchestra.run.smithi162.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-19T02:13:00.971 INFO:teuthology.orchestra.run.smithi162.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 1.1 MB/s | 2.3 MB 00:02 2024-03-19T02:13:02.504 INFO:teuthology.orchestra.run.smithi162.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.2 MB/s | 1.8 MB 00:01 2024-03-19T02:13:02.646 INFO:teuthology.orchestra.run.smithi162.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 297 kB/s | 42 kB 00:00 2024-03-19T02:13:03.038 INFO:teuthology.orchestra.run.smithi162.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 634 kB/s | 248 kB 00:00 2024-03-19T02:13:03.180 INFO:teuthology.orchestra.run.smithi162.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 504 kB/s | 71 kB 00:00 2024-03-19T02:13:03.339 INFO:teuthology.orchestra.run.smithi162.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 230 kB/s | 36 kB 00:00 2024-03-19T02:13:03.614 INFO:teuthology.orchestra.run.smithi162.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 699 kB/s | 192 kB 00:00 2024-03-19T02:13:03.890 INFO:teuthology.orchestra.run.smithi162.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 929 kB/s | 256 kB 00:00 2024-03-19T02:13:03.998 INFO:teuthology.orchestra.run.smithi162.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 213 kB/s | 23 kB 00:00 2024-03-19T02:13:04.107 INFO:teuthology.orchestra.run.smithi162.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-03-19T02:13:04.298 INFO:teuthology.orchestra.run.smithi162.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 809 kB/s | 155 kB 00:00 2024-03-19T02:13:04.497 INFO:teuthology.orchestra.run.smithi162.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 684 kB/s | 136 kB 00:00 2024-03-19T02:13:04.606 INFO:teuthology.orchestra.run.smithi162.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 590 kB/s | 64 kB 00:00 2024-03-19T02:13:04.723 INFO:teuthology.orchestra.run.smithi162.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 246 kB/s | 29 kB 00:00 2024-03-19T02:13:06.064 INFO:teuthology.orchestra.run.smithi162.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 924 kB/s | 1.2 MB 00:01 2024-03-19T02:13:07.990 INFO:teuthology.orchestra.run.smithi162.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.1 MB/s | 2.2 MB 00:01 2024-03-19T02:13:09.898 INFO:teuthology.orchestra.run.smithi162.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 2.3 MB 00:01 2024-03-19T02:13:10.264 INFO:teuthology.orchestra.run.smithi162.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 1.2 MB/s | 465 kB 00:00 2024-03-19T02:13:10.373 INFO:teuthology.orchestra.run.smithi162.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-19T02:13:11.248 INFO:teuthology.orchestra.run.smithi162.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 1.1 MB 00:00 2024-03-19T02:13:11.357 INFO:teuthology.orchestra.run.smithi162.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 686 kB/s | 74 kB 00:00 2024-03-19T02:14:58.933 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:58.933 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:59.226 INFO:teuthology.orchestra.run.smithi162.stdout:(173/174): lua-devel-5.4.4-4.el9.x86_64.rpm 174 B/s | 22 kB 02:10 2024-03-19T02:14:59.976 INFO:teuthology.orchestra.run.smithi162.stdout:(174/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.6 kB/s | 863 kB 02:11 2024-03-19T02:14:59.977 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:14:59.977 INFO:teuthology.orchestra.run.smithi162.stdout:Total 571 kB/s | 224 MB 06:42 2024-03-19T02:15:00.432 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-19T02:15:00.708 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-19T02:15:00.708 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-19T02:15:02.785 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-19T02:15:02.785 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-19T02:15:09.730 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-19T02:15:09.829 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:15:09.855 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:15:09.902 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:15:09.911 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:15:09.951 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:15:10.006 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:15:10.038 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:15:10.072 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:15:10.121 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:15:10.443 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:15:10.498 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:15:10.603 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:15:10.633 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:15:10.707 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:15:10.741 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:15:10.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:15:10.965 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:15:11.012 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:15:11.039 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:15:11.142 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:15:11.165 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:15:11.231 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:15:11.261 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:15:11.284 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:15:11.299 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:15:11.459 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:15:11.493 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:15:11.715 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:15:11.882 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:15:11.942 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:15:12.041 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:15:12.093 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:15:12.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:15:12.406 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:15:12.510 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:15:12.541 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:15:12.567 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:15:12.611 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:15:12.685 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:15:12.739 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:15:12.777 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:15:12.842 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:15:12.876 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:15:12.920 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:15:12.953 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:15:12.980 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:15:13.050 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:15:13.085 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:15:13.120 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:15:13.149 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:15:13.219 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:15:13.272 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:15:13.346 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:15:13.480 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:15:13.541 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:15:14.062 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:15:14.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:15:14.222 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:15:14.265 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:15:14.292 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:15:14.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:15:14.379 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:15:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:15:14.447 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:15:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:15:14.880 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:15:14.968 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:15:15.012 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:15:15.146 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:15:15.199 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:15:15.254 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:15:15.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:15:15.386 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:15:15.428 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:15:15.474 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:15:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:15:15.554 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:15:15.596 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:15:15.729 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:15:15.776 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:15:15.807 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:15:15.849 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:15:15.944 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:15:15.982 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:15:16.023 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:15:16.048 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:15:16.073 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:15:16.153 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:15:16.275 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:15:16.374 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:15:16.424 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:15:16.473 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:15:16.538 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:15:16.719 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:15:16.761 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:15:16.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:15:16.935 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:15:16.966 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:15:16.998 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:15:17.034 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:15:17.068 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:15:17.108 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:15:17.139 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:15:17.169 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:15:17.245 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:15:17.288 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:15:17.322 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:15:17.363 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:15:17.398 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:15:17.495 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:15:17.585 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:15:17.627 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:15:17.667 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:15:17.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:15:17.812 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:15:17.870 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:15:18.599 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:15:18.650 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:15:18.684 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:15:18.740 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:15:18.820 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:15:18.868 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:15:19.046 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:15:19.074 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:15:19.153 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:15:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:15:19.906 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:15:21.151 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:15:21.235 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:15:21.266 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:15:21.297 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:15:21.325 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:15:21.637 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:15:21.672 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:15:21.754 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:15:21.785 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:15:21.821 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:15:22.283 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:15:22.308 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:15:22.379 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:15:22.402 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:15:24.696 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:15:24.741 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:15:25.288 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:15:25.336 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:15:25.428 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:15:25.429 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:15:25.429 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:15:25.429 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:25.462 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:15:37.207 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:15:37.207 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-19T02:15:37.208 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:37.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:15:37.419 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:15:37.419 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:37.419 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:15:37.419 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:38.907 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:15:38.932 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:15:38.932 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:38.932 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:15:38.933 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:15:38.933 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:15:38.933 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:39.158 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:15:39.179 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:15:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:15:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:15:39.180 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:15:39.180 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:39.625 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:15:39.646 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:15:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:15:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:15:39.647 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:15:39.647 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:39.735 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:15:40.111 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:15:40.134 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:15:40.166 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:15:40.227 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:15:40.257 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:15:40.491 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:15:40.525 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:15:41.495 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:15:41.534 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:15:41.633 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:15:41.796 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:15:41.819 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:15:41.842 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:41.887 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:15:41.897 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:15:41.950 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:15:42.823 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:15:42.856 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:15:42.878 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:15:42.878 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:42.879 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:15:42.879 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:15:42.879 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:15:42.879 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:42.923 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:15:42.945 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:15:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:42.945 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:15:42.945 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:42.999 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:15:43.021 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:43.333 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:15:43.355 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:47.821 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:15:47.862 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:15:47.893 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:15:47.920 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:15:48.104 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:15:48.228 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:15:48.269 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:15:48.325 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:15:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:15:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:15:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:15:48.527 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:15:48.557 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:15:48.617 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:15:48.682 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:15:48.722 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:15:49.358 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:15:49.396 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:15:49.416 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:15:50.244 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:15:50.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:15:50.367 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:15:50.464 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:15:50.497 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:15:50.522 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:15:50.547 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:15:50.574 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:15:50.574 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:15:50.662 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:15:50.700 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:15:50.700 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:15:50.793 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:15:50.819 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:15:50.852 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:15:50.878 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:15:50.902 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:15:50.937 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:15:50.993 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:15:51.025 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:15:51.050 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:15:51.075 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:15:51.100 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:15:51.126 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:15:51.132 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:15:51.132 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:15:51.320 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:15:51.321 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:15:51.403 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:15:51.430 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:15:51.463 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:15:51.490 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:15:51.515 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:15:51.582 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:15:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:15:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:15:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:15:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:15:56.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:15:56.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:15:56.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:15:56.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:15:56.494 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:15:56.495 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:15:56.496 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:15:56.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:15:56.498 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:15:56.499 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:15:56.500 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:15:56.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:15:56.502 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:15:56.503 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:15:56.504 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:15:56.505 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:15:56.506 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:15:56.506 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:15:56.506 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:15:59.497 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.498 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:15:59.499 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.500 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:15:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:15:59.503 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:15:59.504 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:15:59.505 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:15:59.506 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:15:59.507 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:15:59.508 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:15:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:15:59.510 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-19T02:15:59.973 DEBUG:teuthology.parallel:result is None 2024-03-19T02:15:59.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:00.194 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:16:00.220 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.220 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.221 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:16:00.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:00.443 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:16:00.469 INFO:teuthology.orchestra.run.smithi154.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.469 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.469 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:16:00.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:00.716 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:16:00.741 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.741 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:16:00.741 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:16:00.743 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T02:16:00.743 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:16:00.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:16:00.777 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:16:00.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:16:00.812 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:16:00.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:16:00.847 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T02:16:00.847 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:16:00.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:16:00.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:16:00.946 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:16:00.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:16:00.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:16:01.046 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:16:01.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:16:01.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:16:01.149 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T02:16:01.150 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:16:01.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:16:01.181 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:16:01.250 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:16:01.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:16:01.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:16:01.351 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:16:01.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:16:01.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:16:01.453 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T02:16:01.453 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:16:01.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:16:01.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:16:01.548 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:16:01.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:16:01.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:16:01.643 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:16:01.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:16:01.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:16:01.743 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-19T02:16:01.837 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'subvols': {'create': 2}} 2024-03-19T02:16:01.837 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:01.837 INFO:tasks.cephadm:Cluster fsid is a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:16:01.838 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-19T02:16:01.838 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.b': '172.21.15.154', 'mon.c': '172.21.15.162'} 2024-03-19T02:16:01.838 INFO:tasks.cephadm:First mon is mon.a on smithi116 2024-03-19T02:16:01.838 INFO:tasks.cephadm:First mgr is x 2024-03-19T02:16:01.838 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-19T02:16:01.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-19T02:16:01.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-03-19T02:16:01.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-03-19T02:16:01.924 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-19T02:16:01.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:02.148 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77736/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2206-gb6ae2491', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2206.gb6ae2491'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:42:27.872142', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240318.181317', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-19T02:16:02.310 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-batrick-testing-20240318.181317, sha1 b6ae24918b03b1b8a19c9263a50655bba80bb810 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:16:02.311 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm 2024-03-19T02:16:02.312 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:02.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:16 /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:02.794 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.284 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:16 /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.284 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.739 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:16 /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.740 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.759 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.777 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:16:03.827 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 on all hosts... 2024-03-19T02:16:03.828 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:16:03.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:16:03.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:16:04.678 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:16:04.745 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:16:04.764 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-03-19T02:16:43.414 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-19T02:16:57.304 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T02:16:57.304 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:16:57.304 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:16:57.304 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-03-19T02:16:57.304 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:16:57.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-03-19T02:16:57.305 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T02:16:57.699 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:16:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:16:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:16:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-19T02:16:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:16:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:16:57.700 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:16:57.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-19T02:16:57.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-03-19T02:16:57.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-03-19T02:16:57.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-19T02:16:57.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-03-19T02:16:57.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-03-19T02:16:57.923 INFO:tasks.cephadm:Writing seed config... 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-19T02:16:57.924 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-19T02:16:57.925 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-19T02:16:57.926 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-19T02:16:57.927 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-19T02:16:57.927 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:16:57.927 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-19T02:16:57.942 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a1da9bbc-e596-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-19T02:16:57.942 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service 2024-03-19T02:16:57.985 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x.service 2024-03-19T02:16:58.028 INFO:tasks.cephadm:Bootstrapping... 2024-03-19T02:16:58.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 -v bootstrap --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:16:58.218 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:16:58.218 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810', '-v', 'bootstrap', '--fsid', 'a1da9bbc-e596-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-19T02:16:58.241 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:16:58.242 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-19T02:16:58.242 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-19T02:16:58.265 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:16:58.265 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-19T02:16:58.265 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-19T02:16:58.273 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T02:16:58.273 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T02:16:58.279 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T02:16:58.279 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:16:58.285 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-19T02:16:58.291 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-19T02:16:58.291 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-19T02:16:58.291 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-19T02:16:58.313 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:16:58.314 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-19T02:16:58.314 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-19T02:16:58.314 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-19T02:16:58.320 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T02:16:58.320 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T02:16:58.325 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T02:16:58.325 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:16:58.331 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 140206776171968 on /run/cephadm/a1da9bbc-e596-11ee-95c9-87774f69a715.lock 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 140206776171968 acquired on /run/cephadm/a1da9bbc-e596-11ee-95c9-87774f69a715.lock 2024-03-19T02:16:58.338 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-19T02:16:58.339 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-19T02:16:58.339 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-19T02:16:58.342 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2024-03-19T02:16:58.342 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-19T02:16:58.344 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-19T02:16:58.344 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-19T02:16:58.344 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-19T02:16:58.346 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-19T02:16:58.346 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-19T02:16:58.347 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-19T02:16:58.348 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-19T02:16:58.348 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-19T02:16:58.349 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:93b5a8d362aeee7e8b1f0bbf18238e1bb256c50573ad5b72aff9e382b1bb3ccb 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T02:16:58.751 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-19T02:16:59.672 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T02:16:59.672 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T02:16:59.673 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-19T02:17:00.552 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-19T02:17:00.553 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-19T02:17:01.442 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCc9fhlxgJlNBAAATOzp6BSplDqn+5xNRs2aQ== 2024-03-19T02:17:02.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCd9fhlXrhNLxAAxY2fne/IvC5RW5PN4RXjdg== 2024-03-19T02:17:03.452 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQCe9fhloPSnMxAAsJNmiUH7n/gnrPyuUtlvjA== 2024-03-19T02:17:03.453 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T02:17:04.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:17:04.360 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-19T02:17:05.256 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.a on 2024-03-19T02:17:05.451 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-19T02:17:05.606 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-19T02:17:05.807 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target. 2024-03-19T02:17:05.808 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target. 2024-03-19T02:17:06.047 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a 2024-03-19T02:17:06.047 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-19T02:17:06.208 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target.wants/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@.service. 2024-03-19T02:17:06.774 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:17:06.774 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T02:17:06.782 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:17:06.782 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:17:06.782 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T02:17:06.782 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T02:17:06.782 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-19T02:17:06.783 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-19T02:17:07.883 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-19T02:17:07.883 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:07.883 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-19T02:17:07.883 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:07.883 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.599268s) 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-19T02:17:07.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.274+0000 7f9b50ac1700 1 Processor -- start 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.274+0000 7f9b50ac1700 1 -- start start 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b50ac1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b50ac1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c108450 con 0x7f9b4c107a60 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b4a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b4a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36474/0 (socket says 172.21.15.116:36474) 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b4a59c700 1 -- 172.21.15.116:0/2656283798 learned_addr learned my addr 172.21.15.116:0/2656283798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:07.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.275+0000 7f9b4a59c700 1 -- 172.21.15.116:0/2656283798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c108c60 con 0x7f9b4c107a60 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b4a59c700 1 --2- 172.21.15.116:0/2656283798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9b400098a0 tx=0x7f9b40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55b42c81f91739f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b4959a700 1 -- 172.21.15.116:0/2656283798 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b40004160 con 0x7f9b4c107a60 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b4959a700 1 -- 172.21.15.116:0/2656283798 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b40004300 con 0x7f9b4c107a60 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b4959a700 1 -- 172.21.15.116:0/2656283798 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b400036a0 con 0x7f9b4c107a60 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 -- 172.21.15.116:0/2656283798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 msgr2=0x7f9b4c107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 --2- 172.21.15.116:0/2656283798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9b400098a0 tx=0x7f9b40009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 -- 172.21.15.116:0/2656283798 shutdown_connections 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 --2- 172.21.15.116:0/2656283798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c107e80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 -- 172.21.15.116:0/2656283798 >> 172.21.15.116:0/2656283798 conn(0x7f9b4c103020 msgr2=0x7f9b4c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 -- 172.21.15.116:0/2656283798 shutdown_connections 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.276+0000 7f9b50ac1700 1 -- 172.21.15.116:0/2656283798 wait complete. 2024-03-19T02:17:07.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b50ac1700 1 Processor -- start 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b50ac1700 1 -- start start 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b50ac1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b50ac1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c109190 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b4a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.277+0000 7f9b4a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36486/0 (socket says 172.21.15.116:36486) 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b4a59c700 1 -- 172.21.15.116:0/3259935831 learned_addr learned my addr 172.21.15.116:0/3259935831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b4a59c700 1 -- 172.21.15.116:0/3259935831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c19b600 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b4a59c700 1 --2- 172.21.15.116:0/3259935831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c1086f0 tx=0x7f9b40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b40003db0 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b40011630 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c19b890 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b40011930 con 0x7f9b4c107a60 2024-03-19T02:17:07.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c19e530 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9b40011ad0 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.278+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9b4001b450 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.279+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b4c063940 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.281+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9b40004260 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.350+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9b4c02dde0 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.350+0000 7f9b3b7fe700 1 -- 172.21.15.116:0/3259935831 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f9b40025480 con 0x7f9b4c107a60 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 msgr2=0x7f9b4c19b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 --2- 172.21.15.116:0/3259935831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c1086f0 tx=0x7f9b40004750 comp rx=0 tx=0).stop 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 shutdown_connections 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 --2- 172.21.15.116:0/3259935831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b4c107a60 0x7f9b4c19b0c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 >> 172.21.15.116:0/3259935831 conn(0x7f9b4c103020 msgr2=0x7f9b4c191dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 shutdown_connections 2024-03-19T02:17:07.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:07.351+0000 7f9b50ac1700 1 -- 172.21.15.116:0/3259935831 wait complete. 2024-03-19T02:17:07.889 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-19T02:17:07.889 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T02:17:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.317+0000 7fb2ad050700 1 Processor -- start 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2ad050700 1 -- start start 2024-03-19T02:17:08.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2ad050700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2ad050700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8108450 con 0x7fb2a8107a60 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47552/0 (socket says 172.21.15.116:47552) 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a6d9d700 1 -- 172.21.15.116:0/1484323353 learned_addr learned my addr 172.21.15.116:0/1484323353 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a6d9d700 1 -- 172.21.15.116:0/1484323353 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a8108c60 con 0x7fb2a8107a60 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a6d9d700 1 --2- 172.21.15.116:0/1484323353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0098a0 tx=0x7fb29c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357882f51fd9f3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.318+0000 7fb2a5d9b700 1 -- 172.21.15.116:0/1484323353 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb29c004160 con 0x7fb2a8107a60 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2a5d9b700 1 -- 172.21.15.116:0/1484323353 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29c004300 con 0x7fb2a8107a60 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2a5d9b700 1 -- 172.21.15.116:0/1484323353 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb29c0036a0 con 0x7fb2a8107a60 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 -- 172.21.15.116:0/1484323353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 msgr2=0x7fb2a8107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 --2- 172.21.15.116:0/1484323353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0098a0 tx=0x7fb29c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 -- 172.21.15.116:0/1484323353 shutdown_connections 2024-03-19T02:17:08.940 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 --2- 172.21.15.116:0/1484323353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a8107e80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 -- 172.21.15.116:0/1484323353 >> 172.21.15.116:0/1484323353 conn(0x7fb2a8103020 msgr2=0x7fb2a8105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 -- 172.21.15.116:0/1484323353 shutdown_connections 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.319+0000 7fb2ad050700 1 -- 172.21.15.116:0/1484323353 wait complete. 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2ad050700 1 Processor -- start 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2ad050700 1 -- start start 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2ad050700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2ad050700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8109190 con 0x7fb2a8107a60 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2a6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2a6d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47554/0 (socket says 172.21.15.116:47554) 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.320+0000 7fb2a6d9d700 1 -- 172.21.15.116:0/4024312195 learned_addr learned my addr 172.21.15.116:0/4024312195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb2a6d9d700 1 -- 172.21.15.116:0/4024312195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a819b5f0 con 0x7fb2a8107a60 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb2a6d9d700 1 --2- 172.21.15.116:0/4024312195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb29c000c00 tx=0x7fb29c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb29c003db0 con 0x7fb2a8107a60 2024-03-19T02:17:08.941 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29c011630 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a819b880 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a819e570 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb29c011950 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb29c011b30 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.321+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb29c01c9c0 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.322+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a8063940 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.324+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb29c011e20 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.391+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fb2a802de20 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.393+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7fb29c004260 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.394+0000 7fb28ffff700 1 -- 172.21.15.116:0/4024312195 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb29c0369d0 con 0x7fb2a8107a60 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 msgr2=0x7fb2a819b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 --2- 172.21.15.116:0/4024312195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb29c000c00 tx=0x7fb29c004750 comp rx=0 tx=0).stop 2024-03-19T02:17:08.942 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 shutdown_connections 2024-03-19T02:17:08.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 --2- 172.21.15.116:0/4024312195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2a8107a60 0x7fb2a819b0b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:08.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 >> 172.21.15.116:0/4024312195 conn(0x7fb2a8103020 msgr2=0x7fb2a8191d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:08.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 shutdown_connections 2024-03-19T02:17:08.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:08.395+0000 7fb2ad050700 1 -- 172.21.15.116:0/4024312195 wait complete. 2024-03-19T02:17:08.943 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-19T02:17:09.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.306+0000 7feefa9e2700 1 Processor -- start 2024-03-19T02:17:09.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.306+0000 7feefa9e2700 1 -- start start 2024-03-19T02:17:09.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.306+0000 7feefa9e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:09.892 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.306+0000 7feefa9e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feef4106740 con 0x7feef4105d50 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47566/0 (socket says 172.21.15.116:47566) 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef3fff700 1 -- 172.21.15.116:0/2688781068 learned_addr learned my addr 172.21.15.116:0/2688781068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef3fff700 1 -- 172.21.15.116:0/2688781068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef4108d80 con 0x7feef4105d50 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef3fff700 1 --2- 172.21.15.116:0/2688781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feee40098a0 tx=0x7feee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18657484f7f4181f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef2ffd700 1 -- 172.21.15.116:0/2688781068 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feee4004160 con 0x7feef4105d50 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef2ffd700 1 -- 172.21.15.116:0/2688781068 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7feee4004300 con 0x7feef4105d50 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.307+0000 7feef2ffd700 1 -- 172.21.15.116:0/2688781068 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feee4011a80 con 0x7feef4105d50 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- 172.21.15.116:0/2688781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 msgr2=0x7feef4106170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 --2- 172.21.15.116:0/2688781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feee40098a0 tx=0x7feee4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- 172.21.15.116:0/2688781068 shutdown_connections 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 --2- 172.21.15.116:0/2688781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef4106170 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- 172.21.15.116:0/2688781068 >> 172.21.15.116:0/2688781068 conn(0x7feef4101680 msgr2=0x7feef4103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- 172.21.15.116:0/2688781068 shutdown_connections 2024-03-19T02:17:09.893 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- 172.21.15.116:0/2688781068 wait complete. 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 Processor -- start 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.308+0000 7feefa9e2700 1 -- start start 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feefa9e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feefa9e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feef410bfb0 con 0x7feef4105d50 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47570/0 (socket says 172.21.15.116:47570) 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef3fff700 1 -- 172.21.15.116:0/3526151678 learned_addr learned my addr 172.21.15.116:0/3526151678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef3fff700 1 -- 172.21.15.116:0/3526151678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef419be20 con 0x7feef4105d50 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef3fff700 1 --2- 172.21.15.116:0/3526151678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feee4000c00 tx=0x7feee4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feee40039f0 con 0x7feef4105d50 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7feee4003b90 con 0x7feef4105d50 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feee401a740 con 0x7feef4105d50 2024-03-19T02:17:09.894 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.309+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef419c0b0 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.310+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef419eda0 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.310+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feee402d3f0 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.310+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feee4019a90 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.310+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef4063940 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.313+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7feee4019d30 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.380+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7feef419f2f0 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feef17fa700 1 -- 172.21.15.116:0/3526151678 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7feee4032030 con 0x7feef4105d50 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 msgr2=0x7feef419b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feefa9e2700 1 --2- 172.21.15.116:0/3526151678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feee4000c00 tx=0x7feee4011b00 comp rx=0 tx=0).stop 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 shutdown_connections 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feefa9e2700 1 --2- 172.21.15.116:0/3526151678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feef4105d50 0x7feef419b8e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.381+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 >> 172.21.15.116:0/3526151678 conn(0x7feef4101680 msgr2=0x7feef41020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.382+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 shutdown_connections 2024-03-19T02:17:09.895 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:09.382+0000 7feefa9e2700 1 -- 172.21.15.116:0/3526151678 wait complete. 2024-03-19T02:17:09.896 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-19T02:17:10.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:09 smithi116 systemd[1]: Stopping Ceph mon.a for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:17:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a[31884]: 2024-03-19T02:17:10.146+0000 7fe022047700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:17:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a[31884]: 2024-03-19T02:17:10.146+0000 7fe022047700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T02:17:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 podman[32133]: 2024-03-19 02:17:10.331208075 +0000 UTC m=+0.285208322 container died cb0d4168829097ec275a671bb90a266b1dd2f664b83592bb2d7fcfee46549e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T02:17:10.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 podman[32133]: 2024-03-19 02:17:10.688665908 +0000 UTC m=+0.642666155 container cleanup cb0d4168829097ec275a671bb90a266b1dd2f664b83592bb2d7fcfee46549e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:17:10.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 bash[32133]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a 2024-03-19T02:17:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:10 smithi116 podman[32147]: 2024-03-19 02:17:10.947727453 +0000 UTC m=+0.611857646 container remove cb0d4168829097ec275a671bb90a266b1dd2f664b83592bb2d7fcfee46549e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:17:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-19T02:17:11.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 systemd[1]: Stopped Ceph mon.a for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:17:11.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 systemd[1]: Starting Ceph mon.a for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 podman[32243]: 2024-03-19 02:17:11.315291542 +0000 UTC m=+0.112300415 container create e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 podman[32243]: 2024-03-19 02:17:11.220556835 +0000 UTC m=+0.017565708 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 podman[32243]: 2024-03-19 02:17:11.534218087 +0000 UTC m=+0.331226970 container init e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 podman[32243]: 2024-03-19 02:17:11.537078579 +0000 UTC m=+0.334087454 container start e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: pidfile_write: ignore empty --pid-file 2024-03-19T02:17:11.565 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: load: jerasure load: lrc 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: RocksDB version: 7.9.2 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Git sha 0 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: DB SUMMARY 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: DB Session ID: 2J8HVWV1DW47QJH51X6P 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: CURRENT file: CURRENT 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-19T02:17:11.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103728 ; 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.error_if_exists: 0 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.create_if_missing: 0 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.paranoid_checks: 1 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.env: 0x556b3aa37c60 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.info_log: 0x556b3cbc5ae0 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.statistics: (nil) 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.use_fsync: 0 2024-03-19T02:17:11.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_log_file_size: 0 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_fallocate: 1 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T02:17:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.use_direct_reads: 0 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.db_log_dir: 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.wal_dir: 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T02:17:11.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T02:17:11.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.write_buffer_manager: 0x556b3cb981e0 2024-03-19T02:17:11.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T02:17:11.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T02:17:11.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T02:17:11.638 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-19T02:17:11.922 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.unordered_write: 0 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.row_cache: None 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.wal_filter: None 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T02:17:11.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.two_write_queues: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.wal_compression: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.atomic_flush: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.log_readahead_size: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T02:17:11.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_background_jobs: 2 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_background_compactions: -1 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_subcompactions: 1 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_open_files: -1 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T02:17:11.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_background_flushes: -1 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Compression algorithms supported: 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kZSTD supported: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kXpressCompression supported: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kZlibCompression supported: 1 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kSnappyCompression supported: 1 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kLZ4Compression supported: 1 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: kBZip2Compression supported: 0 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T02:17:11.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.merge_operator: 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_filter: None 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b3cbc5be0) 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T02:17:11.927 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_type: 0 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_index_type: 0 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_shortening: 1 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: checksum: 4 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: no_block_cache: 0 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache: 0x556b3cbcb090 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_options: 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: capacity : 536870912 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_shard_bits : 4 2024-03-19T02:17:11.928 INFO:journalctl@ceph.mon.a.smithi116.stdout: strict_capacity_limit : 0 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_compressed: (nil) 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: persistent_cache: (nil) 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size: 4096 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size_deviation: 10 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_restart_interval: 16 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_block_restart_interval: 1 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: metadata_block_size: 4096 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: partition_filters: 0 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: use_delta_encoding: 1 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: filter_policy: bloomfilter 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: whole_key_filtering: 1 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: verify_compression: 0 2024-03-19T02:17:11.929 INFO:journalctl@ceph.mon.a.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: format_version: 5 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_index_compression: 1 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_align: 0 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: prepopulate_block_cache: 0 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T02:17:11.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression: NoCompression 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.num_levels: 7 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T02:17:11.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:17:11.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T02:17:11.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T02:17:11.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:17:11.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T02:17:11.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T02:17:11.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T02:17:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.table_properties_collectors: 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.inplace_update_support: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.bloom_locality: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.max_successive_merges: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T02:17:11.936 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.ttl: 2592000 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enable_blob_files: false 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.min_blob_size: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T02:17:11.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8ca1c706-58c8-44fd-9bcf-79a30685206a 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814631566068, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814631567563, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98945, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96924, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13820, "raw_average_key_size": 49, "raw_value_size": 89383, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710814631, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8ca1c706-58c8-44fd-9bcf-79a30685206a", "db_session_id": "2J8HVWV1DW47QJH51X6P", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814631567670, "job": 1, "event": "recovery_finished"} 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-19T02:17:11.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556b3ccbe000 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: DB pointer 0x556b3cca8000 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** DB Stats ** 2024-03-19T02:17:11.939 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:17:11.940 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: L0 2/0 98.46 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: Sum 2/0 98.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:17:11.941 INFO:journalctl@ceph.mon.a.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 105.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:17:11.942 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T02:17:11.943 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative compaction: 0.00 GB write, 22.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval compaction: 0.00 GB write, 22.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T02:17:11.944 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache BinnedLRUCache@0x556b3cbcb090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???) e1 preinit fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).mds e1 new map 2024-03-19T02:17:11.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).mds e1 print_map 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: e1 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: legacy client fscid: -1 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T02:17:11.946 INFO:journalctl@ceph.mon.a.smithi116.stdout: No filesystems configured 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-19T02:17:11.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 bash[32243]: e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 systemd[1]: Started Ceph mon.a for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: monmap epoch 1 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: last_changed 2024-03-19T02:17:03.769369+0000 2024-03-19T02:17:11.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: min_mon_release 19 (squid) 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: election_strategy: 1 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: fsmap 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T02:17:11.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:11 smithi116 ceph-mon[32263]: mgrmap e1: no daemons active 2024-03-19T02:17:12.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.130+0000 7ff680f2f700 1 Processor -- start 2024-03-19T02:17:12.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.130+0000 7ff680f2f700 1 -- start start 2024-03-19T02:17:12.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.130+0000 7ff680f2f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:12.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff680f2f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c106780 con 0x7ff67c105d90 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff67a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff67a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47584/0 (socket says 172.21.15.116:47584) 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff67a59c700 1 -- 172.21.15.116:0/3285790089 learned_addr learned my addr 172.21.15.116:0/3285790089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff67a59c700 1 -- 172.21.15.116:0/3285790089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c108d20 con 0x7ff67c105d90 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.131+0000 7ff67a59c700 1 --2- 172.21.15.116:0/3285790089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff6700098a0 tx=0x7ff670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b92837091a0958 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff67959a700 1 -- 172.21.15.116:0/3285790089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff670004160 con 0x7ff67c105d90 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff67959a700 1 -- 172.21.15.116:0/3285790089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff670004300 con 0x7ff67c105d90 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff67959a700 1 -- 172.21.15.116:0/3285790089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff670011a80 con 0x7ff67c105d90 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 -- 172.21.15.116:0/3285790089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 msgr2=0x7ff67c1061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 --2- 172.21.15.116:0/3285790089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff6700098a0 tx=0x7ff670009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 -- 172.21.15.116:0/3285790089 shutdown_connections 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 --2- 172.21.15.116:0/3285790089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c1061b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 -- 172.21.15.116:0/3285790089 >> 172.21.15.116:0/3285790089 conn(0x7ff67c101720 msgr2=0x7ff67c103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 -- 172.21.15.116:0/3285790089 shutdown_connections 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.132+0000 7ff680f2f700 1 -- 172.21.15.116:0/3285790089 wait complete. 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.133+0000 7ff680f2f700 1 Processor -- start 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.133+0000 7ff680f2f700 1 -- start start 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.133+0000 7ff680f2f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.133+0000 7ff680f2f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10c0d0 con 0x7ff67c105d90 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff67a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff67a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47596/0 (socket says 172.21.15.116:47596) 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff67a59c700 1 -- 172.21.15.116:0/2857141042 learned_addr learned my addr 172.21.15.116:0/2857141042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff67a59c700 1 -- 172.21.15.116:0/2857141042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c19be50 con 0x7ff67c105d90 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff67a59c700 1 --2- 172.21.15.116:0/2857141042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff670006af0 tx=0x7ff670011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6700039e0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff670003b80 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c19c0e0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6700196a0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67c19edd0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff670019ce0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.134+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff670024730 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.135+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67c063940 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.137+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff670043dd0 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.204+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7ff67c106b10 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.206+0000 7ff66b7fe700 1 -- 172.21.15.116:0/2857141042 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7ff670032020 con 0x7ff67c105d90 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 msgr2=0x7ff67c19b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 --2- 172.21.15.116:0/2857141042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff670006af0 tx=0x7ff670011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 shutdown_connections 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 --2- 172.21.15.116:0/2857141042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67c105d90 0x7ff67c19b910 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 >> 172.21.15.116:0/2857141042 conn(0x7ff67c101720 msgr2=0x7ff67c1020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 shutdown_connections 2024-03-19T02:17:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:12.207+0000 7ff680f2f700 1 -- 172.21.15.116:0/2857141042 wait complete. 2024-03-19T02:17:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-19T02:17:12.789 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:17:12.790 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-19T02:17:12.790 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-19T02:17:12.791 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-19T02:17:13.041 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x 2024-03-19T02:17:13.041 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-19T02:17:13.187 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715.target.wants/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@.service. 2024-03-19T02:17:13.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:13 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2857141042' entity='client.admin' 2024-03-19T02:17:13.661 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:13 smithi116 podman[32477]: 2024-03-19 02:17:13.581031157 +0000 UTC m=+0.277101721 container start 32a03d90c7164f57e886350c7a08cbcde2a63939ca49a5d5fc9f5c655133c3ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:17:13.661 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:13 smithi116 bash[32477]: 32a03d90c7164f57e886350c7a08cbcde2a63939ca49a5d5fc9f5c655133c3ef 2024-03-19T02:17:13.674 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:17:13.674 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T02:17:13.681 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:17:13.681 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:17:13.681 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T02:17:13.681 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T02:17:13.689 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:17:13.689 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-19T02:17:13.697 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-19T02:17:13.929 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:13 smithi116 systemd[1]: Started Ceph mgr.x for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:17:14.202 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:14.135+0000 7efd569ed200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:17:14.492 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1242407109' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:17:14.492 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:14.231+0000 7efd569ed200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:17:14.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-19T02:17:14.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:17:14.708 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:17:14.709 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:17:14.710 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:17:06.754009+0000", 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:14.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.121+0000 7f4d35ca8700 1 Processor -- start 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.121+0000 7f4d35ca8700 1 -- start start 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.121+0000 7f4d35ca8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.121+0000 7f4d35ca8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3007bde0 con 0x7f4d301063b0 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.121+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47620/0 (socket says 172.21.15.116:47620) 2024-03-19T02:17:14.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2f7fe700 1 -- 172.21.15.116:0/3034520541 learned_addr learned my addr 172.21.15.116:0/3034520541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2f7fe700 1 -- 172.21.15.116:0/3034520541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d30106d10 con 0x7f4d301063b0 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2f7fe700 1 --2- 172.21.15.116:0/3034520541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4d200098a0 tx=0x7f4d20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fd7219f8a8acdc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2e7fc700 1 -- 172.21.15.116:0/3034520541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d20004160 con 0x7f4d301063b0 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2e7fc700 1 -- 172.21.15.116:0/3034520541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4d20004300 con 0x7f4d301063b0 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.122+0000 7f4d2e7fc700 1 -- 172.21.15.116:0/3034520541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d20011a80 con 0x7f4d301063b0 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 -- 172.21.15.116:0/3034520541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 msgr2=0x7f4d301067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 --2- 172.21.15.116:0/3034520541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4d200098a0 tx=0x7f4d20009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 -- 172.21.15.116:0/3034520541 shutdown_connections 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 --2- 172.21.15.116:0/3034520541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d301067d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:14.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 -- 172.21.15.116:0/3034520541 >> 172.21.15.116:0/3034520541 conn(0x7f4d30101fb0 msgr2=0x7f4d301043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 -- 172.21.15.116:0/3034520541 shutdown_connections 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 -- 172.21.15.116:0/3034520541 wait complete. 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.123+0000 7f4d35ca8700 1 Processor -- start 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d35ca8700 1 -- start start 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d35ca8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d35ca8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30107d40 con 0x7f4d301063b0 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47628/0 (socket says 172.21.15.116:47628) 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d2f7fe700 1 -- 172.21.15.116:0/1242407109 learned_addr learned my addr 172.21.15.116:0/1242407109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:14.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d2f7fe700 1 -- 172.21.15.116:0/1242407109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3019dff0 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.124+0000 7f4d2f7fe700 1 --2- 172.21.15.116:0/1242407109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d20011a50 tx=0x7f4d20011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d20003a80 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3019e280 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4d20003c20 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d2001a650 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d301a0f70 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4d2001ad00 con 0x7f4d301063b0 2024-03-19T02:17:14.715 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.125+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d20019af0 con 0x7f4d301063b0 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.126+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d30063940 con 0x7f4d301063b0 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.130+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4d20019cf0 con 0x7f4d301063b0 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.202+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4d3002dea0 con 0x7f4d301063b0 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.202+0000 7f4d2cff9700 1 -- 172.21.15.116:0/1242407109 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4d20024ad0 con 0x7f4d301063b0 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 msgr2=0x7f4d3019dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 --2- 172.21.15.116:0/1242407109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d20011a50 tx=0x7f4d20011b30 comp rx=0 tx=0).stop 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 shutdown_connections 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 --2- 172.21.15.116:0/1242407109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d301063b0 0x7f4d3019dab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:14.716 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 >> 172.21.15.116:0/1242407109 conn(0x7f4d30101fb0 msgr2=0x7f4d30194770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:14.717 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 shutdown_connections 2024-03-19T02:17:14.717 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:14.203+0000 7f4d35ca8700 1 -- 172.21.15.116:0/1242407109 wait complete. 2024-03-19T02:17:14.717 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-19T02:17:15.192 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:14.872+0000 7efd569ed200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:17:15.192 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:14.962+0000 7efd569ed200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:17:15.192 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.098+0000 7efd569ed200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:17:15.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.192+0000 7efd569ed200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:17:15.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.272+0000 7efd569ed200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:17:15.812 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.530+0000 7efd569ed200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:17:15.812 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.636+0000 7efd569ed200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:17:15.812 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.719+0000 7efd569ed200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:17:15.812 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.811+0000 7efd569ed200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:17:16.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:15.982+0000 7efd569ed200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:17:17.079 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:16 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:16.946+0000 7efd569ed200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:17:17.079 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:17.079+0000 7efd569ed200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:17:17.428 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1092949150' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:17:17.715 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:17.428+0000 7efd569ed200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:17:17.715 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:17.521+0000 7efd569ed200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:17:17.720 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:17.721 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:17:17.722 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:17:17.730 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:17:17.730 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:17:17.730 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:17:17.730 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:17:17.730 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:17.731 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:17:17.732 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:17:06.754009+0000", 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6167ef1700 1 Processor -- start 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6167ef1700 1 -- start start 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6167ef1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6167ef1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f616007be00 con 0x7f6160106390 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6165c8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6165c8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47634/0 (socket says 172.21.15.116:47634) 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.195+0000 7f6165c8d700 1 -- 172.21.15.116:0/4214704655 learned_addr learned my addr 172.21.15.116:0/4214704655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6165c8d700 1 -- 172.21.15.116:0/4214704655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6160106cf0 con 0x7f6160106390 2024-03-19T02:17:17.733 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6165c8d700 1 --2- 172.21.15.116:0/4214704655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f615c0098a0 tx=0x7f615c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eecb897f2e47b7e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6164c8b700 1 -- 172.21.15.116:0/4214704655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f615c004160 con 0x7f6160106390 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6164c8b700 1 -- 172.21.15.116:0/4214704655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f615c004300 con 0x7f6160106390 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6164c8b700 1 -- 172.21.15.116:0/4214704655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f615c011a80 con 0x7f6160106390 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6167ef1700 1 -- 172.21.15.116:0/4214704655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 msgr2=0x7f61601067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.196+0000 7f6167ef1700 1 --2- 172.21.15.116:0/4214704655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f615c0098a0 tx=0x7f615c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 -- 172.21.15.116:0/4214704655 shutdown_connections 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 --2- 172.21.15.116:0/4214704655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f61601067b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 -- 172.21.15.116:0/4214704655 >> 172.21.15.116:0/4214704655 conn(0x7f6160101f90 msgr2=0x7f61601043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 -- 172.21.15.116:0/4214704655 shutdown_connections 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 -- 172.21.15.116:0/4214704655 wait complete. 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 Processor -- start 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.197+0000 7f6167ef1700 1 -- start start 2024-03-19T02:17:17.734 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6167ef1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6167ef1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6160107d20 con 0x7f6160106390 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6165c8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6165c8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47638/0 (socket says 172.21.15.116:47638) 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6165c8d700 1 -- 172.21.15.116:0/1092949150 learned_addr learned my addr 172.21.15.116:0/1092949150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6165c8d700 1 -- 172.21.15.116:0/1092949150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616019e0c0 con 0x7f6160106390 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6165c8d700 1 --2- 172.21.15.116:0/1092949150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f615c006af0 tx=0x7f615c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f615c0039e0 con 0x7f6160106390 2024-03-19T02:17:17.735 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f615c003b80 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616019e350 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f615c0196a0 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.198+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616019e7b0 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.199+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f615c02d440 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.199+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f615c024770 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.200+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6160063940 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.202+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f615c043dd0 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.274+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f616002deb0 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.275+0000 7f6156ffd700 1 -- 172.21.15.116:0/1092949150 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f615c032020 con 0x7f6160106390 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 msgr2=0x7f616019db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 --2- 172.21.15.116:0/1092949150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f615c006af0 tx=0x7f615c011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 shutdown_connections 2024-03-19T02:17:17.736 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 --2- 172.21.15.116:0/1092949150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6160106390 0x7f616019db80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:17.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 >> 172.21.15.116:0/1092949150 conn(0x7f6160101f90 msgr2=0x7f6160191ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:17.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 shutdown_connections 2024-03-19T02:17:17.737 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:17.276+0000 7f6167ef1700 1 -- 172.21.15.116:0/1092949150 wait complete. 2024-03-19T02:17:17.737 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (2/15)... 2024-03-19T02:17:18.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:17.723+0000 7efd569ed200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:17:18.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:17 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:17.911+0000 7efd569ed200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:17:18.450 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:18 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:18.193+0000 7efd569ed200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:17:18.450 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:18 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:18.449+0000 7efd569ed200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:17:18.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:18 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:18.684+0000 7efd569ed200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:17:19.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:19 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:19.477+0000 7efd569ed200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:17:20.156 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:20 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:20.053+0000 7efd569ed200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:17:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:20 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3952161473' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:17:20.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:20 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:20.156+0000 7efd569ed200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:17:20.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:20 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:20.379+0000 7efd569ed200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.834 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:20.835 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:17:20.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:17:20.836 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:17:20.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:17:20.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:20.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:17:06.754009+0000", 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad9ee5700 1 Processor -- start 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad9ee5700 1 -- start start 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad9ee5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad9ee5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4108970 con 0x7f2ad4107f80 2024-03-19T02:17:20.848 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47458/0 (socket says 172.21.15.116:47458) 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.156+0000 7f2ad37fe700 1 -- 172.21.15.116:0/206311276 learned_addr learned my addr 172.21.15.116:0/206311276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.157+0000 7f2ad37fe700 1 -- 172.21.15.116:0/206311276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad4106b90 con 0x7f2ad4107f80 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.157+0000 7f2ad37fe700 1 --2- 172.21.15.116:0/206311276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80098a0 tx=0x7f2ac8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26908cd81ddbe94b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.157+0000 7f2ad27fc700 1 -- 172.21.15.116:0/206311276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ac8004160 con 0x7f2ad4107f80 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.157+0000 7f2ad27fc700 1 -- 172.21.15.116:0/206311276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2ac8004300 con 0x7f2ad4107f80 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.157+0000 7f2ad27fc700 1 -- 172.21.15.116:0/206311276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ac8011a80 con 0x7f2ad4107f80 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/206311276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 msgr2=0x7f2ad41083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 --2- 172.21.15.116:0/206311276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80098a0 tx=0x7f2ac8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/206311276 shutdown_connections 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 --2- 172.21.15.116:0/206311276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad41083a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/206311276 >> 172.21.15.116:0/206311276 conn(0x7f2ad4101720 msgr2=0x7f2ad4103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:20.849 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/206311276 shutdown_connections 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/206311276 wait complete. 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.158+0000 7f2ad9ee5700 1 Processor -- start 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad9ee5700 1 -- start start 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad9ee5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad9ee5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4105a80 con 0x7f2ad4107f80 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47474/0 (socket says 172.21.15.116:47474) 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad37fe700 1 -- 172.21.15.116:0/3952161473 learned_addr learned my addr 172.21.15.116:0/3952161473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad37fe700 1 -- 172.21.15.116:0/3952161473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad410cda0 con 0x7f2ad4107f80 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad37fe700 1 --2- 172.21.15.116:0/3952161473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8006af0 tx=0x7f2ac8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.159+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ac80039e0 con 0x7f2ad4107f80 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2ac8003b80 con 0x7f2ad4107f80 2024-03-19T02:17:20.850 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad410af90 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad410b4d0 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ac80196a0 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ac802d440 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.160+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ac8024c00 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.161+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad4063940 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.163+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2ac8043dd0 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.235+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2ad4194e80 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.236+0000 7f2ad0ff9700 1 -- 172.21.15.116:0/3952161473 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2ac8032020 con 0x7f2ad4107f80 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 msgr2=0x7f2ad410c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 --2- 172.21.15.116:0/3952161473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8006af0 tx=0x7f2ac8011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 shutdown_connections 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 --2- 172.21.15.116:0/3952161473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ad4107f80 0x7f2ad410c860 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 >> 172.21.15.116:0/3952161473 conn(0x7f2ad4101720 msgr2=0x7f2ad41021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:20.851 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 shutdown_connections 2024-03-19T02:17:20.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:20.237+0000 7f2ad9ee5700 1 -- 172.21.15.116:0/3952161473 wait complete. 2024-03-19T02:17:20.852 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (3/15)... 2024-03-19T02:17:21.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: Activating manager daemon x 2024-03-19T02:17:21.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: mgrmap e2: x(active, starting, since 0.00331377s) 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: Manager daemon x is now available 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' 2024-03-19T02:17:21.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:21 smithi116 ceph-mon[32263]: from='mgr.14100 172.21.15.116:0/2771012106' entity='mgr.x' 2024-03-19T02:17:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:22 smithi116 ceph-mon[32263]: mgrmap e3: x(active, since 1.00745s) 2024-03-19T02:17:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:24 smithi116 ceph-mon[32263]: mgrmap e4: x(active, since 3s) 2024-03-19T02:17:26.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:26 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2934387628' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:17:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:17:31.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:17:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:17:31.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:17:31.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:17:06.754009+0000", 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:17:31.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e65696700 1 Processor -- start 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e65696700 1 -- start start 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e65696700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e65696700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e60106790 con 0x7f8e60105da0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47542/0 (socket says 172.21.15.116:47542) 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.844+0000 7f8e5effd700 1 -- 172.21.15.116:0/2023745924 learned_addr learned my addr 172.21.15.116:0/2023745924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e5effd700 1 -- 172.21.15.116:0/2023745924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e60108d30 con 0x7f8e60105da0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e5effd700 1 --2- 172.21.15.116:0/2023745924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009e80 tx=0x7f8e5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5853f5087ed35be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e5dffb700 1 -- 172.21.15.116:0/2023745924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50017070 con 0x7f8e60105da0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e5dffb700 1 -- 172.21.15.116:0/2023745924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e5000b8c0 con 0x7f8e60105da0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e5dffb700 1 -- 172.21.15.116:0/2023745924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50004520 con 0x7f8e60105da0 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e65696700 1 -- 172.21.15.116:0/2023745924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 msgr2=0x7f8e601061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:31.927 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e65696700 1 --2- 172.21.15.116:0/2023745924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009e80 tx=0x7f8e5000b330 comp rx=0 tx=0).stop 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e65696700 1 -- 172.21.15.116:0/2023745924 shutdown_connections 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e65696700 1 --2- 172.21.15.116:0/2023745924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e601061c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.845+0000 7f8e65696700 1 -- 172.21.15.116:0/2023745924 >> 172.21.15.116:0/2023745924 conn(0x7f8e60101770 msgr2=0x7f8e60103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 -- 172.21.15.116:0/2023745924 shutdown_connections 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 -- 172.21.15.116:0/2023745924 wait complete. 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 Processor -- start 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 -- start start 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.846+0000 7f8e65696700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6010c0e0 con 0x7f8e60105da0 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e5effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47556/0 (socket says 172.21.15.116:47556) 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e5effd700 1 -- 172.21.15.116:0/2934387628 learned_addr learned my addr 172.21.15.116:0/2934387628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e5effd700 1 -- 172.21.15.116:0/2934387628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6019be60 con 0x7f8e60105da0 2024-03-19T02:17:31.928 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e5effd700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b7b0 tx=0x7f8e50004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50020020 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6019c0f0 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e5000bba0 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50017040 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.847+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e6019ede0 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.848+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f8e50023020 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.848+0000 7f8e57fff700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 0x7f8e4c03b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.848+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e60063940 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.848+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e5000f050 con 0x7f8e60105da0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.848+0000 7f8e5e7fc700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 0x7f8e4c03b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:31.929 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.849+0000 7f8e5e7fc700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 0x7f8e4c03b400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:25.853+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e5002a020 con 0x7f8e60105da0 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.129+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8e6019efb0 con 0x7f8e60105da0 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.129+0000 7f8e57fff700 1 -- 172.21.15.116:0/2934387628 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8e5002c300 con 0x7f8e60105da0 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 msgr2=0x7f8e4c03b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 0x7f8e4c03b400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 msgr2=0x7f8e6019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b7b0 tx=0x7f8e50004670 comp rx=0 tx=0).stop 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 shutdown_connections 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f8e4c038f40 0x7f8e4c03b400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 --2- 172.21.15.116:0/2934387628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60105da0 0x7f8e6019b920 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 >> 172.21.15.116:0/2934387628 conn(0x7f8e60101770 msgr2=0x7f8e60102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 shutdown_connections 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:26.131+0000 7f8e65696700 1 -- 172.21.15.116:0/2934387628 wait complete. 2024-03-19T02:17:31.930 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-19T02:17:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:33 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/243837531' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T02:17:34.197 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:34.197 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T02:17:34.198 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.465+0000 7f6b24ecc700 1 Processor -- start 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.465+0000 7f6b24ecc700 1 -- start start 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b24ecc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b24ecc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b201073a0 con 0x7f6b201069b0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44934/0 (socket says 172.21.15.116:44934) 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1e59c700 1 -- 172.21.15.116:0/2229433294 learned_addr learned my addr 172.21.15.116:0/2229433294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1e59c700 1 -- 172.21.15.116:0/2229433294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b20107bb0 con 0x7f6b201069b0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1e59c700 1 --2- 172.21.15.116:0/2229433294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b100098a0 tx=0x7f6b10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f761bfba87783c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1d59a700 1 -- 172.21.15.116:0/2229433294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b10004160 con 0x7f6b201069b0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1d59a700 1 -- 172.21.15.116:0/2229433294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6b10004300 con 0x7f6b201069b0 2024-03-19T02:17:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.466+0000 7f6b1d59a700 1 -- 172.21.15.116:0/2229433294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b10011a80 con 0x7f6b201069b0 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 -- 172.21.15.116:0/2229433294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 msgr2=0x7f6b20106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/2229433294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6b100098a0 tx=0x7f6b10009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 -- 172.21.15.116:0/2229433294 shutdown_connections 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/2229433294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b20106dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 -- 172.21.15.116:0/2229433294 >> 172.21.15.116:0/2229433294 conn(0x7f6b20101fb0 msgr2=0x7f6b201043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 -- 172.21.15.116:0/2229433294 shutdown_connections 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.467+0000 7f6b24ecc700 1 -- 172.21.15.116:0/2229433294 wait complete. 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b24ecc700 1 Processor -- start 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b24ecc700 1 -- start start 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b24ecc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b24ecc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2010af60 con 0x7f6b201069b0 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b1e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b1e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44950/0 (socket says 172.21.15.116:44950) 2024-03-19T02:17:34.200 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.468+0000 7f6b1e59c700 1 -- 172.21.15.116:0/243837531 learned_addr learned my addr 172.21.15.116:0/243837531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b1e59c700 1 -- 172.21.15.116:0/243837531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2019ad30 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b1e59c700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b10006af0 tx=0x7f6b10011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b100039e0 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6b10003b80 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2019afc0 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b100196a0 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.469+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b2019dcb0 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.470+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f6b100199a0 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.470+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b20063940 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.470+0000 7f6b0f7fe700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 0x7f6b0803b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.470+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6b1004d510 con 0x7f6b201069b0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.470+0000 7f6b1dd9b700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 0x7f6b0803b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.471+0000 7f6b1dd9b700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 0x7f6b0803b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b14006fd0 tx=0x7f6b14006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:34.201 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.475+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b1004cf30 con 0x7f6b201069b0 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.670+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6b2019de80 con 0x7f6b201069b0 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.671+0000 7f6b0f7fe700 1 -- 172.21.15.116:0/243837531 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f6b2019de80 con 0x7f6b201069b0 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 msgr2=0x7f6b0803b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 0x7f6b0803b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b14006fd0 tx=0x7f6b14006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 msgr2=0x7f6b2019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b10006af0 tx=0x7f6b10011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 shutdown_connections 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6b08038f40 0x7f6b0803b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 --2- 172.21.15.116:0/243837531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b201069b0 0x7f6b2019a7f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 >> 172.21.15.116:0/243837531 conn(0x7f6b20101fb0 msgr2=0x7f6b20102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 shutdown_connections 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:33.673+0000 7f6b24ecc700 1 -- 172.21.15.116:0/243837531 wait complete. 2024-03-19T02:17:34.202 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-19T02:17:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4256645856' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-19T02:17:35.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setuser ceph since I am not root 2024-03-19T02:17:35.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setgroup ceph since I am not root 2024-03-19T02:17:35.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:35.470+0000 7f678375d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:17:35.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:35.565+0000 7f678375d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:17:35.861 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b7227700 1 Processor -- start 2024-03-19T02:17:35.861 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b7227700 1 -- start start 2024-03-19T02:17:35.861 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b7227700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b7227700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0106790 con 0x7f66b0105da0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b4fc3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b4fc3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44954/0 (socket says 172.21.15.116:44954) 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.607+0000 7f66b4fc3700 1 -- 172.21.15.116:0/4098851639 learned_addr learned my addr 172.21.15.116:0/4098851639 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b4fc3700 1 -- 172.21.15.116:0/4098851639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b0108d30 con 0x7f66b0105da0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b4fc3700 1 --2- 172.21.15.116:0/4098851639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66a40098a0 tx=0x7f66a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91ec874237d0eca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66af7fe700 1 -- 172.21.15.116:0/4098851639 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a4004160 con 0x7f66b0105da0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66af7fe700 1 -- 172.21.15.116:0/4098851639 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f66a4004300 con 0x7f66b0105da0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66af7fe700 1 -- 172.21.15.116:0/4098851639 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a4011a80 con 0x7f66b0105da0 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b7227700 1 -- 172.21.15.116:0/4098851639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 msgr2=0x7f66b01061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b7227700 1 --2- 172.21.15.116:0/4098851639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f66a40098a0 tx=0x7f66a4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b7227700 1 -- 172.21.15.116:0/4098851639 shutdown_connections 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b7227700 1 --2- 172.21.15.116:0/4098851639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b01061c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.608+0000 7f66b7227700 1 -- 172.21.15.116:0/4098851639 >> 172.21.15.116:0/4098851639 conn(0x7f66b0101770 msgr2=0x7f66b0103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 -- 172.21.15.116:0/4098851639 shutdown_connections 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 -- 172.21.15.116:0/4098851639 wait complete. 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 Processor -- start 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 -- start start 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b7227700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010c0e0 con 0x7f66b0105da0 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b4fc3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b4fc3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44960/0 (socket says 172.21.15.116:44960) 2024-03-19T02:17:35.863 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.609+0000 7f66b4fc3700 1 -- 172.21.15.116:0/4256645856 learned_addr learned my addr 172.21.15.116:0/4256645856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66b4fc3700 1 -- 172.21.15.116:0/4256645856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b019beb0 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66b4fc3700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f66a4000c00 tx=0x7f66a4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a40039c0 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f66a4003b60 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b019c140 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66a401a6f0 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.610+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b019ee30 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.611+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f66a401a9a0 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.611+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b0063940 con 0x7f66b0105da0 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.611+0000 7f66adffb700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 0x7f66a003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:35.864 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.611+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f66a404d0a0 con 0x7f66b0105da0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.611+0000 7f66affff700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 0x7f66a003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.612+0000 7f66affff700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 0x7f66a003b450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f669c006fd0 tx=0x7f669c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.616+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66a400b7e0 con 0x7f66b0105da0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.865+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f66b019f000 con 0x7f66b0105da0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.917+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f66b019f000 con 0x7f66b0105da0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.919+0000 7f66adffb700 1 -- 172.21.15.116:0/4256645856 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f66a402d3f0 con 0x7f66b0105da0 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 msgr2=0x7f66a003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 0x7f66a003b450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f669c006fd0 tx=0x7f669c006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 msgr2=0x7f66b019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f66a4000c00 tx=0x7f66a4004290 comp rx=0 tx=0).stop 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 shutdown_connections 2024-03-19T02:17:35.865 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f66a0038f90 0x7f66a003b450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:35.866 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 --2- 172.21.15.116:0/4256645856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b0105da0 0x7f66b019b970 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:35.866 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 >> 172.21.15.116:0/4256645856 conn(0x7f66b0101770 msgr2=0x7f66b0102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:35.866 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 shutdown_connections 2024-03-19T02:17:35.866 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:34.920+0000 7f66b7227700 1 -- 172.21.15.116:0/4256645856 wait complete. 2024-03-19T02:17:36.104 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4256645856' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-19T02:17:36.104 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:35 smithi116 ceph-mon[32263]: mgrmap e5: x(active, since 14s) 2024-03-19T02:17:36.463 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.220+0000 7f678375d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:17:36.463 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.318+0000 7f678375d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:17:36.463 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.462+0000 7f678375d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:17:36.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.560+0000 7f678375d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:17:36.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.648+0000 7f678375d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:17:37.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1942501487' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T02:17:37.115 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:36.917+0000 7f678375d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:17:37.115 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:37.029+0000 7f678375d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:17:37.115 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:37.114+0000 7f678375d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.288+0000 7f667f836700 1 Processor -- start 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.288+0000 7f667f836700 1 -- start start 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.288+0000 7f667f836700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.288+0000 7f667f836700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f667807be40 con 0x7f6678108560 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.288+0000 7f667d5d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f667d5d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44986/0 (socket says 172.21.15.116:44986) 2024-03-19T02:17:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f667d5d2700 1 -- 172.21.15.116:0/812043638 learned_addr learned my addr 172.21.15.116:0/812043638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f667d5d2700 1 -- 172.21.15.116:0/812043638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667810ae90 con 0x7f6678108560 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f667d5d2700 1 --2- 172.21.15.116:0/812043638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f66680098a0 tx=0x7f6668009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390b9bc86e135f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f666ffff700 1 -- 172.21.15.116:0/812043638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6668004160 con 0x7f6678108560 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f666ffff700 1 -- 172.21.15.116:0/812043638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6668004300 con 0x7f6678108560 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.289+0000 7f666ffff700 1 -- 172.21.15.116:0/812043638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6668011a80 con 0x7f6678108560 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 -- 172.21.15.116:0/812043638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 msgr2=0x7f667810a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 --2- 172.21.15.116:0/812043638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f66680098a0 tx=0x7f6668009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 -- 172.21.15.116:0/812043638 shutdown_connections 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 --2- 172.21.15.116:0/812043638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667810a950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 -- 172.21.15.116:0/812043638 >> 172.21.15.116:0/812043638 conn(0x7f6678101fb0 msgr2=0x7f66781043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 -- 172.21.15.116:0/812043638 shutdown_connections 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 -- 172.21.15.116:0/812043638 wait complete. 2024-03-19T02:17:37.128 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.290+0000 7f667f836700 1 Processor -- start 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667f836700 1 -- start start 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667f836700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667f836700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f667810be50 con 0x7f6678108560 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667d5d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667d5d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44998/0 (socket says 172.21.15.116:44998) 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667d5d2700 1 -- 172.21.15.116:0/1942501487 learned_addr learned my addr 172.21.15.116:0/1942501487 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667d5d2700 1 -- 172.21.15.116:0/1942501487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667819bdf0 con 0x7f6678108560 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667d5d2700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6668006af0 tx=0x7f6668011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66680039e0 con 0x7f6678108560 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6668003b80 con 0x7f6678108560 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667819c080 con 0x7f6678108560 2024-03-19T02:17:37.129 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.291+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66680196a0 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.292+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667819ed70 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.292+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f66680199a0 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.292+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6678063940 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.293+0000 7f666e7fc700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 0x7f666403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.293+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f666804d410 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.293+0000 7f667cdd1700 1 -- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 msgr2=0x7f666403b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.293+0000 7f667cdd1700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 0x7f666403b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.298+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f666804ce30 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.494+0000 7f667cdd1700 1 -- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 msgr2=0x7f666403b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.494+0000 7f667cdd1700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 0x7f666403b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.558+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f667819ef40 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.558+0000 7f666e7fc700 1 -- 172.21.15.116:0/1942501487 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f667819ef40 con 0x7f6678108560 2024-03-19T02:17:37.130 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 msgr2=0x7f666403b450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 0x7f666403b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 msgr2=0x7f667819b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6668006af0 tx=0x7f6668011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 shutdown_connections 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7f6664038f90 0x7f666403b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 --2- 172.21.15.116:0/1942501487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6678108560 0x7f667819b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 >> 172.21.15.116:0/1942501487 conn(0x7f6678101fb0 msgr2=0x7f66781029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 shutdown_connections 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:36.560+0000 7f667f836700 1 -- 172.21.15.116:0/1942501487 wait complete. 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-19T02:17:37.131 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-19T02:17:37.406 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:37.215+0000 7f678375d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:17:37.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:37.405+0000 7f678375d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:17:38.494 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:38.363+0000 7f678375d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:17:38.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:38.494+0000 7f678375d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:17:39.129 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:38.841+0000 7f678375d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:17:39.129 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:38.931+0000 7f678375d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:17:39.129 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:39.128+0000 7f678375d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:17:39.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:39.311+0000 7f678375d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:17:39.861 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:39.595+0000 7f678375d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:17:39.862 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:39.861+0000 7f678375d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:17:40.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:40 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:40.094+0000 7f678375d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:17:41.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:40 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:40.891+0000 7f678375d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:17:41.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:41.454+0000 7f678375d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:17:41.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:41.552+0000 7f678375d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:17:42.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:17:41.770+0000 7f678375d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:17:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: Active manager daemon x restarted 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: Activating manager daemon x 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: osdmap e2: 0 total, 0 up, 0 in 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: mgrmap e6: x(active, starting, since 0.00329151s) 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: Manager daemon x is now available 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:41 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: Found migration_current of "None". Setting to last migration. 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:43.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:42 smithi116 ceph-mon[32263]: mgrmap e7: x(active, since 1.0064s) 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd866e43700 1 Processor -- start 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd866e43700 1 -- start start 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd866e43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd866e43700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd8601084f0 con 0x7fd860107b00 2024-03-19T02:17:43.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd864bdf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd864bdf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45000/0 (socket says 172.21.15.116:45000) 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.563+0000 7fd864bdf700 1 -- 172.21.15.116:0/1320656773 learned_addr learned my addr 172.21.15.116:0/1320656773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.564+0000 7fd864bdf700 1 -- 172.21.15.116:0/1320656773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd860108cb0 con 0x7fd860107b00 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.564+0000 7fd864bdf700 1 --2- 172.21.15.116:0/1320656773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd8540098a0 tx=0x7fd854009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6a1943c19be81d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.564+0000 7fd85f7fe700 1 -- 172.21.15.116:0/1320656773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd854004160 con 0x7fd860107b00 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.564+0000 7fd85f7fe700 1 -- 172.21.15.116:0/1320656773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd854004300 con 0x7fd860107b00 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.564+0000 7fd85f7fe700 1 -- 172.21.15.116:0/1320656773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd854011a80 con 0x7fd860107b00 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 -- 172.21.15.116:0/1320656773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 msgr2=0x7fd860107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 --2- 172.21.15.116:0/1320656773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd8540098a0 tx=0x7fd854009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 -- 172.21.15.116:0/1320656773 shutdown_connections 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 --2- 172.21.15.116:0/1320656773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd860107f20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 -- 172.21.15.116:0/1320656773 >> 172.21.15.116:0/1320656773 conn(0x7fd8601030c0 msgr2=0x7fd8601054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 -- 172.21.15.116:0/1320656773 shutdown_connections 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 -- 172.21.15.116:0/1320656773 wait complete. 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.565+0000 7fd866e43700 1 Processor -- start 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd866e43700 1 -- start start 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd866e43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd866e43700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd86010c060 con 0x7fd860107b00 2024-03-19T02:17:43.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd864bdf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd864bdf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45004/0 (socket says 172.21.15.116:45004) 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd864bdf700 1 -- 172.21.15.116:0/1918225491 learned_addr learned my addr 172.21.15.116:0/1918225491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd864bdf700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86019beb0 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd864bdf700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd854006af0 tx=0x7fd854011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd854003930 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd854003ad0 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.566+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86019c140 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.567+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8540195e0 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.567+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86019ee30 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fd854019890 con 0x7fd860107b00 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85dffb700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:43.432 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd85003bbb0 con 0x7fd850038fe0 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85ffff700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd85404d230 con 0x7fd860107b00 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.568+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.769+0000 7fd85ffff700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:37.769+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:38.170+0000 7fd85ffff700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:38.170+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:38.971+0000 7fd85ffff700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:38.971+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:17:43.433 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:40.573+0000 7fd85ffff700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1136807551 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:40.573+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:41.786+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fd854019bd0 con 0x7fd860107b00 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:41.786+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 msgr2=0x7fd85003b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:41.786+0000 7fd85dffb700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.788+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fd85401aa90 con 0x7fd860107b00 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.788+0000 7fd85dffb700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 0x7fd85003ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.788+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd854001d30 con 0x7fd85003c8d0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.788+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 0x7fd85003ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.789+0000 7fd85ffff700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 0x7fd85003ecc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd84c003a10 tx=0x7fd84c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.789+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd854001d30 con 0x7fd85003c8d0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd860063940 con 0x7fd85003c8d0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd85dffb700 1 -- 172.21.15.116:0/1918225491 <== mgr.14120 v2:172.21.15.116:6800/647665947 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd860063940 con 0x7fd85003c8d0 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 msgr2=0x7fd85003ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:43.434 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 0x7fd85003ecc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd84c003a10 tx=0x7fd84c0092b0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 msgr2=0x7fd86019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd854006af0 tx=0x7fd854011b70 comp rx=0 tx=0).stop 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 shutdown_connections 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fd85003c8d0 0x7fd85003ecc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860107b00 0x7fd86019b970 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 --2- 172.21.15.116:0/1918225491 >> [v2:172.21.15.116:6800/1136807551,v1:172.21.15.116:6801/1136807551] conn(0x7fd850038fe0 0x7fd85003b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.796+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 >> 172.21.15.116:0/1918225491 conn(0x7fd8601030c0 msgr2=0x7fd860103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.797+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 shutdown_connections 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:42.797+0000 7fd866e43700 1 -- 172.21.15.116:0/1918225491 wait complete. 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-19T02:17:43.435 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-19T02:17:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: [19/Mar/2024:02:17:42] ENGINE Bus STARTING 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: [19/Mar/2024:02:17:42] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: [19/Mar/2024:02:17:42] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: [19/Mar/2024:02:17:42] ENGINE Bus STARTED 2024-03-19T02:17:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:43 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:44.605 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955ceef700 1 Processor -- start 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955ceef700 1 -- start start 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955ceef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955ceef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f955807be40 con 0x7f9558108540 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53060/0 (socket says 172.21.15.116:53060) 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.796+0000 7f955659c700 1 -- 172.21.15.116:0/3392695085 learned_addr learned my addr 172.21.15.116:0/3392695085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f955659c700 1 -- 172.21.15.116:0/3392695085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955810ae70 con 0x7f9558108540 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f955659c700 1 --2- 172.21.15.116:0/3392695085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95480098a0 tx=0x7f9548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e39865ccc78ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f9555d9b700 1 -- 172.21.15.116:0/3392695085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9548004160 con 0x7f9558108540 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f9555d9b700 1 -- 172.21.15.116:0/3392695085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9548004300 con 0x7f9558108540 2024-03-19T02:17:44.606 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f9555d9b700 1 -- 172.21.15.116:0/3392695085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9548011a80 con 0x7f9558108540 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f955ceef700 1 -- 172.21.15.116:0/3392695085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 msgr2=0x7f955810a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.797+0000 7f955ceef700 1 --2- 172.21.15.116:0/3392695085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95480098a0 tx=0x7f9548009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 -- 172.21.15.116:0/3392695085 shutdown_connections 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 --2- 172.21.15.116:0/3392695085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955810a930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 -- 172.21.15.116:0/3392695085 >> 172.21.15.116:0/3392695085 conn(0x7f9558101f30 msgr2=0x7f9558104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 -- 172.21.15.116:0/3392695085 shutdown_connections 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 -- 172.21.15.116:0/3392695085 wait complete. 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.798+0000 7f955ceef700 1 Processor -- start 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955ceef700 1 -- start start 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955ceef700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955ceef700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f955810be30 con 0x7f9558108540 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53066/0 (socket says 172.21.15.116:53066) 2024-03-19T02:17:44.607 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955659c700 1 -- 172.21.15.116:0/547345104 learned_addr learned my addr 172.21.15.116:0/547345104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955659c700 1 -- 172.21.15.116:0/547345104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955819be00 con 0x7f9558108540 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955659c700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9548006af0 tx=0x7f9548011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95480039e0 con 0x7f9558108540 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9548003b80 con 0x7f9558108540 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955819c090 con 0x7f9558108540 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95480196a0 con 0x7f9558108540 2024-03-19T02:17:44.609 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.799+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955819ed80 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.800+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f95480199a0 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.800+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9558063940 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.801+0000 7f954ffff700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 0x7f954403b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.801+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f954804d240 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.801+0000 7f954dd9b700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 0x7f954403b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.801+0000 7f954dd9b700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 0x7f954403b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9540006fd0 tx=0x7f9540006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.806+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9548019d90 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:43.839+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f954802d960 con 0x7f9558108540 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.023+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f9558103090 con 0x7f9544038e60 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.031+0000 7f954ffff700 1 -- 172.21.15.116:0/547345104 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9558103090 con 0x7f9544038e60 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 msgr2=0x7f954403b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 0x7f954403b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9540006fd0 tx=0x7f9540006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 msgr2=0x7f955819b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9548006af0 tx=0x7f9548011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 shutdown_connections 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9544038e60 0x7f954403b320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 --2- 172.21.15.116:0/547345104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9558108540 0x7f955819b8c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 >> 172.21.15.116:0/547345104 conn(0x7f9558101f30 msgr2=0x7f9558102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 shutdown_connections 2024-03-19T02:17:44.611 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:44.034+0000 7f955ceef700 1 -- 172.21.15.116:0/547345104 wait complete. 2024-03-19T02:17:44.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:44 smithi116 ceph-mon[32263]: mgrmap e8: x(active, since 2s) 2024-03-19T02:17:44.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:44 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:44.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:44 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:45.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-19T02:17:45.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.042+0000 7f023eba2700 1 Processor -- start 2024-03-19T02:17:45.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.042+0000 7f023eba2700 1 -- start start 2024-03-19T02:17:45.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.042+0000 7f023eba2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023eba2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f023807be00 con 0x7f02381083a0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023dba0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023dba0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53082/0 (socket says 172.21.15.116:53082) 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023dba0700 1 -- 172.21.15.116:0/1079739822 learned_addr learned my addr 172.21.15.116:0/1079739822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023dba0700 1 -- 172.21.15.116:0/1079739822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023810acd0 con 0x7f02381083a0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023dba0700 1 --2- 172.21.15.116:0/1079739822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f02280098a0 tx=0x7f0228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5ac7c53a783061 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023cb9e700 1 -- 172.21.15.116:0/1079739822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0228004160 con 0x7f02381083a0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023cb9e700 1 -- 172.21.15.116:0/1079739822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0228004300 con 0x7f02381083a0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.043+0000 7f023cb9e700 1 -- 172.21.15.116:0/1079739822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0228011a80 con 0x7f02381083a0 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- 172.21.15.116:0/1079739822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 msgr2=0x7f023810a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 --2- 172.21.15.116:0/1079739822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f02280098a0 tx=0x7f0228009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- 172.21.15.116:0/1079739822 shutdown_connections 2024-03-19T02:17:45.788 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 --2- 172.21.15.116:0/1079739822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023810a790 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- 172.21.15.116:0/1079739822 >> 172.21.15.116:0/1079739822 conn(0x7f0238101d50 msgr2=0x7f0238104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- 172.21.15.116:0/1079739822 shutdown_connections 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- 172.21.15.116:0/1079739822 wait complete. 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 Processor -- start 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.044+0000 7f023eba2700 1 -- start start 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023eba2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023eba2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f023810bc30 con 0x7f02381083a0 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023dba0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023dba0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53088/0 (socket says 172.21.15.116:53088) 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023dba0700 1 -- 172.21.15.116:0/2767397595 learned_addr learned my addr 172.21.15.116:0/2767397595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023dba0700 1 -- 172.21.15.116:0/2767397595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023819bc00 con 0x7f02381083a0 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023dba0700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0228006af0 tx=0x7f0228011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:45.789 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02280039e0 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0228003b80 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02280196a0 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023819be90 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.045+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023819eb80 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.046+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f022801a430 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.046+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0238063940 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.046+0000 7f022effd700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 0x7f022403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.046+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f022804d3d0 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.046+0000 7f023d39f700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 0x7f022403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.047+0000 7f023d39f700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 0x7f022403b450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0234006fd0 tx=0x7f0234006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.051+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0228023e00 con 0x7f02381083a0 2024-03-19T02:17:45.790 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.250+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f0238102e50 con 0x7f0224038f90 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.251+0000 7f022effd700 1 -- 172.21.15.116:0/2767397595 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f0238102e50 con 0x7f0224038f90 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.252+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 msgr2=0x7f022403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 0x7f022403b450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0234006fd0 tx=0x7f0234006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 msgr2=0x7f023819b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0228006af0 tx=0x7f0228011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 shutdown_connections 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0224038f90 0x7f022403b450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 --2- 172.21.15.116:0/2767397595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02381083a0 0x7f023819b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 >> 172.21.15.116:0/2767397595 conn(0x7f0238101d50 msgr2=0x7f0238102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 shutdown_connections 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:45.253+0000 7f023eba2700 1 -- 172.21.15.116:0/2767397595 wait complete. 2024-03-19T02:17:45.791 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-19T02:17:46.098 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:45 smithi116 ceph-mon[32263]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-mon[32263]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:47.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: Generating public/private rsa key pair. 2024-03-19T02:17:47.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: Your identification has been saved in /tmp/tmp14hy726l/key. 2024-03-19T02:17:47.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: Your public key has been saved in /tmp/tmp14hy726l/key.pub. 2024-03-19T02:17:47.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: The key fingerprint is: 2024-03-19T02:17:47.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: SHA256:tZ0v5ghNNfWVwpAe3C80YnCgDDCvgkWzqvzPRpIH/5s ceph-a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: The key's randomart image is: 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: +---[RSA 3072]----+ 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: | oo.. o+o= . o| 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: | . oo o . .* B o.| 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: | o . o + * + .| 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: |.o .. . = + . | 2024-03-19T02:17:47.010 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: |o. .+ S o o . | 2024-03-19T02:17:47.011 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: |o .o + o . | 2024-03-19T02:17:47.011 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: |.. + . . . o . | 2024-03-19T02:17:47.011 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: | . .. ... + . | 2024-03-19T02:17:47.011 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: | .oo E. . . | 2024-03-19T02:17:47.011 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:17:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: +----[SHA256]-----+ 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.197+0000 7f4e275d6700 1 Processor -- start 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.197+0000 7f4e275d6700 1 -- start start 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.197+0000 7f4e275d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.197+0000 7f4e275d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2007be40 con 0x7f4e20108570 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.197+0000 7f4e25372700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e25372700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53098/0 (socket says 172.21.15.116:53098) 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e25372700 1 -- 172.21.15.116:0/1715951045 learned_addr learned my addr 172.21.15.116:0/1715951045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e25372700 1 -- 172.21.15.116:0/1715951045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2010aea0 con 0x7f4e20108570 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e25372700 1 --2- 172.21.15.116:0/1715951045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0098a0 tx=0x7f4e1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e96203aea082c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e17fff700 1 -- 172.21.15.116:0/1715951045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c004160 con 0x7f4e20108570 2024-03-19T02:17:47.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e17fff700 1 -- 172.21.15.116:0/1715951045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4e1c004300 con 0x7f4e20108570 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.198+0000 7f4e17fff700 1 -- 172.21.15.116:0/1715951045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c011a80 con 0x7f4e20108570 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- 172.21.15.116:0/1715951045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 msgr2=0x7f4e2010a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1715951045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0098a0 tx=0x7f4e1c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- 172.21.15.116:0/1715951045 shutdown_connections 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1715951045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2010a960 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- 172.21.15.116:0/1715951045 >> 172.21.15.116:0/1715951045 conn(0x7f4e20101f80 msgr2=0x7f4e201043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- 172.21.15.116:0/1715951045 shutdown_connections 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- 172.21.15.116:0/1715951045 wait complete. 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 Processor -- start 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.199+0000 7f4e275d6700 1 -- start start 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e275d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e275d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010be60 con 0x7f4e20108570 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e25372700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:47.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e25372700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53110/0 (socket says 172.21.15.116:53110) 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e25372700 1 -- 172.21.15.116:0/1968105241 learned_addr learned my addr 172.21.15.116:0/1968105241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e25372700 1 -- 172.21.15.116:0/1968105241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2019be30 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e25372700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c006af0 tx=0x7f4e1c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c0039e0 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4e1c003b80 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2019c0c0 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c0196a0 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.200+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2019edb0 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.201+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f4e1c01a430 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.201+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e20063940 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.201+0000 7f4e167fc700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 0x7f4e0c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.201+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4e1c04d3f0 con 0x7f4e20108570 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.201+0000 7f4e24b71700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 0x7f4e0c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:47.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.202+0000 7f4e24b71700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 0x7f4e0c03b4a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e10006fd0 tx=0x7f4e10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.206+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e1c019950 con 0x7f4e20108570 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.405+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f4e20103090 con 0x7f4e0c038fe0 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.728+0000 7f4e167fc700 1 -- 172.21.15.116:0/1968105241 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4e20103090 con 0x7f4e0c038fe0 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.730+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 msgr2=0x7f4e0c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.730+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 0x7f4e0c03b4a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e10006fd0 tx=0x7f4e10006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.730+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 msgr2=0x7f4e2019b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.730+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c006af0 tx=0x7f4e1c011c40 comp rx=0 tx=0).stop 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 shutdown_connections 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4e0c038fe0 0x7f4e0c03b4a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 --2- 172.21.15.116:0/1968105241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e20108570 0x7f4e2019b8f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 >> 172.21.15.116:0/1968105241 conn(0x7f4e20101f80 msgr2=0x7f4e20102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 shutdown_connections 2024-03-19T02:17:47.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:46.731+0000 7f4e275d6700 1 -- 172.21.15.116:0/1968105241 wait complete. 2024-03-19T02:17:48.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:47 smithi116 ceph-mon[32263]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:48.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:47 smithi116 ceph-mon[32263]: Generating ssh key... 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03b53a9700 1 Processor -- start 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03b53a9700 1 -- start start 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03b53a9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03b53a9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03b01073a0 con 0x7f03b01069b0 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03aeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03aeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53644/0 (socket says 172.21.15.116:53644) 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03aeffd700 1 -- 172.21.15.116:0/929143173 learned_addr learned my addr 172.21.15.116:0/929143173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.704+0000 7f03aeffd700 1 -- 172.21.15.116:0/929143173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b0107bb0 con 0x7f03b01069b0 2024-03-19T02:17:48.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03aeffd700 1 --2- 172.21.15.116:0/929143173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f03a00098a0 tx=0x7f03a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491d781736f9cd3e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03adffb700 1 -- 172.21.15.116:0/929143173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03a0004160 con 0x7f03b01069b0 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03adffb700 1 -- 172.21.15.116:0/929143173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f03a0004300 con 0x7f03b01069b0 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03adffb700 1 -- 172.21.15.116:0/929143173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03a0011a80 con 0x7f03b01069b0 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 -- 172.21.15.116:0/929143173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 msgr2=0x7f03b0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 --2- 172.21.15.116:0/929143173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f03a00098a0 tx=0x7f03a0009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 -- 172.21.15.116:0/929143173 shutdown_connections 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 --2- 172.21.15.116:0/929143173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b0106dd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 -- 172.21.15.116:0/929143173 >> 172.21.15.116:0/929143173 conn(0x7f03b0101fb0 msgr2=0x7f03b01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 -- 172.21.15.116:0/929143173 shutdown_connections 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.705+0000 7f03b53a9700 1 -- 172.21.15.116:0/929143173 wait complete. 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03b53a9700 1 Processor -- start 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03b53a9700 1 -- start start 2024-03-19T02:17:48.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03b53a9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03b53a9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03b010af60 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03aeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03aeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53658/0 (socket says 172.21.15.116:53658) 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.706+0000 7f03aeffd700 1 -- 172.21.15.116:0/142482463 learned_addr learned my addr 172.21.15.116:0/142482463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f03aeffd700 1 -- 172.21.15.116:0/142482463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b019e640 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f03aeffd700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03a0000c00 tx=0x7f03a0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03a00039f0 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f03a0003b90 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b019e810 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03a001a740 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.707+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b019ebb0 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f03a001a9f0 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b0063940 con 0x7f03b01069b0 2024-03-19T02:17:48.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f0397fff700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 0x7f039803b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f03a004d1a0 con 0x7f03b01069b0 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f03ae7fc700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 0x7f039803b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.708+0000 7f03ae7fc700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 0x7f039803b4f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f03a4006fd0 tx=0x7f03a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.713+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03a000b7e0 con 0x7f03b01069b0 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.911+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f03b0103110 con 0x7f0398039030 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.911+0000 7f0397fff700 1 -- 172.21.15.116:0/142482463 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f03b0103110 con 0x7f0398039030 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 msgr2=0x7f039803b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 0x7f039803b4f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f03a4006fd0 tx=0x7f03a4006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 msgr2=0x7f03b007ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03a0000c00 tx=0x7f03a0011b00 comp rx=0 tx=0).stop 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 shutdown_connections 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.913+0000 7f03b53a9700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f0398039030 0x7f039803b4f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.914+0000 7f03b53a9700 1 --2- 172.21.15.116:0/142482463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b01069b0 0x7f03b007ba20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.914+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 >> 172.21.15.116:0/142482463 conn(0x7f03b0101fb0 msgr2=0x7f03b0102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:48.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.914+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 shutdown_connections 2024-03-19T02:17:48.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:47.914+0000 7f03b53a9700 1 -- 172.21.15.116:0/142482463 wait complete. 2024-03-19T02:17:48.530 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-19T02:17:48.530 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-19T02:17:48.530 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-19T02:17:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:48 smithi116 ceph-mon[32263]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:50 smithi116 ceph-mon[32263]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:50 smithi116 ceph-mon[32263]: Deploying cephadm binary to smithi116 2024-03-19T02:17:53.974 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.910+0000 7f8e8af4c700 1 Processor -- start 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.910+0000 7f8e8af4c700 1 -- start start 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e8af4c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e8af4c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e84108490 con 0x7f8e84107aa0 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e88ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e88ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53670/0 (socket says 172.21.15.116:53670) 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e88ce8700 1 -- 172.21.15.116:0/986526399 learned_addr learned my addr 172.21.15.116:0/986526399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e88ce8700 1 -- 172.21.15.116:0/986526399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e84108c50 con 0x7f8e84107aa0 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e88ce8700 1 --2- 172.21.15.116:0/986526399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e80 tx=0x7f8e7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e669716c8f8d27c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e837fe700 1 -- 172.21.15.116:0/986526399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c017070 con 0x7f8e84107aa0 2024-03-19T02:17:53.975 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e837fe700 1 -- 172.21.15.116:0/986526399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e7c00b8c0 con 0x7f8e84107aa0 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.911+0000 7f8e837fe700 1 -- 172.21.15.116:0/986526399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c004520 con 0x7f8e84107aa0 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 -- 172.21.15.116:0/986526399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 msgr2=0x7f8e84107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/986526399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e80 tx=0x7f8e7c00b330 comp rx=0 tx=0).stop 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 -- 172.21.15.116:0/986526399 shutdown_connections 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/986526399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e84107ec0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 -- 172.21.15.116:0/986526399 >> 172.21.15.116:0/986526399 conn(0x7f8e84103020 msgr2=0x7f8e84105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 -- 172.21.15.116:0/986526399 shutdown_connections 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 -- 172.21.15.116:0/986526399 wait complete. 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.912+0000 7f8e8af4c700 1 Processor -- start 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e8af4c700 1 -- start start 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e8af4c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e8af4c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410c000 con 0x7f8e84107aa0 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e88ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:53.976 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e88ce8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53682/0 (socket says 172.21.15.116:53682) 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e88ce8700 1 -- 172.21.15.116:0/2274379023 learned_addr learned my addr 172.21.15.116:0/2274379023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e88ce8700 1 -- 172.21.15.116:0/2274379023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8407e080 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e88ce8700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c003f60 tx=0x7f8e7c004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c020020 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e7c00bcb0 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8407c270 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.913+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c017040 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.914+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8407c7b0 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.914+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8e7c023020 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.914+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e84063940 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.915+0000 7f8e81ffb700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 0x7f8e6c03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.915+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8e7c00f050 con 0x7f8e84107aa0 2024-03-19T02:17:53.977 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.915+0000 7f8e83fff700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 0x7f8e6c03b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.915+0000 7f8e83fff700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 0x7f8e6c03b4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:48.919+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e7c02a020 con 0x7f8e84107aa0 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:49.117+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) -- 0x7f8e8407d010 con 0x7f8e6c039030 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.370+0000 7f8e81ffb700 1 -- 172.21.15.116:0/2274379023 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8e8407d010 con 0x7f8e6c039030 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 msgr2=0x7f8e6c03b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 0x7f8e6c03b4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 msgr2=0x7f8e8407db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c003f60 tx=0x7f8e7c004670 comp rx=0 tx=0).stop 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 shutdown_connections 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f8e6c039030 0x7f8e6c03b4f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 --2- 172.21.15.116:0/2274379023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e84107aa0 0x7f8e8407db40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 >> 172.21.15.116:0/2274379023 conn(0x7f8e84103020 msgr2=0x7f8e84103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 shutdown_connections 2024-03-19T02:17:53.978 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:53.372+0000 7f8e8af4c700 1 -- 172.21.15.116:0/2274379023 wait complete. 2024-03-19T02:17:53.979 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mon service... 2024-03-19T02:17:54.632 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:54 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:54.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:54 smithi116 ceph-mon[32263]: Added host smithi116 2024-03-19T02:17:54.633 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:54 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:17:55.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.396+0000 7fcb314c8700 1 Processor -- start 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.396+0000 7fcb314c8700 1 -- start start 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.396+0000 7fcb314c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.396+0000 7fcb314c8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c07be20 con 0x7fcb2c108580 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.396+0000 7fcb2affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb2affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53684/0 (socket says 172.21.15.116:53684) 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb2affd700 1 -- 172.21.15.116:0/3306236499 learned_addr learned my addr 172.21.15.116:0/3306236499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb2affd700 1 -- 172.21.15.116:0/3306236499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2c10aeb0 con 0x7fcb2c108580 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb2affd700 1 --2- 172.21.15.116:0/3306236499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009e80 tx=0x7fcb1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856277c2f1c1bbbb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb29ffb700 1 -- 172.21.15.116:0/3306236499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb1c017070 con 0x7fcb2c108580 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb29ffb700 1 -- 172.21.15.116:0/3306236499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcb1c00b8c0 con 0x7fcb2c108580 2024-03-19T02:17:55.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.397+0000 7fcb29ffb700 1 -- 172.21.15.116:0/3306236499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb1c004520 con 0x7fcb2c108580 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 -- 172.21.15.116:0/3306236499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 msgr2=0x7fcb2c10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 --2- 172.21.15.116:0/3306236499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009e80 tx=0x7fcb1c00b330 comp rx=0 tx=0).stop 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 -- 172.21.15.116:0/3306236499 shutdown_connections 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 --2- 172.21.15.116:0/3306236499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c10a970 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 -- 172.21.15.116:0/3306236499 >> 172.21.15.116:0/3306236499 conn(0x7fcb2c101fb0 msgr2=0x7fcb2c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 -- 172.21.15.116:0/3306236499 shutdown_connections 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 -- 172.21.15.116:0/3306236499 wait complete. 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.398+0000 7fcb314c8700 1 Processor -- start 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb314c8700 1 -- start start 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb314c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb314c8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c10be20 con 0x7fcb2c108580 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb2affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb2affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53688/0 (socket says 172.21.15.116:53688) 2024-03-19T02:17:55.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb2affd700 1 -- 172.21.15.116:0/1173045168 learned_addr learned my addr 172.21.15.116:0/1173045168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb2affd700 1 -- 172.21.15.116:0/1173045168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2c108340 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb2affd700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009e50 tx=0x7fcb1c004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb1c020020 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcb1c00bba0 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb2c106530 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.399+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb1c017040 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.400+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2c106a70 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.400+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fcb1c023020 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.400+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb2c063940 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.400+0000 7fcb23fff700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 0x7fcb1803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.401+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcb1c00f050 con 0x7fcb2c108580 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.401+0000 7fcb2a7fc700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 0x7fcb1803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.401+0000 7fcb2a7fc700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 0x7fcb1803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcb14006fd0 tx=0x7fcb14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.405+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb1c02a020 con 0x7fcb2c108580 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.632+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fcb2c1072d0 con 0x7fcb18038fe0 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.637+0000 7fcb23fff700 1 -- 172.21.15.116:0/1173045168 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcb2c1072d0 con 0x7fcb18038fe0 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 msgr2=0x7fcb1803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 0x7fcb1803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcb14006fd0 tx=0x7fcb14006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 msgr2=0x7fcb2c107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009e50 tx=0x7fcb1c004670 comp rx=0 tx=0).stop 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 shutdown_connections 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fcb18038fe0 0x7fcb1803b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 --2- 172.21.15.116:0/1173045168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c108580 0x7fcb2c107e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 >> 172.21.15.116:0/1173045168 conn(0x7fcb2c101fb0 msgr2=0x7fcb2c102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 shutdown_connections 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:54.639+0000 7fcb314c8700 1 -- 172.21.15.116:0/1173045168 wait complete. 2024-03-19T02:17:55.395 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mgr service... 2024-03-19T02:17:55.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:55 smithi116 ceph-mon[32263]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:55.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:55 smithi116 ceph-mon[32263]: Saving service mon spec with placement count:5 2024-03-19T02:17:55.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:55 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:56.691 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-19T02:17:56.691 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.811+0000 7f85ee8c2700 1 Processor -- start 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.811+0000 7f85ee8c2700 1 -- start start 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85ee8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85ee8c2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85e8106760 con 0x7f85e8105d70 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53704/0 (socket says 172.21.15.116:53704) 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e7fff700 1 -- 172.21.15.116:0/1601757892 learned_addr learned my addr 172.21.15.116:0/1601757892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e7fff700 1 -- 172.21.15.116:0/1601757892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e8108d00 con 0x7f85e8105d70 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e7fff700 1 --2- 172.21.15.116:0/1601757892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85d80098a0 tx=0x7f85d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29699c2d0373e07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e6ffd700 1 -- 172.21.15.116:0/1601757892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85d8004160 con 0x7f85e8105d70 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e6ffd700 1 -- 172.21.15.116:0/1601757892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f85d8004300 con 0x7f85e8105d70 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.812+0000 7f85e6ffd700 1 -- 172.21.15.116:0/1601757892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85d8011a80 con 0x7f85e8105d70 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 -- 172.21.15.116:0/1601757892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 msgr2=0x7f85e8106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:56.692 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/1601757892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f85d80098a0 tx=0x7f85d8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 -- 172.21.15.116:0/1601757892 shutdown_connections 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/1601757892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e8106190 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 -- 172.21.15.116:0/1601757892 >> 172.21.15.116:0/1601757892 conn(0x7f85e81016e0 msgr2=0x7f85e8103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 -- 172.21.15.116:0/1601757892 shutdown_connections 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.813+0000 7f85ee8c2700 1 -- 172.21.15.116:0/1601757892 wait complete. 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85ee8c2700 1 Processor -- start 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85ee8c2700 1 -- start start 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85ee8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85ee8c2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85e810c0b0 con 0x7f85e8105d70 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53714/0 (socket says 172.21.15.116:53714) 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85e7fff700 1 -- 172.21.15.116:0/3923921114 learned_addr learned my addr 172.21.15.116:0/3923921114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85e7fff700 1 -- 172.21.15.116:0/3923921114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e819bec0 con 0x7f85e8105d70 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.814+0000 7f85e7fff700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85d8000c00 tx=0x7f85d8004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:56.693 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85d80039c0 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e819c150 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f85d8003b60 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85d801a6f0 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e819ee40 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.815+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f85d801a9a0 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.816+0000 7f85e57fa700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 0x7f85d003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.816+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e8063940 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.816+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f85d804d230 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.816+0000 7f85e77fe700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 0x7f85d003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.816+0000 7f85e77fe700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 0x7f85d003b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f85dc006fd0 tx=0x7f85dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:55.821+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85d804cb80 con 0x7f85e8105d70 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.043+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f85e8102890 con 0x7f85d0038fe0 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.050+0000 7f85e57fa700 1 -- 172.21.15.116:0/3923921114 <== mgr.14120 v2:172.21.15.116:6800/647665947 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f85e8102890 con 0x7f85d0038fe0 2024-03-19T02:17:56.694 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 msgr2=0x7f85d003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 0x7f85d003b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f85dc006fd0 tx=0x7f85dc006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 msgr2=0x7f85e819b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85d8000c00 tx=0x7f85d8004290 comp rx=0 tx=0).stop 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 shutdown_connections 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f85d0038fe0 0x7f85d003b4a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 --2- 172.21.15.116:0/3923921114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8105d70 0x7f85e819b980 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 >> 172.21.15.116:0/3923921114 conn(0x7f85e81016e0 msgr2=0x7f85e8102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 shutdown_connections 2024-03-19T02:17:56.695 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:56.052+0000 7f85ee8c2700 1 -- 172.21.15.116:0/3923921114 wait complete. 2024-03-19T02:17:57.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:57 smithi116 ceph-mon[32263]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:17:57.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:57 smithi116 ceph-mon[32263]: Saving service mgr spec with placement count:2 2024-03-19T02:17:57.146 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:57 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:58.369 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.038+0000 7f7d72897700 1 Processor -- start 2024-03-19T02:17:58.369 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.038+0000 7f7d72897700 1 -- start start 2024-03-19T02:17:58.369 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.038+0000 7f7d72897700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.038+0000 7f7d72897700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1084a0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53718/0 (socket says 172.21.15.116:53718) 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6bfff700 1 -- 172.21.15.116:0/3760662716 learned_addr learned my addr 172.21.15.116:0/3760662716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6bfff700 1 -- 172.21.15.116:0/3760662716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c108cb0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6bfff700 1 --2- 172.21.15.116:0/3760662716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0098a0 tx=0x7f7d5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc31e5592ed3774 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6affd700 1 -- 172.21.15.116:0/3760662716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d5c004160 con 0x7f7d6c107ab0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6affd700 1 -- 172.21.15.116:0/3760662716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7d5c004300 con 0x7f7d6c107ab0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.039+0000 7f7d6affd700 1 -- 172.21.15.116:0/3760662716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d5c011a80 con 0x7f7d6c107ab0 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- 172.21.15.116:0/3760662716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 msgr2=0x7f7d6c107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 --2- 172.21.15.116:0/3760662716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0098a0 tx=0x7f7d5c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- 172.21.15.116:0/3760662716 shutdown_connections 2024-03-19T02:17:58.370 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 --2- 172.21.15.116:0/3760662716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c107ed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- 172.21.15.116:0/3760662716 >> 172.21.15.116:0/3760662716 conn(0x7f7d6c1030b0 msgr2=0x7f7d6c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- 172.21.15.116:0/3760662716 shutdown_connections 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- 172.21.15.116:0/3760662716 wait complete. 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 Processor -- start 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.040+0000 7f7d72897700 1 -- start start 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d72897700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d72897700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c10c060 con 0x7f7d6c107ab0 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53728/0 (socket says 172.21.15.116:53728) 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d6bfff700 1 -- 172.21.15.116:0/4081918637 learned_addr learned my addr 172.21.15.116:0/4081918637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d6bfff700 1 -- 172.21.15.116:0/4081918637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c19be30 con 0x7f7d6c107ab0 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d6bfff700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000c00 tx=0x7f7d5c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:58.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d5c0039f0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7d5c003b90 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c19c0c0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.041+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d5c01a740 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.042+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c19edb0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.042+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7d5c01a9f0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.042+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d6c063940 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.043+0000 7f7d697fa700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 0x7f7d5403b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.043+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d5c04d1b0 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.043+0000 7f7d6b7fe700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 0x7f7d5403b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.043+0000 7f7d6b7fe700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 0x7f7d5403b4f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d60006fd0 tx=0x7f7d60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.048+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d5c004300 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.248+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7d6c19f400 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.253+0000 7f7d697fa700 1 -- 172.21.15.116:0/4081918637 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f7d5c020c30 con 0x7f7d6c107ab0 2024-03-19T02:17:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 msgr2=0x7f7d5403b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 0x7f7d5403b4f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d60006fd0 tx=0x7f7d60006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 msgr2=0x7f7d6c19b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000c00 tx=0x7f7d5c011b00 comp rx=0 tx=0).stop 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 shutdown_connections 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f7d54039030 0x7f7d5403b4f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 --2- 172.21.15.116:0/4081918637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d6c107ab0 0x7f7d6c19b8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 >> 172.21.15.116:0/4081918637 conn(0x7f7d6c1030b0 msgr2=0x7f7d6c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 shutdown_connections 2024-03-19T02:17:58.373 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:57.256+0000 7f7d72897700 1 -- 172.21.15.116:0/4081918637 wait complete. 2024-03-19T02:17:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4081918637' entity='client.admin' 2024-03-19T02:17:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:58 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.789+0000 7fad731ad700 1 Processor -- start 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.789+0000 7fad731ad700 1 -- start start 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.789+0000 7fad731ad700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.789+0000 7fad731ad700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c106750 con 0x7fad6c105d60 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad70f49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad70f49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40890/0 (socket says 172.21.15.116:40890) 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad70f49700 1 -- 172.21.15.116:0/87613733 learned_addr learned my addr 172.21.15.116:0/87613733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:59.619 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad70f49700 1 -- 172.21.15.116:0/87613733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c1068d0 con 0x7fad6c105d60 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad70f49700 1 --2- 172.21.15.116:0/87613733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad600098a0 tx=0x7fad60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f961f8acb8963af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad6b7fe700 1 -- 172.21.15.116:0/87613733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad60004160 con 0x7fad6c105d60 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad6b7fe700 1 -- 172.21.15.116:0/87613733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad60004300 con 0x7fad6c105d60 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.790+0000 7fad6b7fe700 1 -- 172.21.15.116:0/87613733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad60011a50 con 0x7fad6c105d60 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 -- 172.21.15.116:0/87613733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 msgr2=0x7fad6c106180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 --2- 172.21.15.116:0/87613733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad600098a0 tx=0x7fad60009bb0 comp rx=0 tx=0).stop 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 -- 172.21.15.116:0/87613733 shutdown_connections 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 --2- 172.21.15.116:0/87613733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c106180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 -- 172.21.15.116:0/87613733 >> 172.21.15.116:0/87613733 conn(0x7fad6c101720 msgr2=0x7fad6c103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 -- 172.21.15.116:0/87613733 shutdown_connections 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.791+0000 7fad731ad700 1 -- 172.21.15.116:0/87613733 wait complete. 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.792+0000 7fad731ad700 1 Processor -- start 2024-03-19T02:17:59.620 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.792+0000 7fad731ad700 1 -- start start 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.792+0000 7fad731ad700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.792+0000 7fad731ad700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c10c010 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad70f49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad70f49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40894/0 (socket says 172.21.15.116:40894) 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad70f49700 1 -- 172.21.15.116:0/373980004 learned_addr learned my addr 172.21.15.116:0/373980004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad70f49700 1 -- 172.21.15.116:0/373980004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c19bdb0 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad70f49700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad60006ba0 tx=0x7fad60011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad60003af0 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c19c040 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad6001a8e0 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad60024900 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.793+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c19ed30 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.794+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fad60022020 con 0x7fad6c105d60 2024-03-19T02:17:59.621 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.795+0000 7fad69ffb700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 0x7fad5c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.795+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6c19c520 con 0x7fad6c105d60 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.795+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fad6004e170 con 0x7fad6c105d60 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.795+0000 7fad6bfff700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 0x7fad5c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.795+0000 7fad6bfff700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 0x7fad5c03b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fad58006fd0 tx=0x7fad58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:58.804+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad6c19c520 con 0x7fad6c105d60 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.008+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fad6c19ef00 con 0x7fad6c105d60 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.010+0000 7fad69ffb700 1 -- 172.21.15.116:0/373980004 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fad6000b3e0 con 0x7fad6c105d60 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 msgr2=0x7fad5c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 0x7fad5c03b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fad58006fd0 tx=0x7fad58006e40 comp rx=0 tx=0).stop 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 msgr2=0x7fad6c19b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad60006ba0 tx=0x7fad60011ad0 comp rx=0 tx=0).stop 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 shutdown_connections 2024-03-19T02:17:59.622 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fad5c038f90 0x7fad5c03b450 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 --2- 172.21.15.116:0/373980004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad6c105d60 0x7fad6c19b870 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:17:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.012+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 >> 172.21.15.116:0/373980004 conn(0x7fad6c101720 msgr2=0x7fad6c102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:17:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.013+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 shutdown_connections 2024-03-19T02:17:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:17:59.013+0000 7fad731ad700 1 -- 172.21.15.116:0/373980004 wait complete. 2024-03-19T02:17:59.623 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-19T02:17:59.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:59 smithi116 ceph-mon[32263]: from='mgr.14120 172.21.15.116:0/3365854731' entity='mgr.x' 2024-03-19T02:17:59.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:17:59 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/373980004' entity='client.admin' 2024-03-19T02:18:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2935262125' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-19T02:18:01.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setuser ceph since I am not root 2024-03-19T02:18:01.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setgroup ceph since I am not root 2024-03-19T02:18:01.335 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:01.215+0000 7f5b8ccaf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:18:01.336 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:01.308+0000 7f5b8ccaf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:18:01.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.070+0000 7f9c41ed4700 1 Processor -- start 2024-03-19T02:18:01.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.070+0000 7f9c41ed4700 1 -- start start 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.070+0000 7f9c41ed4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c41ed4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c07be80 con 0x7f9c3c108500 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40908/0 (socket says 172.21.15.116:40908) 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3b7fe700 1 -- 172.21.15.116:0/1021686443 learned_addr learned my addr 172.21.15.116:0/1021686443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3b7fe700 1 -- 172.21.15.116:0/1021686443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c10ae30 con 0x7f9c3c108500 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3b7fe700 1 --2- 172.21.15.116:0/1021686443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0098a0 tx=0x7f9c2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15aafe91c3b2a415 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3affd700 1 -- 172.21.15.116:0/1021686443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c2c004160 con 0x7f9c3c108500 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3affd700 1 -- 172.21.15.116:0/1021686443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c2c004300 con 0x7f9c3c108500 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.071+0000 7f9c3affd700 1 -- 172.21.15.116:0/1021686443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c2c011a50 con 0x7f9c3c108500 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 -- 172.21.15.116:0/1021686443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 msgr2=0x7f9c3c10a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/1021686443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0098a0 tx=0x7f9c2c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 -- 172.21.15.116:0/1021686443 shutdown_connections 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/1021686443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c10a8f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 -- 172.21.15.116:0/1021686443 >> 172.21.15.116:0/1021686443 conn(0x7f9c3c101f30 msgr2=0x7f9c3c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 -- 172.21.15.116:0/1021686443 shutdown_connections 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.072+0000 7f9c41ed4700 1 -- 172.21.15.116:0/1021686443 wait complete. 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c41ed4700 1 Processor -- start 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c41ed4700 1 -- start start 2024-03-19T02:18:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c41ed4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c41ed4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c10be40 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40924/0 (socket says 172.21.15.116:40924) 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c3b7fe700 1 -- 172.21.15.116:0/2935262125 learned_addr learned my addr 172.21.15.116:0/2935262125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c3b7fe700 1 -- 172.21.15.116:0/2935262125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c19bd90 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c3b7fe700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c011a20 tx=0x7f9c2c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.073+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c2c003b70 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c2c01a8e0 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c19c020 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c2c024990 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c19ed10 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9c2c022020 con 0x7f9c3c108500 2024-03-19T02:18:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.074+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c063940 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.075+0000 7f9c397fa700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 0x7f9c2803b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.075+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9c2c04d680 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.075+0000 7f9c32ffd700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 0x7f9c2803b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.075+0000 7f9c32ffd700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 0x7f9c2803b450 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9c24006fd0 tx=0x7f9c24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.080+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c2c02dc90 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.353+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f9c3c19f2c0 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.652+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f9c2c04fc00 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.654+0000 7f9c397fa700 1 -- 172.21.15.116:0/2935262125 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f9c2c00b3e0 con 0x7f9c3c108500 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 msgr2=0x7f9c2803b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 0x7f9c2803b450 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9c24006fd0 tx=0x7f9c24006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 msgr2=0x7f9c3c19b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c011a20 tx=0x7f9c2c011b00 comp rx=0 tx=0).stop 2024-03-19T02:18:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 shutdown_connections 2024-03-19T02:18:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f9c28038f90 0x7f9c2803b450 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 --2- 172.21.15.116:0/2935262125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c3c108500 0x7f9c3c19b850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 >> 172.21.15.116:0/2935262125 conn(0x7f9c3c101f30 msgr2=0x7f9c3c1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 shutdown_connections 2024-03-19T02:18:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:00.655+0000 7f9c41ed4700 1 -- 172.21.15.116:0/2935262125 wait complete. 2024-03-19T02:18:01.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:01 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2935262125' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-19T02:18:01.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:01 smithi116 ceph-mon[32263]: mgrmap e9: x(active, since 18s) 2024-03-19T02:18:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:01.980+0000 7f5b8ccaf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:18:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.075+0000 7f5b8ccaf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:18:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.223+0000 7f5b8ccaf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:18:02.654 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.320+0000 7f5b8ccaf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:18:02.654 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.404+0000 7f5b8ccaf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.798+0000 7fe280246700 1 Processor -- start 2024-03-19T02:18:02.700 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.798+0000 7fe280246700 1 -- start start 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.798+0000 7fe280246700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.798+0000 7fe280246700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe27807c8a0 con 0x7fe27807de70 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27dfe2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27dfe2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40952/0 (socket says 172.21.15.116:40952) 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27dfe2700 1 -- 172.21.15.116:0/3003044063 learned_addr learned my addr 172.21.15.116:0/3003044063 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27dfe2700 1 -- 172.21.15.116:0/3003044063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27807ca20 con 0x7fe27807de70 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27dfe2700 1 --2- 172.21.15.116:0/3003044063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a346941c50d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27cfe0700 1 -- 172.21.15.116:0/3003044063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe268004160 con 0x7fe27807de70 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27cfe0700 1 -- 172.21.15.116:0/3003044063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe268004300 con 0x7fe27807de70 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.799+0000 7fe27cfe0700 1 -- 172.21.15.116:0/3003044063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe268011a50 con 0x7fe27807de70 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 -- 172.21.15.116:0/3003044063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 msgr2=0x7fe27807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 --2- 172.21.15.116:0/3003044063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2680098a0 tx=0x7fe268009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 -- 172.21.15.116:0/3003044063 shutdown_connections 2024-03-19T02:18:02.701 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 --2- 172.21.15.116:0/3003044063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27807c2d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 -- 172.21.15.116:0/3003044063 >> 172.21.15.116:0/3003044063 conn(0x7fe2781030b0 msgr2=0x7fe2781054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 -- 172.21.15.116:0/3003044063 shutdown_connections 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.800+0000 7fe280246700 1 -- 172.21.15.116:0/3003044063 wait complete. 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe280246700 1 Processor -- start 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe280246700 1 -- start start 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe280246700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe280246700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe27810e0b0 con 0x7fe27807de70 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe27dfe2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe27dfe2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40966/0 (socket says 172.21.15.116:40966) 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe27dfe2700 1 -- 172.21.15.116:0/601333438 learned_addr learned my addr 172.21.15.116:0/601333438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe27dfe2700 1 -- 172.21.15.116:0/601333438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27819dfa0 con 0x7fe27807de70 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe27dfe2700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe268006ba0 tx=0x7fe268011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.801+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe268003af0 con 0x7fe27807de70 2024-03-19T02:18:02.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.802+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe26801a8e0 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.802+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27819e230 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.802+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe268024900 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.802+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781a0f20 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.802+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fe268022020 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.803+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe278063940 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.803+0000 7fe26effd700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 0x7fe26403b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.803+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe26804d430 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.803+0000 7fe27d7e1700 1 -- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 msgr2=0x7fe26403b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.803+0000 7fe27d7e1700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 0x7fe26403b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:01.808+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe26801ab00 con 0x7fe27807de70 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.004+0000 7fe27d7e1700 1 -- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 msgr2=0x7fe26403b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.004+0000 7fe27d7e1700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 0x7fe26403b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.075+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe2781a10f0 con 0x7fe27807de70 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.076+0000 7fe26effd700 1 -- 172.21.15.116:0/601333438 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe26802dc90 con 0x7fe27807de70 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 msgr2=0x7fe26403b500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 0x7fe26403b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 msgr2=0x7fe27819da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe268006ba0 tx=0x7fe268011ad0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 shutdown_connections 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7fe264039040 0x7fe26403b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 --2- 172.21.15.116:0/601333438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe27807de70 0x7fe27819da60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 >> 172.21.15.116:0/601333438 conn(0x7fe2781030b0 msgr2=0x7fe278103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 shutdown_connections 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:02.078+0000 7fe280246700 1 -- 172.21.15.116:0/601333438 wait complete. 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-19T02:18:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-19T02:18:02.915 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/601333438' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T02:18:02.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.668+0000 7f5b8ccaf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:18:02.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.781+0000 7f5b8ccaf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:18:02.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.868+0000 7f5b8ccaf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:18:03.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:02.967+0000 7f5b8ccaf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:18:03.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:03.147+0000 7f5b8ccaf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:18:04.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:04.091+0000 7f5b8ccaf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:18:04.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:04.218+0000 7f5b8ccaf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:18:04.826 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:04.546+0000 7f5b8ccaf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:18:04.827 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:04.634+0000 7f5b8ccaf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:18:04.827 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:04.826+0000 7f5b8ccaf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:18:05.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:05.009+0000 7f5b8ccaf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:18:05.541 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:05.288+0000 7f5b8ccaf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:18:06.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:05.541+0000 7f5b8ccaf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:18:06.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:05.775+0000 7f5b8ccaf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:18:07.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:06.584+0000 7f5b8ccaf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:18:07.498 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:07.178+0000 7f5b8ccaf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:18:07.498 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:07.277+0000 7f5b8ccaf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:18:07.498 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:18:07.498+0000 7f5b8ccaf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:18:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: Active manager daemon x restarted 2024-03-19T02:18:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: Activating manager daemon x 2024-03-19T02:18:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: osdmap e3: 0 total, 0 up, 0 in 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: mgrmap e10: x(active, starting, since 0.00366505s) 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: Manager daemon x is now available 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:18:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:18:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:08 smithi116 ceph-mon[32263]: mgrmap e11: x(active, since 1.00615s) 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.123+0000 7f4645733700 1 Processor -- start 2024-03-19T02:18:09.342 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.123+0000 7f4645733700 1 -- start start 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.123+0000 7f4645733700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.123+0000 7f4645733700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4640106780 con 0x7f4640105d90 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40982/0 (socket says 172.21.15.116:40982) 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463effd700 1 -- 172.21.15.116:0/2347432966 learned_addr learned my addr 172.21.15.116:0/2347432966 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463effd700 1 -- 172.21.15.116:0/2347432966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4640108d20 con 0x7f4640105d90 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463effd700 1 --2- 172.21.15.116:0/2347432966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46340098a0 tx=0x7f4634009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d129dd62e4e61b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463dffb700 1 -- 172.21.15.116:0/2347432966 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4634004160 con 0x7f4640105d90 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463dffb700 1 -- 172.21.15.116:0/2347432966 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4634004300 con 0x7f4640105d90 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.124+0000 7f463dffb700 1 -- 172.21.15.116:0/2347432966 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4634011a50 con 0x7f4640105d90 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 -- 172.21.15.116:0/2347432966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 msgr2=0x7f46401061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 --2- 172.21.15.116:0/2347432966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46340098a0 tx=0x7f4634009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 -- 172.21.15.116:0/2347432966 shutdown_connections 2024-03-19T02:18:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 --2- 172.21.15.116:0/2347432966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401061b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 -- 172.21.15.116:0/2347432966 >> 172.21.15.116:0/2347432966 conn(0x7f4640101720 msgr2=0x7f4640103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 -- 172.21.15.116:0/2347432966 shutdown_connections 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.125+0000 7f4645733700 1 -- 172.21.15.116:0/2347432966 wait complete. 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f4645733700 1 Processor -- start 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f4645733700 1 -- start start 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f4645733700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f4645733700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f464010c0d0 con 0x7f4640105d90 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f463effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f463effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40986/0 (socket says 172.21.15.116:40986) 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f463effd700 1 -- 172.21.15.116:0/154705908 learned_addr learned my addr 172.21.15.116:0/154705908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.126+0000 7f463effd700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4640108ae0 con 0x7f4640105d90 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f463effd700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4634006ba0 tx=0x7f4634011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4634003af0 con 0x7f4640105d90 2024-03-19T02:18:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4640106cd0 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f463401a8e0 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4640107210 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.127+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4634024900 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f4634022020 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f4627fff700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f462803bc10 con 0x7f4628039040 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f463404d250 con 0x7f4640105d90 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f463e7fc700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.128+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.329+0000 7f463e7fc700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.329+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.730+0000 7f463e7fc700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:09.345 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:03.730+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:04.531+0000 7f463e7fc700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:04.531+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:06.132+0000 7f463e7fc700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/647665947 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:06.132+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:07.513+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f463402d3f0 con 0x7f4640105d90 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:07.513+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 msgr2=0x7f462803b500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:07.513+0000 7f4627fff700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.515+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f463404e800 con 0x7f4640105d90 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.515+0000 7f4627fff700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 0x7f462803eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.515+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f463400ae50 con 0x7f462803caf0 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.515+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 0x7f462803eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.517+0000 7f463e7fc700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 0x7f462803eee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4630003a10 tx=0x7f4630009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:09.346 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.517+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f463400ae50 con 0x7f462803caf0 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4640063940 con 0x7f462803caf0 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4627fff700 1 -- 172.21.15.116:0/154705908 <== mgr.14152 v2:172.21.15.116:6800/3573130182 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f4640063940 con 0x7f462803caf0 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 msgr2=0x7f462803eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 0x7f462803eee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4630003a10 tx=0x7f4630009270 comp rx=0 tx=0).stop 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 msgr2=0x7f46401085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4634006ba0 tx=0x7f4634011ad0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 shutdown_connections 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f462803caf0 0x7f462803eee0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4640105d90 0x7f46401085a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 --2- 172.21.15.116:0/154705908 >> [v2:172.21.15.116:6800/647665947,v1:172.21.15.116:6801/647665947] conn(0x7f4628039040 0x7f462803b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 >> 172.21.15.116:0/154705908 conn(0x7f4640101720 msgr2=0x7f4640101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.524+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 shutdown_connections 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:08.525+0000 7f4645733700 1 -- 172.21.15.116:0/154705908 wait complete. 2024-03-19T02:18:09.347 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-19T02:18:09.348 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-19T02:18:09.639 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: [19/Mar/2024:02:18:07] ENGINE Bus STARTING 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: [19/Mar/2024:02:18:07] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: [19/Mar/2024:02:18:07] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: [19/Mar/2024:02:18:07] ENGINE Bus STARTED 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T02:18:09.640 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:10.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-19T02:18:10.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.836+0000 7f93c852c700 1 Processor -- start 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.836+0000 7f93c852c700 1 -- start start 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c852c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c852c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93c0106750 con 0x7f93c0105d60 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c62c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c62c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36456/0 (socket says 172.21.15.116:36456) 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c62c8700 1 -- 172.21.15.116:0/3217649050 learned_addr learned my addr 172.21.15.116:0/3217649050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c62c8700 1 -- 172.21.15.116:0/3217649050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c0108cf0 con 0x7f93c0105d60 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c62c8700 1 --2- 172.21.15.116:0/3217649050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f93b40098a0 tx=0x7f93b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a085713ab00e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c52c6700 1 -- 172.21.15.116:0/3217649050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b4004160 con 0x7f93c0105d60 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c52c6700 1 -- 172.21.15.116:0/3217649050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b4004300 con 0x7f93c0105d60 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.837+0000 7f93c52c6700 1 -- 172.21.15.116:0/3217649050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b4011a50 con 0x7f93c0105d60 2024-03-19T02:18:10.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 -- 172.21.15.116:0/3217649050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 msgr2=0x7f93c0106180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 --2- 172.21.15.116:0/3217649050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f93b40098a0 tx=0x7f93b4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 -- 172.21.15.116:0/3217649050 shutdown_connections 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 --2- 172.21.15.116:0/3217649050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c0106180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 -- 172.21.15.116:0/3217649050 >> 172.21.15.116:0/3217649050 conn(0x7f93c0101690 msgr2=0x7f93c0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 -- 172.21.15.116:0/3217649050 shutdown_connections 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.838+0000 7f93c852c700 1 -- 172.21.15.116:0/3217649050 wait complete. 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c852c700 1 Processor -- start 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c852c700 1 -- start start 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c852c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c852c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010c0a0 con 0x7f93c0105d60 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c62c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c62c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36460/0 (socket says 172.21.15.116:36460) 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c62c8700 1 -- 172.21.15.116:0/1196831446 learned_addr learned my addr 172.21.15.116:0/1196831446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c62c8700 1 -- 172.21.15.116:0/1196831446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c019be60 con 0x7f93c0105d60 2024-03-19T02:18:10.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c62c8700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93b4011a20 tx=0x7f93b4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b4003af0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b401a8e0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b4024860 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.839+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c019c0f0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.840+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c019ede0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.840+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f93b4022020 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.840+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c0063940 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.840+0000 7f93b37fe700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 0x7f93ac03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.841+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f93b404cfe0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.841+0000 7f93c5ac7700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 0x7f93ac03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.841+0000 7f93c5ac7700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 0x7f93ac03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:09.845+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93b4016260 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.031+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f93b402d3f0 con 0x7f93c0105d60 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.079+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f93c01027a0 con 0x7f93ac038f80 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.175+0000 7f93b37fe700 1 -- 172.21.15.116:0/1196831446 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f93c01027a0 con 0x7f93ac038f80 2024-03-19T02:18:10.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 msgr2=0x7f93ac03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 0x7f93ac03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f93bc006fd0 tx=0x7f93bc006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 msgr2=0x7f93c019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93b4011a20 tx=0x7f93b4011b00 comp rx=0 tx=0).stop 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 shutdown_connections 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93ac038f80 0x7f93ac03b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 --2- 172.21.15.116:0/1196831446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c0105d60 0x7f93c019b920 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 >> 172.21.15.116:0/1196831446 conn(0x7f93c0101690 msgr2=0x7f93c0102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 shutdown_connections 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:10.177+0000 7f93c852c700 1 -- 172.21.15.116:0/1196831446 wait complete. 2024-03-19T02:18:10.745 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-19T02:18:11.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:11 smithi116 ceph-mon[32263]: mgrmap e12: x(active, since 2s) 2024-03-19T02:18:11.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:11 smithi116 ceph-mon[32263]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:11.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:11.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:11.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:12.386 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Lpn6DFCOqDqHmVA9ridraOjsEEPjgr3z01KYIreJmdhu7DlnxV1n6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710814691, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-19T02:18:12.386 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.160+0000 7f7b4ffe6700 1 Processor -- start 2024-03-19T02:18:12.386 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.160+0000 7f7b4ffe6700 1 -- start start 2024-03-19T02:18:12.386 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4ffe6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:12.386 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4ffe6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48107380 con 0x7f7b48106990 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4dd82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4dd82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36472/0 (socket says 172.21.15.116:36472) 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4dd82700 1 -- 172.21.15.116:0/804918214 learned_addr learned my addr 172.21.15.116:0/804918214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4dd82700 1 -- 172.21.15.116:0/804918214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b48107b90 con 0x7f7b48106990 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.161+0000 7f7b4dd82700 1 --2- 172.21.15.116:0/804918214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bb40 tx=0x7f7b3800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277baf7a348cd310 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4cd80700 1 -- 172.21.15.116:0/804918214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b3800dbe0 con 0x7f7b48106990 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4cd80700 1 -- 172.21.15.116:0/804918214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b380044e0 con 0x7f7b48106990 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4cd80700 1 -- 172.21.15.116:0/804918214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b38010470 con 0x7f7b48106990 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/804918214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 msgr2=0x7f7b48106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/804918214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bb40 tx=0x7f7b3800be50 comp rx=0 tx=0).stop 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/804918214 shutdown_connections 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/804918214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b48106db0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:12.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/804918214 >> 172.21.15.116:0/804918214 conn(0x7f7b48101f30 msgr2=0x7f7b48104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/804918214 shutdown_connections 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.162+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/804918214 wait complete. 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4ffe6700 1 Processor -- start 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4ffe6700 1 -- start start 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4ffe6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4ffe6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4807afa0 con 0x7f7b48106990 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4dd82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4dd82700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36484/0 (socket says 172.21.15.116:36484) 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.163+0000 7f7b4dd82700 1 -- 172.21.15.116:0/1400361409 learned_addr learned my addr 172.21.15.116:0/1400361409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b4dd82700 1 -- 172.21.15.116:0/1400361409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4819e000 con 0x7f7b48106990 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b4dd82700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d650 tx=0x7f7b38003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:12.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b380108a0 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b3801d070 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4819e290 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b38021c60 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.164+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4819e690 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f7b3801f030 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b48063940 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b3effd700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 0x7f7b3403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7b3804d560 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b4d581700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 0x7f7b3403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.165+0000 7f7b4d581700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 0x7f7b3403b4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006fd0 tx=0x7f7b44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.170+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b38015070 con 0x7f7b48106990 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.402+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f7b481a1100 con 0x7f7b34038fe0 2024-03-19T02:18:12.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.626+0000 7f7b3effd700 1 -- 172.21.15.116:0/1400361409 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f7b481a1100 con 0x7f7b34038fe0 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 msgr2=0x7f7b3403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 0x7f7b3403b4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006fd0 tx=0x7f7b44006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 msgr2=0x7f7b4819dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800d650 tx=0x7f7b38003fb0 comp rx=0 tx=0).stop 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 shutdown_connections 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b34038fe0 0x7f7b3403b4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 --2- 172.21.15.116:0/1400361409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b48106990 0x7f7b4819dac0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 >> 172.21.15.116:0/1400361409 conn(0x7f7b48101f30 msgr2=0x7f7b481940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 shutdown_connections 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:11.628+0000 7f7b4ffe6700 1 -- 172.21.15.116:0/1400361409 wait complete. 2024-03-19T02:18:12.390 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-19T02:18:12.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:12 smithi116 ceph-mon[32263]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:12.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:13 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4009062265' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-19T02:18:14.052 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f91722a1700 1 Processor -- start 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f91722a1700 1 -- start start 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f91722a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f91722a1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f916c106730 con 0x7f916c105d40 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f916bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f916bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36486/0 (socket says 172.21.15.116:36486) 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.233+0000 7f916bfff700 1 -- 172.21.15.116:0/3208446252 learned_addr learned my addr 172.21.15.116:0/3208446252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.234+0000 7f916bfff700 1 -- 172.21.15.116:0/3208446252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c108cd0 con 0x7f916c105d40 2024-03-19T02:18:14.053 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.234+0000 7f916bfff700 1 --2- 172.21.15.116:0/3208446252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f915c0098a0 tx=0x7f915c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183a92a779fa756c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.234+0000 7f916affd700 1 -- 172.21.15.116:0/3208446252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f915c004160 con 0x7f916c105d40 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.234+0000 7f916affd700 1 -- 172.21.15.116:0/3208446252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f915c004300 con 0x7f916c105d40 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.234+0000 7f916affd700 1 -- 172.21.15.116:0/3208446252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f915c011a50 con 0x7f916c105d40 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- 172.21.15.116:0/3208446252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 msgr2=0x7f916c106160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 --2- 172.21.15.116:0/3208446252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f915c0098a0 tx=0x7f915c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- 172.21.15.116:0/3208446252 shutdown_connections 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 --2- 172.21.15.116:0/3208446252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c106160 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- 172.21.15.116:0/3208446252 >> 172.21.15.116:0/3208446252 conn(0x7f916c1016b0 msgr2=0x7f916c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- 172.21.15.116:0/3208446252 shutdown_connections 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- 172.21.15.116:0/3208446252 wait complete. 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 Processor -- start 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.235+0000 7f91722a1700 1 -- start start 2024-03-19T02:18:14.054 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91722a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91722a1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f916c10c080 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f916bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f916bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36492/0 (socket says 172.21.15.116:36492) 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f916bfff700 1 -- 172.21.15.116:0/4009062265 learned_addr learned my addr 172.21.15.116:0/4009062265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f916bfff700 1 -- 172.21.15.116:0/4009062265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c19be50 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f916bfff700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f915c011a20 tx=0x7f915c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f915c003af0 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f915c01a8e0 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f915c024860 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.236+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916c19c0e0 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.237+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916c19edd0 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.237+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f915c022020 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.237+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f916c063940 con 0x7f916c105d40 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.238+0000 7f91697fa700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 0x7f915403b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:14.055 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.238+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f915c04d230 con 0x7f916c105d40 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.238+0000 7f916b7fe700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 0x7f915403b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.238+0000 7f916b7fe700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 0x7f915403b550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9160006fd0 tx=0x7f9160006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.243+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f915c0094f0 con 0x7f916c105d40 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.436+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f916c19f380 con 0x7f916c105d40 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.437+0000 7f91697fa700 1 -- 172.21.15.116:0/4009062265 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f915c02ad60 con 0x7f916c105d40 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.438+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 msgr2=0x7f915403b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.438+0000 7f91722a1700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 0x7f915403b550 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9160006fd0 tx=0x7f9160006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.438+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 msgr2=0x7f916c19b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.438+0000 7f91722a1700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f915c011a20 tx=0x7f915c011b00 comp rx=0 tx=0).stop 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 shutdown_connections 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9154039090 0x7f915403b550 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 --2- 172.21.15.116:0/4009062265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f916c105d40 0x7f916c19b910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 >> 172.21.15.116:0/4009062265 conn(0x7f916c1016b0 msgr2=0x7f916c1020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:14.056 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 shutdown_connections 2024-03-19T02:18:14.057 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:13.439+0000 7f91722a1700 1 -- 172.21.15.116:0/4009062265 wait complete. 2024-03-19T02:18:14.059 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:18:14.059 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T02:18:14.066 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:18:14.066 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T02:18:14.066 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T02:18:14.067 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-19T02:18:14.068 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-19T02:18:14.068 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:14.068 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-19T02:18:14.069 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-19T02:18:14.069 INFO:teuthology.orchestra.run.smithi116.stdout: Password: tc25iktyvj 2024-03-19T02:18:14.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:14.069 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config directory 2024-03-19T02:18:14.069 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-19T02:18:14.966 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:14 smithi116 ceph-mon[32263]: mgrmap e13: x(active, since 6s) 2024-03-19T02:18:15.250 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73ead16700 1 Processor -- start 2024-03-19T02:18:15.250 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73ead16700 1 -- start start 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73ead16700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73ead16700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4107390 con 0x7f73e41069a0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73e8ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73e8ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36500/0 (socket says 172.21.15.116:36500) 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73e8ab2700 1 -- 172.21.15.116:0/3752805171 learned_addr learned my addr 172.21.15.116:0/3752805171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.456+0000 7f73e8ab2700 1 -- 172.21.15.116:0/3752805171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e4107ba0 con 0x7f73e41069a0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73e8ab2700 1 --2- 172.21.15.116:0/3752805171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73d80098a0 tx=0x7f73d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bfaf10edfea0bf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73e37fe700 1 -- 172.21.15.116:0/3752805171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73d8004160 con 0x7f73e41069a0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73e37fe700 1 -- 172.21.15.116:0/3752805171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73d8004300 con 0x7f73e41069a0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73e37fe700 1 -- 172.21.15.116:0/3752805171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73d8011a50 con 0x7f73e41069a0 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 -- 172.21.15.116:0/3752805171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 msgr2=0x7f73e4106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 --2- 172.21.15.116:0/3752805171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73d80098a0 tx=0x7f73d8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:15.251 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 -- 172.21.15.116:0/3752805171 shutdown_connections 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 --2- 172.21.15.116:0/3752805171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e4106dc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 -- 172.21.15.116:0/3752805171 >> 172.21.15.116:0/3752805171 conn(0x7f73e4101f40 msgr2=0x7f73e4104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 -- 172.21.15.116:0/3752805171 shutdown_connections 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.457+0000 7f73ead16700 1 -- 172.21.15.116:0/3752805171 wait complete. 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73ead16700 1 Processor -- start 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73ead16700 1 -- start start 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73ead16700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73ead16700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73e410af50 con 0x7f73e41069a0 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73e8ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73e8ab2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36508/0 (socket says 172.21.15.116:36508) 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73e8ab2700 1 -- 172.21.15.116:0/3940653622 learned_addr learned my addr 172.21.15.116:0/3940653622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.458+0000 7f73e8ab2700 1 -- 172.21.15.116:0/3940653622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e419be60 con 0x7f73e41069a0 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73e8ab2700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f73d8011710 tx=0x7f73d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73d80039c0 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73d8003b60 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e419c0f0 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73d801a690 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.459+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e419ede0 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f73d801a940 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e4063940 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73e1ffb700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 0x7f73d403b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f73d804d870 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73e3fff700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 0x7f73d403b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.460+0000 7f73e3fff700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 0x7f73d403b5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006fd0 tx=0x7f73d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.465+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f73d8019ca0 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.660+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f73e402de70 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.660+0000 7f73e1ffb700 1 -- 172.21.15.116:0/3940653622 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f73d802c350 con 0x7f73e41069a0 2024-03-19T02:18:15.253 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 msgr2=0x7f73d403b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 0x7f73d403b5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f73d0006fd0 tx=0x7f73d0006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 msgr2=0x7f73e419b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f73d8011710 tx=0x7f73d8011ad0 comp rx=0 tx=0).stop 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 shutdown_connections 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73d40390e0 0x7f73d403b5a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 --2- 172.21.15.116:0/3940653622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e41069a0 0x7f73e419b920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.662+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 >> 172.21.15.116:0/3940653622 conn(0x7f73e4101f40 msgr2=0x7f73e4102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.663+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 shutdown_connections 2024-03-19T02:18:15.254 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:14.663+0000 7f73ead16700 1 -- 172.21.15.116:0/3940653622 wait complete. 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.646+0000 7f85057ba700 1 Processor -- start 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.646+0000 7f85057ba700 1 -- start start 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.646+0000 7f85057ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.646+0000 7f85057ba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f850007be20 con 0x7f8500106370 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84feffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84feffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36512/0 (socket says 172.21.15.116:36512) 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84feffd700 1 -- 172.21.15.116:0/854451977 learned_addr learned my addr 172.21.15.116:0/854451977 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84feffd700 1 -- 172.21.15.116:0/854451977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8500106cd0 con 0x7f8500106370 2024-03-19T02:18:16.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84feffd700 1 --2- 172.21.15.116:0/854451977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84f00098a0 tx=0x7f84f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274f6b2800cb1d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84fe7fc700 1 -- 172.21.15.116:0/854451977 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f0004160 con 0x7f8500106370 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84fe7fc700 1 -- 172.21.15.116:0/854451977 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84f0004300 con 0x7f8500106370 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.647+0000 7f84fe7fc700 1 -- 172.21.15.116:0/854451977 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f0011a50 con 0x7f8500106370 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 -- 172.21.15.116:0/854451977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 msgr2=0x7f8500106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 --2- 172.21.15.116:0/854451977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84f00098a0 tx=0x7f84f0009bb0 comp rx=0 tx=0).stop 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 -- 172.21.15.116:0/854451977 shutdown_connections 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 --2- 172.21.15.116:0/854451977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f8500106790 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 -- 172.21.15.116:0/854451977 >> 172.21.15.116:0/854451977 conn(0x7f8500101fb0 msgr2=0x7f85001043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 -- 172.21.15.116:0/854451977 shutdown_connections 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.648+0000 7f85057ba700 1 -- 172.21.15.116:0/854451977 wait complete. 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f85057ba700 1 Processor -- start 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f85057ba700 1 -- start start 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f85057ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:16.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f85057ba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8500107d40 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f84feffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f84feffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36528/0 (socket says 172.21.15.116:36528) 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f84feffd700 1 -- 172.21.15.116:0/1295451279 learned_addr learned my addr 172.21.15.116:0/1295451279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f84feffd700 1 -- 172.21.15.116:0/1295451279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850019dfb0 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.649+0000 7f84feffd700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84f0011a20 tx=0x7f84f0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.650+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f0003b70 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.650+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84f001a8e0 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.650+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850019e240 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.650+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84f0024990 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.650+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85001a0f30 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f84f0022020 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8500063940 con 0x7f8500106370 2024-03-19T02:18:16.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f84fcff9700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 0x7f84ec03b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f84f004d4b0 con 0x7f8500106370 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f84f67fc700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 0x7f84ec03b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.651+0000 7f84f67fc700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 0x7f84ec03b5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84e8006fd0 tx=0x7f84e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.656+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84f0024cf0 con 0x7f8500106370 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.962+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f850002de40 con 0x7f8500106370 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.964+0000 7f84fcff9700 1 -- 172.21.15.116:0/1295451279 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f84f002d910 con 0x7f8500106370 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.966+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 msgr2=0x7f84ec03b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.966+0000 7f85057ba700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 0x7f84ec03b5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84e8006fd0 tx=0x7f84e8006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.966+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 msgr2=0x7f850019da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.966+0000 7f85057ba700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84f0011a20 tx=0x7f84f0011b00 comp rx=0 tx=0).stop 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 shutdown_connections 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f84ec0390e0 0x7f84ec03b5a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 --2- 172.21.15.116:0/1295451279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8500106370 0x7f850019da70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:16.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 >> 172.21.15.116:0/1295451279 conn(0x7f8500101fb0 msgr2=0x7f8500194790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 shutdown_connections 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T02:18:15.967+0000 7f85057ba700 1 -- 172.21.15.116:0/1295451279 wait complete. 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-19T02:18:16.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.536 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-19T02:18:16.536 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.536 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-19T02:18:16.536 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:16.536 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-19T02:18:16.569 INFO:tasks.cephadm:Fetching config... 2024-03-19T02:18:16.569 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:18:16.569 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-19T02:18:16.585 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-19T02:18:16.585 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:18:16.585 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-19T02:18:16.641 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-19T02:18:16.641 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:18:16.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-19T02:18:16.708 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-19T02:18:16.708 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:18:16.708 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-19T02:18:16.763 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-19T02:18:16.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:18:16.844 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:16.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:18:16.903 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:16.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:18:16.963 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmBBZZyiWQ6V7UHN96ZfkJEGMeOIEfNIi+nxhCC39VtVSw7thJy+hNcJyq+F6hD+bAXY0wZfcAmDZU4t389dbnS7JvD7+vpSHZu5Mn0GPKK1HJdrJyI4QlGGFTmi5rD1yEWarF4OceK4stb2aIAZy6fNiMtJqLkus4+Op3uKwflKza/M7aDG3C+drE4hiEZjGFkLRf8Gf9QserG2Fl/rEks3wQLjONUtOb6Hki6YnSCFOYJK+MTYHz3MVFfu9TKRMnOWYjpQkKKS2zunN70+MnU4cV/wngHv3FWetHUcqiGnBTmRvnuUzCjO4GW8Y0HPrf4w1AIe8aceBL9vwcaxjWkiG1DSTfN4k1foV8YZDODQn4Y8ly8zD2PXDxfQoVLjlIhEma1lNH1VeqMx4oNmsk2QwWePYTiUJY0jB5AzS0PyyAUC+S5A5GtTHGahiE+LQayPerFtlTWN3q1wccpFcjHMTQw6CP7bQy1yvbhZ9uOZHIDpNXcS2KIOltyX35JTs= ceph-a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:16.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-19T02:18:17.215 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:16 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1295451279' entity='client.admin' 2024-03-19T02:18:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 -- 172.21.15.116:0/3226174990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 msgr2=0x7f6a8c101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 --2- 172.21.15.116:0/3226174990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c101070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009910 tx=0x7f6a88009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 -- 172.21.15.116:0/3226174990 shutdown_connections 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 --2- 172.21.15.116:0/3226174990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c101070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 -- 172.21.15.116:0/3226174990 >> 172.21.15.116:0/3226174990 conn(0x7f6a8c0fc1d0 msgr2=0x7f6a8c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 -- 172.21.15.116:0/3226174990 shutdown_connections 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.445+0000 7f6a941d6700 1 -- 172.21.15.116:0/3226174990 wait complete. 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a941d6700 1 Processor -- start 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a941d6700 1 -- start start 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a941d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a941d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c106b90 con 0x7f6a8c100c50 2024-03-19T02:18:19.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a91f72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a91f72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45308/0 (socket says 172.21.15.116:45308) 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a91f72700 1 -- 172.21.15.116:0/1094555532 learned_addr learned my addr 172.21.15.116:0/1094555532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a91f72700 1 -- 172.21.15.116:0/1094555532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c196810 con 0x7f6a8c100c50 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.446+0000 7f6a91f72700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a88004fb0 tx=0x7f6a88009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a88027070 con 0x7f6a8c100c50 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a8800be10 con 0x7f6a8c100c50 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8c196aa0 con 0x7f6a8c100c50 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a88022de0 con 0x7f6a8c100c50 2024-03-19T02:18:19.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8c199790 con 0x7f6a8c100c50 2024-03-19T02:18:19.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.447+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f6a8802a070 con 0x7f6a8c100c50 2024-03-19T02:18:19.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.448+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a8c107310 con 0x7f6a8c100c50 2024-03-19T02:18:19.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.448+0000 7f6a7effd700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 0x7f6a7803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:19.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.448+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6a8801e030 con 0x7f6a8c100c50 2024-03-19T02:18:19.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.448+0000 7f6a91771700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 0x7f6a7803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:19.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.448+0000 7f6a91771700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 0x7f6a7803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006fd0 tx=0x7f6a80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.453+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a88013070 con 0x7f6a8c100c50 2024-03-19T02:18:19.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.652+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f6a8c18f630 con 0x7f6a8c100c50 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:19.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:19.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.655+0000 7f6a7effd700 1 -- 172.21.15.116:0/1094555532 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f6a88031020 con 0x7f6a8c100c50 2024-03-19T02:18:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 msgr2=0x7f6a7803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 0x7f6a7803b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a80006fd0 tx=0x7f6a80006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 msgr2=0x7f6a8c1962d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a88004fb0 tx=0x7f6a88009f60 comp rx=0 tx=0).stop 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 shutdown_connections 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6a78039020 0x7f6a7803b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 --2- 172.21.15.116:0/1094555532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a8c100c50 0x7f6a8c1962d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 >> 172.21.15.116:0/1094555532 conn(0x7f6a8c0fc1d0 msgr2=0x7f6a8c0fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 shutdown_connections 2024-03-19T02:18:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:19.658+0000 7f6a941d6700 1 -- 172.21.15.116:0/1094555532 wait complete. 2024-03-19T02:18:20.236 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-19T02:18:20.236 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-19T02:18:20.468 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:20.854 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:20 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1094555532' entity='client.admin' 2024-03-19T02:18:20.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:20.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.950+0000 7f75e0cfe700 1 -- 172.21.15.116:0/2187055080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 msgr2=0x7f75dc102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.950+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/2187055080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc102160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75cc009910 tx=0x7f75cc009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.950+0000 7f75e0cfe700 1 -- 172.21.15.116:0/2187055080 shutdown_connections 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/2187055080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc102160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 -- 172.21.15.116:0/2187055080 >> 172.21.15.116:0/2187055080 conn(0x7f75dc0fd300 msgr2=0x7f75dc0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 -- 172.21.15.116:0/2187055080 shutdown_connections 2024-03-19T02:18:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 -- 172.21.15.116:0/2187055080 wait complete. 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 Processor -- start 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.951+0000 7f75e0cfe700 1 -- start start 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75e0cfe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75e0cfe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc107c80 con 0x7f75dc101d40 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45338/0 (socket says 172.21.15.116:45338) 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75da59c700 1 -- 172.21.15.116:0/1869327672 learned_addr learned my addr 172.21.15.116:0/1869327672 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75da59c700 1 -- 172.21.15.116:0/1869327672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc076510 con 0x7f75dc101d40 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75da59c700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005800 tx=0x7f75cc009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc027070 con 0x7f75dc101d40 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75cc00be10 con 0x7f75dc101d40 2024-03-19T02:18:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc00f460 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc074700 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.952+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc074c40 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.953+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f75cc02a070 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.953+0000 7f75cb7fe700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 0x7f75c403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.953+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f75cc01e030 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.953+0000 7f75d9d9b700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 0x7f75c403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.953+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc108400 con 0x7f75dc101d40 2024-03-19T02:18:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.954+0000 7f75d9d9b700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 0x7f75c403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006fd0 tx=0x7f75d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:21.959+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f75cc013070 con 0x7f75dc101d40 2024-03-19T02:18:22.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.156+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f75dc1905f0 con 0x7f75c4038fc0 2024-03-19T02:18:22.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.160+0000 7f75cb7fe700 1 -- 172.21.15.116:0/1869327672 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f75dc1905f0 con 0x7f75c4038fc0 2024-03-19T02:18:22.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 msgr2=0x7f75c403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:22.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 0x7f75c403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006fd0 tx=0x7f75d0006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:22.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 msgr2=0x7f75dc075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:22.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005800 tx=0x7f75cc009f60 comp rx=0 tx=0).stop 2024-03-19T02:18:22.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 shutdown_connections 2024-03-19T02:18:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f75c4038fc0 0x7f75c403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 --2- 172.21.15.116:0/1869327672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75dc101d40 0x7f75dc075fd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 >> 172.21.15.116:0/1869327672 conn(0x7f75dc0fd300 msgr2=0x7f75dc0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 shutdown_connections 2024-03-19T02:18:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:22.162+0000 7f75e0cfe700 1 -- 172.21.15.116:0/1869327672 wait complete. 2024-03-19T02:18:22.820 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-03-19T02:18:22.820 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:18:22.820 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-03-19T02:18:22.841 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:18:22.841 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:22.899 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-03-19T02:18:22.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch host add smithi154 2024-03-19T02:18:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:23.470 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:18:23.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:23.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:23.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:23.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:23.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:24.434 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:24 smithi116 ceph-mon[32263]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:18:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 -- 172.21.15.116:0/819595688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 msgr2=0x7f9408102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 --2- 172.21.15.116:0/819595688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408102310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009910 tx=0x7f93f8009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 -- 172.21.15.116:0/819595688 shutdown_connections 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 --2- 172.21.15.116:0/819595688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408102310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 -- 172.21.15.116:0/819595688 >> 172.21.15.116:0/819595688 conn(0x7f94080fd7e0 msgr2=0x7f94080ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 -- 172.21.15.116:0/819595688 shutdown_connections 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.635+0000 7f940e7db700 1 -- 172.21.15.116:0/819595688 wait complete. 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f940e7db700 1 Processor -- start 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f940e7db700 1 -- start start 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f940e7db700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f940e7db700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f940810c1b0 con 0x7f9408101f30 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f9407fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f9407fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45362/0 (socket says 172.21.15.116:45362) 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f9407fff700 1 -- 172.21.15.116:0/2315053391 learned_addr learned my addr 172.21.15.116:0/2315053391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.636+0000 7f9407fff700 1 -- 172.21.15.116:0/2315053391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9408197c80 con 0x7f9408101f30 2024-03-19T02:18:24.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f9407fff700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93f80056d0 tx=0x7f93f80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f8026070 con 0x7f9408101f30 2024-03-19T02:18:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9408197f70 con 0x7f9408101f30 2024-03-19T02:18:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93f800bba0 con 0x7f9408101f30 2024-03-19T02:18:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93f80037f0 con 0x7f9408101f30 2024-03-19T02:18:24.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.637+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f940819c010 con 0x7f9408101f30 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f93f8029070 con 0x7f9408101f30 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9408103410 con 0x7f9408101f30 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f94057fa700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 0x7f93f003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f93f801e070 con 0x7f9408101f30 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f94077fe700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 0x7f93f003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:24.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.638+0000 7f94077fe700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 0x7f93f003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f93fc006fd0 tx=0x7f93fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:24.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.643+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93f8013070 con 0x7f9408101f30 2024-03-19T02:18:24.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:24.844+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) -- 0x7f9408198aa0 con 0x7f93f0039020 2024-03-19T02:18:26.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:25 smithi116 ceph-mon[32263]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:26.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:25 smithi116 ceph-mon[32263]: Deploying cephadm binary to smithi154 2024-03-19T02:18:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:28 smithi116 ceph-mon[32263]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:29.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.054+0000 7f94057fa700 1 -- 172.21.15.116:0/2315053391 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9408198aa0 con 0x7f93f0039020 2024-03-19T02:18:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 msgr2=0x7f93f003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 0x7f93f003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f93fc006fd0 tx=0x7f93fc006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 msgr2=0x7f9408197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93f80056d0 tx=0x7f93f80057b0 comp rx=0 tx=0).stop 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 shutdown_connections 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93f0039020 0x7f93f003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 --2- 172.21.15.116:0/2315053391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9408101f30 0x7f9408197740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 >> 172.21.15.116:0/2315053391 conn(0x7f94080fd7e0 msgr2=0x7f9408103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 shutdown_connections 2024-03-19T02:18:29.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:29.056+0000 7f940e7db700 1 -- 172.21.15.116:0/2315053391 wait complete. 2024-03-19T02:18:29.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T02:18:29.865 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:30 smithi116 ceph-mon[32263]: Added host smithi154 2024-03-19T02:18:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:30 smithi116 ceph-mon[32263]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:31.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:31.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 -- 172.21.15.116:0/1658572175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 msgr2=0x7f1cfc101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:31.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 --2- 172.21.15.116:0/1658572175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc101230 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009910 tx=0x7f1cec009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:31.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 -- 172.21.15.116:0/1658572175 shutdown_connections 2024-03-19T02:18:31.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 --2- 172.21.15.116:0/1658572175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc101230 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:31.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 -- 172.21.15.116:0/1658572175 >> 172.21.15.116:0/1658572175 conn(0x7f1cfc0fc720 msgr2=0x7f1cfc0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 -- 172.21.15.116:0/1658572175 shutdown_connections 2024-03-19T02:18:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.454+0000 7f1d01986700 1 -- 172.21.15.116:0/1658572175 wait complete. 2024-03-19T02:18:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1d01986700 1 Processor -- start 2024-03-19T02:18:31.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1d01986700 1 -- start start 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1d01986700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1d01986700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc10b0d0 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1cfaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1cfaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44990/0 (socket says 172.21.15.116:44990) 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.455+0000 7f1cfaffd700 1 -- 172.21.15.116:0/268738616 learned_addr learned my addr 172.21.15.116:0/268738616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1cfaffd700 1 -- 172.21.15.116:0/268738616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cfc198af0 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1cfaffd700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0056d0 tx=0x7f1cec0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cec026070 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cec00bba0 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cfc198de0 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cec0037f0 con 0x7f1cfc100e50 2024-03-19T02:18:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.456+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cfc1992a0 con 0x7f1cfc100e50 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.457+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f1cec029070 con 0x7f1cfc100e50 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.457+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cfc102330 con 0x7f1cfc100e50 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.457+0000 7f1d00984700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 0x7f1ce403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.457+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1cec01e070 con 0x7f1cfc100e50 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.457+0000 7f1cfa7fc700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 0x7f1ce403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:31.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.458+0000 7f1cfa7fc700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 0x7f1ce403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0006fd0 tx=0x7f1cf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:31.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.462+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1cec013070 con 0x7f1cfc100e50 2024-03-19T02:18:31.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.667+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1cfc06a690 con 0x7f1ce4038fc0 2024-03-19T02:18:31.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.668+0000 7f1d00984700 1 -- 172.21.15.116:0/268738616 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f1cfc06a690 con 0x7f1ce4038fc0 2024-03-19T02:18:31.669 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:31.669 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-03-19T02:18:31.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.670+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 msgr2=0x7f1ce403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:31.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 0x7f1ce403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0006fd0 tx=0x7f1cf0006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:31.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 msgr2=0x7f1cfc1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:31.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1cec0056d0 tx=0x7f1cec0057b0 comp rx=0 tx=0).stop 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 shutdown_connections 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1ce4038fc0 0x7f1ce403b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 --2- 172.21.15.116:0/268738616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cfc100e50 0x7f1cfc1985b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 >> 172.21.15.116:0/268738616 conn(0x7f1cfc0fc720 msgr2=0x7f1cfc102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 shutdown_connections 2024-03-19T02:18:31.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:31.671+0000 7f1d01986700 1 -- 172.21.15.116:0/268738616 wait complete. 2024-03-19T02:18:32.278 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-03-19T02:18:32.278 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:18:32.278 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-03-19T02:18:32.297 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:18:32.297 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:32.354 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-03-19T02:18:32.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch host add smithi162 2024-03-19T02:18:32.589 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:32.919 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:32 smithi116 ceph-mon[32263]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:32.919 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:32.919 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:32 smithi116 ceph-mon[32263]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:18:34.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- 172.21.15.116:0/2738970111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 msgr2=0x7f7a5c101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:34.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 --2- 172.21.15.116:0/2738970111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c101230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009910 tx=0x7f7a4c009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:34.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- 172.21.15.116:0/2738970111 shutdown_connections 2024-03-19T02:18:34.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 --2- 172.21.15.116:0/2738970111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c101230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:34.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- 172.21.15.116:0/2738970111 >> 172.21.15.116:0/2738970111 conn(0x7f7a5c0fc720 msgr2=0x7f7a5c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:34.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- 172.21.15.116:0/2738970111 shutdown_connections 2024-03-19T02:18:34.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- 172.21.15.116:0/2738970111 wait complete. 2024-03-19T02:18:34.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 Processor -- start 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.153+0000 7f7a613d8700 1 -- start start 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a613d8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a613d8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c10b0d0 con 0x7f7a5c100e50 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a5affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a5affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45006/0 (socket says 172.21.15.116:45006) 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a5affd700 1 -- 172.21.15.116:0/3492337590 learned_addr learned my addr 172.21.15.116:0/3492337590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a5affd700 1 -- 172.21.15.116:0/3492337590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5c198ae0 con 0x7f7a5c100e50 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a5affd700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0056d0 tx=0x7f7a4c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:34.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a4c026070 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a4c00bba0 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5c198dd0 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a4c0037f0 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.154+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a5c199290 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.155+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f7a4c029070 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.155+0000 7f7a43fff700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 0x7f7a4403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.155+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7a4c01e070 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.155+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a5c102330 con 0x7f7a5c100e50 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.156+0000 7f7a5a7fc700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 0x7f7a4403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:34.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.156+0000 7f7a5a7fc700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 0x7f7a4403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7a50006fd0 tx=0x7f7a50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:34.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.160+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a4c030070 con 0x7f7a5c100e50 2024-03-19T02:18:34.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:34.359+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7f7a5c063940 con 0x7f7a44039020 2024-03-19T02:18:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:34 smithi116 ceph-mon[32263]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:18:36.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:36.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:18:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: Deploying cephadm binary to smithi162 2024-03-19T02:18:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:36 smithi116 ceph-mon[32263]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:18:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.696+0000 7f7a43fff700 1 -- 172.21.15.116:0/3492337590 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7a5c063940 con 0x7f7a44039020 2024-03-19T02:18:38.698 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.698+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 msgr2=0x7f7a4403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.698+0000 7f7a613d8700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 0x7f7a4403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7a50006fd0 tx=0x7f7a50006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.698+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 msgr2=0x7f7a5c1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.698+0000 7f7a613d8700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0056d0 tx=0x7f7a4c0057b0 comp rx=0 tx=0).stop 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.698+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 shutdown_connections 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.699+0000 7f7a613d8700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7a44039020 0x7f7a4403b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.699+0000 7f7a613d8700 1 --2- 172.21.15.116:0/3492337590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a5c100e50 0x7f7a5c1985a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.699+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 >> 172.21.15.116:0/3492337590 conn(0x7f7a5c0fc720 msgr2=0x7f7a5c102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.699+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 shutdown_connections 2024-03-19T02:18:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:38.699+0000 7f7a613d8700 1 -- 172.21.15.116:0/3492337590 wait complete. 2024-03-19T02:18:39.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:38 smithi116 ceph-mon[32263]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:39.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:39.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:39.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:39.271 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T02:18:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:39.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:39.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:39 smithi116 ceph-mon[32263]: Added host smithi162 2024-03-19T02:18:39.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:39.930 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:40.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:40 smithi116 ceph-mon[32263]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:40.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:40 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1690340881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 msgr2=0x7f2a90103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1690340881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90103290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009910 tx=0x7f2a80009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1690340881 shutdown_connections 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1690340881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90103290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1690340881 >> 172.21.15.116:0/1690340881 conn(0x7f2a900fc6e0 msgr2=0x7f2a900feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1690340881 shutdown_connections 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1690340881 wait complete. 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.956+0000 7f2a95e3c700 1 Processor -- start 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a95e3c700 1 -- start start 2024-03-19T02:18:40.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a95e3c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a95e3c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9010c250 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40702/0 (socket says 172.21.15.116:40702) 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8f7fe700 1 -- 172.21.15.116:0/1277286056 learned_addr learned my addr 172.21.15.116:0/1277286056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8f7fe700 1 -- 172.21.15.116:0/1277286056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a90102a00 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8f7fe700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a80006010 tx=0x7f2a80005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a80026070 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a8000bc00 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a90101050 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.957+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a80003910 con 0x7f2a90102eb0 2024-03-19T02:18:40.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.958+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a90101510 con 0x7f2a90102eb0 2024-03-19T02:18:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.958+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f2a80029070 con 0x7f2a90102eb0 2024-03-19T02:18:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.958+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9006a9e0 con 0x7f2a90102eb0 2024-03-19T02:18:40.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.958+0000 7f2a8cff9700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 0x7f2a7803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:40.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.958+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2a8001e070 con 0x7f2a90102eb0 2024-03-19T02:18:40.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.959+0000 7f2a8effd700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 0x7f2a7803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:40.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.959+0000 7f2a8effd700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 0x7f2a7803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:40.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:40.963+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a80013070 con 0x7f2a90102eb0 2024-03-19T02:18:41.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.165+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a901020b0 con 0x7f2a78038fc0 2024-03-19T02:18:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.166+0000 7f2a8cff9700 1 -- 172.21.15.116:0/1277286056 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f2a901020b0 con 0x7f2a78038fc0 2024-03-19T02:18:41.166 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:18:41.166 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-03-19T02:18:41.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 msgr2=0x7f2a7803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 0x7f2a7803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 msgr2=0x7f2a90100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a80006010 tx=0x7f2a80005780 comp rx=0 tx=0).stop 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 shutdown_connections 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a78038fc0 0x7f2a7803b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 --2- 172.21.15.116:0/1277286056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a90102eb0 0x7f2a90100b10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 >> 172.21.15.116:0/1277286056 conn(0x7f2a900fc6e0 msgr2=0x7f2a90105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 shutdown_connections 2024-03-19T02:18:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:41.168+0000 7f2a95e3c700 1 -- 172.21.15.116:0/1277286056 wait complete. 2024-03-19T02:18:41.824 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-19T02:18:41.824 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-19T02:18:42.053 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:18:42.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:42 smithi116 ceph-mon[32263]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:18:42.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:42.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:42 smithi116 ceph-mon[32263]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.399+0000 7fc815cac700 1 -- 172.21.15.116:0/775777207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 msgr2=0x7fc810102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.399+0000 7fc815cac700 1 --2- 172.21.15.116:0/775777207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810102310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc80000b0a0 tx=0x7fc80000b3b0 comp rx=0 tx=0).stop 2024-03-19T02:18:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.399+0000 7fc815cac700 1 -- 172.21.15.116:0/775777207 shutdown_connections 2024-03-19T02:18:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 --2- 172.21.15.116:0/775777207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810102310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 -- 172.21.15.116:0/775777207 >> 172.21.15.116:0/775777207 conn(0x7fc8100fd7e0 msgr2=0x7fc8100ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 -- 172.21.15.116:0/775777207 shutdown_connections 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 -- 172.21.15.116:0/775777207 wait complete. 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 Processor -- start 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.400+0000 7fc815cac700 1 -- start start 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc815cac700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc815cac700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc81010c1b0 con 0x7fc810101f30 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40724/0 (socket says 172.21.15.116:40724) 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80f7fe700 1 -- 172.21.15.116:0/4060343478 learned_addr learned my addr 172.21.15.116:0/4060343478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80f7fe700 1 -- 172.21.15.116:0/4060343478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc810197c90 con 0x7fc810101f30 2024-03-19T02:18:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80f7fe700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8000076a0 tx=0x7fc800003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc800017040 con 0x7fc810101f30 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc800022070 con 0x7fc810101f30 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc810197f20 con 0x7fc810101f30 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc80001d5e0 con 0x7fc810101f30 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.401+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81019c020 con 0x7fc810101f30 2024-03-19T02:18:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.402+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fc80001d850 con 0x7fc810101f30 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.402+0000 7fc80cff9700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 0x7fc7f803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.402+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc80000f050 con 0x7fc810101f30 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.402+0000 7fc80effd700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 0x7fc7f803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.402+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc810103410 con 0x7fc810101f30 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.403+0000 7fc80effd700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 0x7fc7f803b4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.407+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc800025bf0 con 0x7fc810101f30 2024-03-19T02:18:43.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:43.609+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc810063940 con 0x7fc810101f30 2024-03-19T02:18:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.580+0000 7fc80cff9700 1 -- 172.21.15.116:0/4060343478 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc80001c3b0 con 0x7fc810101f30 2024-03-19T02:18:44.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 msgr2=0x7fc7f803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:44.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 0x7fc7f803b4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc804006fd0 tx=0x7fc804006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:44.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 msgr2=0x7fc810197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:44.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8000076a0 tx=0x7fc800003ce0 comp rx=0 tx=0).stop 2024-03-19T02:18:44.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 shutdown_connections 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc7f8039000 0x7fc7f803b4c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 --2- 172.21.15.116:0/4060343478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc810101f30 0x7fc810197750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 >> 172.21.15.116:0/4060343478 conn(0x7fc8100fd7e0 msgr2=0x7fc810104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 shutdown_connections 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:18:44.583+0000 7fc815cac700 1 -- 172.21.15.116:0/4060343478 wait complete. 2024-03-19T02:18:44.583 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4060343478' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:44.842 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:18:44.843 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:44.843 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:45.244 INFO:tasks.cephadm:Adding mon.a on smithi116 2024-03-19T02:18:45.244 INFO:tasks.cephadm:Adding mon.b on smithi154 2024-03-19T02:18:45.245 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-03-19T02:18:45.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c' 2024-03-19T02:18:45.473 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4060343478' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 -- 172.21.15.162:0/2065391285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 msgr2=0x7f319c074a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2065391285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c074a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f318c009910 tx=0x7f318c009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 -- 172.21.15.162:0/2065391285 shutdown_connections 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2065391285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c074a10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 -- 172.21.15.162:0/2065391285 >> 172.21.15.162:0/2065391285 conn(0x7f319c0fd7e0 msgr2=0x7f319c0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:46.745 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 -- 172.21.15.162:0/2065391285 shutdown_connections 2024-03-19T02:18:46.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.745+0000 7f31a1d15700 1 -- 172.21.15.162:0/2065391285 wait complete. 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f31a1d15700 1 Processor -- start 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f31a1d15700 1 -- start start 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f31a1d15700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f31a1d15700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f319c10e410 con 0x7f319c074630 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f319b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f319b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.162:47662/0 (socket says 172.21.15.162:47662) 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.746+0000 7f319b7fe700 1 -- 172.21.15.162:0/2205806868 learned_addr learned my addr 172.21.15.162:0/2205806868 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f319b7fe700 1 -- 172.21.15.162:0/2205806868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319c19fb50 con 0x7f319c074630 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f319b7fe700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f318c0056d0 tx=0x7f318c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f318c026070 con 0x7f319c074630 2024-03-19T02:18:46.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319c19fde0 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f318c004370 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f318c0039b0 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.747+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319c199790 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.748+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f318c029070 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.748+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f319c075d70 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.748+0000 7f3198ff9700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 0x7f318403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.748+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f318c01e070 con 0x7f319c074630 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.748+0000 7f319affd700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 0x7f318403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:46.749 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.749+0000 7f319affd700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 0x7f318403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3190006fd0 tx=0x7f3190006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:46.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.754+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f318c013070 con 0x7f319c074630 2024-03-19T02:18:46.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.963+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}) -- 0x7f319c19acc0 con 0x7f3184038fc0 2024-03-19T02:18:46.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.970+0000 7f3198ff9700 1 -- 172.21.15.162:0/2205806868 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f319c19acc0 con 0x7f3184038fc0 2024-03-19T02:18:46.970 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 msgr2=0x7f318403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 0x7f318403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3190006fd0 tx=0x7f3190006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 msgr2=0x7f319c19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f318c0056d0 tx=0x7f318c0057b0 comp rx=0 tx=0).stop 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 shutdown_connections 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3184038fc0 0x7f318403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 --2- 172.21.15.162:0/2205806868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f319c074630 0x7f319c19f610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 >> 172.21.15.162:0/2205806868 conn(0x7f319c0fd7e0 msgr2=0x7f319c105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 shutdown_connections 2024-03-19T02:18:46.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:46.972+0000 7f31a1d15700 1 -- 172.21.15.162:0/2205806868 wait complete. 2024-03-19T02:18:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:46 smithi116 ceph-mon[32263]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:46 smithi116 ceph-mon[32263]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:18:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:46 smithi116 ceph-mon[32263]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:47.535 DEBUG:teuthology.orchestra.run.smithi154:mon.b> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.b.service 2024-03-19T02:18:47.538 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.c.service 2024-03-19T02:18:47.540 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:18:47.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:18:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c;count:3 2024-03-19T02:18:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: Deploying daemon mon.c on smithi162 2024-03-19T02:18:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:47 smithi116 ceph-mon[32263]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:48.338 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:50 smithi116 ceph-mon[32263]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.040+0000 7fd0661e4700 1 -- 172.21.15.162:0/1580283369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 msgr2=0x7fd060101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.040+0000 7fd0661e4700 1 --2- 172.21.15.162:0/1580283369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060101210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd050009910 tx=0x7fd050009c20 comp rx=0 tx=0).stop 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 -- 172.21.15.162:0/1580283369 shutdown_connections 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 --2- 172.21.15.162:0/1580283369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060101210 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 -- 172.21.15.162:0/1580283369 >> 172.21.15.162:0/1580283369 conn(0x7fd0600fc700 msgr2=0x7fd0600feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 -- 172.21.15.162:0/1580283369 shutdown_connections 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 -- 172.21.15.162:0/1580283369 wait complete. 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 Processor -- start 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.041+0000 7fd0661e4700 1 -- start start 2024-03-19T02:18:51.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd0661e4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:51.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd0661e4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd06010b0b0 con 0x7fd060100e30 2024-03-19T02:18:51.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd05f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:51.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd05f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.162:47674/0 (socket says 172.21.15.162:47674) 2024-03-19T02:18:51.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd05f7fe700 1 -- 172.21.15.162:0/2505446483 learned_addr learned my addr 172.21.15.162:0/2505446483 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.042+0000 7fd05f7fe700 1 -- 172.21.15.162:0/2505446483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd060197c50 con 0x7fd060100e30 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd05f7fe700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd050006010 tx=0x7fd050005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd050026070 con 0x7fd060100e30 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd05000bba0 con 0x7fd060100e30 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd060197f40 con 0x7fd060100e30 2024-03-19T02:18:51.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0500038a0 con 0x7fd060100e30 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.043+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06019bfe0 con 0x7fd060100e30 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.044+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd060102310 con 0x7fd060100e30 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.044+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fd050029070 con 0x7fd060100e30 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.044+0000 7fd05cff9700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 0x7fd04803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.044+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd05001e070 con 0x7fd060100e30 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.044+0000 7fd05effd700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 0x7fd04803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:51.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.045+0000 7fd05effd700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 0x7fd04803b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd054006fd0 tx=0x7fd054006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:51.049 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.049+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd050013070 con 0x7fd060100e30 2024-03-19T02:18:51.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.345+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd060198a70 con 0x7fd060100e30 2024-03-19T02:18:51.346 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:18:51.346 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","modified":"2024-03-19T02:17:03.769369Z","created":"2024-03-19T02:17:03.769369Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T02:18:51.346 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.345+0000 7fd05cff9700 1 -- 172.21.15.162:0/2505446483 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fd050030020 con 0x7fd060100e30 2024-03-19T02:18:51.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 msgr2=0x7fd04803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:51.347 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 0x7fd04803b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd054006fd0 tx=0x7fd054006e40 comp rx=0 tx=0).stop 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 msgr2=0x7fd060197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd050006010 tx=0x7fd050005780 comp rx=0 tx=0).stop 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 shutdown_connections 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd048039020 0x7fd04803b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 --2- 172.21.15.162:0/2505446483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd060100e30 0x7fd060197710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.347+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 >> 172.21.15.162:0/2505446483 conn(0x7fd0600fc700 msgr2=0x7fd060069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.348+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 shutdown_connections 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:51.348+0000 7fd0661e4700 1 -- 172.21.15.162:0/2505446483 wait complete. 2024-03-19T02:18:51.348 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-03-19T02:18:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:51 smithi162 systemd[1]: Starting Ceph mon.c for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:18:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:51 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2505446483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 podman[35742]: 2024-03-19 02:18:51.995772057 +0000 UTC m=+0.018727545 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 podman[35742]: 2024-03-19 02:18:52.108901786 +0000 UTC m=+0.131857266 container create ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 podman[35742]: 2024-03-19 02:18:52.328324263 +0000 UTC m=+0.351279743 container init ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 podman[35742]: 2024-03-19 02:18:52.331237684 +0000 UTC m=+0.354193167 container start ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T02:18:52.356 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pidfile_write: ignore empty --pid-file 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: load: jerasure load: lrc 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: RocksDB version: 7.9.2 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Git sha 0 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: DB SUMMARY 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: DB Session ID: GIZ879RJHNTMM123URRJ 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: CURRENT file: CURRENT 2024-03-19T02:18:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.error_if_exists: 0 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.create_if_missing: 0 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.paranoid_checks: 1 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.env: 0x562b39228c60 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T02:18:52.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.info_log: 0x562b3b959ae0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.statistics: (nil) 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.use_fsync: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_log_file_size: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_fallocate: 1 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T02:18:52.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.use_direct_reads: 0 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.db_log_dir: 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.wal_dir: 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T02:18:52.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.write_buffer_manager: 0x562b3b92c1e0 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T02:18:52.673 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.unordered_write: 0 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.row_cache: None 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.wal_filter: None 2024-03-19T02:18:52.674 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.two_write_queues: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.wal_compression: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.atomic_flush: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.log_readahead_size: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T02:18:52.675 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_background_jobs: 2 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_background_compactions: -1 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_subcompactions: 1 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T02:18:52.676 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_open_files: -1 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_background_flushes: -1 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Compression algorithms supported: 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kZSTD supported: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kXpressCompression supported: 0 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T02:18:52.677 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kZlibCompression supported: 1 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kSnappyCompression supported: 1 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kLZ4Compression supported: 1 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: kBZip2Compression supported: 0 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.merge_operator: 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_filter: None 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T02:18:52.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562b3b959be0) 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_type: 0 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_index_type: 0 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_shortening: 1 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: checksum: 4 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: no_block_cache: 0 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache: 0x562b3b95f090 2024-03-19T02:18:52.679 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_options: 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: capacity : 536870912 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_shard_bits : 4 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: strict_capacity_limit : 0 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_compressed: (nil) 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: persistent_cache: (nil) 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size: 4096 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size_deviation: 10 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_restart_interval: 16 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_block_restart_interval: 1 2024-03-19T02:18:52.680 INFO:journalctl@ceph.mon.c.smithi162.stdout: metadata_block_size: 4096 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: partition_filters: 0 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: use_delta_encoding: 1 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: filter_policy: bloomfilter 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: whole_key_filtering: 1 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: verify_compression: 0 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: format_version: 5 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_index_compression: 1 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_align: 0 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: max_auto_readahead_size: 262144 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: prepopulate_block_cache: 0 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: initial_auto_readahead_size: 8192 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T02:18:52.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression: NoCompression 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.num_levels: 7 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T02:18:52.682 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:18:52.683 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T02:18:52.684 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T02:18:52.685 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.table_properties_collectors: 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.inplace_update_support: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.bloom_locality: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.max_successive_merges: 0 2024-03-19T02:18:52.686 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.ttl: 2592000 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enable_blob_files: false 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.min_blob_size: 0 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T02:18:52.687 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f44ca615-7a95-4606-b7de-8d08fa03bada 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814732363053, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-19T02:18:52.688 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814732363475, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710814732, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f44ca615-7a95-4606-b7de-8d08fa03bada", "db_session_id": "GIZ879RJHNTMM123URRJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710814732363565, "job": 1, "event": "recovery_finished"} 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562b3ba52000 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: DB pointer 0x562b3ba3c000 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** DB Stats ** 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T02:18:52.689 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T02:18:52.690 INFO:journalctl@ceph.mon.c.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T02:18:52.691 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache BinnedLRUCache@0x562b3b95f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: starting mon.c rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(???) e0 preinit fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).mds e1 new map 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: e1 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: legacy client fscid: -1 2024-03-19T02:18:52.692 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout: No filesystems configured 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:18:52.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Deploying cephadm binary to smithi162 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:18:52.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Added host smithi162 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.695 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4060343478' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4060343478' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:18:52.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c;count:3 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: Deploying daemon mon.c on smithi162 2024-03-19T02:18:52.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:52.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2505446483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:18:52.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 ceph-mon[35761]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-19T02:18:52.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 bash[35742]: ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 2024-03-19T02:18:52.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:52 smithi162 systemd[1]: Started Ceph mon.c for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:18:52.943 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:18:52.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:18:53.180 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 -- 172.21.15.162:0/1309698785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46180057d0 msgr2=0x7f4618007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 --2- 172.21.15.162:0/1309698785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46180057d0 0x7f4618007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f462c100c40 tx=0x7f4624009e80 comp rx=0 tx=0).stop 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 -- 172.21.15.162:0/1309698785 shutdown_connections 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 --2- 172.21.15.162:0/1309698785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46180057d0 0x7f4618007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 -- 172.21.15.162:0/1309698785 >> 172.21.15.162:0/1309698785 conn(0x7f462c0fc820 msgr2=0x7f462c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:57.681 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 -- 172.21.15.162:0/1309698785 shutdown_connections 2024-03-19T02:18:57.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.681+0000 7f46343d1700 1 -- 172.21.15.162:0/1309698785 wait complete. 2024-03-19T02:18:57.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 Processor -- start 2024-03-19T02:18:57.683 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 -- start start 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 0x7f462c1955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f462c107bf0 con 0x7f462c19b920 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f46343d1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f462c06b3d0 con 0x7f46180057d0 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:34248/0 (socket says 172.21.15.162:34248) 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 -- 172.21.15.162:0/4047441081 learned_addr learned my addr 172.21.15.162:0/4047441081 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 -- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 msgr2=0x7f462c19b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 -- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 msgr2=0x7f462c19b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463196c700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 0x7f462c1955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:57.684 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.682+0000 7f463216d700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f463196c700 1 -- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 msgr2=0x7f462c19b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f463196c700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f463196c700 1 -- 172.21.15.162:0/4047441081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462c19bf60 con 0x7f462c19b920 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f463196c700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 0x7f462c1955a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b4d0 tx=0x7f461c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f461c011630 con 0x7f462c19b920 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f461c011c80 con 0x7f462c19b920 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462c195ae0 con 0x7f462c19b920 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f461c010da0 con 0x7f462c19b920 2024-03-19T02:18:57.685 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.683+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462c196070 con 0x7f462c19b920 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.684+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f461c018020 con 0x7f462c19b920 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.684+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462c102450 con 0x7f462c19b920 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.685+0000 7f46237fe700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 0x7f461003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.685+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f461c04ba60 con 0x7f462c19b920 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.685+0000 7f463216d700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 0x7f461003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:18:57.686 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.685+0000 7f463216d700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 0x7f461003b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4624009160 tx=0x7f46240090b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:18:57.690 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.690+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f461c01f020 con 0x7f462c19b920 2024-03-19T02:18:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: mon.a calling monitor election 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: mon.c calling monitor election 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: monmap epoch 2 2024-03-19T02:18:57.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: last_changed 2024-03-19T02:18:52.391819+0000 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: min_mon_release 19 (squid) 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: election_strategy: 1 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: fsmap 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: mgrmap e13: x(active, since 49s) 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:18:57.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:18:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: mon.a calling monitor election 2024-03-19T02:18:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: mon.c calling monitor election 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: monmap epoch 2 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: last_changed 2024-03-19T02:18:52.391819+0000 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:18:57.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: min_mon_release 19 (squid) 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: election_strategy: 1 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: fsmap 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: mgrmap e13: x(active, since 49s) 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:18:57.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:18:57.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.966+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f462c196ad0 con 0x7f462c19b920 2024-03-19T02:18:57.967 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.967+0000 7f46237fe700 1 -- 172.21.15.162:0/4047441081 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f461c01ed10 con 0x7f462c19b920 2024-03-19T02:18:57.968 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:18:57.968 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","modified":"2024-03-19T02:18:52.391819Z","created":"2024-03-19T02:17:03.769369Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 msgr2=0x7f461003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 0x7f461003b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4624009160 tx=0x7f46240090b0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 msgr2=0x7f462c1955a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 0x7f462c1955a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b4d0 tx=0x7f461c00b890 comp rx=0 tx=0).stop 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 shutdown_connections 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4610038d60 0x7f461003b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46180057d0 0x7f462c19b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 --2- 172.21.15.162:0/4047441081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f462c19b920 0x7f462c1955a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:18:57.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 >> 172.21.15.162:0/4047441081 conn(0x7f462c0fc820 msgr2=0x7f462c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:18:57.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 shutdown_connections 2024-03-19T02:18:57.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:18:57.969+0000 7f46343d1700 1 -- 172.21.15.162:0/4047441081 wait complete. 2024-03-19T02:18:57.972 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-03-19T02:18:58.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:58 smithi162 ceph-mon[35761]: Deploying daemon mon.b on smithi154 2024-03-19T02:18:58.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:58 smithi162 ceph-mon[35761]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:58.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:58 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/4047441081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:18:58.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:18:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:58 smithi116 ceph-mon[32263]: Deploying daemon mon.b on smithi154 2024-03-19T02:18:58.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:58 smithi116 ceph-mon[32263]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:18:58.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/4047441081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:18:58.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:18:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:18:59.549 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:18:59.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:18:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:19:00.863 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:00 smithi162 ceph-mon[35761]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:00 smithi116 ceph-mon[32263]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:01.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.130+0000 7f59e9081700 1 -- 172.21.15.162:0/1612949990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4102030 msgr2=0x7f59e4102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.130+0000 7f59e9081700 1 --2- 172.21.15.162:0/1612949990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4102030 0x7f59e4102410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b0a0 tx=0x7f59dc00b3b0 comp rx=0 tx=0).stop 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.130+0000 7f59e9081700 1 -- 172.21.15.162:0/1612949990 shutdown_connections 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.130+0000 7f59e9081700 1 --2- 172.21.15.162:0/1612949990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4102030 0x7f59e4102410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.130+0000 7f59e9081700 1 -- 172.21.15.162:0/1612949990 >> 172.21.15.162:0/1612949990 conn(0x7f59e40fd8a0 msgr2=0x7f59e40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.131+0000 7f59e9081700 1 -- 172.21.15.162:0/1612949990 shutdown_connections 2024-03-19T02:19:01.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.131+0000 7f59e9081700 1 -- 172.21.15.162:0/1612949990 wait complete. 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.131+0000 7f59e9081700 1 Processor -- start 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e9081700 1 -- start start 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e9081700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e9081700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e9081700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59e410c2b0 con 0x7f59e4074b50 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e9081700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59e410c130 con 0x7f59e4102030 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.162:49820/0 (socket says 172.21.15.162:49820) 2024-03-19T02:19:01.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 -- 172.21.15.162:0/2791623768 learned_addr learned my addr 172.21.15.162:0/2791623768 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e2d9d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e2d9d700 1 -- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 msgr2=0x7f59e4074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e2d9d700 1 -- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 msgr2=0x7f59e4074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e2d9d700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e2d9d700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 -- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 msgr2=0x7f59e4074610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.132+0000 7f59e259c700 1 -- 172.21.15.162:0/2791623768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e4076530 con 0x7f59e4074b50 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59e259c700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00b700 tx=0x7f59cc00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f59cc011630 con 0x7f59e4074b50 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59cc011c80 con 0x7f59e4074b50 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e41a3c20 con 0x7f59e4074b50 2024-03-19T02:19:01.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f59cc010da0 con 0x7f59e4074b50 2024-03-19T02:19:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.133+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41a4160 con 0x7f59e4074b50 2024-03-19T02:19:01.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.134+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f59cc018020 con 0x7f59e4074b50 2024-03-19T02:19:01.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.134+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e4103510 con 0x7f59e4074b50 2024-03-19T02:19:01.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.134+0000 7f59dbfff700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 0x7f59d003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:01.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.134+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f59cc04bab0 con 0x7f59e4074b50 2024-03-19T02:19:01.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.135+0000 7f59e2d9d700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 0x7f59d003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:01.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.135+0000 7f59e2d9d700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 0x7f59d003b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0062a0 tx=0x7f59dc006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:01.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.140+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59cc01f020 con 0x7f59e4074b50 2024-03-19T02:19:01.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.259+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc01ae40 con 0x7f59e4074b50 2024-03-19T02:19:01.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:01.423+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f59e41a4470 con 0x7f59e4074b50 2024-03-19T02:19:06.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.291+0000 7f59dbfff700 1 -- 172.21.15.162:0/2791623768 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f59cc018310 con 0x7f59e4074b50 2024-03-19T02:19:06.292 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T02:19:06.292 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","modified":"2024-03-19T02:19:01.260038Z","created":"2024-03-19T02:17:03.769369Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T02:19:06.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 msgr2=0x7f59d003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:06.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 0x7f59d003b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0062a0 tx=0x7f59dc006230 comp rx=0 tx=0).stop 2024-03-19T02:19:06.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 msgr2=0x7f59e4074fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:06.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00b700 tx=0x7f59cc00bac0 comp rx=0 tx=0).stop 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 shutdown_connections 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f59d0038d60 0x7f59d003b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59e4102030 0x7f59e4074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 --2- 172.21.15.162:0/2791623768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59e4074b50 0x7f59e4074fd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 >> 172.21.15.162:0/2791623768 conn(0x7f59e40fd8a0 msgr2=0x7f59e4103a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 shutdown_connections 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:06.294+0000 7f59e9081700 1 -- 172.21.15.162:0/2791623768 wait complete. 2024-03-19T02:19:06.296 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: mon.c calling monitor election 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: mon.a calling monitor election 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2791623768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:19:06.643 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: monmap epoch 3 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: last_changed 2024-03-19T02:19:01.260038+0000 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: min_mon_release 19 (squid) 2024-03-19T02:19:06.644 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: election_strategy: 1 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: fsmap 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: mgrmap e13: x(active, since 58s) 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.645 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:19:06.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: mon.c calling monitor election 2024-03-19T02:19:06.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: mon.a calling monitor election 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2791623768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: monmap epoch 3 2024-03-19T02:19:06.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: last_changed 2024-03-19T02:19:01.260038+0000 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: min_mon_release 19 (squid) 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: election_strategy: 1 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: fsmap 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: mgrmap e13: x(active, since 58s) 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T02:19:06.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T02:19:06.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2024-03-19T02:19:06.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:06.895 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-19T02:19:06.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-19T02:19:07.139 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:19:07.561 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:07.561 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:07.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:07.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 -- 172.21.15.116:0/2947415288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 msgr2=0x7f46f4074ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/2947415288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f4074ac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009910 tx=0x7f46f0009c20 comp rx=0 tx=0).stop 2024-03-19T02:19:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 -- 172.21.15.116:0/2947415288 shutdown_connections 2024-03-19T02:19:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/2947415288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f4074ac0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 -- 172.21.15.116:0/2947415288 >> 172.21.15.116:0/2947415288 conn(0x7f46f40fd7e0 msgr2=0x7f46f40ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 -- 172.21.15.116:0/2947415288 shutdown_connections 2024-03-19T02:19:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.773+0000 7f46fb7d9700 1 -- 172.21.15.116:0/2947415288 wait complete. 2024-03-19T02:19:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 Processor -- start 2024-03-19T02:19:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 -- start start 2024-03-19T02:19:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:08.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 0x7f46f4199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4199fb0 0x7f46f419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e440 con 0x7f46f40746e0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e2c0 con 0x7f46f419fdf0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46fb7d9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e5c0 con 0x7f46f4199fb0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9575700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9575700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45376/0 (socket says 172.21.15.116:45376) 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9575700 1 -- 172.21.15.116:0/4290085678 learned_addr learned my addr 172.21.15.116:0/4290085678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f8d74700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 0x7f46f4199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9d76700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4199fb0 0x7f46f419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:08.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9575700 1 -- 172.21.15.116:0/4290085678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4199fb0 msgr2=0x7f46f419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.774+0000 7f46f9575700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4199fb0 0x7f46f419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46f9575700 1 -- 172.21.15.116:0/4290085678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 msgr2=0x7f46f4199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46f9575700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 0x7f46f4199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46f9575700 1 -- 172.21.15.116:0/4290085678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f41a6780 con 0x7f46f40746e0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46f8d74700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 0x7f46f4199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46f9575700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46f0005b40 tx=0x7f46f0004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0027070 con 0x7f46f40746e0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a6950 con 0x7f46f40746e0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46f000be10 con 0x7f46f40746e0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a6e80 con 0x7f46f40746e0 2024-03-19T02:19:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.775+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0022e60 con 0x7f46f40746e0 2024-03-19T02:19:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.776+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f46f002a070 con 0x7f46f40746e0 2024-03-19T02:19:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.776+0000 7f46ea7fc700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 0x7f46e003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.776+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f46f004d0e0 con 0x7f46f40746e0 2024-03-19T02:19:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.776+0000 7f46f8d74700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 0x7f46e003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:08.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.776+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f4063940 con 0x7f46f40746e0 2024-03-19T02:19:08.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.777+0000 7f46f8d74700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 0x7f46e003b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009910 tx=0x7f46e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:08.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:08.781+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f46f0013070 con 0x7f46f40746e0 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.019+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f46f41a7050 con 0x7f46f40746e0 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.020+0000 7f46ea7fc700 1 -- 172.21.15.116:0/4290085678 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f46f0028260 con 0x7f46f40746e0 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:09.020 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 msgr2=0x7f46e003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 0x7f46e003b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009910 tx=0x7f46e4008040 comp rx=0 tx=0).stop 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 msgr2=0x7f46f419f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46f0005b40 tx=0x7f46f0004f80 comp rx=0 tx=0).stop 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 shutdown_connections 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46e0039450 0x7f46e003b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4199fb0 0x7f46f419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:09.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f40746e0 0x7f46f419f8b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:09.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 --2- 172.21.15.116:0/4290085678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f419fdf0 0x7f46f4199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:09.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 >> 172.21.15.116:0/4290085678 conn(0x7f46f40fd7e0 msgr2=0x7f46f4108000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:09.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 shutdown_connections 2024-03-19T02:19:09.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:09.023+0000 7f46fb7d9700 1 -- 172.21.15.116:0/4290085678 wait complete. 2024-03-19T02:19:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mon.b calling monitor election 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mon.b calling monitor election 2024-03-19T02:19:09.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mon.c calling monitor election 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mon.a calling monitor election 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: monmap epoch 3 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: last_changed 2024-03-19T02:19:01.260038+0000 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: min_mon_release 19 (squid) 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: election_strategy: 1 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: fsmap 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:19:09.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: mgrmap e13: x(active, since 60s) 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:09.670 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:09.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: Reconfiguring daemon mon.a on smithi116 2024-03-19T02:19:09.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4290085678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.671 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mon.b calling monitor election 2024-03-19T02:19:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:19:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:19:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:19:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mon.b calling monitor election 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mon.c calling monitor election 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mon.a calling monitor election 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: monmap epoch 3 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: last_changed 2024-03-19T02:19:01.260038+0000 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: min_mon_release 19 (squid) 2024-03-19T02:19:09.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: election_strategy: 1 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: fsmap 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: mgrmap e13: x(active, since 60s) 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: overall HEALTH_OK 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: Reconfiguring daemon mon.a on smithi116 2024-03-19T02:19:09.759 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4290085678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mon.b calling monitor election 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mon.b calling monitor election 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mon.c calling monitor election 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mon.a calling monitor election 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: monmap epoch 3 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: last_changed 2024-03-19T02:19:01.260038+0000 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: created 2024-03-19T02:17:03.769369+0000 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: min_mon_release 19 (squid) 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: election_strategy: 1 2024-03-19T02:19:09.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: fsmap 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: mgrmap e13: x(active, since 60s) 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.762 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: Reconfiguring daemon mon.a on smithi116 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4290085678' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:09.763 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:19:10.169 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-19T02:19:10.169 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:19:10.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:19:10.199 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:19:10.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:19:10.267 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:19:10.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:19:10.303 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:19:10.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:19:10.374 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:19:10.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:19:10.410 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:19:10.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:19:10.480 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-19T02:19:10.480 INFO:tasks.cephadm:Adding mgr.y on smithi154 2024-03-19T02:19:10.480 INFO:tasks.cephadm:Adding mgr.z on smithi162 2024-03-19T02:19:10.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi116=x;smithi154=y;smithi162=z' 2024-03-19T02:19:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:10 smithi116 ceph-mon[32263]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:10.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:10 smithi162 ceph-mon[35761]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:10.754 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:19:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:10 smithi154 ceph-mon[35693]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:11.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:11.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:11 smithi162 ceph-mon[35761]: Reconfiguring daemon mon.b on smithi154 2024-03-19T02:19:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:11 smithi154 ceph-mon[35693]: Reconfiguring daemon mon.b on smithi154 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:11 smithi116 ceph-mon[32263]: Reconfiguring daemon mon.b on smithi154 2024-03-19T02:19:12.186 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.182+0000 7fefdba7f700 1 -- 172.21.15.162:0/778132252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 msgr2=0x7fefd4103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.182+0000 7fefdba7f700 1 --2- 172.21.15.162:0/778132252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4103860 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009910 tx=0x7fefc4009c20 comp rx=0 tx=0).stop 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.182+0000 7fefdba7f700 1 -- 172.21.15.162:0/778132252 shutdown_connections 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.182+0000 7fefdba7f700 1 --2- 172.21.15.162:0/778132252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4103860 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.182+0000 7fefdba7f700 1 -- 172.21.15.162:0/778132252 >> 172.21.15.162:0/778132252 conn(0x7fefd40fba60 msgr2=0x7fefd40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- 172.21.15.162:0/778132252 shutdown_connections 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- 172.21.15.162:0/778132252 wait complete. 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 Processor -- start 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- start start 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 0x7fefd419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4107d20 con 0x7fefd4103440 2024-03-19T02:19:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4107ba0 con 0x7fefd4197aa0 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.183+0000 7fefdba7f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4107ea0 con 0x7fefd4197f20 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefd901a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefd981b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefd901a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.162:60120/0 (socket says 172.21.15.162:60120) 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefd901a700 1 -- 172.21.15.162:0/2520749411 learned_addr learned my addr 172.21.15.162:0/2520749411 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 0x7fefd419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 -- 172.21.15.162:0/2520749411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 msgr2=0x7fefd419cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 -- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 msgr2=0x7fefd4197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 -- 172.21.15.162:0/2520749411 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd419f970 con 0x7fefd4197f20 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefd901a700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:19:12.188 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefda01c700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 0x7fefd419f0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00bd50 tx=0x7fefcc00bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc019070 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd419fc60 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefcc00cbb0 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0125a0 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.184+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd41a0150 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.185+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fefcc01c070 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.185+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd4063940 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.185+0000 7fefcaffd700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 0x7fefc003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.185+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fefcc04b700 con 0x7fefd4197f20 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.186+0000 7fefd981b700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 0x7fefc003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:12.189 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.186+0000 7fefd981b700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 0x7fefc003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefc4005190 tx=0x7fefc401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:12.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.191+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fefcc007290 con 0x7fefd4197f20 2024-03-19T02:19:12.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.407+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi154=y;smithi162=z", "target": ["mon-mgr", ""]}) -- 0x7fefd40627e0 con 0x7fefc0039450 2024-03-19T02:19:12.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.413+0000 7fefcaffd700 1 -- 172.21.15.162:0/2520749411 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fefd40627e0 con 0x7fefc0039450 2024-03-19T02:19:12.415 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 msgr2=0x7fefc003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 0x7fefc003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefc4005190 tx=0x7fefc401a040 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 msgr2=0x7fefd419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 0x7fefd419f0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00bd50 tx=0x7fefcc00bd80 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 shutdown_connections 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fefc0039450 0x7fefc003b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd4197f20 0x7fefd419f0a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefd4103440 0x7fefd4197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 --2- 172.21.15.162:0/2520749411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefd4197aa0 0x7fefd419cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:12.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 >> 172.21.15.162:0/2520749411 conn(0x7fefd40fba60 msgr2=0x7fefd40fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:12.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 shutdown_connections 2024-03-19T02:19:12.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:19:12.416+0000 7fefdba7f700 1 -- 172.21.15.162:0/2520749411 wait complete. 2024-03-19T02:19:12.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:12.943 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.968 DEBUG:teuthology.orchestra.run.smithi154:mgr.y> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.y.service 2024-03-19T02:19:12.971 DEBUG:teuthology.orchestra.run.smithi162:mgr.z> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.z.service 2024-03-19T02:19:12.973 INFO:tasks.cephadm:Deploying OSDs... 2024-03-19T02:19:12.974 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:19:12.974 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:19:12.993 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:19:12.994 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:12.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:13.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:13.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:13.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:19:13.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:19:13.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:13.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 02:18:17.492514588 +0000 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 02:02:48.892358693 +0000 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 02:02:48.892358693 +0000 2024-03-19T02:19:13.027 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 02:02:48.892358693 +0000 2024-03-19T02:19:13.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:19:13.096 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T02:19:13.096 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T02:19:13.097 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00028049 s, 1.8 MB/s 2024-03-19T02:19:13.098 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:19:13.155 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-19T02:19:13.212 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:19:13.212 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.212 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-19T02:19:13.212 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.212 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.213 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 02:18:17.523514058 +0000 2024-03-19T02:19:13.213 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 02:02:49.231352961 +0000 2024-03-19T02:19:13.213 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 02:02:49.231352961 +0000 2024-03-19T02:19:13.213 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 02:02:49.231352961 +0000 2024-03-19T02:19:13.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:19:13.280 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T02:19:13.280 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T02:19:13.280 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000189859 s, 2.7 MB/s 2024-03-19T02:19:13.281 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:19:13.336 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-19T02:19:13.392 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 02:18:17.554513528 +0000 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 02:02:49.589346906 +0000 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 02:02:49.589346906 +0000 2024-03-19T02:19:13.393 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 02:02:49.589346906 +0000 2024-03-19T02:19:13.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:19:13.459 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T02:19:13.459 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T02:19:13.460 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000206528 s, 2.5 MB/s 2024-03-19T02:19:13.461 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:19:13.517 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 02:18:17.584513014 +0000 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 02:02:49.909341495 +0000 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 02:02:49.909341495 +0000 2024-03-19T02:19:13.572 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 02:02:49.909341495 +0000 2024-03-19T02:19:13.573 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:19:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T02:19:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T02:19:13.638 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000186706 s, 2.7 MB/s 2024-03-19T02:19:13.638 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:19:13.695 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-19T02:19:13.696 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:19:13.715 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:19:13.715 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-03-19T02:19:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:13 smithi116 ceph-mon[32263]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:19:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:13 smithi116 ceph-mon[32263]: Reconfiguring daemon mon.c on smithi162 2024-03-19T02:19:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:13 smithi116 ceph-mon[32263]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi154=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:13 smithi116 ceph-mon[32263]: Saving service mgr spec with placement smithi116=x;smithi154=y;smithi162=z;count:3 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-19 02:19:00.683261393 +0000 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-19 02:02:54.047652380 +0000 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-19 02:02:54.047652380 +0000 2024-03-19T02:19:13.773 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-19 02:02:54.047652380 +0000 2024-03-19T02:19:13.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:19:13.842 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-19T02:19:13.843 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-19T02:19:13.843 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000286607 s, 1.8 MB/s 2024-03-19T02:19:13.844 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:19:13.902 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-03-19T02:19:13.958 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:19:13.958 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:13.958 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-19 02:19:00.683261393 +0000 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-19 02:02:54.405642731 +0000 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-19 02:02:54.405642731 +0000 2024-03-19T02:19:13.959 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-19 02:02:54.405642731 +0000 2024-03-19T02:19:13.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:19:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:13 smithi154 ceph-mon[35693]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:19:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:13 smithi154 ceph-mon[35693]: Reconfiguring daemon mon.c on smithi162 2024-03-19T02:19:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:13 smithi154 ceph-mon[35693]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi154=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:13 smithi154 ceph-mon[35693]: Saving service mgr spec with placement smithi116=x;smithi154=y;smithi162=z;count:3 2024-03-19T02:19:14.026 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-19T02:19:14.026 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-19T02:19:14.026 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000194359 s, 2.6 MB/s 2024-03-19T02:19:14.027 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:19:14.084 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-03-19T02:19:14.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:13 smithi162 ceph-mon[35761]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:19:14.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:13 smithi162 ceph-mon[35761]: Reconfiguring daemon mon.c on smithi162 2024-03-19T02:19:14.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:13 smithi162 ceph-mon[35761]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi154=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:14.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:13 smithi162 ceph-mon[35761]: Saving service mgr spec with placement smithi116=x;smithi154=y;smithi162=z;count:3 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:14.141 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-19 02:19:00.684261365 +0000 2024-03-19T02:19:14.142 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-19 02:02:54.749633458 +0000 2024-03-19T02:19:14.142 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-19 02:02:54.749633458 +0000 2024-03-19T02:19:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-19 02:02:54.749633458 +0000 2024-03-19T02:19:14.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:19:14.208 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-19T02:19:14.208 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-19T02:19:14.208 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000194566 s, 2.6 MB/s 2024-03-19T02:19:14.209 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:19:14.265 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-03-19T02:19:14.320 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:19:14.320 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:14.320 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-19T02:19:14.320 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:14.321 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:14.321 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-19 02:19:00.684261365 +0000 2024-03-19T02:19:14.321 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-19 02:02:55.113623647 +0000 2024-03-19T02:19:14.321 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-19 02:02:55.113623647 +0000 2024-03-19T02:19:14.321 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-19 02:02:55.113623647 +0000 2024-03-19T02:19:14.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:19:14.386 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-19T02:19:14.387 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-19T02:19:14.387 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000192824 s, 2.7 MB/s 2024-03-19T02:19:14.388 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:19:14.443 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T02:19:14.443 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:19:14.460 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:19:14.460 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 02:18:51.402132026 +0000 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 01:56:18.733174228 +0000 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 01:56:18.733174228 +0000 2024-03-19T02:19:14.517 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 01:56:18.733174228 +0000 2024-03-19T02:19:14.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:19:14.585 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T02:19:14.585 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T02:19:14.585 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.00021794 s, 2.3 MB/s 2024-03-19T02:19:14.586 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:19:14.643 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 02:18:51.402132026 +0000 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 01:56:19.057167767 +0000 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 01:56:19.057167767 +0000 2024-03-19T02:19:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 01:56:19.057167767 +0000 2024-03-19T02:19:14.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:19:14.766 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T02:19:14.767 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T02:19:14.767 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000167005 s, 3.1 MB/s 2024-03-19T02:19:14.767 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:19:14.821 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-03-19T02:19:14.881 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:19:14.921 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 02:18:51.402132026 +0000 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 01:56:19.392161087 +0000 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 01:56:19.392161087 +0000 2024-03-19T02:19:14.922 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 01:56:19.392161087 +0000 2024-03-19T02:19:14.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:19:14.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:14 smithi162 ceph-mon[35761]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:14.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:14.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:14.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T02:19:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T02:19:14.949 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000214772 s, 2.4 MB/s 2024-03-19T02:19:14.950 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:19:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:14 smithi154 ceph-mon[35693]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:14 smithi116 ceph-mon[32263]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:15.015 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-03-19T02:19:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 02:18:51.403132005 +0000 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 01:56:19.730154347 +0000 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 01:56:19.730154347 +0000 2024-03-19T02:19:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 01:56:19.730154347 +0000 2024-03-19T02:19:15.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:19:15.140 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T02:19:15.140 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T02:19:15.140 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.0002055 s, 2.5 MB/s 2024-03-19T02:19:15.141 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:19:15.198 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-19T02:19:15.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:19:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:19:15.807 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:15.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:15 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:15.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:15 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:15 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:16.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:16 smithi162 ceph-mon[35761]: Deploying daemon mgr.z on smithi162 2024-03-19T02:19:16.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:16 smithi162 ceph-mon[35761]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:16 smithi154 ceph-mon[35693]: Deploying daemon mgr.z on smithi162 2024-03-19T02:19:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:16 smithi154 ceph-mon[35693]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:16 smithi116 ceph-mon[32263]: Deploying daemon mgr.z on smithi162 2024-03-19T02:19:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:16 smithi116 ceph-mon[32263]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:17.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:17 smithi162 systemd[1]: Started Ceph mgr.z for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011142 s, 941 MB/s 2024-03-19T02:19:17.269 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 2024-03-19T02:19:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T02:19:17.917 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:17.572+0000 7f452a5ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:19:17.917 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:17.663+0000 7f452a5ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:19:17.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-19T02:19:18.214 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:19:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: Deploying daemon mgr.y on smithi154 2024-03-19T02:19:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:18 smithi116 ceph-mon[32263]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:18.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.299+0000 7f452a5ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:19:18.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.393+0000 7f452a5ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:19:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: Deploying daemon mgr.y on smithi154 2024-03-19T02:19:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-mon[35761]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: Deploying daemon mgr.y on smithi154 2024-03-19T02:19:18.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:18 smithi154 ceph-mon[35693]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:18.725 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.540+0000 7f452a5ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:19:18.725 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.640+0000 7f452a5ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:19:18.725 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.724+0000 7f452a5ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:19:19.285 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:18.991+0000 7f452a5ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:19:19.285 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:19.100+0000 7f452a5ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:19:19.285 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:19.186+0000 7f452a5ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:19:19.285 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:19.283+0000 7f452a5ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:19:19.666 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:19 smithi154 ceph-mgr[37095]: -- 172.21.15.154:0/2383753216 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56379f3605a0 con 0x56379f335000 2024-03-19T02:19:19.667 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:19.464+0000 7f452a5ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:19:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/4054117134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38101230 msgr2=0x7f2c38101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/4054117134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38101230 0x7f2c38101650 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c280099c0 tx=0x7f2c28009cd0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/4054117134 shutdown_connections 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/4054117134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38102dd0 0x7f2c38109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/4054117134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38101230 0x7f2c38101650 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/4054117134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/4054117134 >> 172.21.15.116:0/4054117134 conn(0x7f2c380fc7d0 msgr2=0x7f2c380fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/4054117134 shutdown_connections 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.749+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/4054117134 wait complete. 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 Processor -- start 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 -- start start 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 0x7f2c38198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810cd20 con 0x7f2c38102dd0 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810cba0 con 0x7f2c38102430 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3cbb9700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810cea0 con 0x7f2c38101230 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c36d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c36d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52264/0 (socket says 172.21.15.116:52264) 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c36d9d700 1 -- 172.21.15.116:0/494035936 learned_addr learned my addr 172.21.15.116:0/494035936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.750+0000 7f2c3659c700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 0x7f2c38198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c35d9b700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 -- 172.21.15.116:0/494035936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 msgr2=0x7f2c38198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 0x7f2c38198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 -- 172.21.15.116:0/494035936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 msgr2=0x7f2c38198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 -- 172.21.15.116:0/494035936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c381a0120 con 0x7f2c38102dd0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c3659c700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 0x7f2c38198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c35d9b700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c36d9d700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ef60 tx=0x7f2c3000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c300092e0 con 0x7f2c38102dd0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381a03b0 con 0x7f2c38102dd0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c30004bb0 con 0x7f2c38102dd0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30004020 con 0x7f2c38102dd0 2024-03-19T02:19:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.751+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a08f0 con 0x7f2c38102dd0 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.752+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f2c3001b030 con 0x7f2c38102dd0 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.752+0000 7f2c277fe700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 0x7f2c2003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.752+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2c3004c5e0 con 0x7f2c38102dd0 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.752+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c38068fd0 con 0x7f2c38102dd0 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.752+0000 7f2c3659c700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 0x7f2c2003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.753+0000 7f2c3659c700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 0x7f2c2003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800b580 tx=0x7f2c28005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:19.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.757+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c30022020 con 0x7f2c38102dd0 2024-03-19T02:19:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:19.971+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2c380fe9b0 con 0x7f2c20039200 2024-03-19T02:19:20.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:19 smithi154 bash[37076]: c129d349b125d44f10fc7f6ce8b463ee8b87c629625d04af28a034c9f08c4074 2024-03-19T02:19:20.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:19 smithi154 systemd[1]: Started Ceph mgr.y for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:19:20.505 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:20.141+0000 7f5bcc2e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:19:20.506 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:20.238+0000 7f5bcc2e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:19:20.799 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:20.461+0000 7f452a5ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:19:20.799 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:20.603+0000 7f452a5ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:19:20.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:20.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:20.860 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:20.860 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:20.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:20.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:20.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.052 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:20.946+0000 7f452a5ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:19:21.053 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:21.035+0000 7f452a5ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:19:21.157 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:20.922+0000 7f5bcc2e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:19:21.157 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.013+0000 7f5bcc2e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:19:21.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:21.234+0000 7f452a5ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:19:21.505 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.156+0000 7f5bcc2e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:19:21.506 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.261+0000 7f5bcc2e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:19:21.506 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.343+0000 7f5bcc2e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:19:21.705 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:21.426+0000 7f452a5ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:19:21.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:21.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:21.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:21.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:21.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:21.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.892 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.609+0000 7f5bcc2e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.715+0000 7f5bcc2e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.799+0000 7f5bcc2e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:21.891+0000 7f5bcc2e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:21.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:21.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:19:21.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:19:21.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:21.962 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:21.705+0000 7f452a5ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:19:21.963 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:21.962+0000 7f452a5ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:19:22.255 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:22.066+0000 7f5bcc2e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:19:22.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:22.192+0000 7f452a5ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:19:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:22 smithi162 ceph-mon[35761]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:19:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:22 smithi162 ceph-mon[35761]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T02:19:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:22 smithi162 ceph-mon[35761]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:22.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:22 smithi116 ceph-mon[32263]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:19:22.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:22 smithi116 ceph-mon[32263]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T02:19:22.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:22 smithi116 ceph-mon[32263]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:22 smithi154 ceph-mon[35693]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:19:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:22 smithi154 ceph-mon[35693]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T02:19:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:22 smithi154 ceph-mon[35693]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:23.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:22.991+0000 7f452a5ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:19:23.480 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.025+0000 7f5bcc2e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:19:23.480 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.151+0000 7f5bcc2e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:19:23.480 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.479+0000 7f5bcc2e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:19:23.755 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.566+0000 7f5bcc2e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]: dispatch 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]': finished 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:23.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2041066208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:23.883 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:23.567+0000 7f452a5ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:23.666+0000 7f452a5ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:19:23.883+0000 7f452a5ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]: dispatch 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]': finished 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:19:23.884 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:23.885 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2041066208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]: dispatch 2024-03-19T02:19:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1703267037' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "228d73fd-9b05-45b9-b2fb-93b859800b76"}]': finished 2024-03-19T02:19:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:19:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2041066208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:24.212 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.758+0000 7f5bcc2e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:19:24.212 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:23.937+0000 7f5bcc2e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:19:24.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:24.211+0000 7f5bcc2e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:19:24.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:24.469+0000 7f5bcc2e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:19:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:24.655+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 14) ==== 91699+0+0 (secure 0 0 0) 0x7f2c30012050 con 0x7f2c38102dd0 2024-03-19T02:19:24.755 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:24.701+0000 7f5bcc2e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: Standby manager daemon z started 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:19:24.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:24 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: Standby manager daemon z started 2024-03-19T02:19:24.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:19:24.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:24.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:19:24.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:24 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: Standby manager daemon z started 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:19:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:24 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/3585831493' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:25.755 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:25 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:25.482+0000 7f5bcc2e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:19:25.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:25 smithi154 ceph-mon[35693]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T02:19:25.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:25 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:19:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:25 smithi162 ceph-mon[35761]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T02:19:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:25 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:19:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:25 smithi116 ceph-mon[32263]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T02:19:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:25 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:19:26.350 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:26.044+0000 7f5bcc2e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:19:26.350 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:26.139+0000 7f5bcc2e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:19:26.350 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:19:26.348+0000 7f5bcc2e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:19:26.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:26.669+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f2c30019d30 con 0x7f2c38102dd0 2024-03-19T02:19:26.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:26.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: Standby manager daemon y started 2024-03-19T02:19:26.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:19:26.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:26.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:19:26.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:26 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: Standby manager daemon y started 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:19:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:26 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: Standby manager daemon y started 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:19:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:26 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/2555730198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:19:27.937 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:27 smithi116 ceph-mon[32263]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-19T02:19:27.938 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:27 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:19:28.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:27 smithi154 ceph-mon[35693]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-19T02:19:28.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:27 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:19:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:27 smithi162 ceph-mon[35761]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-19T02:19:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:27 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:19:28.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:28.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:28.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:28.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:28.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:28.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:28.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:30.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:30 smithi116 ceph-mon[32263]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:30.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:19:30.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:30 smithi154 ceph-mon[35693]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:19:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:30 smithi162 ceph-mon[35761]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:19:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:31 smithi154 ceph-mon[35693]: Deploying daemon osd.0 on smithi116 2024-03-19T02:19:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:31 smithi116 ceph-mon[32263]: Deploying daemon osd.0 on smithi116 2024-03-19T02:19:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:31 smithi162 ceph-mon[35761]: Deploying daemon osd.0 on smithi116 2024-03-19T02:19:33.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:32 smithi154 ceph-mon[35693]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:32 smithi116 ceph-mon[32263]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:32 smithi162 ceph-mon[35761]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:34 smithi154 ceph-mon[35693]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:34 smithi116 ceph-mon[32263]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:34 smithi162 ceph-mon[35761]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:36 smithi154 ceph-mon[35693]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:36 smithi116 ceph-mon[32263]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:36 smithi162 ceph-mon[35761]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:38.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:38.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:38.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:38.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:38.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:38.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:39.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:39.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:39.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.229+0000 7f2c277fe700 1 -- 172.21.15.116:0/494035936 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c380fe9b0 con 0x7f2c20039200 2024-03-19T02:19:39.229 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 msgr2=0x7f2c2003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 0x7f2c2003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800b580 tx=0x7f2c28005fb0 comp rx=0 tx=0).stop 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 msgr2=0x7f2c3819fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ef60 tx=0x7f2c3000ef90 comp rx=0 tx=0).stop 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 shutdown_connections 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c38102dd0 0x7f2c3819fa80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2c20039200 0x7f2c2003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:39.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c38101230 0x7f2c38198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 --2- 172.21.15.116:0/494035936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c38102430 0x7f2c38198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 >> 172.21.15.116:0/494035936 conn(0x7f2c380fc7d0 msgr2=0x7f2c380fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 shutdown_connections 2024-03-19T02:19:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:19:39.231+0000 7f2c3cbb9700 1 -- 172.21.15.116:0/494035936 wait complete. 2024-03-19T02:19:39.848 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.0.service 2024-03-19T02:19:39.851 INFO:tasks.cephadm:Deploying osd.1 on smithi154 with /dev/vg_nvme/lv_4... 2024-03-19T02:19:39.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:19:40.096 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:19:40.454 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:40 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.454 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:40 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.454 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:40 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:40.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:40 smithi154 ceph-mon[35693]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:40.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:40 smithi154 ceph-mon[35693]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:19:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:40 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:40 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:40 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:40.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:40 smithi116 ceph-mon[32263]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:40.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:40 smithi116 ceph-mon[32263]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:19:40.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:40 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:40 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:40.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:40 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:19:40.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:40 smithi162 ceph-mon[35761]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:40.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:40 smithi162 ceph-mon[35761]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:19:41.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:41 smithi154 ceph-mon[35693]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:19:41.452 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:41 smithi154 ceph-mon[35693]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:19:41.452 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:41 smithi154 ceph-mon[35693]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:19:41.452 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:41 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:41.496 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-mon[32263]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:19:41.496 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-mon[32263]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:19:41.496 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-mon[32263]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:19:41.496 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:41.497 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0[44508]: 2024-03-19T02:19:41.221+0000 7f1b9d400700 -1 osd.0 0 waiting for initial osdmap 2024-03-19T02:19:41.497 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:19:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0[44508]: 2024-03-19T02:19:41.226+0000 7f1b991f3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:19:41.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:41 smithi162 ceph-mon[35761]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:19:41.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:41 smithi162 ceph-mon[35761]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:19:41.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:41 smithi162 ceph-mon[35761]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:19:41.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:41 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.031 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:19:42.031 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:19:42.032 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-19T02:19:42.032 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-19T02:19:42.032 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112173 s, 935 MB/s 2024-03-19T02:19:42.032 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-19T02:19:42.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:42 smithi154 ceph-mon[35693]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:19:42.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:42 smithi154 ceph-mon[35693]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:19:42.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:42 smithi154 ceph-mon[35693]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:42 smithi116 ceph-mon[32263]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:19:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:42 smithi116 ceph-mon[32263]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:19:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.424 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:42 smithi116 ceph-mon[32263]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:42.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:42 smithi162 ceph-mon[35761]: from='osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:19:42.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:42 smithi162 ceph-mon[35761]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:19:42.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:42.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:42 smithi162 ceph-mon[35761]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:19:42.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-03-19T02:19:42.935 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:19:43.312 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:19:43.312 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:19:43.312 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] boot 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:19:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:19:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] boot 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:43.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:19:43.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: osd.0 [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] boot 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:43.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:19:43.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:19:44.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:44 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi116 2024-03-19T02:19:44.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:44 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T02:19:44.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:44 smithi154 ceph-mon[35693]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:19:44.366 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:44 smithi154 ceph-mon[35693]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 -- 172.21.15.154:0/2977442045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103560 msgr2=0x7fe4b01039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/2977442045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103560 0x7fe4b01039c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00099c0 tx=0x7fe4a0009cd0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 -- 172.21.15.154:0/2977442045 shutdown_connections 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/2977442045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103f00 0x7fe4b010a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/2977442045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/2977442045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103560 0x7fe4b01039c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 -- 172.21.15.154:0/2977442045 >> 172.21.15.154:0/2977442045 conn(0x7fe4b00fd920 msgr2=0x7fe4b00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 -- 172.21.15.154:0/2977442045 shutdown_connections 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 -- 172.21.15.154:0/2977442045 wait complete. 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.470+0000 7fe4b85ff700 1 Processor -- start 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 -- start start 2024-03-19T02:19:44.472 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 0x7fe4b0198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010de00 con 0x7fe4b0102360 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010dc80 con 0x7fe4b0103f00 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b85ff700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b010df80 con 0x7fe4b0103560 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:58918/0 (socket says 172.21.15.154:58918) 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 -- 172.21.15.154:0/4032933654 learned_addr learned my addr 172.21.15.154:0/4032933654 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b639b700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b5b9a700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 0x7fe4b0198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:44.473 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 -- 172.21.15.154:0/4032933654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 msgr2=0x7fe4b0198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 0x7fe4b0198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 -- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 msgr2=0x7fe4b01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 -- 172.21.15.154:0/4032933654 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b01a0050 con 0x7fe4b0103f00 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b639b700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b5b9a700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 0x7fe4b0198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4b6b9c700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800ada0 tx=0x7fe4a800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8017020 con 0x7fe4b0103f00 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.471+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4a800ce30 con 0x7fe4b0103f00 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.472+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b01a0340 con 0x7fe4b0103f00 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.472+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8005460 con 0x7fe4b0103f00 2024-03-19T02:19:44.474 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.472+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b01a0880 con 0x7fe4b0103f00 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.473+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b0068fd0 con 0x7fe4b0103f00 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.474+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7fe4a800c980 con 0x7fe4b0103f00 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.474+0000 7fe4a77fe700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 0x7fe49c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.475+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe4a80cce30 con 0x7fe4b0103f00 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.475+0000 7fe4b639b700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 0x7fe49c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:19:44.476 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.475+0000 7fe4b639b700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 0x7fe49c0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4b01033c0 tx=0x7fe4ac00a680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:19:44.478 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.478+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4a8021020 con 0x7fe4b0103f00 2024-03-19T02:19:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:44 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi116 2024-03-19T02:19:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:44 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T02:19:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:44 smithi116 ceph-mon[32263]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:19:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:44 smithi116 ceph-mon[32263]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:44.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:44 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi116 2024-03-19T02:19:44.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:44 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T02:19:44.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:44 smithi162 ceph-mon[35761]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:19:44.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:44 smithi162 ceph-mon[35761]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:44.679 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:19:44.679+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe4b0062810 con 0x7fe49c0a2800 2024-03-19T02:19:45.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:45 smithi154 ceph-mon[35693]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:45 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:45 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:45 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:45 smithi116 ceph-mon[32263]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:45.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:45 smithi162 ceph-mon[35761]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:19:45.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:45 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:19:45.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:45 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:19:45.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:45 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:46.452 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:46 smithi154 ceph-mon[35693]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:46 smithi116 ceph-mon[32263]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:46 smithi162 ceph-mon[35761]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:47.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:47 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/1987639350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:47 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:47 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]': finished 2024-03-19T02:19:47.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:47 smithi154 ceph-mon[35693]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:19:47.623 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:47 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:19:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:47 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/1987639350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:47 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:47 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]': finished 2024-03-19T02:19:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:47 smithi162 ceph-mon[35761]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:19:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:47 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:19:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:47 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/1987639350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:47 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]: dispatch 2024-03-19T02:19:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:47 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd497f40-8e3f-473a-8a69-450ad65aa6d9"}]': finished 2024-03-19T02:19:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:47 smithi116 ceph-mon[32263]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:19:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:19:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:48 smithi162 ceph-mon[35761]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:48 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/2949815541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:48 smithi154 ceph-mon[35693]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:48.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:48 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/2949815541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:48.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:48 smithi116 ceph-mon[32263]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:48.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:48 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/2949815541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:19:50.906 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:50 smithi154 ceph-mon[35693]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:50 smithi162 ceph-mon[35761]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:50 smithi116 ceph-mon[32263]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:52 smithi154 ceph-mon[35693]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:52.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:52 smithi116 ceph-mon[32263]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:52.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:52 smithi162 ceph-mon[35761]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:54.911 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:54 smithi154 ceph-mon[35693]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:54.911 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:19:54.911 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:54 smithi162 ceph-mon[35761]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:19:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:54 smithi116 ceph-mon[32263]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:19:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:19:55.810 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:55 smithi154 ceph-mon[35693]: Deploying daemon osd.1 on smithi154 2024-03-19T02:19:55.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:55 smithi162 ceph-mon[35761]: Deploying daemon osd.1 on smithi154 2024-03-19T02:19:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:55 smithi116 ceph-mon[32263]: Deploying daemon osd.1 on smithi154 2024-03-19T02:19:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:56 smithi154 ceph-mon[35693]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:56 smithi162 ceph-mon[35761]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:56 smithi116 ceph-mon[32263]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:58.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:19:58 smithi154 ceph-mon[35693]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:19:58 smithi162 ceph-mon[35761]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:19:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:19:58 smithi116 ceph-mon[32263]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:00.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:00 smithi154 ceph-mon[35693]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:00.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:00 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:00.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:00 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:00.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:00 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:00.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:00 smithi154 ceph-mon[35693]: overall HEALTH_OK 2024-03-19T02:20:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:00 smithi162 ceph-mon[35761]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:00 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:00 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:00.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:00 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:00.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:00 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:20:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:00 smithi116 ceph-mon[32263]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:00 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:00 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:00 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:00 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:20:02.814 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:02 smithi154 ceph-mon[35693]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:02.896 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.895+0000 7fe4a77fe700 1 -- 172.21.15.154:0/4032933654 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe4b0062810 con 0x7fe49c0a2800 2024-03-19T02:20:02.897 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 1 on host 'smithi154' 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 msgr2=0x7fe49c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 0x7fe49c0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4b01033c0 tx=0x7fe4ac00a680 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 msgr2=0x7fe4b019f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800ada0 tx=0x7fe4a800c330 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 shutdown_connections 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4b0103f00 0x7fe4b019f9b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4b0102360 0x7fe4b01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe49c0a2800 0x7fe49c0a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 --2- 172.21.15.154:0/4032933654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4b0103560 0x7fe4b0198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 >> 172.21.15.154:0/4032933654 conn(0x7fe4b00fd920 msgr2=0x7fe4b0106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 shutdown_connections 2024-03-19T02:20:02.902 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:02.898+0000 7fe4b85ff700 1 -- 172.21.15.154:0/4032933654 wait complete. 2024-03-19T02:20:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:02 smithi162 ceph-mon[35761]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:02 smithi116 ceph-mon[32263]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:03.601 DEBUG:teuthology.orchestra.run.smithi154:osd.1> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.1.service 2024-03-19T02:20:03.603 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_4... 2024-03-19T02:20:03.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:20:03.855 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:03.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:03 smithi162 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:03.934 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:20:03.677+0000 7f98c5cf97c0 -1 osd.1 0 log_to_monitors true 2024-03-19T02:20:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:03.935 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:03.935 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:03.935 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:03 smithi154 ceph-mon[35693]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:04.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:03 smithi116 ceph-mon[32263]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:20:05.115 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:04 smithi162 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:20:05.115 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:04 smithi162 ceph-mon[35761]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.115 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:04 smithi162 ceph-mon[35761]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:20:05.115 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:05.115 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:04 smithi162 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.122 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:04 smithi154 ceph-mon[35693]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:20:05.122 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:04 smithi154 ceph-mon[35693]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.123 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:04 smithi154 ceph-mon[35693]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:20:05.123 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:05.123 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:04 smithi154 ceph-mon[35693]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:04 smithi116 ceph-mon[32263]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:20:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:04 smithi116 ceph-mon[32263]: from='osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:04 smithi116 ceph-mon[32263]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:20:05.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:05.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:04 smithi116 ceph-mon[32263]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:20:05.672 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:20:05.672 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:20:05.672 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T02:20:05.673 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T02:20:05.673 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127711 s, 821 MB/s 2024-03-19T02:20:05.673 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T02:20:05.973 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:20:05.695+0000 7f98bb35e700 -1 osd.1 0 waiting for initial osdmap 2024-03-19T02:20:05.974 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:20:05.702+0000 7f98b3149700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:20:06.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-03-19T02:20:06.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:20:06.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:20:06.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.227 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:06.228 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:05 smithi162 ceph-mon[35761]: from='osd.1 ' entity='osd.1' 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:05 smithi154 ceph-mon[35693]: from='osd.1 ' entity='osd.1' 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:20:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:05 smithi116 ceph-mon[32263]: from='osd.1 ' entity='osd.1' 2024-03-19T02:20:06.443 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:20:07.141 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:20:07.141 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:20:07.141 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.141 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] boot 2024-03-19T02:20:07.141 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi154 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] boot 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi154 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:20:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:20:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: osd.1 [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] boot 2024-03-19T02:20:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi154 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 -- 172.21.15.162:0/2387584260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44102e50 msgr2=0x7f4e44105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/2387584260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44102e50 0x7f4e44105240 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009990 tx=0x7f4e40009ca0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 -- 172.21.15.162:0/2387584260 shutdown_connections 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/2387584260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 0x7f4e4410a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/2387584260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44102e50 0x7f4e44105240 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/2387584260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44105780 0x7f4e44107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 -- 172.21.15.162:0/2387584260 >> 172.21.15.162:0/2387584260 conn(0x7f4e440fc800 msgr2=0x7f4e440fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:07.825 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 -- 172.21.15.162:0/2387584260 shutdown_connections 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.822+0000 7f4e4aed1700 1 -- 172.21.15.162:0/2387584260 wait complete. 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 Processor -- start 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 -- start start 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44102e50 0x7f4e441984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 0x7f4e4419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4410de40 con 0x7f4e441080b0 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4410dcc0 con 0x7f4e44102e50 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4aed1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4410dfc0 con 0x7f4e44105780 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:45292/0 (socket says 172.21.15.162:45292) 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 -- 172.21.15.162:0/3722515680 learned_addr learned my addr 172.21.15.162:0/3722515680 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:20:07.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e48c6d700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44102e50 0x7f4e441984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e4946e700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 0x7f4e4419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 -- 172.21.15.162:0/3722515680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44102e50 msgr2=0x7f4e441984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44102e50 0x7f4e441984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 -- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 msgr2=0x7f4e4419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 0x7f4e4419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 -- 172.21.15.162:0/3722515680 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e441a0120 con 0x7f4e44105780 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e3bfff700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c950 tx=0x7f4e3400cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.823+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34007ac0 con 0x7f4e44105780 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.824+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e34004ce0 con 0x7f4e44105780 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.824+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e340055e0 con 0x7f4e44105780 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.824+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e441a0410 con 0x7f4e44105780 2024-03-19T02:20:07.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.824+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e441a09a0 con 0x7f4e44105780 2024-03-19T02:20:07.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.825+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e44068fd0 con 0x7f4e44105780 2024-03-19T02:20:07.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.826+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f4e34004750 con 0x7f4e44105780 2024-03-19T02:20:07.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.826+0000 7f4e39ffb700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 0x7f4e2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.826+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f4e340cccf0 con 0x7f4e44105780 2024-03-19T02:20:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.827+0000 7f4e48c6d700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 0x7f4e2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:07.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.827+0000 7f4e48c6d700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 0x7f4e2c0a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000b5c0 tx=0x7f4e40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:07.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:07.831+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e34014030 con 0x7f4e44105780 2024-03-19T02:20:08.039 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:08.038+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4e44062750 con 0x7f4e2c0a2880 2024-03-19T02:20:08.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:07 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 17088M 2024-03-19T02:20:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:07 smithi154 ceph-mon[35693]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:07 smithi154 ceph-mon[35693]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:20:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:07 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 17088M 2024-03-19T02:20:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:07 smithi116 ceph-mon[32263]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:07 smithi116 ceph-mon[32263]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:20:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:07 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 17088M 2024-03-19T02:20:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:07 smithi162 ceph-mon[35761]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:07 smithi162 ceph-mon[35761]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:20:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:08 smithi154 ceph-mon[35693]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:08 smithi116 ceph-mon[32263]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:08 smithi162 ceph-mon[35761]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:09.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:10.080 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:09 smithi162 ceph-mon[35761]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:10.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:09 smithi154 ceph-mon[35693]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:09 smithi116 ceph-mon[32263]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:10 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/3705329041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:10 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:10 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]': finished 2024-03-19T02:20:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:10 smithi154 ceph-mon[35693]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:20:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:10 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:10 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/3705329041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:10 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:10 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]': finished 2024-03-19T02:20:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:10 smithi116 ceph-mon[32263]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:20:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:10 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:10 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/3705329041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:10 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]: dispatch 2024-03-19T02:20:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:10 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89b45a09-6328-44d9-84ba-890c9225881f"}]': finished 2024-03-19T02:20:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:10 smithi162 ceph-mon[35761]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:20:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:10 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:12.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:11 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/3614415446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:11 smithi154 ceph-mon[35693]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:11 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/3614415446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:11 smithi116 ceph-mon[32263]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:11 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/3614415446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:11 smithi162 ceph-mon[35761]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:14.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:14 smithi162 ceph-mon[35761]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:14 smithi154 ceph-mon[35693]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:14 smithi116 ceph-mon[32263]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:16.790 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:16 smithi162 ceph-mon[35761]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:16 smithi154 ceph-mon[35693]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:16 smithi116 ceph-mon[32263]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:18.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:18 smithi162 ceph-mon[35761]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:18.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:20:18.776 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:18 smithi154 ceph-mon[35693]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:20:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:18 smithi116 ceph-mon[32263]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:20:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:19.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:19 smithi162 ceph-mon[35761]: Deploying daemon osd.2 on smithi162 2024-03-19T02:20:20.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:19 smithi154 ceph-mon[35693]: Deploying daemon osd.2 on smithi162 2024-03-19T02:20:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:19 smithi116 ceph-mon[32263]: Deploying daemon osd.2 on smithi162 2024-03-19T02:20:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:20 smithi162 ceph-mon[35761]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:20 smithi154 ceph-mon[35693]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:20 smithi116 ceph-mon[32263]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:22 smithi162 ceph-mon[35761]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:22 smithi154 ceph-mon[35693]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:22 smithi116 ceph-mon[32263]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:24 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:24 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:24 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:24 smithi154 ceph-mon[35693]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:24 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:24 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:24 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:24 smithi116 ceph-mon[32263]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:24.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:24 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.519 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:24 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:24.520 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:24 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:24.520 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:24 smithi162 ceph-mon[35761]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:26.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.507+0000 7f4e39ffb700 1 -- 172.21.15.162:0/3722515680 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4e44062750 con 0x7f4e2c0a2880 2024-03-19T02:20:26.507 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-03-19T02:20:26.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 msgr2=0x7f4e2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:26.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 0x7f4e2c0a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000b5c0 tx=0x7f4e40005fd0 comp rx=0 tx=0).stop 2024-03-19T02:20:26.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 msgr2=0x7f4e44198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:26.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c950 tx=0x7f4e3400cc60 comp rx=0 tx=0).stop 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 shutdown_connections 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e441080b0 0x7f4e4419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e44102e50 0x7f4e441984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4e2c0a2880 0x7f4e2c0a4d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 --2- 172.21.15.162:0/3722515680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e44105780 0x7f4e44198a00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 >> 172.21.15.162:0/3722515680 conn(0x7f4e440fc800 msgr2=0x7f4e441037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 shutdown_connections 2024-03-19T02:20:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:20:26.509+0000 7f4e4aed1700 1 -- 172.21.15.162:0/3722515680 wait complete. 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:26.808 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:27.122 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.2.service 2024-03-19T02:20:27.165 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-19T02:20:27.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:20:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:20:27.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:27 smithi162 ceph-mon[35761]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:27.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:27 smithi162 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:27.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:27 smithi116 ceph-mon[32263]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:27.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:27 smithi116 ceph-mon[32263]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:28.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:27 smithi154 ceph-mon[35693]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:28.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:27 smithi154 ceph-mon[35693]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:28.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:28 smithi116 ceph-mon[32263]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:28.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:28.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:20:28.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:20:28.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:28.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:28.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:28.918 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2[40891]: 2024-03-19T02:20:28.609+0000 7fcc64f62700 -1 osd.2 0 waiting for initial osdmap 2024-03-19T02:20:28.918 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2[40891]: 2024-03-19T02:20:28.612+0000 7fcc60d55700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: from='osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:28 smithi154 ceph-mon[35693]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111107 s, 944 MB/s 2024-03-19T02:20:29.417 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T02:20:29.897 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:29 smithi162 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:20:29.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:29 smithi162 ceph-mon[35761]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:20:29.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:29 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:29.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:29 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:29 smithi154 ceph-mon[35693]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:20:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:29 smithi154 ceph-mon[35693]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:20:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:29 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:29 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:29 smithi116 ceph-mon[32263]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:20:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:29 smithi116 ceph-mon[32263]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:20:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:29 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:29 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-19T02:20:30.305 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:20:30.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:20:30.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:20:30.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:30.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] boot 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:30.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:20:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:20:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] boot 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: osd.2 [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] boot 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:31.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 -- 172.21.15.116:0/1504897080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 msgr2=0x7f29e0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1504897080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e0100a40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009990 tx=0x7f29d0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 -- 172.21.15.116:0/1504897080 shutdown_connections 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1504897080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e0103fa0 0x7f29e010a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1504897080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e0100a40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.931+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1504897080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 0x7f29e0103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 -- 172.21.15.116:0/1504897080 >> 172.21.15.116:0/1504897080 conn(0x7f29e00fbfa0 msgr2=0x7f29e00fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 -- 172.21.15.116:0/1504897080 shutdown_connections 2024-03-19T02:20:31.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 -- 172.21.15.116:0/1504897080 wait complete. 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 Processor -- start 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 -- start start 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.932+0000 7f29e52e2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 0x7f29e019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e0103fa0 0x7f29e01a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29e010ddf0 con 0x7f29e0100620 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29e010dc70 con 0x7f29e0103fa0 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29e010df70 con 0x7f29e0103600 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39250/0 (socket says 172.21.15.116:39250) 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 -- 172.21.15.116:0/1513195998 learned_addr learned my addr 172.21.15.116:0/1513195998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29de7fc700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 0x7f29e019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 -- 172.21.15.116:0/1513195998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 msgr2=0x7f29e019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:31.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 0x7f29e019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 -- 172.21.15.116:0/1513195998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e0103fa0 msgr2=0x7f29e01a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e0103fa0 0x7f29e01a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 -- 172.21.15.116:0/1513195998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e01a2310 con 0x7f29e0100620 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29deffd700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29d0000c00 tx=0x7f29d0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d0027070 con 0x7f29e0100620 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e01a25a0 con 0x7f29e0100620 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29d0004090 con 0x7f29e0100620 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d0022db0 con 0x7f29e0100620 2024-03-19T02:20:31.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.933+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e01a2ad0 con 0x7f29e0100620 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.934+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e0068fd0 con 0x7f29e0100620 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.936+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f29d002a070 con 0x7f29e0100620 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.936+0000 7f29c7fff700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 0x7f29c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.937+0000 7f29de7fc700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 0x7f29c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.937+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7f29d00ce8d0 con 0x7f29e0100620 2024-03-19T02:20:31.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.937+0000 7f29de7fc700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 0x7f29c80a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d4004630 tx=0x7f29d400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:31.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:31.941+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29d0031070 con 0x7f29e0100620 2024-03-19T02:20:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:32.151+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f29e0102b10 con 0x7f29c80a2800 2024-03-19T02:20:32.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:31 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi162 2024-03-19T02:20:32.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:31 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-19T02:20:32.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:31 smithi116 ceph-mon[32263]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:20:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:31 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi162 2024-03-19T02:20:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:31 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-19T02:20:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:31 smithi162 ceph-mon[35761]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:20:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:31 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi162 2024-03-19T02:20:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:31 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-19T02:20:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:31 smithi154 ceph-mon[35693]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:20:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:33.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:20:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: pgmap v81: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:33.610 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49344]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:33.610 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49344]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:33.610 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49344]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:33.610 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49344]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:33.863 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:33.864 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:33.864 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49349]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:33.864 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:20:33 smithi116 sudo[49349]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:34.138 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:34.138 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:34 smithi154 ceph-mon[35693]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:20:34.138 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:34 smithi154 ceph-mon[35693]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:34.138 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:34.138 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:33 smithi154 sudo[44113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:34.138 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:33 smithi154 sudo[44113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:34.138 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:33 smithi154 sudo[44113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:34.139 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:33 smithi154 sudo[44113]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:34.199 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:34.200 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 ceph-mon[32263]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:20:34.200 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 ceph-mon[32263]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:34.200 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:34.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:20:34.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:34 smithi162 ceph-mon[35761]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:20:34.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:34 smithi162 ceph-mon[35761]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:34.409 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:34.505 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:34 smithi154 sudo[44116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:34.506 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:34 smithi154 sudo[44116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:34.506 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:34 smithi154 sudo[44116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:34.506 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:20:34 smithi154 sudo[44116]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:34.662 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44806]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:34.662 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44806]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:34.662 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44806]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:34.662 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44806]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:34.917 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44809]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:34.917 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44809]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:34.917 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44809]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:34.917 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:20:34 smithi162 sudo[44809]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:35.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 sudo[49421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:35.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 sudo[49421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:35.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 sudo[49421]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:35.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:34 smithi116 sudo[49421]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:35.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:35.205+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f29d0022470 con 0x7f29e0100620 2024-03-19T02:20:35.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 ceph-mon[35761]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44812]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:35.455 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44812]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 ceph-mon[35693]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:20:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 sudo[49424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 sudo[49424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 sudo[49424]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 sudo[49424]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 ceph-mon[32263]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:35 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:35.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44815]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:35.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44815]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:35.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44815]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:35.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:35 smithi162 sudo[44815]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:36.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 sudo[44119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:20:36.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 sudo[44119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:36.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 sudo[44119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:36.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:35 smithi154 sudo[44119]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 sudo[44122]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 sudo[44122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 sudo[44122]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 sudo[44122]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:20:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]: dispatch 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]': finished 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:20:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]': finished 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]: dispatch 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3209195612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6c5e73-72bf-40e9-95b8-8473f091ea90"}]': finished 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:20:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:36.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:20:36.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:20:36.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:20:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:37 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:37 smithi154 ceph-mon[35693]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2000393087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:37 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:37 smithi116 ceph-mon[32263]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2000393087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:37 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:20:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:37 smithi162 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:20:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2000393087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:20:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:38 smithi154 ceph-mon[35693]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:38 smithi116 ceph-mon[32263]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:38.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:38 smithi162 ceph-mon[35761]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:40.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:40 smithi116 ceph-mon[32263]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:40.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:40 smithi162 ceph-mon[35761]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:40 smithi154 ceph-mon[35693]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:42 smithi162 ceph-mon[35761]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:43.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:42 smithi154 ceph-mon[35693]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:42 smithi116 ceph-mon[32263]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:43.693 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:20:43.694 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:20:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:20:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:44.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:44 smithi116 ceph-mon[32263]: Deploying daemon osd.3 on smithi116 2024-03-19T02:20:44.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:44 smithi116 ceph-mon[32263]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:44 smithi162 ceph-mon[35761]: Deploying daemon osd.3 on smithi116 2024-03-19T02:20:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:44 smithi162 ceph-mon[35761]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:44 smithi154 ceph-mon[35693]: Deploying daemon osd.3 on smithi116 2024-03-19T02:20:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:44 smithi154 ceph-mon[35693]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:46.709 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:46 smithi116 ceph-mon[32263]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:46.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:46 smithi162 ceph-mon[35761]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:47.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:46 smithi154 ceph-mon[35693]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:48.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:48 smithi116 ceph-mon[32263]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:48.909 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:48.909 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:48.909 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:48 smithi162 ceph-mon[35761]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:48 smithi154 ceph-mon[35693]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:50.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:50 smithi116 ceph-mon[32263]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:50 smithi162 ceph-mon[35761]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:50 smithi154 ceph-mon[35693]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.125+0000 7f29c7fff700 1 -- 172.21.15.116:0/1513195998 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f29e0102b10 con 0x7f29c80a2800 2024-03-19T02:20:52.126 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2024-03-19T02:20:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 msgr2=0x7f29c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 0x7f29c80a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d4004630 tx=0x7f29d400b3c0 comp rx=0 tx=0).stop 2024-03-19T02:20:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 msgr2=0x7f29e019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29d0000c00 tx=0x7f29d0009580 comp rx=0 tx=0).stop 2024-03-19T02:20:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 shutdown_connections 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e0103fa0 0x7f29e01a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29e0100620 0x7f29e019a6b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29c80a2800 0x7f29c80a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 --2- 172.21.15.116:0/1513195998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e0103600 0x7f29e019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.127+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 >> 172.21.15.116:0/1513195998 conn(0x7f29e00fbfa0 msgr2=0x7f29e01013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.128+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 shutdown_connections 2024-03-19T02:20:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:20:52.128+0000 7f29e52e2700 1 -- 172.21.15.116:0/1513195998 wait complete. 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.195 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:51 smithi116 ceph-mon[32263]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:52.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:51 smithi154 ceph-mon[35693]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:52.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:52.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:52.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:51 smithi162 ceph-mon[35761]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:52.812 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.3.service 2024-03-19T02:20:52.815 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_3... 2024-03-19T02:20:52.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:20:53.059 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:20:53.389 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:53.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:53 smithi116 ceph-mon[32263]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:20:53.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:53.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:53 smithi154 ceph-mon[35693]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:20:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:20:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:53 smithi162 ceph-mon[35761]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:20:54.389 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-mon[32263]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:20:54.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-mon[32263]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:20:54.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-mon[32263]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:20:54.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:54.390 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-mon[32263]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:54.390 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3[51460]: 2024-03-19T02:20:54.122+0000 7f53a40f7700 -1 osd.3 0 waiting for initial osdmap 2024-03-19T02:20:54.390 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:20:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3[51460]: 2024-03-19T02:20:54.127+0000 7f539feea700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:20:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:54 smithi162 ceph-mon[35761]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:20:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:54 smithi162 ceph-mon[35761]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:20:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:54 smithi162 ceph-mon[35761]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:20:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:54 smithi162 ceph-mon[35761]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:54.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:54 smithi154 ceph-mon[35693]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:20:54.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:54 smithi154 ceph-mon[35693]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:20:54.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:54 smithi154 ceph-mon[35693]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:20:54.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:54.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:54 smithi154 ceph-mon[35693]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:55.050 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:20:55.050 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:20:55.050 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-19T02:20:55.050 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-19T02:20:55.050 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113821 s, 921 MB/s 2024-03-19T02:20:55.051 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-19T02:20:55.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:55 smithi154 ceph-mon[35693]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:20:55.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:55 smithi154 ceph-mon[35693]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:20:55.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:55 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:55 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:55 smithi162 ceph-mon[35761]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:20:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:55 smithi162 ceph-mon[35761]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:20:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:55 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:55 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.483 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:55 smithi116 ceph-mon[32263]: from='osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:20:55.484 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:55 smithi116 ceph-mon[32263]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:20:55.484 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:55 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.484 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:55 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:55.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-03-19T02:20:55.959 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:20:56.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:20:56.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:20:56.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] boot 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:56.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:20:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:20:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] boot 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: osd.3 [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] boot 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:20:56.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 -- 172.21.15.154:0/3721748169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102440 msgr2=0x7f04ac1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3721748169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102440 0x7f04ac1028a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f04a80099c0 tx=0x7f04a8009cd0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 -- 172.21.15.154:0/3721748169 shutdown_connections 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3721748169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac102de0 0x7f04ac109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3721748169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac101240 0x7f04ac101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3721748169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102440 0x7f04ac1028a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 -- 172.21.15.154:0/3721748169 >> 172.21.15.154:0/3721748169 conn(0x7f04ac0fc800 msgr2=0x7f04ac0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 -- 172.21.15.154:0/3721748169 shutdown_connections 2024-03-19T02:20:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.391+0000 7f04b44d4700 1 -- 172.21.15.154:0/3721748169 wait complete. 2024-03-19T02:20:57.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:57 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi116 2024-03-19T02:20:57.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:57 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T02:20:57.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:57 smithi154 ceph-mon[35693]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 Processor -- start 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 -- start start 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac101240 0x7f04ac1051d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac102440 0x7f04ac103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac10cd50 con 0x7f04ac101240 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac10cbd0 con 0x7f04ac102de0 2024-03-19T02:20:57.395 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b44d4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac10ced0 con 0x7f04ac102440 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b2a71700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b2a71700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:35618/0 (socket says 172.21.15.154:35618) 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.392+0000 7f04b2a71700 1 -- 172.21.15.154:0/3935023364 learned_addr learned my addr 172.21.15.154:0/3935023364 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2270700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac101240 0x7f04ac1051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b1a6f700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac102440 0x7f04ac103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 -- 172.21.15.154:0/3935023364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac102440 msgr2=0x7f04ac103820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac102440 0x7f04ac103820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 -- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac101240 msgr2=0x7f04ac1051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac101240 0x7f04ac1051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 -- 172.21.15.154:0/3935023364 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac104610 con 0x7f04ac102de0 2024-03-19T02:20:57.396 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b2a71700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007ee0 tx=0x7f04a400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4004bb0 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac1a2690 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04a4004d50 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40058e0 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.393+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac1a2c20 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.395+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac068fd0 con 0x7f04ac102de0 2024-03-19T02:20:57.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.395+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f04a401a020 con 0x7f04ac102de0 2024-03-19T02:20:57.398 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.396+0000 7f04a37fe700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 0x7f04980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:20:57.398 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.396+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7f04a40ccfc0 con 0x7f04ac102de0 2024-03-19T02:20:57.398 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.396+0000 7f04b2270700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 0x7f04980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:20:57.398 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.397+0000 7f04b2270700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 0x7f04980a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04ac1022a0 tx=0x7f049c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:20:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.401+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04a4010070 con 0x7f04ac102de0 2024-03-19T02:20:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:57 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi116 2024-03-19T02:20:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:57 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T02:20:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:57 smithi162 ceph-mon[35761]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:20:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:57 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi116 2024-03-19T02:20:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:57 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T02:20:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:57 smithi116 ceph-mon[32263]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:20:57.606 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:20:57.606+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f04ac0fea30 con 0x7f04980a2890 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:58.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:58.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:20:58.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:58 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:20:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:20:58.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:20:59.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:20:59 smithi154 ceph-mon[35693]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:20:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:20:59 smithi162 ceph-mon[35761]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:20:59.421 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:20:59 smithi116 ceph-mon[32263]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:21:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:00 smithi162 ceph-mon[35761]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:21:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:00 smithi162 ceph-mon[35761]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:00 smithi154 ceph-mon[35693]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:21:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:00 smithi154 ceph-mon[35693]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:00 smithi116 ceph-mon[32263]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:21:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:00 smithi116 ceph-mon[32263]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/2519155170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]': finished 2024-03-19T02:21:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:21:01.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:01.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:01 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/1731044676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/2519155170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]': finished 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:01 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/1731044676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/2519155170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]: dispatch 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4bf282e6-a231-437d-af9b-f8b7a11d67c2"}]': finished 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:01 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/1731044676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:02 smithi154 ceph-mon[35693]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:02 smithi116 ceph-mon[32263]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:02.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:02 smithi162 ceph-mon[35761]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:04 smithi162 ceph-mon[35761]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:04 smithi154 ceph-mon[35693]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:04 smithi116 ceph-mon[32263]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:06 smithi154 ceph-mon[35693]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-19T02:21:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:06 smithi162 ceph-mon[35761]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-19T02:21:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:06 smithi116 ceph-mon[32263]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-19T02:21:07.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:21:07.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:21:08.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:08.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:21:08.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:08 smithi154 ceph-mon[35693]: Deploying daemon osd.4 on smithi154 2024-03-19T02:21:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:08 smithi154 ceph-mon[35693]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T02:21:08.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:08 smithi162 ceph-mon[35761]: Deploying daemon osd.4 on smithi154 2024-03-19T02:21:08.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:08 smithi162 ceph-mon[35761]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T02:21:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:08 smithi116 ceph-mon[32263]: Deploying daemon osd.4 on smithi154 2024-03-19T02:21:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:08 smithi116 ceph-mon[32263]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T02:21:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:10 smithi162 ceph-mon[35761]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:21:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:10 smithi154 ceph-mon[35693]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:21:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:10 smithi116 ceph-mon[32263]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:21:12.890 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:12 smithi154 ceph-mon[35693]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:21:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:12 smithi162 ceph-mon[35761]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:21:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:12 smithi116 ceph-mon[32263]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:21:14.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:13 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:13 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:13 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:14.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:13 smithi154 ceph-mon[35693]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:21:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:13 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:13 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:13 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:13 smithi116 ceph-mon[32263]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:21:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:13 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:13 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:13 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:13 smithi162 ceph-mon[35761]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:21:16.622 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2024-03-19T02:21:16.623 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.622+0000 7f04a37fe700 1 -- 172.21.15.154:0/3935023364 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f04ac0fea30 con 0x7f04980a2890 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 msgr2=0x7f04980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 0x7f04980a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04ac1022a0 tx=0x7f049c009190 comp rx=0 tx=0).stop 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 msgr2=0x7f04ac103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007ee0 tx=0x7f04a400d3b0 comp rx=0 tx=0).stop 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 shutdown_connections 2024-03-19T02:21:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04ac102de0 0x7f04ac103d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04ac101240 0x7f04ac1051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04980a2890 0x7f04980a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 --2- 172.21.15.154:0/3935023364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04ac102440 0x7f04ac103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 >> 172.21.15.154:0/3935023364 conn(0x7f04ac0fc800 msgr2=0x7f04ac0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 shutdown_connections 2024-03-19T02:21:16.625 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:21:16.624+0000 7f04b44d4700 1 -- 172.21.15.154:0/3935023364 wait complete. 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.861 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:16.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:16.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:17.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.339 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.4.service 2024-03-19T02:21:17.341 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-03-19T02:21:17.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:21:17.589 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:21:17.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:17 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:17 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:17.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:17 smithi162 ceph-mon[35761]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:17.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:17 smithi162 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:17.885 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:17 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:17.885 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:17 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:17.885 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:17 smithi154 ceph-mon[35693]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:17.885 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:17 smithi154 ceph-mon[35693]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:17 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:17 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:17 smithi116 ceph-mon[32263]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:17 smithi116 ceph-mon[32263]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:18.868 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:18 smithi154 ceph-mon[35693]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:18.873 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:18 smithi162 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:21:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:21:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: from='osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:18 smithi116 ceph-mon[32263]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:21:19.615 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:21:19.283+0000 7ff612480700 -1 osd.4 0 waiting for initial osdmap 2024-03-19T02:21:19.615 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:21:19.289+0000 7ff609a6a700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115433 s, 908 MB/s 2024-03-19T02:21:19.645 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T02:21:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:19 smithi162 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:21:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:19 smithi162 ceph-mon[35761]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:21:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:19 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:19 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:19 smithi162 ceph-mon[35761]: from='osd.4 ' entity='osd.4' 2024-03-19T02:21:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-mon[35693]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:21:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-mon[35693]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:21:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:19 smithi154 ceph-mon[35693]: from='osd.4 ' entity='osd.4' 2024-03-19T02:21:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:19 smithi116 ceph-mon[32263]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:21:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:19 smithi116 ceph-mon[32263]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:21:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:19 smithi116 ceph-mon[32263]: from='osd.4 ' entity='osd.4' 2024-03-19T02:21:20.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-03-19T02:21:20.550 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] boot 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:20.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:21:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:21:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] boot 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: osd.4 [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] boot 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:21.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:21.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:21 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi154 2024-03-19T02:21:21.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:21 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 8544M 2024-03-19T02:21:22.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:21 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi154 2024-03-19T02:21:22.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:21 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 8544M 2024-03-19T02:21:22.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:21 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi154 2024-03-19T02:21:22.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:21 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 8544M 2024-03-19T02:21:22.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 -- 172.21.15.162:0/62053901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c102340 msgr2=0x7fc96c102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:22.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 --2- 172.21.15.162:0/62053901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c102340 0x7fc96c102760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc96400b0a0 tx=0x7fc96400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 -- 172.21.15.162:0/62053901 shutdown_connections 2024-03-19T02:21:22.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 --2- 172.21.15.162:0/62053901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c103ee0 0x7fc96c10a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 --2- 172.21.15.162:0/62053901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c102340 0x7fc96c102760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 --2- 172.21.15.162:0/62053901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 -- 172.21.15.162:0/62053901 >> 172.21.15.162:0/62053901 conn(0x7fc96c0fd920 msgr2=0x7fc96c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 -- 172.21.15.162:0/62053901 shutdown_connections 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.252+0000 7fc973f31700 1 -- 172.21.15.162:0/62053901 wait complete. 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 Processor -- start 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 -- start start 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c102340 0x7fc96c0760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10de40 con 0x7fc96c103540 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10dcc0 con 0x7fc96c102340 2024-03-19T02:21:22.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc973f31700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10dfc0 con 0x7fc96c103ee0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40944/0 (socket says 172.21.15.162:40944) 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 -- 172.21.15.162:0/2748970591 learned_addr learned my addr 172.21.15.162:0/2748970591 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9714cc700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc971ccd700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c102340 0x7fc96c0760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 -- 172.21.15.162:0/2748970591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c102340 msgr2=0x7fc96c0760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c102340 0x7fc96c0760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.253+0000 7fc9724ce700 1 -- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 msgr2=0x7fc96c074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc9724ce700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc9724ce700 1 -- 172.21.15.162:0/2748970591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c0754f0 con 0x7fc96c103ee0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc9714cc700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc9724ce700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc96800b550 tx=0x7fc96800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968015020 con 0x7fc96c103ee0 2024-03-19T02:21:22.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc968011ce0 con 0x7fc96c103ee0 2024-03-19T02:21:22.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968010cb0 con 0x7fc96c103ee0 2024-03-19T02:21:22.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96c1a26f0 con 0x7fc96c103ee0 2024-03-19T02:21:22.257 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.254+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96c1a2c30 con 0x7fc96c103ee0 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.255+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96c068fd0 con 0x7fc96c103ee0 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.256+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc968018020 con 0x7fc96c103ee0 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.257+0000 7fc962ffd700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 0x7fc9580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.257+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fc9680cd780 con 0x7fc96c103ee0 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.257+0000 7fc971ccd700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 0x7fc9580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:22.258 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.257+0000 7fc971ccd700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 0x7fc9580a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc96400b070 tx=0x7fc9640060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:21:22.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.260+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc96801f020 con 0x7fc96c103ee0 2024-03-19T02:21:22.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:22.466+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc96c062750 con 0x7fc9580a2890 2024-03-19T02:21:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:22 smithi162 ceph-mon[35761]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:22 smithi162 ceph-mon[35761]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:21:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:22 smithi154 ceph-mon[35693]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:22 smithi154 ceph-mon[35693]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:21:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:22 smithi116 ceph-mon[32263]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:22 smithi116 ceph-mon[32263]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:21:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:23.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:23 smithi162 ceph-mon[35761]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:24.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:23 smithi154 ceph-mon[35693]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:23 smithi116 ceph-mon[32263]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:24 smithi162 ceph-mon[35761]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:24 smithi154 ceph-mon[35693]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:24 smithi116 ceph-mon[32263]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:25 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2235621455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:25 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:25 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]': finished 2024-03-19T02:21:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:25 smithi162 ceph-mon[35761]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:21:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:25 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:26.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:25 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/2235621455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:26.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:25 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:26.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:25 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]': finished 2024-03-19T02:21:26.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:25 smithi154 ceph-mon[35693]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:21:26.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:25 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:25 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2235621455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:25 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]: dispatch 2024-03-19T02:21:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:25 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639"}]': finished 2024-03-19T02:21:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:25 smithi116 ceph-mon[32263]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:21:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:25 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:26 smithi162 ceph-mon[35761]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:26 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/1547576767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:26 smithi154 ceph-mon[35693]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:27.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:26 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/1547576767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:26 smithi116 ceph-mon[32263]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:26 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/1547576767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:28.898 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:28 smithi162 ceph-mon[35761]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:29.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:28 smithi154 ceph-mon[35693]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:28 smithi116 ceph-mon[32263]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:30 smithi162 ceph-mon[35761]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:30 smithi154 ceph-mon[35693]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:30 smithi116 ceph-mon[32263]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:32 smithi162 ceph-mon[35761]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:21:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:33.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:32 smithi154 ceph-mon[35693]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:21:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:32 smithi116 ceph-mon[32263]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:21:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:33.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:33 smithi162 ceph-mon[35761]: Deploying daemon osd.5 on smithi162 2024-03-19T02:21:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:33 smithi154 ceph-mon[35693]: Deploying daemon osd.5 on smithi162 2024-03-19T02:21:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:33 smithi116 ceph-mon[32263]: Deploying daemon osd.5 on smithi162 2024-03-19T02:21:34.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:34 smithi162 ceph-mon[35761]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:34 smithi154 ceph-mon[35693]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:34 smithi116 ceph-mon[32263]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:36.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:36 smithi162 ceph-mon[35761]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:36 smithi154 ceph-mon[35693]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:36 smithi116 ceph-mon[32263]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:38.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:38 smithi162 ceph-mon[35761]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:38.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:38.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:38.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:38 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:39.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:38 smithi154 ceph-mon[35693]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:38 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:38 smithi116 ceph-mon[32263]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:38 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:40.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:40 smithi162 ceph-mon[35761]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:41.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:40 smithi154 ceph-mon[35693]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:40 smithi116 ceph-mon[32263]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:41.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.936+0000 7fc962ffd700 1 -- 172.21.15.162:0/2748970591 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc96c062750 con 0x7fc9580a2890 2024-03-19T02:21:41.937 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 msgr2=0x7fc9580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 0x7fc9580a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc96400b070 tx=0x7fc9640060c0 comp rx=0 tx=0).stop 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 msgr2=0x7fc96c074c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc96800b550 tx=0x7fc96800b910 comp rx=0 tx=0).stop 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 shutdown_connections 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc9580a2890 0x7fc9580a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc96c103ee0 0x7fc96c074c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c102340 0x7fc96c0760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:41.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 --2- 172.21.15.162:0/2748970591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc96c103540 0x7fc96c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:41.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 >> 172.21.15.162:0/2748970591 conn(0x7fc96c0fd920 msgr2=0x7fc96c106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:21:41.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 shutdown_connections 2024-03-19T02:21:41.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:21:41.938+0000 7fc973f31700 1 -- 172.21.15.162:0/2748970591 wait complete. 2024-03-19T02:21:42.605 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.5.service 2024-03-19T02:21:42.607 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-19T02:21:42.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:21:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:21:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:42 smithi162 ceph-mon[35761]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:42 smithi162 ceph-mon[35761]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:42 smithi162 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:42.981 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:42 smithi116 ceph-mon[32263]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:42.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:42.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:42.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:42 smithi116 ceph-mon[32263]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:42.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:42 smithi116 ceph-mon[32263]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:42 smithi154 ceph-mon[35693]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:42 smithi154 ceph-mon[35693]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:42 smithi154 ceph-mon[35693]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:21:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:21:43.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:21:43.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:43.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:43.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:43.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-mon[35761]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:44.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:43 smithi154 ceph-mon[35693]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:21:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:43 smithi116 ceph-mon[32263]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:21:44.363 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:21:43.920+0000 7f56708c3700 -1 osd.5 0 waiting for initial osdmap 2024-03-19T02:21:44.363 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:21:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:21:43.926+0000 7f566c6b6700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111977 s, 936 MB/s 2024-03-19T02:21:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T02:21:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:44 smithi162 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:21:45.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:44 smithi162 ceph-mon[35761]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:21:45.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:44 smithi154 ceph-mon[35693]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:21:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:44 smithi154 ceph-mon[35693]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:21:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:44 smithi116 ceph-mon[32263]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:21:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:44 smithi116 ceph-mon[32263]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:21:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:45.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-19T02:21:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:21:46.106 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:21:46.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:21:46.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] boot 2024-03-19T02:21:46.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:21:46.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:46.107 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:45 smithi162 ceph-mon[35761]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:21:46.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:21:46.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:21:46.149 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] boot 2024-03-19T02:21:46.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:21:46.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:46.150 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:45 smithi116 ceph-mon[32263]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:21:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:21:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:21:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: osd.5 [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] boot 2024-03-19T02:21:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:21:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:21:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:45 smithi154 ceph-mon[35693]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi162 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:47.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi162 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:21:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi162 2024-03-19T02:21:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:21:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:21:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 -- 172.21.15.116:0/1523248116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c103ee0 msgr2=0x7fd67c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 --2- 172.21.15.116:0/1523248116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c103ee0 0x7fd67c10a790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd6740099c0 tx=0x7fd674009cd0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 -- 172.21.15.116:0/1523248116 shutdown_connections 2024-03-19T02:21:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 --2- 172.21.15.116:0/1523248116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c103ee0 0x7fd67c10a790 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 --2- 172.21.15.116:0/1523248116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c102340 0x7fd67c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 --2- 172.21.15.116:0/1523248116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103540 0x7fd67c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 -- 172.21.15.116:0/1523248116 >> 172.21.15.116:0/1523248116 conn(0x7fd67c0fd920 msgr2=0x7fd67c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 -- 172.21.15.116:0/1523248116 shutdown_connections 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 -- 172.21.15.116:0/1523248116 wait complete. 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.445+0000 7fd68140b700 1 Processor -- start 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 -- start start 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c103540 0x7fd67c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 0x7fd67c19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10dde0 con 0x7fd67c102340 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10dc60 con 0x7fd67c103540 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10df60 con 0x7fd67c103ee0 2024-03-19T02:21:47.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34540/0 (socket says 172.21.15.116:34540) 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 -- 172.21.15.116:0/2568111617 learned_addr learned my addr 172.21.15.116:0/2568111617 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67b7fe700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 0x7fd67c19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67a7fc700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c103540 0x7fd67c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 -- 172.21.15.116:0/2568111617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 msgr2=0x7fd67c19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 0x7fd67c19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 -- 172.21.15.116:0/2568111617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c103540 msgr2=0x7fd67c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c103540 0x7fd67c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 -- 172.21.15.116:0/2568111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c1a0060 con 0x7fd67c102340 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67b7fe700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 0x7fd67c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd67affd700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00add0 tx=0x7fd66c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c017020 con 0x7fd67c102340 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.446+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c1a0350 con 0x7fd67c102340 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.447+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd66c00c940 con 0x7fd67c102340 2024-03-19T02:21:47.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.447+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c004260 con 0x7fd67c102340 2024-03-19T02:21:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.447+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1a08e0 con 0x7fd67c102340 2024-03-19T02:21:47.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.447+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c068fd0 con 0x7fd67c102340 2024-03-19T02:21:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.448+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd66c00caf0 con 0x7fd67c102340 2024-03-19T02:21:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.449+0000 7fd673fff700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 0x7fd6680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:21:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.449+0000 7fd67a7fc700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 0x7fd6680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:21:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.450+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7fd66c0cd5f0 con 0x7fd67c102340 2024-03-19T02:21:47.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.450+0000 7fd67a7fc700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 0x7fd6680a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd664005950 tx=0x7fd664005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:21:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.453+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd66c021020 con 0x7fd67c102340 2024-03-19T02:21:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:21:47.662+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd67c062750 con 0x7fd6680a2890 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:47 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:21:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:47 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: pgmap v141: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:21:48.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:47 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:49.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:48 smithi116 ceph-mon[32263]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:21:49.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:48 smithi116 ceph-mon[32263]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:21:49.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:48 smithi154 ceph-mon[35693]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:21:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:48 smithi154 ceph-mon[35693]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:21:49.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:48 smithi162 ceph-mon[35761]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:21:49.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:48 smithi162 ceph-mon[35761]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:21:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:49 smithi154 ceph-mon[35693]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:21:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:49 smithi154 ceph-mon[35693]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:49 smithi116 ceph-mon[32263]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:21:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:49 smithi116 ceph-mon[32263]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:49 smithi162 ceph-mon[35761]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:21:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:49 smithi162 ceph-mon[35761]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:50 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]: dispatch 2024-03-19T02:21:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:50 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]': finished 2024-03-19T02:21:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:50 smithi154 ceph-mon[35693]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:21:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:21:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:50 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]: dispatch 2024-03-19T02:21:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:50 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]': finished 2024-03-19T02:21:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:50 smithi116 ceph-mon[32263]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:21:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:21:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:50 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]: dispatch 2024-03-19T02:21:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:50 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2113312280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9047802-fa47-4d12-98d7-b7a23641af81"}]': finished 2024-03-19T02:21:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:50 smithi162 ceph-mon[35761]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:21:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:21:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:51 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3153301605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:51 smithi154 ceph-mon[35693]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:51 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3153301605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:51 smithi116 ceph-mon[32263]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:51 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3153301605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:21:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:51 smithi162 ceph-mon[35761]: pgmap v146: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:21:53.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:53 smithi162 ceph-mon[35761]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:21:53.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:53 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:21:53.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:53 smithi116 ceph-mon[32263]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:21:53.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:53 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:21:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:53 smithi154 ceph-mon[35693]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:21:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:53 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:21:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:54 smithi162 ceph-mon[35761]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:55.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:54 smithi154 ceph-mon[35693]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:54 smithi116 ceph-mon[32263]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:21:56.792 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:56 smithi116 ceph-mon[32263]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-19T02:21:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:56 smithi162 ceph-mon[35761]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-19T02:21:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:56 smithi154 ceph-mon[35693]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-19T02:21:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:58 smithi116 ceph-mon[32263]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-19T02:21:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:21:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:58 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:58 smithi162 ceph-mon[35761]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-19T02:21:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:21:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:58 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:58 smithi154 ceph-mon[35693]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-19T02:21:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:58 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:21:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:58 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:21:59.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:21:59 smithi116 ceph-mon[32263]: Deploying daemon osd.6 on smithi116 2024-03-19T02:22:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:21:59 smithi154 ceph-mon[35693]: Deploying daemon osd.6 on smithi116 2024-03-19T02:22:00.040 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:21:59 smithi162 ceph-mon[35761]: Deploying daemon osd.6 on smithi116 2024-03-19T02:22:00.753 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:00 smithi116 ceph-mon[32263]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:22:00.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:00 smithi154 ceph-mon[35693]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:22:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:00 smithi162 ceph-mon[35761]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:22:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:02 smithi154 ceph-mon[35693]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:22:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:02 smithi116 ceph-mon[32263]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:22:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:02 smithi162 ceph-mon[35761]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-19T02:22:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:04 smithi116 ceph-mon[32263]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:22:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:04 smithi154 ceph-mon[35693]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:22:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:04 smithi162 ceph-mon[35761]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:22:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:06.982 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:06 smithi116 ceph-mon[32263]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:06 smithi154 ceph-mon[35693]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:07.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:06 smithi162 ceph-mon[35761]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:08.157 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-19T02:22:08.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.155+0000 7fd673fff700 1 -- 172.21.15.116:0/2568111617 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd67c062750 con 0x7fd6680a2890 2024-03-19T02:22:08.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 msgr2=0x7fd6680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 0x7fd6680a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd664005950 tx=0x7fd664005eb0 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 msgr2=0x7fd67c198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00add0 tx=0x7fd66c00c330 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 shutdown_connections 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd6680a2890 0x7fd6680a4d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd67c103ee0 0x7fd67c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c102340 0x7fd67c198400 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 --2- 172.21.15.116:0/2568111617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd67c103540 0x7fd67c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 >> 172.21.15.116:0/2568111617 conn(0x7fd67c0fd920 msgr2=0x7fd67c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 shutdown_connections 2024-03-19T02:22:08.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:22:08.158+0000 7fd68140b700 1 -- 172.21.15.116:0/2568111617 wait complete. 2024-03-19T02:22:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:09.000 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.6.service 2024-03-19T02:22:09.002 INFO:tasks.cephadm:Deploying osd.7 on smithi154 with /dev/vg_nvme/lv_2... 2024-03-19T02:22:09.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:22:09.252 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:22:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-mon[32263]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:22:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-mon[32263]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:22:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-mon[32263]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:22:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:09.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:09 smithi154 ceph-mon[35693]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:22:09.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:09 smithi154 ceph-mon[35693]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:22:09.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:09 smithi154 ceph-mon[35693]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:22:09.373 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:09 smithi162 ceph-mon[35761]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:22:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:09 smithi162 ceph-mon[35761]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:22:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:09 smithi162 ceph-mon[35761]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:22:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.206 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:22:09.872+0000 7f0c596d6700 -1 osd.6 0 waiting for initial osdmap 2024-03-19T02:22:10.207 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:22:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:22:09.880+0000 7f0c514c1700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:22:10.500 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:22:10.500 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:22:10.500 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.500 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.501 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:10.501 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' 2024-03-19T02:22:10.501 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:10 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' 2024-03-19T02:22:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:10 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:22:10.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:22:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:22:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: from='osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604]' entity='osd.6' 2024-03-19T02:22:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:10 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0187433 s, 559 MB/s 2024-03-19T02:22:11.280 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-19T02:22:11.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-03-19T02:22:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:22:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:22:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] boot 2024-03-19T02:22:12.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:22:12.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:12.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:11 smithi162 ceph-mon[35761]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:12.179 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:22:12.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:22:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:22:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] boot 2024-03-19T02:22:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:22:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:11 smithi154 ceph-mon[35693]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:22:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:22:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: osd.6 [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] boot 2024-03-19T02:22:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:22:12.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:22:12.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:11 smithi116 ceph-mon[32263]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:12.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:22:12.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi116 2024-03-19T02:22:13.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi116 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:12 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:12 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi116 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T02:22:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:12 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:13.761 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 -- 172.21.15.154:0/318669114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70101240 msgr2=0x7f9c70101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:13.761 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 --2- 172.21.15.154:0/318669114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70101240 0x7f9c70101660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9c600099c0 tx=0x7f9c60009cd0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.761 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 -- 172.21.15.154:0/318669114 shutdown_connections 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 --2- 172.21.15.154:0/318669114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70102de0 0x7f9c70109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 --2- 172.21.15.154:0/318669114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70101240 0x7f9c70101660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 --2- 172.21.15.154:0/318669114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102440 0x7f9c701028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 -- 172.21.15.154:0/318669114 >> 172.21.15.154:0/318669114 conn(0x7f9c700fc7e0 msgr2=0x7f9c700fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 -- 172.21.15.154:0/318669114 shutdown_connections 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.758+0000 7f9c77d04700 1 -- 172.21.15.154:0/318669114 wait complete. 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 Processor -- start 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 -- start start 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70101240 0x7f9c7006ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102de0 0x7f9c7019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:13.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7010ccb0 con 0x7f9c70101240 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7010cb30 con 0x7f9c70102440 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c77d04700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7010ce30 con 0x7f9c70102de0 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50686/0 (socket says 172.21.15.154:50686) 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 -- 172.21.15.154:0/2266931626 learned_addr learned my addr 172.21.15.154:0/2266931626 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c75aa0700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70101240 0x7f9c7006ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c762a1700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102de0 0x7f9c7019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 -- 172.21.15.154:0/2266931626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102de0 msgr2=0x7f9c7019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102de0 0x7f9c7019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 -- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70101240 msgr2=0x7f9c7006ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70101240 0x7f9c7006ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.759+0000 7f9c7529f700 1 -- 172.21.15.154:0/2266931626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c701a00f0 con 0x7f9c70102440 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c7529f700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c920 tx=0x7f9c6c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:22:13.763 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c007b00 con 0x7f9c70102440 2024-03-19T02:22:13.764 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c6c004ce0 con 0x7f9c70102440 2024-03-19T02:22:13.764 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c701a03e0 con 0x7f9c70102440 2024-03-19T02:22:13.764 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c005600 con 0x7f9c70102440 2024-03-19T02:22:13.764 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.760+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c701a0970 con 0x7f9c70102440 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.761+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c70068fd0 con 0x7f9c70102440 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.762+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9c6c007ca0 con 0x7f9c70102440 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.762+0000 7f9c66ffd700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 0x7f9c5c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.763+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f9c6c0cd910 con 0x7f9c70102440 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.763+0000 7f9c75aa0700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 0x7f9c5c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:13.765 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.763+0000 7f9c75aa0700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 0x7f9c5c0a4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c60000c00 tx=0x7f9c60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:22:13.768 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.767+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c6c014030 con 0x7f9c70102440 2024-03-19T02:22:13.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:13.977+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9c70062750 con 0x7f9c5c0a2910 2024-03-19T02:22:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:13 smithi162 ceph-mon[35761]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:13 smithi154 ceph-mon[35693]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:13 smithi116 ceph-mon[32263]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:14 smithi162 ceph-mon[35761]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:15.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:15.244 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:14 smithi154 ceph-mon[35693]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:15.244 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:15.244 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:15.244 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:14 smithi116 ceph-mon[32263]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:15 smithi162 ceph-mon[35761]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:15 smithi154 ceph-mon[35693]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:15 smithi116 ceph-mon[32263]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:17.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:16 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/204617687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:16 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:16 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]': finished 2024-03-19T02:22:17.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:16 smithi162 ceph-mon[35761]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:22:17.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:17.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:16 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/204617687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:16 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:16 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]': finished 2024-03-19T02:22:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:16 smithi154 ceph-mon[35693]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:22:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:16 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/204617687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:16 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]: dispatch 2024-03-19T02:22:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:16 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2ab68c8-d137-4f5c-a8e1-e539b0819897"}]': finished 2024-03-19T02:22:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:16 smithi116 ceph-mon[32263]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:22:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:18.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:17 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/1515295288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:18.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:17 smithi154 ceph-mon[35693]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:17 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/1515295288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:17 smithi162 ceph-mon[35761]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:17 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/1515295288' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:17 smithi116 ceph-mon[32263]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:20.905 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:20 smithi154 ceph-mon[35693]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:20 smithi162 ceph-mon[35761]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:20 smithi116 ceph-mon[32263]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:22.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:22 smithi154 ceph-mon[35693]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:22 smithi162 ceph-mon[35761]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:22 smithi116 ceph-mon[32263]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:24.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:24 smithi154 ceph-mon[35693]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:24.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:24 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:22:24.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:24 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:24 smithi162 ceph-mon[35761]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:24 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:22:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:24 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:24 smithi116 ceph-mon[32263]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:24 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:22:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:24 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:25.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:25 smithi162 ceph-mon[35761]: Deploying daemon osd.7 on smithi154 2024-03-19T02:22:25.957 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:25 smithi154 ceph-mon[35693]: Deploying daemon osd.7 on smithi154 2024-03-19T02:22:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:25 smithi116 ceph-mon[32263]: Deploying daemon osd.7 on smithi154 2024-03-19T02:22:26.904 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:26 smithi154 ceph-mon[35693]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:26 smithi162 ceph-mon[35761]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:26 smithi116 ceph-mon[32263]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:28.805 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:28 smithi154 ceph-mon[35693]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:28.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:28 smithi162 ceph-mon[35761]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:28 smithi116 ceph-mon[32263]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:30.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:30.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:30 smithi162 ceph-mon[35761]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:30.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:30.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:30 smithi154 ceph-mon[35693]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:30 smithi116 ceph-mon[32263]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:32.886 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:32.886 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:32.886 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:32.887 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:32.887 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:32.887 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:32 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:32.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:32.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:32.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:32 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:32 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:33.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.243+0000 7f9c66ffd700 1 -- 172.21.15.154:0/2266931626 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9c70062750 con 0x7f9c5c0a2910 2024-03-19T02:22:33.243 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 7 on host 'smithi154' 2024-03-19T02:22:33.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 msgr2=0x7f9c5c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:33.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 0x7f9c5c0a4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c60000c00 tx=0x7f9c60005fd0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 msgr2=0x7f9c7019d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:33.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c920 tx=0x7f9c6c00cce0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 shutdown_connections 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9c5c0a2910 0x7f9c5c0a4dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c70102de0 0x7f9c7019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c70101240 0x7f9c7006ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 --2- 172.21.15.154:0/2266931626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c70102440 0x7f9c7019d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.245+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 >> 172.21.15.154:0/2266931626 conn(0x7f9c700fc7e0 msgr2=0x7f9c70105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.246+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 shutdown_connections 2024-03-19T02:22:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:22:33.246+0000 7f9c77d04700 1 -- 172.21.15.154:0/2266931626 wait complete. 2024-03-19T02:22:33.968 DEBUG:teuthology.orchestra.run.smithi154:osd.7> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.7.service 2024-03-19T02:22:33.971 INFO:tasks.cephadm:Deploying osd.8 on smithi162 with /dev/vg_nvme/lv_2... 2024-03-19T02:22:33.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:22:34.218 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:34.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:34 smithi162 ceph-mon[35761]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:34 smithi154 ceph-mon[35693]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:34 smithi116 ceph-mon[32263]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:35 smithi154 ceph-mon[35693]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:22:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:35 smithi154 ceph-mon[35693]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:22:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:35 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:35 smithi154 ceph-mon[35693]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:35 smithi154 ceph-mon[35693]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:35 smithi116 ceph-mon[32263]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:22:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:35 smithi116 ceph-mon[32263]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:22:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:35 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:35 smithi116 ceph-mon[32263]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:35 smithi116 ceph-mon[32263]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:35.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:35 smithi162 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:22:35.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:35 smithi162 ceph-mon[35761]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:22:35.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:35 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:35.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:35 smithi162 ceph-mon[35761]: from='osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:35.557 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:35 smithi162 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:36.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-mon[35693]: from='osd.7 ' entity='osd.7' 2024-03-19T02:22:36.396 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:22:36.020+0000 7f6c75869700 -1 osd.7 0 waiting for initial osdmap 2024-03-19T02:22:36.397 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:22:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:22:36.029+0000 7f6c6d654700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:22:36.441 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:22:36.441 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:22:36.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:36.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:36 smithi162 ceph-mon[35761]: from='osd.7 ' entity='osd.7' 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113016 s, 928 MB/s 2024-03-19T02:22:36.481 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:22:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:36 smithi116 ceph-mon[32263]: from='osd.7 ' entity='osd.7' 2024-03-19T02:22:37.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-03-19T02:22:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:22:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] boot 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:37.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.442 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] boot 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:22:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:22:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: osd.7 [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] boot 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:38.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:38 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi154 2024-03-19T02:22:38.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:38 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 5696M 2024-03-19T02:22:38.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:38 smithi162 ceph-mon[35761]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:38.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:38 smithi162 ceph-mon[35761]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:22:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:38 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi154 2024-03-19T02:22:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:38 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 5696M 2024-03-19T02:22:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:38 smithi154 ceph-mon[35693]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:38 smithi154 ceph-mon[35693]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:22:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:38 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi154 2024-03-19T02:22:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:38 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 5696M 2024-03-19T02:22:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:38 smithi116 ceph-mon[32263]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:38 smithi116 ceph-mon[32263]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:22:39.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 -- 172.21.15.162:0/2666949502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 msgr2=0x7fc48c105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:39.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 --2- 172.21.15.162:0/2666949502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c105220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc480009990 tx=0x7fc480009ca0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 -- 172.21.15.162:0/2666949502 shutdown_connections 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 --2- 172.21.15.162:0/2666949502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 --2- 172.21.15.162:0/2666949502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c105220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 --2- 172.21.15.162:0/2666949502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 0x7fc48c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 -- 172.21.15.162:0/2666949502 >> 172.21.15.162:0/2666949502 conn(0x7fc48c0fc7e0 msgr2=0x7fc48c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 -- 172.21.15.162:0/2666949502 shutdown_connections 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.196+0000 7fc492d35700 1 -- 172.21.15.162:0/2666949502 wait complete. 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 Processor -- start 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 -- start start 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 0x7fc48c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:39.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc48c10ddc0 con 0x7fc48c108090 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc48c10dc40 con 0x7fc48c105760 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc492d35700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc48c10df40 con 0x7fc48c102e30 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42802/0 (socket says 172.21.15.162:42802) 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 -- 172.21.15.162:0/1008839389 learned_addr learned my addr 172.21.15.162:0/1008839389 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc4912d2700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc48bfff700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 0x7fc48c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 -- 172.21.15.162:0/1008839389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 msgr2=0x7fc48c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 0x7fc48c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 -- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 msgr2=0x7fc48c19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:39.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 -- 172.21.15.162:0/1008839389 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48c1a0120 con 0x7fc48c102e30 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc4912d2700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.197+0000 7fc490ad1700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4800098e0 tx=0x7fc480009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.198+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480027070 con 0x7fc48c102e30 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.198+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc480004010 con 0x7fc48c102e30 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.198+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48c1a03b0 con 0x7fc48c102e30 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.198+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480022df0 con 0x7fc48c102e30 2024-03-19T02:22:39.201 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.198+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc48c1a0800 con 0x7fc48c102e30 2024-03-19T02:22:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.199+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc48c068fd0 con 0x7fc48c102e30 2024-03-19T02:22:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.201+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc48002a070 con 0x7fc48c102e30 2024-03-19T02:22:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.201+0000 7fc489ffb700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 0x7fc47c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:22:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.201+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5397+0+0 (secure 0 0 0) 0x7fc4800cf240 con 0x7fc48c102e30 2024-03-19T02:22:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.201+0000 7fc48bfff700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 0x7fc47c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:22:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.202+0000 7fc48bfff700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 0x7fc47c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc478005950 tx=0x7fc4780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:22:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.205+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc480031070 con 0x7fc48c102e30 2024-03-19T02:22:39.406 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:39.407+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc48c062750 con 0x7fc47c0a2890 2024-03-19T02:22:39.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:39 smithi162 ceph-mon[35761]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:22:39.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:39.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:39.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:39 smithi154 ceph-mon[35693]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:22:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:39 smithi116 ceph-mon[32263]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:22:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:22:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:22:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:40.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:40 smithi162 ceph-mon[35761]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:40.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:40 smithi162 ceph-mon[35761]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:40.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:40 smithi162 ceph-mon[35761]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:22:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:40 smithi154 ceph-mon[35693]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:40 smithi154 ceph-mon[35693]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:40 smithi154 ceph-mon[35693]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:22:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:40 smithi116 ceph-mon[32263]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:22:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:40 smithi116 ceph-mon[32263]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:40 smithi116 ceph-mon[32263]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:22:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2799054005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]': finished 2024-03-19T02:22:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T02:22:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/2799054005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]': finished 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T02:22:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2799054005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]: dispatch 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba54ef24-b96d-4423-b606-6b742a134ca7"}]': finished 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T02:22:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:22:43.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:43 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2145033532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:44.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:43 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/2145033532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:43 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2145033532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:22:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:44 smithi162 ceph-mon[35761]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:44 smithi154 ceph-mon[35693]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:44 smithi116 ceph-mon[32263]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:46.894 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:46 smithi162 ceph-mon[35761]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:47.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:46 smithi154 ceph-mon[35693]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:46 smithi116 ceph-mon[32263]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:48 smithi162 ceph-mon[35761]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:48 smithi154 ceph-mon[35693]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:48 smithi116 ceph-mon[32263]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:50 smithi162 ceph-mon[35761]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:22:50.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:50.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:50 smithi162 ceph-mon[35761]: Deploying daemon osd.8 on smithi162 2024-03-19T02:22:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:50 smithi154 ceph-mon[35693]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:22:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:50 smithi154 ceph-mon[35693]: Deploying daemon osd.8 on smithi162 2024-03-19T02:22:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:50 smithi116 ceph-mon[32263]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:22:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:50 smithi116 ceph-mon[32263]: Deploying daemon osd.8 on smithi162 2024-03-19T02:22:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:52 smithi162 ceph-mon[35761]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:53.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:52 smithi154 ceph-mon[35693]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:52 smithi116 ceph-mon[32263]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:55.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:54 smithi154 ceph-mon[35693]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:54 smithi116 ceph-mon[32263]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:54 smithi162 ceph-mon[35761]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:56 smithi162 ceph-mon[35761]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:56 smithi154 ceph-mon[35693]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:56 smithi116 ceph-mon[32263]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:22:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:58 smithi162 ceph-mon[35761]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:58 smithi154 ceph-mon[35693]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:58 smithi116 ceph-mon[32263]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:22:59.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.629+0000 7fc489ffb700 1 -- 172.21.15.162:0/1008839389 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc48c062750 con 0x7fc47c0a2890 2024-03-19T02:22:59.630 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 8 on host 'smithi162' 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 msgr2=0x7fc47c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 0x7fc47c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc478005950 tx=0x7fc4780058e0 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 msgr2=0x7fc48c1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4800098e0 tx=0x7fc480009f90 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 shutdown_connections 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c108090 0x7fc48c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc47c0a2890 0x7fc47c0a4d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc48c102e30 0x7fc48c1984c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 --2- 172.21.15.162:0/1008839389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc48c105760 0x7fc48c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 >> 172.21.15.162:0/1008839389 conn(0x7fc48c0fc7e0 msgr2=0x7fc48c103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:22:59.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 shutdown_connections 2024-03-19T02:22:59.633 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:22:59.632+0000 7fc492d35700 1 -- 172.21.15.162:0/1008839389 wait complete. 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:22:59.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:22:59.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:59.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:22:59.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:22:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:22:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:00.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:22:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:00.321 DEBUG:teuthology.orchestra.run.smithi162:osd.8> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.8.service 2024-03-19T02:23:00.324 INFO:tasks.cephadm:Deploying osd.9 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-19T02:23:00.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:23:00.574 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:00.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:00.834 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:23:00.782+0000 7f198c5e5700 -1 osd.8 0 waiting for initial osdmap 2024-03-19T02:23:00.834 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:23:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:23:00.791+0000 7f19883d8700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:00.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:00 smithi116 ceph-mon[32263]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: from='osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:00 smithi154 ceph-mon[35693]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:23:01.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] boot 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T02:23:01.855 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:01 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:23:01.992 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] boot 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T02:23:01.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:01 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: osd.8 [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] boot 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T02:23:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:01 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114065 s, 919 MB/s 2024-03-19T02:23:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T02:23:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:02 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:23:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:02 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:23:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:02 smithi116 ceph-mon[32263]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:23:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:02 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:23:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:02 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:23:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:02 smithi162 ceph-mon[35761]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:23:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:02 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:23:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:02 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:23:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:02 smithi154 ceph-mon[35693]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:23:03.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-19T02:23:03.728 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi162 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:04.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:04 smithi116 ceph-mon[32263]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:23:04.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi162 2024-03-19T02:23:04.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:04.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:04 smithi162 ceph-mon[35761]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:23:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi162 2024-03-19T02:23:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:04.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:04 smithi154 ceph-mon[35693]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 -- 172.21.15.116:0/97475346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 msgr2=0x7f7b701039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 --2- 172.21.15.116:0/97475346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701039a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b600099c0 tx=0x7f7b60009cd0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 -- 172.21.15.116:0/97475346 shutdown_connections 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 --2- 172.21.15.116:0/97475346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70103ee0 0x7f7b7010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 --2- 172.21.15.116:0/97475346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70102340 0x7f7b70102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 --2- 172.21.15.116:0/97475346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701039a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 -- 172.21.15.116:0/97475346 >> 172.21.15.116:0/97475346 conn(0x7f7b700fd920 msgr2=0x7f7b700ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 -- 172.21.15.116:0/97475346 shutdown_connections 2024-03-19T02:23:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.382+0000 7f7b74dee700 1 -- 172.21.15.116:0/97475346 wait complete. 2024-03-19T02:23:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 Processor -- start 2024-03-19T02:23:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 -- start start 2024-03-19T02:23:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 0x7f7b70198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70103ee0 0x7f7b7019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7010dde0 con 0x7f7b70103540 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7010dc60 con 0x7f7b70102340 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b74dee700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7010df60 con 0x7f7b70103ee0 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b6e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 0x7f7b70198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55898/0 (socket says 172.21.15.116:55898) 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b6dd9b700 1 -- 172.21.15.116:0/1474828227 learned_addr learned my addr 172.21.15.116:0/1474828227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.383+0000 7f7b6ed9d700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70103ee0 0x7f7b7019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 -- 172.21.15.116:0/1474828227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70103ee0 msgr2=0x7f7b7019fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70103ee0 0x7f7b7019fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 -- 172.21.15.116:0/1474828227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 msgr2=0x7f7b70198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 0x7f7b70198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 -- 172.21.15.116:0/1474828227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b701a00d0 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6e59c700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 0x7f7b70198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b6dd9b700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b60000c00 tx=0x7f7b600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60027070 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b60004050 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b701a0360 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60022cb0 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.384+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b701a0810 con 0x7f7b70103540 2024-03-19T02:23:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.385+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b70068fd0 con 0x7f7b70103540 2024-03-19T02:23:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.386+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7b6002a070 con 0x7f7b70103540 2024-03-19T02:23:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.387+0000 7f7b5f7fe700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 0x7f7b580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.387+0000 7f7b6e59c700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 0x7f7b580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.387+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f7b600cef50 con 0x7f7b70103540 2024-03-19T02:23:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.387+0000 7f7b6e59c700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 0x7f7b580a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b701033a0 tx=0x7f7b6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:05.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.392+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b6005e740 con 0x7f7b70103540 2024-03-19T02:23:05.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:05.597+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7b70062750 con 0x7f7b580a2910 2024-03-19T02:23:06.475 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:23:06.476 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:06.476 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:06.476 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:06.476 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:06.476 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:06.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:23:06.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:06.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:06.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:06.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:06.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:06.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:07.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:07 smithi116 ceph-mon[32263]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:23:07.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:07 smithi162 ceph-mon[35761]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:23:07.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:07 smithi154 ceph-mon[35693]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:23:08.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:08 smithi162 ceph-mon[35761]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:23:08.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:08 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]: dispatch 2024-03-19T02:23:08.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:08 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]': finished 2024-03-19T02:23:08.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:08 smithi162 ceph-mon[35761]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T02:23:08.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:08.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:08 smithi116 ceph-mon[32263]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:23:08.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:08 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]: dispatch 2024-03-19T02:23:08.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:08 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]': finished 2024-03-19T02:23:08.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:08 smithi116 ceph-mon[32263]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T02:23:08.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:08 smithi154 ceph-mon[35693]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:23:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:08 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]: dispatch 2024-03-19T02:23:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:08 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/608951093' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873"}]': finished 2024-03-19T02:23:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:08 smithi154 ceph-mon[35693]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T02:23:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:09.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:09 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1694447073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:09.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:09 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1694447073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:09.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:09 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1694447073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:10.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:10 smithi162 ceph-mon[35761]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:23:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:10 smithi154 ceph-mon[35693]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:23:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:10 smithi116 ceph-mon[32263]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:23:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:12 smithi162 ceph-mon[35761]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T02:23:12.977 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:12 smithi116 ceph-mon[32263]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T02:23:13.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:12 smithi154 ceph-mon[35693]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T02:23:14.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:14 smithi116 ceph-mon[32263]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:23:14.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:14 smithi162 ceph-mon[35761]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:23:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:14 smithi154 ceph-mon[35693]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:23:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:16 smithi162 ceph-mon[35761]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:23:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:23:16.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:16 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:16.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:16 smithi116 ceph-mon[32263]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:23:16.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:23:16.942 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:16 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:16 smithi154 ceph-mon[35693]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:23:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:23:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:16 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:17.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:17 smithi162 ceph-mon[35761]: Deploying daemon osd.9 on smithi116 2024-03-19T02:23:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:17 smithi154 ceph-mon[35693]: Deploying daemon osd.9 on smithi116 2024-03-19T02:23:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:17 smithi116 ceph-mon[32263]: Deploying daemon osd.9 on smithi116 2024-03-19T02:23:18.907 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:18 smithi154 ceph-mon[35693]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:18.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:18 smithi162 ceph-mon[35761]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:18 smithi116 ceph-mon[32263]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:20.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:20 smithi162 ceph-mon[35761]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:20 smithi154 ceph-mon[35693]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:20 smithi116 ceph-mon[32263]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:22.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:22 smithi116 ceph-mon[32263]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:22.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:22.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:22.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:22 smithi162 ceph-mon[35761]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:22.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:22.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:22 smithi154 ceph-mon[35693]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:24.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:24 smithi116 ceph-mon[32263]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:24 smithi162 ceph-mon[35761]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:24 smithi154 ceph-mon[35693]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:25.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.626+0000 7f7b5f7fe700 1 -- 172.21.15.116:0/1474828227 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7b70062750 con 0x7f7b580a2910 2024-03-19T02:23:25.627 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 9 on host 'smithi116' 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 msgr2=0x7f7b580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 0x7f7b580a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b701033a0 tx=0x7f7b6400b410 comp rx=0 tx=0).stop 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 msgr2=0x7f7b701989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b60000c00 tx=0x7f7b600047b0 comp rx=0 tx=0).stop 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 shutdown_connections 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b580a2910 0x7f7b580a4dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b70103ee0 0x7f7b7019fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b70102340 0x7f7b70198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 --2- 172.21.15.116:0/1474828227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b70103540 0x7f7b701989b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:25.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.628+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 >> 172.21.15.116:0/1474828227 conn(0x7f7b700fd920 msgr2=0x7f7b70106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:23:25.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.629+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 shutdown_connections 2024-03-19T02:23:25.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:23:25.629+0000 7f7b74dee700 1 -- 172.21.15.116:0/1474828227 wait complete. 2024-03-19T02:23:26.365 DEBUG:teuthology.orchestra.run.smithi116:osd.9> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.9.service 2024-03-19T02:23:26.367 INFO:tasks.cephadm:Deploying osd.10 on smithi154 with /dev/vg_nvme/lv_1... 2024-03-19T02:23:26.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:23:26.618 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:23:26.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:26.739 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.739 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.739 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:23:26.739 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.739 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:26 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:23:26.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:26.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:26 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:23:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:27.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:26 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:27.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:27 smithi154 ceph-mon[35693]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:23:27.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:27 smithi154 ceph-mon[35693]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:23:27.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:27 smithi154 ceph-mon[35693]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:23:27.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:27 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:27 smithi116 ceph-mon[32263]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:23:28.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:27 smithi116 ceph-mon[32263]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:23:28.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:27 smithi116 ceph-mon[32263]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:23:28.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:27 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:27 smithi162 ceph-mon[35761]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:23:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:27 smithi162 ceph-mon[35761]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:23:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:27 smithi162 ceph-mon[35761]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T02:23:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:27 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.615 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:23:28.338+0000 7fcb01c4c700 -1 osd.9 0 waiting for initial osdmap 2024-03-19T02:23:28.615 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:23:28.350+0000 7fcaf9236700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:23:28.650 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:23:28.650 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:23:28.651 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-19T02:23:28.651 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-19T02:23:28.651 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113276 s, 926 MB/s 2024-03-19T02:23:28.651 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-19T02:23:28.902 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' 2024-03-19T02:23:28.903 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:28 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.980 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:28.980 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:23:28.980 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:23:28.980 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.981 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:28.981 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' 2024-03-19T02:23:28.981 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:28 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T02:23:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:23:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:29.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: from='osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330]' entity='osd.9' 2024-03-19T02:23:29.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:28 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:29.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-03-19T02:23:29.554 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:23:29.927 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:29 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:23:29.928 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:29 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:23:29.928 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:29 smithi154 ceph-mon[35693]: osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] boot 2024-03-19T02:23:29.928 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:29 smithi154 ceph-mon[35693]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T02:23:29.928 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:29 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:29 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:23:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:29 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:23:30.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:29 smithi116 ceph-mon[32263]: osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] boot 2024-03-19T02:23:30.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:29 smithi116 ceph-mon[32263]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T02:23:30.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:29 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:29 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:23:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:29 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:23:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:29 smithi162 ceph-mon[35761]: osd.9 [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] boot 2024-03-19T02:23:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:29 smithi162 ceph-mon[35761]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T02:23:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:29 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:23:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:23:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:31.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 -- 172.21.15.154:0/2842188267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103ee0 msgr2=0x7f0ef810a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:31.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/2842188267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103ee0 0x7f0ef810a790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000b0d0 tx=0x7f0ef000b3e0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 -- 172.21.15.154:0/2842188267 shutdown_connections 2024-03-19T02:23:31.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/2842188267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103ee0 0x7f0ef810a790 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/2842188267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 0x7f0ef8102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/2842188267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103540 0x7f0ef81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 -- 172.21.15.154:0/2842188267 >> 172.21.15.154:0/2842188267 conn(0x7f0ef80fd920 msgr2=0x7f0ef80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 -- 172.21.15.154:0/2842188267 shutdown_connections 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.214+0000 7f0efd6f1700 1 -- 172.21.15.154:0/2842188267 wait complete. 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 Processor -- start 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 -- start start 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 0x7f0ef80760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103ee0 0x7f0ef8074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810ddc0 con 0x7f0ef8103ee0 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810dc40 con 0x7f0ef8103540 2024-03-19T02:23:31.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0efd6f1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef810df40 con 0x7f0ef8102340 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0ef67fc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0ef67fc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52106/0 (socket says 172.21.15.154:52106) 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0ef67fc700 1 -- 172.21.15.154:0/3585062692 learned_addr learned my addr 172.21.15.154:0/3585062692 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0ef77fe700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103ee0 0x7f0ef8074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.215+0000 7f0ef6ffd700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 0x7f0ef80760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 -- 172.21.15.154:0/3585062692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 msgr2=0x7f0ef80760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 0x7f0ef80760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 -- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103ee0 msgr2=0x7f0ef8074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103ee0 0x7f0ef8074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 -- 172.21.15.154:0/3585062692 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80754f0 con 0x7f0ef8103540 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0ef67fc700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d950 tx=0x7f0eec00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec016070 con 0x7f0ef8103540 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eec00fd40 con 0x7f0ef8103540 2024-03-19T02:23:31.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec00e3b0 con 0x7f0ef8103540 2024-03-19T02:23:31.220 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef81a25a0 con 0x7f0ef8103540 2024-03-19T02:23:31.220 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.216+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef81a2ab0 con 0x7f0ef8103540 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.217+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef8068fd0 con 0x7f0ef8103540 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.219+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0eec019070 con 0x7f0ef8103540 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.220+0000 7f0edffff700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 0x7f0ee00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.220+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f0eec0cdd30 con 0x7f0ef8103540 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.220+0000 7f0ef6ffd700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 0x7f0ee00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:31.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.221+0000 7f0ef6ffd700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 0x7f0ee00a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ef81033a0 tx=0x7f0ee800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:31.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.223+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0eec09b5d0 con 0x7f0ef8103540 2024-03-19T02:23:31.435 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:31.435+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0ef8062750 con 0x7f0ee00a2890 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi116 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:32.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:31 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi116 2024-03-19T02:23:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T02:23:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:23:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:31 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi116 2024-03-19T02:23:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:31 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:32 smithi154 ceph-mon[35693]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:32 smithi154 ceph-mon[35693]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:32 smithi116 ceph-mon[32263]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:32 smithi116 ceph-mon[32263]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:32 smithi162 ceph-mon[35761]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:32 smithi162 ceph-mon[35761]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/1143223832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]': finished 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T02:23:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/1143223832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]': finished 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T02:23:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/1143223832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]: dispatch 2024-03-19T02:23:35.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b644847f-a734-4600-9861-c8574c3f0622"}]': finished 2024-03-19T02:23:35.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T02:23:35.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.154:0/593386700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.154:0/593386700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.154:0/593386700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:23:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:36 smithi154 ceph-mon[35693]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:36 smithi116 ceph-mon[32263]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:36 smithi162 ceph-mon[35761]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:38 smithi154 ceph-mon[35693]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:38 smithi116 ceph-mon[32263]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:38 smithi162 ceph-mon[35761]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:40 smithi154 ceph-mon[35693]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:40 smithi116 ceph-mon[32263]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:40 smithi162 ceph-mon[35761]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:42.844 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:42 smithi154 ceph-mon[35693]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:42.845 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:23:42.845 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:42 smithi116 ceph-mon[32263]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:23:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:42 smithi162 ceph-mon[35761]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:23:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:43.873 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:43 smithi154 ceph-mon[35693]: Deploying daemon osd.10 on smithi154 2024-03-19T02:23:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:43 smithi116 ceph-mon[32263]: Deploying daemon osd.10 on smithi154 2024-03-19T02:23:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:43 smithi162 ceph-mon[35761]: Deploying daemon osd.10 on smithi154 2024-03-19T02:23:44.876 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:44 smithi154 ceph-mon[35693]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:44 smithi116 ceph-mon[32263]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:44 smithi162 ceph-mon[35761]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:46.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:46 smithi154 ceph-mon[35693]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:46 smithi116 ceph-mon[32263]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:46 smithi162 ceph-mon[35761]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:48.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:48.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:48 smithi162 ceph-mon[35761]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:48 smithi154 ceph-mon[35693]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:48 smithi116 ceph-mon[32263]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:50.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:50.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.502 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 10 on host 'smithi154' 2024-03-19T02:23:51.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.502+0000 7f0edffff700 1 -- 172.21.15.154:0/3585062692 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0ef8062750 con 0x7f0ee00a2890 2024-03-19T02:23:51.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 msgr2=0x7f0ee00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 0x7f0ee00a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ef81033a0 tx=0x7f0ee800a380 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 msgr2=0x7f0ef8074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00d950 tx=0x7f0eec00dc60 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 shutdown_connections 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef8103ee0 0x7f0ef8074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0ee00a2890 0x7f0ee00a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ef8102340 0x7f0ef80760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 --2- 172.21.15.154:0/3585062692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef8103540 0x7f0ef8074700 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.504+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 >> 172.21.15.154:0/3585062692 conn(0x7f0ef80fd920 msgr2=0x7f0ef8106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.505+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 shutdown_connections 2024-03-19T02:23:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-19T02:23:51.505+0000 7f0efd6f1700 1 -- 172.21.15.154:0/3585062692 wait complete. 2024-03-19T02:23:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:51 smithi154 ceph-mon[35693]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:51 smithi154 ceph-mon[35693]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:51.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:51 smithi162 ceph-mon[35761]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:51.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:51 smithi162 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:51.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:51.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:51 smithi116 ceph-mon[32263]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:51 smithi116 ceph-mon[32263]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:52.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:23:52.186 DEBUG:teuthology.orchestra.run.smithi154:osd.10> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.10.service 2024-03-19T02:23:52.188 INFO:tasks.cephadm:Deploying osd.11 on smithi162 with /dev/vg_nvme/lv_1... 2024-03-19T02:23:52.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:23:52.435 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:23:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:23:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:52.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:23:52.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:52.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-mon[35693]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:52.757 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:23:52.650+0000 7fb8848dc700 -1 osd.10 0 waiting for initial osdmap 2024-03-19T02:23:52.757 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:23:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:23:52.661+0000 7fb8806cf700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:52.817 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:52 smithi162 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:23:53.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: from='osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:53.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:23:53.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:53.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:52 smithi116 ceph-mon[32263]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-19T02:23:53.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] boot 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T02:23:53.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:53 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:23:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:23:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] boot 2024-03-19T02:23:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T02:23:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:53 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-19T02:23:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:23:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: osd.10 [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] boot 2024-03-19T02:23:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T02:23:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:53 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113176 s, 926 MB/s 2024-03-19T02:23:54.548 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T02:23:54.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:54 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:23:54.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:54 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:23:54.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:54 smithi154 ceph-mon[35693]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:54.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:54 smithi154 ceph-mon[35693]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:23:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:54 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:23:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:54 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:23:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:54 smithi162 ceph-mon[35761]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:54 smithi162 ceph-mon[35761]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:23:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:54 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:23:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:54 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:23:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:54 smithi116 ceph-mon[32263]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:23:55.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:54 smithi116 ceph-mon[32263]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:23:55.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-03-19T02:23:55.575 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:23:56.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi154 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 4272M 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:56.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:23:56.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:56 smithi162 ceph-mon[35761]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi154 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 4272M 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:23:56.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:56 smithi154 ceph-mon[35693]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:23:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi154 2024-03-19T02:23:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 4272M 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:23:56.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:56 smithi116 ceph-mon[32263]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:23:57.413 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 -- 172.21.15.162:0/1152211286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103540 msgr2=0x7f2f841039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 --2- 172.21.15.162:0/1152211286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103540 0x7f2f841039a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f78009990 tx=0x7f2f78009ca0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 -- 172.21.15.162:0/1152211286 shutdown_connections 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 --2- 172.21.15.162:0/1152211286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103ee0 0x7f2f8410a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 --2- 172.21.15.162:0/1152211286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 0x7f2f84102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 --2- 172.21.15.162:0/1152211286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103540 0x7f2f841039a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 -- 172.21.15.162:0/1152211286 >> 172.21.15.162:0/1152211286 conn(0x7f2f840fd920 msgr2=0x7f2f840ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.410+0000 7f2f89405700 1 -- 172.21.15.162:0/1152211286 shutdown_connections 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 -- 172.21.15.162:0/1152211286 wait complete. 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 Processor -- start 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 -- start start 2024-03-19T02:23:57.414 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 0x7f2f84198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103540 0x7f2f84198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8410df20 con 0x7f2f84102340 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8410dda0 con 0x7f2f84103540 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.411+0000 7f2f89405700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8410e0a0 con 0x7f2f84103ee0 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:56412/0 (socket says 172.21.15.162:56412) 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 -- 172.21.15.162:0/2516666996 learned_addr learned my addr 172.21.15.162:0/2516666996 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f827fc700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103540 0x7f2f84198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f82ffd700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 0x7f2f84198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 -- 172.21.15.162:0/2516666996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103540 msgr2=0x7f2f84198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103540 0x7f2f84198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 -- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 msgr2=0x7f2f84198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 0x7f2f84198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 -- 172.21.15.162:0/2516666996 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f841a0290 con 0x7f2f84103ee0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f837fe700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00b9a0 tx=0x7f2f7c00bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c019070 con 0x7f2f84103ee0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f7c00cde0 con 0x7f2f84103ee0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f841a0580 con 0x7f2f84103ee0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c007370 con 0x7f2f84103ee0 2024-03-19T02:23:57.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.412+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f841a0b40 con 0x7f2f84103ee0 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.413+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f84068fd0 con 0x7f2f84103ee0 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.415+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2f7c01c070 con 0x7f2f84103ee0 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.415+0000 7f2f6bfff700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 0x7f2f6c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.416+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f2f7c0ce0e0 con 0x7f2f84103ee0 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.416+0000 7f2f82ffd700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 0x7f2f6c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:23:57.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.416+0000 7f2f82ffd700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 0x7f2f6c0a4dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2f74006fd0 tx=0x7f2f74008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:23:57.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.419+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f7c023070 con 0x7f2f84103ee0 2024-03-19T02:23:57.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:23:57.624+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2f84062750 con 0x7f2f6c0a2910 2024-03-19T02:23:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:57 smithi162 ceph-mon[35761]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:23:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:57 smithi154 ceph-mon[35693]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:23:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:57 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:57 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:57 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:58.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:57 smithi116 ceph-mon[32263]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:23:58.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:23:58.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:23:58.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:23:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:58 smithi162 ceph-mon[35761]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T02:23:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:58 smithi162 ceph-mon[35761]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:23:58 smithi162 ceph-mon[35761]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:23:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:58 smithi154 ceph-mon[35693]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T02:23:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:58 smithi154 ceph-mon[35693]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:59.003 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:23:58 smithi154 ceph-mon[35693]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:23:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:58 smithi116 ceph-mon[32263]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T02:23:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:58 smithi116 ceph-mon[32263]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:23:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:23:58 smithi116 ceph-mon[32263]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:24:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:24:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/671673560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:00.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:00.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]': finished 2024-03-19T02:24:00.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T02:24:00.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:00 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/671673560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]': finished 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T02:24:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:00 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/671673560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]: dispatch 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "883bfd81-b55d-441a-9a0c-2f5faef7f4a0"}]': finished 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T02:24:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:00 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:01 smithi154 ceph-mon[35693]: from='client.? 172.21.15.162:0/2919717006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:24:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:01 smithi116 ceph-mon[32263]: from='client.? 172.21.15.162:0/2919717006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:24:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:01 smithi162 ceph-mon[35761]: from='client.? 172.21.15.162:0/2919717006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:24:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:02 smithi154 ceph-mon[35693]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:24:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:02 smithi116 ceph-mon[32263]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:24:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:02 smithi162 ceph-mon[35761]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T02:24:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:04 smithi162 ceph-mon[35761]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-19T02:24:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:04 smithi154 ceph-mon[35693]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-19T02:24:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:04 smithi116 ceph-mon[32263]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-19T02:24:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:06 smithi154 ceph-mon[35693]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:06 smithi116 ceph-mon[32263]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:07.057 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:06 smithi162 ceph-mon[35761]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:08.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:08 smithi162 ceph-mon[35761]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:08.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:24:08.974 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:08 smithi154 ceph-mon[35693]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:24:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:08 smithi116 ceph-mon[32263]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:24:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:10.003 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:09 smithi162 ceph-mon[35761]: Deploying daemon osd.11 on smithi162 2024-03-19T02:24:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:09 smithi154 ceph-mon[35693]: Deploying daemon osd.11 on smithi162 2024-03-19T02:24:10.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:09 smithi116 ceph-mon[32263]: Deploying daemon osd.11 on smithi162 2024-03-19T02:24:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:10 smithi162 ceph-mon[35761]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:10 smithi154 ceph-mon[35693]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:10 smithi116 ceph-mon[32263]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:13.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:12 smithi154 ceph-mon[35693]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:12 smithi116 ceph-mon[32263]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:13.116 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:12 smithi162 ceph-mon[35761]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:14.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:14 smithi162 ceph-mon[35761]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:14.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:14.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:14.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:14 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:14 smithi154 ceph-mon[35693]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:14 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:14 smithi116 ceph-mon[32263]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:15.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:14 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:16.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:16 smithi162 ceph-mon[35761]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:16 smithi154 ceph-mon[35693]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:16 smithi116 ceph-mon[32263]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:18.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.646+0000 7f2f6bfff700 1 -- 172.21.15.162:0/2516666996 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2f84062750 con 0x7f2f6c0a2910 2024-03-19T02:24:18.647 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 11 on host 'smithi162' 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 msgr2=0x7f2f6c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 0x7f2f6c0a4dd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2f74006fd0 tx=0x7f2f74008040 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 msgr2=0x7f2f8419fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00b9a0 tx=0x7f2f7c00bcb0 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 shutdown_connections 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f6c0a2910 0x7f2f6c0a4dd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f84103ee0 0x7f2f8419fbf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f84102340 0x7f2f84198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 --2- 172.21.15.162:0/2516666996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f84103540 0x7f2f84198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:18.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 >> 172.21.15.162:0/2516666996 conn(0x7f2f840fd920 msgr2=0x7f2f84106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:18.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 shutdown_connections 2024-03-19T02:24:18.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:18.649+0000 7f2f89405700 1 -- 172.21.15.162:0/2516666996 wait complete. 2024-03-19T02:24:18.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:18.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:18.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:18 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:18 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:19.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:19.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:24:19.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:19.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:18 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:19.345 DEBUG:teuthology.orchestra.run.smithi162:osd.11> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.11.service 2024-03-19T02:24:19.348 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-19T02:24:19.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T02:24:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:24:19.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:24:19.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:19.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:24:19.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:19.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:19.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-mon[35761]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:19.921 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:24:19.812+0000 7f40dd4bf700 -1 osd.11 0 waiting for initial osdmap 2024-03-19T02:24:19.922 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:24:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:24:19.826+0000 7f40d92b2700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:19.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:19 smithi116 ceph-mon[32263]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:20.081 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:24:20.081 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: from='osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:20.081 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:24:20.081 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:20.081 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T02:24:20.082 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:19 smithi154 ceph-mon[35693]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:20.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:20 smithi116 ceph-mon[32263]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:24:20.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:20 smithi116 ceph-mon[32263]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:24:20.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:20.995 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:20 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:20 smithi162 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:24:21.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:20 smithi162 ceph-mon[35761]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:24:21.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:20 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:20 smithi154 ceph-mon[35693]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T02:24:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:20 smithi154 ceph-mon[35693]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:24:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:20 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 -- 172.21.15.116:0/2614845581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740105790 msgr2=0x7f3740107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 --2- 172.21.15.116:0/2614845581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740105790 0x7f3740107b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3734009990 tx=0x7f3734009ca0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 -- 172.21.15.116:0/2614845581 shutdown_connections 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 --2- 172.21.15.116:0/2614845581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 0x7f374010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 --2- 172.21.15.116:0/2614845581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740102e60 0x7f3740105250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 --2- 172.21.15.116:0/2614845581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740105790 0x7f3740107b80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 -- 172.21.15.116:0/2614845581 >> 172.21.15.116:0/2614845581 conn(0x7f37400fc830 msgr2=0x7f37400fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 -- 172.21.15.116:0/2614845581 shutdown_connections 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 -- 172.21.15.116:0/2614845581 wait complete. 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.380+0000 7f3744a39700 1 Processor -- start 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 -- start start 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740105790 0x7f3740198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 0x7f374019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f374010de30 con 0x7f3740102e60 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f374010dcb0 con 0x7f3740105790 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f374010dfb0 con 0x7f37401080c0 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51664/0 (socket says 172.21.15.116:51664) 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 -- 172.21.15.116:0/404010230 learned_addr learned my addr 172.21.15.116:0/404010230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373ed9d700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 0x7f374019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373dd9b700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740105790 0x7f3740198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 -- 172.21.15.116:0/404010230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 msgr2=0x7f374019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 0x7f374019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 -- 172.21.15.116:0/404010230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740105790 msgr2=0x7f3740198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740105790 0x7f3740198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 -- 172.21.15.116:0/404010230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37401a00b0 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f373e59c700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f373000d9b0 tx=0x7f373000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730016070 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37401a03a0 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3730004830 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.381+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730005460 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.382+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37401a08e0 con 0x7f3740102e60 2024-03-19T02:24:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.382+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3740068fd0 con 0x7f3740102e60 2024-03-19T02:24:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.384+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f3730005600 con 0x7f3740102e60 2024-03-19T02:24:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.385+0000 7f372f7fe700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 0x7f37280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.385+0000 7f373dd9b700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 0x7f37280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.385+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f37300ce8e0 con 0x7f3740102e60 2024-03-19T02:24:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.385+0000 7f373dd9b700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 0x7f37280a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f373400b5c0 tx=0x7f3734005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:21.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.388+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f373001f020 con 0x7f3740102e60 2024-03-19T02:24:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.585+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3740062f20 con 0x7f3740102e60 2024-03-19T02:24:21.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.586+0000 7f372f7fe700 1 -- 172.21.15.116:0/404010230 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f373009be20 con 0x7f3740102e60 2024-03-19T02:24:21.587 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:21.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 msgr2=0x7f37280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 0x7f37280a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f373400b5c0 tx=0x7f3734005fd0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 msgr2=0x7f3740198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f373000d9b0 tx=0x7f373000dcc0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 shutdown_connections 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37280a2890 0x7f37280a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37401080c0 0x7f374019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740102e60 0x7f3740198450 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 --2- 172.21.15.116:0/404010230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3740105790 0x7f3740198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 >> 172.21.15.116:0/404010230 conn(0x7f37400fc830 msgr2=0x7f3740103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 shutdown_connections 2024-03-19T02:24:21.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:21.588+0000 7f3744a39700 1 -- 172.21.15.116:0/404010230 wait complete. 2024-03-19T02:24:21.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: purged_snaps scrub starts 2024-03-19T02:24:21.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: purged_snaps scrub ok 2024-03-19T02:24:21.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] boot 2024-03-19T02:24:21.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T02:24:21.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:21.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:21.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:21 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/404010230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:24:22.053 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: purged_snaps scrub starts 2024-03-19T02:24:22.053 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: purged_snaps scrub ok 2024-03-19T02:24:22.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] boot 2024-03-19T02:24:22.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T02:24:22.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:22.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:22.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:21 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/404010230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:24:22.194 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710815060,"num_in_osds":12,"osd_in_since":1710815040,"num_remapped_pgs":0} 2024-03-19T02:24:22.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: purged_snaps scrub starts 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: purged_snaps scrub ok 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: osd.11 [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] boot 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:24:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:21 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/404010230' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:24:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:24:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:24:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: Detected new or changed devices on smithi162 2024-03-19T02:24:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:23.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:22 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.233 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:24:23.233 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: Detected new or changed devices on smithi162 2024-03-19T02:24:23.233 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:23.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:22 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: Detected new or changed devices on smithi162 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:23.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:22 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/3987433159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98102360 msgr2=0x7f7d981027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/3987433159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98102360 0x7f7d981027e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c0099c0 tx=0x7f7d8c009cd0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/3987433159 shutdown_connections 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/3987433159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 0x7f7d9810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/3987433159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98108340 0x7f7d98108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/3987433159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98102360 0x7f7d981027e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/3987433159 >> 172.21.15.116:0/3987433159 conn(0x7f7d98079440 msgr2=0x7f7d98079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/3987433159 shutdown_connections 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/3987433159 wait complete. 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 Processor -- start 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.054+0000 7f7d9f1d3700 1 -- start start 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102360 0x7f7d9819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 0x7f7d9819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d98112470 con 0x7f7d98108340 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d981122f0 con 0x7f7d98102d20 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d981125f0 con 0x7f7d98102360 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51678/0 (socket says 172.21.15.116:51678) 2024-03-19T02:24:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 -- 172.21.15.116:0/1605606958 learned_addr learned my addr 172.21.15.116:0/1605606958 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9cf6f700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102360 0x7f7d9819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d97fff700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 0x7f7d9819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 -- 172.21.15.116:0/1605606958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102360 msgr2=0x7f7d9819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102360 0x7f7d9819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 -- 172.21.15.116:0/1605606958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 msgr2=0x7f7d9819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 0x7f7d9819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 -- 172.21.15.116:0/1605606958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d98198f50 con 0x7f7d98108340 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9d770700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d90007d70 tx=0x7f7d9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90016020 con 0x7f7d98108340 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d98199240 con 0x7f7d98108340 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d9000ac90 con 0x7f7d98108340 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.055+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900115e0 con 0x7f7d98108340 2024-03-19T02:24:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.056+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d981a5520 con 0x7f7d98108340 2024-03-19T02:24:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.056+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d981039f0 con 0x7f7d98108340 2024-03-19T02:24:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.057+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7d90019020 con 0x7f7d98108340 2024-03-19T02:24:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.058+0000 7f7d95ffb700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.058+0000 7f7d9cf6f700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.058+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7d900ce530 con 0x7f7d98108340 2024-03-19T02:24:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.058+0000 7f7d9cf6f700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 0x7f7d800a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800ac50 tx=0x7f7d8800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:24.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.063+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d90020020 con 0x7f7d98108340 2024-03-19T02:24:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:23 smithi162 ceph-mon[35761]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:23 smithi154 ceph-mon[35693]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.257+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d98199470 con 0x7f7d98108340 2024-03-19T02:24:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:23 smithi116 ceph-mon[32263]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.258+0000 7f7d95ffb700 1 -- 172.21.15.116:0/1605606958 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18524 (secure 0 0 0) 0x7f7d9009ba70 con 0x7f7d98108340 2024-03-19T02:24:24.259 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:24.259 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:24:21.819283+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 msgr2=0x7f7d800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 0x7f7d800a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800ac50 tx=0x7f7d8800a380 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 msgr2=0x7f7d98198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d90007d70 tx=0x7f7d9000a340 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 shutdown_connections 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d98108340 0x7f7d98198670 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7d800a2890 0x7f7d800a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d98102360 0x7f7d9819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 --2- 172.21.15.116:0/1605606958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d98102d20 0x7f7d9819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 >> 172.21.15.116:0/1605606958 conn(0x7f7d98079440 msgr2=0x7f7d98100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 shutdown_connections 2024-03-19T02:24:24.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:24.261+0000 7f7d9f1d3700 1 -- 172.21.15.116:0/1605606958 wait complete. 2024-03-19T02:24:24.861 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:24:24.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-19T02:24:25.097 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:24:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1605606958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:25.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1605606958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1605606958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:25 smithi162 ceph-mon[35761]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:26.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:25 smithi116 ceph-mon[32263]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:25 smithi154 ceph-mon[35693]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 -- 172.21.15.116:0/3303047581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101e10 msgr2=0x7f04b8105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 --2- 172.21.15.116:0/3303047581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101e10 0x7f04b8105e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009990 tx=0x7f04ac009ca0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 -- 172.21.15.116:0/3303047581 shutdown_connections 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 --2- 172.21.15.116:0/3303047581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 0x7f04b810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 --2- 172.21.15.116:0/3303047581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101460 0x7f04b8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 --2- 172.21.15.116:0/3303047581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101e10 0x7f04b8105e40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 -- 172.21.15.116:0/3303047581 >> 172.21.15.116:0/3303047581 conn(0x7f04b8078180 msgr2=0x7f04b8078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 -- 172.21.15.116:0/3303047581 shutdown_connections 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.859+0000 7f04bc869700 1 -- 172.21.15.116:0/3303047581 wait complete. 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 Processor -- start 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 -- start start 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101e10 0x7f04b8199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:26.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 0x7f04b81a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04b8111390 con 0x7f04b8101460 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04b8111210 con 0x7f04b8101e10 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04bc869700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04b8111510 con 0x7f04b8106520 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04b659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04b659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51694/0 (socket says 172.21.15.116:51694) 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04b659c700 1 -- 172.21.15.116:0/2482103420 learned_addr learned my addr 172.21.15.116:0/2482103420 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04b5d9b700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101e10 0x7f04b8199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.860+0000 7f04b6d9d700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 0x7f04b81a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 -- 172.21.15.116:0/2482103420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 msgr2=0x7f04b81a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:26.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 0x7f04b81a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 -- 172.21.15.116:0/2482103420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101e10 msgr2=0x7f04b8199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101e10 0x7f04b8199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 -- 172.21.15.116:0/2482103420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b81976c0 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04b659c700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04a800d950 tx=0x7f04a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8016070 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b81979b0 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b8197f10 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04a800fd40 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.861+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a800e3b0 con 0x7f04b8101460 2024-03-19T02:24:26.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.862+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0498005330 con 0x7f04b8101460 2024-03-19T02:24:26.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.863+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f04a8019070 con 0x7f04b8101460 2024-03-19T02:24:26.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.864+0000 7f04a77fe700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 0x7f04a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:26.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.864+0000 7f04b5d9b700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 0x7f04a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:26.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.864+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f04a80cdf10 con 0x7f04b8101460 2024-03-19T02:24:26.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.864+0000 7f04b5d9b700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 0x7f04a00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006010 tx=0x7f04ac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:26.868+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04a809b450 con 0x7f04b8101460 2024-03-19T02:24:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.060+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04980058f0 con 0x7f04b8101460 2024-03-19T02:24:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.060+0000 7f04a77fe700 1 -- 172.21.15.116:0/2482103420 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f04a809ad50 con 0x7f04b8101460 2024-03-19T02:24:27.060 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 msgr2=0x7f04a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 0x7f04a00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006010 tx=0x7f04ac005960 comp rx=0 tx=0).stop 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 msgr2=0x7f04b81998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04a800d950 tx=0x7f04a800dc60 comp rx=0 tx=0).stop 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 shutdown_connections 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f04a00a2890 0x7f04a00a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04b8106520 0x7f04b81a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b8101460 0x7f04b81998d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 --2- 172.21.15.116:0/2482103420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04b8101e10 0x7f04b8199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 >> 172.21.15.116:0/2482103420 conn(0x7f04b8078180 msgr2=0x7f04b80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 shutdown_connections 2024-03-19T02:24:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:27.062+0000 7f04bc869700 1 -- 172.21.15.116:0/2482103420 wait complete. 2024-03-19T02:24:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:27 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2482103420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:27.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:27 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2482103420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:27 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2482103420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:27.834 INFO:tasks.cephadm:Adding mds.a on smithi116 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.d on smithi116 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.g on smithi116 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.j on smithi116 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.b on smithi154 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.e on smithi154 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.h on smithi154 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.k on smithi154 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.c on smithi162 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.f on smithi162 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.i on smithi162 2024-03-19T02:24:27.835 INFO:tasks.cephadm:Adding mds.l on smithi162 2024-03-19T02:24:27.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l' 2024-03-19T02:24:28.087 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:24:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:28 smithi162 ceph-mon[35761]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:28 smithi154 ceph-mon[35693]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:28.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:28 smithi116 ceph-mon[32263]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1007474014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103ee0 msgr2=0x7f8a4c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:29.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1007474014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c10a790 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400b0a0 tx=0x7f8a4400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1007474014 shutdown_connections 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1007474014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c10a790 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1007474014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 0x7f8a4c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1007474014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103540 0x7f8a4c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1007474014 >> 172.21.15.162:0/1007474014 conn(0x7f8a4c0fd920 msgr2=0x7f8a4c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1007474014 shutdown_connections 2024-03-19T02:24:29.748 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.747+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1007474014 wait complete. 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 Processor -- start 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 -- start start 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 0x7f8a4c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:29.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10de30 con 0x7f8a4c102340 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10dcb0 con 0x7f8a4c103ee0 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a53ffc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10dfb0 con 0x7f8a4c103540 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a51597700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a51597700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42606/0 (socket says 172.21.15.162:42606) 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a51597700 1 -- 172.21.15.162:0/1440020639 learned_addr learned my addr 172.21.15.162:0/1440020639 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a51d98700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 0x7f8a4c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.748+0000 7f8a52599700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 -- 172.21.15.162:0/1440020639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 msgr2=0x7f8a4c19fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c19fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 -- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 msgr2=0x7f8a4c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 0x7f8a4c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.751 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 -- 172.21.15.162:0/1440020639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c1a00a0 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a52599700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c19fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a51597700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b560 tx=0x7f8a4800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48015020 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a48011ce0 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48010d60 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c1a0390 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.749+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c1a0880 con 0x7f8a4c103540 2024-03-19T02:24:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.750+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c068fd0 con 0x7f8a4c103540 2024-03-19T02:24:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.751+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8a48018020 con 0x7f8a4c103540 2024-03-19T02:24:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.752+0000 7f8a42ffd700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 0x7f8a380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.752+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8a480ce280 con 0x7f8a4c103540 2024-03-19T02:24:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.752+0000 7f8a51d98700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 0x7f8a380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:29.753 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.752+0000 7f8a51d98700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 0x7f8a380a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c1033a0 tx=0x7f8a3c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:29.755 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.755+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a4801f020 con 0x7f8a4c103540 2024-03-19T02:24:29.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.961+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}) -- 0x7f8a4c062950 con 0x7f8a380a2910 2024-03-19T02:24:29.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.968+0000 7f8a42ffd700 1 -- 172.21.15.162:0/1440020639 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f8a4c062950 con 0x7f8a380a2910 2024-03-19T02:24:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mds.all update... 2024-03-19T02:24:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 msgr2=0x7f8a380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:29.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 0x7f8a380a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c1033a0 tx=0x7f8a3c007400 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 msgr2=0x7f8a4c198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b560 tx=0x7f8a4800b870 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 shutdown_connections 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c103ee0 0x7f8a4c19fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a4c102340 0x7f8a4c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8a380a2910 0x7f8a380a4dd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 --2- 172.21.15.162:0/1440020639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a4c103540 0x7f8a4c198980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 >> 172.21.15.162:0/1440020639 conn(0x7f8a4c0fd920 msgr2=0x7f8a4c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 shutdown_connections 2024-03-19T02:24:29.971 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T02:24:29.970+0000 7f8a53ffc700 1 -- 172.21.15.162:0/1440020639 wait complete. 2024-03-19T02:24:30.645 DEBUG:teuthology.orchestra.run.smithi116:mds.a> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.a.service 2024-03-19T02:24:30.647 DEBUG:teuthology.orchestra.run.smithi116:mds.d> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.d.service 2024-03-19T02:24:30.691 DEBUG:teuthology.orchestra.run.smithi116:mds.g> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.g.service 2024-03-19T02:24:30.734 DEBUG:teuthology.orchestra.run.smithi116:mds.j> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.j.service 2024-03-19T02:24:30.777 DEBUG:teuthology.orchestra.run.smithi154:mds.b> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.b.service 2024-03-19T02:24:30.781 DEBUG:teuthology.orchestra.run.smithi154:mds.e> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.e.service 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:30.803 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:30.804 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:30 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:30.804 DEBUG:teuthology.orchestra.run.smithi154:mds.h> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.h.service 2024-03-19T02:24:30.847 DEBUG:teuthology.orchestra.run.smithi154:mds.k> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.k.service 2024-03-19T02:24:30.891 DEBUG:teuthology.orchestra.run.smithi162:mds.c> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.c.service 2024-03-19T02:24:30.894 DEBUG:teuthology.orchestra.run.smithi162:mds.f> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.f.service 2024-03-19T02:24:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:30.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:30.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:30 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:30.919 DEBUG:teuthology.orchestra.run.smithi162:mds.i> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.i.service 2024-03-19T02:24:30.963 DEBUG:teuthology.orchestra.run.smithi162:mds.l> sudo journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.l.service 2024-03-19T02:24:31.006 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-19T02:24:31.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:31.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:30 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 -- 172.21.15.116:0/1084674438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 msgr2=0x7f7404101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 --2- 172.21.15.116:0/1084674438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f7404101830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009900 tx=0x7f73fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 -- 172.21.15.116:0/1084674438 shutdown_connections 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 --2- 172.21.15.116:0/1084674438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f7404101830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 --2- 172.21.15.116:0/1084674438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7404100a10 0x7f7404100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 --2- 172.21.15.116:0/1084674438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f74041000b0 0x7f74041004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 -- 172.21.15.116:0/1084674438 >> 172.21.15.116:0/1084674438 conn(0x7f74040fbcb0 msgr2=0x7f74040fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 -- 172.21.15.116:0/1084674438 shutdown_connections 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 -- 172.21.15.116:0/1084674438 wait complete. 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.120+0000 7f7408fe8640 1 Processor -- start 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 -- start start 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74041000b0 0x7f7404199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7404100a10 0x7f740419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f740410f5b0 con 0x7f74041013b0 2024-03-19T02:24:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f740410f430 con 0x7f74041000b0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f740410f730 con 0x7f7404100a10 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58786/0 (socket says 172.21.15.116:58786) 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 -- 172.21.15.116:0/3534013708 learned_addr learned my addr 172.21.15.116:0/3534013708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f740259c640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74041000b0 0x7f7404199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7401d9b640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7404100a10 0x7f740419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 -- 172.21.15.116:0/3534013708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7404100a10 msgr2=0x7f740419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7404100a10 0x7f740419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 -- 172.21.15.116:0/3534013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74041000b0 msgr2=0x7f7404199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74041000b0 0x7f7404199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 -- 172.21.15.116:0/3534013708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74041a18c0 con 0x7f74041013b0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7402d9d640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009230 tx=0x7f73fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc043020 con 0x7f74041013b0 2024-03-19T02:24:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74041a1b50 con 0x7f74041013b0 2024-03-19T02:24:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73fc0044c0 con 0x7f74041013b0 2024-03-19T02:24:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.121+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc003bc0 con 0x7f74041013b0 2024-03-19T02:24:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.122+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74041a2120 con 0x7f74041013b0 2024-03-19T02:24:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.122+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740410aff0 con 0x7f74041013b0 2024-03-19T02:24:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.123+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f73fc046020 con 0x7f74041013b0 2024-03-19T02:24:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.124+0000 7f73f37fe640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 0x7f73ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.124+0000 7f740259c640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 0x7f73ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.124+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f73fc0f0870 con 0x7f74041013b0 2024-03-19T02:24:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.124+0000 7f740259c640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 0x7f73ec0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f740419ab50 tx=0x7f73f400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.128+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f73fc04d020 con 0x7f74041013b0 2024-03-19T02:24:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.305+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74040624c0 con 0x7f74041013b0 2024-03-19T02:24:31.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.306+0000 7f73f37fe640 1 -- 172.21.15.116:0/3534013708 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18524 (secure 0 0 0) 0x7f73fc0dd030 con 0x7f74041013b0 2024-03-19T02:24:31.306 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:31.306 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:24:21.819283+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:24:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 msgr2=0x7f73ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 0x7f73ec0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f740419ab50 tx=0x7f73f400d040 comp rx=0 tx=0).stop 2024-03-19T02:24:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 msgr2=0x7f74041a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009230 tx=0x7f73fc0040c0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 shutdown_connections 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f73ec0a2800 0x7f73ec0a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74041013b0 0x7f74041a1220 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7404100a10 0x7f740419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 --2- 172.21.15.116:0/3534013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74041000b0 0x7f7404199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 >> 172.21.15.116:0/3534013708 conn(0x7f74040fbcb0 msgr2=0x7f74040fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.308+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 shutdown_connections 2024-03-19T02:24:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.309+0000 7f7408fe8640 1 -- 172.21.15.116:0/3534013708 wait complete. 2024-03-19T02:24:31.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:24:31.319 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 -- 172.21.15.116:0/2308311097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208100730 msgr2=0x7fe208100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 --2- 172.21.15.116:0/2308311097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208100730 0x7fe208100b50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009900 tx=0x7fe1fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 -- 172.21.15.116:0/2308311097 shutdown_connections 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 --2- 172.21.15.116:0/2308311097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 0x7fe208108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 --2- 172.21.15.116:0/2308311097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208101920 0x7fe208101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 --2- 172.21.15.116:0/2308311097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208100730 0x7fe208100b50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 -- 172.21.15.116:0/2308311097 >> 172.21.15.116:0/2308311097 conn(0x7fe2080fbcc0 msgr2=0x7fe2080fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 -- 172.21.15.116:0/2308311097 shutdown_connections 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.424+0000 7fe20db09640 1 -- 172.21.15.116:0/2308311097 wait complete. 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 Processor -- start 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 -- start start 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 0x7fe20806a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 0x7fe20819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe20810c260 con 0x7fe208101920 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe20810c0e0 con 0x7fe208100730 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe20db09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe20810c3e0 con 0x7fe2081022e0 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58798/0 (socket says 172.21.15.116:58798) 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 -- 172.21.15.116:0/4049764052 learned_addr learned my addr 172.21.15.116:0/4049764052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe2077fe640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 0x7fe20806a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe207fff640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 0x7fe20819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 -- 172.21.15.116:0/4049764052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 msgr2=0x7fe20819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 0x7fe20819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 -- 172.21.15.116:0/4049764052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 msgr2=0x7fe20806a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 0x7fe20806a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe206ffd640 1 -- 172.21.15.116:0/4049764052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20819f6d0 con 0x7fe208101920 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.425+0000 7fe2077fe640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 0x7fe20806a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe206ffd640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8017030 con 0x7fe208101920 2024-03-19T02:24:31.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1f8004020 con 0x7fe208101920 2024-03-19T02:24:31.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20819f9c0 con 0x7fe208101920 2024-03-19T02:24:31.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80047a0 con 0x7fe208101920 2024-03-19T02:24:31.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.426+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20819ff10 con 0x7fe208101920 2024-03-19T02:24:31.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.427+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe208107b70 con 0x7fe208101920 2024-03-19T02:24:31.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.428+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe1f80041d0 con 0x7fe208101920 2024-03-19T02:24:31.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.428+0000 7fe204ff9640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.429+0000 7fe2077fe640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.429+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe1f80ce170 con 0x7fe208101920 2024-03-19T02:24:31.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.429+0000 7fe2077fe640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc006010 tx=0x7fe1fc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.433+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1f809b6b0 con 0x7fe208101920 2024-03-19T02:24:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.609+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe20802d410 con 0x7fe208101920 2024-03-19T02:24:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.609+0000 7fe204ff9640 1 -- 172.21.15.116:0/4049764052 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fe1f809afb0 con 0x7fe208101920 2024-03-19T02:24:31.609 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:24:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 msgr2=0x7fe1f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc006010 tx=0x7fe1fc02c7e0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 msgr2=0x7fe20819c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800de10 comp rx=0 tx=0).stop 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 shutdown_connections 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2081022e0 0x7fe20819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe208101920 0x7fe20819c7d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 --2- 172.21.15.116:0/4049764052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe208100730 0x7fe20806a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 >> 172.21.15.116:0/4049764052 conn(0x7fe2080fbcc0 msgr2=0x7fe208104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 shutdown_connections 2024-03-19T02:24:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.612+0000 7fe20db09640 1 -- 172.21.15.116:0/4049764052 wait complete. 2024-03-19T02:24:31.622 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 -- 172.21.15.116:0/2289817978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74701022f0 msgr2=0x7f7470108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 --2- 172.21.15.116:0/2289817978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74701022f0 0x7f7470108b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7468009ca0 tx=0x7f746802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 -- 172.21.15.116:0/2289817978 shutdown_connections 2024-03-19T02:24:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 --2- 172.21.15.116:0/2289817978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74701022f0 0x7f7470108b80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 --2- 172.21.15.116:0/2289817978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 0x7f7470101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 --2- 172.21.15.116:0/2289817978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7470100740 0x7f7470100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 -- 172.21.15.116:0/2289817978 >> 172.21.15.116:0/2289817978 conn(0x7f74700fbcb0 msgr2=0x7f74700fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 -- 172.21.15.116:0/2289817978 shutdown_connections 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.724+0000 7f74765e6640 1 -- 172.21.15.116:0/2289817978 wait complete. 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 Processor -- start 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 -- start start 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 0x7f747019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74701022f0 0x7f74701a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c270 con 0x7f7470100740 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c0f0 con 0x7f74701022f0 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f74765e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f747010c3f0 con 0x7f7470101930 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58814/0 (socket says 172.21.15.116:58814) 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 -- 172.21.15.116:0/3271603004 learned_addr learned my addr 172.21.15.116:0/3271603004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f7474b83640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74701022f0 0x7f74701a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746f7fe640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 0x7f747019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 -- 172.21.15.116:0/3271603004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 msgr2=0x7f747019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 0x7f747019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 -- 172.21.15.116:0/3271603004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74701022f0 msgr2=0x7f74701a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74701022f0 0x7f74701a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.725+0000 7f746ffff640 1 -- 172.21.15.116:0/3271603004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74701a1880 con 0x7f7470100740 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f746ffff640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f746000d770 tx=0x7f746000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460017070 con 0x7f7470100740 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7460009c30 con 0x7f7470100740 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460003d00 con 0x7f7470100740 2024-03-19T02:24:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74701a1b70 con 0x7f7470100740 2024-03-19T02:24:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.726+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74701a20c0 con 0x7f7470100740 2024-03-19T02:24:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.727+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7470107a60 con 0x7f7470100740 2024-03-19T02:24:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.728+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7460003ea0 con 0x7f7470100740 2024-03-19T02:24:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.728+0000 7f746d7fa640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.728+0000 7f746f7fe640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.728+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f74600ce2c0 con 0x7f7470100740 2024-03-19T02:24:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.729+0000 7f746f7fe640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 0x7f74580a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7464005fd0 tx=0x7f7464005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.732+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f746009b800 con 0x7f7470100740 2024-03-19T02:24:31.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:31 smithi162 ceph-mon[35761]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:24:31.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:31 smithi162 ceph-mon[35761]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T02:24:31.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:31 smithi162 ceph-mon[35761]: Deploying daemon mds.l on smithi162 2024-03-19T02:24:31.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:31 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3534013708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:31.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:31 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4049764052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:31.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.944+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f747002d1f0 con 0x7f7470100740 2024-03-19T02:24:31.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.944+0000 7f746d7fa640 1 -- 172.21.15.116:0/3271603004 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7460009de0 con 0x7f7470100740 2024-03-19T02:24:31.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:31.944 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T02:24:31.945 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 1 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 msgr2=0x7f74580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 0x7f74580a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7464005fd0 tx=0x7f7464005850 comp rx=0 tx=0).stop 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 msgr2=0x7f7470069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f746000d770 tx=0x7f746000dc30 comp rx=0 tx=0).stop 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 shutdown_connections 2024-03-19T02:24:31.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f74580a2800 0x7f74580a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74701022f0 0x7f74701a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7470101930 0x7f747019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 --2- 172.21.15.116:0/3271603004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7470100740 0x7f7470069510 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.946+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 >> 172.21.15.116:0/3271603004 conn(0x7f74700fbcb0 msgr2=0x7f7470104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.947+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 shutdown_connections 2024-03-19T02:24:31.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:31.947+0000 7f74765e6640 1 -- 172.21.15.116:0/3271603004 wait complete. 2024-03-19T02:24:31.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:31 smithi116 ceph-mon[32263]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:24:31.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:31 smithi116 ceph-mon[32263]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T02:24:31.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:31 smithi116 ceph-mon[32263]: Deploying daemon mds.l on smithi162 2024-03-19T02:24:31.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:31 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3534013708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:31.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:31 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4049764052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:31 smithi154 ceph-mon[35693]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:24:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:31 smithi154 ceph-mon[35693]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T02:24:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:31 smithi154 ceph-mon[35693]: Deploying daemon mds.l on smithi162 2024-03-19T02:24:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:31 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3534013708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:24:32.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:31 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4049764052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:24:32.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:32 smithi162 ceph-mon[35761]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:32.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:32 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3271603004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:32 smithi154 ceph-mon[35693]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:32 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3271603004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:32 smithi116 ceph-mon[32263]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:32 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3271603004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:33.063 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:24:33 smithi162 ceph-mds[65757]: -- 172.21.15.162:0/1720814847 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638d0fdba40 con 0x5638d0fae800 2024-03-19T02:24:33.063 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:24:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: starting mds.l at 2024-03-19T02:24:33.417 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:24:33 smithi162 bash[65738]: 11fb4d09207d92d965a77a490ae3d8207e0b2154c5f654fa4ee5f11cdd9b8884 2024-03-19T02:24:33.417 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:24:33 smithi162 systemd[1]: Started Ceph mds.l for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:33.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.071+0000 7f98c93b3640 1 -- 172.21.15.116:0/1890039781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c40ffea0 msgr2=0x7f98c41002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.071+0000 7f98c93b3640 1 --2- 172.21.15.116:0/1890039781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c40ffea0 0x7f98c41002c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009ca0 tx=0x7f98b402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.071+0000 7f98c93b3640 1 -- 172.21.15.116:0/1890039781 shutdown_connections 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.071+0000 7f98c93b3640 1 --2- 172.21.15.116:0/1890039781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c4101a40 0x7f98c4108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.071+0000 7f98c93b3640 1 --2- 172.21.15.116:0/1890039781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101080 0x7f98c4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 --2- 172.21.15.116:0/1890039781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c40ffea0 0x7f98c41002c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 -- 172.21.15.116:0/1890039781 >> 172.21.15.116:0/1890039781 conn(0x7f98c40fb460 msgr2=0x7f98c40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 -- 172.21.15.116:0/1890039781 shutdown_connections 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 -- 172.21.15.116:0/1890039781 wait complete. 2024-03-19T02:24:34.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 Processor -- start 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.072+0000 7f98c93b3640 1 -- start start 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 0x7f98c4196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101a40 0x7f98c419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98c410c290 con 0x7f98c4101080 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98c410c110 con 0x7f98c4101a40 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98c410c410 con 0x7f98c40ffea0 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58830/0 (socket says 172.21.15.116:58830) 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 -- 172.21.15.116:0/895756392 learned_addr learned my addr 172.21.15.116:0/895756392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c37fe640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101a40 0x7f98c419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:34.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c2ffd640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 0x7f98c4196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 -- 172.21.15.116:0/895756392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 msgr2=0x7f98c4196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 0x7f98c4196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 -- 172.21.15.116:0/895756392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101a40 msgr2=0x7f98c419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101a40 0x7f98c419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 -- 172.21.15.116:0/895756392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c419e550 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c2ffd640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 0x7f98c4196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c27fc640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e9f0 tx=0x7f98b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b800cd80 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c419e840 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98b80108e0 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8018920 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.073+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c419ed90 con 0x7f98c4101080 2024-03-19T02:24:34.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.074+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c4107260 con 0x7f98c4101080 2024-03-19T02:24:34.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.075+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f98b8010a90 con 0x7f98c4101080 2024-03-19T02:24:34.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.076+0000 7f98abfff640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 0x7f98ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:34.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.076+0000 7f98c2ffd640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 0x7f98ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:34.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.076+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f98b80ce960 con 0x7f98c4101080 2024-03-19T02:24:34.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.076+0000 7f98c2ffd640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 0x7f98ac0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009430 tx=0x7f98b4009a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:34.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.080+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98b809bea0 con 0x7f98c4101080 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6832/750284800,v1:172.21.15.162:6833/750284800] up:boot 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: fsmap 1 up:standby 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:34.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:34.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:34.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: Deploying daemon mds.k on smithi154 2024-03-19T02:24:34.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:34 smithi154 ceph-mon[35693]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6832/750284800,v1:172.21.15.162:6833/750284800] up:boot 2024-03-19T02:24:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: fsmap 1 up:standby 2024-03-19T02:24:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:24:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:34.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:34.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:34.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: Deploying daemon mds.k on smithi154 2024-03-19T02:24:34.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:34 smithi116 ceph-mon[32263]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.296+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c402d1f0 con 0x7f98c4101080 2024-03-19T02:24:34.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.296+0000 7f98abfff640 1 -- 172.21.15.116:0/895756392 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f98b809b7a0 con 0x7f98c4101080 2024-03-19T02:24:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:34.299 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:24:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-19T02:24:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 msgr2=0x7f98ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 0x7f98ac0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009430 tx=0x7f98b4009a20 comp rx=0 tx=0).stop 2024-03-19T02:24:34.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 msgr2=0x7f98c4196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e9f0 tx=0x7f98b800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 shutdown_connections 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f98ac0a2800 0x7f98ac0a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98c4101a40 0x7f98c419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98c4101080 0x7f98c4196e70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 --2- 172.21.15.116:0/895756392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98c40ffea0 0x7f98c4196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 >> 172.21.15.116:0/895756392 conn(0x7f98c40fb460 msgr2=0x7f98c4104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 shutdown_connections 2024-03-19T02:24:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:34.298+0000 7f98c93b3640 1 -- 172.21.15.116:0/895756392 wait complete. 2024-03-19T02:24:34.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6832/750284800,v1:172.21.15.162:6833/750284800] up:boot 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: fsmap 1 up:standby 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: Deploying daemon mds.k on smithi154 2024-03-19T02:24:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:34 smithi162 ceph-mon[35761]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/895756392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:35.506 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 systemd[1]: Starting Ceph mds.k for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/895756392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:35.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/895756392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:35.850 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 podman[65188]: 2024-03-19 02:24:35.596441236 +0000 UTC m=+0.103177371 container create 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:24:35.850 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 podman[65188]: 2024-03-19 02:24:35.511945584 +0000 UTC m=+0.018681733 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:35.850 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 podman[65188]: 2024-03-19 02:24:35.824747321 +0000 UTC m=+0.331483460 container init 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T02:24:35.850 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 podman[65188]: 2024-03-19 02:24:35.827366298 +0000 UTC m=+0.334102442 container start 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-19T02:24:36.245 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 ceph-mds[65207]: -- 172.21.15.154:0/3888104017 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556e2571a40 con 0x5556e2544800 2024-03-19T02:24:36.245 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: starting mds.k at 2024-03-19T02:24:36.245 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 bash[65188]: 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df 2024-03-19T02:24:36.245 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:24:35 smithi154 systemd[1]: Started Ceph mds.k for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:36.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- 172.21.15.116:0/2771096311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48101930 msgr2=0x7fbc48101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- 172.21.15.116:0/2771096311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48101930 0x7fbc48101db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009900 tx=0x7fbc3802bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- 172.21.15.116:0/2771096311 shutdown_connections 2024-03-19T02:24:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- 172.21.15.116:0/2771096311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc481022f0 0x7fbc48108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- 172.21.15.116:0/2771096311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48101930 0x7fbc48101db0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- 172.21.15.116:0/2771096311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc48100740 0x7fbc48100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- 172.21.15.116:0/2771096311 >> 172.21.15.116:0/2771096311 conn(0x7fbc480fbcb0 msgr2=0x7fbc480fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- 172.21.15.116:0/2771096311 shutdown_connections 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- 172.21.15.116:0/2771096311 wait complete. 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 Processor -- start 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- start start 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 0x7fbc481980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc481022f0 0x7fbc4819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4810c2e0 con 0x7fbc48100740 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4810c160 con 0x7fbc48101930 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.414+0000 7fbc4f881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4810c460 con 0x7fbc481022f0 2024-03-19T02:24:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58850/0 (socket says 172.21.15.116:58850) 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 -- 172.21.15.116:0/3473309441 learned_addr learned my addr 172.21.15.116:0/3473309441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4de1e640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc481022f0 0x7fbc4819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4ce1c640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 0x7fbc481980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 -- 172.21.15.116:0/3473309441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc481022f0 msgr2=0x7fbc4819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc481022f0 0x7fbc4819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 -- 172.21.15.116:0/3473309441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 msgr2=0x7fbc481980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 0x7fbc481980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 -- 172.21.15.116:0/3473309441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4819f780 con 0x7fbc48100740 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4ce1c640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 0x7fbc481980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4d61d640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400b450 tx=0x7fbc4400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44015020 con 0x7fbc48100740 2024-03-19T02:24:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc44011d40 con 0x7fbc48100740 2024-03-19T02:24:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4819fa70 con 0x7fbc48100740 2024-03-19T02:24:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4400f340 con 0x7fbc48100740 2024-03-19T02:24:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.415+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4819ffc0 con 0x7fbc48100740 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.416+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc481a0250 con 0x7fbc48100740 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.417+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fbc44018020 con 0x7fbc48100740 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.418+0000 7fbc3e7fc640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 0x7fbc340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.418+0000 7fbc4ce1c640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 0x7fbc340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.418+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbc440ce160 con 0x7fbc48100740 2024-03-19T02:24:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.418+0000 7fbc4ce1c640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 0x7fbc340a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc48199070 tx=0x7fbc3802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:36.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.422+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc4401f020 con 0x7fbc48100740 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:36 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:36 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:36.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:36.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:36 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:36.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.670+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc481a0420 con 0x7fbc48100740 2024-03-19T02:24:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.671+0000 7fbc3e7fc640 1 -- 172.21.15.116:0/3473309441 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fbc4400d040 con 0x7fbc48100740 2024-03-19T02:24:36.671 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:36.671 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:24:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-19T02:24:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 msgr2=0x7fbc340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 0x7fbc340a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbc48199070 tx=0x7fbc3802c7e0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 msgr2=0x7fbc48197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400b450 tx=0x7fbc4400b910 comp rx=0 tx=0).stop 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 shutdown_connections 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbc340a2880 0x7fbc340a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc481022f0 0x7fbc4819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc48101930 0x7fbc481980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 --2- 172.21.15.116:0/3473309441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc48100740 0x7fbc48197b60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 >> 172.21.15.116:0/3473309441 conn(0x7fbc480fbcb0 msgr2=0x7fbc48068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 shutdown_connections 2024-03-19T02:24:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:36.673+0000 7fbc4f881640 1 -- 172.21.15.116:0/3473309441 wait complete. 2024-03-19T02:24:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:37 smithi154 ceph-mon[35693]: Deploying daemon mds.j on smithi116 2024-03-19T02:24:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:37 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6832/1010748955,v1:172.21.15.154:6833/1010748955] up:boot 2024-03-19T02:24:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:37 smithi154 ceph-mon[35693]: fsmap 2 up:standby 2024-03-19T02:24:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:37 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:24:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3473309441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:37 smithi116 ceph-mon[32263]: Deploying daemon mds.j on smithi116 2024-03-19T02:24:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:37 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6832/1010748955,v1:172.21.15.154:6833/1010748955] up:boot 2024-03-19T02:24:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:37 smithi116 ceph-mon[32263]: fsmap 2 up:standby 2024-03-19T02:24:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:37 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:24:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3473309441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:37 smithi162 ceph-mon[35761]: Deploying daemon mds.j on smithi116 2024-03-19T02:24:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:37 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6832/1010748955,v1:172.21.15.154:6833/1010748955] up:boot 2024-03-19T02:24:37.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:37 smithi162 ceph-mon[35761]: fsmap 2 up:standby 2024-03-19T02:24:37.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:37 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:24:37.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3473309441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:38.250 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 systemd[1]: Starting Ceph mds.j for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:38.503 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:38 smithi116 ceph-mon[32263]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:38.504 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 podman[70458]: 2024-03-19 02:24:38.277465499 +0000 UTC m=+0.017457871 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:38.504 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 podman[70458]: 2024-03-19 02:24:38.381085526 +0000 UTC m=+0.121077893 container create 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:24:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:38 smithi154 ceph-mon[35693]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:38.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:38 smithi162 ceph-mon[35761]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:38.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:38.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 -- 172.21.15.116:0/3867346464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44101850 msgr2=0x7f7c44101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/3867346464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44101850 0x7f7c44101c70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009900 tx=0x7f7c3802bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 -- 172.21.15.116:0/3867346464 shutdown_connections 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/3867346464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 0x7f7c44109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/3867346464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44102a40 0x7f7c44102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/3867346464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44101850 0x7f7c44101c70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 -- 172.21.15.116:0/3867346464 >> 172.21.15.116:0/3867346464 conn(0x7f7c440fcdc0 msgr2=0x7f7c440ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 -- 172.21.15.116:0/3867346464 shutdown_connections 2024-03-19T02:24:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.793+0000 7f7c48cdd640 1 -- 172.21.15.116:0/3867346464 wait complete. 2024-03-19T02:24:38.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 Processor -- start 2024-03-19T02:24:38.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 -- start start 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 0x7f7c44197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 0x7f7c4419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4410d380 con 0x7f7c44102a40 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4410d200 con 0x7f7c44101850 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c48cdd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4410d500 con 0x7f7c44103400 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c41d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c41d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55056/0 (socket says 172.21.15.116:55056) 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c41d9b640 1 -- 172.21.15.116:0/1511241159 learned_addr learned my addr 172.21.15.116:0/1511241159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 0x7f7c44197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c42d9d640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 0x7f7c4419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 -- 172.21.15.116:0/1511241159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 msgr2=0x7f7c4419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 0x7f7c4419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 -- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 msgr2=0x7f7c44198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c4259c640 1 -- 172.21.15.116:0/1511241159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4419f6e0 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.794+0000 7f7c41d9b640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c4259c640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 0x7f7c44197ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c38036040 tx=0x7f7c38005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38043020 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4419f970 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c38004350 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4419fee0 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.795+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38003ab0 con 0x7f7c44101850 2024-03-19T02:24:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.796+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c44108c90 con 0x7f7c44101850 2024-03-19T02:24:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.797+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7c38046020 con 0x7f7c44101850 2024-03-19T02:24:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.798+0000 7f7c337fe640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.798+0000 7f7c41d9b640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.798+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7c380f0cd0 con 0x7f7c44101850 2024-03-19T02:24:38.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.798+0000 7f7c41d9b640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c44198fd0 tx=0x7f7c34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:38.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:38.805+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c3804d020 con 0x7f7c44101850 2024-03-19T02:24:39.008 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 podman[70458]: 2024-03-19 02:24:38.642303385 +0000 UTC m=+0.382295753 container init 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T02:24:39.008 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 podman[70458]: 2024-03-19 02:24:38.644843921 +0000 UTC m=+0.384836289 container start 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T02:24:39.008 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 ceph-mds[70477]: -- 172.21.15.116:0/2570732123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b8f2c7a40 con 0x564b8f29a800 2024-03-19T02:24:39.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j[70473]: starting mds.j at 2024-03-19T02:24:39.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 bash[70458]: 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 2024-03-19T02:24:39.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:24:38 smithi116 systemd[1]: Started Ceph mds.j for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.029+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c4402d1f0 con 0x7f7c44101850 2024-03-19T02:24:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.030+0000 7f7c337fe640 1 -- 172.21.15.116:0/1511241159 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f7c380be210 con 0x7f7c44101850 2024-03-19T02:24:39.031 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:39.031 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:24:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 msgr2=0x7f7c2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c44198fd0 tx=0x7f7c34005f90 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 msgr2=0x7f7c44197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 0x7f7c44197ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c38036040 tx=0x7f7c38005dc0 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 shutdown_connections 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7c2c0a2800 0x7f7c2c0a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c44103400 0x7f7c4419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c44102a40 0x7f7c44198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 --2- 172.21.15.116:0/1511241159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c44101850 0x7f7c44197ac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 >> 172.21.15.116:0/1511241159 conn(0x7f7c440fcdc0 msgr2=0x7f7c44105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 shutdown_connections 2024-03-19T02:24:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:39.032+0000 7f7c48cdd640 1 -- 172.21.15.116:0/1511241159 wait complete. 2024-03-19T02:24:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: Deploying daemon mds.d on smithi116 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1511241159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6834/1165879311,v1:172.21.15.116:6835/1165879311] up:boot 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: fsmap 3 up:standby 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:24:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:39 smithi116 ceph-mon[32263]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: Deploying daemon mds.d on smithi116 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1511241159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6834/1165879311,v1:172.21.15.116:6835/1165879311] up:boot 2024-03-19T02:24:40.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: fsmap 3 up:standby 2024-03-19T02:24:40.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:24:40.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:39 smithi162 ceph-mon[35761]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: Deploying daemon mds.d on smithi116 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1511241159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6834/1165879311,v1:172.21.15.116:6835/1165879311] up:boot 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: fsmap 3 up:standby 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:24:40.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:39 smithi154 ceph-mon[35693]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:41.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:41.074 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 systemd[1]: Starting Ceph mds.d for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:41.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- 172.21.15.116:0/3473288267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 msgr2=0x7fa8cc0fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 --2- 172.21.15.116:0/3473288267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc0fff70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009930 tx=0x7fa8c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- 172.21.15.116:0/3473288267 shutdown_connections 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 --2- 172.21.15.116:0/3473288267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8cc103580 0x7fa8cc109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 --2- 172.21.15.116:0/3473288267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8cc102bc0 0x7fa8cc103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 --2- 172.21.15.116:0/3473288267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc0fff70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- 172.21.15.116:0/3473288267 >> 172.21.15.116:0/3473288267 conn(0x7fa8cc0fb460 msgr2=0x7fa8cc0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- 172.21.15.116:0/3473288267 shutdown_connections 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- 172.21.15.116:0/3473288267 wait complete. 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 Processor -- start 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.166+0000 7fa8d4663640 1 -- start start 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:41.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8cc102bc0 0x7fa8cc1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8cc103580 0x7fa8cc19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10d410 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10d290 con 0x7fa8cc103580 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc10d590 con 0x7fa8cc102bc0 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55066/0 (socket says 172.21.15.116:55066) 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 -- 172.21.15.116:0/38485102 learned_addr learned my addr 172.21.15.116:0/38485102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 -- 172.21.15.116:0/38485102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8cc102bc0 msgr2=0x7fa8cc1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8cc102bc0 0x7fa8cc1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 -- 172.21.15.116:0/38485102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8cc103580 msgr2=0x7fa8cc19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8cc103580 0x7fa8cc19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 -- 172.21.15.116:0/38485102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc19f7c0 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d23ff640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009230 tx=0x7fa8c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8043020 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8cc19fa50 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8c8004aa0 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8003db0 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.167+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8cc1a0020 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.168+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8cc108e10 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.169+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa8c8046020 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.170+0000 7fa8c37fe640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 0x7fa8b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:41.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.170+0000 7fa8d1bfe640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 0x7fa8b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:41.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.170+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa8c80f0a70 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.170+0000 7fa8d1bfe640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 0x7fa8b80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc1990b0 tx=0x7fa8bc006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:41.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.174+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8c804d020 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.390+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8cc02d1f0 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.390+0000 7fa8c37fe640 1 -- 172.21.15.116:0/38485102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fa8c80bdfb0 con 0x7fa8cc0ffb50 2024-03-19T02:24:41.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:41.392 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:24:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 4 2024-03-19T02:24:41.393 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 podman[70825]: 2024-03-19 02:24:41.126619035 +0000 UTC m=+0.019800369 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:41.393 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 podman[70825]: 2024-03-19 02:24:41.264901405 +0000 UTC m=+0.158082703 container create f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 msgr2=0x7fa8b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 0x7fa8b80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc1990b0 tx=0x7fa8bc006c90 comp rx=0 tx=0).stop 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 msgr2=0x7fa8cc197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009230 tx=0x7fa8c80042f0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 shutdown_connections 2024-03-19T02:24:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8b80a2800 0x7fa8b80a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8cc103580 0x7fa8cc19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8cc102bc0 0x7fa8cc1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 --2- 172.21.15.116:0/38485102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8cc0ffb50 0x7fa8cc197ba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.392+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 >> 172.21.15.116:0/38485102 conn(0x7fa8cc0fb460 msgr2=0x7fa8cc105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.393+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 shutdown_connections 2024-03-19T02:24:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:41.393+0000 7fa8d4663640 1 -- 172.21.15.116:0/38485102 wait complete. 2024-03-19T02:24:41.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/38485102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:41.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/38485102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:41.758 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 podman[70825]: 2024-03-19 02:24:41.50126895 +0000 UTC m=+0.394450273 container init f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-19T02:24:41.758 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 podman[70825]: 2024-03-19 02:24:41.503950506 +0000 UTC m=+0.397131806 container start f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T02:24:41.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 ceph-mds[70865]: -- 172.21.15.116:0/2972231539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a488fb1a40 con 0x55a488f84c00 2024-03-19T02:24:41.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d[70859]: starting mds.d at 2024-03-19T02:24:41.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 bash[70825]: f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 2024-03-19T02:24:41.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:24:41 smithi116 systemd[1]: Started Ceph mds.d for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:41.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/38485102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: Deploying daemon mds.e on smithi154 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6836/261800782,v1:172.21.15.116:6837/261800782] up:boot 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: fsmap 4 up:standby 2024-03-19T02:24:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:42 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: Deploying daemon mds.e on smithi154 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6836/261800782,v1:172.21.15.116:6837/261800782] up:boot 2024-03-19T02:24:42.965 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: fsmap 4 up:standby 2024-03-19T02:24:42.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:42 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: Deploying daemon mds.e on smithi154 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6836/261800782,v1:172.21.15.116:6837/261800782] up:boot 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: fsmap 4 up:standby 2024-03-19T02:24:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:42 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:24:43.405 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:43.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.519+0000 7f844811b640 1 -- 172.21.15.116:0/1403730895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440100740 msgr2=0x7f8440100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:43.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.519+0000 7f844811b640 1 --2- 172.21.15.116:0/1403730895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440100740 0x7f8440100b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f843c009900 tx=0x7f843c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:43.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- 172.21.15.116:0/1403730895 shutdown_connections 2024-03-19T02:24:43.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- 172.21.15.116:0/1403730895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401022f0 0x7f8440108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- 172.21.15.116:0/1403730895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8440101930 0x7f8440101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- 172.21.15.116:0/1403730895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440100740 0x7f8440100b60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- 172.21.15.116:0/1403730895 >> 172.21.15.116:0/1403730895 conn(0x7f84400fbcb0 msgr2=0x7f84400fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- 172.21.15.116:0/1403730895 shutdown_connections 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- 172.21.15.116:0/1403730895 wait complete. 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 Processor -- start 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- start start 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440100740 0x7f8440069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 0x7f844019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f844010c240 con 0x7f8440101930 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f844010c0c0 con 0x7f8440100740 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f844811b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f844010c3c0 con 0x7f84401022f0 2024-03-19T02:24:43.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.520+0000 7f84456b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55096/0 (socket says 172.21.15.116:55096) 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 -- 172.21.15.116:0/3988988513 learned_addr learned my addr 172.21.15.116:0/3988988513 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f8445eb7640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440100740 0x7f8440069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84466b8640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 0x7f844019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 -- 172.21.15.116:0/3988988513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 msgr2=0x7f844019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 0x7f844019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 -- 172.21.15.116:0/3988988513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440100740 msgr2=0x7f8440069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440100740 0x7f8440069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 -- 172.21.15.116:0/3988988513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844019f640 con 0x7f8440101930 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84466b8640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 0x7f844019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f84456b6640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f843000c920 tx=0x7f843000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430007c70 con 0x7f8440101930 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8430007e10 con 0x7f8440101930 2024-03-19T02:24:43.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844019f930 con 0x7f8440101930 2024-03-19T02:24:43.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430018820 con 0x7f8440101930 2024-03-19T02:24:43.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.521+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844019fe80 con 0x7f8440101930 2024-03-19T02:24:43.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.522+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8440107ab0 con 0x7f8440101930 2024-03-19T02:24:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.523+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f84300189c0 con 0x7f8440101930 2024-03-19T02:24:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.524+0000 7f8436ffd640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.524+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f84300ce710 con 0x7f8440101930 2024-03-19T02:24:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.524+0000 7f8445eb7640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:43.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.524+0000 7f8445eb7640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 0x7f842c0a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f843c005ab0 tx=0x7f843c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:43.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.528+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8430014030 con 0x7f8440101930 2024-03-19T02:24:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.745+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f844002d3a0 con 0x7f8440101930 2024-03-19T02:24:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.745+0000 7f8436ffd640 1 -- 172.21.15.116:0/3988988513 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f843009bc50 con 0x7f8440101930 2024-03-19T02:24:43.746 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:43.746 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:24:43.746 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 5 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 msgr2=0x7f842c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 0x7f842c0a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f843c005ab0 tx=0x7f843c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 msgr2=0x7f844019c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f843000c920 tx=0x7f843000cde0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 shutdown_connections 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84401022f0 0x7f844019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8440101930 0x7f844019c740 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 --2- 172.21.15.116:0/3988988513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8440100740 0x7f8440069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 >> 172.21.15.116:0/3988988513 conn(0x7f84400fbcb0 msgr2=0x7f8440104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:43.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 shutdown_connections 2024-03-19T02:24:43.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:43.747+0000 7f844811b640 1 -- 172.21.15.116:0/3988988513 wait complete. 2024-03-19T02:24:44.006 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:43 smithi154 systemd[1]: Starting Ceph mds.e for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:44.305 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 podman[65505]: 2024-03-19 02:24:43.945263056 +0000 UTC m=+0.017179333 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:44.306 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 podman[65505]: 2024-03-19 02:24:44.061291376 +0000 UTC m=+0.133207640 container create 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-19T02:24:44.306 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 podman[65505]: 2024-03-19 02:24:44.306176421 +0000 UTC m=+0.378092698 container init 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:24:44.648 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 podman[65505]: 2024-03-19 02:24:44.309144979 +0000 UTC m=+0.381061251 container start 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:24:44.648 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mds[65524]: -- 172.21.15.154:0/1585106788 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c81b061a40 con 0x55c81b034c00 2024-03-19T02:24:44.648 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: starting mds.e at 2024-03-19T02:24:44.648 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 bash[65505]: 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 2024-03-19T02:24:44.648 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:24:44 smithi154 systemd[1]: Started Ceph mds.e for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3988988513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:44.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:44.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:44 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3988988513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:44 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3988988513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:44 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:45.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:45.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- 172.21.15.116:0/3464402721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44104c50 msgr2=0x7f0f44107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 --2- 172.21.15.116:0/3464402721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44104c50 0x7f0f44107040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009900 tx=0x7f0f2c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- 172.21.15.116:0/3464402721 shutdown_connections 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 --2- 172.21.15.116:0/3464402721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44107580 0x7f0f44109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 --2- 172.21.15.116:0/3464402721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44104c50 0x7f0f44107040 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 --2- 172.21.15.116:0/3464402721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44102320 0x7f0f44104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- 172.21.15.116:0/3464402721 >> 172.21.15.116:0/3464402721 conn(0x7f0f440fbcb0 msgr2=0x7f0f440fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- 172.21.15.116:0/3464402721 shutdown_connections 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- 172.21.15.116:0/3464402721 wait complete. 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 Processor -- start 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.874+0000 7f0f49c61640 1 -- start start 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44102320 0x7f0f44197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:45.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44104c50 0x7f0f44198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410d370 con 0x7f0f44107580 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410d1f0 con 0x7f0f44104c50 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4410d4f0 con 0x7f0f44102320 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55126/0 (socket says 172.21.15.116:55126) 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f437fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44102320 0x7f0f44197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 -- 172.21.15.116:0/4086438617 learned_addr learned my addr 172.21.15.116:0/4086438617 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f3affd640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44104c50 0x7f0f44198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 -- 172.21.15.116:0/4086438617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44102320 msgr2=0x7f0f44197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44102320 0x7f0f44197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 -- 172.21.15.116:0/4086438617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44104c50 msgr2=0x7f0f44198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44104c50 0x7f0f44198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:45.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 -- 172.21.15.116:0/4086438617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4419f6e0 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f43fff640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eb10 tx=0x7f0f3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c004b20 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f3c004cc0 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4419f9d0 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c010690 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.875+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4419fef0 con 0x7f0f44107580 2024-03-19T02:24:45.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.876+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f440fd7a0 con 0x7f0f44107580 2024-03-19T02:24:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.878+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0f3c005000 con 0x7f0f44107580 2024-03-19T02:24:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.879+0000 7f0f417fa640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 0x7f0f300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.879+0000 7f0f437fe640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 0x7f0f300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.879+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0f3c0cf000 con 0x7f0f44107580 2024-03-19T02:24:45.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.879+0000 7f0f437fe640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 0x7f0f300a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400bbe0 tx=0x7f0f3400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:45.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:45.882+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f3c09c540 con 0x7f0f44107580 2024-03-19T02:24:45.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:45 smithi162 ceph-mon[35761]: Deploying daemon mds.f on smithi162 2024-03-19T02:24:45.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:45 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6834/1464153745,v1:172.21.15.154:6835/1464153745] up:boot 2024-03-19T02:24:45.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:45 smithi162 ceph-mon[35761]: fsmap 5 up:standby 2024-03-19T02:24:45.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:45 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:24:46.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:45 smithi154 ceph-mon[35693]: Deploying daemon mds.f on smithi162 2024-03-19T02:24:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:45 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6834/1464153745,v1:172.21.15.154:6835/1464153745] up:boot 2024-03-19T02:24:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:45 smithi154 ceph-mon[35693]: fsmap 5 up:standby 2024-03-19T02:24:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:45 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:24:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:45 smithi116 ceph-mon[32263]: Deploying daemon mds.f on smithi162 2024-03-19T02:24:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:45 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6834/1464153745,v1:172.21.15.154:6835/1464153745] up:boot 2024-03-19T02:24:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:45 smithi116 ceph-mon[32263]: fsmap 5 up:standby 2024-03-19T02:24:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:45 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:24:46.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.097+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f4402d1f0 con 0x7f0f44107580 2024-03-19T02:24:46.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.098+0000 7f0f417fa640 1 -- 172.21.15.116:0/4086438617 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f0f3c09be40 con 0x7f0f44107580 2024-03-19T02:24:46.098 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:46.099 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T02:24:46.099 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-19T02:24:46.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 msgr2=0x7f0f300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:46.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 0x7f0f300a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400bbe0 tx=0x7f0f3400b390 comp rx=0 tx=0).stop 2024-03-19T02:24:46.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 msgr2=0x7f0f4419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00eb10 tx=0x7f0f3c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 shutdown_connections 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f300a2800 0x7f0f300a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f44107580 0x7f0f4419f070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f44104c50 0x7f0f44198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 --2- 172.21.15.116:0/4086438617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f44102320 0x7f0f44197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 >> 172.21.15.116:0/4086438617 conn(0x7f0f440fbcb0 msgr2=0x7f0f44102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 shutdown_connections 2024-03-19T02:24:46.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:46.100+0000 7f0f49c61640 1 -- 172.21.15.116:0/4086438617 wait complete. 2024-03-19T02:24:46.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:46 smithi162 ceph-mon[35761]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:46.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:46 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4086438617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:47.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:46 smithi154 ceph-mon[35693]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:47.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:46 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4086438617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:46 smithi116 ceph-mon[32263]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:46 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4086438617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:47.113 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:24:47 smithi162 podman[66049]: 2024-03-19 02:24:47.091112098 +0000 UTC m=+0.362311175 container start 3c3ca9b06120bddef210931bc6539ed9673c40bccb5be14ed803d768cae8a851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-19T02:24:47.417 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:24:47 smithi162 ceph-mds[66068]: -- 172.21.15.162:0/1914464286 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556831d45a40 con 0x556831d18c00 2024-03-19T02:24:47.417 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:24:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: starting mds.f at 2024-03-19T02:24:47.417 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:24:47 smithi162 bash[66049]: 3c3ca9b06120bddef210931bc6539ed9673c40bccb5be14ed803d768cae8a851 2024-03-19T02:24:47.417 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:24:47 smithi162 systemd[1]: Started Ceph mds.f for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:48.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 -- 172.21.15.116:0/3417486858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e8102a20 msgr2=0x7fa5e8102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/3417486858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e8102a20 0x7fa5e8102ea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009900 tx=0x7fa5d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 -- 172.21.15.116:0/3417486858 shutdown_connections 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/3417486858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e81033e0 0x7fa5e8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/3417486858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e8102a20 0x7fa5e8102ea0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/3417486858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 0x7fa5e8101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 -- 172.21.15.116:0/3417486858 >> 172.21.15.116:0/3417486858 conn(0x7fa5e80fce00 msgr2=0x7fa5e80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 -- 172.21.15.116:0/3417486858 shutdown_connections 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 -- 172.21.15.116:0/3417486858 wait complete. 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.224+0000 7fa5efbe6640 1 Processor -- start 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 -- start start 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 0x7fa5e81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e8102a20 0x7fa5e8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e810d340 con 0x7fa5e81033e0 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e810d1c0 con 0x7fa5e8101830 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e810d4c0 con 0x7fa5e8102a20 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ed181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e8102a20 0x7fa5e8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:48.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51752/0 (socket says 172.21.15.116:51752) 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 -- 172.21.15.116:0/4037974808 learned_addr learned my addr 172.21.15.116:0/4037974808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ed982640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 0x7fa5e81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 -- 172.21.15.116:0/4037974808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e8102a20 msgr2=0x7fa5e8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e8102a20 0x7fa5e8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 -- 172.21.15.116:0/4037974808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 msgr2=0x7fa5e81979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 0x7fa5e81979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 -- 172.21.15.116:0/4037974808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e819f5d0 con 0x7fa5e81033e0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5ee183640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000eaa0 tx=0x7fa5e000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e000cdb0 con 0x7fa5e81033e0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5e0005020 con 0x7fa5e81033e0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e819f8c0 con 0x7fa5e81033e0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.225+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00116d0 con 0x7fa5e81033e0 2024-03-19T02:24:48.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.226+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e819fde0 con 0x7fa5e81033e0 2024-03-19T02:24:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.227+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e8108b50 con 0x7fa5e81033e0 2024-03-19T02:24:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.228+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa5e00051d0 con 0x7fa5e81033e0 2024-03-19T02:24:48.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.229+0000 7fa5deffd640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 0x7fa5d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:48.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.229+0000 7fa5ed982640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 0x7fa5d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:48.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.229+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa5e00ceb10 con 0x7fa5e81033e0 2024-03-19T02:24:48.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.229+0000 7fa5ed982640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 0x7fa5d40a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40058c0 tx=0x7fa5e4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:48.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.233+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5e0016030 con 0x7fa5e81033e0 2024-03-19T02:24:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.448+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5e802d1f0 con 0x7fa5e81033e0 2024-03-19T02:24:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.448+0000 7fa5deffd640 1 -- 172.21.15.116:0/4037974808 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7fa5e009c0d0 con 0x7fa5e81033e0 2024-03-19T02:24:48.449 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:48.449 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-19T02:24:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 msgr2=0x7fa5d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 0x7fa5d40a4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40058c0 tx=0x7fa5e4005850 comp rx=0 tx=0).stop 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 msgr2=0x7fa5e819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000eaa0 tx=0x7fa5e000ef60 comp rx=0 tx=0).stop 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 shutdown_connections 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa5d40a2880 0x7fa5d40a4d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5e81033e0 0x7fa5e819ef60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e8102a20 0x7fa5e8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 --2- 172.21.15.116:0/4037974808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e8101830 0x7fa5e81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.450+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 >> 172.21.15.116:0/4037974808 conn(0x7fa5e80fce00 msgr2=0x7fa5e8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.451+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 shutdown_connections 2024-03-19T02:24:48.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:48.451+0000 7fa5efbe6640 1 -- 172.21.15.116:0/4037974808 wait complete. 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: Deploying daemon mds.c on smithi162 2024-03-19T02:24:48.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:48.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:boot 2024-03-19T02:24:48.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: fsmap 6 up:standby 2024-03-19T02:24:48.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:48 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: Deploying daemon mds.c on smithi162 2024-03-19T02:24:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:boot 2024-03-19T02:24:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: fsmap 6 up:standby 2024-03-19T02:24:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:48 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:24:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: Deploying daemon mds.c on smithi162 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:boot 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: fsmap 6 up:standby 2024-03-19T02:24:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:48 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:24:49.661 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:49 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4037974808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:49.662 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:49 smithi162 systemd[1]: Starting Ceph mds.c for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:49.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:49 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4037974808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:49.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:49 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4037974808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:49.917 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:49 smithi162 podman[66357]: 2024-03-19 02:24:49.564201606 +0000 UTC m=+0.017762685 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:49.917 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:49 smithi162 podman[66357]: 2024-03-19 02:24:49.700317564 +0000 UTC m=+0.153878684 container create 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:24:50.290 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 podman[66357]: 2024-03-19 02:24:50.010493387 +0000 UTC m=+0.464054477 container init 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-19T02:24:50.290 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 podman[66357]: 2024-03-19 02:24:50.013210228 +0000 UTC m=+0.466771309 container start 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T02:24:50.290 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mds[66376]: -- 172.21.15.162:0/2790680327 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb530cda40 con 0x55bb530a0800 2024-03-19T02:24:50.290 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: starting mds.c at 2024-03-19T02:24:50.290 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 bash[66357]: 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 2024-03-19T02:24:50.291 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:24:50 smithi162 systemd[1]: Started Ceph mds.c for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:50.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:50.541 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:50.541 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.541 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.541 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.541 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:50.542 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:50.542 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:50 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:50.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 -- 172.21.15.116:0/1798000673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 msgr2=0x7f8dcc101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:50.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 --2- 172.21.15.116:0/1798000673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc101c60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009900 tx=0x7f8dbc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 -- 172.21.15.116:0/1798000673 shutdown_connections 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 --2- 172.21.15.116:0/1798000673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 0x7f8dcc109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 --2- 172.21.15.116:0/1798000673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 0x7f8dcc102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 --2- 172.21.15.116:0/1798000673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc101c60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 -- 172.21.15.116:0/1798000673 >> 172.21.15.116:0/1798000673 conn(0x7f8dcc0fcdd0 msgr2=0x7f8dcc0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 -- 172.21.15.116:0/1798000673 shutdown_connections 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.577+0000 7f8dd473b640 1 -- 172.21.15.116:0/1798000673 wait complete. 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 Processor -- start 2024-03-19T02:24:50.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 -- start start 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 0x7f8dcc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 0x7f8dcc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc10d370 con 0x7f8dcc101840 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc10d1f0 con 0x7f8dcc102a30 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd473b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc10d4f0 con 0x7f8dcc1033f0 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51774/0 (socket says 172.21.15.116:51774) 2024-03-19T02:24:50.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 -- 172.21.15.116:0/2830041869 learned_addr learned my addr 172.21.15.116:0/2830041869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd1cd6640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 0x7f8dcc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd2cd8640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 0x7f8dcc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 -- 172.21.15.116:0/2830041869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 msgr2=0x7f8dcc19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 0x7f8dcc19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 -- 172.21.15.116:0/2830041869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 msgr2=0x7f8dcc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 0x7f8dcc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 -- 172.21.15.116:0/2830041869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc19f690 con 0x7f8dcc101840 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.578+0000 7f8dd24d7640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000c00 tx=0x7f8dbc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.579+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc043020 con 0x7f8dcc101840 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.579+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dbc004440 con 0x7f8dcc101840 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.579+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc19f920 con 0x7f8dcc101840 2024-03-19T02:24:50.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.579+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc003c90 con 0x7f8dcc101840 2024-03-19T02:24:50.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.579+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc19fe70 con 0x7f8dcc101840 2024-03-19T02:24:50.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.580+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc108bb0 con 0x7f8dcc101840 2024-03-19T02:24:50.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.581+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8dbc046020 con 0x7f8dcc101840 2024-03-19T02:24:50.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.581+0000 7f8dc37fe640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 0x7f8db80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:50.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.581+0000 7f8dd1cd6640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 0x7f8db80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:50.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.581+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8dbc0f07d0 con 0x7f8dcc101840 2024-03-19T02:24:50.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.582+0000 7f8dd1cd6640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 0x7f8db80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc198f80 tx=0x7f8dc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:50.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.586+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8dbc04d020 con 0x7f8dcc101840 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:50 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:50 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:50.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.799+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dcc062420 con 0x7f8dcc101840 2024-03-19T02:24:50.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.799+0000 7f8dc37fe640 1 -- 172.21.15.116:0/2830041869 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f8dbc0bdd10 con 0x7f8dcc101840 2024-03-19T02:24:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:50.800 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-19T02:24:50.801 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2024-03-19T02:24:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 msgr2=0x7f8db80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 0x7f8db80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc198f80 tx=0x7f8dc800b410 comp rx=0 tx=0).stop 2024-03-19T02:24:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 msgr2=0x7f8dcc197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc000c00 tx=0x7f8dbc004040 comp rx=0 tx=0).stop 2024-03-19T02:24:50.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 shutdown_connections 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8db80a2800 0x7f8db80a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dcc1033f0 0x7f8dcc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc102a30 0x7f8dcc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 --2- 172.21.15.116:0/2830041869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dcc101840 0x7f8dcc197a70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.801+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 >> 172.21.15.116:0/2830041869 conn(0x7f8dcc0fcdd0 msgr2=0x7f8dcc105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.802+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 shutdown_connections 2024-03-19T02:24:50.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:50.802+0000 7f8dd473b640 1 -- 172.21.15.116:0/2830041869 wait complete. 2024-03-19T02:24:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:51 smithi154 ceph-mon[35693]: Deploying daemon mds.h on smithi154 2024-03-19T02:24:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:51 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:boot 2024-03-19T02:24:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:51 smithi154 ceph-mon[35693]: fsmap 7 up:standby 2024-03-19T02:24:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:51 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:24:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:51 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2830041869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:51 smithi162 ceph-mon[35761]: Deploying daemon mds.h on smithi154 2024-03-19T02:24:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:51 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:boot 2024-03-19T02:24:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:51 smithi162 ceph-mon[35761]: fsmap 7 up:standby 2024-03-19T02:24:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:51 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:24:51.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:51 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2830041869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:51.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:51 smithi116 ceph-mon[32263]: Deploying daemon mds.h on smithi154 2024-03-19T02:24:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:51 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:boot 2024-03-19T02:24:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:51 smithi116 ceph-mon[32263]: fsmap 7 up:standby 2024-03-19T02:24:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:51 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:24:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:51 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2830041869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:52.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:52 smithi162 ceph-mon[35761]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:52.693 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:52 smithi154 ceph-mon[35693]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:52.694 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:52 smithi154 systemd[1]: Starting Ceph mds.h for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:52.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:52 smithi116 ceph-mon[32263]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:52.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:52.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 -- 172.21.15.116:0/3754514878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8100740 msgr2=0x7f2da8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:52.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/3754514878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8100740 0x7f2da8100b60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009900 tx=0x7f2d9802bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:52.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 -- 172.21.15.116:0/3754514878 shutdown_connections 2024-03-19T02:24:52.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/3754514878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 0x7f2da8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/3754514878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8101930 0x7f2da8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/3754514878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8100740 0x7f2da8100b60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 -- 172.21.15.116:0/3754514878 >> 172.21.15.116:0/3754514878 conn(0x7f2da80fbcb0 msgr2=0x7f2da80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 -- 172.21.15.116:0/3754514878 shutdown_connections 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.927+0000 7f2dae0ff640 1 -- 172.21.15.116:0/3754514878 wait complete. 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 Processor -- start 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 -- start start 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8100740 0x7f2da8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 0x7f2da819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810d350 con 0x7f2da8101930 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810d1d0 con 0x7f2da81022f0 2024-03-19T02:24:52.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2dae0ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2da810d4d0 con 0x7f2da8100740 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51788/0 (socket says 172.21.15.116:51788) 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 -- 172.21.15.116:0/21414714 learned_addr learned my addr 172.21.15.116:0/21414714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da7fff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 0x7f2da819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da77fe640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8100740 0x7f2da8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 -- 172.21.15.116:0/21414714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8100740 msgr2=0x7f2da8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8100740 0x7f2da8197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:52.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 -- 172.21.15.116:0/21414714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 msgr2=0x7f2da819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 0x7f2da819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.928+0000 7f2da6ffd640 1 -- 172.21.15.116:0/21414714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da819f6a0 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2da6ffd640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00c890 tx=0x7f2d9c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c004220 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d9c0043c0 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c004a00 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da819f990 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.929+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da819ff30 con 0x7f2da8101930 2024-03-19T02:24:52.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.930+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da8107ad0 con 0x7f2da8101930 2024-03-19T02:24:52.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.931+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2d9c004ba0 con 0x7f2da8101930 2024-03-19T02:24:52.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.932+0000 7f2da4ff9640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 0x7f2d900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:52.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.932+0000 7f2da77fe640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 0x7f2d900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:52.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.932+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2d9c0ce210 con 0x7f2da8101930 2024-03-19T02:24:52.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.932+0000 7f2da77fe640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 0x7f2d900a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006010 tx=0x7f2d98005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:52.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:52.936+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d9c014030 con 0x7f2da8101930 2024-03-19T02:24:52.995 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:52 smithi154 podman[65813]: 2024-03-19 02:24:52.596887104 +0000 UTC m=+0.017993403 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:52.995 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:52 smithi154 podman[65813]: 2024-03-19 02:24:52.70974026 +0000 UTC m=+0.130846547 container create fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T02:24:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.151+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2da802d1f0 con 0x7f2da8101930 2024-03-19T02:24:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.153+0000 7f2da4ff9640 1 -- 172.21.15.116:0/21414714 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f2d9c09b750 con 0x7f2da8101930 2024-03-19T02:24:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:53.154 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-19T02:24:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 msgr2=0x7f2d900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 0x7f2d900a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006010 tx=0x7f2d98005be0 comp rx=0 tx=0).stop 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 msgr2=0x7f2da8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00c890 tx=0x7f2d9c00cd50 comp rx=0 tx=0).stop 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 shutdown_connections 2024-03-19T02:24:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2d900a2880 0x7f2d900a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da81022f0 0x7f2da819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da8101930 0x7f2da8197fc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 --2- 172.21.15.116:0/21414714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2da8100740 0x7f2da8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 >> 172.21.15.116:0/21414714 conn(0x7f2da80fbcb0 msgr2=0x7f2da8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 shutdown_connections 2024-03-19T02:24:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:53.155+0000 7f2dae0ff640 1 -- 172.21.15.116:0/21414714 wait complete. 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:52 smithi154 podman[65813]: 2024-03-19 02:24:52.996138719 +0000 UTC m=+0.417245018 container init fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:52 smithi154 podman[65813]: 2024-03-19 02:24:52.998852482 +0000 UTC m=+0.419958777 container start fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:53 smithi154 ceph-mds[65832]: -- 172.21.15.154:0/494631472 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ba685da40 con 0x556ba6830c00 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: starting mds.h at 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:53 smithi154 bash[65813]: fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 2024-03-19T02:24:53.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:24:53 smithi154 systemd[1]: Started Ceph mds.h for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/21414714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: Deploying daemon mds.i on smithi162 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:boot 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: fsmap 8 up:standby 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:24:54.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:54 smithi162 ceph-mon[35761]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/21414714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: Deploying daemon mds.i on smithi162 2024-03-19T02:24:54.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:boot 2024-03-19T02:24:54.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: fsmap 8 up:standby 2024-03-19T02:24:54.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:24:54.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:54 smithi154 ceph-mon[35693]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/21414714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: Deploying daemon mds.i on smithi162 2024-03-19T02:24:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:boot 2024-03-19T02:24:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: fsmap 8 up:standby 2024-03-19T02:24:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:24:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:54 smithi116 ceph-mon[32263]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:55.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:55.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 -- 172.21.15.116:0/1683305345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0101950 msgr2=0x7f13b0101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:55.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 --2- 172.21.15.116:0/1683305345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0101950 0x7f13b0101dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f13a4009900 tx=0x7f13a402bc30 comp rx=0 tx=0).stop 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 -- 172.21.15.116:0/1683305345 shutdown_connections 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 --2- 172.21.15.116:0/1683305345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b0108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 --2- 172.21.15.116:0/1683305345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0101950 0x7f13b0101dd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 --2- 172.21.15.116:0/1683305345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0100760 0x7f13b0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 -- 172.21.15.116:0/1683305345 >> 172.21.15.116:0/1683305345 conn(0x7f13b00fbcd0 msgr2=0x7f13b00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 -- 172.21.15.116:0/1683305345 shutdown_connections 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.282+0000 7f13b5a73640 1 -- 172.21.15.116:0/1683305345 wait complete. 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 Processor -- start 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 -- start start 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 0x7f13b0196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:55.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010c2f0 con 0x7f13b0100760 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010c170 con 0x7f13b0102310 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13b5a73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010c470 con 0x7f13b0101950 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51798/0 (socket says 172.21.15.116:51798) 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13af7fe640 1 -- 172.21.15.116:0/2255149725 learned_addr learned my addr 172.21.15.116:0/2255149725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13aeffd640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 0x7f13b0196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.283+0000 7f13affff640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 -- 172.21.15.116:0/2255149725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 msgr2=0x7f13b0196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 0x7f13b0196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 -- 172.21.15.116:0/2255149725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 msgr2=0x7f13b019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 -- 172.21.15.116:0/2255149725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b019e630 con 0x7f13b0100760 2024-03-19T02:24:55.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13aeffd640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 0x7f13b0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13affff640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13af7fe640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d770 tx=0x7f13a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0017070 con 0x7f13b0100760 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13a0009c30 con 0x7f13b0100760 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b019e920 con 0x7f13b0100760 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0003d00 con 0x7f13b0100760 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.284+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b019eef0 con 0x7f13b0100760 2024-03-19T02:24:55.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.285+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b0068a60 con 0x7f13b0100760 2024-03-19T02:24:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.286+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f13a0003ea0 con 0x7f13b0100760 2024-03-19T02:24:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.287+0000 7f13acff9640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 0x7f13980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.287+0000 7f13aeffd640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 0x7f13980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.287+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f13a00ce3e0 con 0x7f13b0100760 2024-03-19T02:24:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.287+0000 7f13aeffd640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 0x7f13980a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f13b0197f20 tx=0x7f13a402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.290+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13a009b920 con 0x7f13b0100760 2024-03-19T02:24:55.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 systemd[1]: Starting Ceph mds.i for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:24:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.502+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13b002d5b0 con 0x7f13b0100760 2024-03-19T02:24:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.503+0000 7f13acff9640 1 -- 172.21.15.116:0/2255149725 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7f13a0009de0 con 0x7f13b0100760 2024-03-19T02:24:55.503 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:55.503 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T02:24:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-03-19T02:24:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.504+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 msgr2=0x7f13980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.504+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 0x7f13980a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f13b0197f20 tx=0x7f13a402c7e0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.504+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 msgr2=0x7f13b0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.504+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d770 tx=0x7f13a000dc30 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 shutdown_connections 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f13980a2800 0x7f13980a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b0102310 0x7f13b019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f13b0101950 0x7f13b0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 --2- 172.21.15.116:0/2255149725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b0100760 0x7f13b0196a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 >> 172.21.15.116:0/2255149725 conn(0x7f13b00fbcd0 msgr2=0x7f13b0104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 shutdown_connections 2024-03-19T02:24:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:55.505+0000 7f13b5a73640 1 -- 172.21.15.116:0/2255149725 wait complete. 2024-03-19T02:24:55.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:55 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2255149725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:55.789 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 podman[66675]: 2024-03-19 02:24:55.537032596 +0000 UTC m=+0.110401423 container create e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T02:24:55.789 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 podman[66675]: 2024-03-19 02:24:55.444977526 +0000 UTC m=+0.018346366 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:24:55.789 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 podman[66675]: 2024-03-19 02:24:55.789750704 +0000 UTC m=+0.363119545 container init e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-19T02:24:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:55 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2255149725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:55 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2255149725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:56.167 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 podman[66675]: 2024-03-19 02:24:55.792619938 +0000 UTC m=+0.365988765 container start e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-19T02:24:56.167 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 ceph-mds[66695]: -- 172.21.15.162:0/1623114491 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fee34d1a40 con 0x55fee34a4c00 2024-03-19T02:24:56.167 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: starting mds.i at 2024-03-19T02:24:56.167 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 bash[66675]: e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d 2024-03-19T02:24:56.168 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:24:55 smithi162 systemd[1]: Started Ceph mds.i for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:56 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:56.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:56.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:56 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:24:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:56 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:24:57.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- 172.21.15.116:0/1547933371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 msgr2=0x7fc45c1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 --2- 172.21.15.116:0/1547933371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c1099b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc454009ca0 tx=0x7fc45402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- 172.21.15.116:0/1547933371 shutdown_connections 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 --2- 172.21.15.116:0/1547933371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c1099b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 --2- 172.21.15.116:0/1547933371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 --2- 172.21.15.116:0/1547933371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 0x7fc45c104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- 172.21.15.116:0/1547933371 >> 172.21.15.116:0/1547933371 conn(0x7fc45c0fbcb0 msgr2=0x7fc45c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:57.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- 172.21.15.116:0/1547933371 shutdown_connections 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- 172.21.15.116:0/1547933371 wait complete. 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 Processor -- start 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.625+0000 7fc46366a640 1 -- start start 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 0x7fc45c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c10d390 con 0x7fc45c1075c0 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c10d210 con 0x7fc45c102360 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c10d510 con 0x7fc45c104c90 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49950/0 (socket says 172.21.15.116:49950) 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 -- 172.21.15.116:0/518964437 learned_addr learned my addr 172.21.15.116:0/518964437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:57.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461406640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 0x7fc45c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc460c05640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 -- 172.21.15.116:0/518964437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 msgr2=0x7fc45c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 -- 172.21.15.116:0/518964437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 msgr2=0x7fc45c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 0x7fc45c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 -- 172.21.15.116:0/518964437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45c19f6b0 con 0x7fc45c1075c0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc460c05640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc461c07640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc45402c700 tx=0x7fc4540328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454041020 con 0x7fc45c1075c0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc454003e20 con 0x7fc45c1075c0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45c19f940 con 0x7fc45c1075c0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454039040 con 0x7fc45c1075c0 2024-03-19T02:24:57.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.626+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc45c19fe30 con 0x7fc45c1075c0 2024-03-19T02:24:57.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.627+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc45c0fd7a0 con 0x7fc45c1075c0 2024-03-19T02:24:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.628+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc454044020 con 0x7fc45c1075c0 2024-03-19T02:24:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.629+0000 7fc4527fc640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 0x7fc4480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.629+0000 7fc461406640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 0x7fc4480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.629+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc4540f0e80 con 0x7fc45c1075c0 2024-03-19T02:24:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.630+0000 7fc461406640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 0x7fc4480a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc45800bbe0 tx=0x7fc45800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.633+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc45404b020 con 0x7fc45c1075c0 2024-03-19T02:24:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:57 smithi116 ceph-mon[32263]: Deploying daemon mds.g on smithi116 2024-03-19T02:24:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:57 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:boot 2024-03-19T02:24:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:57 smithi116 ceph-mon[32263]: fsmap 9 up:standby 2024-03-19T02:24:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:57 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:24:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.859+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc45c02d1f0 con 0x7fc45c1075c0 2024-03-19T02:24:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.860+0000 7fc4527fc640 1 -- 172.21.15.116:0/518964437 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7fc4540a8090 con 0x7fc45c1075c0 2024-03-19T02:24:57.861 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:24:57.861 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-19T02:24:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 msgr2=0x7fc4480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 0x7fc4480a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc45800bbe0 tx=0x7fc45800b390 comp rx=0 tx=0).stop 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 msgr2=0x7fc45c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc45402c700 tx=0x7fc4540328a0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 shutdown_connections 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc4480a2800 0x7fc4480a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c1075c0 0x7fc45c19f040 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc45c104c90 0x7fc45c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 --2- 172.21.15.116:0/518964437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc45c102360 0x7fc45c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 >> 172.21.15.116:0/518964437 conn(0x7fc45c0fbcb0 msgr2=0x7fc45c102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 shutdown_connections 2024-03-19T02:24:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:57.862+0000 7fc46366a640 1 -- 172.21.15.116:0/518964437 wait complete. 2024-03-19T02:24:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:57 smithi162 ceph-mon[35761]: Deploying daemon mds.g on smithi116 2024-03-19T02:24:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:57 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:boot 2024-03-19T02:24:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:57 smithi162 ceph-mon[35761]: fsmap 9 up:standby 2024-03-19T02:24:57.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:57 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:24:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:57 smithi154 ceph-mon[35693]: Deploying daemon mds.g on smithi116 2024-03-19T02:24:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:57 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:boot 2024-03-19T02:24:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:57 smithi154 ceph-mon[35693]: fsmap 9 up:standby 2024-03-19T02:24:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:57 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:24:58.508 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 podman[71445]: 2024-03-19 02:24:58.403962106 +0000 UTC m=+0.123633411 container create 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-19T02:24:58.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:58 smithi162 ceph-mon[35761]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:58.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:58 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/518964437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:58 smithi154 ceph-mon[35693]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:58 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/518964437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:58 smithi116 ceph-mon[32263]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:24:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/518964437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:24:59.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 podman[71445]: 2024-03-19 02:24:58.640283355 +0000 UTC m=+0.359954633 container init 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-19T02:24:59.009 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 podman[71445]: 2024-03-19 02:24:58.643104382 +0000 UTC m=+0.362775646 container start 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T02:24:59.009 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 ceph-mds[71465]: -- 172.21.15.116:0/3645755806 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55831f983a40 con 0x55831f957000 2024-03-19T02:24:59.009 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: starting mds.g at 2024-03-19T02:24:59.009 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 bash[71445]: 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 2024-03-19T02:24:59.009 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:24:58 smithi116 systemd[1]: Started Ceph mds.g for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:24:59.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- 172.21.15.116:0/4218896215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850102110 msgr2=0x7f08501089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- 172.21.15.116:0/4218896215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850102110 0x7f08501089a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0848009ca0 tx=0x7f084802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- 172.21.15.116:0/4218896215 shutdown_connections 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- 172.21.15.116:0/4218896215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850102110 0x7f08501089a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- 172.21.15.116:0/4218896215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850101750 0x7f0850101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- 172.21.15.116:0/4218896215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850100560 0x7f0850100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- 172.21.15.116:0/4218896215 >> 172.21.15.116:0/4218896215 conn(0x7f08500fbaf0 msgr2=0x7f08500fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- 172.21.15.116:0/4218896215 shutdown_connections 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- 172.21.15.116:0/4218896215 wait complete. 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 Processor -- start 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- start start 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850101750 0x7f0850197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 0x7f085019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f085010c0a0 con 0x7f0850100560 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f085010bf20 con 0x7f0850101750 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.981+0000 7f084ffff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f085010c220 con 0x7f0850102110 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49990/0 (socket says 172.21.15.116:49990) 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 -- 172.21.15.116:0/283616501 learned_addr learned my addr 172.21.15.116:0/283616501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:24:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084f7fe640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 0x7f085019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 -- 172.21.15.116:0/283616501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 msgr2=0x7f085019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 0x7f085019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f0847fff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850101750 0x7f0850197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 -- 172.21.15.116:0/283616501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850101750 msgr2=0x7f0850197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850101750 0x7f0850197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 -- 172.21.15.116:0/283616501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085019f460 con 0x7f0850100560 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084f7fe640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 0x7f085019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084effd640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f084000c970 tx=0x7f084000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840004220 con 0x7f0850100560 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08400043c0 con 0x7f0850100560 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085019f750 con 0x7f0850100560 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084000f450 con 0x7f0850100560 2024-03-19T02:24:59.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.982+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085019fcf0 con 0x7f0850100560 2024-03-19T02:24:59.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.983+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085019ffb0 con 0x7f0850100560 2024-03-19T02:24:59.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.984+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f08400079b0 con 0x7f0850100560 2024-03-19T02:24:59.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.985+0000 7f08549fd640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 0x7f083c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:24:59.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.985+0000 7f0847fff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 0x7f083c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:24:59.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.985+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f08400ce390 con 0x7f0850100560 2024-03-19T02:24:59.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.985+0000 7f0847fff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 0x7f083c0a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0838005fd0 tx=0x7f0838005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:24:59.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:24:59.989+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0840014030 con 0x7f0850100560 2024-03-19T02:25:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: Deploying daemon mds.a on smithi116 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6838/1891911264,v1:172.21.15.116:6839/1891911264] up:boot 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: fsmap 10 up:standby 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:25:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:24:59 smithi162 ceph-mon[35761]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.204+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08501a0180 con 0x7f0850100560 2024-03-19T02:25:00.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.205+0000 7f08549fd640 1 -- 172.21.15.116:0/283616501 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f084009b8d0 con 0x7f0850100560 2024-03-19T02:25:00.205 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:00.206 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-19T02:25:00.206 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-19T02:25:00.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 msgr2=0x7f083c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 0x7f083c0a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0838005fd0 tx=0x7f0838005850 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 msgr2=0x7f0850197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f084000c970 tx=0x7f084000ce30 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 shutdown_connections 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f083c0a2880 0x7f083c0a4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0850102110 0x7f085019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0850101750 0x7f0850197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 --2- 172.21.15.116:0/283616501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0850100560 0x7f0850197840 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 >> 172.21.15.116:0/283616501 conn(0x7f08500fbaf0 msgr2=0x7f08500fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 shutdown_connections 2024-03-19T02:25:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:00.207+0000 7f084ffff640 1 -- 172.21.15.116:0/283616501 wait complete. 2024-03-19T02:25:00.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:00.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: Deploying daemon mds.a on smithi116 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6838/1891911264,v1:172.21.15.116:6839/1891911264] up:boot 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: fsmap 10 up:standby 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:25:00.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:24:59 smithi116 ceph-mon[32263]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: Deploying daemon mds.a on smithi116 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6838/1891911264,v1:172.21.15.116:6839/1891911264] up:boot 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: fsmap 10 up:standby 2024-03-19T02:25:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:25:00.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:24:59 smithi154 ceph-mon[35693]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/283616501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:01.173 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/283616501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:01.173 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:00 smithi116 systemd[1]: Starting Ceph mds.a for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:25:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/283616501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:01.441 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 podman[71804]: 2024-03-19 02:25:01.076237716 +0000 UTC m=+0.017824522 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:25:01.442 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 podman[71804]: 2024-03-19 02:25:01.185000458 +0000 UTC m=+0.126587258 container create 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T02:25:01.442 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 podman[71804]: 2024-03-19 02:25:01.44198604 +0000 UTC m=+0.383572840 container init 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:25:01.758 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 podman[71804]: 2024-03-19 02:25:01.444682414 +0000 UTC m=+0.386269221 container start 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:25:01.758 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 ceph-mds[71824]: -- 172.21.15.116:0/3633486519 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564170257a40 con 0x56417022ac00 2024-03-19T02:25:01.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: starting mds.a at 2024-03-19T02:25:01.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 bash[71804]: 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 2024-03-19T02:25:01.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:25:01 smithi116 systemd[1]: Started Ceph mds.a for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:25:02.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:25:02.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- 172.21.15.116:0/1762480705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 msgr2=0x7fdd40104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- 172.21.15.116:0/1762480705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40104720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009ca0 tx=0x7fdd3c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- 172.21.15.116:0/1762480705 shutdown_connections 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- 172.21.15.116:0/1762480705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd40107590 0x7fdd40109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- 172.21.15.116:0/1762480705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 0x7fdd40107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- 172.21.15.116:0/1762480705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40104720 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- 172.21.15.116:0/1762480705 >> 172.21.15.116:0/1762480705 conn(0x7fdd400fbce0 msgr2=0x7fdd400fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- 172.21.15.116:0/1762480705 shutdown_connections 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- 172.21.15.116:0/1762480705 wait complete. 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 Processor -- start 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- start start 2024-03-19T02:25:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 0x7fdd40198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd40107590 0x7fdd4019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d3f0 con 0x7fdd40102330 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d270 con 0x7fdd40104c60 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd47e85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d570 con 0x7fdd40107590 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.331+0000 7fdd45c21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50022/0 (socket says 172.21.15.116:50022) 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 -- 172.21.15.116:0/3023659321 learned_addr learned my addr 172.21.15.116:0/3023659321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45420640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 0x7fdd40198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 -- 172.21.15.116:0/3023659321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd40107590 msgr2=0x7fdd4019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd40107590 0x7fdd4019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 -- 172.21.15.116:0/3023659321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 msgr2=0x7fdd40198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 0x7fdd40198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 -- 172.21.15.116:0/3023659321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4019f6f0 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd45c21640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009dc0 tx=0x7fdd3c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c041020 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4019f980 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd3c003e20 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c039040 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.332+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4019ff40 con 0x7fdd40102330 2024-03-19T02:25:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.333+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd400fd7d0 con 0x7fdd40102330 2024-03-19T02:25:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.334+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fdd3c044020 con 0x7fdd40102330 2024-03-19T02:25:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.335+0000 7fdd36ffd640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 0x7fdd2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.335+0000 7fdd45420640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 0x7fdd2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.335+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdd3c0f08f0 con 0x7fdd40102330 2024-03-19T02:25:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.335+0000 7fdd45420640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 0x7fdd2c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdd40198fe0 tx=0x7fdd3000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.339+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd3c04b020 con 0x7fdd40102330 2024-03-19T02:25:02.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.552+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd4002d1f0 con 0x7fdd40102330 2024-03-19T02:25:02.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.553+0000 7fdd36ffd640 1 -- 172.21.15.116:0/3023659321 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9668 (secure 0 0 0) 0x7fdd3c0bde30 con 0x7fdd40102330 2024-03-19T02:25:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:02.555 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-19T02:25:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 12 2024-03-19T02:25:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 msgr2=0x7fdd2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 0x7fdd2c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdd40198fe0 tx=0x7fdd3000b390 comp rx=0 tx=0).stop 2024-03-19T02:25:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 msgr2=0x7fdd40197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009dc0 tx=0x7fdd3c0328a0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 shutdown_connections 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd2c0a2800 0x7fdd2c0a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd40107590 0x7fdd4019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd40104c60 0x7fdd40198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 --2- 172.21.15.116:0/3023659321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40102330 0x7fdd40197ad0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:02.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 >> 172.21.15.116:0/3023659321 conn(0x7fdd400fbce0 msgr2=0x7fdd40102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:02.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 shutdown_connections 2024-03-19T02:25:02.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:02.555+0000 7fdd47e85640 1 -- 172.21.15.116:0/3023659321 wait complete. 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: Deploying daemon mds.b on smithi154 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6840/3941624950,v1:172.21.15.116:6841/3941624950] up:boot 2024-03-19T02:25:02.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: fsmap 11 up:standby 2024-03-19T02:25:02.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:25:02.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:02 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3023659321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:02.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: Deploying daemon mds.b on smithi154 2024-03-19T02:25:02.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6840/3941624950,v1:172.21.15.116:6841/3941624950] up:boot 2024-03-19T02:25:02.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: fsmap 11 up:standby 2024-03-19T02:25:02.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:25:02.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:02 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3023659321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: Deploying daemon mds.b on smithi154 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6840/3941624950,v1:172.21.15.116:6841/3941624950] up:boot 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: fsmap 11 up:standby 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:25:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:02 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3023659321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:04.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:03 smithi154 systemd[1]: Starting Ceph mds.b for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:25:04.378 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 podman[66120]: 2024-03-19 02:25:03.999880874 +0000 UTC m=+0.019312486 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:25:04.378 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 podman[66120]: 2024-03-19 02:25:04.116978644 +0000 UTC m=+0.136410255 container create 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-19T02:25:04.378 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 podman[66120]: 2024-03-19 02:25:04.378164034 +0000 UTC m=+0.397595651 container init 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T02:25:04.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:25:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 -- 172.21.15.116:0/3781300016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d41022f0 msgr2=0x7fd8d4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 --2- 172.21.15.116:0/3781300016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d41022f0 0x7fd8d4108b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009ca0 tx=0x7fd8cc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 -- 172.21.15.116:0/3781300016 shutdown_connections 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 --2- 172.21.15.116:0/3781300016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d41022f0 0x7fd8d4108b80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 --2- 172.21.15.116:0/3781300016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d4101930 0x7fd8d4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 --2- 172.21.15.116:0/3781300016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4100740 0x7fd8d4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.682+0000 7fd8db510640 1 -- 172.21.15.116:0/3781300016 >> 172.21.15.116:0/3781300016 conn(0x7fd8d40fbcb0 msgr2=0x7fd8d40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- 172.21.15.116:0/3781300016 shutdown_connections 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- 172.21.15.116:0/3781300016 wait complete. 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 Processor -- start 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- start start 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 0x7fd8d4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 0x7fd8d419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d410d350 con 0x7fd8d4100740 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d410d1d0 con 0x7fd8d41022f0 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8db510640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d410d4d0 con 0x7fd8d4101930 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8d92ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8d92ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50038/0 (socket says 172.21.15.116:50038) 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8d92ac640 1 -- 172.21.15.116:0/1227683048 learned_addr learned my addr 172.21.15.116:0/1227683048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.683+0000 7fd8d8aab640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 0x7fd8d4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d9aad640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 0x7fd8d419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 -- 172.21.15.116:0/1227683048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 msgr2=0x7fd8d4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 0x7fd8d4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 -- 172.21.15.116:0/1227683048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 msgr2=0x7fd8d419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 0x7fd8d419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 -- 172.21.15.116:0/1227683048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d419f5f0 con 0x7fd8d4100740 2024-03-19T02:25:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d9aad640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 0x7fd8d419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d8aab640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 0x7fd8d4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8d92ac640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000d770 tx=0x7fd8d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0017070 con 0x7fd8d4100740 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8d0009c30 con 0x7fd8d4100740 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d419f8e0 con 0x7fd8d4100740 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0003d00 con 0x7fd8d4100740 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.684+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d419fe30 con 0x7fd8d4100740 2024-03-19T02:25:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.685+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d4107a60 con 0x7fd8d4100740 2024-03-19T02:25:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.686+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd8d0003ea0 con 0x7fd8d4100740 2024-03-19T02:25:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.687+0000 7fd8ca7fc640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.687+0000 7fd8d8aab640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.687+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd8d00ce3b0 con 0x7fd8d4100740 2024-03-19T02:25:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.687+0000 7fd8d8aab640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4198ee0 tx=0x7fd8c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:04.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.690+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd8d009b8f0 con 0x7fd8d4100740 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 podman[66120]: 2024-03-19 02:25:04.380859945 +0000 UTC m=+0.400291563 container start 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True) 2024-03-19T02:25:04.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-mds[66139]: -- 172.21.15.154:0/2335245763 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560cc51a7a40 con 0x560cc517ac00 2024-03-19T02:25:04.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: starting mds.b at 2024-03-19T02:25:04.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 bash[66120]: 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f 2024-03-19T02:25:04.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:04 smithi154 systemd[1]: Started Ceph mds.b for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:25:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:04 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.927+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8d402d1f0 con 0x7fd8d4100740 2024-03-19T02:25:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.927+0000 7fd8ca7fc640 1 -- 172.21.15.116:0/1227683048 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10497 (secure 0 0 0) 0x7fd8d0009de0 con 0x7fd8d4100740 2024-03-19T02:25:04.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:04.928 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-19T02:25:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 13 2024-03-19T02:25:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 msgr2=0x7fd8c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4198ee0 tx=0x7fd8c4005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 msgr2=0x7fd8d41979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000d770 tx=0x7fd8d000dc30 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 shutdown_connections 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8d41022f0 0x7fd8d419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8d4101930 0x7fd8d4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 --2- 172.21.15.116:0/1227683048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8d4100740 0x7fd8d41979d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 >> 172.21.15.116:0/1227683048 conn(0x7fd8d40fbcb0 msgr2=0x7fd8d4104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 shutdown_connections 2024-03-19T02:25:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:04.930+0000 7fd8db510640 1 -- 172.21.15.116:0/1227683048 wait complete. 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:04 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:04.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 -- 172.21.15.116:0/1319031203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a501022f0 msgr2=0x7f4a50108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/1319031203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a501022f0 0x7f4a50108b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009ca0 tx=0x7f4a4802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 -- 172.21.15.116:0/1319031203 shutdown_connections 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/1319031203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a501022f0 0x7f4a50108b80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/1319031203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101930 0x7f4a50101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/1319031203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 0x7f4a50100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 -- 172.21.15.116:0/1319031203 >> 172.21.15.116:0/1319031203 conn(0x7f4a500fbcb0 msgr2=0x7f4a500fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 -- 172.21.15.116:0/1319031203 shutdown_connections 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.044+0000 7f4a54e9a640 1 -- 172.21.15.116:0/1319031203 wait complete. 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 Processor -- start 2024-03-19T02:25:05.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 -- start start 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 0x7f4a50196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a501022f0 0x7f4a5019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010c270 con 0x7f4a50101930 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010c0f0 con 0x7f4a50100740 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a54e9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010c3f0 con 0x7f4a501022f0 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a4dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a4dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50064/0 (socket says 172.21.15.116:50064) 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a4dd9b640 1 -- 172.21.15.116:0/3852666086 learned_addr learned my addr 172.21.15.116:0/3852666086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a4e59c640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 0x7f4a50196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.045+0000 7f4a4ed9d640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a501022f0 0x7f4a5019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 -- 172.21.15.116:0/3852666086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a501022f0 msgr2=0x7f4a5019dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a501022f0 0x7f4a5019dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 -- 172.21.15.116:0/3852666086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 msgr2=0x7f4a50196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 0x7f4a50196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 -- 172.21.15.116:0/3852666086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5019e550 con 0x7f4a50101930 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a4dd9b640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400e9f0 tx=0x7f4a4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4400cd80 con 0x7f4a50101930 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5019e840 con 0x7f4a50101930 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a440054d0 con 0x7f4a50101930 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44010490 con 0x7f4a50101930 2024-03-19T02:25:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.046+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5019ed90 con 0x7f4a50101930 2024-03-19T02:25:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.047+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a50107a60 con 0x7f4a50101930 2024-03-19T02:25:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.048+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4a44005000 con 0x7f4a50101930 2024-03-19T02:25:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.049+0000 7f4a3f7fe640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 0x7f4a380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.049+0000 7f4a4e59c640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 0x7f4a380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.049+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4a440ceaf0 con 0x7f4a50101930 2024-03-19T02:25:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.049+0000 7f4a4e59c640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 0x7f4a380a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a40005fd0 tx=0x7f4a4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.057+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a44053740 con 0x7f4a50101930 2024-03-19T02:25:05.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.249+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a5002d290 con 0x7f4a50101930 2024-03-19T02:25:05.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.250+0000 7f4a3f7fe640 1 -- 172.21.15.116:0/3852666086 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18524 (secure 0 0 0) 0x7f4a440052f0 con 0x7f4a50101930 2024-03-19T02:25:05.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:05.251 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:24:21.819283+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 msgr2=0x7f4a380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 0x7f4a380a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a40005fd0 tx=0x7f4a4000b410 comp rx=0 tx=0).stop 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 msgr2=0x7f4a50196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400e9f0 tx=0x7f4a4400edf0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 shutdown_connections 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a380a2800 0x7f4a380a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a501022f0 0x7f4a5019dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50101930 0x7f4a50196e70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 --2- 172.21.15.116:0/3852666086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a50100740 0x7f4a50196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 >> 172.21.15.116:0/3852666086 conn(0x7f4a500fbcb0 msgr2=0x7f4a50104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 shutdown_connections 2024-03-19T02:25:05.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.253+0000 7f4a54e9a640 1 -- 172.21.15.116:0/3852666086 wait complete. 2024-03-19T02:25:05.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:25:05.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:25:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 -- 172.21.15.116:0/3559529556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00100730 msgr2=0x7f1b00100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3559529556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00100730 0x7f1b00100b50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1afc009900 tx=0x7f1afc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 -- 172.21.15.116:0/3559529556 shutdown_connections 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3559529556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b001022e0 0x7f1b00108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3559529556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b00101920 0x7f1b00101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3559529556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00100730 0x7f1b00100b50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 -- 172.21.15.116:0/3559529556 >> 172.21.15.116:0/3559529556 conn(0x7f1b000fbcc0 msgr2=0x7f1b000fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 -- 172.21.15.116:0/3559529556 shutdown_connections 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.379+0000 7f1b07c68640 1 -- 172.21.15.116:0/3559529556 wait complete. 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 Processor -- start 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 -- start start 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b00100730 0x7f1b0006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b001022e0 0x7f1b001a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0010c260 con 0x7f1b00101920 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0010c0e0 con 0x7f1b001022e0 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b07c68640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0010c3e0 con 0x7f1b00100730 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50072/0 (socket says 172.21.15.116:50072) 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 -- 172.21.15.116:0/3677490644 learned_addr learned my addr 172.21.15.116:0/3677490644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b06205640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b001022e0 0x7f1b001a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05a04640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b00100730 0x7f1b0006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 -- 172.21.15.116:0/3677490644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b00100730 msgr2=0x7f1b0006a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b00100730 0x7f1b0006a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 -- 172.21.15.116:0/3677490644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b001022e0 msgr2=0x7f1b001a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b001022e0 0x7f1b001a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 -- 172.21.15.116:0/3677490644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b001a1920 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1b05203640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e9f0 tx=0x7f1af000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.380+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af000cd80 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.381+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1af00108e0 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.381+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b001a1bb0 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.381+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0018920 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.381+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b001a2100 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.382+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b00107ad0 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.383+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1af0010a90 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.383+0000 7f1af6ffd640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 0x7f1aec0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.383+0000 7f1b05a04640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 0x7f1aec0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.383+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1af00ce860 con 0x7f1b00101920 2024-03-19T02:25:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.384+0000 7f1b05a04640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 0x7f1aec0a4cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1afc005ab0 tx=0x7f1afc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.388+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1af009b750 con 0x7f1b00101920 2024-03-19T02:25:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.564+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b0002d3a0 con 0x7f1b00101920 2024-03-19T02:25:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.565+0000 7f1af6ffd640 1 -- 172.21.15.116:0/3677490644 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f1af009f7f0 con 0x7f1b00101920 2024-03-19T02:25:05.565 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 msgr2=0x7f1aec0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 0x7f1aec0a4cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1afc005ab0 tx=0x7f1afc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 msgr2=0x7f1b0019ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1af000e9f0 tx=0x7f1af000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 shutdown_connections 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1aec0a2830 0x7f1aec0a4cf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b001022e0 0x7f1b001a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b00101920 0x7f1b0019ebe0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 --2- 172.21.15.116:0/3677490644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b00100730 0x7f1b0006a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 >> 172.21.15.116:0/3677490644 conn(0x7f1b000fbcc0 msgr2=0x7f1b00104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 shutdown_connections 2024-03-19T02:25:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.567+0000 7f1b07c68640 1 -- 172.21.15.116:0/3677490644 wait complete. 2024-03-19T02:25:05.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T02:25:05.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 -- 172.21.15.116:0/299720175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c100740 msgr2=0x7f917c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 --2- 172.21.15.116:0/299720175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c100740 0x7f917c100b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f916c009900 tx=0x7f916c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 -- 172.21.15.116:0/299720175 shutdown_connections 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 --2- 172.21.15.116:0/299720175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c1022f0 0x7f917c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 --2- 172.21.15.116:0/299720175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c101930 0x7f917c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 --2- 172.21.15.116:0/299720175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c100740 0x7f917c100b60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 -- 172.21.15.116:0/299720175 >> 172.21.15.116:0/299720175 conn(0x7f917c0fbcb0 msgr2=0x7f917c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 -- 172.21.15.116:0/299720175 shutdown_connections 2024-03-19T02:25:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.686+0000 7f918224b640 1 -- 172.21.15.116:0/299720175 wait complete. 2024-03-19T02:25:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 Processor -- start 2024-03-19T02:25:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 -- start start 2024-03-19T02:25:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 0x7f917c195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 0x7f917c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f917c10b170 con 0x7f917c101930 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f917c10aff0 con 0x7f917c1022f0 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f918224b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f917c10b2f0 con 0x7f917c100740 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f917affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f917affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50094/0 (socket says 172.21.15.116:50094) 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f917affd640 1 -- 172.21.15.116:0/4084666903 learned_addr learned my addr 172.21.15.116:0/4084666903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f917b7fe640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 0x7f917c195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.687+0000 7f917bfff640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 0x7f917c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 -- 172.21.15.116:0/4084666903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 msgr2=0x7f917c195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 0x7f917c195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 -- 172.21.15.116:0/4084666903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 msgr2=0x7f917c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 0x7f917c19ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 -- 172.21.15.116:0/4084666903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c19d4a0 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917b7fe640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 0x7f917c195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917bfff640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 0x7f917c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f917affd640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9170009be0 tx=0x7f91700071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170007e00 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f917001e070 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c19d790 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170018730 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.688+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c19dd30 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.689+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c107a80 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.690+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9170019b30 con 0x7f917c101930 2024-03-19T02:25:05.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.691+0000 7f9178ff9640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 0x7f91640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.691+0000 7f917b7fe640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 0x7f91640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.691+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f91700d20f0 con 0x7f917c101930 2024-03-19T02:25:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.691+0000 7f917b7fe640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 0x7f91640a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f916c005ab0 tx=0x7f916c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:05.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.696+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9170016070 con 0x7f917c101930 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:boot 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: fsmap 12 up:standby 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1227683048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3852666086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.824 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:05 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3677490644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.887+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f917c02d3a0 con 0x7f917c101930 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.888+0000 7f9178ff9640 1 -- 172.21.15.116:0/4084666903 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f917009f630 con 0x7f917c101930 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:25:05.888 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-03-19T02:25:05.889 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-03-19T02:25:05.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:25:05.889 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-19T02:25:05.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 msgr2=0x7f91640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 0x7f91640a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f916c005ab0 tx=0x7f916c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 msgr2=0x7f917c195dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9170009be0 tx=0x7f91700071c0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 shutdown_connections 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f91640a2880 0x7f91640a4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f917c1022f0 0x7f917c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f917c101930 0x7f917c195dc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 --2- 172.21.15.116:0/4084666903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f917c100740 0x7f917c195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:05.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 >> 172.21.15.116:0/4084666903 conn(0x7f917c0fbcb0 msgr2=0x7f917c104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:05.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 shutdown_connections 2024-03-19T02:25:05.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:05.890+0000 7f918224b640 1 -- 172.21.15.116:0/4084666903 wait complete. 2024-03-19T02:25:05.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:boot 2024-03-19T02:25:05.900 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: fsmap 12 up:standby 2024-03-19T02:25:05.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:25:05.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1227683048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3852666086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.901 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:05 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3677490644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:05.902 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T02:25:05.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T02:25:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:boot 2024-03-19T02:25:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: fsmap 12 up:standby 2024-03-19T02:25:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:25:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1227683048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3852666086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:05 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3677490644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 -- 172.21.15.116:0/1808593767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 msgr2=0x7f3164108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 --2- 172.21.15.116:0/1808593767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f3164108b80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f315c009ca0 tx=0x7f315c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 -- 172.21.15.116:0/1808593767 shutdown_connections 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 --2- 172.21.15.116:0/1808593767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f3164108b80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 --2- 172.21.15.116:0/1808593767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 0x7f3164101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 --2- 172.21.15.116:0/1808593767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 0x7f3164100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 -- 172.21.15.116:0/1808593767 >> 172.21.15.116:0/1808593767 conn(0x7f31640fbcd0 msgr2=0x7f31640fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 -- 172.21.15.116:0/1808593767 shutdown_connections 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 -- 172.21.15.116:0/1808593767 wait complete. 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.008+0000 7f316907e640 1 Processor -- start 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 -- start start 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 0x7f31641979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 0x7f3164197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d390 con 0x7f31641022f0 2024-03-19T02:25:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d210 con 0x7f3164100740 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316907e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f316410d510 con 0x7f3164101930 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50110/0 (socket says 172.21.15.116:50110) 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 -- 172.21.15.116:0/3173788915 learned_addr learned my addr 172.21.15.116:0/3173788915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f3162d9d640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 0x7f31641979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316259c640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 0x7f3164197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 -- 172.21.15.116:0/3173788915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 msgr2=0x7f3164197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 0x7f3164197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 -- 172.21.15.116:0/3173788915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 msgr2=0x7f31641979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 0x7f31641979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 -- 172.21.15.116:0/3173788915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316419f610 con 0x7f31641022f0 2024-03-19T02:25:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.009+0000 7f316359e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f315c02c700 tx=0x7f315c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.010+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c041020 con 0x7f31641022f0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.010+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f315c003e20 con 0x7f31641022f0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.010+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316419f8a0 con 0x7f31641022f0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.010+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c039040 con 0x7f31641022f0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.010+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316419fd90 con 0x7f31641022f0 2024-03-19T02:25:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.011+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3164107ae0 con 0x7f31641022f0 2024-03-19T02:25:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.012+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f315c044020 con 0x7f31641022f0 2024-03-19T02:25:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.013+0000 7f314bfff640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 0x7f314c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.013+0000 7f3162d9d640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 0x7f314c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.013+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f315c0f0da0 con 0x7f31641022f0 2024-03-19T02:25:06.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.014+0000 7f3162d9d640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 0x7f314c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3164101790 tx=0x7f315400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:06.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.017+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f315c04b020 con 0x7f31641022f0 2024-03-19T02:25:06.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.244+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f316402d370 con 0x7f31641022f0 2024-03-19T02:25:06.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.620+0000 7f314bfff640 1 -- 172.21.15.116:0/3173788915 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f315c0be2e0 con 0x7f31641022f0 2024-03-19T02:25:06.621 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 msgr2=0x7f314c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 0x7f314c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3164101790 tx=0x7f315400b390 comp rx=0 tx=0).stop 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 msgr2=0x7f316419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f315c02c700 tx=0x7f315c0328a0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 shutdown_connections 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f314c0a2800 0x7f314c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31641022f0 0x7f316419efa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3164101930 0x7f3164197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 --2- 172.21.15.116:0/3173788915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3164100740 0x7f31641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 >> 172.21.15.116:0/3173788915 conn(0x7f31640fbcd0 msgr2=0x7f3164104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 shutdown_connections 2024-03-19T02:25:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.622+0000 7f316907e640 1 -- 172.21.15.116:0/3173788915 wait complete. 2024-03-19T02:25:06.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-19T02:25:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 -- 172.21.15.116:0/1619619380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8101910 msgr2=0x7f76d8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 --2- 172.21.15.116:0/1619619380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8101910 0x7f76d8101d90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009900 tx=0x7f76c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:06.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 -- 172.21.15.116:0/1619619380 shutdown_connections 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 --2- 172.21.15.116:0/1619619380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d81022d0 0x7f76d8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 --2- 172.21.15.116:0/1619619380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8101910 0x7f76d8101d90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 --2- 172.21.15.116:0/1619619380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.744+0000 7f76e007e640 1 -- 172.21.15.116:0/1619619380 >> 172.21.15.116:0/1619619380 conn(0x7f76d80fbcb0 msgr2=0x7f76d80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- 172.21.15.116:0/1619619380 shutdown_connections 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- 172.21.15.116:0/1619619380 wait complete. 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 Processor -- start 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- start start 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 0x7f76d8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 0x7f76d819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810d370 con 0x7f76d81022d0 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810d1f0 con 0x7f76d8101910 2024-03-19T02:25:06.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76e007e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810d4f0 con 0x7f76d8100720 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76dd619640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 0x7f76d8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76dde1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76de61b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 0x7f76d819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76dd619640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 0x7f76d8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48680/0 (socket says 172.21.15.116:48680) 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76dd619640 1 -- 172.21.15.116:0/314738237 learned_addr learned my addr 172.21.15.116:0/314738237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.745+0000 7f76dde1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41070/0 (socket says 172.21.15.116:41070) 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 -- 172.21.15.116:0/314738237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 msgr2=0x7f76d8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 -- 172.21.15.116:0/314738237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 msgr2=0x7f76d8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:06.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 0x7f76d8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 -- 172.21.15.116:0/314738237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d819f6f0 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76de61b640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 0x7f76d819f080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76d000d6b0 tx=0x7f76d000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0004230 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76d00043d0 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d819f9e0 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0003de0 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.746+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d819fee0 con 0x7f76d81022d0 2024-03-19T02:25:06.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.747+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d8107b60 con 0x7f76d81022d0 2024-03-19T02:25:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.748+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f76d00035f0 con 0x7f76d81022d0 2024-03-19T02:25:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.749+0000 7f76ceffd640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 0x7f76c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.749+0000 7f76dde1a640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 0x7f76c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.749+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f76d00ce4b0 con 0x7f76d81022d0 2024-03-19T02:25:06.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.749+0000 7f76dde1a640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 0x7f76c40a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f76d8101770 tx=0x7f76d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.752+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76d0017020 con 0x7f76d81022d0 2024-03-19T02:25:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4084666903' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:25:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:25:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:06 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:06.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:06.932+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f76d802d310 con 0x7f76d81022d0 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4084666903' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:06 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4084666903' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:06 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.624+0000 7f76ceffd640 1 -- 172.21.15.116:0/314738237 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7f76d009b230 con 0x7f76d81022d0 2024-03-19T02:25:07.625 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 msgr2=0x7f76c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 0x7f76c40a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f76d8101770 tx=0x7f76d4009500 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 msgr2=0x7f76d819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 0x7f76d819f080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76d000d6b0 tx=0x7f76d000db70 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 shutdown_connections 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c40a2800 0x7f76c40a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81022d0 0x7f76d819f080 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d8101910 0x7f76d8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 --2- 172.21.15.116:0/314738237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d8100720 0x7f76d8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 >> 172.21.15.116:0/314738237 conn(0x7f76d80fbcb0 msgr2=0x7f76d8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 shutdown_connections 2024-03-19T02:25:07.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.626+0000 7f76e007e640 1 -- 172.21.15.116:0/314738237 wait complete. 2024-03-19T02:25:07.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T02:25:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 -- 172.21.15.116:0/2651885674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf88074200 msgr2=0x7faf88074660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 --2- 172.21.15.116:0/2651885674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf88074200 0x7faf88074660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009900 tx=0x7faf7c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 -- 172.21.15.116:0/2651885674 shutdown_connections 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 --2- 172.21.15.116:0/2651885674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf881096e0 0x7faf8810bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 --2- 172.21.15.116:0/2651885674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf88074200 0x7faf88074660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 --2- 172.21.15.116:0/2651885674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf880757e0 0x7faf88073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 -- 172.21.15.116:0/2651885674 >> 172.21.15.116:0/2651885674 conn(0x7faf880fcd80 msgr2=0x7faf880ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 -- 172.21.15.116:0/2651885674 shutdown_connections 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.759+0000 7faf8e281640 1 -- 172.21.15.116:0/2651885674 wait complete. 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 Processor -- start 2024-03-19T02:25:07.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 -- start start 2024-03-19T02:25:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf880757e0 0x7faf8819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:07.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810f560 con 0x7faf881096e0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810f3e0 con 0x7faf880757e0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.760+0000 7faf8e281640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf8810f6e0 con 0x7faf88074200 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf87fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf87fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47752/0 (socket says 172.21.15.116:47752) 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf87fff640 1 -- 172.21.15.116:0/23938154 learned_addr learned my addr 172.21.15.116:0/23938154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59264/0 (socket says 172.21.15.116:59264) 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf877fe640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf880757e0 0x7faf8819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 -- 172.21.15.116:0/23938154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 msgr2=0x7faf88199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 -- 172.21.15.116:0/23938154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf880757e0 msgr2=0x7faf8819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf880757e0 0x7faf8819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 -- 172.21.15.116:0/23938154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf881a18f0 con 0x7faf881096e0 2024-03-19T02:25:07.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf87fff640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8c81e640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e940 tx=0x7faf8000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80004d20 con 0x7faf881096e0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf80005020 con 0x7faf881096e0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf881a1be0 con 0x7faf881096e0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf800116d0 con 0x7faf881096e0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.761+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf881a20b0 con 0x7faf881096e0 2024-03-19T02:25:07.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.763+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf881084a0 con 0x7faf881096e0 2024-03-19T02:25:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.763+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7faf8000cb20 con 0x7faf881096e0 2024-03-19T02:25:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.764+0000 7faf857fa640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 0x7faf700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.764+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7faf800ce970 con 0x7faf881096e0 2024-03-19T02:25:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.764+0000 7faf87fff640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 0x7faf700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:07.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.765+0000 7faf87fff640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 0x7faf700a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf8819ab80 tx=0x7faf7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:07.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:07.770+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf80016030 con 0x7faf881096e0 2024-03-19T02:25:07.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:25:07.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:25:07.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:25:07.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:07.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:07.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:07.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:07.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:25:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:25:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:25:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:08.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:08.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:08.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3173788915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:25:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:25:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:25:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:25:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.013+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7faf88062490 con 0x7faf881096e0 2024-03-19T02:25:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.649+0000 7faf857fa640 1 -- 172.21.15.116:0/23938154 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7faf8009b570 con 0x7faf881096e0 2024-03-19T02:25:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 msgr2=0x7faf700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 0x7faf700a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf8819ab80 tx=0x7faf7800b410 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 msgr2=0x7faf881a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e940 tx=0x7faf8000ee00 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 shutdown_connections 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faf700a2800 0x7faf700a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf881096e0 0x7faf881a1250 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf880757e0 0x7faf8819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 --2- 172.21.15.116:0/23938154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf88074200 0x7faf88199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 >> 172.21.15.116:0/23938154 conn(0x7faf880fcd80 msgr2=0x7faf88105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 shutdown_connections 2024-03-19T02:25:08.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.651+0000 7faf8e281640 1 -- 172.21.15.116:0/23938154 wait complete. 2024-03-19T02:25:08.668 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-19T02:25:08.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-19T02:25:08.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.798+0000 7fdd6c313640 1 -- 172.21.15.116:0/3680675184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 msgr2=0x7fdd64101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.798+0000 7fdd6c313640 1 --2- 172.21.15.116:0/3680675184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd64101d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009900 tx=0x7fdd5402bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- 172.21.15.116:0/3680675184 shutdown_connections 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- 172.21.15.116:0/3680675184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd641022d0 0x7fdd64108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- 172.21.15.116:0/3680675184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd64101d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- 172.21.15.116:0/3680675184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd64100720 0x7fdd64100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- 172.21.15.116:0/3680675184 >> 172.21.15.116:0/3680675184 conn(0x7fdd640fbcb0 msgr2=0x7fdd640fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- 172.21.15.116:0/3680675184 shutdown_connections 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- 172.21.15.116:0/3680675184 wait complete. 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 Processor -- start 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- start start 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 0x7fdd64069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:08.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd641022d0 0x7fdd6419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410c250 con 0x7fdd64101910 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410c0d0 con 0x7fdd641022d0 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd6c313640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6410c3d0 con 0x7fdd64100720 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.799+0000 7fdd698ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd698ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59286/0 (socket says 172.21.15.116:59286) 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd698ae640 1 -- 172.21.15.116:0/26512731 learned_addr learned my addr 172.21.15.116:0/26512731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a8b0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd641022d0 0x7fdd6419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 0x7fdd64069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 -- 172.21.15.116:0/26512731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd641022d0 msgr2=0x7fdd6419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd641022d0 0x7fdd6419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 -- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 msgr2=0x7fdd6419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:08.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 -- 172.21.15.116:0/26512731 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6419f6a0 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd698ae640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6a0af640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 0x7fdd64069550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000e9c0 tx=0x7fdd6000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000ccc0 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6419f990 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd600035f0 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6000ce60 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.800+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd6419fee0 con 0x7fdd64100720 2024-03-19T02:25:08.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.801+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd64107b80 con 0x7fdd64100720 2024-03-19T02:25:08.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.802+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fdd600037a0 con 0x7fdd64100720 2024-03-19T02:25:08.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.803+0000 7fdd5b7fe640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 0x7fdd500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:08.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.803+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fdd600ced20 con 0x7fdd64100720 2024-03-19T02:25:08.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.803+0000 7fdd698ae640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 0x7fdd500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:08.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.803+0000 7fdd698ae640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 0x7fdd500a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd640692c0 tx=0x7fdd5402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:08.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:08.807+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd6009bf10 con 0x7fdd64100720 2024-03-19T02:25:08.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: pgmap v283: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:08.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:25:08.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:25:08.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:25:08.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:25:08.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:08.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:08 smithi162 ceph-mon[35761]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: pgmap v283: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:08 smithi154 ceph-mon[35693]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: pgmap v283: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/314738237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:08 smithi116 ceph-mon[32263]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:25:09.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.017+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fdd6402d1f0 con 0x7fdd64100720 2024-03-19T02:25:09.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.191+0000 7fdd5b7fe640 1 -- 172.21.15.116:0/26512731 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fdd6009b810 con 0x7fdd64100720 2024-03-19T02:25:09.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 msgr2=0x7fdd500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:09.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 0x7fdd500a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd640692c0 tx=0x7fdd5402c7e0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 msgr2=0x7fdd64069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 0x7fdd64069550 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000e9c0 tx=0x7fdd6000ee80 comp rx=0 tx=0).stop 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 shutdown_connections 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdd500a2800 0x7fdd500a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd641022d0 0x7fdd6419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd64101910 0x7fdd6419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 --2- 172.21.15.116:0/26512731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd64100720 0x7fdd64069550 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 >> 172.21.15.116:0/26512731 conn(0x7fdd640fbcb0 msgr2=0x7fdd64104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:09.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 shutdown_connections 2024-03-19T02:25:09.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.194+0000 7fdd6c313640 1 -- 172.21.15.116:0/26512731 wait complete. 2024-03-19T02:25:09.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T02:25:09.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- 172.21.15.116:0/1366674738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb398101070 msgr2=0x7fb3981014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 --2- 172.21.15.116:0/1366674738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb398101070 0x7fb3981014f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb384009900 tx=0x7fb38402bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- 172.21.15.116:0/1366674738 shutdown_connections 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 --2- 172.21.15.116:0/1366674738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101a30 0x7fb398108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 --2- 172.21.15.116:0/1366674738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb398101070 0x7fb3981014f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 --2- 172.21.15.116:0/1366674738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3980ffe90 0x7fb3981002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- 172.21.15.116:0/1366674738 >> 172.21.15.116:0/1366674738 conn(0x7fb3980fb450 msgr2=0x7fb3980fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- 172.21.15.116:0/1366674738 shutdown_connections 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- 172.21.15.116:0/1366674738 wait complete. 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 Processor -- start 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.320+0000 7fb39ece8640 1 -- start start 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:09.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101070 0x7fb3981980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb398101a30 0x7fb39819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb39810c310 con 0x7fb3980ffe90 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb39810c190 con 0x7fb398101a30 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb39810c490 con 0x7fb398101070 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59308/0 (socket says 172.21.15.116:59308) 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 -- 172.21.15.116:0/4170101269 learned_addr learned my addr 172.21.15.116:0/4170101269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb397fff640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101070 0x7fb3981980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 -- 172.21.15.116:0/4170101269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101070 msgr2=0x7fb3981980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101070 0x7fb3981980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 -- 172.21.15.116:0/4170101269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb398101a30 msgr2=0x7fb39819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb398101a30 0x7fb39819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 -- 172.21.15.116:0/4170101269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39819f790 con 0x7fb3980ffe90 2024-03-19T02:25:09.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ca84640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d770 tx=0x7fb38c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c017070 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb38c009c30 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39819fa80 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c003d00 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.321+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3981a0050 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.322+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3981a0280 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.323+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb38c003ea0 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.324+0000 7fb395ffb640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.324+0000 7fb397fff640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.324+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fb38c0ce870 con 0x7fb3980ffe90 2024-03-19T02:25:09.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.324+0000 7fb397fff640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 0x7fb3880a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb384036040 tx=0x7fb384000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:09.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.328+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb38c09b8e0 con 0x7fb3980ffe90 2024-03-19T02:25:09.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:09.504+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb3981a0a50 con 0x7fb3980ffe90 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/26512731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:25:09.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:09 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:25:10.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/26512731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:10.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:09 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/23938154' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.010 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/26512731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:25:10.011 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:09 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:25:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: pgmap v287: 97 pgs: 10 creating+peering, 71 unknown, 16 active+clean; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-19T02:25:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:active 2024-03-19T02:25:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:10.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:10.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:25:10.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:10 smithi162 ceph-mon[35761]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: pgmap v287: 97 pgs: 10 creating+peering, 71 unknown, 16 active+clean; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:active 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:25:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:10 smithi154 ceph-mon[35693]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: pgmap v287: 97 pgs: 10 creating+peering, 71 unknown, 16 active+clean; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] up:active 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:25:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:10 smithi116 ceph-mon[32263]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:25:11.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.192+0000 7fb395ffb640 1 -- 172.21.15.116:0/4170101269 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fb38c009de0 con 0x7fb3980ffe90 2024-03-19T02:25:11.193 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' created 2024-03-19T02:25:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 msgr2=0x7fb3880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 0x7fb3880a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb384036040 tx=0x7fb384000bc0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 msgr2=0x7fb398197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d770 tx=0x7fb38c00dc30 comp rx=0 tx=0).stop 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 shutdown_connections 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3880a2800 0x7fb3880a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb398101a30 0x7fb39819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb398101070 0x7fb3981980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 --2- 172.21.15.116:0/4170101269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3980ffe90 0x7fb398197b70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 >> 172.21.15.116:0/4170101269 conn(0x7fb3980fb450 msgr2=0x7fb3980fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 shutdown_connections 2024-03-19T02:25:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.195+0000 7fb39ece8640 1 -- 172.21.15.116:0/4170101269 wait complete. 2024-03-19T02:25:11.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-19T02:25:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 -- 172.21.15.116:0/4268833612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 msgr2=0x7f25e0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 --2- 172.21.15.116:0/4268833612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0101db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009900 tx=0x7f25d002bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 -- 172.21.15.116:0/4268833612 shutdown_connections 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 --2- 172.21.15.116:0/4268833612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 0x7f25e0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 --2- 172.21.15.116:0/4268833612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0101db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 --2- 172.21.15.116:0/4268833612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 0x7f25e0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 -- 172.21.15.116:0/4268833612 >> 172.21.15.116:0/4268833612 conn(0x7f25e00fbcd0 msgr2=0x7f25e00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 -- 172.21.15.116:0/4268833612 shutdown_connections 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.325+0000 7f25e81ab640 1 -- 172.21.15.116:0/4268833612 wait complete. 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 Processor -- start 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 -- start start 2024-03-19T02:25:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 0x7f25e0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 0x7f25e019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25e010c2e0 con 0x7f25e0101930 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25e010c160 con 0x7f25e01022f0 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e81ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25e010c460 con 0x7f25e0100740 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59324/0 (socket says 172.21.15.116:59324) 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 -- 172.21.15.116:0/1653189124 learned_addr learned my addr 172.21.15.116:0/1653189124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e6748640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 0x7f25e019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5f47640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 0x7f25e0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 -- 172.21.15.116:0/1653189124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 msgr2=0x7f25e0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 0x7f25e0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 -- 172.21.15.116:0/1653189124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 msgr2=0x7f25e019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 0x7f25e019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 -- 172.21.15.116:0/1653189124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e019f750 con 0x7f25e0101930 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25e5746640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25d0006010 tx=0x7f25d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.326+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0043020 con 0x7f25e0101930 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.327+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d0004440 con 0x7f25e0101930 2024-03-19T02:25:11.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.327+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e019f9e0 con 0x7f25e0101930 2024-03-19T02:25:11.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.327+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0003bf0 con 0x7f25e0101930 2024-03-19T02:25:11.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.327+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e019ffb0 con 0x7f25e0101930 2024-03-19T02:25:11.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.328+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e01a0180 con 0x7f25e0101930 2024-03-19T02:25:11.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.329+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f25d0046020 con 0x7f25e0101930 2024-03-19T02:25:11.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.330+0000 7f25d6ffd640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 0x7f25cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:11.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.330+0000 7f25e5f47640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 0x7f25cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:11.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.330+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f25d00f15d0 con 0x7f25e0101930 2024-03-19T02:25:11.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.330+0000 7f25e5f47640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 0x7f25cc0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25e0101790 tx=0x7f25dc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:11.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.334+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25d004d020 con 0x7f25e0101930 2024-03-19T02:25:11.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:11.508+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f25e01a0350 con 0x7f25e0101930 2024-03-19T02:25:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.199+0000 7f25d6ffd640 1 -- 172.21.15.116:0/1653189124 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f25d00be450 con 0x7f25e0101930 2024-03-19T02:25:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:set pool 4 allow_ec_overwrites to true 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 msgr2=0x7f25cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 0x7f25cc0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25e0101790 tx=0x7f25dc008040 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 msgr2=0x7f25e0198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25d0006010 tx=0x7f25d0004040 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 shutdown_connections 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25cc0a2800 0x7f25cc0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e01022f0 0x7f25e019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101930 0x7f25e0198070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 --2- 172.21.15.116:0/1653189124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25e0100740 0x7f25e0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 >> 172.21.15.116:0/1653189124 conn(0x7f25e00fbcd0 msgr2=0x7f25e0069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 shutdown_connections 2024-03-19T02:25:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.201+0000 7f25e81ab640 1 -- 172.21.15.116:0/1653189124 wait complete. 2024-03-19T02:25:12.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-19T02:25:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- 172.21.15.116:0/1189121508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101830 msgr2=0x7f3998101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- 172.21.15.116:0/1189121508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101830 0x7f3998101c50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3988009900 tx=0x7f3988030040 comp rx=0 tx=0).stop 2024-03-19T02:25:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- 172.21.15.116:0/1189121508 shutdown_connections 2024-03-19T02:25:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- 172.21.15.116:0/1189121508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39981033e0 0x7f3998109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- 172.21.15.116:0/1189121508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 0x7f3998102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- 172.21.15.116:0/1189121508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3998101830 0x7f3998101c50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- 172.21.15.116:0/1189121508 >> 172.21.15.116:0/1189121508 conn(0x7f39980fcdc0 msgr2=0x7f39980ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- 172.21.15.116:0/1189121508 shutdown_connections 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- 172.21.15.116:0/1189121508 wait complete. 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 Processor -- start 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- start start 2024-03-19T02:25:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3998101830 0x7f3998195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 0x7f3998195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f399810b1a0 con 0x7f39981033e0 2024-03-19T02:25:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.327+0000 7f399e44b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f399810b020 con 0x7f3998102a20 2024-03-19T02:25:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399e44b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f399810b320 con 0x7f3998101830 2024-03-19T02:25:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59338/0 (socket says 172.21.15.116:59338) 2024-03-19T02:25:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 -- 172.21.15.116:0/682097430 learned_addr learned my addr 172.21.15.116:0/682097430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f39977fe640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 0x7f3998195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f3997fff640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3998101830 0x7f3998195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 -- 172.21.15.116:0/682097430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3998101830 msgr2=0x7f3998195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3998101830 0x7f3998195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 -- 172.21.15.116:0/682097430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 msgr2=0x7f3998195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 0x7f3998195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 -- 172.21.15.116:0/682097430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399819d540 con 0x7f39981033e0 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399c9e8640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f399000b970 tx=0x7f399000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990019070 con 0x7f39981033e0 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3990014440 con 0x7f39981033e0 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39900075d0 con 0x7f39981033e0 2024-03-19T02:25:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399819d830 con 0x7f39981033e0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.328+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399819ddb0 con 0x7f39981033e0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.329+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3998108c20 con 0x7f39981033e0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.330+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f399001c070 con 0x7f39981033e0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.331+0000 7f39957fa640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 0x7f39800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.331+0000 7f3997fff640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 0x7f39800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.331+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f39900ce200 con 0x7f39981033e0 2024-03-19T02:25:12.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.331+0000 7f3997fff640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 0x7f39800a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3988005c40 tx=0x7f3988005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:12.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.336+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f399009aa30 con 0x7f39981033e0 2024-03-19T02:25:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:12 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:25:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:12 smithi154 ceph-mon[35693]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:25:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:12 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:25:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:12 smithi154 ceph-mon[35693]: pgmap v290: 129 pgs: 10 creating+peering, 57 unknown, 62 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:25:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:12 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:25:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:12 smithi116 ceph-mon[32263]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:25:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:12 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:25:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:12 smithi116 ceph-mon[32263]: pgmap v290: 129 pgs: 10 creating+peering, 57 unknown, 62 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:25:12.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:12.552+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f3998062420 con 0x7f39981033e0 2024-03-19T02:25:12.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:12 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4170101269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:25:12.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:12 smithi162 ceph-mon[35761]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:25:12.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:12 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:25:12.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:12 smithi162 ceph-mon[35761]: pgmap v290: 129 pgs: 10 creating+peering, 57 unknown, 62 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:25:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.200+0000 7f39957fa640 1 -- 172.21.15.116:0/682097430 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f399009ead0 con 0x7f39981033e0 2024-03-19T02:25:13.201 INFO:teuthology.orchestra.run.smithi116.stderr:added data pool 4 to fsmap 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 msgr2=0x7f39800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 0x7f39800a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3988005c40 tx=0x7f3988005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 msgr2=0x7f399819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f399000b970 tx=0x7f399000be30 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 shutdown_connections 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f39800a2800 0x7f39800a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39981033e0 0x7f399819ced0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3998102a20 0x7f3998195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 --2- 172.21.15.116:0/682097430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3998101830 0x7f3998195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 >> 172.21.15.116:0/682097430 conn(0x7f39980fcdc0 msgr2=0x7f3998105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 shutdown_connections 2024-03-19T02:25:13.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.203+0000 7f399e44b640 1 -- 172.21.15.116:0/682097430 wait complete. 2024-03-19T02:25:13.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 -- 172.21.15.116:0/2625568590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e4101930 msgr2=0x7ff1e4101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 --2- 172.21.15.116:0/2625568590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e4101930 0x7ff1e4101db0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009900 tx=0x7ff1cc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 -- 172.21.15.116:0/2625568590 shutdown_connections 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 --2- 172.21.15.116:0/2625568590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e41022f0 0x7ff1e4108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 --2- 172.21.15.116:0/2625568590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e4101930 0x7ff1e4101db0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 --2- 172.21.15.116:0/2625568590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 0x7ff1e4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 -- 172.21.15.116:0/2625568590 >> 172.21.15.116:0/2625568590 conn(0x7ff1e40fbcd0 msgr2=0x7ff1e40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 -- 172.21.15.116:0/2625568590 shutdown_connections 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 -- 172.21.15.116:0/2625568590 wait complete. 2024-03-19T02:25:13.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.320+0000 7ff1e9268640 1 Processor -- start 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 -- start start 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 0x7ff1e406a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 0x7ff1e419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e410c270 con 0x7ff1e41022f0 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e410c0f0 con 0x7ff1e4101930 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e9268640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e410c3f0 con 0x7ff1e4100740 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59354/0 (socket says 172.21.15.116:59354) 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 -- 172.21.15.116:0/3991049322 learned_addr learned my addr 172.21.15.116:0/3991049322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1da7fc640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 0x7ff1e419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e2ffd640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 0x7ff1e406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 -- 172.21.15.116:0/3991049322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 msgr2=0x7ff1e406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 0x7ff1e406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 -- 172.21.15.116:0/3991049322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 msgr2=0x7ff1e419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 0x7ff1e419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 -- 172.21.15.116:0/3991049322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e419f730 con 0x7ff1e41022f0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1da7fc640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 0x7ff1e419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e37fe640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00efd0 tx=0x7ff1dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.321+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc003b70 con 0x7ff1e41022f0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.322+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1dc003d10 con 0x7ff1e41022f0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.322+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e419f9c0 con 0x7ff1e41022f0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.322+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc010980 con 0x7ff1e41022f0 2024-03-19T02:25:13.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.322+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e419fe90 con 0x7ff1e41022f0 2024-03-19T02:25:13.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.323+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e4107b80 con 0x7ff1e41022f0 2024-03-19T02:25:13.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.323+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff1dc004020 con 0x7ff1e41022f0 2024-03-19T02:25:13.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.324+0000 7ff1e0ff9640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 0x7ff1d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.324+0000 7ff1e2ffd640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 0x7ff1d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.324+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff1dc0cf100 con 0x7ff1e41022f0 2024-03-19T02:25:13.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.324+0000 7ff1e2ffd640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 0x7ff1d00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005fd0 tx=0x7ff1d400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.328+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1dc09bf60 con 0x7ff1e41022f0 2024-03-19T02:25:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:13 smithi154 ceph-mon[35693]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:13 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:25:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:13 smithi154 ceph-mon[35693]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:25:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:13 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:25:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:13 smithi116 ceph-mon[32263]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:13 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:25:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:13 smithi116 ceph-mon[32263]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:25:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:13 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:25:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.543+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1e4062420 con 0x7ff1e41022f0 2024-03-19T02:25:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.544+0000 7ff1e0ff9640 1 -- 172.21.15.116:0/3991049322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11815 (secure 0 0 0) 0x7ff1dc09b860 con 0x7ff1e41022f0 2024-03-19T02:25:13.545 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:13.545 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:12.554074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-03-19T02:25:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 msgr2=0x7ff1d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 0x7ff1d00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005fd0 tx=0x7ff1d400b390 comp rx=0 tx=0).stop 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 msgr2=0x7ff1e419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00efd0 tx=0x7ff1dc00c6a0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 shutdown_connections 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1d00a2800 0x7ff1d00a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41022f0 0x7ff1e419ee20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1e4101930 0x7ff1e419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 --2- 172.21.15.116:0/3991049322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1e4100740 0x7ff1e406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 >> 172.21.15.116:0/3991049322 conn(0x7ff1e40fbcd0 msgr2=0x7ff1e4104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 shutdown_connections 2024-03-19T02:25:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.546+0000 7ff1e9268640 1 -- 172.21.15.116:0/3991049322 wait complete. 2024-03-19T02:25:13.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:13.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 -- 172.21.15.116:0/2465055794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca88102360 msgr2=0x7fca88104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/2465055794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca88102360 0x7fca88104750 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fca78009900 tx=0x7fca7802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:13.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 -- 172.21.15.116:0/2465055794 shutdown_connections 2024-03-19T02:25:13.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/2465055794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca881075c0 0x7fca881099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/2465055794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88104c90 0x7fca88107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/2465055794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca88102360 0x7fca88104750 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 -- 172.21.15.116:0/2465055794 >> 172.21.15.116:0/2465055794 conn(0x7fca880fbcb0 msgr2=0x7fca880fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 -- 172.21.15.116:0/2465055794 shutdown_connections 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.659+0000 7fca8d8ea640 1 -- 172.21.15.116:0/2465055794 wait complete. 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 Processor -- start 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 -- start start 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 0x7fca88197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca88104c90 0x7fca88197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810d380 con 0x7fca881075c0 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810d200 con 0x7fca88102360 2024-03-19T02:25:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca8d8ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810d500 con 0x7fca88104c90 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59372/0 (socket says 172.21.15.116:59372) 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca86ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 0x7fca88197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 -- 172.21.15.116:0/4172330546 learned_addr learned my addr 172.21.15.116:0/4172330546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca867fc640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca88104c90 0x7fca88197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 -- 172.21.15.116:0/4172330546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca88104c90 msgr2=0x7fca88197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca88104c90 0x7fca88197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 -- 172.21.15.116:0/4172330546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 msgr2=0x7fca88197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 0x7fca88197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca877fe640 1 -- 172.21.15.116:0/4172330546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8819f6d0 con 0x7fca881075c0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.660+0000 7fca86ffd640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 0x7fca88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca877fe640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fca8000eaa0 tx=0x7fca8000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8000cdb0 con 0x7fca881075c0 2024-03-19T02:25:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca80005020 con 0x7fca881075c0 2024-03-19T02:25:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8819f9c0 con 0x7fca881075c0 2024-03-19T02:25:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca800116d0 con 0x7fca881075c0 2024-03-19T02:25:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.661+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8819ff10 con 0x7fca881075c0 2024-03-19T02:25:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.662+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca880fd7a0 con 0x7fca881075c0 2024-03-19T02:25:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.663+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fca800051d0 con 0x7fca881075c0 2024-03-19T02:25:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.663+0000 7fca8c8e8640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 0x7fca700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.663+0000 7fca86ffd640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 0x7fca700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.664+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fca800cf150 con 0x7fca881075c0 2024-03-19T02:25:13.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.664+0000 7fca86ffd640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 0x7fca700a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca78009230 tx=0x7fca78005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.667+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca80016030 con 0x7fca881075c0 2024-03-19T02:25:13.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:13 smithi162 ceph-mon[35761]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:25:13.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:13 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1653189124' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:25:13.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:13 smithi162 ceph-mon[35761]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:25:13.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:13 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:25:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.843+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca8802d440 con 0x7fca881075c0 2024-03-19T02:25:13.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.844+0000 7fca8c8e8640 1 -- 172.21.15.116:0/4172330546 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7fca8009b960 con 0x7fca881075c0 2024-03-19T02:25:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:13.845 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 msgr2=0x7fca700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 0x7fca700a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca78009230 tx=0x7fca78005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 msgr2=0x7fca8819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fca8000eaa0 tx=0x7fca8000ef60 comp rx=0 tx=0).stop 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 shutdown_connections 2024-03-19T02:25:13.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca700a2800 0x7fca700a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca881075c0 0x7fca8819f060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca88104c90 0x7fca88197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 --2- 172.21.15.116:0/4172330546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca88102360 0x7fca88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.846+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 >> 172.21.15.116:0/4172330546 conn(0x7fca880fbcb0 msgr2=0x7fca88102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.847+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 shutdown_connections 2024-03-19T02:25:13.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.847+0000 7fca8d8ea640 1 -- 172.21.15.116:0/4172330546 wait complete. 2024-03-19T02:25:13.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:13.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.959+0000 7f4c6dae9640 1 -- 172.21.15.116:0/23519377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 msgr2=0x7f4c68107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.959+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/23519377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68107080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009900 tx=0x7f4c5c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- 172.21.15.116:0/23519377 shutdown_connections 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/23519377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c681075c0 0x7f4c681099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/23519377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68107080 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/23519377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 0x7f4c68104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- 172.21.15.116:0/23519377 >> 172.21.15.116:0/23519377 conn(0x7f4c680fbcd0 msgr2=0x7f4c680fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- 172.21.15.116:0/23519377 shutdown_connections 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- 172.21.15.116:0/23519377 wait complete. 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 Processor -- start 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- start start 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 0x7f4c68197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c681075c0 0x7f4c6819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810d390 con 0x7f4c68104c90 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810d210 con 0x7f4c68102360 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.960+0000 7f4c6dae9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810d510 con 0x7f4c681075c0 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59396/0 (socket says 172.21.15.116:59396) 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 -- 172.21.15.116:0/2185116574 learned_addr learned my addr 172.21.15.116:0/2185116574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c677fe640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 0x7f4c68197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 -- 172.21.15.116:0/2185116574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c681075c0 msgr2=0x7f4c6819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c681075c0 0x7f4c6819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 -- 172.21.15.116:0/2185116574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 msgr2=0x7f4c68197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 0x7f4c68197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 -- 172.21.15.116:0/2185116574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6819f740 con 0x7f4c68104c90 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c66ffd640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c000c00 tx=0x7f4c5c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c043020 con 0x7f4c68104c90 2024-03-19T02:25:13.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6819f9d0 con 0x7f4c68104c90 2024-03-19T02:25:13.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c5c004350 con 0x7f4c68104c90 2024-03-19T02:25:13.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c003b80 con 0x7f4c68104c90 2024-03-19T02:25:13.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.961+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c6819ff20 con 0x7f4c68104c90 2024-03-19T02:25:13.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.962+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c680fd7c0 con 0x7f4c68104c90 2024-03-19T02:25:13.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.963+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4c5c046020 con 0x7f4c68104c90 2024-03-19T02:25:13.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.964+0000 7f4c64ff9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 0x7f4c500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:13.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.964+0000 7f4c677fe640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 0x7f4c500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:13.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.964+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4c5c0f1230 con 0x7f4c68104c90 2024-03-19T02:25:13.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.964+0000 7f4c677fe640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 0x7f4c500a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4c580095f0 tx=0x7f4c58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:13.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:13.967+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4c5c04d020 con 0x7f4c68104c90 2024-03-19T02:25:14.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.143+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c6802d290 con 0x7f4c68104c90 2024-03-19T02:25:14.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.145+0000 7f4c64ff9640 1 -- 172.21.15.116:0/2185116574 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f4c5c0be130 con 0x7f4c68104c90 2024-03-19T02:25:14.145 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:14.146 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:14.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 msgr2=0x7f4c500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 0x7f4c500a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4c580095f0 tx=0x7f4c58005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:14.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 msgr2=0x7f4c68198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c000c00 tx=0x7f4c5c005e00 comp rx=0 tx=0).stop 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 shutdown_connections 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4c500a2800 0x7f4c500a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c681075c0 0x7f4c6819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c68104c90 0x7f4c68198060 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 --2- 172.21.15.116:0/2185116574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c68102360 0x7f4c68197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 >> 172.21.15.116:0/2185116574 conn(0x7f4c680fbcd0 msgr2=0x7f4c68104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 shutdown_connections 2024-03-19T02:25:14.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.147+0000 7f4c6dae9640 1 -- 172.21.15.116:0/2185116574 wait complete. 2024-03-19T02:25:14.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- 172.21.15.116:0/4146953961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d70102340 msgr2=0x7f1d70104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 --2- 172.21.15.116:0/4146953961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d70102340 0x7f1d70104730 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009900 tx=0x7f1d6002bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- 172.21.15.116:0/4146953961 shutdown_connections 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 --2- 172.21.15.116:0/4146953961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d701075a0 0x7f1d70109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 --2- 172.21.15.116:0/4146953961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 --2- 172.21.15.116:0/4146953961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d70102340 0x7f1d70104730 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- 172.21.15.116:0/4146953961 >> 172.21.15.116:0/4146953961 conn(0x7f1d700fbcd0 msgr2=0x7f1d700fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- 172.21.15.116:0/4146953961 shutdown_connections 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- 172.21.15.116:0/4146953961 wait complete. 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 Processor -- start 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.278+0000 7f1d767e6640 1 -- start start 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 0x7f1d70197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7010d3b0 con 0x7f1d701075a0 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7010d230 con 0x7f1d70102340 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7010d530 con 0x7f1d70104c70 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59410/0 (socket says 172.21.15.116:59410) 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 -- 172.21.15.116:0/3355568377 learned_addr learned my addr 172.21.15.116:0/3355568377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6ffff640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 0x7f1d70197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6f7fe640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 -- 172.21.15.116:0/3355568377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 msgr2=0x7f1d70198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 -- 172.21.15.116:0/3355568377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 msgr2=0x7f1d70197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 0x7f1d70197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 -- 172.21.15.116:0/3355568377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7019f700 con 0x7f1d701075a0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6f7fe640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6ffff640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 0x7f1d70197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d74d83640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800eb10 tx=0x7f1d6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68004b20 con 0x7f1d701075a0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7019f9f0 con 0x7f1d701075a0 2024-03-19T02:25:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d68004cc0 con 0x7f1d701075a0 2024-03-19T02:25:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.279+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68010690 con 0x7f1d701075a0 2024-03-19T02:25:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.280+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7019fef0 con 0x7f1d701075a0 2024-03-19T02:25:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.281+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d700fd7c0 con 0x7f1d701075a0 2024-03-19T02:25:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.281+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1d68005020 con 0x7f1d701075a0 2024-03-19T02:25:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.282+0000 7f1d6d7fa640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 0x7f1d580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.282+0000 7f1d6ffff640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 0x7f1d580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.282+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1d680cf2e0 con 0x7f1d701075a0 2024-03-19T02:25:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.282+0000 7f1d6ffff640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 0x7f1d580a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009230 tx=0x7f1d60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.286+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d6809c140 con 0x7f1d701075a0 2024-03-19T02:25:14.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.498+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d70062420 con 0x7f1d701075a0 2024-03-19T02:25:14.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.498+0000 7f1d6d7fa640 1 -- 172.21.15.116:0/3355568377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11815 (secure 0 0 0) 0x7f1d680afe20 con 0x7f1d701075a0 2024-03-19T02:25:14.499 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:14.499 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:12.554074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-03-19T02:25:14.500 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-03-19T02:25:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 msgr2=0x7f1d580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 0x7f1d580a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009230 tx=0x7f1d60005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 msgr2=0x7f1d7019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1d6800eb10 tx=0x7f1d6800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 shutdown_connections 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1d580a2800 0x7f1d580a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.500+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d701075a0 0x7f1d7019f090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.501+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d70104c70 0x7f1d70198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.501+0000 7f1d767e6640 1 --2- 172.21.15.116:0/3355568377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d70102340 0x7f1d70197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.501+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 >> 172.21.15.116:0/3355568377 conn(0x7f1d700fbcd0 msgr2=0x7f1d70102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.501+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 shutdown_connections 2024-03-19T02:25:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.501+0000 7f1d767e6640 1 -- 172.21.15.116:0/3355568377 wait complete. 2024-03-19T02:25:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3991049322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: pgmap v293: 129 pgs: 5 creating+peering, 24 unknown, 100 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4172330546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2185116574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3991049322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: pgmap v293: 129 pgs: 5 creating+peering, 24 unknown, 100 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4172330546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2185116574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.512 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:14.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- 172.21.15.116:0/3159030893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a4100740 msgr2=0x7f20a4100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- 172.21.15.116:0/3159030893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a4100740 0x7f20a4100b60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2094009900 tx=0x7f209402bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- 172.21.15.116:0/3159030893 shutdown_connections 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- 172.21.15.116:0/3159030893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a41022f0 0x7f20a4108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- 172.21.15.116:0/3159030893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4101930 0x7f20a4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- 172.21.15.116:0/3159030893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a4100740 0x7f20a4100b60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- 172.21.15.116:0/3159030893 >> 172.21.15.116:0/3159030893 conn(0x7f20a40fbcd0 msgr2=0x7f20a40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- 172.21.15.116:0/3159030893 shutdown_connections 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- 172.21.15.116:0/3159030893 wait complete. 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 Processor -- start 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- start start 2024-03-19T02:25:14.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4100740 0x7f20a406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a4101930 0x7f20a419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20a410c240 con 0x7f20a41022f0 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20a410c0c0 con 0x7f20a4100740 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.614+0000 7f20a935b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20a410c3c0 con 0x7f20a4101930 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59428/0 (socket says 172.21.15.116:59428) 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 -- 172.21.15.116:0/323956566 learned_addr learned my addr 172.21.15.116:0/323956566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a2ffd640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4100740 0x7f20a406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f209bfff640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a4101930 0x7f20a419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 -- 172.21.15.116:0/323956566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a4101930 msgr2=0x7f20a419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a4101930 0x7f20a419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 -- 172.21.15.116:0/323956566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4100740 msgr2=0x7f20a406aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4100740 0x7f20a406aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 -- 172.21.15.116:0/323956566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a419f6a0 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a37fe640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f209c00cce0 tx=0x7f209c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c00f450 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f209c00faa0 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a419f990 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c018b70 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.615+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a419fe90 con 0x7f20a41022f0 2024-03-19T02:25:14.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.616+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a4107a80 con 0x7f20a41022f0 2024-03-19T02:25:14.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.617+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f209c018d10 con 0x7f20a41022f0 2024-03-19T02:25:14.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.618+0000 7f20a0ff9640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.618+0000 7f20a2ffd640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.618+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f209c0cf190 con 0x7f20a41022f0 2024-03-19T02:25:14.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.618+0000 7f20a2ffd640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 0x7f20900a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2094005ab0 tx=0x7f2094005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.622+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f209c014030 con 0x7f20a41022f0 2024-03-19T02:25:14.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/682097430' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:25:14.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:25:14.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:14.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3991049322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: pgmap v293: 129 pgs: 5 creating+peering, 24 unknown, 100 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-19T02:25:14.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4172330546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:14 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2185116574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.797+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20a402d3f0 con 0x7f20a41022f0 2024-03-19T02:25:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.799+0000 7f20a0ff9640 1 -- 172.21.15.116:0/323956566 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f209c09c080 con 0x7f20a41022f0 2024-03-19T02:25:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:14.800 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 msgr2=0x7f20900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 0x7f20900a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2094005ab0 tx=0x7f2094005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 msgr2=0x7f20a419ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f209c00cce0 tx=0x7f209c00a620 comp rx=0 tx=0).stop 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 shutdown_connections 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20a41022f0 0x7f20a419ede0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20a4101930 0x7f20a419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 --2- 172.21.15.116:0/323956566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20a4100740 0x7f20a406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 >> 172.21.15.116:0/323956566 conn(0x7f20a40fbcd0 msgr2=0x7f20a4104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 shutdown_connections 2024-03-19T02:25:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.801+0000 7f20a935b640 1 -- 172.21.15.116:0/323956566 wait complete. 2024-03-19T02:25:14.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T02:25:08.015174+0000', 'modified': '2024-03-19T02:25:12.554074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/4009274564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 4009274564}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 4009274564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24494, 'qdb_cluster': [24494]} 2024-03-19T02:25:14.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T02:25:14.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- 172.21.15.116:0/4181165185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 msgr2=0x7f41e0109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- 172.21.15.116:0/4181165185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e0109950 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009900 tx=0x7f41d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- 172.21.15.116:0/4181165185 shutdown_connections 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- 172.21.15.116:0/4181165185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e0109950 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- 172.21.15.116:0/4181165185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- 172.21.15.116:0/4181165185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e01046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- 172.21.15.116:0/4181165185 >> 172.21.15.116:0/4181165185 conn(0x7f41e00fbcb0 msgr2=0x7f41e00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- 172.21.15.116:0/4181165185 shutdown_connections 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- 172.21.15.116:0/4181165185 wait complete. 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 Processor -- start 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- start start 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41e010d350 con 0x7f41e0107540 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41e010d1d0 con 0x7f41e0104c10 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e75b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41e010d4d0 con 0x7f41e01022e0 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.916+0000 7f41e5b50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59438/0 (socket says 172.21.15.116:59438) 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 -- 172.21.15.116:0/920198627 learned_addr learned my addr 172.21.15.116:0/920198627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e534f640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 -- 172.21.15.116:0/920198627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 msgr2=0x7f41e0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 -- 172.21.15.116:0/920198627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 msgr2=0x7f41e0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 -- 172.21.15.116:0/920198627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e019f680 con 0x7f41e0107540 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e534f640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e5b50640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009230 tx=0x7f41d80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41d67fc640 1 -- 172.21.15.116:0/920198627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8043020 con 0x7f41e0107540 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e019f910 con 0x7f41e0107540 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41d67fc640 1 -- 172.21.15.116:0/920198627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d80044c0 con 0x7f41e0107540 2024-03-19T02:25:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41d67fc640 1 -- 172.21.15.116:0/920198627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8003bc0 con 0x7f41e0107540 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.917+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e019feb0 con 0x7f41e0107540 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.919+0000 7f41d67fc640 1 -- 172.21.15.116:0/920198627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f41d8046020 con 0x7f41e0107540 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 Processor -- start 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 -- start start 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41d67fc640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 0x7f41cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56ac64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 0x55dd56ac8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd56a25de0 con 0x55dd56ac19e0 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd569cec80 con 0x55dd56ac6b50 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41ea196740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd56a21e80 con 0x55dd56ac0ff0 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 0x7f41cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41d67fc640 1 -- 172.21.15.116:0/920198627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f41d80f0fc0 con 0x7f41e0107540 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e4b4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56ac64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e4b4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56ac64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59452/0 (socket says 172.21.15.116:59452) 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e4b4e640 1 -- 172.21.15.116:0/1183035997 learned_addr learned my addr 172.21.15.116:0/1183035997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e5b50640 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 0x55dd56ac8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e534f640 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e5b50640 1 -- 172.21.15.116:0/1183035997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac0ff0 msgr2=0x55dd56ac1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.920+0000 7f41e5b50640 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41e5b50640 1 -- 172.21.15.116:0/1183035997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 msgr2=0x55dd56ac64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41e5b50640 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56ac64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41e5b50640 1 -- 172.21.15.116:0/1183035997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd569e0fb0 con 0x55dd56ac6b50 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 0x7f41cc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009fd0 tx=0x7f41d0009f20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41e5b50640 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 0x55dd56ac8fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41d80eebf0 tx=0x7f41d80bd4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c41be381f57ea33a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41cb7fe640 1 -- 172.21.15.116:0/1183035997 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d80a1450 con 0x55dd56ac6b50 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41cb7fe640 1 -- 172.21.15.116:0/1183035997 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d80bde20 con 0x55dd56ac6b50 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.921+0000 7f41cb7fe640 1 -- 172.21.15.116:0/1183035997 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d80452c0 con 0x55dd56ac6b50 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- 172.21.15.116:0/1183035997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 msgr2=0x55dd56ac8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 0x55dd56ac8fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41d80eebf0 tx=0x7f41d80bd4f0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- 172.21.15.116:0/1183035997 shutdown_connections 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac6b50 0x55dd56ac8fe0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56ac64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- 172.21.15.116:0/1183035997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- 172.21.15.116:0/1183035997 >> 172.21.15.116:0/1183035997 conn(0x55dd56abc560 msgr2=0x55dd56abc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- 172.21.15.116:0/1183035997 shutdown_connections 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- 172.21.15.116:0/1183035997 wait complete. 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- start start 2024-03-19T02:25:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 Processor -- start 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ac98c0 con 0x55dd56ac19e0 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd569e1fc0 con 0x55dd56ac0ff0 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ac9a40 con 0x55dd56ac6b50 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41ea196740 10 client.0 _ll_register_callbacks cb 0x55dd56a1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:25:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41e4b4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41e4b4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59456/0 (socket says 172.21.15.116:59456) 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41e4b4e640 1 -- 172.21.15.116:0/3616873328 learned_addr learned my addr 172.21.15.116:0/3616873328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.922+0000 7f41e534f640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 -- 172.21.15.116:0/3616873328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 msgr2=0x55dd56ad27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 -- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac0ff0 msgr2=0x55dd56acb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 -- 172.21.15.116:0/3616873328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5673a2d0 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005870 tx=0x7f41d001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aba0f36af935108 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0024070 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d80bde20 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f41d0011db0 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f41d001edb0 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41ea196740 10 client.14760 fetch_fsmap learned FSMap version 17 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55dd568046b0 con 0x55dd56ac19e0 2024-03-19T02:25:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.923+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d80bde20 con 0x55dd56ac19e0 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f41d0029b50 con 0x55dd56ac19e0 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 10 client.14760 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 20 client.14760 populate_metadata read hostname 'smithi116' 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56ad81a0 0x55dd56ad9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 4 client.14760 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd56a33320 con 0x55dd56ad81a0 2024-03-19T02:25:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41e534f640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56ad81a0 0x55dd56ad9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41e534f640 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56ad81a0 0x55dd56ad9610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0403d97d2425fb server_cookie=79e00a6be01a0c7a in_seq=0 out_seq=0 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.924+0000 7f41c8ff9640 10 client.14760 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3616873328 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd56a33320 con 0x55dd56ad81a0 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41c8ff9640 10 client.14760 handle_command_reply: tid=0 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 client.14760 shutdown 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 msgr2=0x55dd56acb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005870 tx=0x7f41d001d000 comp rx=0 tx=0).stop 2024-03-19T02:25:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 shutdown_connections 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56ad81a0 0x55dd56ad9610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41c8ff9640 0 client.14760 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 --2- 172.21.15.116:0/3616873328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 >> 172.21.15.116:0/3616873328 conn(0x55dd56abc560 msgr2=0x55dd56abb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 shutdown_connections 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 1 -- 172.21.15.116:0/3616873328 wait complete. 2024-03-19T02:25:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.925+0000 7f41ea196740 20 client.14760 trim_cache size 0 max 16384 2024-03-19T02:25:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 Processor -- start 2024-03-19T02:25:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 -- start start 2024-03-19T02:25:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad7950 con 0x55dd56ac0ff0 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad8020 con 0x55dd56ac19e0 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41ea196740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad86f0 con 0x55dd56ac6b50 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59470/0 (socket says 172.21.15.116:59470) 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 -- 172.21.15.116:0/2309073489 learned_addr learned my addr 172.21.15.116:0/2309073489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 -- 172.21.15.116:0/2309073489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 msgr2=0x55dd56ad27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 -- 172.21.15.116:0/2309073489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 msgr2=0x55dd56acb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.943+0000 7f41e534f640 1 -- 172.21.15.116:0/2309073489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd56abd2d0 con 0x55dd56ac0ff0 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41e534f640 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55dd56ac2240 tx=0x7f41dc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd10ad5d2f8bc4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41caffd640 1 -- 172.21.15.116:0/2309073489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc020500 con 0x55dd56ac0ff0 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41caffd640 1 -- 172.21.15.116:0/2309073489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41dc024020 con 0x55dd56ac0ff0 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41caffd640 1 -- 172.21.15.116:0/2309073489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004660 con 0x55dd56ac0ff0 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 -- 172.21.15.116:0/2309073489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 msgr2=0x55dd56acb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55dd56ac2240 tx=0x7f41dc000f40 comp rx=0 tx=0).stop 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 -- 172.21.15.116:0/2309073489 shutdown_connections 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56ad27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56acb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 --2- 172.21.15.116:0/2309073489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56acb2d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 -- 172.21.15.116:0/2309073489 >> 172.21.15.116:0/2309073489 conn(0x55dd56abc560 msgr2=0x55dd56ad6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 -- 172.21.15.116:0/2309073489 shutdown_connections 2024-03-19T02:25:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.944+0000 7f41ea196740 1 -- 172.21.15.116:0/2309073489 wait complete. 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 -- start start 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 Processor -- start 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56ad27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56b3bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad9490 con 0x55dd56ac0ff0 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad8dc0 con 0x55dd56ac19e0 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd56ad9b60 con 0x55dd56ac6b50 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41ea196740 10 client.0 _ll_register_callbacks cb 0x7f41d00115e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59476/0 (socket says 172.21.15.116:59476) 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 -- 172.21.15.116:0/3703765818 learned_addr learned my addr 172.21.15.116:0/3703765818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56ad27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e5b50640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56b3bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 -- 172.21.15.116:0/3703765818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 msgr2=0x55dd56b3bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56b3bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 -- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 msgr2=0x55dd56ad27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56ad27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.945+0000 7f41e534f640 1 -- 172.21.15.116:0/3703765818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd56b3c730 con 0x55dd56ac0ff0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41e534f640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41dc002230 tx=0x7f41dc020660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5b410344a8b5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004c50 con 0x55dd56ac0ff0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f41dc004df0 con 0x55dd56ac0ff0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41dc024020 con 0x55dd56ac0ff0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f41dc0043e0 con 0x55dd56ac0ff0 2024-03-19T02:25:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 10 client.14766 fetch_fsmap learned FSMap version 17 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55dd56b3da30 con 0x55dd56ac0ff0 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41dc024020 con 0x55dd56ac0ff0 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f41dc00f040 con 0x55dd56ac0ff0 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 10 client.14766 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 20 client.14766 populate_metadata read hostname 'smithi116' 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56b42480 0x55dd56b41190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 4 client.14766 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "status"}) -- 0x55dd56a2d670 con 0x55dd56b42480 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.946+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56b42480 0x55dd56b41190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41c8ff9640 10 client.14766 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41e4b4e640 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56b42480 0x55dd56b41190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f70374e02be90b0 server_cookie=d6ff8059e48c950a in_seq=0 out_seq=0 2024-03-19T02:25:14.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41c8ff9640 1 -- 172.21.15.116:0/3703765818 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dd56a2d670 con 0x55dd56b42480 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41c8ff9640 10 client.14766 handle_command_reply: tid=0 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 client.14766 shutdown 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 msgr2=0x55dd56ac59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41dc002230 tx=0x7f41dc020660 comp rx=0 tx=0).stop 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 shutdown_connections 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd56b42480 0x55dd56b41190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd56ac6b50 0x55dd56b3bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd56ac19e0 0x55dd56ad27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41c8ff9640 0 client.14766 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 --2- 172.21.15.116:0/3703765818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd56ac0ff0 0x55dd56ac59f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 >> 172.21.15.116:0/3703765818 conn(0x55dd56abc560 msgr2=0x55dd56ad6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 shutdown_connections 2024-03-19T02:25:14.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 1 -- 172.21.15.116:0/3703765818 wait complete. 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.947+0000 7f41ea196740 20 client.14766 trim_cache size 0 max 16384 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24494, 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:25:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 6.2973793569999996, 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 17, 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 90, 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 10.541917423999999 2024-03-19T02:25:14.957 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 msgr2=0x7f41cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 0x7f41cc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009fd0 tx=0x7f41d0009f20 comp rx=0 tx=0).stop 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 msgr2=0x7f41e019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009230 tx=0x7f41d80040c0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 shutdown_connections 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f41cc0a2800 0x7f41cc0a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41e0107540 0x7f41e019f010 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41e0104c10 0x7f41e0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 --2- 172.21.15.116:0/920198627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e01022e0 0x7f41e0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 >> 172.21.15.116:0/920198627 conn(0x7f41e00fbcb0 msgr2=0x7f41e0102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 shutdown_connections 2024-03-19T02:25:14.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:14.948+0000 7f41e75b3640 1 -- 172.21.15.116:0/920198627 wait complete. 2024-03-19T02:25:14.960 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T02:25:14.960 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:14.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T02:25:15.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:25:14.604+0000 7fbadc802700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T02:25:15.197 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:15.461 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:15 smithi116 ceph-mon[32263]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:15.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:15 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:25:15.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:15 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3355568377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:15.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:15 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/323956566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:15 smithi154 ceph-mon[35693]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:15 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:25:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:15 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3355568377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:15 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/323956566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:15.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:15 smithi162 ceph-mon[35761]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:25:15.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:15 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:25:15.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:15 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3355568377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:15.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:15 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/323956566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:16 smithi154 ceph-mon[35693]: pgmap v294: 129 pgs: 5 creating+peering, 16 unknown, 108 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-19T02:25:16.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:16 smithi116 ceph-mon[32263]: pgmap v294: 129 pgs: 5 creating+peering, 16 unknown, 108 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-19T02:25:16.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:16 smithi162 ceph-mon[35761]: pgmap v294: 129 pgs: 5 creating+peering, 16 unknown, 108 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-19T02:25:17.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 -- 172.21.15.116:0/3065441873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 msgr2=0x7f6df810f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 --2- 172.21.15.116:0/3065441873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df810f160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6df000b0d0 tx=0x7f6df000b3e0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 -- 172.21.15.116:0/3065441873 shutdown_connections 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 --2- 172.21.15.116:0/3065441873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df810f160 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 --2- 172.21.15.116:0/3065441873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8108340 0x7f6df8108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 --2- 172.21.15.116:0/3065441873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8102360 0x7f6df81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.010+0000 7f6dff327700 1 -- 172.21.15.116:0/3065441873 >> 172.21.15.116:0/3065441873 conn(0x7f6df8079440 msgr2=0x7f6df8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.011+0000 7f6dff327700 1 -- 172.21.15.116:0/3065441873 shutdown_connections 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.011+0000 7f6dff327700 1 -- 172.21.15.116:0/3065441873 wait complete. 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.011+0000 7f6dff327700 1 Processor -- start 2024-03-19T02:25:17.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.011+0000 7f6dff327700 1 -- start start 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 0x7f6df819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8108340 0x7f6df8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6df8112470 con 0x7f6df8102d20 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81122f0 con 0x7f6df8102360 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81125f0 con 0x7f6df8108340 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:17.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59496/0 (socket says 172.21.15.116:59496) 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 -- 172.21.15.116:0/1828571936 learned_addr learned my addr 172.21.15.116:0/1828571936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfd0c3700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 0x7f6df819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfd8c4700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8108340 0x7f6df8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 -- 172.21.15.116:0/1828571936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8108340 msgr2=0x7f6df8198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8108340 0x7f6df8198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 -- 172.21.15.116:0/1828571936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 msgr2=0x7f6df819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 0x7f6df819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 -- 172.21.15.116:0/1828571936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df8198f70 con 0x7f6df8102d20 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfd0c3700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 0x7f6df819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dfc8c2700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c920 tx=0x7f6df400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4004a30 con 0x7f6df8102d20 2024-03-19T02:25:17.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6df40054d0 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df8199260 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4007c30 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.012+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df8074540 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.013+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df81039f0 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.014+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6df4004bd0 con 0x7f6df8102d20 2024-03-19T02:25:17.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.015+0000 7f6dee7fc700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 0x7f6de40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:17.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.015+0000 7f6dfd0c3700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 0x7f6de40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:17.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.015+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6df40cec00 con 0x7f6df8102d20 2024-03-19T02:25:17.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.015+0000 7f6dfd0c3700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 0x7f6de40a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009fd0 tx=0x7f6de8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:17.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.019+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6df4014030 con 0x7f6df8102d20 2024-03-19T02:25:17.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.257+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6df802dbb0 con 0x7f6df8102d20 2024-03-19T02:25:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.257+0000 7f6dee7fc700 1 -- 172.21.15.116:0/1828571936 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f6df409bae0 con 0x7f6df8102d20 2024-03-19T02:25:17.258 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.259+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 msgr2=0x7f6de40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.259+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 0x7f6de40a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009fd0 tx=0x7f6de8009260 comp rx=0 tx=0).stop 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.259+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 msgr2=0x7f6df819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.259+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c920 tx=0x7f6df400cc30 comp rx=0 tx=0).stop 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 shutdown_connections 2024-03-19T02:25:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6de40a2910 0x7f6de40a4dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6df8108340 0x7f6df8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6df8102360 0x7f6df819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 --2- 172.21.15.116:0/1828571936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df8102d20 0x7f6df819eb50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 >> 172.21.15.116:0/1828571936 conn(0x7f6df8079440 msgr2=0x7f6df8101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 shutdown_connections 2024-03-19T02:25:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:17.260+0000 7f6dff327700 1 -- 172.21.15.116:0/1828571936 wait complete. 2024-03-19T02:25:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:17 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1828571936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:25:17.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:17 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1828571936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:25:17.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:17 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1828571936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T02:25:17.959 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T02:25:17.960 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T02:25:17.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:17.960 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:25:17.976 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 41077sec preferred_lft 41077sec 2024-03-19T02:25:17.977 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-19T02:25:17.978 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:25:17.978 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T02:25:17.978 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:25:17.978 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:25:17.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-19T02:25:17.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-19T02:25:17.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-19T02:25:17.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T02:25:17.979 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:25:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:25:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:18.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:18.184 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T02:25:18.211 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-19T02:25:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:18.213 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-19T02:25:18.282 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-19T02:25:18.282 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T02:25:18.282 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T02:25:18.282 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T02:25:18.282 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T02:25:18.282 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:25:18.283 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:25:18.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T02:25:18.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T02:25:18.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T02:25:18.283 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:25:18.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:25:18.475 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:18 smithi116 ceph-mon[32263]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s wr, 3 op/s 2024-03-19T02:25:18.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:18.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:18.565 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T02:25:18.568 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:18 smithi162 ceph-mon[35761]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s wr, 3 op/s 2024-03-19T02:25:18.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:18.580 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-19T02:25:18.635 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:25:18.635 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:25:18.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:25:18.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-19T02:25:18.635 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:25:18.657 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:18 smithi154 ceph-mon[35693]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s wr, 3 op/s 2024-03-19T02:25:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:25:18.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:18.757 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-19T02:25:18.757 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:25:18.757 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:25:18.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-19T02:25:18.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T02:25:18.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-19T02:25:18.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-19T02:25:18.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-19T02:25:18.758 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:25:18.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:25:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:18.894 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:25:18.894 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:25:18.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-19T02:25:18.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T02:25:18.894 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:25:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:25:18.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:18.975 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-19T02:25:18.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:25:18.976 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:18.991 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T02:25:18.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:25:18.991 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:19.046 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-19T02:25:19.113 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:25:19.171 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-19T02:25:19.171 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-19T02:25:19.171 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-19T02:25:19.171 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-19T02:25:19.172 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:25:19.173 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/105c939f0328de1668168813dcbfde3e8eac64353870a865edc6e64881aafc43/merged 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/1ca6329da29ba9cc1ded52d1fda3fa7f4b282c5d6c2cc11a5dbe3497e0b413e8/merged 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:25:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/73a69ff94c6f368cf169b44a6d3a6ad7a62ffe99dd12cd8959b91ac1a9f9a7bf/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/776d3185faea4a5790e60d649621a6c756987875b5f3e54f8f1a6d47dbb4ae2c/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/101d9845f1265f94cc82f49e12f8fa48bbddbce39c7a3fc6d104394b56d90024/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/8a251936db2ed6b341d5b7f1c962f8d2f4cdee7a8f13f657a58c0aa0db2c59fa/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6ce183691416a01ad903095c2847c36f5ff96371767d62f353ea226d8b796e7a/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/69b48faf2550c0d2579eb5fdd431271f2fd54aa710d23c4fb41af23559ce35f1/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/52d53c064a0f81dee0cede53b46458f5c9011d803f5831b430af148b5061e918/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d9e070467a714fd4702b0b56c587243707ab26dc972ca1ea99c3ce18b3d975f5/merged 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:25:19.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:25:19.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:19.176 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-19T02:25:19.225 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T02:25:19.226 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-19T02:25:19.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-19T02:25:19.295 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:25:19.351 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-19T02:25:19.351 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-19T02:25:19.351 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-19T02:25:19.351 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-19T02:25:19.352 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:25:19.353 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/105c939f0328de1668168813dcbfde3e8eac64353870a865edc6e64881aafc43/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/1ca6329da29ba9cc1ded52d1fda3fa7f4b282c5d6c2cc11a5dbe3497e0b413e8/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/73a69ff94c6f368cf169b44a6d3a6ad7a62ffe99dd12cd8959b91ac1a9f9a7bf/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/776d3185faea4a5790e60d649621a6c756987875b5f3e54f8f1a6d47dbb4ae2c/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/101d9845f1265f94cc82f49e12f8fa48bbddbce39c7a3fc6d104394b56d90024/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/8a251936db2ed6b341d5b7f1c962f8d2f4cdee7a8f13f657a58c0aa0db2c59fa/merged 2024-03-19T02:25:19.354 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6ce183691416a01ad903095c2847c36f5ff96371767d62f353ea226d8b796e7a/merged 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/69b48faf2550c0d2579eb5fdd431271f2fd54aa710d23c4fb41af23559ce35f1/merged 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/52d53c064a0f81dee0cede53b46458f5c9011d803f5831b430af148b5061e918/merged 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d9e070467a714fd4702b0b56c587243707ab26dc972ca1ea99c3ce18b3d975f5/merged 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:25:19.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:19.355 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-19T02:25:19.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:2024-03-19T02:25:19.363+0000 7fbb3d373480 -1 init, newargv = 0x5568c47781f0 newargc=15 2024-03-19T02:25:19.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[73928]: starting ceph client 2024-03-19T02:25:19.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[73928]: starting fuse 2024-03-19T02:25:19.379 INFO:teuthology.orchestra.run.smithi116.stdout:119 2024-03-19T02:25:19.379 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-19T02:25:19.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-19T02:25:19.380 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-19T02:25:19.381 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.admin")) 2024-03-19T02:25:19.382 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-19T02:25:19.499 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.admin.73928.asok 2024-03-19T02:25:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:19.510 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73928.asok 2024-03-19T02:25:19.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:19.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73928.asok status 2024-03-19T02:25:19.647 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "admin", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "73928", 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:25:19.648 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 14778, 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 14778 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1183946056 2024-03-19T02:25:19.649 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1183946056 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.14778 192.168.0.1:0/1183946056", 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1183946056", 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 17, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 90, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-19T02:25:19.650 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-19T02:25:19.651 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-19T02:25:19.651 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:25:19.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:19.656 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-19T02:25:19.656 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-19T02:25:19.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:19.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:19.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:19.741 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-19T02:25:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T02:25:19.849 INFO:teuthology.orchestra.run.smithi116.stdout:# file: . 2024-03-19T02:25:19.849 INFO:teuthology.orchestra.run.smithi116.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-19T02:25:19.849 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:25:19.855 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:25:19.855 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:25:19.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:20.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- 172.21.15.116:0/2674935627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb01022f0 msgr2=0x7f5fb0108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- 172.21.15.116:0/2674935627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb01022f0 0x7f5fb0108b80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009ca0 tx=0x7f5fa802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- 172.21.15.116:0/2674935627 shutdown_connections 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- 172.21.15.116:0/2674935627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb01022f0 0x7f5fb0108b80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- 172.21.15.116:0/2674935627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0101930 0x7f5fb0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- 172.21.15.116:0/2674935627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb0100740 0x7f5fb0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- 172.21.15.116:0/2674935627 >> 172.21.15.116:0/2674935627 conn(0x7f5fb00fbcb0 msgr2=0x7f5fb00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- 172.21.15.116:0/2674935627 shutdown_connections 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- 172.21.15.116:0/2674935627 wait complete. 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 Processor -- start 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- start start 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0100740 0x7f5fb00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb01022f0 0x7f5fb019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010c270 con 0x7f5fb0101930 2024-03-19T02:25:20.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010c0f0 con 0x7f5fb01022f0 2024-03-19T02:25:20.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb7377640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb010c3f0 con 0x7f5fb0100740 2024-03-19T02:25:20.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb4912640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb4912640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39436/0 (socket says 172.21.15.116:39436) 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.012+0000 7f5fb4912640 1 -- 172.21.15.116:0/3724284268 learned_addr learned my addr 172.21.15.116:0/3724284268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb5914640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb01022f0 0x7f5fb019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb5113640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0100740 0x7f5fb00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 -- 172.21.15.116:0/3724284268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0100740 msgr2=0x7f5fb00694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0100740 0x7f5fb00694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 -- 172.21.15.116:0/3724284268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb01022f0 msgr2=0x7f5fb019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb01022f0 0x7f5fb019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 -- 172.21.15.116:0/3724284268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb019f6b0 con 0x7f5fb0101930 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb4912640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c970 tx=0x7f5fac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac007bb0 con 0x7f5fb0101930 2024-03-19T02:25:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fac007d50 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb019f940 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac00f450 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.013+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb019fe40 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.014+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb0107a60 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.015+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f5fac004f00 con 0x7f5fb0101930 2024-03-19T02:25:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.015+0000 7f5fa67fc640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 0x7f5f9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.016+0000 7f5fb5113640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 0x7f5f9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.016+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5fac0ceda0 con 0x7f5fb0101930 2024-03-19T02:25:20.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.016+0000 7f5fb5113640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 0x7f5f9c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0101790 tx=0x7f5fa0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.019+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fac014030 con 0x7f5fb0101930 2024-03-19T02:25:20.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.195+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fb002d3c0 con 0x7f5fb0101930 2024-03-19T02:25:20.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.196+0000 7f5fa67fc640 1 -- 172.21.15.116:0/3724284268 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f5fac09bc00 con 0x7f5fb0101930 2024-03-19T02:25:20.198 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:20.198 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 msgr2=0x7f5f9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 0x7f5f9c0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0101790 tx=0x7f5fa0005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 msgr2=0x7f5fb019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c970 tx=0x7f5fac00ce30 comp rx=0 tx=0).stop 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 shutdown_connections 2024-03-19T02:25:20.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5f9c0a2800 0x7f5f9c0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb01022f0 0x7f5fb019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fb0101930 0x7f5fb019c780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 --2- 172.21.15.116:0/3724284268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fb0100740 0x7f5fb00694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.198+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 >> 172.21.15.116:0/3724284268 conn(0x7f5fb00fbcb0 msgr2=0x7f5fb0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.199+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 shutdown_connections 2024-03-19T02:25:20.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.199+0000 7f5fb7377640 1 -- 172.21.15.116:0/3724284268 wait complete. 2024-03-19T02:25:20.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:25:20.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:25:20.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 -- 172.21.15.116:0/2291256025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c101910 msgr2=0x7f373c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 --2- 172.21.15.116:0/2291256025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c101910 0x7f373c101d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3730009860 tx=0x7f373002bb70 comp rx=0 tx=0).stop 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 -- 172.21.15.116:0/2291256025 shutdown_connections 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 --2- 172.21.15.116:0/2291256025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 --2- 172.21.15.116:0/2291256025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c101910 0x7f373c101d90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 --2- 172.21.15.116:0/2291256025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c100720 0x7f373c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 -- 172.21.15.116:0/2291256025 >> 172.21.15.116:0/2291256025 conn(0x7f373c0fbcb0 msgr2=0x7f373c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.316+0000 7f374253b640 1 -- 172.21.15.116:0/2291256025 shutdown_connections 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 -- 172.21.15.116:0/2291256025 wait complete. 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 Processor -- start 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 -- start start 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 0x7f373c1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10c230 con 0x7f373c101910 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10c0b0 con 0x7f373c100720 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f374253b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f373c10c3b0 con 0x7f373c1022d0 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.317+0000 7f373b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 0x7f373c1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39456/0 (socket says 172.21.15.116:39456) 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373b7fe640 1 -- 172.21.15.116:0/3580620335 learned_addr learned my addr 172.21.15.116:0/3580620335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f3740ad8640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 -- 172.21.15.116:0/3580620335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 msgr2=0x7f373c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 -- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 msgr2=0x7f373c196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 -- 172.21.15.116:0/3580620335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c19e5c0 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373b7fe640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f3740ad8640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f373bfff640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 0x7f373c1969a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f373c101770 tx=0x7f372c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c004440 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f372c00cd70 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c19e8b0 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c019480 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.318+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c19edd0 con 0x7f373c100720 2024-03-19T02:25:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.319+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373c107bb0 con 0x7f373c100720 2024-03-19T02:25:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.320+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f372c003710 con 0x7f373c100720 2024-03-19T02:25:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.321+0000 7f37397fa640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 0x7f37240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.321+0000 7f373b7fe640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 0x7f37240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.321+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f372c0cf1d0 con 0x7f373c100720 2024-03-19T02:25:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.321+0000 7f373b7fe640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 0x7f37240a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f373002c180 tx=0x7f37300058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.324+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f372c009510 con 0x7f373c100720 2024-03-19T02:25:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.499+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f373c062420 con 0x7f373c100720 2024-03-19T02:25:20.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.500+0000 7f37397fa640 1 -- 172.21.15.116:0/3580620335 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f372c09c0c0 con 0x7f373c100720 2024-03-19T02:25:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:25:20.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 msgr2=0x7f37240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 0x7f37240a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f373002c180 tx=0x7f37300058b0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 msgr2=0x7f373c1969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 0x7f373c1969a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f373c101770 tx=0x7f372c00ee50 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 shutdown_connections 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f37240a2800 0x7f37240a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f373c1022d0 0x7f373c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f373c101910 0x7f373c196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 --2- 172.21.15.116:0/3580620335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f373c100720 0x7f373c1969a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 >> 172.21.15.116:0/3580620335 conn(0x7f373c0fbcb0 msgr2=0x7f373c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 shutdown_connections 2024-03-19T02:25:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.502+0000 7f374253b640 1 -- 172.21.15.116:0/3580620335 wait complete. 2024-03-19T02:25:20.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 -- 172.21.15.116:0/361381231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 msgr2=0x7f1dd0109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/361381231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd0109960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009ca0 tx=0x7f1dc802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 -- 172.21.15.116:0/361381231 shutdown_connections 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/361381231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd0109960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/361381231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0104c40 0x7f1dd0107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/361381231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0102310 0x7f1dd0104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 -- 172.21.15.116:0/361381231 >> 172.21.15.116:0/361381231 conn(0x7f1dd00fbcc0 msgr2=0x7f1dd00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 -- 172.21.15.116:0/361381231 shutdown_connections 2024-03-19T02:25:20.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 -- 172.21.15.116:0/361381231 wait complete. 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.618+0000 7f1dd54ff640 1 Processor -- start 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 -- start start 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0102310 0x7f1dd0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0104c40 0x7f1dd0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd010d380 con 0x7f1dd0107570 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd010d200 con 0x7f1dd0104c40 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dd54ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd010d500 con 0x7f1dd0102310 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39476/0 (socket says 172.21.15.116:39476) 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 -- 172.21.15.116:0/3789893313 learned_addr learned my addr 172.21.15.116:0/3789893313 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dc67fc640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0104c40 0x7f1dd0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dceffd640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0102310 0x7f1dd0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 -- 172.21.15.116:0/3789893313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0102310 msgr2=0x7f1dd0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0102310 0x7f1dd0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 -- 172.21.15.116:0/3789893313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0104c40 msgr2=0x7f1dd0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0104c40 0x7f1dd0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 -- 172.21.15.116:0/3789893313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd019f680 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dcf7fe640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1dc802c700 tx=0x7f1dc80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.619+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8041020 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.620+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dc8003e20 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.620+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd019f910 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.620+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8039040 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.620+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd019fe50 con 0x7f1dd0107570 2024-03-19T02:25:20.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.621+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd00fd7b0 con 0x7f1dd0107570 2024-03-19T02:25:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.621+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1dc8044020 con 0x7f1dd0107570 2024-03-19T02:25:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.622+0000 7f1dccff9640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 0x7f1dbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.622+0000 7f1dceffd640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 0x7f1dbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.622+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1dc80f0e70 con 0x7f1dd0107570 2024-03-19T02:25:20.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.623+0000 7f1dceffd640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 0x7f1dbc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009b60 tx=0x7f1dc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.627+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1dc804b020 con 0x7f1dd0107570 2024-03-19T02:25:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.804+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1dd0062420 con 0x7f1dd0107570 2024-03-19T02:25:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.805+0000 7f1dccff9640 1 -- 172.21.15.116:0/3789893313 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f1dc80bdcd0 con 0x7f1dd0107570 2024-03-19T02:25:20.805 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:25:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.806+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 msgr2=0x7f1dbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.806+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 0x7f1dbc0a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009b60 tx=0x7f1dc0005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 msgr2=0x7f1dd019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1dc802c700 tx=0x7f1dc80328a0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 shutdown_connections 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dbc0a2880 0x7f1dbc0a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dd0107570 0x7f1dd019f010 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd0104c40 0x7f1dd0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 --2- 172.21.15.116:0/3789893313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dd0102310 0x7f1dd0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 >> 172.21.15.116:0/3789893313 conn(0x7f1dd00fbcc0 msgr2=0x7f1dd0102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 shutdown_connections 2024-03-19T02:25:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.807+0000 7f1dd54ff640 1 -- 172.21.15.116:0/3789893313 wait complete. 2024-03-19T02:25:20.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:20 smithi116 ceph-mon[32263]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-19T02:25:20.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:20 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3724284268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:20.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:20 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3580620335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:20.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:25:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:20 smithi162 ceph-mon[35761]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-19T02:25:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:20 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3724284268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:20.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:20 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3580620335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.922+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2286762235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0101850 msgr2=0x7f4fd0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.922+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2286762235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0101850 0x7f4fd0101c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009900 tx=0x7f4fc002bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2286762235 shutdown_connections 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2286762235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0103400 0x7f4fd0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2286762235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0102a40 0x7f4fd0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2286762235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0101850 0x7f4fd0101c70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2286762235 >> 172.21.15.116:0/2286762235 conn(0x7f4fd00fcdc0 msgr2=0x7f4fd00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2286762235 shutdown_connections 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2286762235 wait complete. 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 Processor -- start 2024-03-19T02:25:20.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- start start 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 0x7f4fd0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0103400 0x7f4fd019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd010d370 con 0x7f4fd0102a40 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd010d1f0 con 0x7f4fd0101850 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd7ac6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd010d4f0 con 0x7f4fd0103400 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd5061640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd5862640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 0x7f4fd0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.923+0000 7f4fd5061640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39488/0 (socket says 172.21.15.116:39488) 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 -- 172.21.15.116:0/2300753924 learned_addr learned my addr 172.21.15.116:0/2300753924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd6063640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0103400 0x7f4fd019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 -- 172.21.15.116:0/2300753924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0103400 msgr2=0x7f4fd019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0103400 0x7f4fd019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 -- 172.21.15.116:0/2300753924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 msgr2=0x7f4fd0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 0x7f4fd0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 -- 172.21.15.116:0/2300753924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd019f710 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5862640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 0x7f4fd0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd5061640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c970 tx=0x7f4fcc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc007bb0 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd019fa00 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fcc007d50 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc00f450 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.924+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd019ff50 con 0x7f4fd0102a40 2024-03-19T02:25:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.925+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd0108c90 con 0x7f4fd0102a40 2024-03-19T02:25:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.926+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4fcc004f00 con 0x7f4fd0102a40 2024-03-19T02:25:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.927+0000 7f4fc6ffd640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 0x7f4fbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.927+0000 7f4fd5862640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 0x7f4fbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.927+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4fcc0ced60 con 0x7f4fd0102a40 2024-03-19T02:25:20.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.927+0000 7f4fd5862640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 0x7f4fbc0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0005ab0 tx=0x7f4fc0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:20.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:20.930+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fcc014030 con 0x7f4fd0102a40 2024-03-19T02:25:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:20 smithi154 ceph-mon[35693]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-19T02:25:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:20 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3724284268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:20 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3580620335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.106+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4fd002d400 con 0x7f4fd0102a40 2024-03-19T02:25:21.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.106+0000 7f4fc6ffd640 1 -- 172.21.15.116:0/2300753924 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f4fcc09bc40 con 0x7f4fd0102a40 2024-03-19T02:25:21.107 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 msgr2=0x7f4fbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 0x7f4fbc0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0005ab0 tx=0x7f4fc0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 msgr2=0x7f4fd0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c970 tx=0x7f4fcc00ce30 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 shutdown_connections 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4fbc0a2880 0x7f4fbc0a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fd0103400 0x7f4fd019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd0102a40 0x7f4fd0198030 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 --2- 172.21.15.116:0/2300753924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd0101850 0x7f4fd0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 >> 172.21.15.116:0/2300753924 conn(0x7f4fd00fcdc0 msgr2=0x7f4fd0105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 shutdown_connections 2024-03-19T02:25:21.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.108+0000 7f4fd7ac6640 1 -- 172.21.15.116:0/2300753924 wait complete. 2024-03-19T02:25:21.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- 172.21.15.116:0/1503500078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f41022f0 msgr2=0x7f07f4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 --2- 172.21.15.116:0/1503500078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f41022f0 0x7f07f4108b80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009ca0 tx=0x7f07ec02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- 172.21.15.116:0/1503500078 shutdown_connections 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 --2- 172.21.15.116:0/1503500078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f41022f0 0x7f07f4108b80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 --2- 172.21.15.116:0/1503500078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f4101930 0x7f07f4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 --2- 172.21.15.116:0/1503500078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 0x7f07f4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- 172.21.15.116:0/1503500078 >> 172.21.15.116:0/1503500078 conn(0x7f07f40fbcd0 msgr2=0x7f07f40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- 172.21.15.116:0/1503500078 shutdown_connections 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- 172.21.15.116:0/1503500078 wait complete. 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 Processor -- start 2024-03-19T02:25:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.224+0000 7f07fc540640 1 -- start start 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 0x7f07f4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f41022f0 0x7f07f419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07f410d380 con 0x7f07f4101930 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07f410d200 con 0x7f07f41022f0 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07f410d500 con 0x7f07f4100740 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39496/0 (socket says 172.21.15.116:39496) 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 -- 172.21.15.116:0/2842872486 learned_addr learned my addr 172.21.15.116:0/2842872486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07faadd640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f41022f0 0x7f07f419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fa2dc640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 0x7f07f4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 -- 172.21.15.116:0/2842872486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 msgr2=0x7f07f4197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 0x7f07f4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 -- 172.21.15.116:0/2842872486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f41022f0 msgr2=0x7f07f419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f41022f0 0x7f07f419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 -- 172.21.15.116:0/2842872486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f419f630 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07f9adb640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f07e400e9f0 tx=0x7f07e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e400cd80 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f419f920 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07e40108e0 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4018920 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.225+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f419fec0 con 0x7f07f4101930 2024-03-19T02:25:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.226+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4107a60 con 0x7f07f4101930 2024-03-19T02:25:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.227+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f07e4010a90 con 0x7f07f4101930 2024-03-19T02:25:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.228+0000 7f07eb7fe640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 0x7f07e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.228+0000 7f07fa2dc640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 0x7f07e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.228+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f07e40cf040 con 0x7f07f4101930 2024-03-19T02:25:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.228+0000 7f07fa2dc640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 0x7f07e00a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f07f0005fd0 tx=0x7f07f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.232+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f07e409bea0 con 0x7f07f4101930 2024-03-19T02:25:21.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.407+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f07f4062420 con 0x7f07f4101930 2024-03-19T02:25:21.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.407+0000 7f07eb7fe640 1 -- 172.21.15.116:0/2842872486 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f07e409b7a0 con 0x7f07f4101930 2024-03-19T02:25:21.408 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 msgr2=0x7f07e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 0x7f07e00a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f07f0005fd0 tx=0x7f07f0005850 comp rx=0 tx=0).stop 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 msgr2=0x7f07f4197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f07e400e9f0 tx=0x7f07e400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 shutdown_connections 2024-03-19T02:25:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f07e00a2800 0x7f07e00a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07f41022f0 0x7f07f419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07f4101930 0x7f07f4197f50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 --2- 172.21.15.116:0/2842872486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07f4100740 0x7f07f4197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 >> 172.21.15.116:0/2842872486 conn(0x7f07f40fbcd0 msgr2=0x7f07f4104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 shutdown_connections 2024-03-19T02:25:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.409+0000 7f07fc540640 1 -- 172.21.15.116:0/2842872486 wait complete. 2024-03-19T02:25:21.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 -- 172.21.15.116:0/3874644370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 msgr2=0x7efdf4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 --2- 172.21.15.116:0/3874644370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4100b50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efde4009900 tx=0x7efde402bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 -- 172.21.15.116:0/3874644370 shutdown_connections 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 --2- 172.21.15.116:0/3874644370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 0x7efdf4108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 --2- 172.21.15.116:0/3874644370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 0x7efdf4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 --2- 172.21.15.116:0/3874644370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4100b50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 -- 172.21.15.116:0/3874644370 >> 172.21.15.116:0/3874644370 conn(0x7efdf40fbcc0 msgr2=0x7efdf40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 -- 172.21.15.116:0/3874644370 shutdown_connections 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.522+0000 7efdf8836640 1 -- 172.21.15.116:0/3874644370 wait complete. 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 Processor -- start 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 -- start start 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 0x7efdf4196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 0x7efdf419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efdf410c230 con 0x7efdf4100730 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efdf410c0b0 con 0x7efdf4101920 2024-03-19T02:25:21.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf8836640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efdf410c3b0 con 0x7efdf41022e0 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39512/0 (socket says 172.21.15.116:39512) 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdf259c640 1 -- 172.21.15.116:0/2686376856 learned_addr learned my addr 172.21.15.116:0/2686376856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.523+0000 7efdebfff640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 0x7efdf4196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf2d9d640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 0x7efdf419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 -- 172.21.15.116:0/2686376856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 msgr2=0x7efdf419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 0x7efdf419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 -- 172.21.15.116:0/2686376856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 msgr2=0x7efdf4196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 0x7efdf4196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 -- 172.21.15.116:0/2686376856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf419e590 con 0x7efdf4100730 2024-03-19T02:25:21.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf259c640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efde4000c00 tx=0x7efde4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4043020 con 0x7efdf4100730 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efde4004440 con 0x7efdf4100730 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf419e820 con 0x7efdf4100730 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4003c90 con 0x7efdf4100730 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.524+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf419ed70 con 0x7efdf4100730 2024-03-19T02:25:21.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.525+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf4107b70 con 0x7efdf4100730 2024-03-19T02:25:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.526+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efde4046020 con 0x7efdf4100730 2024-03-19T02:25:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.527+0000 7efdeb7fe640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 0x7efde00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.527+0000 7efdebfff640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 0x7efde00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.527+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efde40f1330 con 0x7efdf4100730 2024-03-19T02:25:21.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.527+0000 7efdebfff640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 0x7efde00a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efdf4197e80 tx=0x7efddc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.530+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efde404d020 con 0x7efdf4100730 2024-03-19T02:25:21.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.720+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efdf402d5e0 con 0x7efdf4100730 2024-03-19T02:25:21.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.720+0000 7efdeb7fe640 1 -- 172.21.15.116:0/2686376856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+598 (secure 0 0 0) 0x7efde40be190 con 0x7efdf4100730 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1993666663 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/707840070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/996139070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2583747917 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3953610795 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:21.721 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2020074950 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:21.722 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 msgr2=0x7efde00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 0x7efde00a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efdf4197e80 tx=0x7efddc005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 msgr2=0x7efdf4196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efde4000c00 tx=0x7efde4004040 comp rx=0 tx=0).stop 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 shutdown_connections 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efde00a2800 0x7efde00a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdf41022e0 0x7efdf419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdf4101920 0x7efdf4196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 --2- 172.21.15.116:0/2686376856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf4100730 0x7efdf4196970 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 >> 172.21.15.116:0/2686376856 conn(0x7efdf40fbcc0 msgr2=0x7efdf4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 shutdown_connections 2024-03-19T02:25:21.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.722+0000 7efdf8836640 1 -- 172.21.15.116:0/2686376856 wait complete. 2024-03-19T02:25:21.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:21 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3789893313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:21 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2300753924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:21 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2842872486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.734 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:25:21.734 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:25:21.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:21.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2027227495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 msgr2=0x7f9f38101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2027227495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38101db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009900 tx=0x7f9f2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2027227495 shutdown_connections 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2027227495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 0x7f9f38108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2027227495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38101db0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2027227495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 0x7f9f38100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.892+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2027227495 >> 172.21.15.116:0/2027227495 conn(0x7f9f380fbcb0 msgr2=0x7f9f380fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2027227495 shutdown_connections 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2027227495 wait complete. 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 Processor -- start 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- start start 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 0x7f9f381969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 0x7f9f3819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810c270 con 0x7f9f38101930 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810c0f0 con 0x7f9f38100740 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3fc4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3810c3f0 con 0x7f9f381022f0 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3d1e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3d1e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39530/0 (socket says 172.21.15.116:39530) 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.893+0000 7f9f3d1e6640 1 -- 172.21.15.116:0/2074580648 learned_addr learned my addr 172.21.15.116:0/2074580648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d9e7640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 0x7f9f381969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3e1e8640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 0x7f9f3819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 -- 172.21.15.116:0/2074580648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 msgr2=0x7f9f3819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 0x7f9f3819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 -- 172.21.15.116:0/2074580648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 msgr2=0x7f9f381969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:21.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 0x7f9f381969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 -- 172.21.15.116:0/2074580648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3819e5d0 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3d1e6640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f28006010 tx=0x7f9f28004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f28043020 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f28004440 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3819e860 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f28003bf0 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.894+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3819ee00 con 0x7f9f38101930 2024-03-19T02:25:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.895+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f38107ad0 con 0x7f9f38101930 2024-03-19T02:25:21.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.896+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9f28046020 con 0x7f9f38101930 2024-03-19T02:25:21.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.897+0000 7f9f2effd640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 0x7f9f240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:21.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.897+0000 7f9f3d9e7640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 0x7f9f240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.897+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9f280f0f30 con 0x7f9f38101930 2024-03-19T02:25:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.897+0000 7f9f3d9e7640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 0x7f9f240a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f38101790 tx=0x7f9f34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:21.901+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9f2804d020 con 0x7f9f38101930 2024-03-19T02:25:21.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:21 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3789893313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:21 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2300753924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:21.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:21 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2842872486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:22.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:21 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3789893313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:22.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:21 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2300753924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:22.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:21 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2842872486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:22.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.081+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f38062420 con 0x7f9f38101930 2024-03-19T02:25:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.083+0000 7f9f2effd640 1 -- 172.21.15.116:0/2074580648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f9f280bdd90 con 0x7f9f38101930 2024-03-19T02:25:22.084 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:22.084 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 msgr2=0x7f9f240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 0x7f9f240a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f38101790 tx=0x7f9f34005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 msgr2=0x7f9f38196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f28006010 tx=0x7f9f28004040 comp rx=0 tx=0).stop 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 shutdown_connections 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f9f240a2880 0x7f9f240a4d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f381022f0 0x7f9f3819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f38101930 0x7f9f38196ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 --2- 172.21.15.116:0/2074580648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f38100740 0x7f9f381969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 >> 172.21.15.116:0/2074580648 conn(0x7f9f380fbcb0 msgr2=0x7f9f38104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 shutdown_connections 2024-03-19T02:25:22.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.085+0000 7f9f3fc4b640 1 -- 172.21.15.116:0/2074580648 wait complete. 2024-03-19T02:25:22.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:25:22.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 -- 172.21.15.116:0/2143564130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178104c30 msgr2=0x7fb178107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 --2- 172.21.15.116:0/2143564130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178104c30 0x7fb178107020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009ca0 tx=0x7fb16c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 -- 172.21.15.116:0/2143564130 shutdown_connections 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 --2- 172.21.15.116:0/2143564130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178107560 0x7fb178109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 --2- 172.21.15.116:0/2143564130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178104c30 0x7fb178107020 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 --2- 172.21.15.116:0/2143564130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178102300 0x7fb1781046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 -- 172.21.15.116:0/2143564130 >> 172.21.15.116:0/2143564130 conn(0x7fb1780fbcb0 msgr2=0x7fb1780fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 -- 172.21.15.116:0/2143564130 shutdown_connections 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.201+0000 7fb17d881640 1 -- 172.21.15.116:0/2143564130 wait complete. 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 Processor -- start 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 -- start start 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 0x7fb178197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178104c30 0x7fb178198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb17810d340 con 0x7fb178107560 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb17810d1c0 con 0x7fb178102300 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb17d881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb17810d4c0 con 0x7fb178104c30 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb1777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb176ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 0x7fb178197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb1777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39552/0 (socket says 172.21.15.116:39552) 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.202+0000 7fb1777fe640 1 -- 172.21.15.116:0/4070513074 learned_addr learned my addr 172.21.15.116:0/4070513074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb1767fc640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178104c30 0x7fb178198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 -- 172.21.15.116:0/4070513074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178104c30 msgr2=0x7fb178198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178104c30 0x7fb178198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 -- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 msgr2=0x7fb17819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 -- 172.21.15.116:0/4070513074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17819f6e0 con 0x7fb178102300 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb1777fe640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb176ffd640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 0x7fb178197ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb17000e9f0 tx=0x7fb17000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17000ccc0 con 0x7fb178102300 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17819f9d0 con 0x7fb178102300 2024-03-19T02:25:22.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb170005020 con 0x7fb178102300 2024-03-19T02:25:22.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170004a00 con 0x7fb178102300 2024-03-19T02:25:22.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.203+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17819fef0 con 0x7fb178102300 2024-03-19T02:25:22.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.204+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1780fd7a0 con 0x7fb178102300 2024-03-19T02:25:22.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.205+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb170004ba0 con 0x7fb178102300 2024-03-19T02:25:22.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.206+0000 7fb17c87f640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 0x7fb1600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.206+0000 7fb1767fc640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 0x7fb1600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.206+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb1700cec20 con 0x7fb178102300 2024-03-19T02:25:22.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.206+0000 7fb1767fc640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 0x7fb1600a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb16c032380 tx=0x7fb16c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.212+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb17009ba80 con 0x7fb178102300 2024-03-19T02:25:22.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.385+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb178062450 con 0x7fb178102300 2024-03-19T02:25:22.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.386+0000 7fb17c87f640 1 -- 172.21.15.116:0/4070513074 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fb17009b380 con 0x7fb178102300 2024-03-19T02:25:22.386 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 msgr2=0x7fb1600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 0x7fb1600a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb16c032380 tx=0x7fb16c000f40 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 msgr2=0x7fb178197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 0x7fb178197ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb17000e9f0 tx=0x7fb17000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 shutdown_connections 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb1600a2800 0x7fb1600a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb178107560 0x7fb17819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb178104c30 0x7fb178198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 --2- 172.21.15.116:0/4070513074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb178102300 0x7fb178197ac0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 >> 172.21.15.116:0/4070513074 conn(0x7fb1780fbcb0 msgr2=0x7fb178102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 shutdown_connections 2024-03-19T02:25:22.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.388+0000 7fb17d881640 1 -- 172.21.15.116:0/4070513074 wait complete. 2024-03-19T02:25:22.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:25:22.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 -- 172.21.15.116:0/3175603510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 msgr2=0x7efd4c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 --2- 172.21.15.116:0/3175603510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c103240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efd40009860 tx=0x7efd4002bb70 comp rx=0 tx=0).stop 2024-03-19T02:25:22.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 -- 172.21.15.116:0/3175603510 shutdown_connections 2024-03-19T02:25:22.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 --2- 172.21.15.116:0/3175603510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 --2- 172.21.15.116:0/3175603510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 --2- 172.21.15.116:0/3175603510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c103240 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 -- 172.21.15.116:0/3175603510 >> 172.21.15.116:0/3175603510 conn(0x7efd4c0fb450 msgr2=0x7efd4c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 -- 172.21.15.116:0/3175603510 shutdown_connections 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.503+0000 7efd50abf640 1 -- 172.21.15.116:0/3175603510 wait complete. 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 Processor -- start 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 -- start start 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d370 con 0x7efd4c100250 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d1f0 con 0x7efd4c102e20 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd50abf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd4c10d4f0 con 0x7efd4c0ff890 2024-03-19T02:25:22.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39578/0 (socket says 172.21.15.116:39578) 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 -- 172.21.15.116:0/3749250996 learned_addr learned my addr 172.21.15.116:0/3749250996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd4ad9d640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd4a59c640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 -- 172.21.15.116:0/3749250996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 msgr2=0x7efd4c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 -- 172.21.15.116:0/3749250996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 msgr2=0x7efd4c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 -- 172.21.15.116:0/3749250996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4c19f6c0 con 0x7efd4c100250 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd4a59c640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.504+0000 7efd49d9b640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e9f0 tx=0x7efd3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.505+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c00cd80 con 0x7efd4c100250 2024-03-19T02:25:22.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.505+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4c19f9b0 con 0x7efd4c100250 2024-03-19T02:25:22.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.505+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd3c0054d0 con 0x7efd4c100250 2024-03-19T02:25:22.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.505+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c010490 con 0x7efd4c100250 2024-03-19T02:25:22.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.505+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4c19feb0 con 0x7efd4c100250 2024-03-19T02:25:22.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.506+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd4c108e50 con 0x7efd4c100250 2024-03-19T02:25:22.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.507+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efd3c005020 con 0x7efd4c100250 2024-03-19T02:25:22.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.507+0000 7efd3b7fe640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 0x7efd340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.507+0000 7efd4a59c640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 0x7efd340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.507+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efd3c0cf0d0 con 0x7efd4c100250 2024-03-19T02:25:22.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.509+0000 7efd4a59c640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 0x7efd340a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd4002c180 tx=0x7efd40034980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.513+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd3c09bf30 con 0x7efd4c100250 2024-03-19T02:25:22.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.691+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd4c062420 con 0x7efd4c100250 2024-03-19T02:25:22.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.691+0000 7efd3b7fe640 1 -- 172.21.15.116:0/3749250996 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7efd3c09b830 con 0x7efd4c100250 2024-03-19T02:25:22.692 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:25:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 msgr2=0x7efd340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 0x7efd340a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd4002c180 tx=0x7efd40034980 comp rx=0 tx=0).stop 2024-03-19T02:25:22.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 msgr2=0x7efd4c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e9f0 tx=0x7efd3c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 shutdown_connections 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efd340a2800 0x7efd340a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd4c102e20 0x7efd4c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.693+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd4c100250 0x7efd4c197fe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.694+0000 7efd50abf640 1 --2- 172.21.15.116:0/3749250996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd4c0ff890 0x7efd4c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.694+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 >> 172.21.15.116:0/3749250996 conn(0x7efd4c0fb450 msgr2=0x7efd4c106020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.694+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 shutdown_connections 2024-03-19T02:25:22.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.694+0000 7efd50abf640 1 -- 172.21.15.116:0/3749250996 wait complete. 2024-03-19T02:25:22.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- 172.21.15.116:0/3885576273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e501033e0 msgr2=0x7f2e50109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 --2- 172.21.15.116:0/3885576273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e501033e0 0x7f2e50109c70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009ca0 tx=0x7f2e4802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- 172.21.15.116:0/3885576273 shutdown_connections 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 --2- 172.21.15.116:0/3885576273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e501033e0 0x7f2e50109c70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 --2- 172.21.15.116:0/3885576273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 --2- 172.21.15.116:0/3885576273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e50101830 0x7f2e50101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- 172.21.15.116:0/3885576273 >> 172.21.15.116:0/3885576273 conn(0x7f2e500fcdc0 msgr2=0x7f2e500ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- 172.21.15.116:0/3885576273 shutdown_connections 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- 172.21.15.116:0/3885576273 wait complete. 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 Processor -- start 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.809+0000 7f2e565c4640 1 -- start start 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e501033e0 0x7f2e5019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5010d360 con 0x7f2e50101830 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5010d1e0 con 0x7f2e501033e0 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e565c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5010d4e0 con 0x7f2e50102a20 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39594/0 (socket says 172.21.15.116:39594) 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 -- 172.21.15.116:0/728988136 learned_addr learned my addr 172.21.15.116:0/728988136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:22.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e54b61640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e501033e0 0x7f2e5019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4f7fe640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 -- 172.21.15.116:0/728988136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 msgr2=0x7f2e50197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 -- 172.21.15.116:0/728988136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e501033e0 msgr2=0x7f2e5019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e501033e0 0x7f2e5019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 -- 172.21.15.116:0/728988136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5019f630 con 0x7f2e50101830 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4f7fe640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4ffff640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e9c0 tx=0x7f2e4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.810+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000cd80 con 0x7f2e50101830 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.811+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5019f920 con 0x7f2e50101830 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.811+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e400035f0 con 0x7f2e50101830 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.811+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40010690 con 0x7f2e50101830 2024-03-19T02:25:22.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.811+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5019fe70 con 0x7f2e50101830 2024-03-19T02:25:22.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.812+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e50108b50 con 0x7f2e50101830 2024-03-19T02:25:22.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.812+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2e400037a0 con 0x7f2e50101830 2024-03-19T02:25:22.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.813+0000 7f2e4d7fa640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 0x7f2e380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:22.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.813+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2e400cf0e0 con 0x7f2e50101830 2024-03-19T02:25:22.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.813+0000 7f2e4f7fe640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 0x7f2e380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:22.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.814+0000 7f2e4f7fe640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 0x7f2e380a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e50198f20 tx=0x7f2e44005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:22.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.817+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e4009bf40 con 0x7f2e50101830 2024-03-19T02:25:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:22 smithi162 ceph-mon[35761]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:22 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2686376856' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:22 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2074580648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:22.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:22 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4070513074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:22.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.992+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e50062420 con 0x7f2e50101830 2024-03-19T02:25:22.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.992+0000 7f2e4d7fa640 1 -- 172.21.15.116:0/728988136 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f2e4009b840 con 0x7f2e50101830 2024-03-19T02:25:22.992 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 msgr2=0x7f2e380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 0x7f2e380a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e50198f20 tx=0x7f2e44005e50 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 msgr2=0x7f2e50197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e9c0 tx=0x7f2e4000ee80 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 shutdown_connections 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2e380a2800 0x7f2e380a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e501033e0 0x7f2e5019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e50102a20 0x7f2e50197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 --2- 172.21.15.116:0/728988136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e50101830 0x7f2e50197a10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 >> 172.21.15.116:0/728988136 conn(0x7f2e500fcdc0 msgr2=0x7f2e50105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 shutdown_connections 2024-03-19T02:25:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:22.994+0000 7f2e565c4640 1 -- 172.21.15.116:0/728988136 wait complete. 2024-03-19T02:25:23.003 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:22 smithi116 ceph-mon[32263]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:23.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:22 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2686376856' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:23.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:22 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2074580648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:23.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:22 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4070513074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:23.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:25:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:22 smithi154 ceph-mon[35693]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:22 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2686376856' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:22 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2074580648' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:22 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4070513074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 -- 172.21.15.116:0/3681764562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc100720 msgr2=0x7f30cc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 --2- 172.21.15.116:0/3681764562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc100720 0x7f30cc100b40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009900 tx=0x7f30bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 -- 172.21.15.116:0/3681764562 shutdown_connections 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 --2- 172.21.15.116:0/3681764562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 0x7f30cc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 --2- 172.21.15.116:0/3681764562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc101910 0x7f30cc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 --2- 172.21.15.116:0/3681764562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc100720 0x7f30cc100b40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 -- 172.21.15.116:0/3681764562 >> 172.21.15.116:0/3681764562 conn(0x7f30cc0fbcb0 msgr2=0x7f30cc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:23.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 -- 172.21.15.116:0/3681764562 shutdown_connections 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 -- 172.21.15.116:0/3681764562 wait complete. 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.111+0000 7f30d1219640 1 Processor -- start 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 -- start start 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc100720 0x7f30cc199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 0x7f30cc1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10b110 con 0x7f30cc101910 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10af90 con 0x7f30cc1022d0 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30d1219640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc10b290 con 0x7f30cc100720 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39620/0 (socket says 172.21.15.116:39620) 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 -- 172.21.15.116:0/2374168539 learned_addr learned my addr 172.21.15.116:0/2374168539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:23.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30cad9d640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc100720 0x7f30cc199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30cb59e640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 0x7f30cc1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 -- 172.21.15.116:0/2374168539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc100720 msgr2=0x7f30cc199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc100720 0x7f30cc199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 -- 172.21.15.116:0/2374168539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 msgr2=0x7f30cc1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 0x7f30cc1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 -- 172.21.15.116:0/2374168539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc1a1900 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c3fff640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f30b400d950 tx=0x7f30b400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4017070 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30b400de80 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.112+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400f620 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.113+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc1a1bf0 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.113+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30cc1a20c0 con 0x7f30cc101910 2024-03-19T02:25:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.113+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc068610 con 0x7f30cc101910 2024-03-19T02:25:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.115+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f30b4009c30 con 0x7f30cc101910 2024-03-19T02:25:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.115+0000 7f30c8d99640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 0x7f30b80a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.115+0000 7f30cad9d640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 0x7f30b80a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.115+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f30b40ce810 con 0x7f30cc101910 2024-03-19T02:25:23.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.116+0000 7f30cad9d640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 0x7f30b80a4d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30bc005ab0 tx=0x7f30bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:23.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.119+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30b409b6f0 con 0x7f30cc101910 2024-03-19T02:25:23.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.293+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f30cc1a26c0 con 0x7f30cc101910 2024-03-19T02:25:23.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.294+0000 7f30c8d99640 1 -- 172.21.15.116:0/2374168539 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f30b409aff0 con 0x7f30cc101910 2024-03-19T02:25:23.294 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 msgr2=0x7f30b80a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 0x7f30b80a4d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30bc005ab0 tx=0x7f30bc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 msgr2=0x7f30cc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f30b400d950 tx=0x7f30b400dd50 comp rx=0 tx=0).stop 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 shutdown_connections 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f30b80a2860 0x7f30b80a4d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30cc1022d0 0x7f30cc1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc101910 0x7f30cc19a230 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 --2- 172.21.15.116:0/2374168539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30cc100720 0x7f30cc199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 >> 172.21.15.116:0/2374168539 conn(0x7f30cc0fbcb0 msgr2=0x7f30cc107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:23.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 shutdown_connections 2024-03-19T02:25:23.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.296+0000 7f30d1219640 1 -- 172.21.15.116:0/2374168539 wait complete. 2024-03-19T02:25:23.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:25:23.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 -- 172.21.15.116:0/2343459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301033e0 msgr2=0x7fca30109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 --2- 172.21.15.116:0/2343459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301033e0 0x7fca30109c70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca28009ca0 tx=0x7fca2802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 -- 172.21.15.116:0/2343459906 shutdown_connections 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 --2- 172.21.15.116:0/2343459906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301033e0 0x7fca30109c70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 --2- 172.21.15.116:0/2343459906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30102a20 0x7fca30102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 --2- 172.21.15.116:0/2343459906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca30101830 0x7fca30101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 -- 172.21.15.116:0/2343459906 >> 172.21.15.116:0/2343459906 conn(0x7fca300fcdc0 msgr2=0x7fca300ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 -- 172.21.15.116:0/2343459906 shutdown_connections 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.410+0000 7fca36ef1640 1 -- 172.21.15.116:0/2343459906 wait complete. 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 Processor -- start 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 -- start start 2024-03-19T02:25:23.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30101830 0x7fca30197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca301033e0 0x7fca3019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010d340 con 0x7fca30102a20 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010d1c0 con 0x7fca30101830 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca36ef1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca3010d4c0 con 0x7fca301033e0 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca34c8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30101830 0x7fca30197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39638/0 (socket says 172.21.15.116:39638) 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 -- 172.21.15.116:0/526911218 learned_addr learned my addr 172.21.15.116:0/526911218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca3548e640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca301033e0 0x7fca3019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 -- 172.21.15.116:0/526911218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca301033e0 msgr2=0x7fca3019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca301033e0 0x7fca3019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 -- 172.21.15.116:0/526911218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30101830 msgr2=0x7fca30197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30101830 0x7fca30197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 -- 172.21.15.116:0/526911218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3019f5a0 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.411+0000 7fca2ffff640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00ea60 tx=0x7fca1c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.412+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c00cde0 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.412+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3019f890 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.412+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca1c0035f0 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.412+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c010690 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.412+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3019fde0 con 0x7fca30102a20 2024-03-19T02:25:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.413+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca30108bd0 con 0x7fca30102a20 2024-03-19T02:25:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.414+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fca1c0037a0 con 0x7fca30102a20 2024-03-19T02:25:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.414+0000 7fca2dffb640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 0x7fca200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.414+0000 7fca34c8d640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 0x7fca200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.414+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fca1c0cef30 con 0x7fca30102a20 2024-03-19T02:25:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.415+0000 7fca34c8d640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 0x7fca200a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca30102880 tx=0x7fca24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:23.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.418+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca1c09bd90 con 0x7fca30102a20 2024-03-19T02:25:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.605+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fca3002d1f0 con 0x7fca30102a20 2024-03-19T02:25:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.605+0000 7fca2dffb640 1 -- 172.21.15.116:0/526911218 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+598 (secure 0 0 0) 0x7fca1c09b690 con 0x7fca30102a20 2024-03-19T02:25:23.605 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1993666663 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:23.605 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/707840070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/996139070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2583747917 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3953610795 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2020074950 2024-03-20T02:17:41.784026+0000 2024-03-19T02:25:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-19T02:25:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.607+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 msgr2=0x7fca200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.607+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 0x7fca200a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca30102880 tx=0x7fca24005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 msgr2=0x7fca30197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00ea60 tx=0x7fca1c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 shutdown_connections 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fca200a2800 0x7fca200a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca301033e0 0x7fca3019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca30102a20 0x7fca30197ec0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 --2- 172.21.15.116:0/526911218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30101830 0x7fca30197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 >> 172.21.15.116:0/526911218 conn(0x7fca300fcdc0 msgr2=0x7fca30105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 shutdown_connections 2024-03-19T02:25:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:23.608+0000 7fca36ef1640 1 -- 172.21.15.116:0/526911218 wait complete. 2024-03-19T02:25:23.618 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-19T02:25:23.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:23.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:25:23.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[73928]: fuse finished with error 0 and tester_r 0 2024-03-19T02:25:23.646 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:25:23.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3749250996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:23.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/728988136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:23.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2374168539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:23.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/526911218' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:24.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3749250996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/728988136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2374168539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/526911218' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3749250996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/728988136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2374168539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:25:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/526911218' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:25:24.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:24 smithi162 ceph-mon[35761]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:25.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:24 smithi154 ceph-mon[35693]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:24 smithi116 ceph-mon[32263]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:25:27.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:26 smithi154 ceph-mon[35693]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:26 smithi116 ceph-mon[32263]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:26 smithi162 ceph-mon[35761]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:28.786 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:28 smithi116 ceph-mon[32263]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:29.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:28 smithi154 ceph-mon[35693]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:28 smithi162 ceph-mon[35761]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:29.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:25:29.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:25:29.748 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-19T02:25:29.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 -- 172.21.15.116:0/3971310295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 msgr2=0x7f9368101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 --2- 172.21.15.116:0/3971310295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368101db0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f935c009900 tx=0x7f935c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 -- 172.21.15.116:0/3971310295 shutdown_connections 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 --2- 172.21.15.116:0/3971310295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f9368108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 --2- 172.21.15.116:0/3971310295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368101db0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:29.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 --2- 172.21.15.116:0/3971310295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 0x7f9368100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 -- 172.21.15.116:0/3971310295 >> 172.21.15.116:0/3971310295 conn(0x7f93680fbcb0 msgr2=0x7f93680fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 -- 172.21.15.116:0/3971310295 shutdown_connections 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.929+0000 7f9370268640 1 -- 172.21.15.116:0/3971310295 wait complete. 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 Processor -- start 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 -- start start 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 0x7f9368197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f936819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f936810c270 con 0x7f9368101930 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f936810c0f0 con 0x7f93681022f0 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f9370268640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f936810c3f0 con 0x7f9368100740 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:29.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34654/0 (socket says 172.21.15.116:34654) 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 -- 172.21.15.116:0/1981404967 learned_addr learned my addr 172.21.15.116:0/1981404967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936e004640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 0x7f9368197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936e805640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f936819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 -- 172.21.15.116:0/1981404967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 msgr2=0x7f9368197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 0x7f9368197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 -- 172.21.15.116:0/1981404967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 msgr2=0x7f936819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f936819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 -- 172.21.15.116:0/1981404967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936819f6b0 con 0x7f9368101930 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936e805640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f936819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.930+0000 7f936d803640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f935c006010 tx=0x7f935c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.931+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c043020 con 0x7f9368101930 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.931+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936819f940 con 0x7f9368101930 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.931+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f935c0043d0 con 0x7f9368101930 2024-03-19T02:25:29.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.931+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c003b80 con 0x7f9368101930 2024-03-19T02:25:29.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.931+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936819fe90 con 0x7f9368101930 2024-03-19T02:25:29.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.932+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93681a0060 con 0x7f9368101930 2024-03-19T02:25:29.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.933+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f935c046020 con 0x7f9368101930 2024-03-19T02:25:29.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.933+0000 7f935b7fe640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 0x7f93540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:29.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.934+0000 7f936e004640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 0x7f93540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:29.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.934+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f935c0f0f60 con 0x7f9368101930 2024-03-19T02:25:29.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.934+0000 7f936e004640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 0x7f93540a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9368101790 tx=0x7f9364005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:29.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:29.938+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f935c04d020 con 0x7f9368101930 2024-03-19T02:25:30.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.116+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93681a0350 con 0x7f9368101930 2024-03-19T02:25:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.118+0000 7f935b7fe640 1 -- 172.21.15.116:0/1981404967 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f935c0bddc0 con 0x7f9368101930 2024-03-19T02:25:30.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:30.118 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 msgr2=0x7f93540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 0x7f93540a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9368101790 tx=0x7f9364005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 msgr2=0x7f9368197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f935c006010 tx=0x7f935c005e40 comp rx=0 tx=0).stop 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 shutdown_connections 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f93540a2800 0x7f93540a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93681022f0 0x7f936819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9368101930 0x7f9368197fd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 --2- 172.21.15.116:0/1981404967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9368100740 0x7f9368197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 >> 172.21.15.116:0/1981404967 conn(0x7f93680fbcb0 msgr2=0x7f93680fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 shutdown_connections 2024-03-19T02:25:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.120+0000 7f9370268640 1 -- 172.21.15.116:0/1981404967 wait complete. 2024-03-19T02:25:30.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.233+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/4202422184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41075a0 msgr2=0x7fcdc4109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.233+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/4202422184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41075a0 0x7fcdc4109990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009ca0 tx=0x7fcdbc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/4202422184 shutdown_connections 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/4202422184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41075a0 0x7fcdc4109990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/4202422184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4104c70 0x7fcdc4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/4202422184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102340 0x7fcdc4104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/4202422184 >> 172.21.15.116:0/4202422184 conn(0x7fcdc40fbcd0 msgr2=0x7fcdc40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/4202422184 shutdown_connections 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/4202422184 wait complete. 2024-03-19T02:25:30.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 Processor -- start 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- start start 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4102340 0x7fcdc4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc41075a0 0x7fcdc419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410d3b0 con 0x7fcdc4104c70 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410d230 con 0x7fcdc41075a0 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.234+0000 7fcdc8ac0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410d530 con 0x7fcdc4102340 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34674/0 (socket says 172.21.15.116:34674) 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 -- 172.21.15.116:0/2783208896 learned_addr learned my addr 172.21.15.116:0/2783208896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdc2d9d640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc41075a0 0x7fcdc419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 -- 172.21.15.116:0/2783208896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4102340 msgr2=0x7fcdc4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4102340 0x7fcdc4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 -- 172.21.15.116:0/2783208896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc41075a0 msgr2=0x7fcdc419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc41075a0 0x7fcdc419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 -- 172.21.15.116:0/2783208896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc419f6b0 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbbfff640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00c970 tx=0x7fcdac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac007bb0 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac007d50 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc419f9a0 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac018730 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.235+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc419fea0 con 0x7fcdc4104c70 2024-03-19T02:25:30.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.236+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc40fd7c0 con 0x7fcdc4104c70 2024-03-19T02:25:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.237+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fcdac00f450 con 0x7fcdc4104c70 2024-03-19T02:25:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.238+0000 7fcdbb7fe640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 0x7fcdb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.238+0000 7fcdc259c640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 0x7fcdb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.238+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcdac0cf080 con 0x7fcdc4104c70 2024-03-19T02:25:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.238+0000 7fcdc259c640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 0x7fcdb00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400ba80 tx=0x7fcdb4005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.241+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdac014030 con 0x7fcdc4104c70 2024-03-19T02:25:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.416+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdc402d460 con 0x7fcdc4104c70 2024-03-19T02:25:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.418+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/2783208896 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7fcdac09bee0 con 0x7fcdc4104c70 2024-03-19T02:25:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:30.418 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 msgr2=0x7fcdb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 0x7fcdb00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400ba80 tx=0x7fcdb4005ed0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 msgr2=0x7fcdc4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00c970 tx=0x7fcdac00ce30 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 shutdown_connections 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdb00a2800 0x7fcdb00a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc41075a0 0x7fcdc419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4104c70 0x7fcdc4197fd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 --2- 172.21.15.116:0/2783208896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4102340 0x7fcdc4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 >> 172.21.15.116:0/2783208896 conn(0x7fcdc40fbcd0 msgr2=0x7fcdc4102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 shutdown_connections 2024-03-19T02:25:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.420+0000 7fcdc8ac0640 1 -- 172.21.15.116:0/2783208896 wait complete. 2024-03-19T02:25:30.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T02:25:30.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 -- 172.21.15.116:0/1966231260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 msgr2=0x7ff584100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 --2- 172.21.15.116:0/1966231260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584100b60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009900 tx=0x7ff56c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:30.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 -- 172.21.15.116:0/1966231260 shutdown_connections 2024-03-19T02:25:30.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 --2- 172.21.15.116:0/1966231260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5841022f0 0x7ff584108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 --2- 172.21.15.116:0/1966231260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff584101930 0x7ff584101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 --2- 172.21.15.116:0/1966231260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584100b60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 -- 172.21.15.116:0/1966231260 >> 172.21.15.116:0/1966231260 conn(0x7ff5840fbcd0 msgr2=0x7ff5840fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 -- 172.21.15.116:0/1966231260 shutdown_connections 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.536+0000 7ff589f1b640 1 -- 172.21.15.116:0/1966231260 wait complete. 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 Processor -- start 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 -- start start 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff584101930 0x7ff5841980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff58419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c2c0 con 0x7ff584100740 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c140 con 0x7ff584101930 2024-03-19T02:25:30.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff589f1b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c440 con 0x7ff5841022f0 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34690/0 (socket says 172.21.15.116:34690) 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 -- 172.21.15.116:0/833580824 learned_addr learned my addr 172.21.15.116:0/833580824 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff582ffd640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff584101930 0x7ff5841980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff583fff640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff58419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 -- 172.21.15.116:0/833580824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 msgr2=0x7ff58419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff58419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 -- 172.21.15.116:0/833580824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff584101930 msgr2=0x7ff5841980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff584101930 0x7ff5841980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff5837fe640 1 -- 172.21.15.116:0/833580824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58419f750 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.537+0000 7ff583fff640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff58419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff5837fe640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c043020 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff56c004440 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58419f9e0 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c003c20 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.538+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58419ffb0 con 0x7ff584100740 2024-03-19T02:25:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.539+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5841a0180 con 0x7ff584100740 2024-03-19T02:25:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.540+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff56c046020 con 0x7ff584100740 2024-03-19T02:25:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.541+0000 7ff580ff9640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.541+0000 7ff582ffd640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.541+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff56c0f0f90 con 0x7ff584100740 2024-03-19T02:25:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.541+0000 7ff582ffd640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 0x7ff5700a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff584199040 tx=0x7ff574009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.545+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff56c04d020 con 0x7ff584100740 2024-03-19T02:25:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:30 smithi116 ceph-mon[32263]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:30 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1981404967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:30 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2783208896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:30.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:30.759+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff5841a03b0 con 0x7ff584100740 2024-03-19T02:25:31.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:30 smithi154 ceph-mon[35693]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:30 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1981404967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:31.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:30 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2783208896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:30 smithi162 ceph-mon[35761]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:30 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1981404967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:30 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2783208896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:31.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.679+0000 7ff580ff9640 1 -- 172.21.15.116:0/833580824 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7ff56c0bddf0 con 0x7ff584100740 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 msgr2=0x7ff5700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 0x7ff5700a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff584199040 tx=0x7ff574009110 comp rx=0 tx=0).stop 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 msgr2=0x7ff584197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c004040 comp rx=0 tx=0).stop 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 shutdown_connections 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff5700a2800 0x7ff5700a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff58419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff584101930 0x7ff5841980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 --2- 172.21.15.116:0/833580824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584197b80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 >> 172.21.15.116:0/833580824 conn(0x7ff5840fbcd0 msgr2=0x7ff584069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:31.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 shutdown_connections 2024-03-19T02:25:31.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.682+0000 7ff589f1b640 1 -- 172.21.15.116:0/833580824 wait complete. 2024-03-19T02:25:31.695 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-19T02:25:31.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-19T02:25:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3710194708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 msgr2=0x7f4a50102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3710194708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50102ea0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009900 tx=0x7f4a4002bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3710194708 shutdown_connections 2024-03-19T02:25:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3710194708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a50109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3710194708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50102ea0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3710194708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3710194708 >> 172.21.15.116:0/3710194708 conn(0x7f4a500fcdc0 msgr2=0x7f4a500ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3710194708 shutdown_connections 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3710194708 wait complete. 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.806+0000 7f4a57d7e640 1 Processor -- start 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 -- start start 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a5019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010d360 con 0x7f4a50102a20 2024-03-19T02:25:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010d1e0 con 0x7f4a501033e0 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5010d4e0 con 0x7f4a50101830 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55319640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55319640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34718/0 (socket says 172.21.15.116:34718) 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55319640 1 -- 172.21.15.116:0/3029170412 learned_addr learned my addr 172.21.15.116:0/3029170412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a5019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55b1a640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 -- 172.21.15.116:0/3029170412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 msgr2=0x7f4a50197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 -- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 msgr2=0x7f4a50198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 -- 172.21.15.116:0/3029170412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5019f6e0 con 0x7f4a501033e0 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55319640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a55b1a640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a5631b640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a5019f070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009fd0 tx=0x7f4a4800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800ce20 con 0x7f4a501033e0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5019f9d0 con 0x7f4a501033e0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.807+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a48005020 con 0x7f4a501033e0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.808+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48010690 con 0x7f4a501033e0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.808+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5019fed0 con 0x7f4a501033e0 2024-03-19T02:25:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.808+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a50108c60 con 0x7f4a501033e0 2024-03-19T02:25:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.809+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4a480051d0 con 0x7f4a501033e0 2024-03-19T02:25:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.810+0000 7f4a46ffd640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.810+0000 7f4a55b1a640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.810+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4a480cf720 con 0x7f4a501033e0 2024-03-19T02:25:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.810+0000 7f4a55b1a640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0058c0 tx=0x7f4a4c005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:31.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:31.818+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a4809c580 con 0x7f4a501033e0 2024-03-19T02:25:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:31 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:25:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:31 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.030+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f4a5002d380 con 0x7f4a501033e0 2024-03-19T02:25:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:31 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:25:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.682+0000 7f4a46ffd640 1 -- 172.21.15.116:0/3029170412 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f4a4809be80 con 0x7f4a501033e0 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 msgr2=0x7f4a3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0058c0 tx=0x7f4a4c005ec0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 msgr2=0x7f4a5019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a5019f070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009fd0 tx=0x7f4a4800ef10 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 shutdown_connections 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501033e0 0x7f4a5019f070 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a50102a20 0x7f4a50198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 --2- 172.21.15.116:0/3029170412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a50101830 0x7f4a50197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 >> 172.21.15.116:0/3029170412 conn(0x7f4a500fcdc0 msgr2=0x7f4a50105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 shutdown_connections 2024-03-19T02:25:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.685+0000 7f4a57d7e640 1 -- 172.21.15.116:0/3029170412 wait complete. 2024-03-19T02:25:32.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-19T02:25:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 -- 172.21.15.116:0/664404634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 msgr2=0x7f8f2010bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 --2- 172.21.15.116:0/664404634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f2010bb70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009900 tx=0x7f8f1002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 -- 172.21.15.116:0/664404634 shutdown_connections 2024-03-19T02:25:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 --2- 172.21.15.116:0/664404634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f2010bb70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 --2- 172.21.15.116:0/664404634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 --2- 172.21.15.116:0/664404634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 0x7f8f20073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 -- 172.21.15.116:0/664404634 >> 172.21.15.116:0/664404634 conn(0x7f8f200fcdc0 msgr2=0x7f8f200ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.804+0000 7f8f25662640 1 -- 172.21.15.116:0/664404634 shutdown_connections 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 -- 172.21.15.116:0/664404634 wait complete. 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 Processor -- start 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 -- start start 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 0x7f8f2019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010f5c0 con 0x7f8f201096e0 2024-03-19T02:25:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010f440 con 0x7f8f200757e0 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f25662640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2010f740 con 0x7f8f20074200 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34736/0 (socket says 172.21.15.116:34736) 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.805+0000 7f8f1f7fe640 1 -- 172.21.15.116:0/3738758794 learned_addr learned my addr 172.21.15.116:0/3738758794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1e7fc640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 0x7f8f2019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1effd640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 -- 172.21.15.116:0/3738758794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 msgr2=0x7f8f20199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 -- 172.21.15.116:0/3738758794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 msgr2=0x7f8f2019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 0x7f8f2019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 -- 172.21.15.116:0/3738758794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f201a18f0 con 0x7f8f201096e0 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1effd640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:25:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f1f7fe640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009230 tx=0x7f8f1002da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10043020 con 0x7f8f201096e0 2024-03-19T02:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f1002de10 con 0x7f8f201096e0 2024-03-19T02:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f201a1b80 con 0x7f8f201096e0 2024-03-19T02:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10003b50 con 0x7f8f201096e0 2024-03-19T02:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.806+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f201a2150 con 0x7f8f201096e0 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.807+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f201084a0 con 0x7f8f201096e0 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.808+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8f10046020 con 0x7f8f201096e0 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.809+0000 7f8f07fff640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.809+0000 7f8f1effd640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.809+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8f100f0fc0 con 0x7f8f201096e0 2024-03-19T02:25:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.809+0000 7f8f1effd640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 0x7f8f080a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8f2019ab80 tx=0x7f8f18005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:32.813+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f1004d020 con 0x7f8f201096e0 2024-03-19T02:25:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:32 smithi154 ceph-mon[35693]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:32 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:25:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:32 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:32 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3029170412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:32 smithi154 ceph-mon[35693]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:32 smithi116 ceph-mon[32263]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:32 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:25:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:32 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:32 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3029170412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:32 smithi116 ceph-mon[32263]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.025+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8f20062420 con 0x7f8f201096e0 2024-03-19T02:25:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:32 smithi162 ceph-mon[35761]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:32 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/833580824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:25:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:32 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:32 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3029170412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:32 smithi162 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T02:25:33.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.691+0000 7f8f07fff640 1 -- 172.21.15.116:0/3738758794 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f8f100bde20 con 0x7f8f201096e0 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 msgr2=0x7f8f080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 0x7f8f080a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8f2019ab80 tx=0x7f8f18005e70 comp rx=0 tx=0).stop 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 msgr2=0x7f8f201a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009230 tx=0x7f8f1002da10 comp rx=0 tx=0).stop 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 shutdown_connections 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8f080a2800 0x7f8f080a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f201096e0 0x7f8f201a1250 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f200757e0 0x7f8f2019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 --2- 172.21.15.116:0/3738758794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f20074200 0x7f8f20199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 >> 172.21.15.116:0/3738758794 conn(0x7f8f200fcdc0 msgr2=0x7f8f20105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:33.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 shutdown_connections 2024-03-19T02:25:33.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.694+0000 7f8f25662640 1 -- 172.21.15.116:0/3738758794 wait complete. 2024-03-19T02:25:33.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T02:25:33.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2279595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac8101830 msgr2=0x7f4ac8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:33.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2279595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac8101830 0x7f4ac8101c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009900 tx=0x7f4ab802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:33.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2279595982 shutdown_connections 2024-03-19T02:25:33.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2279595982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac81033e0 0x7f4ac8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2279595982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 0x7f4ac8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2279595982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac8101830 0x7f4ac8101c50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2279595982 >> 172.21.15.116:0/2279595982 conn(0x7f4ac80fcda0 msgr2=0x7f4ac80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2279595982 shutdown_connections 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.820+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2279595982 wait complete. 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 Processor -- start 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 -- start start 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 0x7f4ac8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 0x7f4ac8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810d330 con 0x7f4ac81033e0 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810d1b0 con 0x7f4ac8101830 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ace4e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810d4b0 con 0x7f4ac8102a20 2024-03-19T02:25:33.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4acca85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4acca85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34760/0 (socket says 172.21.15.116:34760) 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4acca85640 1 -- 172.21.15.116:0/2500478231 learned_addr learned my addr 172.21.15.116:0/2500478231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ac7fff640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 0x7f4ac8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.821+0000 7f4ac77fe640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 0x7f4ac8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 -- 172.21.15.116:0/2500478231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 msgr2=0x7f4ac8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 0x7f4ac8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 -- 172.21.15.116:0/2500478231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 msgr2=0x7f4ac8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 0x7f4ac8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 -- 172.21.15.116:0/2500478231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac819f6e0 con 0x7f4ac81033e0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ac7fff640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 0x7f4ac8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4acca85640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000eb10 tx=0x7f4ac000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0004b20 con 0x7f4ac81033e0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac0004cc0 con 0x7f4ac81033e0 2024-03-19T02:25:33.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0010690 con 0x7f4ac81033e0 2024-03-19T02:25:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac819f9d0 con 0x7f4ac81033e0 2024-03-19T02:25:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.822+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac819fed0 con 0x7f4ac81033e0 2024-03-19T02:25:33.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.823+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac8108cc0 con 0x7f4ac81033e0 2024-03-19T02:25:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.824+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4ac0005020 con 0x7f4ac81033e0 2024-03-19T02:25:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.825+0000 7f4ac57fa640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 0x7f4ab00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.825+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4ac00cf230 con 0x7f4ac81033e0 2024-03-19T02:25:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.825+0000 7f4ac7fff640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 0x7f4ab00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:33.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.825+0000 7f4ac7fff640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 0x7f4ab00a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8005ab0 tx=0x7f4ab8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:33.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:33.829+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ac009c090 con 0x7f4ac81033e0 2024-03-19T02:25:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-19T02:25:34.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:33 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:25:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:25:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T02:25:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:25:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-19T02:25:34.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-19T02:25:34.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:33 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:25:34.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.042+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f4ac8062450 con 0x7f4ac81033e0 2024-03-19T02:25:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:25:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T02:25:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:25:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:25:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-19T02:25:34.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-19T02:25:34.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:33 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:25:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.697+0000 7f4ac57fa640 1 -- 172.21.15.116:0/2500478231 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f4ac009b990 con 0x7f4ac81033e0 2024-03-19T02:25:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 msgr2=0x7f4ab00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 0x7f4ab00a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8005ab0 tx=0x7f4ab8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 msgr2=0x7f4ac819f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:34.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000eb10 tx=0x7f4ac000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 shutdown_connections 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4ab00a2800 0x7f4ab00a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.700+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac81033e0 0x7f4ac819f070 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.701+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ac8102a20 0x7f4ac8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.701+0000 7f4ace4e8640 1 --2- 172.21.15.116:0/2500478231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac8101830 0x7f4ac8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.701+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 >> 172.21.15.116:0/2500478231 conn(0x7f4ac80fcda0 msgr2=0x7f4ac8105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.701+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 shutdown_connections 2024-03-19T02:25:34.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.701+0000 7f4ace4e8640 1 -- 172.21.15.116:0/2500478231 wait complete. 2024-03-19T02:25:34.718 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-19T02:25:34.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- 172.21.15.116:0/1362019616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664107520 msgr2=0x7fe664109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/1362019616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664107520 0x7fe664109950 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009ca0 tx=0x7fe65c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- 172.21.15.116:0/1362019616 shutdown_connections 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/1362019616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664107520 0x7fe664109950 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/1362019616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe664104bf0 0x7fe664106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/1362019616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6641022c0 0x7fe6641046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- 172.21.15.116:0/1362019616 >> 172.21.15.116:0/1362019616 conn(0x7fe6640fbcb0 msgr2=0x7fe6640fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- 172.21.15.116:0/1362019616 shutdown_connections 2024-03-19T02:25:34.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- 172.21.15.116:0/1362019616 wait complete. 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 Processor -- start 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- start start 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6641022c0 0x7fe664199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe664107520 0x7fe6641a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe66410d370 con 0x7fe664104bf0 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe66410d1f0 con 0x7fe664107520 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.829+0000 7fe66c2b5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe66410d4f0 con 0x7fe6641022c0 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34782/0 (socket says 172.21.15.116:34782) 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 -- 172.21.15.116:0/965413305 learned_addr learned my addr 172.21.15.116:0/965413305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe66a051640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6641022c0 0x7fe664199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:34.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe66a852640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe664107520 0x7fe6641a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 -- 172.21.15.116:0/965413305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6641022c0 msgr2=0x7fe664199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6641022c0 0x7fe664199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 -- 172.21.15.116:0/965413305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe664107520 msgr2=0x7fe6641a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe664107520 0x7fe6641a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 -- 172.21.15.116:0/965413305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6641a18d0 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe669850640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe65400e9f0 tx=0x7fe65400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65400ccc0 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6540054d0 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654010430 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6641a1bc0 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.830+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6641a20c0 con 0x7fe664104bf0 2024-03-19T02:25:34.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.831+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe664109010 con 0x7fe664104bf0 2024-03-19T02:25:34.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.832+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe654005020 con 0x7fe664104bf0 2024-03-19T02:25:34.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.833+0000 7fe65b7fe640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:34.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.833+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe6540cf060 con 0x7fe664104bf0 2024-03-19T02:25:34.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.833+0000 7fe66a051640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:34.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.833+0000 7fe66a051640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 0x7fe6500a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe66419ab60 tx=0x7fe660005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:34.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:34.837+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe65409bec0 con 0x7fe664104bf0 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:25:34.605+0000 7fbadc802700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:active 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T02:25:35.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:active 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-19T02:25:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-19T02:25:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T02:25:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:25:35.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.037+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe664061c60 con 0x7fe6500a2800 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3738758794' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] up:active 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-19T02:25:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-19T02:25:35.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T02:25:35.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:25:35.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:25:35.042+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:25:35.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:25:35.042+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:25:35.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:25:35.042+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:25:35.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:25:35.042+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:25:35.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:25:35.042+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:25:35.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.729+0000 7fe65b7fe640 1 -- 172.21.15.116:0/965413305 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe664061c60 con 0x7fe6500a2800 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 msgr2=0x7fe6500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 0x7fe6500a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe66419ab60 tx=0x7fe660005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 msgr2=0x7fe66419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe65400e9f0 tx=0x7fe65400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 shutdown_connections 2024-03-19T02:25:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe664107520 0x7fe6641a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe664104bf0 0x7fe66419a1d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 --2- 172.21.15.116:0/965413305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6641022c0 0x7fe664199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 >> 172.21.15.116:0/965413305 conn(0x7fe6640fbcb0 msgr2=0x7fe664101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 shutdown_connections 2024-03-19T02:25:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.731+0000 7fe66c2b5640 1 -- 172.21.15.116:0/965413305 wait complete. 2024-03-19T02:25:35.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 -- 172.21.15.116:0/2814030203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac107560 msgr2=0x7ffbac109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 --2- 172.21.15.116:0/2814030203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac107560 0x7ffbac109950 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009ca0 tx=0x7ffba402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 -- 172.21.15.116:0/2814030203 shutdown_connections 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 --2- 172.21.15.116:0/2814030203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac107560 0x7ffbac109950 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 --2- 172.21.15.116:0/2814030203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac104c30 0x7ffbac107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 --2- 172.21.15.116:0/2814030203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac102300 0x7ffbac1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 -- 172.21.15.116:0/2814030203 >> 172.21.15.116:0/2814030203 conn(0x7ffbac0fbcb0 msgr2=0x7ffbac0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 -- 172.21.15.116:0/2814030203 shutdown_connections 2024-03-19T02:25:35.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.857+0000 7ffbb3663640 1 -- 172.21.15.116:0/2814030203 wait complete. 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 Processor -- start 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 -- start start 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac104c30 0x7ffbac197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac107560 0x7ffbac19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac10d350 con 0x7ffbac102300 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac10d1d0 con 0x7ffbac104c30 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb3663640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac10d4d0 con 0x7ffbac107560 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34824/0 (socket says 172.21.15.116:34824) 2024-03-19T02:25:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 -- 172.21.15.116:0/1488547871 learned_addr learned my addr 172.21.15.116:0/1488547871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb0bfe640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac104c30 0x7ffbac197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb1c00640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac107560 0x7ffbac19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 -- 172.21.15.116:0/1488547871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac107560 msgr2=0x7ffbac19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac107560 0x7ffbac19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 -- 172.21.15.116:0/1488547871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac104c30 msgr2=0x7ffbac197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac104c30 0x7ffbac197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 -- 172.21.15.116:0/1488547871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac19f6a0 con 0x7ffbac102300 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.858+0000 7ffbb13ff640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e9f0 tx=0x7ffba800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.859+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800cd80 con 0x7ffbac102300 2024-03-19T02:25:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.859+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffba80108e0 con 0x7ffbac102300 2024-03-19T02:25:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.859+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac19f990 con 0x7ffbac102300 2024-03-19T02:25:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.859+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8018920 con 0x7ffbac102300 2024-03-19T02:25:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.859+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac19fee0 con 0x7ffbac102300 2024-03-19T02:25:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.860+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbac068610 con 0x7ffbac102300 2024-03-19T02:25:35.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.861+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ffba8010a90 con 0x7ffbac102300 2024-03-19T02:25:35.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.861+0000 7ffba27fc640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 0x7ffb980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:35.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.861+0000 7ffbb0bfe640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 0x7ffb980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:35.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.861+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ffba80cef90 con 0x7ffbac102300 2024-03-19T02:25:35.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.862+0000 7ffbb0bfe640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 0x7ffb980a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffbac198f90 tx=0x7ffb9c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:35.866+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffba809b7a0 con 0x7ffbac102300 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:active 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-19T02:25:36.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:35 smithi154 ceph-mon[35693]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:active 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-19T02:25:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:35 smithi116 ceph-mon[32263]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-19T02:25:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.066+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffbac061cf0 con 0x7ffb980a2800 2024-03-19T02:25:36.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.088+0000 7ffba27fc640 1 -- 172.21.15.116:0/1488547871 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffbac061cf0 con 0x7ffb980a2800 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 msgr2=0x7ffb980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 0x7ffb980a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffbac198f90 tx=0x7ffb9c00b410 comp rx=0 tx=0).stop 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 msgr2=0x7ffbac197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e9f0 tx=0x7ffba800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 shutdown_connections 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ffb980a2800 0x7ffb980a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbac107560 0x7ffbac19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbac104c30 0x7ffbac197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 --2- 172.21.15.116:0/1488547871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbac102300 0x7ffbac197a80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 >> 172.21.15.116:0/1488547871 conn(0x7ffbac0fbcb0 msgr2=0x7ffbac102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 shutdown_connections 2024-03-19T02:25:36.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.090+0000 7ffbb3663640 1 -- 172.21.15.116:0/1488547871 wait complete. 2024-03-19T02:25:36.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-19T02:25:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2500478231' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:25:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] up:active 2024-03-19T02:25:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T02:25:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-19T02:25:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-19T02:25:36.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:35 smithi162 ceph-mon[35761]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-19T02:25:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- 172.21.15.116:0/4117732902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8107530 msgr2=0x7fadc8109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/4117732902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8107530 0x7fadc8109960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009ca0 tx=0x7fadc002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- 172.21.15.116:0/4117732902 shutdown_connections 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/4117732902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8107530 0x7fadc8109960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/4117732902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc8104c00 0x7fadc8106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/4117732902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc81022d0 0x7fadc81046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- 172.21.15.116:0/4117732902 >> 172.21.15.116:0/4117732902 conn(0x7fadc80fbcc0 msgr2=0x7fadc80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- 172.21.15.116:0/4117732902 shutdown_connections 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- 172.21.15.116:0/4117732902 wait complete. 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 Processor -- start 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.208+0000 7fadcfdb3640 1 -- start start 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 0x7fadc8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc8107530 0x7fadc819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d360 con 0x7fadc8104c00 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d1e0 con 0x7fadc81022d0 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d4e0 con 0x7fadc8107530 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcdb4f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 0x7fadc8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34846/0 (socket says 172.21.15.116:34846) 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 -- 172.21.15.116:0/3076964 learned_addr learned my addr 172.21.15.116:0/3076964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadce350640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc8107530 0x7fadc819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 -- 172.21.15.116:0/3076964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc8107530 msgr2=0x7fadc819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc8107530 0x7fadc819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 -- 172.21.15.116:0/3076964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 msgr2=0x7fadc8197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 0x7fadc8197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 -- 172.21.15.116:0/3076964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc819f690 con 0x7fadc8104c00 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcdb4f640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 0x7fadc8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:36.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcd34e640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fadb800e9c0 tx=0x7fadb800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb800cd80 con 0x7fadc8104c00 2024-03-19T02:25:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadb80054d0 con 0x7fadc8104c00 2024-03-19T02:25:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8010430 con 0x7fadc8104c00 2024-03-19T02:25:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc819f980 con 0x7fadc8104c00 2024-03-19T02:25:36.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.209+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc819fe80 con 0x7fadc8104c00 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.210+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc80fd7b0 con 0x7fadc8104c00 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.211+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fadb8004fd0 con 0x7fadc8104c00 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.212+0000 7fadbeffd640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 0x7fadb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.212+0000 7fadcdb4f640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 0x7fadb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.212+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fadb80cf070 con 0x7fadc8104c00 2024-03-19T02:25:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.212+0000 7fadcdb4f640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 0x7fadb40a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009fc0 tx=0x7fadc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.217+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fadb809bed0 con 0x7fadc8104c00 2024-03-19T02:25:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.420+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fadc8061c60 con 0x7fadb40a2800 2024-03-19T02:25:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.424+0000 7fadbeffd640 1 -- 172.21.15.116:0/3076964 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fadc8061c60 con 0x7fadb40a2800 2024-03-19T02:25:36.425 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_0/479ec2b2-297d-461d-8f26-3eef92c9c7c9 2024-03-19T02:25:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 msgr2=0x7fadb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 0x7fadb40a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009fc0 tx=0x7fadc4005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 msgr2=0x7fadc8197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fadb800e9c0 tx=0x7fadb800ee80 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 shutdown_connections 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fadb40a2800 0x7fadb40a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc8107530 0x7fadc819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc8104c00 0x7fadc8197fb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 --2- 172.21.15.116:0/3076964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadc81022d0 0x7fadc8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 >> 172.21.15.116:0/3076964 conn(0x7fadc80fbcc0 msgr2=0x7fadc8102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 shutdown_connections 2024-03-19T02:25:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.426+0000 7fadcfdb3640 1 -- 172.21.15.116:0/3076964 wait complete. 2024-03-19T02:25:36.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-19T02:25:36.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- 172.21.15.116:0/2533635435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec101920 msgr2=0x7f99ec101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/2533635435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec101920 0x7f99ec101da0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009900 tx=0x7f99d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- 172.21.15.116:0/2533635435 shutdown_connections 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/2533635435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec1022e0 0x7f99ec108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/2533635435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec101920 0x7f99ec101da0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/2533635435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec100730 0x7f99ec100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- 172.21.15.116:0/2533635435 >> 172.21.15.116:0/2533635435 conn(0x7f99ec0fbcc0 msgr2=0x7f99ec0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- 172.21.15.116:0/2533635435 shutdown_connections 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- 172.21.15.116:0/2533635435 wait complete. 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 Processor -- start 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- start start 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec101920 0x7f99ec198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec1022e0 0x7f99ec19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c240 con 0x7f99ec100730 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c0c0 con 0x7f99ec1022e0 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f2f3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10c3c0 con 0x7f99ec101920 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.544+0000 7f99f0cd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34866/0 (socket says 172.21.15.116:34866) 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 -- 172.21.15.116:0/1453239823 learned_addr learned my addr 172.21.15.116:0/1453239823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f14d8640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec1022e0 0x7f99ec19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99ebfff640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec101920 0x7f99ec198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 -- 172.21.15.116:0/1453239823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec101920 msgr2=0x7f99ec198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec101920 0x7f99ec198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 -- 172.21.15.116:0/1453239823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec1022e0 msgr2=0x7f99ec19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec1022e0 0x7f99ec19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 -- 172.21.15.116:0/1453239823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec19f6e0 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f0cd7640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99e000e9c0 tx=0x7f99e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e000cd80 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec19f9d0 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e00035f0 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0010690 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.545+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec19ff40 con 0x7f99ec100730 2024-03-19T02:25:36.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.546+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec1a0170 con 0x7f99ec100730 2024-03-19T02:25:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.547+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f99e00037a0 con 0x7f99ec100730 2024-03-19T02:25:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.548+0000 7f99e9ffb640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 0x7f99dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.548+0000 7f99ebfff640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 0x7f99dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.548+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f99e00cf030 con 0x7f99ec100730 2024-03-19T02:25:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.548+0000 7f99ebfff640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 0x7f99dc0a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009230 tx=0x7f99d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.552+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99e009be90 con 0x7f99ec100730 2024-03-19T02:25:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.726+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f99e0097e00 con 0x7f99ec100730 2024-03-19T02:25:36.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.763+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f99ec06a570 con 0x7f99dc0a2800 2024-03-19T02:25:36.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.786+0000 7f99e9ffb640 1 -- 172.21.15.116:0/1453239823 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f99ec06a570 con 0x7f99dc0a2800 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 msgr2=0x7f99dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 0x7f99dc0a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009230 tx=0x7f99d8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 msgr2=0x7f99ec197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99e000e9c0 tx=0x7f99e000ee80 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 shutdown_connections 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f99dc0a2800 0x7f99dc0a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec1022e0 0x7f99ec19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99ec101920 0x7f99ec198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 --2- 172.21.15.116:0/1453239823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99ec100730 0x7f99ec197ac0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 >> 172.21.15.116:0/1453239823 conn(0x7f99ec0fbcc0 msgr2=0x7f99ec069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 shutdown_connections 2024-03-19T02:25:36.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.788+0000 7f99f2f3b640 1 -- 172.21.15.116:0/1453239823 wait complete. 2024-03-19T02:25:36.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 -- 172.21.15.116:0/3685391277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034101830 msgr2=0x7f3034101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 --2- 172.21.15.116:0/3685391277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034101830 0x7f3034101c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3024009900 tx=0x7f302402bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 -- 172.21.15.116:0/3685391277 shutdown_connections 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 --2- 172.21.15.116:0/3685391277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 0x7f3034109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 --2- 172.21.15.116:0/3685391277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034102a20 0x7f3034102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 --2- 172.21.15.116:0/3685391277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034101830 0x7f3034101c50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 -- 172.21.15.116:0/3685391277 >> 172.21.15.116:0/3685391277 conn(0x7f30340fcdc0 msgr2=0x7f30340ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:36.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 -- 172.21.15.116:0/3685391277 shutdown_connections 2024-03-19T02:25:36.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.904+0000 7f3038f09640 1 -- 172.21.15.116:0/3685391277 wait complete. 2024-03-19T02:25:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 Processor -- start 2024-03-19T02:25:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 -- start start 2024-03-19T02:25:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034101830 0x7f3034197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 0x7f303419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f303410d360 con 0x7f3034102a20 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f303410d1e0 con 0x7f3034101830 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3038f09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f303410d4e0 con 0x7f30341033e0 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34884/0 (socket says 172.21.15.116:34884) 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 -- 172.21.15.116:0/2014919743 learned_addr learned my addr 172.21.15.116:0/2014919743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3032d9d640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 0x7f303419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f303259c640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034101830 0x7f3034197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 -- 172.21.15.116:0/2014919743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 msgr2=0x7f303419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 0x7f303419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.905+0000 7f3031d9b640 1 -- 172.21.15.116:0/2014919743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034101830 msgr2=0x7f3034197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f3031d9b640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034101830 0x7f3034197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f3031d9b640 1 -- 172.21.15.116:0/2014919743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303419f6c0 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f3031d9b640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f302800e9f0 tx=0x7f302800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302800cd80 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30280108e0 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303419f9b0 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3028018920 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.906+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303419ff00 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.907+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3034108c70 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.908+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f3028010a90 con 0x7f3034102a20 2024-03-19T02:25:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.908+0000 7f30237fe640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 0x7f301c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.909+0000 7f303259c640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 0x7f301c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.909+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f30280cf120 con 0x7f3034102a20 2024-03-19T02:25:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.909+0000 7f303259c640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 0x7f301c0a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3024005ab0 tx=0x7f3024005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:36.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:36.913+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f302809b7d0 con 0x7f3034102a20 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:active 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-19T02:25:37.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:active 2024-03-19T02:25:37.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:36 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:25:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-03-19T02:25:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:active 2024-03-19T02:25:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T02:25:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:active 2024-03-19T02:25:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:36 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:25:37.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.117+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3034061cf0 con 0x7f301c0a2890 2024-03-19T02:25:37.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.121+0000 7f30237fe640 1 -- 172.21.15.116:0/2014919743 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3034061cf0 con 0x7f301c0a2890 2024-03-19T02:25:37.121 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_1/4ac05b26-83f8-42ac-89ab-f6275022c9cd 2024-03-19T02:25:37.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 msgr2=0x7f301c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 0x7f301c0a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3024005ab0 tx=0x7f3024005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 msgr2=0x7f3034197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f302800e9f0 tx=0x7f302800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 shutdown_connections 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f301c0a2890 0x7f301c0a4d50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30341033e0 0x7f303419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3034102a20 0x7f3034197fe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 --2- 172.21.15.116:0/2014919743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3034101830 0x7f3034197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 >> 172.21.15.116:0/2014919743 conn(0x7f30340fcdc0 msgr2=0x7f3034105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 shutdown_connections 2024-03-19T02:25:37.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.123+0000 7f3038f09640 1 -- 172.21.15.116:0/2014919743 wait complete. 2024-03-19T02:25:37.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] up:active 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:25:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:25:37.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-03-19T02:25:37.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T02:25:37.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-19T02:25:37.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] up:active 2024-03-19T02:25:37.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:36 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 -- 172.21.15.116:0/126821350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f98102a30 msgr2=0x7f7f98102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 --2- 172.21.15.116:0/126821350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f98102a30 0x7f7f98102eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009900 tx=0x7f7f8802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 -- 172.21.15.116:0/126821350 shutdown_connections 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 --2- 172.21.15.116:0/126821350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981033f0 0x7f7f98109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 --2- 172.21.15.116:0/126821350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f98102a30 0x7f7f98102eb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 --2- 172.21.15.116:0/126821350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f98101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 -- 172.21.15.116:0/126821350 >> 172.21.15.116:0/126821350 conn(0x7f7f980fcdd0 msgr2=0x7f7f980ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 -- 172.21.15.116:0/126821350 shutdown_connections 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.235+0000 7f7f9f485640 1 -- 172.21.15.116:0/126821350 wait complete. 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 Processor -- start 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 -- start start 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 0x7f7f98073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f981033f0 0x7f7f98074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810d350 con 0x7f7f98101840 2024-03-19T02:25:37.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810d1d0 con 0x7f7f98102a30 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9810d4d0 con 0x7f7f981033f0 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34912/0 (socket says 172.21.15.116:34912) 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 -- 172.21.15.116:0/3761563655 learned_addr learned my addr 172.21.15.116:0/3761563655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9ca20640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 0x7f7f98073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9da22640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f981033f0 0x7f7f98074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 -- 172.21.15.116:0/3761563655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f981033f0 msgr2=0x7f7f98074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f981033f0 0x7f7f98074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 -- 172.21.15.116:0/3761563655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 msgr2=0x7f7f98073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 0x7f7f98073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 -- 172.21.15.116:0/3761563655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f980749e0 con 0x7f7f98101840 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9ca20640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 0x7f7f98073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9d221640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000ebc0 tx=0x7f7f9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90003970 con 0x7f7f98101840 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.236+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f981a1ba0 con 0x7f7f98101840 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.237+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f90004030 con 0x7f7f98101840 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.237+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900196e0 con 0x7f7f98101840 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.237+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f981a2110 con 0x7f7f98101840 2024-03-19T02:25:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.238+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f98108bd0 con 0x7f7f98101840 2024-03-19T02:25:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.239+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7f900041e0 con 0x7f7f98101840 2024-03-19T02:25:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.239+0000 7f7f8e7fc640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 0x7f7f840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.239+0000 7f7f9ca20640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 0x7f7f840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.239+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7f900cf730 con 0x7f7f98101840 2024-03-19T02:25:37.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.240+0000 7f7f9ca20640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 0x7f7f840a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009230 tx=0x7f7f88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.243+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f9009c430 con 0x7f7f98101840 2024-03-19T02:25:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.456+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f9802d420 con 0x7f7f98101840 2024-03-19T02:25:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.457+0000 7f7f8e7fc640 1 -- 172.21.15.116:0/3761563655 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f7f9009bd30 con 0x7f7f98101840 2024-03-19T02:25:37.458 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:37.458 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:25:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 msgr2=0x7f7f840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 0x7f7f840a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009230 tx=0x7f7f88005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 msgr2=0x7f7f980755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000ebc0 tx=0x7f7f9000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 shutdown_connections 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7f840a2890 0x7f7f840a4d50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f981033f0 0x7f7f98074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f98102a30 0x7f7f98073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 --2- 172.21.15.116:0/3761563655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f98101840 0x7f7f980755d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 >> 172.21.15.116:0/3761563655 conn(0x7f7f980fcdd0 msgr2=0x7f7f98105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 shutdown_connections 2024-03-19T02:25:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.459+0000 7f7f9f485640 1 -- 172.21.15.116:0/3761563655 wait complete. 2024-03-19T02:25:37.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:25:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- 172.21.15.116:0/1233762801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4107580 msgr2=0x7febf4109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- 172.21.15.116:0/1233762801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4107580 0x7febf4109970 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7febec009ca0 tx=0x7febec02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- 172.21.15.116:0/1233762801 shutdown_connections 2024-03-19T02:25:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- 172.21.15.116:0/1233762801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4107580 0x7febf4109970 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- 172.21.15.116:0/1233762801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4104c50 0x7febf4107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- 172.21.15.116:0/1233762801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 0x7febf4104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- 172.21.15.116:0/1233762801 >> 172.21.15.116:0/1233762801 conn(0x7febf40fbcb0 msgr2=0x7febf40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- 172.21.15.116:0/1233762801 shutdown_connections 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- 172.21.15.116:0/1233762801 wait complete. 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 Processor -- start 2024-03-19T02:25:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- start start 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 0x7febf4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107580 0x7febf419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febf410d390 con 0x7febf4104c50 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febf410d210 con 0x7febf4102320 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febf9fd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febf410d510 con 0x7febf4107580 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.572+0000 7febebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34928/0 (socket says 172.21.15.116:34928) 2024-03-19T02:25:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 -- 172.21.15.116:0/4163933166 learned_addr learned my addr 172.21.15.116:0/4163933166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf3fff640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107580 0x7febf419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf37fe640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 0x7febf4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 -- 172.21.15.116:0/4163933166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107580 msgr2=0x7febf419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107580 0x7febf419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 -- 172.21.15.116:0/4163933166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 msgr2=0x7febf4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 0x7febf4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 -- 172.21.15.116:0/4163933166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf419f670 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febebfff640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e9f0 tx=0x7febdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc00ccc0 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febdc0054d0 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf419f960 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc010430 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.573+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf419fe60 con 0x7febf4104c50 2024-03-19T02:25:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.574+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf40fd7a0 con 0x7febf4104c50 2024-03-19T02:25:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.575+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7febdc005020 con 0x7febf4104c50 2024-03-19T02:25:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.575+0000 7febf17fa640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 0x7febe00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.576+0000 7febf37fe640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 0x7febe00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.576+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7febdc0cf220 con 0x7febf4104c50 2024-03-19T02:25:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.576+0000 7febf37fe640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 0x7febe00a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7febe4009bd0 tx=0x7febe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.580+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febdc09bec0 con 0x7febf4104c50 2024-03-19T02:25:37.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.761+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febf402d400 con 0x7febf4104c50 2024-03-19T02:25:37.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.763+0000 7febf17fa640 1 -- 172.21.15.116:0/4163933166 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7febdc09b7c0 con 0x7febf4104c50 2024-03-19T02:25:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:37.764 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:37.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 msgr2=0x7febe00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 0x7febe00a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7febe4009bd0 tx=0x7febe4005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 msgr2=0x7febf4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e9f0 tx=0x7febdc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 shutdown_connections 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7febe00a2890 0x7febe00a4d50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf4107580 0x7febf419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf4104c50 0x7febf4197fe0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 --2- 172.21.15.116:0/4163933166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf4102320 0x7febf4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 >> 172.21.15.116:0/4163933166 conn(0x7febf40fbcb0 msgr2=0x7febf4102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 shutdown_connections 2024-03-19T02:25:37.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.765+0000 7febf9fd4640 1 -- 172.21.15.116:0/4163933166 wait complete. 2024-03-19T02:25:37.777 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T02:25:37.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:25:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3248097991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098100740 msgr2=0x7fc098100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3248097991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098100740 0x7fc098100b60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009900 tx=0x7fc08c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3248097991 shutdown_connections 2024-03-19T02:25:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3248097991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0981022f0 0x7fc098108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3248097991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098101930 0x7fc098101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3248097991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc098100740 0x7fc098100b60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.877+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3248097991 >> 172.21.15.116:0/3248097991 conn(0x7fc0980fbcb0 msgr2=0x7fc0980fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3248097991 shutdown_connections 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3248097991 wait complete. 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 Processor -- start 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- start start 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 0x7fc098197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc098101930 0x7fc098197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc09810d390 con 0x7fc0981022f0 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc09810d210 con 0x7fc098100740 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.878+0000 7fc09cb1b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc09810d510 con 0x7fc098101930 2024-03-19T02:25:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53456/0 (socket says 172.21.15.116:53456) 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 -- 172.21.15.116:0/3648438555 learned_addr learned my addr 172.21.15.116:0/3648438555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc095d9b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc098101930 0x7fc098197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc09659c640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 0x7fc098197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 -- 172.21.15.116:0/3648438555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc098101930 msgr2=0x7fc098197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc098101930 0x7fc098197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 -- 172.21.15.116:0/3648438555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 msgr2=0x7fc098197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 0x7fc098197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 -- 172.21.15.116:0/3648438555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09819f6c0 con 0x7fc0981022f0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc09659c640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 0x7fc098197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc096d9d640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc09000eb10 tx=0x7fc09000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090004b20 con 0x7fc0981022f0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc090004cc0 con 0x7fc0981022f0 2024-03-19T02:25:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09819f9b0 con 0x7fc0981022f0 2024-03-19T02:25:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090010690 con 0x7fc0981022f0 2024-03-19T02:25:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.879+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09819ff00 con 0x7fc0981022f0 2024-03-19T02:25:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.880+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc098107a80 con 0x7fc0981022f0 2024-03-19T02:25:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.881+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc090005020 con 0x7fc0981022f0 2024-03-19T02:25:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.882+0000 7fc0877fe640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 0x7fc0800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.882+0000 7fc09659c640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 0x7fc0800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.882+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc0900cf4f0 con 0x7fc0981022f0 2024-03-19T02:25:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.882+0000 7fc09659c640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 0x7fc0800a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc08c005ab0 tx=0x7fc08c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:37.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:37.886+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc09009c1f0 con 0x7fc0981022f0 2024-03-19T02:25:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:37 smithi154 ceph-mon[35693]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:37 smithi154 ceph-mon[35693]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:37 smithi154 ceph-mon[35693]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:25:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:37 smithi154 ceph-mon[35693]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3761563655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:37 smithi116 ceph-mon[32263]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:37 smithi116 ceph-mon[32263]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:37 smithi116 ceph-mon[32263]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:25:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:37 smithi116 ceph-mon[32263]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3761563655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.056+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc098061c60 con 0x7fc0800a2890 2024-03-19T02:25:38.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.066+0000 7fc0877fe640 1 -- 172.21.15.116:0/3648438555 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340356 (secure 0 0 0) 0x7fc098061c60 con 0x7fc0800a2890 2024-03-19T02:25:38.071 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:38.073 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:25:38.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 msgr2=0x7fc0800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 0x7fc0800a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc08c005ab0 tx=0x7fc08c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 msgr2=0x7fc09819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc09000eb10 tx=0x7fc09000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 shutdown_connections 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc0800a2890 0x7fc0800a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0981022f0 0x7fc09819f050 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc098101930 0x7fc098197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 --2- 172.21.15.116:0/3648438555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc098100740 0x7fc098197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 >> 172.21.15.116:0/3648438555 conn(0x7fc0980fbcb0 msgr2=0x7fc098104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 shutdown_connections 2024-03-19T02:25:38.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.070+0000 7fc09cb1b640 1 -- 172.21.15.116:0/3648438555 wait complete. 2024-03-19T02:25:38.081 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-19T02:25:37.600263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600535,"num_objects":87,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":146,"num_read_kb":128,"num_write":358,"num_write_kb":3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371148,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570676,"statfs":{"total":1151940427776,"available":1151560372224,"internally_reserved":0,"allocated":13991936,"data_stored":7587385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5527,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":129,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003629"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247463+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:25:13.247463+0000","last_peered":"2024-03-19T02:25:13.247463+0000","last_clean":"2024-03-19T02:25:13.247463+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:25:13.247463+0000","last_undegraded":"2024-03-19T02:25:13.247463+0000","last_fullsized":"2024-03-19T02:25:13.247463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522944+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:25:13.522944+0000","last_peered":"2024-03-19T02:25:13.522944+0000","last_clean":"2024-03-19T02:25:13.522944+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:25:13.522944+0000","last_undegraded":"2024-03-19T02:25:13.522944+0000","last_fullsized":"2024-03-19T02:25:13.522944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522440+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:25:13.522440+0000","last_peered":"2024-03-19T02:25:13.522440+0000","last_clean":"2024-03-19T02:25:13.522440+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:25:13.522440+0000","last_undegraded":"2024-03-19T02:25:13.522440+0000","last_fullsized":"2024-03-19T02:25:13.522440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523473+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:25:13.523473+0000","last_peered":"2024-03-19T02:25:13.523473+0000","last_clean":"2024-03-19T02:25:13.523473+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:25:13.523473+0000","last_undegraded":"2024-03-19T02:25:13.523473+0000","last_fullsized":"2024-03-19T02:25:13.523473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380625+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:25:13.380625+0000","last_peered":"2024-03-19T02:25:13.380625+0000","last_clean":"2024-03-19T02:25:13.380625+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:25:13.380625+0000","last_undegraded":"2024-03-19T02:25:13.380625+0000","last_fullsized":"2024-03-19T02:25:13.380625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522906+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:25:13.522906+0000","last_peered":"2024-03-19T02:25:13.522906+0000","last_clean":"2024-03-19T02:25:13.522906+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:25:13.522906+0000","last_undegraded":"2024-03-19T02:25:13.522906+0000","last_fullsized":"2024-03-19T02:25:13.522906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523815+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:25:13.523815+0000","last_peered":"2024-03-19T02:25:13.523815+0000","last_clean":"2024-03-19T02:25:13.523815+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:25:13.523815+0000","last_undegraded":"2024-03-19T02:25:13.523815+0000","last_fullsized":"2024-03-19T02:25:13.523815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522103+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:25:13.522103+0000","last_peered":"2024-03-19T02:25:13.522103+0000","last_clean":"2024-03-19T02:25:13.522103+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:25:13.522103+0000","last_undegraded":"2024-03-19T02:25:13.522103+0000","last_fullsized":"2024-03-19T02:25:13.522103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204384+0000","last_change":"2024-03-19T02:25:08.201590+0000","last_active":"2024-03-19T02:25:13.204384+0000","last_peered":"2024-03-19T02:25:13.204384+0000","last_clean":"2024-03-19T02:25:13.204384+0000","last_became_active":"2024-03-19T02:25:08.201274+0000","last_became_peered":"2024-03-19T02:25:08.201274+0000","last_unstale":"2024-03-19T02:25:13.204384+0000","last_undegraded":"2024-03-19T02:25:13.204384+0000","last_fullsized":"2024-03-19T02:25:13.204384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:43:25.989358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523575+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:25:13.523575+0000","last_peered":"2024-03-19T02:25:13.523575+0000","last_clean":"2024-03-19T02:25:13.523575+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:25:13.523575+0000","last_undegraded":"2024-03-19T02:25:13.523575+0000","last_fullsized":"2024-03-19T02:25:13.523575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523801+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:25:13.523801+0000","last_peered":"2024-03-19T02:25:13.523801+0000","last_clean":"2024-03-19T02:25:13.523801+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:25:13.523801+0000","last_undegraded":"2024-03-19T02:25:13.523801+0000","last_fullsized":"2024-03-19T02:25:13.523801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523373+0000","last_change":"2024-03-19T02:25:08.212302+0000","last_active":"2024-03-19T02:25:13.523373+0000","last_peered":"2024-03-19T02:25:13.523373+0000","last_clean":"2024-03-19T02:25:13.523373+0000","last_became_active":"2024-03-19T02:25:08.210842+0000","last_became_peered":"2024-03-19T02:25:08.210842+0000","last_unstale":"2024-03-19T02:25:13.523373+0000","last_undegraded":"2024-03-19T02:25:13.523373+0000","last_fullsized":"2024-03-19T02:25:13.523373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:54:44.550780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203680+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:25:13.203680+0000","last_peered":"2024-03-19T02:25:13.203680+0000","last_clean":"2024-03-19T02:25:13.203680+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:25:13.203680+0000","last_undegraded":"2024-03-19T02:25:13.203680+0000","last_fullsized":"2024-03-19T02:25:13.203680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522096+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:25:13.522096+0000","last_peered":"2024-03-19T02:25:13.522096+0000","last_clean":"2024-03-19T02:25:13.522096+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:25:13.522096+0000","last_undegraded":"2024-03-19T02:25:13.522096+0000","last_fullsized":"2024-03-19T02:25:13.522096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522284+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:25:13.522284+0000","last_peered":"2024-03-19T02:25:13.522284+0000","last_clean":"2024-03-19T02:25:13.522284+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:25:13.522284+0000","last_undegraded":"2024-03-19T02:25:13.522284+0000","last_fullsized":"2024-03-19T02:25:13.522284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203669+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:25:13.203669+0000","last_peered":"2024-03-19T02:25:13.203669+0000","last_clean":"2024-03-19T02:25:13.203669+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:25:13.203669+0000","last_undegraded":"2024-03-19T02:25:13.203669+0000","last_fullsized":"2024-03-19T02:25:13.203669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247288+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:25:13.247288+0000","last_peered":"2024-03-19T02:25:13.247288+0000","last_clean":"2024-03-19T02:25:13.247288+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:25:13.247288+0000","last_undegraded":"2024-03-19T02:25:13.247288+0000","last_fullsized":"2024-03-19T02:25:13.247288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523648+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:25:13.523648+0000","last_peered":"2024-03-19T02:25:13.523648+0000","last_clean":"2024-03-19T02:25:13.523648+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:25:13.523648+0000","last_undegraded":"2024-03-19T02:25:13.523648+0000","last_fullsized":"2024-03-19T02:25:13.523648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523139+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:25:13.523139+0000","last_peered":"2024-03-19T02:25:13.523139+0000","last_clean":"2024-03-19T02:25:13.523139+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:25:13.523139+0000","last_undegraded":"2024-03-19T02:25:13.523139+0000","last_fullsized":"2024-03-19T02:25:13.523139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522816+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:25:13.522816+0000","last_peered":"2024-03-19T02:25:13.522816+0000","last_clean":"2024-03-19T02:25:13.522816+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:25:13.522816+0000","last_undegraded":"2024-03-19T02:25:13.522816+0000","last_fullsized":"2024-03-19T02:25:13.522816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708487+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:25:33.708487+0000","last_peered":"2024-03-19T02:25:33.708487+0000","last_clean":"2024-03-19T02:25:33.708487+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:25:33.708487+0000","last_undegraded":"2024-03-19T02:25:33.708487+0000","last_fullsized":"2024-03-19T02:25:33.708487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522486+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:25:13.522486+0000","last_peered":"2024-03-19T02:25:13.522486+0000","last_clean":"2024-03-19T02:25:13.522486+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:25:13.522486+0000","last_undegraded":"2024-03-19T02:25:13.522486+0000","last_fullsized":"2024-03-19T02:25:13.522486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522705+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:25:13.522705+0000","last_peered":"2024-03-19T02:25:13.522705+0000","last_clean":"2024-03-19T02:25:13.522705+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:25:13.522705+0000","last_undegraded":"2024-03-19T02:25:13.522705+0000","last_fullsized":"2024-03-19T02:25:13.522705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522943+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:25:13.522943+0000","last_peered":"2024-03-19T02:25:13.522943+0000","last_clean":"2024-03-19T02:25:13.522943+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:25:13.522943+0000","last_undegraded":"2024-03-19T02:25:13.522943+0000","last_fullsized":"2024-03-19T02:25:13.522943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523354+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:25:13.523354+0000","last_peered":"2024-03-19T02:25:13.523354+0000","last_clean":"2024-03-19T02:25:13.523354+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:25:13.523354+0000","last_undegraded":"2024-03-19T02:25:13.523354+0000","last_fullsized":"2024-03-19T02:25:13.523354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523238+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:25:13.523238+0000","last_peered":"2024-03-19T02:25:13.523238+0000","last_clean":"2024-03-19T02:25:13.523238+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:25:13.523238+0000","last_undegraded":"2024-03-19T02:25:13.523238+0000","last_fullsized":"2024-03-19T02:25:13.523238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:36.680062+0000","last_change":"2024-03-19T02:25:07.648715+0000","last_active":"2024-03-19T02:25:36.680062+0000","last_peered":"2024-03-19T02:25:36.680062+0000","last_clean":"2024-03-19T02:25:36.680062+0000","last_became_active":"2024-03-19T02:25:07.646184+0000","last_became_peered":"2024-03-19T02:25:07.646184+0000","last_unstale":"2024-03-19T02:25:36.680062+0000","last_undegraded":"2024-03-19T02:25:36.680062+0000","last_fullsized":"2024-03-19T02:25:36.680062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:04.444659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203903+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:25:13.203903+0000","last_peered":"2024-03-19T02:25:13.203903+0000","last_clean":"2024-03-19T02:25:13.203903+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:25:13.203903+0000","last_undegraded":"2024-03-19T02:25:13.203903+0000","last_fullsized":"2024-03-19T02:25:13.203903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522484+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:25:13.522484+0000","last_peered":"2024-03-19T02:25:13.522484+0000","last_clean":"2024-03-19T02:25:13.522484+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:25:13.522484+0000","last_undegraded":"2024-03-19T02:25:13.522484+0000","last_fullsized":"2024-03-19T02:25:13.522484+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.728255+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:25:35.728255+0000","last_peered":"2024-03-19T02:25:35.728255+0000","last_clean":"2024-03-19T02:25:35.728255+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:25:35.728255+0000","last_undegraded":"2024-03-19T02:25:35.728255+0000","last_fullsized":"2024-03-19T02:25:35.728255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207032+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:25:13.207032+0000","last_peered":"2024-03-19T02:25:13.207032+0000","last_clean":"2024-03-19T02:25:13.207032+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:25:13.207032+0000","last_undegraded":"2024-03-19T02:25:13.207032+0000","last_fullsized":"2024-03-19T02:25:13.207032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523141+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:25:13.523141+0000","last_peered":"2024-03-19T02:25:13.523141+0000","last_clean":"2024-03-19T02:25:13.523141+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:25:13.523141+0000","last_undegraded":"2024-03-19T02:25:13.523141+0000","last_fullsized":"2024-03-19T02:25:13.523141+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.697666+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:25:32.697666+0000","last_peered":"2024-03-19T02:25:32.697666+0000","last_clean":"2024-03-19T02:25:32.697666+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:25:32.697666+0000","last_undegraded":"2024-03-19T02:25:32.697666+0000","last_fullsized":"2024-03-19T02:25:32.697666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380659+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:25:13.380659+0000","last_peered":"2024-03-19T02:25:13.380659+0000","last_clean":"2024-03-19T02:25:13.380659+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:25:13.380659+0000","last_undegraded":"2024-03-19T02:25:13.380659+0000","last_fullsized":"2024-03-19T02:25:13.380659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522918+0000","last_change":"2024-03-19T02:25:12.217847+0000","last_active":"2024-03-19T02:25:13.522918+0000","last_peered":"2024-03-19T02:25:13.522918+0000","last_clean":"2024-03-19T02:25:13.522918+0000","last_became_active":"2024-03-19T02:25:12.217350+0000","last_became_peered":"2024-03-19T02:25:12.217350+0000","last_unstale":"2024-03-19T02:25:13.522918+0000","last_undegraded":"2024-03-19T02:25:13.522918+0000","last_fullsized":"2024-03-19T02:25:13.522918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:22.562971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522874+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:25:13.522874+0000","last_peered":"2024-03-19T02:25:13.522874+0000","last_clean":"2024-03-19T02:25:13.522874+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:25:13.522874+0000","last_undegraded":"2024-03-19T02:25:13.522874+0000","last_fullsized":"2024-03-19T02:25:13.522874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206955+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:25:13.206955+0000","last_peered":"2024-03-19T02:25:13.206955+0000","last_clean":"2024-03-19T02:25:13.206955+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:25:13.206955+0000","last_undegraded":"2024-03-19T02:25:13.206955+0000","last_fullsized":"2024-03-19T02:25:13.206955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247361+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:25:13.247361+0000","last_peered":"2024-03-19T02:25:13.247361+0000","last_clean":"2024-03-19T02:25:13.247361+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:25:13.247361+0000","last_undegraded":"2024-03-19T02:25:13.247361+0000","last_fullsized":"2024-03-19T02:25:13.247361+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708552+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:25:33.708552+0000","last_peered":"2024-03-19T02:25:33.708552+0000","last_clean":"2024-03-19T02:25:33.708552+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:25:33.708552+0000","last_undegraded":"2024-03-19T02:25:33.708552+0000","last_fullsized":"2024-03-19T02:25:33.708552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522214+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:25:13.522214+0000","last_peered":"2024-03-19T02:25:13.522214+0000","last_clean":"2024-03-19T02:25:13.522214+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:25:13.522214+0000","last_undegraded":"2024-03-19T02:25:13.522214+0000","last_fullsized":"2024-03-19T02:25:13.522214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522845+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:25:13.522845+0000","last_peered":"2024-03-19T02:25:13.522845+0000","last_clean":"2024-03-19T02:25:13.522845+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:25:13.522845+0000","last_undegraded":"2024-03-19T02:25:13.522845+0000","last_fullsized":"2024-03-19T02:25:13.522845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.696034+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:25:32.696034+0000","last_peered":"2024-03-19T02:25:32.696034+0000","last_clean":"2024-03-19T02:25:32.696034+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:25:32.696034+0000","last_undegraded":"2024-03-19T02:25:32.696034+0000","last_fullsized":"2024-03-19T02:25:32.696034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522955+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:25:13.522955+0000","last_peered":"2024-03-19T02:25:13.522955+0000","last_clean":"2024-03-19T02:25:13.522955+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:25:13.522955+0000","last_undegraded":"2024-03-19T02:25:13.522955+0000","last_fullsized":"2024-03-19T02:25:13.522955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522897+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:25:13.522897+0000","last_peered":"2024-03-19T02:25:13.522897+0000","last_clean":"2024-03-19T02:25:13.522897+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:25:13.522897+0000","last_undegraded":"2024-03-19T02:25:13.522897+0000","last_fullsized":"2024-03-19T02:25:13.522897+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695407+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:25:32.695407+0000","last_peered":"2024-03-19T02:25:32.695407+0000","last_clean":"2024-03-19T02:25:32.695407+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:25:32.695407+0000","last_undegraded":"2024-03-19T02:25:32.695407+0000","last_fullsized":"2024-03-19T02:25:32.695407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204281+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:25:13.204281+0000","last_peered":"2024-03-19T02:25:13.204281+0000","last_clean":"2024-03-19T02:25:13.204281+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:25:13.204281+0000","last_undegraded":"2024-03-19T02:25:13.204281+0000","last_fullsized":"2024-03-19T02:25:13.204281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523286+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:25:13.523286+0000","last_peered":"2024-03-19T02:25:13.523286+0000","last_clean":"2024-03-19T02:25:13.523286+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:25:13.523286+0000","last_undegraded":"2024-03-19T02:25:13.523286+0000","last_fullsized":"2024-03-19T02:25:13.523286+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.727158+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:25:35.727158+0000","last_peered":"2024-03-19T02:25:35.727158+0000","last_clean":"2024-03-19T02:25:35.727158+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:25:35.727158+0000","last_undegraded":"2024-03-19T02:25:35.727158+0000","last_fullsized":"2024-03-19T02:25:35.727158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523461+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:25:13.523461+0000","last_peered":"2024-03-19T02:25:13.523461+0000","last_clean":"2024-03-19T02:25:13.523461+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:25:13.523461+0000","last_undegraded":"2024-03-19T02:25:13.523461+0000","last_fullsized":"2024-03-19T02:25:13.523461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523330+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:25:13.523330+0000","last_peered":"2024-03-19T02:25:13.523330+0000","last_clean":"2024-03-19T02:25:13.523330+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:25:13.523330+0000","last_undegraded":"2024-03-19T02:25:13.523330+0000","last_fullsized":"2024-03-19T02:25:13.523330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695359+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:25:32.695359+0000","last_peered":"2024-03-19T02:25:32.695359+0000","last_clean":"2024-03-19T02:25:32.695359+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:25:32.695359+0000","last_undegraded":"2024-03-19T02:25:32.695359+0000","last_fullsized":"2024-03-19T02:25:32.695359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204123+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:25:13.204123+0000","last_peered":"2024-03-19T02:25:13.204123+0000","last_clean":"2024-03-19T02:25:13.204123+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:25:13.204123+0000","last_undegraded":"2024-03-19T02:25:13.204123+0000","last_fullsized":"2024-03-19T02:25:13.204123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523217+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:25:13.523217+0000","last_peered":"2024-03-19T02:25:13.523217+0000","last_clean":"2024-03-19T02:25:13.523217+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:25:13.523217+0000","last_undegraded":"2024-03-19T02:25:13.523217+0000","last_fullsized":"2024-03-19T02:25:13.523217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.696591+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:25:32.696591+0000","last_peered":"2024-03-19T02:25:32.696591+0000","last_clean":"2024-03-19T02:25:32.696591+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:25:32.696591+0000","last_undegraded":"2024-03-19T02:25:32.696591+0000","last_fullsized":"2024-03-19T02:25:32.696591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380880+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:25:13.380880+0000","last_peered":"2024-03-19T02:25:13.380880+0000","last_clean":"2024-03-19T02:25:13.380880+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:25:13.380880+0000","last_undegraded":"2024-03-19T02:25:13.380880+0000","last_fullsized":"2024-03-19T02:25:13.380880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207381+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:25:13.207381+0000","last_peered":"2024-03-19T02:25:13.207381+0000","last_clean":"2024-03-19T02:25:13.207381+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:25:13.207381+0000","last_undegraded":"2024-03-19T02:25:13.207381+0000","last_fullsized":"2024-03-19T02:25:13.207381+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708508+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:25:33.708508+0000","last_peered":"2024-03-19T02:25:33.708508+0000","last_clean":"2024-03-19T02:25:33.708508+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:25:33.708508+0000","last_undegraded":"2024-03-19T02:25:33.708508+0000","last_fullsized":"2024-03-19T02:25:33.708508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203754+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:25:13.203754+0000","last_peered":"2024-03-19T02:25:13.203754+0000","last_clean":"2024-03-19T02:25:13.203754+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:25:13.203754+0000","last_undegraded":"2024-03-19T02:25:13.203754+0000","last_fullsized":"2024-03-19T02:25:13.203754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247064+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:25:13.247064+0000","last_peered":"2024-03-19T02:25:13.247064+0000","last_clean":"2024-03-19T02:25:13.247064+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:25:13.247064+0000","last_undegraded":"2024-03-19T02:25:13.247064+0000","last_fullsized":"2024-03-19T02:25:13.247064+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695994+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:25:32.695994+0000","last_peered":"2024-03-19T02:25:32.695994+0000","last_clean":"2024-03-19T02:25:32.695994+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:25:32.695994+0000","last_undegraded":"2024-03-19T02:25:32.695994+0000","last_fullsized":"2024-03-19T02:25:32.695994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523811+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:25:13.523811+0000","last_peered":"2024-03-19T02:25:13.523811+0000","last_clean":"2024-03-19T02:25:13.523811+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:25:13.523811+0000","last_undegraded":"2024-03-19T02:25:13.523811+0000","last_fullsized":"2024-03-19T02:25:13.523811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523108+0000","last_change":"2024-03-19T02:25:12.228992+0000","last_active":"2024-03-19T02:25:13.523108+0000","last_peered":"2024-03-19T02:25:13.523108+0000","last_clean":"2024-03-19T02:25:13.523108+0000","last_became_active":"2024-03-19T02:25:12.228457+0000","last_became_peered":"2024-03-19T02:25:12.228457+0000","last_unstale":"2024-03-19T02:25:13.523108+0000","last_undegraded":"2024-03-19T02:25:13.523108+0000","last_fullsized":"2024-03-19T02:25:13.523108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:14:04.599952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.710979+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:25:34.710979+0000","last_peered":"2024-03-19T02:25:34.710979+0000","last_clean":"2024-03-19T02:25:34.710979+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:25:34.710979+0000","last_undegraded":"2024-03-19T02:25:34.710979+0000","last_fullsized":"2024-03-19T02:25:34.710979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523188+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:25:13.523188+0000","last_peered":"2024-03-19T02:25:13.523188+0000","last_clean":"2024-03-19T02:25:13.523188+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:25:13.523188+0000","last_undegraded":"2024-03-19T02:25:13.523188+0000","last_fullsized":"2024-03-19T02:25:13.523188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203995+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:25:13.203995+0000","last_peered":"2024-03-19T02:25:13.203995+0000","last_clean":"2024-03-19T02:25:13.203995+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:25:13.203995+0000","last_undegraded":"2024-03-19T02:25:13.203995+0000","last_fullsized":"2024-03-19T02:25:13.203995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.726393+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:25:35.726393+0000","last_peered":"2024-03-19T02:25:35.726393+0000","last_clean":"2024-03-19T02:25:35.726393+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:25:35.726393+0000","last_undegraded":"2024-03-19T02:25:35.726393+0000","last_fullsized":"2024-03-19T02:25:35.726393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206374+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:25:13.206374+0000","last_peered":"2024-03-19T02:25:13.206374+0000","last_clean":"2024-03-19T02:25:13.206374+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:25:13.206374+0000","last_undegraded":"2024-03-19T02:25:13.206374+0000","last_fullsized":"2024-03-19T02:25:13.206374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206575+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:25:13.206575+0000","last_peered":"2024-03-19T02:25:13.206575+0000","last_clean":"2024-03-19T02:25:13.206575+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:25:13.206575+0000","last_undegraded":"2024-03-19T02:25:13.206575+0000","last_fullsized":"2024-03-19T02:25:13.206575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.702691+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:25:34.702691+0000","last_peered":"2024-03-19T02:25:34.702691+0000","last_clean":"2024-03-19T02:25:34.702691+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:25:34.702691+0000","last_undegraded":"2024-03-19T02:25:34.702691+0000","last_fullsized":"2024-03-19T02:25:34.702691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247273+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:25:13.247273+0000","last_peered":"2024-03-19T02:25:13.247273+0000","last_clean":"2024-03-19T02:25:13.247273+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:25:13.247273+0000","last_undegraded":"2024-03-19T02:25:13.247273+0000","last_fullsized":"2024-03-19T02:25:13.247273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380706+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:25:13.380706+0000","last_peered":"2024-03-19T02:25:13.380706+0000","last_clean":"2024-03-19T02:25:13.380706+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:25:13.380706+0000","last_undegraded":"2024-03-19T02:25:13.380706+0000","last_fullsized":"2024-03-19T02:25:13.380706+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.524292+0000","last_change":"2024-03-19T02:25:07.637031+0000","last_active":"2024-03-19T02:25:13.524292+0000","last_peered":"2024-03-19T02:25:13.524292+0000","last_clean":"2024-03-19T02:25:13.524292+0000","last_became_active":"2024-03-19T02:25:07.636712+0000","last_became_peered":"2024-03-19T02:25:07.636712+0000","last_unstale":"2024-03-19T02:25:13.524292+0000","last_undegraded":"2024-03-19T02:25:13.524292+0000","last_fullsized":"2024-03-19T02:25:13.524292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:03.071971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523378+0000","last_change":"2024-03-19T02:25:08.215661+0000","last_active":"2024-03-19T02:25:13.523378+0000","last_peered":"2024-03-19T02:25:13.523378+0000","last_clean":"2024-03-19T02:25:13.523378+0000","last_became_active":"2024-03-19T02:25:08.215314+0000","last_became_peered":"2024-03-19T02:25:08.215314+0000","last_unstale":"2024-03-19T02:25:13.523378+0000","last_undegraded":"2024-03-19T02:25:13.523378+0000","last_fullsized":"2024-03-19T02:25:13.523378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:56.493024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203918+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:25:13.203918+0000","last_peered":"2024-03-19T02:25:13.203918+0000","last_clean":"2024-03-19T02:25:13.203918+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:25:13.203918+0000","last_undegraded":"2024-03-19T02:25:13.203918+0000","last_fullsized":"2024-03-19T02:25:13.203918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:36.681922+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:25:36.681922+0000","last_peered":"2024-03-19T02:25:36.681922+0000","last_clean":"2024-03-19T02:25:36.681922+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:25:36.681922+0000","last_undegraded":"2024-03-19T02:25:36.681922+0000","last_fullsized":"2024-03-19T02:25:36.681922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522776+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:25:13.522776+0000","last_peered":"2024-03-19T02:25:13.522776+0000","last_clean":"2024-03-19T02:25:13.522776+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:25:13.522776+0000","last_undegraded":"2024-03-19T02:25:13.522776+0000","last_fullsized":"2024-03-19T02:25:13.522776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203965+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:25:13.203965+0000","last_peered":"2024-03-19T02:25:13.203965+0000","last_clean":"2024-03-19T02:25:13.203965+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:25:13.203965+0000","last_undegraded":"2024-03-19T02:25:13.203965+0000","last_fullsized":"2024-03-19T02:25:13.203965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206559+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:25:13.206559+0000","last_peered":"2024-03-19T02:25:13.206559+0000","last_clean":"2024-03-19T02:25:13.206559+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:25:13.206559+0000","last_undegraded":"2024-03-19T02:25:13.206559+0000","last_fullsized":"2024-03-19T02:25:13.206559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708896+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:25:33.708896+0000","last_peered":"2024-03-19T02:25:33.708896+0000","last_clean":"2024-03-19T02:25:33.708896+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:25:33.708896+0000","last_undegraded":"2024-03-19T02:25:33.708896+0000","last_fullsized":"2024-03-19T02:25:33.708896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523727+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:25:13.523727+0000","last_peered":"2024-03-19T02:25:13.523727+0000","last_clean":"2024-03-19T02:25:13.523727+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:25:13.523727+0000","last_undegraded":"2024-03-19T02:25:13.523727+0000","last_fullsized":"2024-03-19T02:25:13.523727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522365+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:25:13.522365+0000","last_peered":"2024-03-19T02:25:13.522365+0000","last_clean":"2024-03-19T02:25:13.522365+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:25:13.522365+0000","last_undegraded":"2024-03-19T02:25:13.522365+0000","last_fullsized":"2024-03-19T02:25:13.522365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.712345+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:25:34.712345+0000","last_peered":"2024-03-19T02:25:34.712345+0000","last_clean":"2024-03-19T02:25:34.712345+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:25:34.712345+0000","last_undegraded":"2024-03-19T02:25:34.712345+0000","last_fullsized":"2024-03-19T02:25:34.712345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523083+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:25:13.523083+0000","last_peered":"2024-03-19T02:25:13.523083+0000","last_clean":"2024-03-19T02:25:13.523083+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:25:13.523083+0000","last_undegraded":"2024-03-19T02:25:13.523083+0000","last_fullsized":"2024-03-19T02:25:13.523083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207199+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:25:13.207199+0000","last_peered":"2024-03-19T02:25:13.207199+0000","last_clean":"2024-03-19T02:25:13.207199+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:25:13.207199+0000","last_undegraded":"2024-03-19T02:25:13.207199+0000","last_fullsized":"2024-03-19T02:25:13.207199+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203384+0000","last_change":"2024-03-19T02:25:08.201769+0000","last_active":"2024-03-19T02:25:13.203384+0000","last_peered":"2024-03-19T02:25:13.203384+0000","last_clean":"2024-03-19T02:25:13.203384+0000","last_became_active":"2024-03-19T02:25:08.201477+0000","last_became_peered":"2024-03-19T02:25:08.201477+0000","last_unstale":"2024-03-19T02:25:13.203384+0000","last_undegraded":"2024-03-19T02:25:13.203384+0000","last_fullsized":"2024-03-19T02:25:13.203384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:27.064654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708485+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:25:33.708485+0000","last_peered":"2024-03-19T02:25:33.708485+0000","last_clean":"2024-03-19T02:25:33.708485+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:25:33.708485+0000","last_undegraded":"2024-03-19T02:25:33.708485+0000","last_fullsized":"2024-03-19T02:25:33.708485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247074+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:25:13.247074+0000","last_peered":"2024-03-19T02:25:13.247074+0000","last_clean":"2024-03-19T02:25:13.247074+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:25:13.247074+0000","last_undegraded":"2024-03-19T02:25:13.247074+0000","last_fullsized":"2024-03-19T02:25:13.247074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.727731+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:25:35.727731+0000","last_peered":"2024-03-19T02:25:35.727731+0000","last_clean":"2024-03-19T02:25:35.727731+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:25:35.727731+0000","last_undegraded":"2024-03-19T02:25:35.727731+0000","last_fullsized":"2024-03-19T02:25:35.727731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204014+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:25:13.204014+0000","last_peered":"2024-03-19T02:25:13.204014+0000","last_clean":"2024-03-19T02:25:13.204014+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:25:13.204014+0000","last_undegraded":"2024-03-19T02:25:13.204014+0000","last_fullsized":"2024-03-19T02:25:13.204014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523324+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:25:13.523324+0000","last_peered":"2024-03-19T02:25:13.523324+0000","last_clean":"2024-03-19T02:25:13.523324+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:25:13.523324+0000","last_undegraded":"2024-03-19T02:25:13.523324+0000","last_fullsized":"2024-03-19T02:25:13.523324+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.706824+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:25:33.706824+0000","last_peered":"2024-03-19T02:25:33.706824+0000","last_clean":"2024-03-19T02:25:33.706824+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:25:33.706824+0000","last_undegraded":"2024-03-19T02:25:33.706824+0000","last_fullsized":"2024-03-19T02:25:33.706824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522722+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:25:13.522722+0000","last_peered":"2024-03-19T02:25:13.522722+0000","last_clean":"2024-03-19T02:25:13.522722+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:25:13.522722+0000","last_undegraded":"2024-03-19T02:25:13.522722+0000","last_fullsized":"2024-03-19T02:25:13.522722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522672+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:25:13.522672+0000","last_peered":"2024-03-19T02:25:13.522672+0000","last_clean":"2024-03-19T02:25:13.522672+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:25:13.522672+0000","last_undegraded":"2024-03-19T02:25:13.522672+0000","last_fullsized":"2024-03-19T02:25:13.522672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695737+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:25:32.695737+0000","last_peered":"2024-03-19T02:25:32.695737+0000","last_clean":"2024-03-19T02:25:32.695737+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:25:32.695737+0000","last_undegraded":"2024-03-19T02:25:32.695737+0000","last_fullsized":"2024-03-19T02:25:32.695737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523036+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:25:13.523036+0000","last_peered":"2024-03-19T02:25:13.523036+0000","last_clean":"2024-03-19T02:25:13.523036+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:25:13.523036+0000","last_undegraded":"2024-03-19T02:25:13.523036+0000","last_fullsized":"2024-03-19T02:25:13.523036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522908+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:25:13.522908+0000","last_peered":"2024-03-19T02:25:13.522908+0000","last_clean":"2024-03-19T02:25:13.522908+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:25:13.522908+0000","last_undegraded":"2024-03-19T02:25:13.522908+0000","last_fullsized":"2024-03-19T02:25:13.522908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695920+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:25:32.695920+0000","last_peered":"2024-03-19T02:25:32.695920+0000","last_clean":"2024-03-19T02:25:32.695920+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:25:32.695920+0000","last_undegraded":"2024-03-19T02:25:32.695920+0000","last_fullsized":"2024-03-19T02:25:32.695920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380452+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:25:13.380452+0000","last_peered":"2024-03-19T02:25:13.380452+0000","last_clean":"2024-03-19T02:25:13.380452+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:25:13.380452+0000","last_undegraded":"2024-03-19T02:25:13.380452+0000","last_fullsized":"2024-03-19T02:25:13.380452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523277+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:25:13.523277+0000","last_peered":"2024-03-19T02:25:13.523277+0000","last_clean":"2024-03-19T02:25:13.523277+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:25:13.523277+0000","last_undegraded":"2024-03-19T02:25:13.523277+0000","last_fullsized":"2024-03-19T02:25:13.523277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.725350+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:25:35.725350+0000","last_peered":"2024-03-19T02:25:35.725350+0000","last_clean":"2024-03-19T02:25:35.725350+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:25:35.725350+0000","last_undegraded":"2024-03-19T02:25:35.725350+0000","last_fullsized":"2024-03-19T02:25:35.725350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.246847+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:25:13.246847+0000","last_peered":"2024-03-19T02:25:13.246847+0000","last_clean":"2024-03-19T02:25:13.246847+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:25:13.246847+0000","last_undegraded":"2024-03-19T02:25:13.246847+0000","last_fullsized":"2024-03-19T02:25:13.246847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523245+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:25:13.523245+0000","last_peered":"2024-03-19T02:25:13.523245+0000","last_clean":"2024-03-19T02:25:13.523245+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:25:13.523245+0000","last_undegraded":"2024-03-19T02:25:13.523245+0000","last_fullsized":"2024-03-19T02:25:13.523245+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.691496+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:25:34.691496+0000","last_peered":"2024-03-19T02:25:34.691496+0000","last_clean":"2024-03-19T02:25:34.691496+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:25:34.691496+0000","last_undegraded":"2024-03-19T02:25:34.691496+0000","last_fullsized":"2024-03-19T02:25:34.691496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522698+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:25:13.522698+0000","last_peered":"2024-03-19T02:25:13.522698+0000","last_clean":"2024-03-19T02:25:13.522698+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:25:13.522698+0000","last_undegraded":"2024-03-19T02:25:13.522698+0000","last_fullsized":"2024-03-19T02:25:13.522698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522681+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:25:13.522681+0000","last_peered":"2024-03-19T02:25:13.522681+0000","last_clean":"2024-03-19T02:25:13.522681+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:25:13.522681+0000","last_undegraded":"2024-03-19T02:25:13.522681+0000","last_fullsized":"2024-03-19T02:25:13.522681+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.697057+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:25:32.697057+0000","last_peered":"2024-03-19T02:25:32.697057+0000","last_clean":"2024-03-19T02:25:32.697057+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:25:32.697057+0000","last_undegraded":"2024-03-19T02:25:32.697057+0000","last_fullsized":"2024-03-19T02:25:32.697057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522891+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:25:13.522891+0000","last_peered":"2024-03-19T02:25:13.522891+0000","last_clean":"2024-03-19T02:25:13.522891+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:25:13.522891+0000","last_undegraded":"2024-03-19T02:25:13.522891+0000","last_fullsized":"2024-03-19T02:25:13.522891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203807+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:25:13.203807+0000","last_peered":"2024-03-19T02:25:13.203807+0000","last_clean":"2024-03-19T02:25:13.203807+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:25:13.203807+0000","last_undegraded":"2024-03-19T02:25:13.203807+0000","last_fullsized":"2024-03-19T02:25:13.203807+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.710440+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:25:34.710440+0000","last_peered":"2024-03-19T02:25:34.710440+0000","last_clean":"2024-03-19T02:25:34.710440+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:25:34.710440+0000","last_undegraded":"2024-03-19T02:25:34.710440+0000","last_fullsized":"2024-03-19T02:25:34.710440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523857+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:25:13.523857+0000","last_peered":"2024-03-19T02:25:13.523857+0000","last_clean":"2024-03-19T02:25:13.523857+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:25:13.523857+0000","last_undegraded":"2024-03-19T02:25:13.523857+0000","last_fullsized":"2024-03-19T02:25:13.523857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522166+0000","last_change":"2024-03-19T02:25:12.220335+0000","last_active":"2024-03-19T02:25:13.522166+0000","last_peered":"2024-03-19T02:25:13.522166+0000","last_clean":"2024-03-19T02:25:13.522166+0000","last_became_active":"2024-03-19T02:25:12.219063+0000","last_became_peered":"2024-03-19T02:25:12.219063+0000","last_unstale":"2024-03-19T02:25:13.522166+0000","last_undegraded":"2024-03-19T02:25:13.522166+0000","last_fullsized":"2024-03-19T02:25:13.522166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:10.439897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207271+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:25:13.207271+0000","last_peered":"2024-03-19T02:25:13.207271+0000","last_clean":"2024-03-19T02:25:13.207271+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:25:13.207271+0000","last_undegraded":"2024-03-19T02:25:13.207271+0000","last_fullsized":"2024-03-19T02:25:13.207271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523468+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:25:13.523468+0000","last_peered":"2024-03-19T02:25:13.523468+0000","last_clean":"2024-03-19T02:25:13.523468+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:25:13.523468+0000","last_undegraded":"2024-03-19T02:25:13.523468+0000","last_fullsized":"2024-03-19T02:25:13.523468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206337+0000","last_change":"2024-03-19T02:25:12.218644+0000","last_active":"2024-03-19T02:25:13.206337+0000","last_peered":"2024-03-19T02:25:13.206337+0000","last_clean":"2024-03-19T02:25:13.206337+0000","last_became_active":"2024-03-19T02:25:12.216439+0000","last_became_peered":"2024-03-19T02:25:12.216439+0000","last_unstale":"2024-03-19T02:25:13.206337+0000","last_undegraded":"2024-03-19T02:25:13.206337+0000","last_fullsized":"2024-03-19T02:25:13.206337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:29.734963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.725773+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:25:35.725773+0000","last_peered":"2024-03-19T02:25:35.725773+0000","last_clean":"2024-03-19T02:25:35.725773+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:25:35.725773+0000","last_undegraded":"2024-03-19T02:25:35.725773+0000","last_fullsized":"2024-03-19T02:25:35.725773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523607+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:25:13.523607+0000","last_peered":"2024-03-19T02:25:13.523607+0000","last_clean":"2024-03-19T02:25:13.523607+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:25:13.523607+0000","last_undegraded":"2024-03-19T02:25:13.523607+0000","last_fullsized":"2024-03-19T02:25:13.523607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.246922+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:25:13.246922+0000","last_peered":"2024-03-19T02:25:13.246922+0000","last_clean":"2024-03-19T02:25:13.246922+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:25:13.246922+0000","last_undegraded":"2024-03-19T02:25:13.246922+0000","last_fullsized":"2024-03-19T02:25:13.246922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522812+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:25:13.522812+0000","last_peered":"2024-03-19T02:25:13.522812+0000","last_clean":"2024-03-19T02:25:13.522812+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:25:13.522812+0000","last_undegraded":"2024-03-19T02:25:13.522812+0000","last_fullsized":"2024-03-19T02:25:13.522812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522639+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:25:13.522639+0000","last_peered":"2024-03-19T02:25:13.522639+0000","last_clean":"2024-03-19T02:25:13.522639+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:25:13.522639+0000","last_undegraded":"2024-03-19T02:25:13.522639+0000","last_fullsized":"2024-03-19T02:25:13.522639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522400+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:25:13.522400+0000","last_peered":"2024-03-19T02:25:13.522400+0000","last_clean":"2024-03-19T02:25:13.522400+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:25:13.522400+0000","last_undegraded":"2024-03-19T02:25:13.522400+0000","last_fullsized":"2024-03-19T02:25:13.522400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203937+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:25:13.203937+0000","last_peered":"2024-03-19T02:25:13.203937+0000","last_clean":"2024-03-19T02:25:13.203937+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:25:13.203937+0000","last_undegraded":"2024-03-19T02:25:13.203937+0000","last_fullsized":"2024-03-19T02:25:13.203937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247308+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:25:13.247308+0000","last_peered":"2024-03-19T02:25:13.247308+0000","last_clean":"2024-03-19T02:25:13.247308+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:25:13.247308+0000","last_undegraded":"2024-03-19T02:25:13.247308+0000","last_fullsized":"2024-03-19T02:25:13.247308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523097+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:25:13.523097+0000","last_peered":"2024-03-19T02:25:13.523097+0000","last_clean":"2024-03-19T02:25:13.523097+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:25:13.523097+0000","last_undegraded":"2024-03-19T02:25:13.523097+0000","last_fullsized":"2024-03-19T02:25:13.523097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522884+0000","last_change":"2024-03-19T02:25:08.204302+0000","last_active":"2024-03-19T02:25:13.522884+0000","last_peered":"2024-03-19T02:25:13.522884+0000","last_clean":"2024-03-19T02:25:13.522884+0000","last_became_active":"2024-03-19T02:25:08.203929+0000","last_became_peered":"2024-03-19T02:25:08.203929+0000","last_unstale":"2024-03-19T02:25:13.522884+0000","last_undegraded":"2024-03-19T02:25:13.522884+0000","last_fullsized":"2024-03-19T02:25:13.522884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:06.436193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206616+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:25:13.206616+0000","last_peered":"2024-03-19T02:25:13.206616+0000","last_clean":"2024-03-19T02:25:13.206616+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:25:13.206616+0000","last_undegraded":"2024-03-19T02:25:13.206616+0000","last_fullsized":"2024-03-19T02:25:13.206616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523046+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:25:13.523046+0000","last_peered":"2024-03-19T02:25:13.523046+0000","last_clean":"2024-03-19T02:25:13.523046+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:25:13.523046+0000","last_undegraded":"2024-03-19T02:25:13.523046+0000","last_fullsized":"2024-03-19T02:25:13.523046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203920+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:25:13.203920+0000","last_peered":"2024-03-19T02:25:13.203920+0000","last_clean":"2024-03-19T02:25:13.203920+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:25:13.203920+0000","last_undegraded":"2024-03-19T02:25:13.203920+0000","last_fullsized":"2024-03-19T02:25:13.203920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203926+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:25:13.203926+0000","last_peered":"2024-03-19T02:25:13.203926+0000","last_clean":"2024-03-19T02:25:13.203926+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:25:13.203926+0000","last_undegraded":"2024-03-19T02:25:13.203926+0000","last_fullsized":"2024-03-19T02:25:13.203926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247092+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:25:13.247092+0000","last_peered":"2024-03-19T02:25:13.247092+0000","last_clean":"2024-03-19T02:25:13.247092+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:25:13.247092+0000","last_undegraded":"2024-03-19T02:25:13.247092+0000","last_fullsized":"2024-03-19T02:25:13.247092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10050,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":176,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":64237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":485554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612629,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1075432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":481084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":481766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1077872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462956,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":480988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":505159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":480356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":485458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":480674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":480932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:25:38.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:25:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:37 smithi162 ceph-mon[35761]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:37 smithi162 ceph-mon[35761]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:37 smithi162 ceph-mon[35761]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:25:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:37 smithi162 ceph-mon[35761]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3761563655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:25:38.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.222+0000 7f0f844b2640 1 -- 172.21.15.116:0/4282561106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c1022f0 msgr2=0x7f0f7c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.222+0000 7f0f844b2640 1 --2- 172.21.15.116:0/4282561106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c108b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009900 tx=0x7f0f7802bc30 comp rx=0 tx=0).stop 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- 172.21.15.116:0/4282561106 shutdown_connections 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- 172.21.15.116:0/4282561106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c108b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- 172.21.15.116:0/4282561106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c101930 0x7f0f7c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- 172.21.15.116:0/4282561106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c100740 0x7f0f7c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- 172.21.15.116:0/4282561106 >> 172.21.15.116:0/4282561106 conn(0x7f0f7c0fbcb0 msgr2=0x7f0f7c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- 172.21.15.116:0/4282561106 shutdown_connections 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- 172.21.15.116:0/4282561106 wait complete. 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 Processor -- start 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- start start 2024-03-19T02:25:38.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c101930 0x7f0f7c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10c270 con 0x7f0f7c100740 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10c0f0 con 0x7f0f7c101930 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f844b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c10c3f0 con 0x7f0f7c1022f0 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f8224e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f8224e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53462/0 (socket says 172.21.15.116:53462) 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.223+0000 7f0f8224e640 1 -- 172.21.15.116:0/1638644565 learned_addr learned my addr 172.21.15.116:0/1638644565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f81a4d640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c101930 0x7f0f7c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f82a4f640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 -- 172.21.15.116:0/1638644565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c1022f0 msgr2=0x7f0f7c19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 -- 172.21.15.116:0/1638644565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c101930 msgr2=0x7f0f7c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c101930 0x7f0f7c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 -- 172.21.15.116:0/1638644565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c19f6b0 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f8224e640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0f7400ebc0 tx=0x7f0f7400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74003b70 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f74003d10 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c19f940 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74019840 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.224+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c19fe90 con 0x7f0f7c100740 2024-03-19T02:25:38.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.225+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7c068610 con 0x7f0f7c100740 2024-03-19T02:25:38.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.226+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f0f740199e0 con 0x7f0f7c100740 2024-03-19T02:25:38.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.227+0000 7f0f737fe640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 0x7f0f680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:38.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.227+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0f740cf7d0 con 0x7f0f7c100740 2024-03-19T02:25:38.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.227+0000 7f0f81a4d640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 0x7f0f680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:38.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.227+0000 7f0f81a4d640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 0x7f0f680a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c069260 tx=0x7f0f6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:38.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.230+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f7409c560 con 0x7f0f7c100740 2024-03-19T02:25:38.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.401+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f7c061c60 con 0x7f0f680a2890 2024-03-19T02:25:38.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.410+0000 7f0f737fe640 1 -- 172.21.15.116:0/1638644565 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340356 (secure 0 0 0) 0x7f0f7c061c60 con 0x7f0f680a2890 2024-03-19T02:25:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:38.416 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 msgr2=0x7f0f680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 0x7f0f680a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c069260 tx=0x7f0f6c005f90 comp rx=0 tx=0).stop 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 msgr2=0x7f0f7c0694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0f7400ebc0 tx=0x7f0f7400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 shutdown_connections 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0f680a2890 0x7f0f680a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f7c1022f0 0x7f0f7c19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f7c101930 0x7f0f7c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 --2- 172.21.15.116:0/1638644565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c100740 0x7f0f7c0694f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:38.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 >> 172.21.15.116:0/1638644565 conn(0x7f0f7c0fbcb0 msgr2=0x7f0f7c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:38.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 shutdown_connections 2024-03-19T02:25:38.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:38.414+0000 7f0f844b2640 1 -- 172.21.15.116:0/1638644565 wait complete. 2024-03-19T02:25:38.425 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-19T02:25:37.600263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600535,"num_objects":87,"num_object_clones":0,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":146,"num_read_kb":128,"num_write":358,"num_write_kb":3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371148,"kb_used_data":13664,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570676,"statfs":{"total":1151940427776,"available":1151560372224,"internally_reserved":0,"allocated":13991936,"data_stored":7587385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5527,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":129,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003629"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247463+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:25:13.247463+0000","last_peered":"2024-03-19T02:25:13.247463+0000","last_clean":"2024-03-19T02:25:13.247463+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:25:13.247463+0000","last_undegraded":"2024-03-19T02:25:13.247463+0000","last_fullsized":"2024-03-19T02:25:13.247463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522944+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:25:13.522944+0000","last_peered":"2024-03-19T02:25:13.522944+0000","last_clean":"2024-03-19T02:25:13.522944+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:25:13.522944+0000","last_undegraded":"2024-03-19T02:25:13.522944+0000","last_fullsized":"2024-03-19T02:25:13.522944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522440+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:25:13.522440+0000","last_peered":"2024-03-19T02:25:13.522440+0000","last_clean":"2024-03-19T02:25:13.522440+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:25:13.522440+0000","last_undegraded":"2024-03-19T02:25:13.522440+0000","last_fullsized":"2024-03-19T02:25:13.522440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523473+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:25:13.523473+0000","last_peered":"2024-03-19T02:25:13.523473+0000","last_clean":"2024-03-19T02:25:13.523473+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:25:13.523473+0000","last_undegraded":"2024-03-19T02:25:13.523473+0000","last_fullsized":"2024-03-19T02:25:13.523473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380625+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:25:13.380625+0000","last_peered":"2024-03-19T02:25:13.380625+0000","last_clean":"2024-03-19T02:25:13.380625+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:25:13.380625+0000","last_undegraded":"2024-03-19T02:25:13.380625+0000","last_fullsized":"2024-03-19T02:25:13.380625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522906+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:25:13.522906+0000","last_peered":"2024-03-19T02:25:13.522906+0000","last_clean":"2024-03-19T02:25:13.522906+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:25:13.522906+0000","last_undegraded":"2024-03-19T02:25:13.522906+0000","last_fullsized":"2024-03-19T02:25:13.522906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523815+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:25:13.523815+0000","last_peered":"2024-03-19T02:25:13.523815+0000","last_clean":"2024-03-19T02:25:13.523815+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:25:13.523815+0000","last_undegraded":"2024-03-19T02:25:13.523815+0000","last_fullsized":"2024-03-19T02:25:13.523815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522103+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:25:13.522103+0000","last_peered":"2024-03-19T02:25:13.522103+0000","last_clean":"2024-03-19T02:25:13.522103+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:25:13.522103+0000","last_undegraded":"2024-03-19T02:25:13.522103+0000","last_fullsized":"2024-03-19T02:25:13.522103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204384+0000","last_change":"2024-03-19T02:25:08.201590+0000","last_active":"2024-03-19T02:25:13.204384+0000","last_peered":"2024-03-19T02:25:13.204384+0000","last_clean":"2024-03-19T02:25:13.204384+0000","last_became_active":"2024-03-19T02:25:08.201274+0000","last_became_peered":"2024-03-19T02:25:08.201274+0000","last_unstale":"2024-03-19T02:25:13.204384+0000","last_undegraded":"2024-03-19T02:25:13.204384+0000","last_fullsized":"2024-03-19T02:25:13.204384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:43:25.989358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523575+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:25:13.523575+0000","last_peered":"2024-03-19T02:25:13.523575+0000","last_clean":"2024-03-19T02:25:13.523575+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:25:13.523575+0000","last_undegraded":"2024-03-19T02:25:13.523575+0000","last_fullsized":"2024-03-19T02:25:13.523575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523801+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:25:13.523801+0000","last_peered":"2024-03-19T02:25:13.523801+0000","last_clean":"2024-03-19T02:25:13.523801+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:25:13.523801+0000","last_undegraded":"2024-03-19T02:25:13.523801+0000","last_fullsized":"2024-03-19T02:25:13.523801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523373+0000","last_change":"2024-03-19T02:25:08.212302+0000","last_active":"2024-03-19T02:25:13.523373+0000","last_peered":"2024-03-19T02:25:13.523373+0000","last_clean":"2024-03-19T02:25:13.523373+0000","last_became_active":"2024-03-19T02:25:08.210842+0000","last_became_peered":"2024-03-19T02:25:08.210842+0000","last_unstale":"2024-03-19T02:25:13.523373+0000","last_undegraded":"2024-03-19T02:25:13.523373+0000","last_fullsized":"2024-03-19T02:25:13.523373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:54:44.550780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203680+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:25:13.203680+0000","last_peered":"2024-03-19T02:25:13.203680+0000","last_clean":"2024-03-19T02:25:13.203680+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:25:13.203680+0000","last_undegraded":"2024-03-19T02:25:13.203680+0000","last_fullsized":"2024-03-19T02:25:13.203680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522096+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:25:13.522096+0000","last_peered":"2024-03-19T02:25:13.522096+0000","last_clean":"2024-03-19T02:25:13.522096+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:25:13.522096+0000","last_undegraded":"2024-03-19T02:25:13.522096+0000","last_fullsized":"2024-03-19T02:25:13.522096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522284+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:25:13.522284+0000","last_peered":"2024-03-19T02:25:13.522284+0000","last_clean":"2024-03-19T02:25:13.522284+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:25:13.522284+0000","last_undegraded":"2024-03-19T02:25:13.522284+0000","last_fullsized":"2024-03-19T02:25:13.522284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203669+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:25:13.203669+0000","last_peered":"2024-03-19T02:25:13.203669+0000","last_clean":"2024-03-19T02:25:13.203669+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:25:13.203669+0000","last_undegraded":"2024-03-19T02:25:13.203669+0000","last_fullsized":"2024-03-19T02:25:13.203669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247288+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:25:13.247288+0000","last_peered":"2024-03-19T02:25:13.247288+0000","last_clean":"2024-03-19T02:25:13.247288+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:25:13.247288+0000","last_undegraded":"2024-03-19T02:25:13.247288+0000","last_fullsized":"2024-03-19T02:25:13.247288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523648+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:25:13.523648+0000","last_peered":"2024-03-19T02:25:13.523648+0000","last_clean":"2024-03-19T02:25:13.523648+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:25:13.523648+0000","last_undegraded":"2024-03-19T02:25:13.523648+0000","last_fullsized":"2024-03-19T02:25:13.523648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523139+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:25:13.523139+0000","last_peered":"2024-03-19T02:25:13.523139+0000","last_clean":"2024-03-19T02:25:13.523139+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:25:13.523139+0000","last_undegraded":"2024-03-19T02:25:13.523139+0000","last_fullsized":"2024-03-19T02:25:13.523139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522816+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:25:13.522816+0000","last_peered":"2024-03-19T02:25:13.522816+0000","last_clean":"2024-03-19T02:25:13.522816+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:25:13.522816+0000","last_undegraded":"2024-03-19T02:25:13.522816+0000","last_fullsized":"2024-03-19T02:25:13.522816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708487+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:25:33.708487+0000","last_peered":"2024-03-19T02:25:33.708487+0000","last_clean":"2024-03-19T02:25:33.708487+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:25:33.708487+0000","last_undegraded":"2024-03-19T02:25:33.708487+0000","last_fullsized":"2024-03-19T02:25:33.708487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522486+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:25:13.522486+0000","last_peered":"2024-03-19T02:25:13.522486+0000","last_clean":"2024-03-19T02:25:13.522486+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:25:13.522486+0000","last_undegraded":"2024-03-19T02:25:13.522486+0000","last_fullsized":"2024-03-19T02:25:13.522486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522705+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:25:13.522705+0000","last_peered":"2024-03-19T02:25:13.522705+0000","last_clean":"2024-03-19T02:25:13.522705+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:25:13.522705+0000","last_undegraded":"2024-03-19T02:25:13.522705+0000","last_fullsized":"2024-03-19T02:25:13.522705+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522943+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:25:13.522943+0000","last_peered":"2024-03-19T02:25:13.522943+0000","last_clean":"2024-03-19T02:25:13.522943+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:25:13.522943+0000","last_undegraded":"2024-03-19T02:25:13.522943+0000","last_fullsized":"2024-03-19T02:25:13.522943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523354+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:25:13.523354+0000","last_peered":"2024-03-19T02:25:13.523354+0000","last_clean":"2024-03-19T02:25:13.523354+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:25:13.523354+0000","last_undegraded":"2024-03-19T02:25:13.523354+0000","last_fullsized":"2024-03-19T02:25:13.523354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523238+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:25:13.523238+0000","last_peered":"2024-03-19T02:25:13.523238+0000","last_clean":"2024-03-19T02:25:13.523238+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:25:13.523238+0000","last_undegraded":"2024-03-19T02:25:13.523238+0000","last_fullsized":"2024-03-19T02:25:13.523238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:36.680062+0000","last_change":"2024-03-19T02:25:07.648715+0000","last_active":"2024-03-19T02:25:36.680062+0000","last_peered":"2024-03-19T02:25:36.680062+0000","last_clean":"2024-03-19T02:25:36.680062+0000","last_became_active":"2024-03-19T02:25:07.646184+0000","last_became_peered":"2024-03-19T02:25:07.646184+0000","last_unstale":"2024-03-19T02:25:36.680062+0000","last_undegraded":"2024-03-19T02:25:36.680062+0000","last_fullsized":"2024-03-19T02:25:36.680062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:04.444659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203903+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:25:13.203903+0000","last_peered":"2024-03-19T02:25:13.203903+0000","last_clean":"2024-03-19T02:25:13.203903+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:25:13.203903+0000","last_undegraded":"2024-03-19T02:25:13.203903+0000","last_fullsized":"2024-03-19T02:25:13.203903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522484+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:25:13.522484+0000","last_peered":"2024-03-19T02:25:13.522484+0000","last_clean":"2024-03-19T02:25:13.522484+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:25:13.522484+0000","last_undegraded":"2024-03-19T02:25:13.522484+0000","last_fullsized":"2024-03-19T02:25:13.522484+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.728255+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:25:35.728255+0000","last_peered":"2024-03-19T02:25:35.728255+0000","last_clean":"2024-03-19T02:25:35.728255+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:25:35.728255+0000","last_undegraded":"2024-03-19T02:25:35.728255+0000","last_fullsized":"2024-03-19T02:25:35.728255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207032+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:25:13.207032+0000","last_peered":"2024-03-19T02:25:13.207032+0000","last_clean":"2024-03-19T02:25:13.207032+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:25:13.207032+0000","last_undegraded":"2024-03-19T02:25:13.207032+0000","last_fullsized":"2024-03-19T02:25:13.207032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523141+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:25:13.523141+0000","last_peered":"2024-03-19T02:25:13.523141+0000","last_clean":"2024-03-19T02:25:13.523141+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:25:13.523141+0000","last_undegraded":"2024-03-19T02:25:13.523141+0000","last_fullsized":"2024-03-19T02:25:13.523141+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.697666+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:25:32.697666+0000","last_peered":"2024-03-19T02:25:32.697666+0000","last_clean":"2024-03-19T02:25:32.697666+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:25:32.697666+0000","last_undegraded":"2024-03-19T02:25:32.697666+0000","last_fullsized":"2024-03-19T02:25:32.697666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380659+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:25:13.380659+0000","last_peered":"2024-03-19T02:25:13.380659+0000","last_clean":"2024-03-19T02:25:13.380659+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:25:13.380659+0000","last_undegraded":"2024-03-19T02:25:13.380659+0000","last_fullsized":"2024-03-19T02:25:13.380659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522918+0000","last_change":"2024-03-19T02:25:12.217847+0000","last_active":"2024-03-19T02:25:13.522918+0000","last_peered":"2024-03-19T02:25:13.522918+0000","last_clean":"2024-03-19T02:25:13.522918+0000","last_became_active":"2024-03-19T02:25:12.217350+0000","last_became_peered":"2024-03-19T02:25:12.217350+0000","last_unstale":"2024-03-19T02:25:13.522918+0000","last_undegraded":"2024-03-19T02:25:13.522918+0000","last_fullsized":"2024-03-19T02:25:13.522918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:22.562971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522874+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:25:13.522874+0000","last_peered":"2024-03-19T02:25:13.522874+0000","last_clean":"2024-03-19T02:25:13.522874+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:25:13.522874+0000","last_undegraded":"2024-03-19T02:25:13.522874+0000","last_fullsized":"2024-03-19T02:25:13.522874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206955+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:25:13.206955+0000","last_peered":"2024-03-19T02:25:13.206955+0000","last_clean":"2024-03-19T02:25:13.206955+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:25:13.206955+0000","last_undegraded":"2024-03-19T02:25:13.206955+0000","last_fullsized":"2024-03-19T02:25:13.206955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247361+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:25:13.247361+0000","last_peered":"2024-03-19T02:25:13.247361+0000","last_clean":"2024-03-19T02:25:13.247361+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:25:13.247361+0000","last_undegraded":"2024-03-19T02:25:13.247361+0000","last_fullsized":"2024-03-19T02:25:13.247361+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708552+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:25:33.708552+0000","last_peered":"2024-03-19T02:25:33.708552+0000","last_clean":"2024-03-19T02:25:33.708552+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:25:33.708552+0000","last_undegraded":"2024-03-19T02:25:33.708552+0000","last_fullsized":"2024-03-19T02:25:33.708552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522214+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:25:13.522214+0000","last_peered":"2024-03-19T02:25:13.522214+0000","last_clean":"2024-03-19T02:25:13.522214+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:25:13.522214+0000","last_undegraded":"2024-03-19T02:25:13.522214+0000","last_fullsized":"2024-03-19T02:25:13.522214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522845+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:25:13.522845+0000","last_peered":"2024-03-19T02:25:13.522845+0000","last_clean":"2024-03-19T02:25:13.522845+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:25:13.522845+0000","last_undegraded":"2024-03-19T02:25:13.522845+0000","last_fullsized":"2024-03-19T02:25:13.522845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.696034+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:25:32.696034+0000","last_peered":"2024-03-19T02:25:32.696034+0000","last_clean":"2024-03-19T02:25:32.696034+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:25:32.696034+0000","last_undegraded":"2024-03-19T02:25:32.696034+0000","last_fullsized":"2024-03-19T02:25:32.696034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522955+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:25:13.522955+0000","last_peered":"2024-03-19T02:25:13.522955+0000","last_clean":"2024-03-19T02:25:13.522955+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:25:13.522955+0000","last_undegraded":"2024-03-19T02:25:13.522955+0000","last_fullsized":"2024-03-19T02:25:13.522955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522897+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:25:13.522897+0000","last_peered":"2024-03-19T02:25:13.522897+0000","last_clean":"2024-03-19T02:25:13.522897+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:25:13.522897+0000","last_undegraded":"2024-03-19T02:25:13.522897+0000","last_fullsized":"2024-03-19T02:25:13.522897+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695407+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:25:32.695407+0000","last_peered":"2024-03-19T02:25:32.695407+0000","last_clean":"2024-03-19T02:25:32.695407+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:25:32.695407+0000","last_undegraded":"2024-03-19T02:25:32.695407+0000","last_fullsized":"2024-03-19T02:25:32.695407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204281+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:25:13.204281+0000","last_peered":"2024-03-19T02:25:13.204281+0000","last_clean":"2024-03-19T02:25:13.204281+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:25:13.204281+0000","last_undegraded":"2024-03-19T02:25:13.204281+0000","last_fullsized":"2024-03-19T02:25:13.204281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523286+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:25:13.523286+0000","last_peered":"2024-03-19T02:25:13.523286+0000","last_clean":"2024-03-19T02:25:13.523286+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:25:13.523286+0000","last_undegraded":"2024-03-19T02:25:13.523286+0000","last_fullsized":"2024-03-19T02:25:13.523286+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.727158+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:25:35.727158+0000","last_peered":"2024-03-19T02:25:35.727158+0000","last_clean":"2024-03-19T02:25:35.727158+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:25:35.727158+0000","last_undegraded":"2024-03-19T02:25:35.727158+0000","last_fullsized":"2024-03-19T02:25:35.727158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523461+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:25:13.523461+0000","last_peered":"2024-03-19T02:25:13.523461+0000","last_clean":"2024-03-19T02:25:13.523461+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:25:13.523461+0000","last_undegraded":"2024-03-19T02:25:13.523461+0000","last_fullsized":"2024-03-19T02:25:13.523461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523330+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:25:13.523330+0000","last_peered":"2024-03-19T02:25:13.523330+0000","last_clean":"2024-03-19T02:25:13.523330+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:25:13.523330+0000","last_undegraded":"2024-03-19T02:25:13.523330+0000","last_fullsized":"2024-03-19T02:25:13.523330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695359+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:25:32.695359+0000","last_peered":"2024-03-19T02:25:32.695359+0000","last_clean":"2024-03-19T02:25:32.695359+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:25:32.695359+0000","last_undegraded":"2024-03-19T02:25:32.695359+0000","last_fullsized":"2024-03-19T02:25:32.695359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204123+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:25:13.204123+0000","last_peered":"2024-03-19T02:25:13.204123+0000","last_clean":"2024-03-19T02:25:13.204123+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:25:13.204123+0000","last_undegraded":"2024-03-19T02:25:13.204123+0000","last_fullsized":"2024-03-19T02:25:13.204123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523217+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:25:13.523217+0000","last_peered":"2024-03-19T02:25:13.523217+0000","last_clean":"2024-03-19T02:25:13.523217+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:25:13.523217+0000","last_undegraded":"2024-03-19T02:25:13.523217+0000","last_fullsized":"2024-03-19T02:25:13.523217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.696591+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:25:32.696591+0000","last_peered":"2024-03-19T02:25:32.696591+0000","last_clean":"2024-03-19T02:25:32.696591+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:25:32.696591+0000","last_undegraded":"2024-03-19T02:25:32.696591+0000","last_fullsized":"2024-03-19T02:25:32.696591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380880+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:25:13.380880+0000","last_peered":"2024-03-19T02:25:13.380880+0000","last_clean":"2024-03-19T02:25:13.380880+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:25:13.380880+0000","last_undegraded":"2024-03-19T02:25:13.380880+0000","last_fullsized":"2024-03-19T02:25:13.380880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207381+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:25:13.207381+0000","last_peered":"2024-03-19T02:25:13.207381+0000","last_clean":"2024-03-19T02:25:13.207381+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:25:13.207381+0000","last_undegraded":"2024-03-19T02:25:13.207381+0000","last_fullsized":"2024-03-19T02:25:13.207381+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708508+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:25:33.708508+0000","last_peered":"2024-03-19T02:25:33.708508+0000","last_clean":"2024-03-19T02:25:33.708508+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:25:33.708508+0000","last_undegraded":"2024-03-19T02:25:33.708508+0000","last_fullsized":"2024-03-19T02:25:33.708508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203754+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:25:13.203754+0000","last_peered":"2024-03-19T02:25:13.203754+0000","last_clean":"2024-03-19T02:25:13.203754+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:25:13.203754+0000","last_undegraded":"2024-03-19T02:25:13.203754+0000","last_fullsized":"2024-03-19T02:25:13.203754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247064+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:25:13.247064+0000","last_peered":"2024-03-19T02:25:13.247064+0000","last_clean":"2024-03-19T02:25:13.247064+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:25:13.247064+0000","last_undegraded":"2024-03-19T02:25:13.247064+0000","last_fullsized":"2024-03-19T02:25:13.247064+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695994+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:25:32.695994+0000","last_peered":"2024-03-19T02:25:32.695994+0000","last_clean":"2024-03-19T02:25:32.695994+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:25:32.695994+0000","last_undegraded":"2024-03-19T02:25:32.695994+0000","last_fullsized":"2024-03-19T02:25:32.695994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523811+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:25:13.523811+0000","last_peered":"2024-03-19T02:25:13.523811+0000","last_clean":"2024-03-19T02:25:13.523811+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:25:13.523811+0000","last_undegraded":"2024-03-19T02:25:13.523811+0000","last_fullsized":"2024-03-19T02:25:13.523811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523108+0000","last_change":"2024-03-19T02:25:12.228992+0000","last_active":"2024-03-19T02:25:13.523108+0000","last_peered":"2024-03-19T02:25:13.523108+0000","last_clean":"2024-03-19T02:25:13.523108+0000","last_became_active":"2024-03-19T02:25:12.228457+0000","last_became_peered":"2024-03-19T02:25:12.228457+0000","last_unstale":"2024-03-19T02:25:13.523108+0000","last_undegraded":"2024-03-19T02:25:13.523108+0000","last_fullsized":"2024-03-19T02:25:13.523108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:14:04.599952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.710979+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:25:34.710979+0000","last_peered":"2024-03-19T02:25:34.710979+0000","last_clean":"2024-03-19T02:25:34.710979+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:25:34.710979+0000","last_undegraded":"2024-03-19T02:25:34.710979+0000","last_fullsized":"2024-03-19T02:25:34.710979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523188+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:25:13.523188+0000","last_peered":"2024-03-19T02:25:13.523188+0000","last_clean":"2024-03-19T02:25:13.523188+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:25:13.523188+0000","last_undegraded":"2024-03-19T02:25:13.523188+0000","last_fullsized":"2024-03-19T02:25:13.523188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203995+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:25:13.203995+0000","last_peered":"2024-03-19T02:25:13.203995+0000","last_clean":"2024-03-19T02:25:13.203995+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:25:13.203995+0000","last_undegraded":"2024-03-19T02:25:13.203995+0000","last_fullsized":"2024-03-19T02:25:13.203995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.726393+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:25:35.726393+0000","last_peered":"2024-03-19T02:25:35.726393+0000","last_clean":"2024-03-19T02:25:35.726393+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:25:35.726393+0000","last_undegraded":"2024-03-19T02:25:35.726393+0000","last_fullsized":"2024-03-19T02:25:35.726393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206374+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:25:13.206374+0000","last_peered":"2024-03-19T02:25:13.206374+0000","last_clean":"2024-03-19T02:25:13.206374+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:25:13.206374+0000","last_undegraded":"2024-03-19T02:25:13.206374+0000","last_fullsized":"2024-03-19T02:25:13.206374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206575+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:25:13.206575+0000","last_peered":"2024-03-19T02:25:13.206575+0000","last_clean":"2024-03-19T02:25:13.206575+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:25:13.206575+0000","last_undegraded":"2024-03-19T02:25:13.206575+0000","last_fullsized":"2024-03-19T02:25:13.206575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.702691+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:25:34.702691+0000","last_peered":"2024-03-19T02:25:34.702691+0000","last_clean":"2024-03-19T02:25:34.702691+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:25:34.702691+0000","last_undegraded":"2024-03-19T02:25:34.702691+0000","last_fullsized":"2024-03-19T02:25:34.702691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247273+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:25:13.247273+0000","last_peered":"2024-03-19T02:25:13.247273+0000","last_clean":"2024-03-19T02:25:13.247273+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:25:13.247273+0000","last_undegraded":"2024-03-19T02:25:13.247273+0000","last_fullsized":"2024-03-19T02:25:13.247273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380706+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:25:13.380706+0000","last_peered":"2024-03-19T02:25:13.380706+0000","last_clean":"2024-03-19T02:25:13.380706+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:25:13.380706+0000","last_undegraded":"2024-03-19T02:25:13.380706+0000","last_fullsized":"2024-03-19T02:25:13.380706+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.524292+0000","last_change":"2024-03-19T02:25:07.637031+0000","last_active":"2024-03-19T02:25:13.524292+0000","last_peered":"2024-03-19T02:25:13.524292+0000","last_clean":"2024-03-19T02:25:13.524292+0000","last_became_active":"2024-03-19T02:25:07.636712+0000","last_became_peered":"2024-03-19T02:25:07.636712+0000","last_unstale":"2024-03-19T02:25:13.524292+0000","last_undegraded":"2024-03-19T02:25:13.524292+0000","last_fullsized":"2024-03-19T02:25:13.524292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:03.071971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523378+0000","last_change":"2024-03-19T02:25:08.215661+0000","last_active":"2024-03-19T02:25:13.523378+0000","last_peered":"2024-03-19T02:25:13.523378+0000","last_clean":"2024-03-19T02:25:13.523378+0000","last_became_active":"2024-03-19T02:25:08.215314+0000","last_became_peered":"2024-03-19T02:25:08.215314+0000","last_unstale":"2024-03-19T02:25:13.523378+0000","last_undegraded":"2024-03-19T02:25:13.523378+0000","last_fullsized":"2024-03-19T02:25:13.523378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:56.493024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203918+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:25:13.203918+0000","last_peered":"2024-03-19T02:25:13.203918+0000","last_clean":"2024-03-19T02:25:13.203918+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:25:13.203918+0000","last_undegraded":"2024-03-19T02:25:13.203918+0000","last_fullsized":"2024-03-19T02:25:13.203918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:36.681922+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:25:36.681922+0000","last_peered":"2024-03-19T02:25:36.681922+0000","last_clean":"2024-03-19T02:25:36.681922+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:25:36.681922+0000","last_undegraded":"2024-03-19T02:25:36.681922+0000","last_fullsized":"2024-03-19T02:25:36.681922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522776+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:25:13.522776+0000","last_peered":"2024-03-19T02:25:13.522776+0000","last_clean":"2024-03-19T02:25:13.522776+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:25:13.522776+0000","last_undegraded":"2024-03-19T02:25:13.522776+0000","last_fullsized":"2024-03-19T02:25:13.522776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203965+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:25:13.203965+0000","last_peered":"2024-03-19T02:25:13.203965+0000","last_clean":"2024-03-19T02:25:13.203965+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:25:13.203965+0000","last_undegraded":"2024-03-19T02:25:13.203965+0000","last_fullsized":"2024-03-19T02:25:13.203965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206559+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:25:13.206559+0000","last_peered":"2024-03-19T02:25:13.206559+0000","last_clean":"2024-03-19T02:25:13.206559+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:25:13.206559+0000","last_undegraded":"2024-03-19T02:25:13.206559+0000","last_fullsized":"2024-03-19T02:25:13.206559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708896+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:25:33.708896+0000","last_peered":"2024-03-19T02:25:33.708896+0000","last_clean":"2024-03-19T02:25:33.708896+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:25:33.708896+0000","last_undegraded":"2024-03-19T02:25:33.708896+0000","last_fullsized":"2024-03-19T02:25:33.708896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523727+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:25:13.523727+0000","last_peered":"2024-03-19T02:25:13.523727+0000","last_clean":"2024-03-19T02:25:13.523727+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:25:13.523727+0000","last_undegraded":"2024-03-19T02:25:13.523727+0000","last_fullsized":"2024-03-19T02:25:13.523727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522365+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:25:13.522365+0000","last_peered":"2024-03-19T02:25:13.522365+0000","last_clean":"2024-03-19T02:25:13.522365+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:25:13.522365+0000","last_undegraded":"2024-03-19T02:25:13.522365+0000","last_fullsized":"2024-03-19T02:25:13.522365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.712345+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:25:34.712345+0000","last_peered":"2024-03-19T02:25:34.712345+0000","last_clean":"2024-03-19T02:25:34.712345+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:25:34.712345+0000","last_undegraded":"2024-03-19T02:25:34.712345+0000","last_fullsized":"2024-03-19T02:25:34.712345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523083+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:25:13.523083+0000","last_peered":"2024-03-19T02:25:13.523083+0000","last_clean":"2024-03-19T02:25:13.523083+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:25:13.523083+0000","last_undegraded":"2024-03-19T02:25:13.523083+0000","last_fullsized":"2024-03-19T02:25:13.523083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207199+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:25:13.207199+0000","last_peered":"2024-03-19T02:25:13.207199+0000","last_clean":"2024-03-19T02:25:13.207199+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:25:13.207199+0000","last_undegraded":"2024-03-19T02:25:13.207199+0000","last_fullsized":"2024-03-19T02:25:13.207199+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203384+0000","last_change":"2024-03-19T02:25:08.201769+0000","last_active":"2024-03-19T02:25:13.203384+0000","last_peered":"2024-03-19T02:25:13.203384+0000","last_clean":"2024-03-19T02:25:13.203384+0000","last_became_active":"2024-03-19T02:25:08.201477+0000","last_became_peered":"2024-03-19T02:25:08.201477+0000","last_unstale":"2024-03-19T02:25:13.203384+0000","last_undegraded":"2024-03-19T02:25:13.203384+0000","last_fullsized":"2024-03-19T02:25:13.203384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:27.064654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.708485+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:25:33.708485+0000","last_peered":"2024-03-19T02:25:33.708485+0000","last_clean":"2024-03-19T02:25:33.708485+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:25:33.708485+0000","last_undegraded":"2024-03-19T02:25:33.708485+0000","last_fullsized":"2024-03-19T02:25:33.708485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247074+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:25:13.247074+0000","last_peered":"2024-03-19T02:25:13.247074+0000","last_clean":"2024-03-19T02:25:13.247074+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:25:13.247074+0000","last_undegraded":"2024-03-19T02:25:13.247074+0000","last_fullsized":"2024-03-19T02:25:13.247074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.727731+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:25:35.727731+0000","last_peered":"2024-03-19T02:25:35.727731+0000","last_clean":"2024-03-19T02:25:35.727731+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:25:35.727731+0000","last_undegraded":"2024-03-19T02:25:35.727731+0000","last_fullsized":"2024-03-19T02:25:35.727731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.204014+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:25:13.204014+0000","last_peered":"2024-03-19T02:25:13.204014+0000","last_clean":"2024-03-19T02:25:13.204014+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:25:13.204014+0000","last_undegraded":"2024-03-19T02:25:13.204014+0000","last_fullsized":"2024-03-19T02:25:13.204014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523324+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:25:13.523324+0000","last_peered":"2024-03-19T02:25:13.523324+0000","last_clean":"2024-03-19T02:25:13.523324+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:25:13.523324+0000","last_undegraded":"2024-03-19T02:25:13.523324+0000","last_fullsized":"2024-03-19T02:25:13.523324+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:33.706824+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:25:33.706824+0000","last_peered":"2024-03-19T02:25:33.706824+0000","last_clean":"2024-03-19T02:25:33.706824+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:25:33.706824+0000","last_undegraded":"2024-03-19T02:25:33.706824+0000","last_fullsized":"2024-03-19T02:25:33.706824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522722+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:25:13.522722+0000","last_peered":"2024-03-19T02:25:13.522722+0000","last_clean":"2024-03-19T02:25:13.522722+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:25:13.522722+0000","last_undegraded":"2024-03-19T02:25:13.522722+0000","last_fullsized":"2024-03-19T02:25:13.522722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522672+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:25:13.522672+0000","last_peered":"2024-03-19T02:25:13.522672+0000","last_clean":"2024-03-19T02:25:13.522672+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:25:13.522672+0000","last_undegraded":"2024-03-19T02:25:13.522672+0000","last_fullsized":"2024-03-19T02:25:13.522672+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695737+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:25:32.695737+0000","last_peered":"2024-03-19T02:25:32.695737+0000","last_clean":"2024-03-19T02:25:32.695737+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:25:32.695737+0000","last_undegraded":"2024-03-19T02:25:32.695737+0000","last_fullsized":"2024-03-19T02:25:32.695737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523036+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:25:13.523036+0000","last_peered":"2024-03-19T02:25:13.523036+0000","last_clean":"2024-03-19T02:25:13.523036+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:25:13.523036+0000","last_undegraded":"2024-03-19T02:25:13.523036+0000","last_fullsized":"2024-03-19T02:25:13.523036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522908+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:25:13.522908+0000","last_peered":"2024-03-19T02:25:13.522908+0000","last_clean":"2024-03-19T02:25:13.522908+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:25:13.522908+0000","last_undegraded":"2024-03-19T02:25:13.522908+0000","last_fullsized":"2024-03-19T02:25:13.522908+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.695920+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:25:32.695920+0000","last_peered":"2024-03-19T02:25:32.695920+0000","last_clean":"2024-03-19T02:25:32.695920+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:25:32.695920+0000","last_undegraded":"2024-03-19T02:25:32.695920+0000","last_fullsized":"2024-03-19T02:25:32.695920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.380452+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:25:13.380452+0000","last_peered":"2024-03-19T02:25:13.380452+0000","last_clean":"2024-03-19T02:25:13.380452+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:25:13.380452+0000","last_undegraded":"2024-03-19T02:25:13.380452+0000","last_fullsized":"2024-03-19T02:25:13.380452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523277+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:25:13.523277+0000","last_peered":"2024-03-19T02:25:13.523277+0000","last_clean":"2024-03-19T02:25:13.523277+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:25:13.523277+0000","last_undegraded":"2024-03-19T02:25:13.523277+0000","last_fullsized":"2024-03-19T02:25:13.523277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.725350+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:25:35.725350+0000","last_peered":"2024-03-19T02:25:35.725350+0000","last_clean":"2024-03-19T02:25:35.725350+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:25:35.725350+0000","last_undegraded":"2024-03-19T02:25:35.725350+0000","last_fullsized":"2024-03-19T02:25:35.725350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.246847+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:25:13.246847+0000","last_peered":"2024-03-19T02:25:13.246847+0000","last_clean":"2024-03-19T02:25:13.246847+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:25:13.246847+0000","last_undegraded":"2024-03-19T02:25:13.246847+0000","last_fullsized":"2024-03-19T02:25:13.246847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523245+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:25:13.523245+0000","last_peered":"2024-03-19T02:25:13.523245+0000","last_clean":"2024-03-19T02:25:13.523245+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:25:13.523245+0000","last_undegraded":"2024-03-19T02:25:13.523245+0000","last_fullsized":"2024-03-19T02:25:13.523245+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.691496+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:25:34.691496+0000","last_peered":"2024-03-19T02:25:34.691496+0000","last_clean":"2024-03-19T02:25:34.691496+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:25:34.691496+0000","last_undegraded":"2024-03-19T02:25:34.691496+0000","last_fullsized":"2024-03-19T02:25:34.691496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522698+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:25:13.522698+0000","last_peered":"2024-03-19T02:25:13.522698+0000","last_clean":"2024-03-19T02:25:13.522698+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:25:13.522698+0000","last_undegraded":"2024-03-19T02:25:13.522698+0000","last_fullsized":"2024-03-19T02:25:13.522698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522681+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:25:13.522681+0000","last_peered":"2024-03-19T02:25:13.522681+0000","last_clean":"2024-03-19T02:25:13.522681+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:25:13.522681+0000","last_undegraded":"2024-03-19T02:25:13.522681+0000","last_fullsized":"2024-03-19T02:25:13.522681+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:32.697057+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:25:32.697057+0000","last_peered":"2024-03-19T02:25:32.697057+0000","last_clean":"2024-03-19T02:25:32.697057+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:25:32.697057+0000","last_undegraded":"2024-03-19T02:25:32.697057+0000","last_fullsized":"2024-03-19T02:25:32.697057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522891+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:25:13.522891+0000","last_peered":"2024-03-19T02:25:13.522891+0000","last_clean":"2024-03-19T02:25:13.522891+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:25:13.522891+0000","last_undegraded":"2024-03-19T02:25:13.522891+0000","last_fullsized":"2024-03-19T02:25:13.522891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203807+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:25:13.203807+0000","last_peered":"2024-03-19T02:25:13.203807+0000","last_clean":"2024-03-19T02:25:13.203807+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:25:13.203807+0000","last_undegraded":"2024-03-19T02:25:13.203807+0000","last_fullsized":"2024-03-19T02:25:13.203807+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:34.710440+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:25:34.710440+0000","last_peered":"2024-03-19T02:25:34.710440+0000","last_clean":"2024-03-19T02:25:34.710440+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:25:34.710440+0000","last_undegraded":"2024-03-19T02:25:34.710440+0000","last_fullsized":"2024-03-19T02:25:34.710440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523857+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:25:13.523857+0000","last_peered":"2024-03-19T02:25:13.523857+0000","last_clean":"2024-03-19T02:25:13.523857+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:25:13.523857+0000","last_undegraded":"2024-03-19T02:25:13.523857+0000","last_fullsized":"2024-03-19T02:25:13.523857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522166+0000","last_change":"2024-03-19T02:25:12.220335+0000","last_active":"2024-03-19T02:25:13.522166+0000","last_peered":"2024-03-19T02:25:13.522166+0000","last_clean":"2024-03-19T02:25:13.522166+0000","last_became_active":"2024-03-19T02:25:12.219063+0000","last_became_peered":"2024-03-19T02:25:12.219063+0000","last_unstale":"2024-03-19T02:25:13.522166+0000","last_undegraded":"2024-03-19T02:25:13.522166+0000","last_fullsized":"2024-03-19T02:25:13.522166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:10.439897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.207271+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:25:13.207271+0000","last_peered":"2024-03-19T02:25:13.207271+0000","last_clean":"2024-03-19T02:25:13.207271+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:25:13.207271+0000","last_undegraded":"2024-03-19T02:25:13.207271+0000","last_fullsized":"2024-03-19T02:25:13.207271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523468+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:25:13.523468+0000","last_peered":"2024-03-19T02:25:13.523468+0000","last_clean":"2024-03-19T02:25:13.523468+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:25:13.523468+0000","last_undegraded":"2024-03-19T02:25:13.523468+0000","last_fullsized":"2024-03-19T02:25:13.523468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206337+0000","last_change":"2024-03-19T02:25:12.218644+0000","last_active":"2024-03-19T02:25:13.206337+0000","last_peered":"2024-03-19T02:25:13.206337+0000","last_clean":"2024-03-19T02:25:13.206337+0000","last_became_active":"2024-03-19T02:25:12.216439+0000","last_became_peered":"2024-03-19T02:25:12.216439+0000","last_unstale":"2024-03-19T02:25:13.206337+0000","last_undegraded":"2024-03-19T02:25:13.206337+0000","last_fullsized":"2024-03-19T02:25:13.206337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:29.734963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:35.725773+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:25:35.725773+0000","last_peered":"2024-03-19T02:25:35.725773+0000","last_clean":"2024-03-19T02:25:35.725773+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:25:35.725773+0000","last_undegraded":"2024-03-19T02:25:35.725773+0000","last_fullsized":"2024-03-19T02:25:35.725773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523607+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:25:13.523607+0000","last_peered":"2024-03-19T02:25:13.523607+0000","last_clean":"2024-03-19T02:25:13.523607+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:25:13.523607+0000","last_undegraded":"2024-03-19T02:25:13.523607+0000","last_fullsized":"2024-03-19T02:25:13.523607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.246922+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:25:13.246922+0000","last_peered":"2024-03-19T02:25:13.246922+0000","last_clean":"2024-03-19T02:25:13.246922+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:25:13.246922+0000","last_undegraded":"2024-03-19T02:25:13.246922+0000","last_fullsized":"2024-03-19T02:25:13.246922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522812+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:25:13.522812+0000","last_peered":"2024-03-19T02:25:13.522812+0000","last_clean":"2024-03-19T02:25:13.522812+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:25:13.522812+0000","last_undegraded":"2024-03-19T02:25:13.522812+0000","last_fullsized":"2024-03-19T02:25:13.522812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522639+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:25:13.522639+0000","last_peered":"2024-03-19T02:25:13.522639+0000","last_clean":"2024-03-19T02:25:13.522639+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:25:13.522639+0000","last_undegraded":"2024-03-19T02:25:13.522639+0000","last_fullsized":"2024-03-19T02:25:13.522639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522400+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:25:13.522400+0000","last_peered":"2024-03-19T02:25:13.522400+0000","last_clean":"2024-03-19T02:25:13.522400+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:25:13.522400+0000","last_undegraded":"2024-03-19T02:25:13.522400+0000","last_fullsized":"2024-03-19T02:25:13.522400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203937+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:25:13.203937+0000","last_peered":"2024-03-19T02:25:13.203937+0000","last_clean":"2024-03-19T02:25:13.203937+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:25:13.203937+0000","last_undegraded":"2024-03-19T02:25:13.203937+0000","last_fullsized":"2024-03-19T02:25:13.203937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247308+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:25:13.247308+0000","last_peered":"2024-03-19T02:25:13.247308+0000","last_clean":"2024-03-19T02:25:13.247308+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:25:13.247308+0000","last_undegraded":"2024-03-19T02:25:13.247308+0000","last_fullsized":"2024-03-19T02:25:13.247308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523097+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:25:13.523097+0000","last_peered":"2024-03-19T02:25:13.523097+0000","last_clean":"2024-03-19T02:25:13.523097+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:25:13.523097+0000","last_undegraded":"2024-03-19T02:25:13.523097+0000","last_fullsized":"2024-03-19T02:25:13.523097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.522884+0000","last_change":"2024-03-19T02:25:08.204302+0000","last_active":"2024-03-19T02:25:13.522884+0000","last_peered":"2024-03-19T02:25:13.522884+0000","last_clean":"2024-03-19T02:25:13.522884+0000","last_became_active":"2024-03-19T02:25:08.203929+0000","last_became_peered":"2024-03-19T02:25:08.203929+0000","last_unstale":"2024-03-19T02:25:13.522884+0000","last_undegraded":"2024-03-19T02:25:13.522884+0000","last_fullsized":"2024-03-19T02:25:13.522884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:06.436193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.206616+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:25:13.206616+0000","last_peered":"2024-03-19T02:25:13.206616+0000","last_clean":"2024-03-19T02:25:13.206616+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:25:13.206616+0000","last_undegraded":"2024-03-19T02:25:13.206616+0000","last_fullsized":"2024-03-19T02:25:13.206616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.523046+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:25:13.523046+0000","last_peered":"2024-03-19T02:25:13.523046+0000","last_clean":"2024-03-19T02:25:13.523046+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:25:13.523046+0000","last_undegraded":"2024-03-19T02:25:13.523046+0000","last_fullsized":"2024-03-19T02:25:13.523046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203920+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:25:13.203920+0000","last_peered":"2024-03-19T02:25:13.203920+0000","last_clean":"2024-03-19T02:25:13.203920+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:25:13.203920+0000","last_undegraded":"2024-03-19T02:25:13.203920+0000","last_fullsized":"2024-03-19T02:25:13.203920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.203926+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:25:13.203926+0000","last_peered":"2024-03-19T02:25:13.203926+0000","last_clean":"2024-03-19T02:25:13.203926+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:25:13.203926+0000","last_undegraded":"2024-03-19T02:25:13.203926+0000","last_fullsized":"2024-03-19T02:25:13.203926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T02:25:13.247092+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:25:13.247092+0000","last_peered":"2024-03-19T02:25:13.247092+0000","last_clean":"2024-03-19T02:25:13.247092+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:25:13.247092+0000","last_undegraded":"2024-03-19T02:25:13.247092+0000","last_fullsized":"2024-03-19T02:25:13.247092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10050,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":176,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":64237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":485554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612629,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1075432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":481084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":481766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1077872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462956,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":480988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":505159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":480356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":485458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":480674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":480932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":42223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:25:38.430 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T02:25:38.430 INFO:tasks.cephadm:Setting up client nodes... 2024-03-19T02:25:38.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T02:25:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:38.974 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:38 smithi116 ceph-mon[32263]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:38.975 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:38 smithi116 ceph-mon[32263]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-19T02:25:38.975 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:38 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4163933166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:39.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:38 smithi154 ceph-mon[35693]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:38 smithi154 ceph-mon[35693]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-19T02:25:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:38 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4163933166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:38 smithi162 ceph-mon[35761]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:25:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:38 smithi162 ceph-mon[35761]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-19T02:25:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:38 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4163933166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:39 smithi154 ceph-mon[35693]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:39 smithi154 ceph-mon[35693]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:25:39.606+0000 7fbadc802700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-19T02:25:40.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:25:39.606+0000 7fbadc802700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-19T02:25:40.006 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:25:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:25:39.606+0000 7fbadc802700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-19T02:25:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:39 smithi116 ceph-mon[32263]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:39 smithi116 ceph-mon[32263]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:39 smithi162 ceph-mon[35761]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:39 smithi162 ceph-mon[35761]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:25:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 -- 172.21.15.116:0/3601955244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c103f90 msgr2=0x7f730c10a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 --2- 172.21.15.116:0/3601955244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c103f90 0x7f730c10a840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009990 tx=0x7f72fc009ca0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 -- 172.21.15.116:0/3601955244 shutdown_connections 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 --2- 172.21.15.116:0/3601955244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c103f90 0x7f730c10a840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 --2- 172.21.15.116:0/3601955244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c100610 0x7f730c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 --2- 172.21.15.116:0/3601955244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c1035f0 0x7f730c103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 -- 172.21.15.116:0/3601955244 >> 172.21.15.116:0/3601955244 conn(0x7f730c0fbfb0 msgr2=0x7f730c0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 -- 172.21.15.116:0/3601955244 shutdown_connections 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 -- 172.21.15.116:0/3601955244 wait complete. 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.557+0000 7f73115d5700 1 Processor -- start 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 -- start start 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c1035f0 0x7f730c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c103f90 0x7f730c19f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:40.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f730c10ddc0 con 0x7f730c100610 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f730c10dc40 con 0x7f730c1035f0 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f73115d5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f730c10df40 con 0x7f730c103f90 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53482/0 (socket says 172.21.15.116:53482) 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 -- 172.21.15.116:0/4255553451 learned_addr learned my addr 172.21.15.116:0/4255553451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730a7fc700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c1035f0 0x7f730c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730b7fe700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c103f90 0x7f730c19f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 -- 172.21.15.116:0/4255553451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c103f90 msgr2=0x7f730c19f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c103f90 0x7f730c19f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 -- 172.21.15.116:0/4255553451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c1035f0 msgr2=0x7f730c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c1035f0 0x7f730c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.558+0000 7f730affd700 1 -- 172.21.15.116:0/4255553451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c19ffb0 con 0x7f730c100610 2024-03-19T02:25:40.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f730affd700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7304009f40 tx=0x7f730400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73040092e0 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730400c9c0 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c1a0240 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304004470 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.559+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c1a0780 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.560+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c050820 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.561+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f730401b030 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.561+0000 7f72f3fff700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 0x7f72f40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.561+0000 7f730a7fc700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 0x7f72f40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.562+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f73040ced20 con 0x7f730c100610 2024-03-19T02:25:40.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.562+0000 7f730a7fc700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 0x7f72f40a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7300005950 tx=0x7f730000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:40.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.566+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7304022020 con 0x7f730c100610 2024-03-19T02:25:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.848+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f730c02dcf0 con 0x7f730c100610 2024-03-19T02:25:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.850+0000 7f72f3fff700 1 -- 172.21.15.116:0/4255553451 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f730c02dcf0 con 0x7f730c100610 2024-03-19T02:25:40.851 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-19T02:25:40.851 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCk9/hl4ZCtMhAA6uqmJdqAv5SS2XCIrB2r6w== 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 msgr2=0x7f72f40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 0x7f72f40a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7300005950 tx=0x7f730000b410 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 msgr2=0x7f730c198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7304009f40 tx=0x7f730400c3a0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 shutdown_connections 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72f40a29d0 0x7f72f40a4e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f730c103f90 0x7f730c19f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f730c100610 0x7f730c198350 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 --2- 172.21.15.116:0/4255553451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f730c1035f0 0x7f730c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.852+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 >> 172.21.15.116:0/4255553451 conn(0x7f730c0fbfb0 msgr2=0x7f730c106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.853+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 shutdown_connections 2024-03-19T02:25:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:40.853+0000 7f73115d5700 1 -- 172.21.15.116:0/4255553451 wait complete. 2024-03-19T02:25:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:40 smithi154 ceph-mon[35693]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-03-19T02:25:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:40 smithi116 ceph-mon[32263]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-03-19T02:25:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:40 smithi162 ceph-mon[35761]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-03-19T02:25:41.677 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:25:41.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-19T02:25:41.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T02:25:41.723 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T02:25:41.723 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-19T02:25:41.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-19T02:25:41.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:25:41.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:25:42.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:25:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:25:42.014 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:25:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4255553451' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:25:43.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:42 smithi154 ceph-mon[35693]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:42 smithi116 ceph-mon[32263]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:42 smithi162 ceph-mon[35761]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.908+0000 7fab413bf700 1 -- 172.21.15.116:0/3455297027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1064f0 msgr2=0x7fab3c10e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.908+0000 7fab413bf700 1 --2- 172.21.15.116:0/3455297027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1064f0 0x7fab3c10e0c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0099c0 tx=0x7fab2c009cd0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 -- 172.21.15.116:0/3455297027 shutdown_connections 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 --2- 172.21.15.116:0/3455297027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1064f0 0x7fab3c10e0c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 --2- 172.21.15.116:0/3455297027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c1014a0 0x7fab3c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 --2- 172.21.15.116:0/3455297027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c101e50 0x7fab3c105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 -- 172.21.15.116:0/3455297027 >> 172.21.15.116:0/3455297027 conn(0x7fab3c077260 msgr2=0x7fab3c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 -- 172.21.15.116:0/3455297027 shutdown_connections 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 -- 172.21.15.116:0/3455297027 wait complete. 2024-03-19T02:25:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 Processor -- start 2024-03-19T02:25:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.909+0000 7fab413bf700 1 -- start start 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c101e50 0x7fab3c1033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c1064f0 0x7fab3c103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c1113b0 con 0x7fab3c1014a0 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c111230 con 0x7fab3c1064f0 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab413bf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c111530 con 0x7fab3c101e50 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:43.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53492/0 (socket says 172.21.15.116:53492) 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 -- 172.21.15.116:0/212728571 learned_addr learned my addr 172.21.15.116:0/212728571 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3a7fc700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c101e50 0x7fab3c1033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3b7fe700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c1064f0 0x7fab3c103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 -- 172.21.15.116:0/212728571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c101e50 msgr2=0x7fab3c1033f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c101e50 0x7fab3c1033f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 -- 172.21.15.116:0/212728571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c1064f0 msgr2=0x7fab3c103930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c1064f0 0x7fab3c103930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 -- 172.21.15.116:0/212728571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c1a6ee0 con 0x7fab3c1014a0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab3affd700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fab3400ef30 tx=0x7fab3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.910+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab340092e0 con 0x7fab3c1014a0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.911+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c1a7170 con 0x7fab3c1014a0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.911+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab34004bb0 con 0x7fab3c1014a0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.911+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34004040 con 0x7fab3c1014a0 2024-03-19T02:25:43.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.911+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c1a76b0 con 0x7fab3c1014a0 2024-03-19T02:25:43.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.912+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab3c068fd0 con 0x7fab3c1014a0 2024-03-19T02:25:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.913+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fab3401b030 con 0x7fab3c1014a0 2024-03-19T02:25:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.914+0000 7fab23fff700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 0x7fab240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.914+0000 7fab3a7fc700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 0x7fab240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.914+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fab340ce5e0 con 0x7fab3c1014a0 2024-03-19T02:25:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.914+0000 7fab3a7fc700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 0x7fab240a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fab3c107b50 tx=0x7fab30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:43.918+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab34010040 con 0x7fab3c1014a0 2024-03-19T02:25:44.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.173+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fab3c02dbb0 con 0x7fab3c1014a0 2024-03-19T02:25:44.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.177+0000 7fab23fff700 1 -- 172.21.15.116:0/212728571 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262669 (secure 0 0 0) 0x7fab3409b370 con 0x7fab3c1014a0 2024-03-19T02:25:44.182 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 msgr2=0x7fab240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 0x7fab240a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fab3c107b50 tx=0x7fab30009260 comp rx=0 tx=0).stop 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 msgr2=0x7fab3c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fab3400ef30 tx=0x7fab3400c3a0 comp rx=0 tx=0).stop 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 shutdown_connections 2024-03-19T02:25:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab3c1064f0 0x7fab3c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1014a0 0x7fab3c102eb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fab240a2920 0x7fab240a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 --2- 172.21.15.116:0/212728571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab3c101e50 0x7fab3c1033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 >> 172.21.15.116:0/212728571 conn(0x7fab3c077260 msgr2=0x7fab3c06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 shutdown_connections 2024-03-19T02:25:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:44.181+0000 7fab413bf700 1 -- 172.21.15.116:0/212728571 wait complete. 2024-03-19T02:25:44.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:44 smithi116 ceph-mon[32263]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:44.839 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/212728571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:25:44.867 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":3573130182},{"type":"v1","addr":"172.21.15.116:6801","nonce":3573130182}]},"active_addr":"172.21.15.116:6801/3573130182","active_change":"2024-03-19T02:18:07.512344+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2284944336}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1256325072}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":953095802}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2648298519}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2172219038}]}]} 2024-03-19T02:25:44.872 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-19T02:25:44.872 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-19T02:25:44.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:25:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:44 smithi154 ceph-mon[35693]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/212728571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:25:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:44 smithi162 ceph-mon[35761]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.8 KiB/s wr, 20 op/s 2024-03-19T02:25:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/212728571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:25:46.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:46 smithi116 ceph-mon[32263]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-19T02:25:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 -- 172.21.15.116:0/2962637358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 msgr2=0x7efecc108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 --2- 172.21.15.116:0/2962637358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc108740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efebc0099c0 tx=0x7efebc009cd0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 -- 172.21.15.116:0/2962637358 shutdown_connections 2024-03-19T02:25:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 --2- 172.21.15.116:0/2962637358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc06a960 0x7efecc10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 --2- 172.21.15.116:0/2962637358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc108740 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.987+0000 7efed19b8700 1 --2- 172.21.15.116:0/2962637358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc069fc0 0x7efecc06a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.988+0000 7efed19b8700 1 -- 172.21.15.116:0/2962637358 >> 172.21.15.116:0/2962637358 conn(0x7efecc078140 msgr2=0x7efecc078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.988+0000 7efed19b8700 1 -- 172.21.15.116:0/2962637358 shutdown_connections 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.988+0000 7efed19b8700 1 -- 172.21.15.116:0/2962637358 wait complete. 2024-03-19T02:25:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.988+0000 7efed19b8700 1 Processor -- start 2024-03-19T02:25:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.988+0000 7efed19b8700 1 -- start start 2024-03-19T02:25:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc069fc0 0x7efecc19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 0x7efecc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efecc112450 con 0x7efecc108360 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efecc1122d0 con 0x7efecc06a960 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efecc1125d0 con 0x7efecc069fc0 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53520/0 (socket says 172.21.15.116:53520) 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 -- 172.21.15.116:0/2468306427 learned_addr learned my addr 172.21.15.116:0/2468306427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efeca7fc700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 0x7efecc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecaffd700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc069fc0 0x7efecc19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 -- 172.21.15.116:0/2468306427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc069fc0 msgr2=0x7efecc19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc069fc0 0x7efecc19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 -- 172.21.15.116:0/2468306427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 msgr2=0x7efecc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 0x7efecc19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 -- 172.21.15.116:0/2468306427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efecc198f70 con 0x7efecc108360 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efeca7fc700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 0x7efecc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efecb7fe700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efec400ef30 tx=0x7efec400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40092e0 con 0x7efecc108360 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efec4004bb0 con 0x7efecc108360 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efecc199200 con 0x7efecc108360 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4004020 con 0x7efecc108360 2024-03-19T02:25:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.989+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efecc1a5470 con 0x7efecc108360 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.991+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efecc103a10 con 0x7efecc108360 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.991+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7efec401b030 con 0x7efecc108360 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.992+0000 7efed09b6700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 0x7efeb40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.992+0000 7efecaffd700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 0x7efeb40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.992+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efec40ce880 con 0x7efecc108360 2024-03-19T02:25:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.992+0000 7efecaffd700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 0x7efeb40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efebc003760 tx=0x7efebc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:46.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:46.997+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efec4022020 con 0x7efecc108360 2024-03-19T02:25:47.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:46 smithi154 ceph-mon[35693]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-19T02:25:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:46 smithi162 ceph-mon[35761]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-19T02:25:47.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.191+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efecc02dbb0 con 0x7efecc108360 2024-03-19T02:25:47.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.192+0000 7efed09b6700 1 -- 172.21.15.116:0/2468306427 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7efec4010040 con 0x7efecc108360 2024-03-19T02:25:47.193 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:47.193 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:47.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 msgr2=0x7efeb40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 0x7efeb40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efebc003760 tx=0x7efebc005fb0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 msgr2=0x7efecc198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efec400ef30 tx=0x7efec400c3a0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 shutdown_connections 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efecc108360 0x7efecc198790 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efeb40a2920 0x7efeb40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efecc069fc0 0x7efecc19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 --2- 172.21.15.116:0/2468306427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efecc06a960 0x7efecc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.194+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 >> 172.21.15.116:0/2468306427 conn(0x7efecc078140 msgr2=0x7efecc0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.195+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 shutdown_connections 2024-03-19T02:25:47.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:47.195+0000 7efed19b8700 1 -- 172.21.15.116:0/2468306427 wait complete. 2024-03-19T02:25:47.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:47 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2468306427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:47.994 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-19T02:25:47.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:25:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:47 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2468306427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:48.068 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:47 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2468306427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:48.239 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:48 smithi154 ceph-mon[35693]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.3 KiB/s wr, 23 op/s 2024-03-19T02:25:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:48 smithi116 ceph-mon[32263]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.3 KiB/s wr, 23 op/s 2024-03-19T02:25:49.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:48 smithi162 ceph-mon[35761]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.3 KiB/s wr, 23 op/s 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2438665762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 msgr2=0x7f52f01087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2438665762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f01087f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009990 tx=0x7f52e0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2438665762 shutdown_connections 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2438665762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102d20 0x7f52f010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2438665762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f01087f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2438665762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102360 0x7f52f01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.021+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2438665762 >> 172.21.15.116:0/2438665762 conn(0x7f52f0079440 msgr2=0x7f52f0079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.022+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2438665762 shutdown_connections 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.022+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2438665762 wait complete. 2024-03-19T02:25:50.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.022+0000 7f52f4c1f700 1 Processor -- start 2024-03-19T02:25:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.022+0000 7f52f4c1f700 1 -- start start 2024-03-19T02:25:50.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 0x7f52f01a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 0x7f52f01a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52f0112500 con 0x7f52f0108410 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52f0112380 con 0x7f52f0102d20 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52f4c1f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52f0112680 con 0x7f52f0102360 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:50.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50152/0 (socket says 172.21.15.116:50152) 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 -- 172.21.15.116:0/2712297400 learned_addr learned my addr 172.21.15.116:0/2712297400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52ee59c700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 0x7f52f01a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52edd9b700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 0x7f52f01a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 -- 172.21.15.116:0/2712297400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 msgr2=0x7f52f01a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 0x7f52f01a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 -- 172.21.15.116:0/2712297400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 msgr2=0x7f52f01a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 0x7f52f01a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 -- 172.21.15.116:0/2712297400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f019b120 con 0x7f52f0108410 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52edd9b700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 0x7f52f01a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52ee59c700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 0x7f52f01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52eed9d700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f52e800ef30 tx=0x7f52e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.023+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80092e0 con 0x7f52f0108410 2024-03-19T02:25:50.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.024+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f019b3b0 con 0x7f52f0108410 2024-03-19T02:25:50.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.024+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52e8004bb0 con 0x7f52f0108410 2024-03-19T02:25:50.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.024+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8004020 con 0x7f52f0108410 2024-03-19T02:25:50.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.024+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f01a76d0 con 0x7f52f0108410 2024-03-19T02:25:50.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.025+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f0068fd0 con 0x7f52f0108410 2024-03-19T02:25:50.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.026+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f52e801b030 con 0x7f52f0108410 2024-03-19T02:25:50.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.026+0000 7f52df7fe700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 0x7f52d80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:50.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.026+0000 7f52ee59c700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 0x7f52d80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:50.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.026+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f52e80ce870 con 0x7f52f0108410 2024-03-19T02:25:50.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.027+0000 7f52ee59c700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 0x7f52d80a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f52e000b2e0 tx=0x7f52e001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:50.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.031+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52e8022020 con 0x7f52f0108410 2024-03-19T02:25:50.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.225+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52f019bef0 con 0x7f52f0108410 2024-03-19T02:25:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.227+0000 7f52df7fe700 1 -- 172.21.15.116:0/2712297400 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24196 (secure 0 0 0) 0x7f52e8010040 con 0x7f52f0108410 2024-03-19T02:25:50.228 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:25:50.228 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":90,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:25:12.553908+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:25:50.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 msgr2=0x7f52d80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 0x7f52d80a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f52e000b2e0 tx=0x7f52e001a040 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 msgr2=0x7f52f019a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f52e800ef30 tx=0x7f52e800c3a0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 shutdown_connections 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52f0108410 0x7f52f019a9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f52d80a29a0 0x7f52d80a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52f0102360 0x7f52f01a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 --2- 172.21.15.116:0/2712297400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f0102d20 0x7f52f01a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 >> 172.21.15.116:0/2712297400 conn(0x7f52f0079440 msgr2=0x7f52f0074990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 shutdown_connections 2024-03-19T02:25:50.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:50.229+0000 7f52f4c1f700 1 -- 172.21.15.116:0/2712297400 wait complete. 2024-03-19T02:25:50.945 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:50 smithi116 ceph-mon[32263]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s 2024-03-19T02:25:50.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:50 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2712297400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:50.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-19T02:25:50.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-19T02:25:50.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-19T02:25:50.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-19T02:25:50.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-19T02:25:50.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-19T02:25:50.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-19T02:25:50.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-19T02:25:50.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-19T02:25:50.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-19T02:25:50.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-19T02:25:50.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-19T02:25:51.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:50 smithi154 ceph-mon[35693]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s 2024-03-19T02:25:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:50 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2712297400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:50 smithi162 ceph-mon[35761]: pgmap v311: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 15 op/s 2024-03-19T02:25:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:50 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2712297400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:25:51.669 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:51.673 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:51.674 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:52 smithi116 ceph-mon[32263]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-03-19T02:25:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:52 smithi162 ceph-mon[35761]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-03-19T02:25:53.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:52 smithi154 ceph-mon[35693]: pgmap v312: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-03-19T02:25:54.804 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:54 smithi116 ceph-mon[32263]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-19T02:25:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 -- 172.21.15.116:0/3488146600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40101bb0 msgr2=0x7fce4010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3488146600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40101bb0 0x7fce4010e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b0a0 tx=0x7fce3800b3b0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 -- 172.21.15.116:0/3488146600 shutdown_connections 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3488146600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40101bb0 0x7fce4010e090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3488146600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40107250 0x7fce40107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3488146600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce40101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 -- 172.21.15.116:0/3488146600 >> 172.21.15.116:0/3488146600 conn(0x7fce40078140 msgr2=0x7fce40078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 -- 172.21.15.116:0/3488146600 shutdown_connections 2024-03-19T02:25:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.807+0000 7fce44d7a700 1 -- 172.21.15.116:0/3488146600 wait complete. 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 Processor -- start 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 -- start start 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce401a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40101bb0 0x7fce401a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 0x7fce4019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce40111380 con 0x7fce40107250 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce40111200 con 0x7fce40101bb0 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce44d7a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce40111500 con 0x7fce40101210 2024-03-19T02:25:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 0x7fce4019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce401a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 0x7fce4019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50158/0 (socket says 172.21.15.116:50158) 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3e59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce401a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52862/0 (socket says 172.21.15.116:52862) 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 -- 172.21.15.116:0/3661709571 learned_addr learned my addr 172.21.15.116:0/3661709571 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce35d9b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40101bb0 0x7fce401a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 -- 172.21.15.116:0/3661709571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 msgr2=0x7fce401a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce401a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 -- 172.21.15.116:0/3661709571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40101bb0 msgr2=0x7fce401a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40101bb0 0x7fce401a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3ed9d700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4019b1b0 con 0x7fce40107250 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce35d9b700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce40101bb0 0x7fce401a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.808+0000 7fce3e59c700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce40101210 0x7fce401a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:25:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce3ed9d700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 0x7fce4019aa50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b780 tx=0x7fce38004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38021070 con 0x7fce40107250 2024-03-19T02:25:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce38007e30 con 0x7fce40107250 2024-03-19T02:25:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38004700 con 0x7fce40107250 2024-03-19T02:25:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4019b440 con 0x7fce40107250 2024-03-19T02:25:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.809+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce401a7760 con 0x7fce40107250 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.810+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce4006aaa0 con 0x7fce40107250 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.811+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fce38007980 con 0x7fce40107250 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.811+0000 7fce37fff700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fce2c0a2920 0x7fce2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce3e59c700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fce2c0a2920 0x7fce2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fce3800f020 con 0x7fce40107250 2024-03-19T02:25:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce37fff700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] conn(0x7fce2c0ad5e0 0x7fce2c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce3801ce50 con 0x7fce2c0ad5e0 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fce380d2630 con 0x7fce40107250 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce35d9b700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] conn(0x7fce2c0ad5e0 0x7fce2c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce3e59c700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fce2c0a2920 0x7fce2c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce4019bb90 tx=0x7fce30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.812+0000 7fce35d9b700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] conn(0x7fce2c0ad5e0 0x7fce2c0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:25:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.813+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== osd.0 v2:172.21.15.116:6802/3168976854 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce3801ce50 con 0x7fce2c0ad5e0 2024-03-19T02:25:54.825 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:54.828 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:54.828 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:54.828 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:54.828 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:54.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.839+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 --> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce40102900 con 0x7fce2c0ad5e0 2024-03-19T02:25:54.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.840+0000 7fce37fff700 1 -- 172.21.15.116:0/3661709571 <== osd.0 v2:172.21.15.116:6802/3168976854 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fce40102900 con 0x7fce2c0ad5e0 2024-03-19T02:25:54.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.841+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] conn(0x7fce2c0ad5e0 msgr2=0x7fce2c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.841+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] conn(0x7fce2c0ad5e0 0x7fce2c0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:25:54.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.841+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fce2c0a2920 msgr2=0x7fce2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.842+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fce2c0a2920 0x7fce2c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce4019bb90 tx=0x7fce30009260 comp rx=0 tx=0).stop 2024-03-19T02:25:54.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.842+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 msgr2=0x7fce4019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:25:54.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.842+0000 7fce44d7a700 1 --2- 172.21.15.116:0/3661709571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce40107250 0x7fce4019aa50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b780 tx=0x7fce38004d00 comp rx=0 tx=0).stop 2024-03-19T02:25:54.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.842+0000 7fce3ed9d700 1 -- 172.21.15.116:0/3661709571 reap_dead start 2024-03-19T02:25:54.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.843+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 shutdown_connections 2024-03-19T02:25:54.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.843+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 >> 172.21.15.116:0/3661709571 conn(0x7fce40078140 msgr2=0x7fce4006a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:25:54.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.843+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 shutdown_connections 2024-03-19T02:25:54.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:25:54.843+0000 7fce44d7a700 1 -- 172.21.15.116:0/3661709571 wait complete. 2024-03-19T02:25:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:54 smithi162 ceph-mon[35761]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-19T02:25:55.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:54 smithi154 ceph-mon[35693]: pgmap v313: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-19T02:25:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:56 smithi162 ceph-mon[35761]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-19T02:25:57.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:56 smithi116 ceph-mon[32263]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-19T02:25:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:56 smithi154 ceph-mon[35693]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-19T02:25:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:25:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:25:58 smithi162 ceph-mon[35761]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-19T02:25:59.225 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:25:58 smithi116 ceph-mon[32263]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-19T02:25:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:25:58 smithi154 ceph-mon[35693]: pgmap v315: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-19T02:26:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:00 smithi116 ceph-mon[32263]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-19T02:26:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:00 smithi162 ceph-mon[35761]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-19T02:26:01.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:00 smithi154 ceph-mon[35693]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-19T02:26:02.972 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:02 smithi116 ceph-mon[32263]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:02 smithi154 ceph-mon[35693]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:02 smithi162 ceph-mon[35761]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- 172.21.15.116:0/1904611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806ac70 msgr2=0x7f039806b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 --2- 172.21.15.116:0/1904611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806ac70 0x7f039806b050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f038c0099c0 tx=0x7f038c009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- 172.21.15.116:0/1904611116 shutdown_connections 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 --2- 172.21.15.116:0/1904611116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0398107690 0x7f039810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 --2- 172.21.15.116:0/1904611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806ac70 0x7f039806b050 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 --2- 172.21.15.116:0/1904611116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806b620 0x7f0398106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- 172.21.15.116:0/1904611116 >> 172.21.15.116:0/1904611116 conn(0x7f0398077230 msgr2=0x7f0398077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- 172.21.15.116:0/1904611116 shutdown_connections 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- 172.21.15.116:0/1904611116 wait complete. 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 Processor -- start 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.629+0000 7f039f032700 1 -- start start 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806ac70 0x7f039819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 0x7f039819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0398107690 0x7f03981988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0398112550 con 0x7f039806b620 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03981123d0 con 0x7f039806ac70 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03981126d0 con 0x7f0398107690 2024-03-19T02:26:03.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0397fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 0x7f039819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0397fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 0x7f039819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46074/0 (socket says 172.21.15.116:46074) 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0397fff700 1 -- 172.21.15.116:0/3018006064 learned_addr learned my addr 172.21.15.116:0/3018006064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806ac70 0x7f039819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039d5cf700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0398107690 0x7f03981988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 -- 172.21.15.116:0/3018006064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0398107690 msgr2=0x7f03981988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0398107690 0x7f03981988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 -- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 msgr2=0x7f039819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 0x7f039819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0398199050 con 0x7f039806ac70 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0397fff700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f039806b620 0x7f039819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039cdce700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806ac70 0x7f039819e6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f038c0098e0 tx=0x7f038c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c026070 con 0x7f039806ac70 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03981992e0 con 0x7f039806ac70 2024-03-19T02:26:03.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f038c003c70 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.630+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c00f9e0 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.631+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03981a55b0 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.631+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0398062a60 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.633+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f038c029070 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.633+0000 7f0395ffb700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f03880a2920 0x7f03880a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.633+0000 7f0397fff700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f03880a2920 0x7f03880a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.633+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f038c0cfbd0 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f0395ffb700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] conn(0x7f03880ad4b0 0x7f03880af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f038c00b790 con 0x7f03880ad4b0 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f038c0cff80 con 0x7f039806ac70 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f039d5cf700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] conn(0x7f03880ad4b0 0x7f03880af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f0397fff700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f03880a2920 0x7f03880a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0398100f00 tx=0x7f0384009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.634+0000 7f039d5cf700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] conn(0x7f03880ad4b0 0x7f03880af8f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.635+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== osd.6 v2:172.21.15.116:6818/119271604 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f038c00b790 con 0x7f03880ad4b0 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 --> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0398062e60 con 0x7f03880ad4b0 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f0395ffb700 1 -- 172.21.15.116:0/3018006064 <== osd.6 v2:172.21.15.116:6818/119271604 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0398062e60 con 0x7f03880ad4b0 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] conn(0x7f03880ad4b0 msgr2=0x7f03880af8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] conn(0x7f03880ad4b0 0x7f03880af8f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f03880a2920 msgr2=0x7f03880a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f03880a2920 0x7f03880a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0398100f00 tx=0x7f0384009260 comp rx=0 tx=0).stop 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806ac70 msgr2=0x7f039819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 --2- 172.21.15.116:0/3018006064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f039806ac70 0x7f039819e6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f038c0098e0 tx=0x7f038c004270 comp rx=0 tx=0).stop 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039d5cf700 1 -- 172.21.15.116:0/3018006064 reap_dead start 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 shutdown_connections 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 >> 172.21.15.116:0/3018006064 conn(0x7f0398077230 msgr2=0x7f03980ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 shutdown_connections 2024-03-19T02:26:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:02.651+0000 7f039f032700 1 -- 172.21.15.116:0/3018006064 wait complete. 2024-03-19T02:26:04.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/1649933521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 msgr2=0x7fdb04100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/1649933521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb04100d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009990 tx=0x7fdaf4009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/1649933521 shutdown_connections 2024-03-19T02:26:04.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/1649933521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb041012c0 0x7fdb0410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/1649933521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb04107380 0x7fdb04107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/1649933521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb04100d80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/1649933521 >> 172.21.15.116:0/1649933521 conn(0x7fdb04077810 msgr2=0x7fdb040fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/1649933521 shutdown_connections 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.309+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/1649933521 wait complete. 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 Processor -- start 2024-03-19T02:26:04.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 -- start start 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb0419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041012c0 0x7fdb0419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb04107380 0x7fdb041989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112610 con 0x7fdb04100900 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112490 con 0x7fdb041012c0 2024-03-19T02:26:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb0a8f2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04112790 con 0x7fdb04107380 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb0419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb037fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041012c0 0x7fdb0419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb0419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46092/0 (socket says 172.21.15.116:46092) 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb03fff700 1 -- 172.21.15.116:0/958536568 learned_addr learned my addr 172.21.15.116:0/958536568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.310+0000 7fdb08e8f700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb04107380 0x7fdb041989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 -- 172.21.15.116:0/958536568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb04107380 msgr2=0x7fdb041989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb04107380 0x7fdb041989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 -- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 msgr2=0x7fdb0419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb04100900 0x7fdb0419e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb04199110 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb08e8f700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb04107380 0x7fdb041989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb037fe700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041012c0 0x7fdb0419ed30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40059d0 tx=0x7fdaf4004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4027070 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb041993a0 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdaf4022470 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400b610 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.311+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb041a56c0 con 0x7fdb041012c0 2024-03-19T02:26:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.312+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb04062a20 con 0x7fdb041012c0 2024-03-19T02:26:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.313+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fdaf402a070 con 0x7fdb041012c0 2024-03-19T02:26:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb017fa700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdaec0a2920 0x7fdaec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fdaf40d02d0 con 0x7fdb041012c0 2024-03-19T02:26:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb03fff700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdaec0a2920 0x7fdaec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb017fa700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] conn(0x7fdaec0ad5e0 0x7fdaec0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdaf400eea0 con 0x7fdaec0ad5e0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdaf40d0680 con 0x7fdb041012c0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb08e8f700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] conn(0x7fdaec0ad5e0 0x7fdaec0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb03fff700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdaec0a2920 0x7fdaec0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdb04199af0 tx=0x7fdaf8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.314+0000 7fdb08e8f700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] conn(0x7fdaec0ad5e0 0x7fdaec0afa20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.315+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== osd.10 v2:172.21.15.154:6824/1964548863 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdaf400eea0 con 0x7fdaec0ad5e0 2024-03-19T02:26:04.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.332+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 --> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb04062e20 con 0x7fdaec0ad5e0 2024-03-19T02:26:04.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.332+0000 7fdb017fa700 1 -- 172.21.15.116:0/958536568 <== osd.10 v2:172.21.15.154:6824/1964548863 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdb04062e20 con 0x7fdaec0ad5e0 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.332+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] conn(0x7fdaec0ad5e0 msgr2=0x7fdaec0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.332+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] conn(0x7fdaec0ad5e0 0x7fdaec0afa20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdaec0a2920 msgr2=0x7fdaec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdaec0a2920 0x7fdaec0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdb04199af0 tx=0x7fdaf8009260 comp rx=0 tx=0).stop 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041012c0 msgr2=0x7fdb0419ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 --2- 172.21.15.116:0/958536568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041012c0 0x7fdb0419ed30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40059d0 tx=0x7fdaf4004790 comp rx=0 tx=0).stop 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb08e8f700 1 -- 172.21.15.116:0/958536568 reap_dead start 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 shutdown_connections 2024-03-19T02:26:04.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 >> 172.21.15.116:0/958536568 conn(0x7fdb04077810 msgr2=0x7fdb041053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:04.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 shutdown_connections 2024-03-19T02:26:04.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.333+0000 7fdb0a8f2700 1 -- 172.21.15.116:0/958536568 wait complete. 2024-03-19T02:26:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 -- 172.21.15.116:0/1736075335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102360 msgr2=0x7fe61c1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1736075335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102360 0x7fe61c1027e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009990 tx=0x7fe60c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 -- 172.21.15.116:0/1736075335 shutdown_connections 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1736075335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102d20 0x7fe61c10f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1736075335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 0x7fe61c1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1736075335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102360 0x7fe61c1027e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 -- 172.21.15.116:0/1736075335 >> 172.21.15.116:0/1736075335 conn(0x7fe61c079440 msgr2=0x7fe61c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 -- 172.21.15.116:0/1736075335 shutdown_connections 2024-03-19T02:26:04.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.729+0000 7fe623f5c700 1 -- 172.21.15.116:0/1736075335 wait complete. 2024-03-19T02:26:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 Processor -- start 2024-03-19T02:26:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 -- start start 2024-03-19T02:26:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102360 0x7fe61c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 0x7fe61c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 0x7fe61c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c1124d0 con 0x7fe61c102d20 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c112350 con 0x7fe61c108410 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe623f5c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c112650 con 0x7fe61c102360 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe6214f7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 0x7fe61c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe6224f9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 0x7fe61c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe6214f7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 0x7fe61c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46122/0 (socket says 172.21.15.116:46122) 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe6214f7700 1 -- 172.21.15.116:0/1803207326 learned_addr learned my addr 172.21.15.116:0/1803207326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.730+0000 7fe621cf8700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102360 0x7fe61c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 -- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102360 msgr2=0x7fe61c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102360 0x7fe61c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 -- 172.21.15.116:0/1803207326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 msgr2=0x7fe61c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 0x7fe61c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61c198f90 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6224f9700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c108410 0x7fe61c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe621cf8700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe61c102360 0x7fe61c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe6214f7700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 0x7fe61c19eb70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe60c000c00 tx=0x7fe60c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c027070 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61c199220 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe60c022470 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c00b610 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.731+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61c1a5490 con 0x7fe61c102d20 2024-03-19T02:26:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.732+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe61c062a20 con 0x7fe61c102d20 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.733+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fe60c02a070 con 0x7fe61c102d20 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe612ffd700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6080a2920 0x7fe6080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe621cf8700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6080a2920 0x7fe6080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe60c0d0210 con 0x7fe61c102d20 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe612ffd700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] conn(0x7fe6080ad5e0 0x7fe6080afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe60c001d30 con 0x7fe6080ad5e0 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe60c0d06f0 con 0x7fe61c102d20 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe621cf8700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6080a2920 0x7fe6080a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe618011490 tx=0x7fe618009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.734+0000 7fe6224f9700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] conn(0x7fe6080ad5e0 0x7fe6080afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.735+0000 7fe6224f9700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] conn(0x7fe6080ad5e0 0x7fe6080afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.736+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== osd.11 v2:172.21.15.162:6824/3174333504 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe60c001d30 con 0x7fe6080ad5e0 2024-03-19T02:26:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.752+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 --> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe61c062e20 con 0x7fe6080ad5e0 2024-03-19T02:26:04.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.752+0000 7fe612ffd700 1 -- 172.21.15.116:0/1803207326 <== osd.11 v2:172.21.15.162:6824/3174333504 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe61c062e20 con 0x7fe6080ad5e0 2024-03-19T02:26:04.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] conn(0x7fe6080ad5e0 msgr2=0x7fe6080afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] conn(0x7fe6080ad5e0 0x7fe6080afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:04.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6080a2920 msgr2=0x7fe6080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe6080a2920 0x7fe6080a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe618011490 tx=0x7fe618009260 comp rx=0 tx=0).stop 2024-03-19T02:26:04.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 msgr2=0x7fe61c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 --2- 172.21.15.116:0/1803207326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe61c102d20 0x7fe61c19eb70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe60c000c00 tx=0x7fe60c004790 comp rx=0 tx=0).stop 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe6224f9700 1 -- 172.21.15.116:0/1803207326 reap_dead start 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 shutdown_connections 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 >> 172.21.15.116:0/1803207326 conn(0x7fe61c079440 msgr2=0x7fe61c1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 shutdown_connections 2024-03-19T02:26:04.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:04.753+0000 7fe623f5c700 1 -- 172.21.15.116:0/1803207326 wait complete. 2024-03-19T02:26:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:04 smithi116 ceph-mon[32263]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:04 smithi162 ceph-mon[35761]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:05.225 INFO:teuthology.orchestra.run.smithi116.stdout:201863462962 2024-03-19T02:26:05.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-19T02:26:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:04 smithi154 ceph-mon[35693]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 -- 172.21.15.116:0/1999069452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 msgr2=0x7f227c06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 --2- 172.21.15.116:0/1999069452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c06ae70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f22700099c0 tx=0x7f2270009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 -- 172.21.15.116:0/1999069452 shutdown_connections 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 --2- 172.21.15.116:0/1999069452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c06b3b0 0x7f227c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 --2- 172.21.15.116:0/1999069452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c108390 0x7f227c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 --2- 172.21.15.116:0/1999069452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c06ae70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 -- 172.21.15.116:0/1999069452 >> 172.21.15.116:0/1999069452 conn(0x7f227c077230 msgr2=0x7f227c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 -- 172.21.15.116:0/1999069452 shutdown_connections 2024-03-19T02:26:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 -- 172.21.15.116:0/1999069452 wait complete. 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.422+0000 7f2282488700 1 Processor -- start 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 -- start start 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c06b3b0 0x7f227c1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c108390 0x7f227c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1124c0 con 0x7f227c06aa10 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f227c112340 con 0x7f227c06b3b0 2024-03-19T02:26:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2282488700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f227c112640 con 0x7f227c108390 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c1a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46140/0 (socket says 172.21.15.116:46140) 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227bfff700 1 -- 172.21.15.116:0/170663227 learned_addr learned my addr 172.21.15.116:0/170663227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227b7fe700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c06b3b0 0x7f227c1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f2280a25700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c108390 0x7f227c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227bfff700 1 -- 172.21.15.116:0/170663227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c108390 msgr2=0x7f227c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.423+0000 7f227bfff700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c108390 0x7f227c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f227bfff700 1 -- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c06b3b0 msgr2=0x7f227c1a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f227bfff700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c06b3b0 0x7f227c1a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f227bfff700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c19b150 con 0x7f227c06aa10 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f2280a25700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227c108390 0x7f227c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f227b7fe700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f227c06b3b0 0x7f227c1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f227bfff700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c1a0830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9a0 tx=0x7f226c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c00ca10 con 0x7f227c06aa10 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f226c00cbb0 con 0x7f227c06aa10 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c19b440 con 0x7f227c06aa10 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c018680 con 0x7f227c06aa10 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.424+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c1a7700 con 0x7f227c06aa10 2024-03-19T02:26:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.425+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f227c10df80 con 0x7f227c06aa10 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.426+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f226c010430 con 0x7f227c06aa10 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f22797fa700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22640a29a0 0x7f22640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f227b7fe700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22640a29a0 0x7f22640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f226c0cf360 con 0x7f227c06aa10 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f22797fa700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] conn(0x7f22640ad660 0x7f22640afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f226c01bec0 con 0x7f22640ad660 2024-03-19T02:26:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f226c0cf7b0 con 0x7f227c06aa10 2024-03-19T02:26:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f2280a25700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] conn(0x7f22640ad660 0x7f22640afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.427+0000 7f227b7fe700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22640a29a0 0x7f22640a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f227c103370 tx=0x7f2270005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.428+0000 7f2280a25700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] conn(0x7f22640ad660 0x7f22640afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.428+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== osd.7 v2:172.21.15.154:6816/2515714011 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f226c01bec0 con 0x7f22640ad660 2024-03-19T02:26:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.444+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 --> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f227c103a40 con 0x7f22640ad660 2024-03-19T02:26:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f22797fa700 1 -- 172.21.15.116:0/170663227 <== osd.7 v2:172.21.15.154:6816/2515714011 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f227c103a40 con 0x7f22640ad660 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] conn(0x7f22640ad660 msgr2=0x7f22640afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] conn(0x7f22640ad660 0x7f22640afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22640a29a0 msgr2=0x7f22640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22640a29a0 0x7f22640a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f227c103370 tx=0x7f2270005fb0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 msgr2=0x7f227c1a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 --2- 172.21.15.116:0/170663227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227c06aa10 0x7f227c1a0830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9a0 tx=0x7f226c00ecb0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2280a25700 1 -- 172.21.15.116:0/170663227 reap_dead start 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 shutdown_connections 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 >> 172.21.15.116:0/170663227 conn(0x7f227c077230 msgr2=0x7f227c10d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 shutdown_connections 2024-03-19T02:26:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.445+0000 7f2282488700 1 -- 172.21.15.116:0/170663227 wait complete. 2024-03-19T02:26:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 -- 172.21.15.116:0/1447089003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664107250 msgr2=0x7fb664107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 --2- 172.21.15.116:0/1447089003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664107250 0x7fb664107630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00b0a0 tx=0x7fb65c00b3b0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 -- 172.21.15.116:0/1447089003 shutdown_connections 2024-03-19T02:26:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 --2- 172.21.15.116:0/1447089003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb664101bb0 0x7fb66410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 --2- 172.21.15.116:0/1447089003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664107250 0x7fb664107630 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 --2- 172.21.15.116:0/1447089003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb664101210 0x7fb664101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 -- 172.21.15.116:0/1447089003 >> 172.21.15.116:0/1447089003 conn(0x7fb664078140 msgr2=0x7fb664078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 -- 172.21.15.116:0/1447089003 shutdown_connections 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.756+0000 7fb668e52700 1 -- 172.21.15.116:0/1447089003 wait complete. 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 Processor -- start 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 -- start start 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb664101210 0x7fb6641997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 0x7fb664199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb664107250 0x7fb6641a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb664111360 con 0x7fb664101bb0 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6641111e0 con 0x7fb664101210 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb668e52700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6641114e0 con 0x7fb664107250 2024-03-19T02:26:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb661d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 0x7fb664199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb661d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 0x7fb664199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46168/0 (socket says 172.21.15.116:46168) 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb661d9b700 1 -- 172.21.15.116:0/1767534679 learned_addr learned my addr 172.21.15.116:0/1767534679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb66259c700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb664101210 0x7fb6641997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.757+0000 7fb662d9d700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb664107250 0x7fb6641a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 -- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb664107250 msgr2=0x7fb6641a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb664107250 0x7fb6641a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 -- 172.21.15.116:0/1767534679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb664101210 msgr2=0x7fb6641997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb664101210 0x7fb6641997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6641a4da0 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb661d9b700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 0x7fb664199d20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb65800e9c0 tx=0x7fb65800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65800cbd0 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb65800ee60 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6641a4fd0 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6580055b0 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.758+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6641a5430 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.759+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6640629e0 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.760+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb658004750 con 0x7fb664101bb0 2024-03-19T02:26:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb6537fe700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb64c0a2920 0x7fb64c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb66259c700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb64c0a2920 0x7fb64c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb6580cf4f0 con 0x7fb664101bb0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb6537fe700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] conn(0x7fb64c0ad5e0 0x7fb64c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb658004500 con 0x7fb64c0ad5e0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb6580cf940 con 0x7fb664101bb0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb662d9d700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] conn(0x7fb64c0ad5e0 0x7fb64c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb66259c700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb64c0a2920 0x7fb64c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009e60 tx=0x7fb65c00b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.761+0000 7fb662d9d700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] conn(0x7fb64c0ad5e0 0x7fb64c0afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.763+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== osd.8 v2:172.21.15.162:6816/2609002747 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb658004500 con 0x7fb64c0ad5e0 2024-03-19T02:26:05.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.782+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 --> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb664062e60 con 0x7fb64c0ad5e0 2024-03-19T02:26:05.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb6537fe700 1 -- 172.21.15.116:0/1767534679 <== osd.8 v2:172.21.15.162:6816/2609002747 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb664062e60 con 0x7fb64c0ad5e0 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] conn(0x7fb64c0ad5e0 msgr2=0x7fb64c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] conn(0x7fb64c0ad5e0 0x7fb64c0afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb64c0a2920 msgr2=0x7fb64c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb64c0a2920 0x7fb64c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009e60 tx=0x7fb65c00b700 comp rx=0 tx=0).stop 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 msgr2=0x7fb664199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 --2- 172.21.15.116:0/1767534679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb664101bb0 0x7fb664199d20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb65800e9c0 tx=0x7fb65800ecd0 comp rx=0 tx=0).stop 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb662d9d700 1 -- 172.21.15.116:0/1767534679 reap_dead start 2024-03-19T02:26:05.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 shutdown_connections 2024-03-19T02:26:05.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 >> 172.21.15.116:0/1767534679 conn(0x7fb664078140 msgr2=0x7fb664100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:05.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 shutdown_connections 2024-03-19T02:26:05.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:05.783+0000 7fb668e52700 1 -- 172.21.15.116:0/1767534679 wait complete. 2024-03-19T02:26:07.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:06 smithi162 ceph-mon[35761]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:07.252 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:06 smithi116 ceph-mon[32263]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:07.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:06 smithi154 ceph-mon[35693]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:07.698 INFO:teuthology.orchestra.run.smithi116.stdout:313532612636 2024-03-19T02:26:07.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-19T02:26:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 -- 172.21.15.116:0/3618001843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 msgr2=0x7f7b7410f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/3618001843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7410f1a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00b0d0 tx=0x7f7b6c00b3e0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 -- 172.21.15.116:0/3618001843 shutdown_connections 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/3618001843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7410f1a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/3618001843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 0x7f7b74108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/3618001843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b74069fc0 0x7f7b7406a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.895+0000 7f7b7ad79700 1 -- 172.21.15.116:0/3618001843 >> 172.21.15.116:0/3618001843 conn(0x7f7b74078140 msgr2=0x7f7b74078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- 172.21.15.116:0/3618001843 shutdown_connections 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- 172.21.15.116:0/3618001843 wait complete. 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 Processor -- start 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- start start 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b74069fc0 0x7f7b7419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 0x7f7b74198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:07.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74112490 con 0x7f7b7406a960 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74112310 con 0x7f7b74069fc0 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b7ad79700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7b74112610 con 0x7f7b74108360 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.896+0000 7f7b73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56214/0 (socket says 172.21.15.116:56214) 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 -- 172.21.15.116:0/2609139287 learned_addr learned my addr 172.21.15.116:0/2609139287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b79316700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 0x7f7b74198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b78b15700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b74069fc0 0x7f7b7419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 -- 172.21.15.116:0/2609139287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 msgr2=0x7f7b74198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 0x7f7b74198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 -- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b74069fc0 msgr2=0x7f7b7419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b74069fc0 0x7f7b7419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b74199050 con 0x7f7b7406a960 2024-03-19T02:26:07.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b79316700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b74108360 0x7f7b74198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b73fff700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7419ebe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000e9a0 tx=0x7f7b6000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000ca10 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b6000cbb0 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b74199340 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60018680 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.897+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b741a5620 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.898+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b74062a60 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.899+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7b60010430 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b71ffb700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b640a2920 0x7f7b640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b78b15700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b640a2920 0x7f7b640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7b600cf570 con 0x7f7b7406a960 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b71ffb700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] conn(0x7f7b640ada80 0x7f7b640aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:07.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b6001bec0 con 0x7f7b640ada80 2024-03-19T02:26:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7b600cf9c0 con 0x7f7b7406a960 2024-03-19T02:26:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b78b15700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b640a2920 0x7f7b640a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009be0 tx=0x7f7b68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.900+0000 7f7b79316700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] conn(0x7f7b640ada80 0x7f7b640aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.901+0000 7f7b79316700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] conn(0x7f7b640ada80 0x7f7b640aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:07.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.902+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== osd.1 v2:172.21.15.154:6800/3251901849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b6001bec0 con 0x7f7b640ada80 2024-03-19T02:26:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.924+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 --> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b74062ee0 con 0x7f7b640ada80 2024-03-19T02:26:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b71ffb700 1 -- 172.21.15.116:0/2609139287 <== osd.1 v2:172.21.15.154:6800/3251901849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b74062ee0 con 0x7f7b640ada80 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] conn(0x7f7b640ada80 msgr2=0x7f7b640aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] conn(0x7f7b640ada80 0x7f7b640aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b640a2920 msgr2=0x7f7b640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f7b640a2920 0x7f7b640a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009be0 tx=0x7f7b68009260 comp rx=0 tx=0).stop 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 msgr2=0x7f7b7419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b7ad79700 1 --2- 172.21.15.116:0/2609139287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b7406a960 0x7f7b7419ebe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000e9a0 tx=0x7f7b6000ecb0 comp rx=0 tx=0).stop 2024-03-19T02:26:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.925+0000 7f7b79316700 1 -- 172.21.15.116:0/2609139287 reap_dead start 2024-03-19T02:26:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.926+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 shutdown_connections 2024-03-19T02:26:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.926+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 >> 172.21.15.116:0/2609139287 conn(0x7f7b74078140 msgr2=0x7f7b740fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.926+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 shutdown_connections 2024-03-19T02:26:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:07.926+0000 7f7b7ad79700 1 -- 172.21.15.116:0/2609139287 wait complete. 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:07 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-19T02:26:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:07 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T02:26:08.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:07 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-19T02:26:08.385 INFO:teuthology.orchestra.run.smithi116.stdout:223338299437 2024-03-19T02:26:08.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-19T02:26:08.889 INFO:teuthology.orchestra.run.smithi116.stdout:34359738446 2024-03-19T02:26:08.889 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-19T02:26:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-19T02:26:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-19T02:26:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-19T02:26:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-19T02:26:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:08 smithi162 ceph-mon[35761]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:26:09.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:09.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-19T02:26:09.190 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-19T02:26:09.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:08 smithi116 ceph-mon[32263]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-19T02:26:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:26:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T02:26:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-19T02:26:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:08 smithi154 ceph-mon[35693]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:26:09.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:09 smithi116 ceph-mon[32263]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:26:09.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:26:09.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:09 smithi116 ceph-mon[32263]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:26:09.845 INFO:teuthology.orchestra.run.smithi116.stdout:55834574924 2024-03-19T02:26:09.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-19T02:26:10.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:09 smithi162 ceph-mon[35761]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:26:10.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:26:10.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:09 smithi162 ceph-mon[35761]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:26:10.186 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:09 smithi154 ceph-mon[35693]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:26:10.186 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:26:10.186 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:09 smithi154 ceph-mon[35693]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:26:10.500 INFO:teuthology.orchestra.run.smithi116.stdout:347892351000 2024-03-19T02:26:10.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-19T02:26:11.261 INFO:teuthology.orchestra.run.smithi116.stdout:257698037799 2024-03-19T02:26:11.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-19T02:26:12.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:11 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:11 smithi154 ceph-mon[35693]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-19T02:26:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:11 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:11 smithi116 ceph-mon[32263]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-19T02:26:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:11 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:11 smithi162 ceph-mon[35761]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-19T02:26:13.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 -- 172.21.15.116:0/1768384359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041011f0 msgr2=0x7fb504101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 --2- 172.21.15.116:0/1768384359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041011f0 0x7fb504101670 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009990 tx=0x7fb4f8009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 -- 172.21.15.116:0/1768384359 shutdown_connections 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 --2- 172.21.15.116:0/1768384359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb504101bb0 0x7fb50410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 --2- 172.21.15.116:0/1768384359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5041072a0 0x7fb504107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 --2- 172.21.15.116:0/1768384359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5041011f0 0x7fb504101670 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 -- 172.21.15.116:0/1768384359 >> 172.21.15.116:0/1768384359 conn(0x7fb504078220 msgr2=0x7fb504078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 -- 172.21.15.116:0/1768384359 shutdown_connections 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 -- 172.21.15.116:0/1768384359 wait complete. 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.897+0000 7fb5094f7700 1 Processor -- start 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 -- start start 2024-03-19T02:26:13.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5041011f0 0x7fb504102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 0x7fb5041033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5041072a0 0x7fb504103920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5041113d0 con 0x7fb504101bb0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb504111250 con 0x7fb5041011f0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5094f7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb504111550 con 0x7fb5041072a0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 0x7fb5041033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 0x7fb5041033e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56232/0 (socket says 172.21.15.116:56232) 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 -- 172.21.15.116:0/3893861103 learned_addr learned my addr 172.21.15.116:0/3893861103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb502ffd700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5041011f0 0x7fb504102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5037fe700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5041072a0 0x7fb504103920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 -- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5041072a0 msgr2=0x7fb504103920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5041072a0 0x7fb504103920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 -- 172.21.15.116:0/3893861103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5041011f0 msgr2=0x7fb504102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5041011f0 0x7fb504102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5041a6ee0 con 0x7fb504101bb0 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb5027fc700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 0x7fb5041033e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005910 tx=0x7fb4f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.898+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8027070 con 0x7fb504101bb0 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.899+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5041a7170 con 0x7fb504101bb0 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.899+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4f8022470 con 0x7fb504101bb0 2024-03-19T02:26:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.899+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800b610 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.899+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5041a7650 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.899+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb504105310 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.901+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb4f802a070 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.901+0000 7fb4ebfff700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb4ec0a2920 0x7fb4ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb502ffd700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb4ec0a2920 0x7fb4ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb4f80d00a0 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb4ebfff700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] conn(0x7fb4ec0ada80 0x7fb4ec0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4ec0b05f0 con 0x7fb4ec0ada80 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb4f80d04f0 con 0x7fb504101bb0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb502ffd700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb4ec0a2920 0x7fb4ec0a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009be0 tx=0x7fb4f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb5037fe700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] conn(0x7fb4ec0ada80 0x7fb4ec0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.902+0000 7fb5037fe700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] conn(0x7fb4ec0ada80 0x7fb4ec0aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.903+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== osd.5 v2:172.21.15.162:6808/1110895518 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4ec0b05f0 con 0x7fb4ec0ada80 2024-03-19T02:26:13.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.919+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 --> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb5041028e0 con 0x7fb4ec0ada80 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb4ebfff700 1 -- 172.21.15.116:0/3893861103 <== osd.5 v2:172.21.15.162:6808/1110895518 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb5041028e0 con 0x7fb4ec0ada80 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] conn(0x7fb4ec0ada80 msgr2=0x7fb4ec0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] conn(0x7fb4ec0ada80 0x7fb4ec0aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb4ec0a2920 msgr2=0x7fb4ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb4ec0a2920 0x7fb4ec0a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009be0 tx=0x7fb4f4009260 comp rx=0 tx=0).stop 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 msgr2=0x7fb5041033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5094f7700 1 --2- 172.21.15.116:0/3893861103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb504101bb0 0x7fb5041033e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005910 tx=0x7fb4f80047b0 comp rx=0 tx=0).stop 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.920+0000 7fb5037fe700 1 -- 172.21.15.116:0/3893861103 reap_dead start 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.921+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 shutdown_connections 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.921+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 >> 172.21.15.116:0/3893861103 conn(0x7fb504078220 msgr2=0x7fb504100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.921+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 shutdown_connections 2024-03-19T02:26:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:10.921+0000 7fb5094f7700 1 -- 172.21.15.116:0/3893861103 wait complete. 2024-03-19T02:26:13.901 INFO:teuthology.orchestra.run.smithi116.stdout:167503724600 2024-03-19T02:26:13.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-19T02:26:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:13 smithi154 ceph-mon[35693]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T02:26:14.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:13 smithi116 ceph-mon[32263]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T02:26:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:13 smithi162 ceph-mon[35761]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/1237170812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4106a30 msgr2=0x7f2bb4106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/1237170812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4106a30 0x7f2bb4106e10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00c210 tx=0x7f2bac00c520 comp rx=0 tx=0).stop 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/1237170812 shutdown_connections 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/1237170812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb4101610 0x7f2bb410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/1237170812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4106a30 0x7f2bb4106e10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/1237170812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 0x7f2bb41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.465+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/1237170812 >> 172.21.15.116:0/1237170812 conn(0x7f2bb4076570 msgr2=0x7f2bb4076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/1237170812 shutdown_connections 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/1237170812 wait complete. 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 Processor -- start 2024-03-19T02:26:14.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- start start 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 0x7f2bb4102550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 0x7f2bb4102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb4106a30 0x7f2bb4102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb41102d0 con 0x7f2bb4101610 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4110150 con 0x7f2bb4100c50 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bbc0fa700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4110450 con 0x7f2bb4106a30 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.466+0000 7f2bb9695700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 0x7f2bb4102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 0x7f2bb4102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56252/0 (socket says 172.21.15.116:56252) 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 -- 172.21.15.116:0/834378583 learned_addr learned my addr 172.21.15.116:0/834378583 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9e96700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 0x7f2bb4102550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bba697700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb4106a30 0x7f2bb4102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:14.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 -- 172.21.15.116:0/834378583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb4106a30 msgr2=0x7f2bb4102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb4106a30 0x7f2bb4102fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 -- 172.21.15.116:0/834378583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 msgr2=0x7f2bb4102550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 0x7f2bb4102550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb41a6f20 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9e96700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb4100c50 0x7f2bb4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bb9695700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 0x7f2bb4102a90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400eb30 tx=0x7f2ba400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400cbd0 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ba40044d0 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4003e30 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb41a7210 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.467+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41a7750 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.468+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bb4062a20 con 0x7f2bb4101610 2024-03-19T02:26:14.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.469+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2ba4004000 con 0x7f2bb4101610 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2baaffd700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2ba00a2920 0x7f2ba00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2bb9e96700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2ba00a2920 0x7f2ba00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2ba40cf6f0 con 0x7f2bb4101610 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2baaffd700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] conn(0x7f2ba00ada80 0x7f2ba00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ba4012ec0 con 0x7f2ba00ada80 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f2ba40cfb40 con 0x7f2bb4101610 2024-03-19T02:26:14.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2bba697700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] conn(0x7f2ba00ada80 0x7f2ba00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2bb9e96700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2ba00a2920 0x7f2ba00a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00a700 tx=0x7f2bac00c870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.470+0000 7f2bba697700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] conn(0x7f2ba00ada80 0x7f2ba00aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.471+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== osd.9 v2:172.21.15.116:6826/3348335330 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ba4012ec0 con 0x7f2ba00ada80 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 --> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bb4062ea0 con 0x7f2ba00ada80 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2baaffd700 1 -- 172.21.15.116:0/834378583 <== osd.9 v2:172.21.15.116:6826/3348335330 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bb4062ea0 con 0x7f2ba00ada80 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] conn(0x7f2ba00ada80 msgr2=0x7f2ba00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] conn(0x7f2ba00ada80 0x7f2ba00aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2ba00a2920 msgr2=0x7f2ba00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2ba00a2920 0x7f2ba00a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00a700 tx=0x7f2bac00c870 comp rx=0 tx=0).stop 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 msgr2=0x7f2bb4102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bbc0fa700 1 --2- 172.21.15.116:0/834378583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb4101610 0x7f2bb4102a90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400eb30 tx=0x7f2ba400ee40 comp rx=0 tx=0).stop 2024-03-19T02:26:14.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.487+0000 7f2bba697700 1 -- 172.21.15.116:0/834378583 reap_dead start 2024-03-19T02:26:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.488+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 shutdown_connections 2024-03-19T02:26:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.488+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 >> 172.21.15.116:0/834378583 conn(0x7f2bb4076570 msgr2=0x7f2bb40fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.488+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 shutdown_connections 2024-03-19T02:26:14.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:14.488+0000 7f2bbc0fa700 1 -- 172.21.15.116:0/834378583 wait complete. 2024-03-19T02:26:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:14 smithi154 ceph-mon[35693]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-03-19T02:26:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:14 smithi116 ceph-mon[32263]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-03-19T02:26:15.139 INFO:teuthology.orchestra.run.smithi116.stdout:292057776164 2024-03-19T02:26:15.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-19T02:26:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:14 smithi162 ceph-mon[35761]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 -- 172.21.15.116:0/4140697932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 msgr2=0x7f4708101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 --2- 172.21.15.116:0/4140697932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f4708101670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009990 tx=0x7f46f8009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 -- 172.21.15.116:0/4140697932 shutdown_connections 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 --2- 172.21.15.116:0/4140697932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708101bb0 0x7f470810e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 --2- 172.21.15.116:0/4140697932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4708107250 0x7f4708107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 --2- 172.21.15.116:0/4140697932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f4708101670 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 -- 172.21.15.116:0/4140697932 >> 172.21.15.116:0/4140697932 conn(0x7f4708078140 msgr2=0x7f4708078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 -- 172.21.15.116:0/4140697932 shutdown_connections 2024-03-19T02:26:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.281+0000 7f4710288700 1 -- 172.21.15.116:0/4140697932 wait complete. 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 Processor -- start 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 -- start start 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f470819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4708101bb0 0x7f470819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708107250 0x7f47081976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47081113b0 con 0x7f4708101210 2024-03-19T02:26:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4708111230 con 0x7f4708107250 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f4710288700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4708111530 con 0x7f4708101bb0 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f470e024700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f470819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f470e024700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f470819d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56270/0 (socket says 172.21.15.116:56270) 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f470e024700 1 -- 172.21.15.116:0/1475510142 learned_addr learned my addr 172.21.15.116:0/1475510142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f470d823700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4708101bb0 0x7f470819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.282+0000 7f470e825700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708107250 0x7f47081976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 -- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4708101bb0 msgr2=0x7f470819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4708101bb0 0x7f470819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 -- 172.21.15.116:0/1475510142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708107250 msgr2=0x7f47081976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708107250 0x7f47081976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4708197fd0 con 0x7f4708101210 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e825700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4708107250 0x7f47081976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:26:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f470e024700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f470819d620 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f470400eab0 tx=0x7f470400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470401c070 con 0x7f4708101210 2024-03-19T02:26:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4708198260 con 0x7f4708101210 2024-03-19T02:26:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4704017410 con 0x7f4708101210 2024-03-19T02:26:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704016610 con 0x7f4708101210 2024-03-19T02:26:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.283+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470806a190 con 0x7f4708101210 2024-03-19T02:26:15.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.284+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f470806a7a0 con 0x7f4708101210 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.285+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f470401f070 con 0x7f4708101210 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f46ff7fe700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46f40a2920 0x7f46f40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f470d823700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46f40a2920 0x7f46f40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f47040d0360 con 0x7f4708101210 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f46ff7fe700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] conn(0x7f46f40ada80 0x7f46f40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4704011e50 con 0x7f46f40ada80 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f47040d7050 con 0x7f4708101210 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f470d823700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46f40a2920 0x7f46f40a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4708198eb0 tx=0x7f46f800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.286+0000 7f470e825700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] conn(0x7f46f40ada80 0x7f46f40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.287+0000 7f470e825700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] conn(0x7f46f40ada80 0x7f46f40aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:15.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.288+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== osd.2 v2:172.21.15.162:6800/2685600713 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4704011e50 con 0x7f46f40ada80 2024-03-19T02:26:15.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.304+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 --> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f470806aba0 con 0x7f46f40ada80 2024-03-19T02:26:15.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f46ff7fe700 1 -- 172.21.15.116:0/1475510142 <== osd.2 v2:172.21.15.162:6800/2685600713 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f470806aba0 con 0x7f46f40ada80 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] conn(0x7f46f40ada80 msgr2=0x7f46f40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] conn(0x7f46f40ada80 0x7f46f40aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46f40a2920 msgr2=0x7f46f40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46f40a2920 0x7f46f40a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4708198eb0 tx=0x7f46f800b260 comp rx=0 tx=0).stop 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 msgr2=0x7f470819d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 --2- 172.21.15.116:0/1475510142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4708101210 0x7f470819d620 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f470400eab0 tx=0x7f470400edc0 comp rx=0 tx=0).stop 2024-03-19T02:26:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f470e825700 1 -- 172.21.15.116:0/1475510142 reap_dead start 2024-03-19T02:26:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 shutdown_connections 2024-03-19T02:26:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 >> 172.21.15.116:0/1475510142 conn(0x7f4708078140 msgr2=0x7f4708100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 shutdown_connections 2024-03-19T02:26:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:15.305+0000 7f4710288700 1 -- 172.21.15.116:0/1475510142 wait complete. 2024-03-19T02:26:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:15 smithi116 ceph-mon[32263]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T02:26:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:15 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:26:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:15 smithi162 ceph-mon[35761]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T02:26:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:15 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:26:16.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:15 smithi154 ceph-mon[35693]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T02:26:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:15 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.516+0000 7fc32f6a6700 1 -- 172.21.15.116:0/442364524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 msgr2=0x7fc328101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.516+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/442364524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328101880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc324009990 tx=0x7fc324009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 -- 172.21.15.116:0/442364524 shutdown_connections 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/442364524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc328106530 0x7fc32810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/442364524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328101880 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/442364524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc328101e50 0x7fc328105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 -- 172.21.15.116:0/442364524 >> 172.21.15.116:0/442364524 conn(0x7fc328077230 msgr2=0x7fc328077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 -- 172.21.15.116:0/442364524 shutdown_connections 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 -- 172.21.15.116:0/442364524 wait complete. 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 Processor -- start 2024-03-19T02:26:16.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 -- start start 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.517+0000 7fc32f6a6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc328101e50 0x7fc328197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc328106530 0x7fc3281980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3281113d0 con 0x7fc3281014a0 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc328111250 con 0x7fc328101e50 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc328111550 con 0x7fc328106530 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56288/0 (socket says 172.21.15.116:56288) 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 -- 172.21.15.116:0/2375542118 learned_addr learned my addr 172.21.15.116:0/2375542118 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:16.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32cc41700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc328101e50 0x7fc328197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32dc43700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc328106530 0x7fc3281980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 -- 172.21.15.116:0/2375542118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc328106530 msgr2=0x7fc3281980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc328106530 0x7fc3281980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 -- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc328101e50 msgr2=0x7fc328197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc328101e50 0x7fc328197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3281a6ed0 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32d442700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328197660 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc324000c00 tx=0x7fc324004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324027070 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc324022470 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3281a7160 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32400b610 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.518+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3281a75b0 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.519+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc328062a60 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.520+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc32402a070 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc31e7fc700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc3140a2920 0x7fc3140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc32cc41700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc3140a2920 0x7fc3140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fc3240d00d0 con 0x7fc3281014a0 2024-03-19T02:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc31e7fc700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] conn(0x7fc3140ada80 0x7fc3140aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc324024ec0 con 0x7fc3140ada80 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fc3240d0520 con 0x7fc3281014a0 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc32dc43700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] conn(0x7fc3140ada80 0x7fc3140aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.521+0000 7fc32cc41700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc3140a2920 0x7fc3140a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc3281997c0 tx=0x7fc318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.522+0000 7fc32dc43700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] conn(0x7fc3140ada80 0x7fc3140aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.523+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== osd.4 v2:172.21.15.154:6808/719513760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc324024ec0 con 0x7fc3140ada80 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 --> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc328062ee0 con 0x7fc3140ada80 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc31e7fc700 1 -- 172.21.15.116:0/2375542118 <== osd.4 v2:172.21.15.154:6808/719513760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc328062ee0 con 0x7fc3140ada80 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] conn(0x7fc3140ada80 msgr2=0x7fc3140aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] conn(0x7fc3140ada80 0x7fc3140aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc3140a2920 msgr2=0x7fc3140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc3140a2920 0x7fc3140a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc3281997c0 tx=0x7fc318009260 comp rx=0 tx=0).stop 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 msgr2=0x7fc328197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:16.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 --2- 172.21.15.116:0/2375542118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3281014a0 0x7fc328197660 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc324000c00 tx=0x7fc324004790 comp rx=0 tx=0).stop 2024-03-19T02:26:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32dc43700 1 -- 172.21.15.116:0/2375542118 reap_dead start 2024-03-19T02:26:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 shutdown_connections 2024-03-19T02:26:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.539+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 >> 172.21.15.116:0/2375542118 conn(0x7fc328077230 msgr2=0x7fc3281076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.540+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 shutdown_connections 2024-03-19T02:26:16.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:16.540+0000 7fc32f6a6700 1 -- 172.21.15.116:0/2375542118 wait complete. 2024-03-19T02:26:17.161 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:16 smithi116 ceph-mon[32263]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T02:26:17.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:16 smithi162 ceph-mon[35761]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T02:26:17.193 INFO:teuthology.orchestra.run.smithi116.stdout:77309411400 2024-03-19T02:26:17.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-19T02:26:17.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:16 smithi154 ceph-mon[35693]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T02:26:17.893 INFO:teuthology.orchestra.run.smithi116.stdout:146028888127 2024-03-19T02:26:17.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-19T02:26:18.094 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:18.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 -- 172.21.15.116:0/2259606129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006aa60 msgr2=0x7f2bb006aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 --2- 172.21.15.116:0/2259606129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006aa60 0x7f2bb006aee0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40099c0 tx=0x7f2ba4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 -- 172.21.15.116:0/2259606129 shutdown_connections 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 --2- 172.21.15.116:0/2259606129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006b420 0x7f2bb010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 --2- 172.21.15.116:0/2259606129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 --2- 172.21.15.116:0/2259606129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006aa60 0x7f2bb006aee0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 -- 172.21.15.116:0/2259606129 >> 172.21.15.116:0/2259606129 conn(0x7f2bb0078220 msgr2=0x7f2bb0078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 -- 172.21.15.116:0/2259606129 shutdown_connections 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.330+0000 7f2bb613d700 1 -- 172.21.15.116:0/2259606129 wait complete. 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 Processor -- start 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 -- start start 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006aa60 0x7f2bb01011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006b420 0x7f2bb0101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:18.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb01124e0 con 0x7f2bb006b420 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0112360 con 0x7f2bb006aa60 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2bb613d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0112660 con 0x7f2bb0108340 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.331+0000 7f2baffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37242/0 (socket says 172.21.15.116:37242) 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 -- 172.21.15.116:0/1199245727 learned_addr learned my addr 172.21.15.116:0/1199245727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 -- 172.21.15.116:0/1199245727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006aa60 msgr2=0x7f2bb01011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baf7fe700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006aa60 0x7f2bb01011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006aa60 0x7f2bb01011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 -- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006b420 msgr2=0x7f2bb0101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baeffd700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006b420 0x7f2bb0101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:18.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006b420 0x7f2bb0101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb0072e50 con 0x7f2bb0108340 2024-03-19T02:26:18.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baf7fe700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb006aa60 0x7f2bb01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:26:18.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baeffd700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb006b420 0x7f2bb0101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:26:18.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.332+0000 7f2baffff700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0101c70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e940 tx=0x7f2ba800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.333+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba800cbd0 con 0x7f2bb0108340 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.333+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ba800cd70 con 0x7f2bb0108340 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.333+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8005690 con 0x7f2bb0108340 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.333+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb0073140 con 0x7f2bb0108340 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.333+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb0073680 con 0x7f2bb0108340 2024-03-19T02:26:18.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.334+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bb0073c10 con 0x7f2bb0108340 2024-03-19T02:26:18.337 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:18.337 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:18.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.335+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2ba8004750 con 0x7f2bb0108340 2024-03-19T02:26:18.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.336+0000 7f2bacff9700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2b980a29a0 0x7f2b980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.336+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2ba80cf380 con 0x7f2bb0108340 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.336+0000 7f2bacff9700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] conn(0x7f2b980adb00 0x7f2b980affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.336+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b980b0670 con 0x7f2b980adb00 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.336+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f2ba80cf730 con 0x7f2bb0108340 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.338+0000 7f2baf7fe700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2b980a29a0 0x7f2b980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:18.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.338+0000 7f2baeffd700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] conn(0x7f2b980adb00 0x7f2b980affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:18.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.338+0000 7f2baf7fe700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2b980a29a0 0x7f2b980a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009fd0 tx=0x7f2ba000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:18.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.339+0000 7f2baeffd700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] conn(0x7f2b980adb00 0x7f2b980affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:18.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.340+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== osd.3 v2:172.21.15.116:6810/2748385188 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b980b0670 con 0x7f2b980adb00 2024-03-19T02:26:18.368 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:18.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.373+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 --> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bb0074090 con 0x7f2b980adb00 2024-03-19T02:26:18.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.374+0000 7f2bacff9700 1 -- 172.21.15.116:0/1199245727 <== osd.3 v2:172.21.15.116:6810/2748385188 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bb0074090 con 0x7f2b980adb00 2024-03-19T02:26:18.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.374+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] conn(0x7f2b980adb00 msgr2=0x7f2b980affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.374+0000 7f2bb613d700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] conn(0x7f2b980adb00 0x7f2b980affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:18.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2b980a29a0 msgr2=0x7f2b980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2b980a29a0 0x7f2b980a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009fd0 tx=0x7f2ba000b040 comp rx=0 tx=0).stop 2024-03-19T02:26:18.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 msgr2=0x7f2bb0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 --2- 172.21.15.116:0/1199245727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bb0108340 0x7f2bb0101c70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e940 tx=0x7f2ba800ed00 comp rx=0 tx=0).stop 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2baffff700 1 -- 172.21.15.116:0/1199245727 reap_dead start 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 shutdown_connections 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.375+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 >> 172.21.15.116:0/1199245727 conn(0x7f2bb0078220 msgr2=0x7f2bb00ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.376+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 shutdown_connections 2024-03-19T02:26:18.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:18.376+0000 7f2bb613d700 1 -- 172.21.15.116:0/1199245727 wait complete. 2024-03-19T02:26:19.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:18 smithi116 ceph-mon[32263]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-19T02:26:19.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:18 smithi154 ceph-mon[35693]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-19T02:26:19.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:18 smithi162 ceph-mon[35761]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-19T02:26:20.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:20 smithi116 ceph-mon[32263]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:21.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:20 smithi162 ceph-mon[35761]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:20 smithi154 ceph-mon[35693]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:22 smithi116 ceph-mon[32263]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:22 smithi162 ceph-mon[35761]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:22 smithi154 ceph-mon[35693]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 -- 172.21.15.116:0/2113457088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b41014a0 msgr2=0x7f83b4101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:23.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 --2- 172.21.15.116:0/2113457088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b41014a0 0x7f83b4101880 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009990 tx=0x7f83a4009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 -- 172.21.15.116:0/2113457088 shutdown_connections 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 --2- 172.21.15.116:0/2113457088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b4106530 0x7f83b410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 --2- 172.21.15.116:0/2113457088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b41014a0 0x7f83b4101880 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 --2- 172.21.15.116:0/2113457088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b4105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 -- 172.21.15.116:0/2113457088 >> 172.21.15.116:0/2113457088 conn(0x7f83b4077230 msgr2=0x7f83b4077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 -- 172.21.15.116:0/2113457088 shutdown_connections 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.346+0000 7f83b9540700 1 -- 172.21.15.116:0/2113457088 wait complete. 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 Processor -- start 2024-03-19T02:26:23.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 -- start start 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b4106530 0x7f83b4198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83b41113a0 con 0x7f83b4106530 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4111220 con 0x7f83b4101e50 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b9540700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83b4111520 con 0x7f83b41014a0 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.347+0000 7f83b27fc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37262/0 (socket says 172.21.15.116:37262) 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 -- 172.21.15.116:0/1410468828 learned_addr learned my addr 172.21.15.116:0/1410468828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b27fc700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55812/0 (socket says 172.21.15.116:55812) 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b37fe700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b4106530 0x7f83b4198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 -- 172.21.15.116:0/1410468828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 msgr2=0x7f83b419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 -- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b4106530 msgr2=0x7f83b4198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b4106530 0x7f83b4198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 -- 172.21.15.116:0/1410468828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b4199030 con 0x7f83b41014a0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b2ffd700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000c00 tx=0x7f83a4004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4027070 con 0x7f83b41014a0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b41992c0 con 0x7f83b41014a0 2024-03-19T02:26:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83a4022470 con 0x7f83b41014a0 2024-03-19T02:26:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a400b610 con 0x7f83b41014a0 2024-03-19T02:26:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.348+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b41a54e0 con 0x7f83b41014a0 2024-03-19T02:26:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.349+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b4102930 con 0x7f83b41014a0 2024-03-19T02:26:23.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.350+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f83a402a070 con 0x7f83b41014a0 2024-03-19T02:26:23.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.351+0000 7f839bfff700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 0x7f839c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:23.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.351+0000 7f83b27fc700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 0x7f839c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:23.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.351+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f83a40cfe50 con 0x7f83b41014a0 2024-03-19T02:26:23.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.352+0000 7f83b27fc700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 0x7f839c0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f83b4100d50 tx=0x7f83a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:23.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.357+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83a4013070 con 0x7f83b41014a0 2024-03-19T02:26:23.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.557+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f83b41a56b0 con 0x7f83b41014a0 2024-03-19T02:26:23.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.557+0000 7f839bfff700 1 -- 172.21.15.116:0/1410468828 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f83a409ca60 con 0x7f83b41014a0 2024-03-19T02:26:23.557 INFO:teuthology.orchestra.run.smithi116.stdout:223338299440 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 msgr2=0x7f839c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 0x7f839c0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f83b4100d50 tx=0x7f83a8009260 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 msgr2=0x7f83b419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000c00 tx=0x7f83a4004790 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 shutdown_connections 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b4106530 0x7f83b4198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f839c0a2920 0x7f839c0a4de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b41014a0 0x7f83b419e680 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 --2- 172.21.15.116:0/1410468828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83b4101e50 0x7f83b419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:23.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 >> 172.21.15.116:0/1410468828 conn(0x7f83b4077230 msgr2=0x7f83b41076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:23.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 shutdown_connections 2024-03-19T02:26:23.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:23.559+0000 7f83b9540700 1 -- 172.21.15.116:0/1410468828 wait complete. 2024-03-19T02:26:23.696 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1410468828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:26:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1410468828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:26:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1410468828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:26:24.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:24 smithi116 ceph-mon[32263]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:24.971 INFO:teuthology.orchestra.run.smithi116.stdout:111669149763 2024-03-19T02:26:24.972 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-19T02:26:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:24 smithi162 ceph-mon[35761]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:24 smithi154 ceph-mon[35693]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:26.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:26 smithi116 ceph-mon[32263]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:26 smithi162 ceph-mon[35761]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:26 smithi154 ceph-mon[35693]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 -- 172.21.15.116:0/4004959306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 msgr2=0x7f22dc10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 --2- 172.21.15.116:0/4004959306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc10e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009990 tx=0x7f22cc009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 -- 172.21.15.116:0/4004959306 shutdown_connections 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 --2- 172.21.15.116:0/4004959306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc10e090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 --2- 172.21.15.116:0/4004959306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 0x7f22dc107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 --2- 172.21.15.116:0/4004959306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 -- 172.21.15.116:0/4004959306 >> 172.21.15.116:0/4004959306 conn(0x7f22dc078140 msgr2=0x7f22dc078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 -- 172.21.15.116:0/4004959306 shutdown_connections 2024-03-19T02:26:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.041+0000 7f22e0f47700 1 -- 172.21.15.116:0/4004959306 wait complete. 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 Processor -- start 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 -- start start 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc1967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 0x7f22dc19b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc1102d0 con 0x7f22dc101bb0 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc110150 con 0x7f22dc107250 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22e0f47700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc110450 con 0x7f22dc101210 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22da59c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc1967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50820/0 (socket says 172.21.15.116:50820) 2024-03-19T02:26:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 -- 172.21.15.116:0/1548041093 learned_addr learned my addr 172.21.15.116:0/1548041093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22dad9d700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 0x7f22dc19b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 -- 172.21.15.116:0/1548041093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 msgr2=0x7f22dc1967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc1967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 -- 172.21.15.116:0/1548041093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 msgr2=0x7f22dc19b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 0x7f22dc19b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22d9d9b700 1 -- 172.21.15.116:0/1548041093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc19b760 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.042+0000 7f22da59c700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22d9d9b700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f22d00123f0 tx=0x7f22d000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0004bb0 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d0004d50 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc19b9f0 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d00043e0 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.043+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc1a32d0 con 0x7f22dc101bb0 2024-03-19T02:26:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.044+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc068fd0 con 0x7f22dc101bb0 2024-03-19T02:26:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.045+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f22d001f030 con 0x7f22dc101bb0 2024-03-19T02:26:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.046+0000 7f22cb7fe700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 0x7f22c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.046+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f22d00d3400 con 0x7f22dc101bb0 2024-03-19T02:26:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.046+0000 7f22da59c700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 0x7f22c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.046+0000 7f22da59c700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 0x7f22c40a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22d400bf70 tx=0x7f22d400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.049+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22d0026020 con 0x7f22dc101bb0 2024-03-19T02:26:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.252+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f22dc02dbb0 con 0x7f22dc101bb0 2024-03-19T02:26:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.252+0000 7f22cb7fe700 1 -- 172.21.15.116:0/1548041093 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f22d00a0070 con 0x7f22dc101bb0 2024-03-19T02:26:28.252 INFO:teuthology.orchestra.run.smithi116.stdout:34359738452 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 msgr2=0x7f22c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 0x7f22c40a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f22d400bf70 tx=0x7f22d400d040 comp rx=0 tx=0).stop 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 msgr2=0x7f22dc196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f22d00123f0 tx=0x7f22d000ef90 comp rx=0 tx=0).stop 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 shutdown_connections 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22dc107250 0x7f22dc19b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22c40a2920 0x7f22c40a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22dc101210 0x7f22dc1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 --2- 172.21.15.116:0/1548041093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22dc101bb0 0x7f22dc196ce0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.254+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 >> 172.21.15.116:0/1548041093 conn(0x7f22dc078140 msgr2=0x7f22dc100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.255+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 shutdown_connections 2024-03-19T02:26:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:28.255+0000 7f22e0f47700 1 -- 172.21.15.116:0/1548041093 wait complete. 2024-03-19T02:26:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:28 smithi162 ceph-mon[35761]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:28 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3003732366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:26:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:28 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1548041093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:26:29.215 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:28 smithi116 ceph-mon[32263]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:29.215 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:28 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3003732366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:26:29.215 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:28 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1548041093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:26:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:28 smithi154 ceph-mon[35693]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:28 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3003732366' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:26:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:28 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1548041093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:26:29.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 -- 172.21.15.116:0/2670549776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff254101e50 msgr2=0x7ff254105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 --2- 172.21.15.116:0/2670549776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff254101e50 0x7ff254105e10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff2440099c0 tx=0x7ff244009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 -- 172.21.15.116:0/2670549776 shutdown_connections 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 --2- 172.21.15.116:0/2670549776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2541064f0 0x7ff25410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 --2- 172.21.15.116:0/2670549776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff254101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 --2- 172.21.15.116:0/2670549776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff254101e50 0x7ff254105e10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 -- 172.21.15.116:0/2670549776 >> 172.21.15.116:0/2670549776 conn(0x7ff254077260 msgr2=0x7ff254077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 -- 172.21.15.116:0/2670549776 shutdown_connections 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.689+0000 7ff25be27700 1 -- 172.21.15.116:0/2670549776 wait complete. 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 Processor -- start 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 -- start start 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff25419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff254101e50 0x7ff25419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff254111380 con 0x7ff2541064f0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff254111200 con 0x7ff2541014a0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25be27700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff254111500 con 0x7ff254101e50 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.690+0000 7ff25a3c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff259bc3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff25419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff2593c2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff254101e50 0x7ff25419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50800/0 (socket says 172.21.15.116:50800) 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 -- 172.21.15.116:0/3003732366 learned_addr learned my addr 172.21.15.116:0/3003732366 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 -- 172.21.15.116:0/3003732366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff254101e50 msgr2=0x7ff25419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff254101e50 0x7ff25419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 -- 172.21.15.116:0/3003732366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 msgr2=0x7ff25419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff25419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 -- 172.21.15.116:0/3003732366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff254197f60 con 0x7ff2541064f0 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff259bc3700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff25419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25a3c4700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ef30 tx=0x7ff24c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stdout:313532612640 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0092e0 con 0x7ff2541064f0 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2541981f0 con 0x7ff2541064f0 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff24c004bb0 con 0x7ff2541064f0 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c004020 con 0x7ff2541064f0 2024-03-19T02:26:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.691+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25406a850 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.692+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2541028f0 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.693+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7ff24c01b030 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.694+0000 7ff24affd700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 0x7ff2400a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.694+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff24c0cf2f0 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.694+0000 7ff259bc3700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 0x7ff2400a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.694+0000 7ff259bc3700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 0x7ff2400a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff25000e530 tx=0x7ff250010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.697+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff24c022020 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.917+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff25406b280 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.917+0000 7ff24affd700 1 -- 172.21.15.116:0/3003732366 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff24c010040 con 0x7ff2541064f0 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 msgr2=0x7ff2400a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 0x7ff2400a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff25000e530 tx=0x7ff250010040 comp rx=0 tx=0).stop 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 msgr2=0x7ff254197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00ef30 tx=0x7ff24c00c3a0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 shutdown_connections 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2541064f0 0x7ff254197780 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2541014a0 0x7ff25419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff2400a2920 0x7ff2400a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 --2- 172.21.15.116:0/3003732366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff254101e50 0x7ff25419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 >> 172.21.15.116:0/3003732366 conn(0x7ff254077260 msgr2=0x7ff2540ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 shutdown_connections 2024-03-19T02:26:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:27.919+0000 7ff25be27700 1 -- 172.21.15.116:0/3003732366 wait complete. 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 -- 172.21.15.116:0/1534766694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 msgr2=0x7f900010e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 --2- 172.21.15.116:0/1534766694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f900010e7d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009990 tx=0x7f8ff0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 -- 172.21.15.116:0/1534766694 shutdown_connections 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 --2- 172.21.15.116:0/1534766694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 0x7f9000111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 --2- 172.21.15.116:0/1534766694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f9000074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 --2- 172.21.15.116:0/1534766694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f900010e7d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 -- 172.21.15.116:0/1534766694 >> 172.21.15.116:0/1534766694 conn(0x7f9000079470 msgr2=0x7f9000079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 -- 172.21.15.116:0/1534766694 shutdown_connections 2024-03-19T02:26:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.982+0000 7f90069de700 1 -- 172.21.15.116:0/1534766694 wait complete. 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 Processor -- start 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 -- start start 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f90001a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 0x7f900019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9000114790 con 0x7f9000074e60 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9000114610 con 0x7f9000074540 2024-03-19T02:26:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f90069de700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9000114910 con 0x7f900010ed10 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8fff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f90001a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8fff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50840/0 (socket says 172.21.15.116:50840) 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8fff7fe700 1 -- 172.21.15.116:0/2152675031 learned_addr learned my addr 172.21.15.116:0/2152675031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f9004f7b700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 0x7f900019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 -- 172.21.15.116:0/2152675031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 msgr2=0x7f900019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 0x7f900019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 -- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 msgr2=0x7f90001a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.983+0000 7f8ffffff700 1 -- 172.21.15.116:0/2152675031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900019b190 con 0x7f9000074540 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f8fff7fe700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f8ffffff700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f90001a0870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e9f0 tx=0x7f8ff400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff400c9c0 con 0x7f9000074540 2024-03-19T02:26:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f900019b480 con 0x7f9000074540 2024-03-19T02:26:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ff400cb60 con 0x7f9000074540 2024-03-19T02:26:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4018680 con 0x7f9000074540 2024-03-19T02:26:29.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.984+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90001a7740 con 0x7f9000074540 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.985+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9000068fd0 con 0x7f9000074540 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.986+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f8ff4010430 con 0x7f9000074540 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.986+0000 7f8ffd7fa700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 0x7f8fe80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.987+0000 7f8fff7fe700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 0x7f8fe80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.987+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f8ff40cf0b0 con 0x7f9000074540 2024-03-19T02:26:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.987+0000 7f8fff7fe700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 0x7f8fe80a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9000101e80 tx=0x7f8ff001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:29.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:29.996+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ff409bc30 con 0x7f9000074540 2024-03-19T02:26:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:29 smithi116 ceph-mon[32263]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:26:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:29 smithi162 ceph-mon[35761]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.201+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f900002dbb0 con 0x7f9000074540 2024-03-19T02:26:30.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.201+0000 7f8ffd7fa700 1 -- 172.21.15.116:0/2152675031 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8ff409b530 con 0x7f9000074540 2024-03-19T02:26:30.202 INFO:teuthology.orchestra.run.smithi116.stdout:55834574928 2024-03-19T02:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 msgr2=0x7f8fe80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 0x7f8fe80a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9000101e80 tx=0x7f8ff001a040 comp rx=0 tx=0).stop 2024-03-19T02:26:30.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 msgr2=0x7f90001a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f90001a0870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400e9f0 tx=0x7f8ff400ed00 comp rx=0 tx=0).stop 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 shutdown_connections 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fe80a2920 0x7f8fe80a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f900010ed10 0x7f900019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9000074540 0x7f90001a0870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 --2- 172.21.15.116:0/2152675031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9000074e60 0x7f90001a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 >> 172.21.15.116:0/2152675031 conn(0x7f9000079470 msgr2=0x7f90000ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 shutdown_connections 2024-03-19T02:26:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.204+0000 7f90069de700 1 -- 172.21.15.116:0/2152675031 wait complete. 2024-03-19T02:26:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 -- 172.21.15.116:0/2308478673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238100ac0 msgr2=0x7f6238100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2308478673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238100ac0 0x7f6238100ea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6234009920 tx=0x7f6234009c30 comp rx=0 tx=0).stop 2024-03-19T02:26:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 -- 172.21.15.116:0/2308478673 shutdown_connections 2024-03-19T02:26:30.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2308478673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62381076e0 0x7f623810f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2308478673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238100ac0 0x7f6238100ea0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2308478673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6238101470 0x7f6238106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 -- 172.21.15.116:0/2308478673 >> 172.21.15.116:0/2308478673 conn(0x7f6238077810 msgr2=0x7f62380feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 -- 172.21.15.116:0/2308478673 shutdown_connections 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.236+0000 7f623f9d4700 1 -- 172.21.15.116:0/2308478673 wait complete. 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 Processor -- start 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 -- start start 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238101470 0x7f62381a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 0x7f623819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6238103870 con 0x7f6238100ac0 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62381036f0 con 0x7f6238101470 2024-03-19T02:26:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623f9d4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f623819b130 con 0x7f62381076e0 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50858/0 (socket says 172.21.15.116:50858) 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 -- 172.21.15.116:0/2520256499 learned_addr learned my addr 172.21.15.116:0/2520256499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623df71700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 0x7f623819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623cf6f700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238101470 0x7f62381a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 -- 172.21.15.116:0/2520256499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 msgr2=0x7f623819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 0x7f623819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 -- 172.21.15.116:0/2520256499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238101470 msgr2=0x7f62381a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238101470 0x7f62381a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623d770700 1 -- 172.21.15.116:0/2520256499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623819b330 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.237+0000 7f623df71700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 0x7f623819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f623d770700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6234009230 tx=0x7f623400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6234027070 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f623819b5c0 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62381a7930 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62340055f0 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.238+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6234004930 con 0x7f6238100ac0 2024-03-19T02:26:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.239+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f621c005330 con 0x7f6238100ac0 2024-03-19T02:26:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.240+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f623402a070 con 0x7f6238100ac0 2024-03-19T02:26:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.241+0000 7f622e7fc700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 0x7f62240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.241+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f62340cfb70 con 0x7f6238100ac0 2024-03-19T02:26:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.241+0000 7f623cf6f700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 0x7f62240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.241+0000 7f623cf6f700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 0x7f62240a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f62380fff10 tx=0x7f6228009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:30.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.248+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6234031020 con 0x7f6238100ac0 2024-03-19T02:26:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:29 smithi154 ceph-mon[35693]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:26:30.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.455+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f621c0058f0 con 0x7f6238100ac0 2024-03-19T02:26:30.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.455+0000 7f622e7fc700 1 -- 172.21.15.116:0/2520256499 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6234013030 con 0x7f6238100ac0 2024-03-19T02:26:30.456 INFO:teuthology.orchestra.run.smithi116.stdout:201863462967 2024-03-19T02:26:30.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 msgr2=0x7f62240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 0x7f62240a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f62380fff10 tx=0x7f6228009260 comp rx=0 tx=0).stop 2024-03-19T02:26:30.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 msgr2=0x7f62381a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:30.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6234009230 tx=0x7f623400fa00 comp rx=0 tx=0).stop 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 shutdown_connections 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f62240a2920 0x7f62240a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62381076e0 0x7f623819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6238100ac0 0x7f62381a0890 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 --2- 172.21.15.116:0/2520256499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6238101470 0x7f62381a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.457+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 >> 172.21.15.116:0/2520256499 conn(0x7f6238077810 msgr2=0x7f62380fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.458+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 shutdown_connections 2024-03-19T02:26:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:30.458+0000 7f623f9d4700 1 -- 172.21.15.116:0/2520256499 wait complete. 2024-03-19T02:26:31.078 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:30 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2152675031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:26:31.078 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:30 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2520256499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:26:31.103 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612636 got 313532612640 for osd.10 2024-03-19T02:26:31.104 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:30 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2152675031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:26:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:30 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2520256499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:26:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:30 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2152675031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:26:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:30 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2520256499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:26:32.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738452 for osd.0 2024-03-19T02:26:32.038 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:32.111 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:31 smithi116 ceph-mon[32263]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:31 smithi162 ceph-mon[35761]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:31 smithi154 ceph-mon[35693]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:26:32.451 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:32.899 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299437 got 223338299440 for osd.7 2024-03-19T02:26:32.899 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:33.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574928 for osd.1 2024-03-19T02:26:33.668 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:34.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:34 smithi116 ceph-mon[32263]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:35.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:34 smithi154 ceph-mon[35693]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:34 smithi162 ceph-mon[35761]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 -- 172.21.15.116:0/1174016514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648102fa0 msgr2=0x7f4648106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 --2- 172.21.15.116:0/1174016514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648102fa0 0x7f4648106f40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4638009990 tx=0x7f4638009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 -- 172.21.15.116:0/1174016514 shutdown_connections 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 --2- 172.21.15.116:0/1174016514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4648107620 0x7f464810f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 --2- 172.21.15.116:0/1174016514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46481025f0 0x7f46481029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 --2- 172.21.15.116:0/1174016514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648102fa0 0x7f4648106f40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 -- 172.21.15.116:0/1174016514 >> 172.21.15.116:0/1174016514 conn(0x7f46480793a0 msgr2=0x7f46480797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 -- 172.21.15.116:0/1174016514 shutdown_connections 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.831+0000 7f464ff49700 1 -- 172.21.15.116:0/1174016514 wait complete. 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 Processor -- start 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 -- start start 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 0x7f464819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 0x7f464819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648107620 0x7f46481987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112490 con 0x7f4648107620 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112310 con 0x7f46481025f0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464ff49700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112610 con 0x7f4648102fa0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464d4e4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 0x7f464819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464d4e4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 0x7f464819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54850/0 (socket says 172.21.15.116:54850) 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464d4e4700 1 -- 172.21.15.116:0/1599414105 learned_addr learned my addr 172.21.15.116:0/1599414105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.832+0000 7f464dce5700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 0x7f464819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 -- 172.21.15.116:0/1599414105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 msgr2=0x7f464819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 0x7f464819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 -- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648107620 msgr2=0x7f46481987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648107620 0x7f46481987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 -- 172.21.15.116:0/1599414105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4648198fb0 con 0x7f46481025f0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464dce5700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 0x7f464819e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f464400e9f0 tx=0x7f464400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644004220 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f464400ce00 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46481992a0 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644005020 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.833+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46481a54b0 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.834+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4648103a20 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.835+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f464400c950 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.836+0000 7f463effd700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 0x7f46340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.836+0000 7f464d4e4700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 0x7f46340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.836+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f46440ceff0 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.836+0000 7f464d4e4700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 0x7f46340a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4648101fa0 tx=0x7f463800b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:34.839+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f464409bc60 con 0x7f46481025f0 2024-03-19T02:26:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.046+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f464802dbb0 con 0x7f46481025f0 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stdout:146028888130 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.047+0000 7f463effd700 1 -- 172.21.15.116:0/1599414105 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f464409b560 con 0x7f46481025f0 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 msgr2=0x7f46340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 0x7f46340a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4648101fa0 tx=0x7f463800b610 comp rx=0 tx=0).stop 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 msgr2=0x7f464819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 0x7f464819e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f464400e9f0 tx=0x7f464400ed00 comp rx=0 tx=0).stop 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 shutdown_connections 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4648107620 0x7f46481987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46481025f0 0x7f464819e650 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f46340a2920 0x7f46340a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 --2- 172.21.15.116:0/1599414105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4648102fa0 0x7f464819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 >> 172.21.15.116:0/1599414105 conn(0x7f46480793a0 msgr2=0x7f4648100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 shutdown_connections 2024-03-19T02:26:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:35.049+0000 7f464ff49700 1 -- 172.21.15.116:0/1599414105 wait complete. 2024-03-19T02:26:35.991 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1599414105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:26:36.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1599414105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:26:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1599414105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:26:36.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888127 got 146028888130 for osd.4 2024-03-19T02:26:36.725 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:36.925 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:36 smithi116 ceph-mon[32263]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:37.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:36 smithi154 ceph-mon[35693]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:36 smithi162 ceph-mon[35761]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- 172.21.15.116:0/665940379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c100eb0 msgr2=0x7f687c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 --2- 172.21.15.116:0/665940379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c100eb0 0x7f687c101290 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f68780099c0 tx=0x7f6878009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- 172.21.15.116:0/665940379 shutdown_connections 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 --2- 172.21.15.116:0/665940379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 --2- 172.21.15.116:0/665940379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c100eb0 0x7f687c101290 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 --2- 172.21.15.116:0/665940379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c101860 0x7f687c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- 172.21.15.116:0/665940379 >> 172.21.15.116:0/665940379 conn(0x7f687c076550 msgr2=0x7f687c076960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- 172.21.15.116:0/665940379 shutdown_connections 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- 172.21.15.116:0/665940379 wait complete. 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 Processor -- start 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.925+0000 7f688362e700 1 -- start start 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 0x7f687c101e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c1028b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:37.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f687c112510 con 0x7f687c101860 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f687c112390 con 0x7f687c1076d0 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f688362e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f687c112690 con 0x7f687c100eb0 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50906/0 (socket says 172.21.15.116:50906) 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 -- 172.21.15.116:0/1727520732 learned_addr learned my addr 172.21.15.116:0/1727520732 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f68813ca700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 0x7f687c101e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6881bcb700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c1028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 -- 172.21.15.116:0/1727520732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 msgr2=0x7f687c101e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 0x7f687c101e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 -- 172.21.15.116:0/1727520732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 msgr2=0x7f687c1028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c1028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 -- 172.21.15.116:0/1727520732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c1a6f90 con 0x7f687c101860 2024-03-19T02:26:37.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6881bcb700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f68813ca700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 0x7f687c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.926+0000 7f6880bc9700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f686c00e940 tx=0x7f686c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c00c9e0 con 0x7f687c101860 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687c1a7280 con 0x7f687c101860 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686c00cb80 con 0x7f687c101860 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c018680 con 0x7f687c101860 2024-03-19T02:26:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f687c1a77c0 con 0x7f687c101860 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.927+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f687c068fd0 con 0x7f687c101860 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.929+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f686c010430 con 0x7f687c101860 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.930+0000 7f68727fc700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 0x7f68680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.930+0000 7f68813ca700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 0x7f68680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.930+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f686c0cf390 con 0x7f687c101860 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.930+0000 7f68813ca700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 0x7f68680a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6878003760 tx=0x7f6878005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:37.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:36.933+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f686c09bf70 con 0x7f687c101860 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.134+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f687c10a910 con 0x7f687c101860 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.134+0000 7f68727fc700 1 -- 172.21.15.116:0/1727520732 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f686c09b870 con 0x7f687c101860 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 msgr2=0x7f68680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 0x7f68680a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6878003760 tx=0x7f6878005fb0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 msgr2=0x7f687c102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f686c00e940 tx=0x7f686c00ec50 comp rx=0 tx=0).stop 2024-03-19T02:26:37.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 shutdown_connections 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f687c1076d0 0x7f687c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f68680a2920 0x7f68680a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f687c100eb0 0x7f687c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 --2- 172.21.15.116:0/1727520732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f687c101860 0x7f687c102370 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 >> 172.21.15.116:0/1727520732 conn(0x7f687c076550 msgr2=0x7f687c100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 shutdown_connections 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:37.136+0000 7f688362e700 1 -- 172.21.15.116:0/1727520732 wait complete. 2024-03-19T02:26:37.661 INFO:teuthology.orchestra.run.smithi116.stdout:347892351006 2024-03-19T02:26:38.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1727520732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:26:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1727520732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:26:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1727520732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:26:38.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:38 smithi116 ceph-mon[32263]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:39.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:38 smithi154 ceph-mon[35693]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:38 smithi162 ceph-mon[35761]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:39.875 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:39 smithi116 ceph-mon[32263]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 -- 172.21.15.116:0/2509336868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b810ca00 msgr2=0x7f48b810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 --2- 172.21.15.116:0/2509336868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b810ca00 0x7f48b810ee30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f48b000b0a0 tx=0x7f48b000b3b0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 -- 172.21.15.116:0/2509336868 shutdown_connections 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 --2- 172.21.15.116:0/2509336868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b810ca00 0x7f48b810ee30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 --2- 172.21.15.116:0/2509336868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b8103620 0x7f48b8103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 --2- 172.21.15.116:0/2509336868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103f40 0x7f48b810c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.875+0000 7f48bda27700 1 -- 172.21.15.116:0/2509336868 >> 172.21.15.116:0/2509336868 conn(0x7f48b8078340 msgr2=0x7f48b8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:39.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.876+0000 7f48bda27700 1 -- 172.21.15.116:0/2509336868 shutdown_connections 2024-03-19T02:26:39.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.876+0000 7f48bda27700 1 -- 172.21.15.116:0/2509336868 wait complete. 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.876+0000 7f48bda27700 1 Processor -- start 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 -- start start 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103620 0x7f48b8101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 0x7f48b8101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:39.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f48b81125e0 con 0x7f48b8103f40 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8112460 con 0x7f48b810ca00 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48bda27700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8112760 con 0x7f48b8103620 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 0x7f48b8101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33390/0 (socket says 172.21.15.116:33390) 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 -- 172.21.15.116:0/659139464 learned_addr learned my addr 172.21.15.116:0/659139464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b6ffd700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103620 0x7f48b8101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 -- 172.21.15.116:0/659139464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103620 msgr2=0x7f48b8101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103620 0x7f48b8101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 -- 172.21.15.116:0/659139464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 msgr2=0x7f48b8101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 0x7f48b8101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 -- 172.21.15.116:0/659139464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b81a7000 con 0x7f48b8103f40 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b77fe700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 0x7f48b8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:39.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.877+0000 7f48b67fc700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00e9f0 tx=0x7f48ac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.878+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac004a30 con 0x7f48b8103f40 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.878+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b81a72f0 con 0x7f48b8103f40 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.878+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48ac00cc70 con 0x7f48b8103f40 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.878+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0055f0 con 0x7f48b8103f40 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.878+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81a7830 con 0x7f48b8103f40 2024-03-19T02:26:39.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.879+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b8068fd0 con 0x7f48b8103f40 2024-03-19T02:26:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.880+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f48ac00c7c0 con 0x7f48b8103f40 2024-03-19T02:26:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.880+0000 7f48bca25700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 0x7f48a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.880+0000 7f48b6ffd700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 0x7f48a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.881+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f48ac0cf160 con 0x7f48b8103f40 2024-03-19T02:26:39.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.881+0000 7f48b6ffd700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 0x7f48a00a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009680 tx=0x7f48a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:39.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:39.884+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48ac053740 con 0x7f48b8103f40 2024-03-19T02:26:40.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:39 smithi154 ceph-mon[35693]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T02:26:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.103+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f48b81027b0 con 0x7f48b8103f40 2024-03-19T02:26:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.103+0000 7f48bca25700 1 -- 172.21.15.116:0/659139464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f48ac09bea0 con 0x7f48b8103f40 2024-03-19T02:26:40.104 INFO:teuthology.orchestra.run.smithi116.stdout:77309411405 2024-03-19T02:26:40.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 msgr2=0x7f48a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:40.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 0x7f48a00a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009680 tx=0x7f48a8009260 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 msgr2=0x7f48b81017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00e9f0 tx=0x7f48ac00ed00 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 shutdown_connections 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48b810ca00 0x7f48b8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f48a00a2890 0x7f48a00a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8103620 0x7f48b8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 --2- 172.21.15.116:0/659139464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48b8103f40 0x7f48b81017c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 >> 172.21.15.116:0/659139464 conn(0x7f48b8078340 msgr2=0x7f48b80fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 shutdown_connections 2024-03-19T02:26:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:40.105+0000 7f48bda27700 1 -- 172.21.15.116:0/659139464 wait complete. 2024-03-19T02:26:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:39 smithi162 ceph-mon[35761]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T02:26:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:40 smithi116 ceph-mon[32263]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:40.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:40 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/659139464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:26:40.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411405 for osd.2 2024-03-19T02:26:40.789 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:41.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:40 smithi154 ceph-mon[35693]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:41.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:40 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/659139464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:26:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:40 smithi162 ceph-mon[35761]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:40 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/659139464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 -- 172.21.15.116:0/1754419709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 msgr2=0x7fa7e0107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 --2- 172.21.15.116:0/1754419709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e0107760 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009990 tx=0x7fa7d0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 -- 172.21.15.116:0/1754419709 shutdown_connections 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 --2- 172.21.15.116:0/1754419709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e01012c0 0x7fa7e010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 --2- 172.21.15.116:0/1754419709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e0107760 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 --2- 172.21.15.116:0/1754419709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e0100900 0x7fa7e0100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 -- 172.21.15.116:0/1754419709 >> 172.21.15.116:0/1754419709 conn(0x7fa7e0077810 msgr2=0x7fa7e00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:41.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 -- 172.21.15.116:0/1754419709 shutdown_connections 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.141+0000 7fa7e6453700 1 -- 172.21.15.116:0/1754419709 wait complete. 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 Processor -- start 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 -- start start 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e0100900 0x7fa7e019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e01125f0 con 0x7fa7e0107380 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0112470 con 0x7fa7e01012c0 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e6453700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0112770 con 0x7fa7e0100900 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e49f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60744/0 (socket says 172.21.15.116:60744) 2024-03-19T02:26:41.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 -- 172.21.15.116:0/4237094313 learned_addr learned my addr 172.21.15.116:0/4237094313 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7dffff700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e0100900 0x7fa7e019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.391 INFO:teuthology.orchestra.run.smithi116.stdout:257698037806 2024-03-19T02:26:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 -- 172.21.15.116:0/4237094313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e0100900 msgr2=0x7fa7e019e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e0100900 0x7fa7e019e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 -- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 msgr2=0x7fa7e01988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e01988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7df7fe700 1 -- 172.21.15.116:0/4237094313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e0199010 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.142+0000 7fa7e49f0700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e01988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7df7fe700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9a0 tx=0x7fa7d400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400ca10 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7d400cbb0 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e0199300 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4018680 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.143+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e01a55c0 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.144+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e010a950 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.145+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa7d4047a60 con 0x7fa7e01012c0 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.145+0000 7fa7dd7fa700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 0x7fa7c80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.145+0000 7fa7dffff700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 0x7fa7c80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.146+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa7d40cef90 con 0x7fa7e01012c0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.146+0000 7fa7dffff700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 0x7fa7c80a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000b2e0 tx=0x7fa7d001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.151+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7d409bc70 con 0x7fa7e01012c0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.364+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa7e0199de0 con 0x7fa7e01012c0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.364+0000 7fa7dd7fa700 1 -- 172.21.15.116:0/4237094313 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa7d409b570 con 0x7fa7e01012c0 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 msgr2=0x7fa7c80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 0x7fa7c80a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000b2e0 tx=0x7fa7d001a040 comp rx=0 tx=0).stop 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 msgr2=0x7fa7e019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9a0 tx=0x7fa7d400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 shutdown_connections 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7e0107380 0x7fa7e01988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa7c80a2910 0x7fa7c80a4dd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7e0100900 0x7fa7e019e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 --2- 172.21.15.116:0/4237094313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7e01012c0 0x7fa7e019ec30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 >> 172.21.15.116:0/4237094313 conn(0x7fa7e0077810 msgr2=0x7fa7e01054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 shutdown_connections 2024-03-19T02:26:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.366+0000 7fa7e6453700 1 -- 172.21.15.116:0/4237094313 wait complete. 2024-03-19T02:26:41.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 -- 172.21.15.116:0/425935980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4102360 msgr2=0x7f76d41027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 --2- 172.21.15.116:0/425935980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4102360 0x7f76d41027e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f76c40099c0 tx=0x7f76c4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 -- 172.21.15.116:0/425935980 shutdown_connections 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 --2- 172.21.15.116:0/425935980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102d20 0x7f76d410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 --2- 172.21.15.116:0/425935980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4108340 0x7f76d4108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 --2- 172.21.15.116:0/425935980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4102360 0x7f76d41027e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 -- 172.21.15.116:0/425935980 >> 172.21.15.116:0/425935980 conn(0x7f76d4079440 msgr2=0x7f76d4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 -- 172.21.15.116:0/425935980 shutdown_connections 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.019+0000 7f76dc015700 1 -- 172.21.15.116:0/425935980 wait complete. 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 Processor -- start 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 -- start start 2024-03-19T02:26:41.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4102d20 0x7f76d419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4112490 con 0x7f76d4108340 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4112310 con 0x7f76d4102360 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76dc015700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4112610 con 0x7f76d4102d20 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76da5b2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d95b0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4102d20 0x7f76d419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60734/0 (socket says 172.21.15.116:60734) 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76da5b2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33408/0 (socket says 172.21.15.116:33408) 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 -- 172.21.15.116:0/3391924739 learned_addr learned my addr 172.21.15.116:0/3391924739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 -- 172.21.15.116:0/3391924739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4102d20 msgr2=0x7f76d419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4102d20 0x7f76d419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 -- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 msgr2=0x7f76d4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.020+0000 7f76d9db1700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76d9db1700 1 -- 172.21.15.116:0/3391924739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d4198f70 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76da5b2700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76d9db1700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f76d000e9c0 tx=0x7f76d000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d000cbd0 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76d000ee60 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00055b0 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d4199260 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.021+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d41a5470 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.022+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d41039f0 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.024+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f76d0004750 con 0x7f76d4102360 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.025+0000 7f76caffd700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 0x7f76c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.025+0000 7f76d95b0700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 0x7f76c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:41.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.025+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f76d00cf1d0 con 0x7f76d4102360 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.025+0000 7f76d95b0700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 0x7f76c00a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76d4079ab0 tx=0x7f76c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.027+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76d0053740 con 0x7f76d4102360 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.259+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f76d402dbb0 con 0x7f76d4102360 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.260+0000 7f76caffd700 1 -- 172.21.15.116:0/3391924739 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f76d009bf10 con 0x7f76d4102360 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stdout:167503724605 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 msgr2=0x7f76c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 0x7f76c00a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76d4079ab0 tx=0x7f76c4005fb0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 msgr2=0x7f76d419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f76d000e9c0 tx=0x7f76d000ecd0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 shutdown_connections 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d4108340 0x7f76d4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76d4102360 0x7f76d419e610 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76c00a2910 0x7f76c00a4dd0 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76d4079ab0 tx=0x7f76c4005fb0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 --2- 172.21.15.116:0/3391924739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76d4102d20 0x7f76d419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 >> 172.21.15.116:0/3391924739 conn(0x7f76d4079440 msgr2=0x7f76d41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.262+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 shutdown_connections 2024-03-19T02:26:41.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:41.263+0000 7f76dc015700 1 -- 172.21.15.116:0/3391924739 wait complete. 2024-03-19T02:26:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3391924739' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:26:42.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4237094313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:26:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3391924739' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:26:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4237094313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:26:42.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 -- 172.21.15.116:0/1713099835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec102360 msgr2=0x7fcfec1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:42.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 --2- 172.21.15.116:0/1713099835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec102360 0x7fcfec1027e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0099c0 tx=0x7fcfdc009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 -- 172.21.15.116:0/1713099835 shutdown_connections 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 --2- 172.21.15.116:0/1713099835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102d20 0x7fcfec10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 --2- 172.21.15.116:0/1713099835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec108340 0x7fcfec108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 --2- 172.21.15.116:0/1713099835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec102360 0x7fcfec1027e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 -- 172.21.15.116:0/1713099835 >> 172.21.15.116:0/1713099835 conn(0x7fcfec079440 msgr2=0x7fcfec079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 -- 172.21.15.116:0/1713099835 shutdown_connections 2024-03-19T02:26:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.128+0000 7fcff426e700 1 -- 172.21.15.116:0/1713099835 wait complete. 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 Processor -- start 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 -- start start 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102360 0x7fcfec19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec102d20 0x7fcfec19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec112470 con 0x7fcfec108340 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1122f0 con 0x7fcfec102d20 2024-03-19T02:26:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff426e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1125f0 con 0x7fcfec102360 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33466/0 (socket says 172.21.15.116:33466) 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 -- 172.21.15.116:0/1000243594 learned_addr learned my addr 172.21.15.116:0/1000243594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff200a700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102360 0x7fcfec19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff1809700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec102d20 0x7fcfec19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 -- 172.21.15.116:0/1000243594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102360 msgr2=0x7fcfec19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102360 0x7fcfec19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 -- 172.21.15.116:0/1000243594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec102d20 msgr2=0x7fcfec19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec102d20 0x7fcfec19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 -- 172.21.15.116:0/1000243594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec198fd0 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.129+0000 7fcff280b700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ef30 tx=0x7fcfe400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.130+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe40092e0 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.130+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe4004bb0 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.130+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec199260 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.130+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4004020 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.130+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfec1a54d0 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.131+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfec1039f0 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.132+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fcfe401b030 con 0x7fcfec108340 2024-03-19T02:26:42.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.132+0000 7fcfe37fe700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 0x7fcfd80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:42.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.133+0000 7fcff200a700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 0x7fcfd80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:42.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.133+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fcfe40ce830 con 0x7fcfec108340 2024-03-19T02:26:42.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.133+0000 7fcff200a700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 0x7fcfd80a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:42.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.137+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcfe4022020 con 0x7fcfec108340 2024-03-19T02:26:42.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3391924739' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:26:42.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4237094313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:26:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.334+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fcfec02dbb0 con 0x7fcfec108340 2024-03-19T02:26:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.335+0000 7fcfe37fe700 1 -- 172.21.15.116:0/1000243594 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcfe4010040 con 0x7fcfec108340 2024-03-19T02:26:42.336 INFO:teuthology.orchestra.run.smithi116.stdout:292057776170 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 msgr2=0x7fcfd80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 0x7fcfd80a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009680 tx=0x7fcfe8009260 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 msgr2=0x7fcfec1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400ef30 tx=0x7fcfe400c3a0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 shutdown_connections 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec108340 0x7fcfec1987f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcfd80a2910 0x7fcfd80a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfec102360 0x7fcfec19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 --2- 172.21.15.116:0/1000243594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfec102d20 0x7fcfec19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 >> 172.21.15.116:0/1000243594 conn(0x7fcfec079440 msgr2=0x7fcfec101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 shutdown_connections 2024-03-19T02:26:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:42.337+0000 7fcff426e700 1 -- 172.21.15.116:0/1000243594 wait complete. 2024-03-19T02:26:42.821 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462962 got 201863462967 for osd.6 2024-03-19T02:26:42.821 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:42.934 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:42 smithi116 ceph-mon[32263]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:42.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1000243594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:26:43.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:42 smithi154 ceph-mon[35693]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1000243594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:26:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:42 smithi162 ceph-mon[35761]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1000243594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:26:43.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724605 for osd.5 2024-03-19T02:26:43.421 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:44.449 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351000 got 347892351006 for osd.11 2024-03-19T02:26:44.449 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:45.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:44 smithi154 ceph-mon[35693]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:44 smithi116 ceph-mon[32263]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:45.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776164 got 292057776170 for osd.9 2024-03-19T02:26:45.137 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:44 smithi162 ceph-mon[35761]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.115+0000 7fdc8de7d700 1 -- 172.21.15.116:0/1596748810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88107400 msgr2=0x7fdc881077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.115+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/1596748810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88107400 0x7fdc881077e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc78009990 tx=0x7fdc78009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 -- 172.21.15.116:0/1596748810 shutdown_connections 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/1596748810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc88101eb0 0x7fdc8810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/1596748810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88107400 0x7fdc881077e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/1596748810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc881014f0 0x7fdc88101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 -- 172.21.15.116:0/1596748810 >> 172.21.15.116:0/1596748810 conn(0x7fdc88078070 msgr2=0x7fdc88078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 -- 172.21.15.116:0/1596748810 shutdown_connections 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 -- 172.21.15.116:0/1596748810 wait complete. 2024-03-19T02:26:46.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 Processor -- start 2024-03-19T02:26:46.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.116+0000 7fdc8de7d700 1 -- start start 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc881014f0 0x7fdc8819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc88107400 0x7fdc88197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc88111440 con 0x7fdc88101eb0 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc881112c0 con 0x7fdc881014f0 2024-03-19T02:26:46.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc8de7d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc881115c0 con 0x7fdc88107400 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33486/0 (socket says 172.21.15.116:33486) 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 -- 172.21.15.116:0/3817731862 learned_addr learned my addr 172.21.15.116:0/3817731862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc877fe700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc881014f0 0x7fdc8819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc87fff700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc88107400 0x7fdc88197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 -- 172.21.15.116:0/3817731862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc88107400 msgr2=0x7fdc88197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc88107400 0x7fdc88197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 -- 172.21.15.116:0/3817731862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc881014f0 msgr2=0x7fdc8819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc881014f0 0x7fdc8819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 -- 172.21.15.116:0/3817731862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc88197fe0 con 0x7fdc88101eb0 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc86ffd700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00e9c0 tx=0x7fdc7c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:46.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.117+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c00cbd0 con 0x7fdc88101eb0 2024-03-19T02:26:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.118+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc7c00ee60 con 0x7fdc88101eb0 2024-03-19T02:26:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.118+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc881982d0 con 0x7fdc88101eb0 2024-03-19T02:26:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.118+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc881a4490 con 0x7fdc88101eb0 2024-03-19T02:26:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.118+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0055b0 con 0x7fdc88101eb0 2024-03-19T02:26:46.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.119+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc881097f0 con 0x7fdc88101eb0 2024-03-19T02:26:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.120+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fdc7c004750 con 0x7fdc88101eb0 2024-03-19T02:26:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.120+0000 7fdc84ff9700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 0x7fdc700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.120+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdc7c0cf030 con 0x7fdc88101eb0 2024-03-19T02:26:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.120+0000 7fdc877fe700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 0x7fdc700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:46.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.121+0000 7fdc877fe700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 0x7fdc700a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800b2e0 tx=0x7fdc78005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:46.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.125+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc7c09bd70 con 0x7fdc88101eb0 2024-03-19T02:26:46.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.328+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fdc88198ce0 con 0x7fdc88101eb0 2024-03-19T02:26:46.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.328+0000 7fdc84ff9700 1 -- 172.21.15.116:0/3817731862 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdc7c09b670 con 0x7fdc88101eb0 2024-03-19T02:26:46.329 INFO:teuthology.orchestra.run.smithi116.stdout:111669149768 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 msgr2=0x7fdc700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 0x7fdc700a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800b2e0 tx=0x7fdc78005960 comp rx=0 tx=0).stop 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 msgr2=0x7fdc8819db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00e9c0 tx=0x7fdc7c00ecd0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 shutdown_connections 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdc700a2890 0x7fdc700a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc88107400 0x7fdc88197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc881014f0 0x7fdc8819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 --2- 172.21.15.116:0/3817731862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc88101eb0 0x7fdc8819db70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:46.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 >> 172.21.15.116:0/3817731862 conn(0x7fdc88078070 msgr2=0x7fdc881002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:46.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.330+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 shutdown_connections 2024-03-19T02:26:46.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:46.331+0000 7fdc8de7d700 1 -- 172.21.15.116:0/3817731862 wait complete. 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:26:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:46 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:26:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:46 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-19T02:26:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:26:46.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:26:46.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:46 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' 2024-03-19T02:26:46.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037806 for osd.8 2024-03-19T02:26:46.669 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:47.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:47 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3817731862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:26:47.458 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149763 got 111669149768 for osd.3 2024-03-19T02:26:47.458 DEBUG:teuthology.parallel:result is None 2024-03-19T02:26:47.458 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-19T02:26:47.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T02:26:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:47 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3817731862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:26:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:47 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3817731862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:26:47.699 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:48 smithi154 ceph-mon[35693]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:48.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:48 smithi116 ceph-mon[32263]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:48.568 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:48 smithi162 ceph-mon[35761]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:49.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.513+0000 7f5b3343c700 1 -- 172.21.15.116:0/1599114687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1076a0 msgr2=0x7f5b2c10f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:49.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.513+0000 7f5b3343c700 1 --2- 172.21.15.116:0/1599114687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c10f220 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400b0d0 tx=0x7f5b2400b3e0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.513+0000 7f5b3343c700 1 -- 172.21.15.116:0/1599114687 shutdown_connections 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.513+0000 7f5b3343c700 1 --2- 172.21.15.116:0/1599114687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c10f220 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.513+0000 7f5b3343c700 1 --2- 172.21.15.116:0/1599114687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 --2- 172.21.15.116:0/1599114687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c102fa0 0x7f5b2c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- 172.21.15.116:0/1599114687 >> 172.21.15.116:0/1599114687 conn(0x7f5b2c079430 msgr2=0x7f5b2c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- 172.21.15.116:0/1599114687 shutdown_connections 2024-03-19T02:26:49.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- 172.21.15.116:0/1599114687 wait complete. 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 Processor -- start 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- start start 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c102fa0 0x7f5b2c074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c1124f0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c112370 con 0x7f5b2c1076a0 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.514+0000 7f5b3343c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c112670 con 0x7f5b2c102fa0 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:49.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44768/0 (socket says 172.21.15.116:44768) 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 -- 172.21.15.116:0/3835908465 learned_addr learned my addr 172.21.15.116:0/3835908465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b309d7700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c102fa0 0x7f5b2c074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b319d9700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 -- 172.21.15.116:0/3835908465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c102fa0 msgr2=0x7f5b2c074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c102fa0 0x7f5b2c074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 -- 172.21.15.116:0/3835908465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c1076a0 msgr2=0x7f5b2c075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 -- 172.21.15.116:0/3835908465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c1a6eb0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b311d8700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d980 tx=0x7f5b2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:49.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28016070 con 0x7f5b2c1025f0 2024-03-19T02:26:49.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b28004830 con 0x7f5b2c1025f0 2024-03-19T02:26:49.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c1a71a0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28005530 con 0x7f5b2c1025f0 2024-03-19T02:26:49.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.515+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c1a76e0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.516+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c068fd0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.517+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5b280049e0 con 0x7f5b2c1025f0 2024-03-19T02:26:49.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.518+0000 7f5b227fc700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 0x7f5b180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:49.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.518+0000 7f5b309d7700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 0x7f5b180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:49.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.518+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f5b280cf090 con 0x7f5b2c1025f0 2024-03-19T02:26:49.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.519+0000 7f5b309d7700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 0x7f5b180a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c108b80 tx=0x7f5b1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:49.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.522+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b2801f020 con 0x7f5b2c1025f0 2024-03-19T02:26:49.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.713+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b2c062750 con 0x7f5b180a2890 2024-03-19T02:26:49.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.724+0000 7f5b227fc700 1 -- 172.21.15.116:0/3835908465 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340517 (secure 0 0 0) 0x7f5b2c062750 con 0x7f5b180a2890 2024-03-19T02:26:49.730 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 msgr2=0x7f5b180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 0x7f5b180a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c108b80 tx=0x7f5b1c009260 comp rx=0 tx=0).stop 2024-03-19T02:26:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 msgr2=0x7f5b2c074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d980 tx=0x7f5b2800dc90 comp rx=0 tx=0).stop 2024-03-19T02:26:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 shutdown_connections 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c1076a0 0x7f5b2c075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1025f0 0x7f5b2c074610 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b180a2890 0x7f5b180a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 --2- 172.21.15.116:0/3835908465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c102fa0 0x7f5b2c074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 >> 172.21.15.116:0/3835908465 conn(0x7f5b2c079430 msgr2=0x7f5b2c101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 shutdown_connections 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:49.729+0000 7f5b3343c700 1 -- 172.21.15.116:0/3835908465 wait complete. 2024-03-19T02:26:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:26:50.455 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-19T02:26:49.623810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659048,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":877,"num_read_kb":567,"num_write":426,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":850,"ondisk_log_size":850,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371796,"kb_used_data":14324,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570028,"statfs":{"total":1151940427776,"available":1151559708672,"internally_reserved":0,"allocated":14667776,"data_stored":7999658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":66,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003448"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647254+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:26:37.647254+0000","last_peered":"2024-03-19T02:26:37.647254+0000","last_clean":"2024-03-19T02:26:37.647254+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:26:37.647254+0000","last_undegraded":"2024-03-19T02:26:37.647254+0000","last_fullsized":"2024-03-19T02:26:37.647254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.626244+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:26:37.626244+0000","last_peered":"2024-03-19T02:26:37.626244+0000","last_clean":"2024-03-19T02:26:37.626244+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:26:37.626244+0000","last_undegraded":"2024-03-19T02:26:37.626244+0000","last_fullsized":"2024-03-19T02:26:37.626244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643815+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:26:37.643815+0000","last_peered":"2024-03-19T02:26:37.643815+0000","last_clean":"2024-03-19T02:26:37.643815+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:26:37.643815+0000","last_undegraded":"2024-03-19T02:26:37.643815+0000","last_fullsized":"2024-03-19T02:26:37.643815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630617+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:26:37.630617+0000","last_peered":"2024-03-19T02:26:37.630617+0000","last_clean":"2024-03-19T02:26:37.630617+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:26:37.630617+0000","last_undegraded":"2024-03-19T02:26:37.630617+0000","last_fullsized":"2024-03-19T02:26:37.630617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.619443+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:26:37.619443+0000","last_peered":"2024-03-19T02:26:37.619443+0000","last_clean":"2024-03-19T02:26:37.619443+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:26:37.619443+0000","last_undegraded":"2024-03-19T02:26:37.619443+0000","last_fullsized":"2024-03-19T02:26:37.619443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636947+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:26:37.636947+0000","last_peered":"2024-03-19T02:26:37.636947+0000","last_clean":"2024-03-19T02:26:37.636947+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:26:37.636947+0000","last_undegraded":"2024-03-19T02:26:37.636947+0000","last_fullsized":"2024-03-19T02:26:37.636947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634368+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:26:37.634368+0000","last_peered":"2024-03-19T02:26:37.634368+0000","last_clean":"2024-03-19T02:26:37.634368+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:26:37.634368+0000","last_undegraded":"2024-03-19T02:26:37.634368+0000","last_fullsized":"2024-03-19T02:26:37.634368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636293+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:26:37.636293+0000","last_peered":"2024-03-19T02:26:37.636293+0000","last_clean":"2024-03-19T02:26:37.636293+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:26:37.636293+0000","last_undegraded":"2024-03-19T02:26:37.636293+0000","last_fullsized":"2024-03-19T02:26:37.636293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642534+0000","last_change":"2024-03-19T02:26:08.850490+0000","last_active":"2024-03-19T02:26:37.642534+0000","last_peered":"2024-03-19T02:26:37.642534+0000","last_clean":"2024-03-19T02:26:37.642534+0000","last_became_active":"2024-03-19T02:26:08.850087+0000","last_became_peered":"2024-03-19T02:26:08.850087+0000","last_unstale":"2024-03-19T02:26:37.642534+0000","last_undegraded":"2024-03-19T02:26:37.642534+0000","last_fullsized":"2024-03-19T02:26:37.642534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636720+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:26:37.636720+0000","last_peered":"2024-03-19T02:26:37.636720+0000","last_clean":"2024-03-19T02:26:37.636720+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:26:37.636720+0000","last_undegraded":"2024-03-19T02:26:37.636720+0000","last_fullsized":"2024-03-19T02:26:37.636720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645253+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:26:37.645253+0000","last_peered":"2024-03-19T02:26:37.645253+0000","last_clean":"2024-03-19T02:26:37.645253+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:26:37.645253+0000","last_undegraded":"2024-03-19T02:26:37.645253+0000","last_fullsized":"2024-03-19T02:26:37.645253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.623229+0000","last_change":"2024-03-19T02:26:08.789057+0000","last_active":"2024-03-19T02:26:37.623229+0000","last_peered":"2024-03-19T02:26:37.623229+0000","last_clean":"2024-03-19T02:26:37.623229+0000","last_became_active":"2024-03-19T02:26:08.788689+0000","last_became_peered":"2024-03-19T02:26:08.788689+0000","last_unstale":"2024-03-19T02:26:37.623229+0000","last_undegraded":"2024-03-19T02:26:37.623229+0000","last_fullsized":"2024-03-19T02:26:37.623229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:08.295772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643445+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:26:37.643445+0000","last_peered":"2024-03-19T02:26:37.643445+0000","last_clean":"2024-03-19T02:26:37.643445+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:26:37.643445+0000","last_undegraded":"2024-03-19T02:26:37.643445+0000","last_fullsized":"2024-03-19T02:26:37.643445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.627588+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:26:37.627588+0000","last_peered":"2024-03-19T02:26:37.627588+0000","last_clean":"2024-03-19T02:26:37.627588+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:26:37.627588+0000","last_undegraded":"2024-03-19T02:26:37.627588+0000","last_fullsized":"2024-03-19T02:26:37.627588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.626405+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:26:37.626405+0000","last_peered":"2024-03-19T02:26:37.626405+0000","last_clean":"2024-03-19T02:26:37.626405+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:26:37.626405+0000","last_undegraded":"2024-03-19T02:26:37.626405+0000","last_fullsized":"2024-03-19T02:26:37.626405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.646720+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:26:37.646720+0000","last_peered":"2024-03-19T02:26:37.646720+0000","last_clean":"2024-03-19T02:26:37.646720+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:26:37.646720+0000","last_undegraded":"2024-03-19T02:26:37.646720+0000","last_fullsized":"2024-03-19T02:26:37.646720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647365+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:26:37.647365+0000","last_peered":"2024-03-19T02:26:37.647365+0000","last_clean":"2024-03-19T02:26:37.647365+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:26:37.647365+0000","last_undegraded":"2024-03-19T02:26:37.647365+0000","last_fullsized":"2024-03-19T02:26:37.647365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.624143+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:26:37.624143+0000","last_peered":"2024-03-19T02:26:37.624143+0000","last_clean":"2024-03-19T02:26:37.624143+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:26:37.624143+0000","last_undegraded":"2024-03-19T02:26:37.624143+0000","last_fullsized":"2024-03-19T02:26:37.624143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.640895+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:26:37.640895+0000","last_peered":"2024-03-19T02:26:37.640895+0000","last_clean":"2024-03-19T02:26:37.640895+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:26:37.640895+0000","last_undegraded":"2024-03-19T02:26:37.640895+0000","last_fullsized":"2024-03-19T02:26:37.640895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784660+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:26:08.784660+0000","last_peered":"2024-03-19T02:26:08.784660+0000","last_clean":"2024-03-19T02:26:08.784660+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:26:08.784660+0000","last_undegraded":"2024-03-19T02:26:08.784660+0000","last_fullsized":"2024-03-19T02:26:08.784660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:42.753077+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:26:42.753077+0000","last_peered":"2024-03-19T02:26:42.753077+0000","last_clean":"2024-03-19T02:26:42.753077+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:26:42.753077+0000","last_undegraded":"2024-03-19T02:26:42.753077+0000","last_fullsized":"2024-03-19T02:26:42.753077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638692+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:26:37.638692+0000","last_peered":"2024-03-19T02:26:37.638692+0000","last_clean":"2024-03-19T02:26:37.638692+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:26:37.638692+0000","last_undegraded":"2024-03-19T02:26:37.638692+0000","last_fullsized":"2024-03-19T02:26:37.638692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848768+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:26:08.848768+0000","last_peered":"2024-03-19T02:26:08.848768+0000","last_clean":"2024-03-19T02:26:08.848768+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:26:08.848768+0000","last_undegraded":"2024-03-19T02:26:08.848768+0000","last_fullsized":"2024-03-19T02:26:08.848768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788785+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:26:08.788785+0000","last_peered":"2024-03-19T02:26:08.788785+0000","last_clean":"2024-03-19T02:26:08.788785+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:26:08.788785+0000","last_undegraded":"2024-03-19T02:26:08.788785+0000","last_fullsized":"2024-03-19T02:26:08.788785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.631931+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:26:37.631931+0000","last_peered":"2024-03-19T02:26:37.631931+0000","last_clean":"2024-03-19T02:26:37.631931+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:26:37.631931+0000","last_undegraded":"2024-03-19T02:26:37.631931+0000","last_fullsized":"2024-03-19T02:26:37.631931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849957+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:26:08.849957+0000","last_peered":"2024-03-19T02:26:08.849957+0000","last_clean":"2024-03-19T02:26:08.849957+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:26:08.849957+0000","last_undegraded":"2024-03-19T02:26:08.849957+0000","last_fullsized":"2024-03-19T02:26:08.849957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783299+0000","last_change":"2024-03-19T02:25:07.648715+0000","last_active":"2024-03-19T02:26:08.783299+0000","last_peered":"2024-03-19T02:26:08.783299+0000","last_clean":"2024-03-19T02:26:08.783299+0000","last_became_active":"2024-03-19T02:25:07.646184+0000","last_became_peered":"2024-03-19T02:25:07.646184+0000","last_unstale":"2024-03-19T02:26:08.783299+0000","last_undegraded":"2024-03-19T02:26:08.783299+0000","last_fullsized":"2024-03-19T02:26:08.783299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:04.444659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647017+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:26:37.647017+0000","last_peered":"2024-03-19T02:26:37.647017+0000","last_clean":"2024-03-19T02:26:37.647017+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:26:37.647017+0000","last_undegraded":"2024-03-19T02:26:37.647017+0000","last_fullsized":"2024-03-19T02:26:37.647017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848523+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:26:08.848523+0000","last_peered":"2024-03-19T02:26:08.848523+0000","last_clean":"2024-03-19T02:26:08.848523+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:26:08.848523+0000","last_undegraded":"2024-03-19T02:26:08.848523+0000","last_fullsized":"2024-03-19T02:26:08.848523+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783590+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:26:08.783590+0000","last_peered":"2024-03-19T02:26:08.783590+0000","last_clean":"2024-03-19T02:26:08.783590+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:26:08.783590+0000","last_undegraded":"2024-03-19T02:26:08.783590+0000","last_fullsized":"2024-03-19T02:26:08.783590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643998+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:26:37.643998+0000","last_peered":"2024-03-19T02:26:37.643998+0000","last_clean":"2024-03-19T02:26:37.643998+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:26:37.643998+0000","last_undegraded":"2024-03-19T02:26:37.643998+0000","last_fullsized":"2024-03-19T02:26:37.643998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784836+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:26:08.784836+0000","last_peered":"2024-03-19T02:26:08.784836+0000","last_clean":"2024-03-19T02:26:08.784836+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:26:08.784836+0000","last_undegraded":"2024-03-19T02:26:08.784836+0000","last_fullsized":"2024-03-19T02:26:08.784836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:43.751100+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:26:43.751100+0000","last_peered":"2024-03-19T02:26:43.751100+0000","last_clean":"2024-03-19T02:26:43.751100+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:26:43.751100+0000","last_undegraded":"2024-03-19T02:26:43.751100+0000","last_fullsized":"2024-03-19T02:26:43.751100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.624742+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:26:37.624742+0000","last_peered":"2024-03-19T02:26:37.624742+0000","last_clean":"2024-03-19T02:26:37.624742+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:26:37.624742+0000","last_undegraded":"2024-03-19T02:26:37.624742+0000","last_fullsized":"2024-03-19T02:26:37.624742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850745+0000","last_change":"2024-03-19T02:26:08.850745+0000","last_active":"2024-03-19T02:26:08.850745+0000","last_peered":"2024-03-19T02:26:08.850745+0000","last_clean":"2024-03-19T02:26:08.850745+0000","last_became_active":"2024-03-19T02:26:08.850285+0000","last_became_peered":"2024-03-19T02:26:08.850285+0000","last_unstale":"2024-03-19T02:26:08.850745+0000","last_undegraded":"2024-03-19T02:26:08.850745+0000","last_fullsized":"2024-03-19T02:26:08.850745+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:17.233944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782535+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:26:08.782535+0000","last_peered":"2024-03-19T02:26:08.782535+0000","last_clean":"2024-03-19T02:26:08.782535+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:26:08.782535+0000","last_undegraded":"2024-03-19T02:26:08.782535+0000","last_fullsized":"2024-03-19T02:26:08.782535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.632423+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:26:37.632423+0000","last_peered":"2024-03-19T02:26:37.632423+0000","last_clean":"2024-03-19T02:26:37.632423+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:26:37.632423+0000","last_undegraded":"2024-03-19T02:26:37.632423+0000","last_fullsized":"2024-03-19T02:26:37.632423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850073+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:26:08.850073+0000","last_peered":"2024-03-19T02:26:08.850073+0000","last_clean":"2024-03-19T02:26:08.850073+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:26:08.850073+0000","last_undegraded":"2024-03-19T02:26:08.850073+0000","last_fullsized":"2024-03-19T02:26:08.850073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787342+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:26:08.787342+0000","last_peered":"2024-03-19T02:26:08.787342+0000","last_clean":"2024-03-19T02:26:08.787342+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:26:08.787342+0000","last_undegraded":"2024-03-19T02:26:08.787342+0000","last_fullsized":"2024-03-19T02:26:08.787342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641978+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:26:37.641978+0000","last_peered":"2024-03-19T02:26:37.641978+0000","last_clean":"2024-03-19T02:26:37.641978+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:26:37.641978+0000","last_undegraded":"2024-03-19T02:26:37.641978+0000","last_fullsized":"2024-03-19T02:26:37.641978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784932+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:26:08.784932+0000","last_peered":"2024-03-19T02:26:08.784932+0000","last_clean":"2024-03-19T02:26:08.784932+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:26:08.784932+0000","last_undegraded":"2024-03-19T02:26:08.784932+0000","last_fullsized":"2024-03-19T02:26:08.784932+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848513+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:26:08.848513+0000","last_peered":"2024-03-19T02:26:08.848513+0000","last_clean":"2024-03-19T02:26:08.848513+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:26:08.848513+0000","last_undegraded":"2024-03-19T02:26:08.848513+0000","last_fullsized":"2024-03-19T02:26:08.848513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641892+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:26:37.641892+0000","last_peered":"2024-03-19T02:26:37.641892+0000","last_clean":"2024-03-19T02:26:37.641892+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:26:37.641892+0000","last_undegraded":"2024-03-19T02:26:37.641892+0000","last_fullsized":"2024-03-19T02:26:37.641892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786503+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:26:08.786503+0000","last_peered":"2024-03-19T02:26:08.786503+0000","last_clean":"2024-03-19T02:26:08.786503+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:26:08.786503+0000","last_undegraded":"2024-03-19T02:26:08.786503+0000","last_fullsized":"2024-03-19T02:26:08.786503+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":175,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:47.754612+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:26:47.754612+0000","last_peered":"2024-03-19T02:26:47.754612+0000","last_clean":"2024-03-19T02:26:47.754612+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:26:47.754612+0000","last_undegraded":"2024-03-19T02:26:47.754612+0000","last_fullsized":"2024-03-19T02:26:47.754612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.646981+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:26:37.646981+0000","last_peered":"2024-03-19T02:26:37.646981+0000","last_clean":"2024-03-19T02:26:37.646981+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:26:37.646981+0000","last_undegraded":"2024-03-19T02:26:37.646981+0000","last_fullsized":"2024-03-19T02:26:37.646981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.781931+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:26:08.781931+0000","last_peered":"2024-03-19T02:26:08.781931+0000","last_clean":"2024-03-19T02:26:08.781931+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:26:08.781931+0000","last_undegraded":"2024-03-19T02:26:08.781931+0000","last_fullsized":"2024-03-19T02:26:08.781931+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849105+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:26:08.849105+0000","last_peered":"2024-03-19T02:26:08.849105+0000","last_clean":"2024-03-19T02:26:08.849105+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:26:08.849105+0000","last_undegraded":"2024-03-19T02:26:08.849105+0000","last_fullsized":"2024-03-19T02:26:08.849105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636537+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:26:37.636537+0000","last_peered":"2024-03-19T02:26:37.636537+0000","last_clean":"2024-03-19T02:26:37.636537+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:26:37.636537+0000","last_undegraded":"2024-03-19T02:26:37.636537+0000","last_fullsized":"2024-03-19T02:26:37.636537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850194+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:26:08.850194+0000","last_peered":"2024-03-19T02:26:08.850194+0000","last_clean":"2024-03-19T02:26:08.850194+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:26:08.850194+0000","last_undegraded":"2024-03-19T02:26:08.850194+0000","last_fullsized":"2024-03-19T02:26:08.850194+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849001+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:26:08.849001+0000","last_peered":"2024-03-19T02:26:08.849001+0000","last_clean":"2024-03-19T02:26:08.849001+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:26:08.849001+0000","last_undegraded":"2024-03-19T02:26:08.849001+0000","last_fullsized":"2024-03-19T02:26:08.849001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647373+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:26:37.647373+0000","last_peered":"2024-03-19T02:26:37.647373+0000","last_clean":"2024-03-19T02:26:37.647373+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:26:37.647373+0000","last_undegraded":"2024-03-19T02:26:37.647373+0000","last_fullsized":"2024-03-19T02:26:37.647373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782070+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:26:08.782070+0000","last_peered":"2024-03-19T02:26:08.782070+0000","last_clean":"2024-03-19T02:26:08.782070+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:26:08.782070+0000","last_undegraded":"2024-03-19T02:26:08.782070+0000","last_fullsized":"2024-03-19T02:26:08.782070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:47.754665+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:26:47.754665+0000","last_peered":"2024-03-19T02:26:47.754665+0000","last_clean":"2024-03-19T02:26:47.754665+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:26:47.754665+0000","last_undegraded":"2024-03-19T02:26:47.754665+0000","last_fullsized":"2024-03-19T02:26:47.754665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62024,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":136,"num_write":38,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.628671+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:26:37.628671+0000","last_peered":"2024-03-19T02:26:37.628671+0000","last_clean":"2024-03-19T02:26:37.628671+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:26:37.628671+0000","last_undegraded":"2024-03-19T02:26:37.628671+0000","last_fullsized":"2024-03-19T02:26:37.628671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849227+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:26:08.849227+0000","last_peered":"2024-03-19T02:26:08.849227+0000","last_clean":"2024-03-19T02:26:08.849227+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:26:08.849227+0000","last_undegraded":"2024-03-19T02:26:08.849227+0000","last_fullsized":"2024-03-19T02:26:08.849227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784722+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:26:08.784722+0000","last_peered":"2024-03-19T02:26:08.784722+0000","last_clean":"2024-03-19T02:26:08.784722+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:26:08.784722+0000","last_undegraded":"2024-03-19T02:26:08.784722+0000","last_fullsized":"2024-03-19T02:26:08.784722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643157+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:26:37.643157+0000","last_peered":"2024-03-19T02:26:37.643157+0000","last_clean":"2024-03-19T02:26:37.643157+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:26:37.643157+0000","last_undegraded":"2024-03-19T02:26:37.643157+0000","last_fullsized":"2024-03-19T02:26:37.643157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849704+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:26:08.849704+0000","last_peered":"2024-03-19T02:26:08.849704+0000","last_clean":"2024-03-19T02:26:08.849704+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:26:08.849704+0000","last_undegraded":"2024-03-19T02:26:08.849704+0000","last_fullsized":"2024-03-19T02:26:08.849704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849370+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:26:08.849370+0000","last_peered":"2024-03-19T02:26:08.849370+0000","last_clean":"2024-03-19T02:26:08.849370+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:26:08.849370+0000","last_undegraded":"2024-03-19T02:26:08.849370+0000","last_fullsized":"2024-03-19T02:26:08.849370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642815+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:26:37.642815+0000","last_peered":"2024-03-19T02:26:37.642815+0000","last_clean":"2024-03-19T02:26:37.642815+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:26:37.642815+0000","last_undegraded":"2024-03-19T02:26:37.642815+0000","last_fullsized":"2024-03-19T02:26:37.642815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.852290+0000","last_change":"2024-03-19T02:26:08.850986+0000","last_active":"2024-03-19T02:26:08.852290+0000","last_peered":"2024-03-19T02:26:08.852290+0000","last_clean":"2024-03-19T02:26:08.852290+0000","last_became_active":"2024-03-19T02:26:08.850619+0000","last_became_peered":"2024-03-19T02:26:08.850619+0000","last_unstale":"2024-03-19T02:26:08.852290+0000","last_undegraded":"2024-03-19T02:26:08.852290+0000","last_fullsized":"2024-03-19T02:26:08.852290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:15.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:43.754520+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:26:43.754520+0000","last_peered":"2024-03-19T02:26:43.754520+0000","last_clean":"2024-03-19T02:26:43.754520+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:26:43.754520+0000","last_undegraded":"2024-03-19T02:26:43.754520+0000","last_fullsized":"2024-03-19T02:26:43.754520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.633138+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:26:37.633138+0000","last_peered":"2024-03-19T02:26:37.633138+0000","last_clean":"2024-03-19T02:26:37.633138+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:26:37.633138+0000","last_undegraded":"2024-03-19T02:26:37.633138+0000","last_fullsized":"2024-03-19T02:26:37.633138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787187+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:26:08.787187+0000","last_peered":"2024-03-19T02:26:08.787187+0000","last_clean":"2024-03-19T02:26:08.787187+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:26:08.787187+0000","last_undegraded":"2024-03-19T02:26:08.787187+0000","last_fullsized":"2024-03-19T02:26:08.787187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782479+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:26:08.782479+0000","last_peered":"2024-03-19T02:26:08.782479+0000","last_clean":"2024-03-19T02:26:08.782479+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:26:08.782479+0000","last_undegraded":"2024-03-19T02:26:08.782479+0000","last_fullsized":"2024-03-19T02:26:08.782479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630792+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:26:37.630792+0000","last_peered":"2024-03-19T02:26:37.630792+0000","last_clean":"2024-03-19T02:26:37.630792+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:26:37.630792+0000","last_undegraded":"2024-03-19T02:26:37.630792+0000","last_fullsized":"2024-03-19T02:26:37.630792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848652+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:26:08.848652+0000","last_peered":"2024-03-19T02:26:08.848652+0000","last_clean":"2024-03-19T02:26:08.848652+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:26:08.848652+0000","last_undegraded":"2024-03-19T02:26:08.848652+0000","last_fullsized":"2024-03-19T02:26:08.848652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783432+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:26:08.783432+0000","last_peered":"2024-03-19T02:26:08.783432+0000","last_clean":"2024-03-19T02:26:08.783432+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:26:08.783432+0000","last_undegraded":"2024-03-19T02:26:08.783432+0000","last_fullsized":"2024-03-19T02:26:08.783432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.639006+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:26:37.639006+0000","last_peered":"2024-03-19T02:26:37.639006+0000","last_clean":"2024-03-19T02:26:37.639006+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:26:37.639006+0000","last_undegraded":"2024-03-19T02:26:37.639006+0000","last_fullsized":"2024-03-19T02:26:37.639006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782714+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:26:08.782714+0000","last_peered":"2024-03-19T02:26:08.782714+0000","last_clean":"2024-03-19T02:26:08.782714+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:26:08.782714+0000","last_undegraded":"2024-03-19T02:26:08.782714+0000","last_fullsized":"2024-03-19T02:26:08.782714+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.851073+0000","last_change":"2024-03-19T02:26:08.851073+0000","last_active":"2024-03-19T02:26:08.851073+0000","last_peered":"2024-03-19T02:26:08.851073+0000","last_clean":"2024-03-19T02:26:08.851073+0000","last_became_active":"2024-03-19T02:26:08.850702+0000","last_became_peered":"2024-03-19T02:26:08.850702+0000","last_unstale":"2024-03-19T02:26:08.851073+0000","last_undegraded":"2024-03-19T02:26:08.851073+0000","last_fullsized":"2024-03-19T02:26:08.851073+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:11.668203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642312+0000","last_change":"2024-03-19T02:26:08.788232+0000","last_active":"2024-03-19T02:26:37.642312+0000","last_peered":"2024-03-19T02:26:37.642312+0000","last_clean":"2024-03-19T02:26:37.642312+0000","last_became_active":"2024-03-19T02:26:08.787993+0000","last_became_peered":"2024-03-19T02:26:08.787993+0000","last_unstale":"2024-03-19T02:26:37.642312+0000","last_undegraded":"2024-03-19T02:26:37.642312+0000","last_fullsized":"2024-03-19T02:26:37.642312+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:40.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786844+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:26:08.786844+0000","last_peered":"2024-03-19T02:26:08.786844+0000","last_clean":"2024-03-19T02:26:08.786844+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:26:08.786844+0000","last_undegraded":"2024-03-19T02:26:08.786844+0000","last_fullsized":"2024-03-19T02:26:08.786844+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:45.756761+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:26:45.756761+0000","last_peered":"2024-03-19T02:26:45.756761+0000","last_clean":"2024-03-19T02:26:45.756761+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:26:45.756761+0000","last_undegraded":"2024-03-19T02:26:45.756761+0000","last_fullsized":"2024-03-19T02:26:45.756761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636395+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:26:37.636395+0000","last_peered":"2024-03-19T02:26:37.636395+0000","last_clean":"2024-03-19T02:26:37.636395+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:26:37.636395+0000","last_undegraded":"2024-03-19T02:26:37.636395+0000","last_fullsized":"2024-03-19T02:26:37.636395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788849+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:26:08.788849+0000","last_peered":"2024-03-19T02:26:08.788849+0000","last_clean":"2024-03-19T02:26:08.788849+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:26:08.788849+0000","last_undegraded":"2024-03-19T02:26:08.788849+0000","last_fullsized":"2024-03-19T02:26:08.788849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630761+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:26:37.630761+0000","last_peered":"2024-03-19T02:26:37.630761+0000","last_clean":"2024-03-19T02:26:37.630761+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:26:37.630761+0000","last_undegraded":"2024-03-19T02:26:37.630761+0000","last_fullsized":"2024-03-19T02:26:37.630761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:44.755169+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:26:44.755169+0000","last_peered":"2024-03-19T02:26:44.755169+0000","last_clean":"2024-03-19T02:26:44.755169+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:26:44.755169+0000","last_undegraded":"2024-03-19T02:26:44.755169+0000","last_fullsized":"2024-03-19T02:26:44.755169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848454+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:26:08.848454+0000","last_peered":"2024-03-19T02:26:08.848454+0000","last_clean":"2024-03-19T02:26:08.848454+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:26:08.848454+0000","last_undegraded":"2024-03-19T02:26:08.848454+0000","last_fullsized":"2024-03-19T02:26:08.848454+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.644175+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:26:37.644175+0000","last_peered":"2024-03-19T02:26:37.644175+0000","last_clean":"2024-03-19T02:26:37.644175+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:26:37.644175+0000","last_undegraded":"2024-03-19T02:26:37.644175+0000","last_fullsized":"2024-03-19T02:26:37.644175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850130+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:26:08.850130+0000","last_peered":"2024-03-19T02:26:08.850130+0000","last_clean":"2024-03-19T02:26:08.850130+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:26:08.850130+0000","last_undegraded":"2024-03-19T02:26:08.850130+0000","last_fullsized":"2024-03-19T02:26:08.850130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":361,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848703+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:26:08.848703+0000","last_peered":"2024-03-19T02:26:08.848703+0000","last_clean":"2024-03-19T02:26:08.848703+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:26:08.848703+0000","last_undegraded":"2024-03-19T02:26:08.848703+0000","last_fullsized":"2024-03-19T02:26:08.848703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848937+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:26:08.848937+0000","last_peered":"2024-03-19T02:26:08.848937+0000","last_clean":"2024-03-19T02:26:08.848937+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:26:08.848937+0000","last_undegraded":"2024-03-19T02:26:08.848937+0000","last_fullsized":"2024-03-19T02:26:08.848937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642438+0000","last_change":"2024-03-19T02:26:08.788444+0000","last_active":"2024-03-19T02:26:37.642438+0000","last_peered":"2024-03-19T02:26:37.642438+0000","last_clean":"2024-03-19T02:26:37.642438+0000","last_became_active":"2024-03-19T02:26:08.788084+0000","last_became_peered":"2024-03-19T02:26:08.788084+0000","last_unstale":"2024-03-19T02:26:37.642438+0000","last_undegraded":"2024-03-19T02:26:37.642438+0000","last_fullsized":"2024-03-19T02:26:37.642438+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:18.879498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848917+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:26:08.848917+0000","last_peered":"2024-03-19T02:26:08.848917+0000","last_clean":"2024-03-19T02:26:08.848917+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:26:08.848917+0000","last_undegraded":"2024-03-19T02:26:08.848917+0000","last_fullsized":"2024-03-19T02:26:08.848917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849421+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:26:08.849421+0000","last_peered":"2024-03-19T02:26:08.849421+0000","last_clean":"2024-03-19T02:26:08.849421+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:26:08.849421+0000","last_undegraded":"2024-03-19T02:26:08.849421+0000","last_fullsized":"2024-03-19T02:26:08.849421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783039+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:26:08.783039+0000","last_peered":"2024-03-19T02:26:08.783039+0000","last_clean":"2024-03-19T02:26:08.783039+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:26:08.783039+0000","last_undegraded":"2024-03-19T02:26:08.783039+0000","last_fullsized":"2024-03-19T02:26:08.783039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642676+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:26:37.642676+0000","last_peered":"2024-03-19T02:26:37.642676+0000","last_clean":"2024-03-19T02:26:37.642676+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:26:37.642676+0000","last_undegraded":"2024-03-19T02:26:37.642676+0000","last_fullsized":"2024-03-19T02:26:37.642676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788585+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:26:08.788585+0000","last_peered":"2024-03-19T02:26:08.788585+0000","last_clean":"2024-03-19T02:26:08.788585+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:26:08.788585+0000","last_undegraded":"2024-03-19T02:26:08.788585+0000","last_fullsized":"2024-03-19T02:26:08.788585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782703+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:26:08.782703+0000","last_peered":"2024-03-19T02:26:08.782703+0000","last_clean":"2024-03-19T02:26:08.782703+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:26:08.782703+0000","last_undegraded":"2024-03-19T02:26:08.782703+0000","last_fullsized":"2024-03-19T02:26:08.782703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641744+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:26:37.641744+0000","last_peered":"2024-03-19T02:26:37.641744+0000","last_clean":"2024-03-19T02:26:37.641744+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:26:37.641744+0000","last_undegraded":"2024-03-19T02:26:37.641744+0000","last_fullsized":"2024-03-19T02:26:37.641744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784197+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:26:08.784197+0000","last_peered":"2024-03-19T02:26:08.784197+0000","last_clean":"2024-03-19T02:26:08.784197+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:26:08.784197+0000","last_undegraded":"2024-03-19T02:26:08.784197+0000","last_fullsized":"2024-03-19T02:26:08.784197+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850898+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:26:08.850898+0000","last_peered":"2024-03-19T02:26:08.850898+0000","last_clean":"2024-03-19T02:26:08.850898+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:26:08.850898+0000","last_undegraded":"2024-03-19T02:26:08.850898+0000","last_fullsized":"2024-03-19T02:26:08.850898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.635563+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:26:37.635563+0000","last_peered":"2024-03-19T02:26:37.635563+0000","last_clean":"2024-03-19T02:26:37.635563+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:26:37.635563+0000","last_undegraded":"2024-03-19T02:26:37.635563+0000","last_fullsized":"2024-03-19T02:26:37.635563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850604+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:26:08.850604+0000","last_peered":"2024-03-19T02:26:08.850604+0000","last_clean":"2024-03-19T02:26:08.850604+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:26:08.850604+0000","last_undegraded":"2024-03-19T02:26:08.850604+0000","last_fullsized":"2024-03-19T02:26:08.850604+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786068+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:26:08.786068+0000","last_peered":"2024-03-19T02:26:08.786068+0000","last_clean":"2024-03-19T02:26:08.786068+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:26:08.786068+0000","last_undegraded":"2024-03-19T02:26:08.786068+0000","last_fullsized":"2024-03-19T02:26:08.786068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.597134+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:26:37.597134+0000","last_peered":"2024-03-19T02:26:37.597134+0000","last_clean":"2024-03-19T02:26:37.597134+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:26:37.597134+0000","last_undegraded":"2024-03-19T02:26:37.597134+0000","last_fullsized":"2024-03-19T02:26:37.597134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782948+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:26:08.782948+0000","last_peered":"2024-03-19T02:26:08.782948+0000","last_clean":"2024-03-19T02:26:08.782948+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:26:08.782948+0000","last_undegraded":"2024-03-19T02:26:08.782948+0000","last_fullsized":"2024-03-19T02:26:08.782948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786609+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:26:08.786609+0000","last_peered":"2024-03-19T02:26:08.786609+0000","last_clean":"2024-03-19T02:26:08.786609+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:26:08.786609+0000","last_undegraded":"2024-03-19T02:26:08.786609+0000","last_fullsized":"2024-03-19T02:26:08.786609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638015+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:26:37.638015+0000","last_peered":"2024-03-19T02:26:37.638015+0000","last_clean":"2024-03-19T02:26:37.638015+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:26:37.638015+0000","last_undegraded":"2024-03-19T02:26:37.638015+0000","last_fullsized":"2024-03-19T02:26:37.638015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850823+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:26:08.850823+0000","last_peered":"2024-03-19T02:26:08.850823+0000","last_clean":"2024-03-19T02:26:08.850823+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:26:08.850823+0000","last_undegraded":"2024-03-19T02:26:08.850823+0000","last_fullsized":"2024-03-19T02:26:08.850823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784484+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:26:08.784484+0000","last_peered":"2024-03-19T02:26:08.784484+0000","last_clean":"2024-03-19T02:26:08.784484+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:26:08.784484+0000","last_undegraded":"2024-03-19T02:26:08.784484+0000","last_fullsized":"2024-03-19T02:26:08.784484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630482+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:26:37.630482+0000","last_peered":"2024-03-19T02:26:37.630482+0000","last_clean":"2024-03-19T02:26:37.630482+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:26:37.630482+0000","last_undegraded":"2024-03-19T02:26:37.630482+0000","last_fullsized":"2024-03-19T02:26:37.630482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850460+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:26:08.850460+0000","last_peered":"2024-03-19T02:26:08.850460+0000","last_clean":"2024-03-19T02:26:08.850460+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:26:08.850460+0000","last_undegraded":"2024-03-19T02:26:08.850460+0000","last_fullsized":"2024-03-19T02:26:08.850460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782712+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:26:08.782712+0000","last_peered":"2024-03-19T02:26:08.782712+0000","last_clean":"2024-03-19T02:26:08.782712+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:26:08.782712+0000","last_undegraded":"2024-03-19T02:26:08.782712+0000","last_fullsized":"2024-03-19T02:26:08.782712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645735+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:26:37.645735+0000","last_peered":"2024-03-19T02:26:37.645735+0000","last_clean":"2024-03-19T02:26:37.645735+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:26:37.645735+0000","last_undegraded":"2024-03-19T02:26:37.645735+0000","last_fullsized":"2024-03-19T02:26:37.645735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.785663+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:26:08.785663+0000","last_peered":"2024-03-19T02:26:08.785663+0000","last_clean":"2024-03-19T02:26:08.785663+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:26:08.785663+0000","last_undegraded":"2024-03-19T02:26:08.785663+0000","last_fullsized":"2024-03-19T02:26:08.785663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:43.754561+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:26:43.754561+0000","last_peered":"2024-03-19T02:26:43.754561+0000","last_clean":"2024-03-19T02:26:43.754561+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:26:43.754561+0000","last_undegraded":"2024-03-19T02:26:43.754561+0000","last_fullsized":"2024-03-19T02:26:43.754561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645144+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:26:37.645144+0000","last_peered":"2024-03-19T02:26:37.645144+0000","last_clean":"2024-03-19T02:26:37.645144+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:26:37.645144+0000","last_undegraded":"2024-03-19T02:26:37.645144+0000","last_fullsized":"2024-03-19T02:26:37.645144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849926+0000","last_change":"2024-03-19T02:26:08.849926+0000","last_active":"2024-03-19T02:26:08.849926+0000","last_peered":"2024-03-19T02:26:08.849926+0000","last_clean":"2024-03-19T02:26:08.849926+0000","last_became_active":"2024-03-19T02:26:08.849548+0000","last_became_peered":"2024-03-19T02:26:08.849548+0000","last_unstale":"2024-03-19T02:26:08.849926+0000","last_undegraded":"2024-03-19T02:26:08.849926+0000","last_fullsized":"2024-03-19T02:26:08.849926+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:44:35.078313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849115+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:26:08.849115+0000","last_peered":"2024-03-19T02:26:08.849115+0000","last_clean":"2024-03-19T02:26:08.849115+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:26:08.849115+0000","last_undegraded":"2024-03-19T02:26:08.849115+0000","last_fullsized":"2024-03-19T02:26:08.849115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.628214+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:26:37.628214+0000","last_peered":"2024-03-19T02:26:37.628214+0000","last_clean":"2024-03-19T02:26:37.628214+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:26:37.628214+0000","last_undegraded":"2024-03-19T02:26:37.628214+0000","last_fullsized":"2024-03-19T02:26:37.628214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.851525+0000","last_change":"2024-03-19T02:26:08.850968+0000","last_active":"2024-03-19T02:26:08.851525+0000","last_peered":"2024-03-19T02:26:08.851525+0000","last_clean":"2024-03-19T02:26:08.851525+0000","last_became_active":"2024-03-19T02:26:08.850571+0000","last_became_peered":"2024-03-19T02:26:08.850571+0000","last_unstale":"2024-03-19T02:26:08.851525+0000","last_undegraded":"2024-03-19T02:26:08.851525+0000","last_fullsized":"2024-03-19T02:26:08.851525+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:29.532645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787300+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:26:08.787300+0000","last_peered":"2024-03-19T02:26:08.787300+0000","last_clean":"2024-03-19T02:26:08.787300+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:26:08.787300+0000","last_undegraded":"2024-03-19T02:26:08.787300+0000","last_fullsized":"2024-03-19T02:26:08.787300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634016+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:26:37.634016+0000","last_peered":"2024-03-19T02:26:37.634016+0000","last_clean":"2024-03-19T02:26:37.634016+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:26:37.634016+0000","last_undegraded":"2024-03-19T02:26:37.634016+0000","last_fullsized":"2024-03-19T02:26:37.634016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641108+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:26:37.641108+0000","last_peered":"2024-03-19T02:26:37.641108+0000","last_clean":"2024-03-19T02:26:37.641108+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:26:37.641108+0000","last_undegraded":"2024-03-19T02:26:37.641108+0000","last_fullsized":"2024-03-19T02:26:37.641108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.625170+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:26:37.625170+0000","last_peered":"2024-03-19T02:26:37.625170+0000","last_clean":"2024-03-19T02:26:37.625170+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:26:37.625170+0000","last_undegraded":"2024-03-19T02:26:37.625170+0000","last_fullsized":"2024-03-19T02:26:37.625170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.625408+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:26:37.625408+0000","last_peered":"2024-03-19T02:26:37.625408+0000","last_clean":"2024-03-19T02:26:37.625408+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:26:37.625408+0000","last_undegraded":"2024-03-19T02:26:37.625408+0000","last_fullsized":"2024-03-19T02:26:37.625408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634739+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:26:37.634739+0000","last_peered":"2024-03-19T02:26:37.634739+0000","last_clean":"2024-03-19T02:26:37.634739+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:26:37.634739+0000","last_undegraded":"2024-03-19T02:26:37.634739+0000","last_fullsized":"2024-03-19T02:26:37.634739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641270+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:26:37.641270+0000","last_peered":"2024-03-19T02:26:37.641270+0000","last_clean":"2024-03-19T02:26:37.641270+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:26:37.641270+0000","last_undegraded":"2024-03-19T02:26:37.641270+0000","last_fullsized":"2024-03-19T02:26:37.641270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638949+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:26:37.638949+0000","last_peered":"2024-03-19T02:26:37.638949+0000","last_clean":"2024-03-19T02:26:37.638949+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:26:37.638949+0000","last_undegraded":"2024-03-19T02:26:37.638949+0000","last_fullsized":"2024-03-19T02:26:37.638949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.617197+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:26:37.617197+0000","last_peered":"2024-03-19T02:26:37.617197+0000","last_clean":"2024-03-19T02:26:37.617197+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:26:37.617197+0000","last_undegraded":"2024-03-19T02:26:37.617197+0000","last_fullsized":"2024-03-19T02:26:37.617197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638115+0000","last_change":"2024-03-19T02:26:08.790951+0000","last_active":"2024-03-19T02:26:37.638115+0000","last_peered":"2024-03-19T02:26:37.638115+0000","last_clean":"2024-03-19T02:26:37.638115+0000","last_became_active":"2024-03-19T02:26:08.790587+0000","last_became_peered":"2024-03-19T02:26:08.790587+0000","last_unstale":"2024-03-19T02:26:37.638115+0000","last_undegraded":"2024-03-19T02:26:37.638115+0000","last_fullsized":"2024-03-19T02:26:37.638115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:11.410269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.629113+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:26:37.629113+0000","last_peered":"2024-03-19T02:26:37.629113+0000","last_clean":"2024-03-19T02:26:37.629113+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:26:37.629113+0000","last_undegraded":"2024-03-19T02:26:37.629113+0000","last_fullsized":"2024-03-19T02:26:37.629113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642650+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:26:37.642650+0000","last_peered":"2024-03-19T02:26:37.642650+0000","last_clean":"2024-03-19T02:26:37.642650+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:26:37.642650+0000","last_undegraded":"2024-03-19T02:26:37.642650+0000","last_fullsized":"2024-03-19T02:26:37.642650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647350+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:26:37.647350+0000","last_peered":"2024-03-19T02:26:37.647350+0000","last_clean":"2024-03-19T02:26:37.647350+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:26:37.647350+0000","last_undegraded":"2024-03-19T02:26:37.647350+0000","last_fullsized":"2024-03-19T02:26:37.647350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.644245+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:26:37.644245+0000","last_peered":"2024-03-19T02:26:37.644245+0000","last_clean":"2024-03-19T02:26:37.644245+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:26:37.644245+0000","last_undegraded":"2024-03-19T02:26:37.644245+0000","last_fullsized":"2024-03-19T02:26:37.644245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638496+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:26:37.638496+0000","last_peered":"2024-03-19T02:26:37.638496+0000","last_clean":"2024-03-19T02:26:37.638496+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:26:37.638496+0000","last_undegraded":"2024-03-19T02:26:37.638496+0000","last_fullsized":"2024-03-19T02:26:37.638496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68446,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":741,"num_read_kb":448,"num_write":243,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":504492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1153332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":508518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":504800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1100906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":499926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":562980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":501382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":563200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":499634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":504000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1096488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:26:50.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T02:26:50.676 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:50 smithi116 ceph-mon[32263]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:50.677 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:50 smithi116 ceph-mon[32263]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:50 smithi154 ceph-mon[35693]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:51.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:50 smithi154 ceph-mon[35693]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:50 smithi162 ceph-mon[35761]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T02:26:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:50 smithi162 ceph-mon[35761]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 -- 172.21.15.116:0/2387753315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100900 msgr2=0x7f8fe0100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/2387753315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100900 0x7f8fe0100d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009990 tx=0x7f8fdc009ca0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 -- 172.21.15.116:0/2387753315 shutdown_connections 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/2387753315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe01012c0 0x7f8fe010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/2387753315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 0x7f8fe0107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/2387753315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100900 0x7f8fe0100d80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 -- 172.21.15.116:0/2387753315 >> 172.21.15.116:0/2387753315 conn(0x7f8fe0077810 msgr2=0x7f8fe00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 -- 172.21.15.116:0/2387753315 shutdown_connections 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.440+0000 7f8fe72e5700 1 -- 172.21.15.116:0/2387753315 wait complete. 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 Processor -- start 2024-03-19T02:26:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 -- start start 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 0x7f8fe019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 0x7f8fe0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0112610 con 0x7f8fe01012c0 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0112490 con 0x7f8fe0100900 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe72e5700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0112790 con 0x7f8fe0107380 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe4880700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe5081700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 0x7f8fe019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe4880700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44788/0 (socket says 172.21.15.116:44788) 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe5882700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 0x7f8fe0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe5081700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 0x7f8fe019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34340/0 (socket says 172.21.15.116:34340) 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.441+0000 7f8fe4880700 1 -- 172.21.15.116:0/792854687 learned_addr learned my addr 172.21.15.116:0/792854687 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 -- 172.21.15.116:0/792854687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 msgr2=0x7f8fe0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 0x7f8fe0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 -- 172.21.15.116:0/792854687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 msgr2=0x7f8fe019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 0x7f8fe019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 -- 172.21.15.116:0/792854687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe01990d0 con 0x7f8fe01012c0 2024-03-19T02:26:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe4880700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc027070 con 0x7f8fe01012c0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe0199360 con 0x7f8fe01012c0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fdc022470 con 0x7f8fe01012c0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00b610 con 0x7f8fe01012c0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.442+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe0101a10 con 0x7f8fe01012c0 2024-03-19T02:26:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.443+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe010a8e0 con 0x7f8fe01012c0 2024-03-19T02:26:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.444+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8fdc02a070 con 0x7f8fe01012c0 2024-03-19T02:26:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.445+0000 7f8fd67fc700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 0x7f8fcc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.445+0000 7f8fe5081700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 0x7f8fcc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:52.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.445+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f8fdc0cfc60 con 0x7f8fe01012c0 2024-03-19T02:26:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.445+0000 7f8fe5081700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 0x7f8fcc0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0199ab0 tx=0x7f8fd0012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.448+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fdc013070 con 0x7f8fe01012c0 2024-03-19T02:26:52.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.640+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fe0199ea0 con 0x7f8fcc0a2890 2024-03-19T02:26:52.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.649+0000 7f8fd67fc700 1 -- 172.21.15.116:0/792854687 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340515 (secure 0 0 0) 0x7f8fe0199ea0 con 0x7f8fcc0a2890 2024-03-19T02:26:52.657 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 msgr2=0x7f8fcc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 0x7f8fcc0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0199ab0 tx=0x7f8fd0012040 comp rx=0 tx=0).stop 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 msgr2=0x7f8fe019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc004790 comp rx=0 tx=0).stop 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 shutdown_connections 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8fcc0a2890 0x7f8fcc0a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe0107380 0x7f8fe0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100900 0x7f8fe019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 --2- 172.21.15.116:0/792854687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe01012c0 0x7f8fe019ecf0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 >> 172.21.15.116:0/792854687 conn(0x7f8fe0077810 msgr2=0x7f8fe0105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 shutdown_connections 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:52.652+0000 7f8fe72e5700 1 -- 172.21.15.116:0/792854687 wait complete. 2024-03-19T02:26:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:26:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:52 smithi154 ceph-mon[35693]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-19T02:26:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:52 smithi116 ceph-mon[32263]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-19T02:26:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:52 smithi162 ceph-mon[35761]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-19T02:26:53.455 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-03-19T02:26:51.624278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659048,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":577,"num_write":426,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":850,"ondisk_log_size":850,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371796,"kb_used_data":14324,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570028,"statfs":{"total":1151940427776,"available":1151559708672,"internally_reserved":0,"allocated":14667776,"data_stored":7999658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":60,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003089"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647254+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:26:37.647254+0000","last_peered":"2024-03-19T02:26:37.647254+0000","last_clean":"2024-03-19T02:26:37.647254+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:26:37.647254+0000","last_undegraded":"2024-03-19T02:26:37.647254+0000","last_fullsized":"2024-03-19T02:26:37.647254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.626244+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:26:37.626244+0000","last_peered":"2024-03-19T02:26:37.626244+0000","last_clean":"2024-03-19T02:26:37.626244+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:26:37.626244+0000","last_undegraded":"2024-03-19T02:26:37.626244+0000","last_fullsized":"2024-03-19T02:26:37.626244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643815+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:26:37.643815+0000","last_peered":"2024-03-19T02:26:37.643815+0000","last_clean":"2024-03-19T02:26:37.643815+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:26:37.643815+0000","last_undegraded":"2024-03-19T02:26:37.643815+0000","last_fullsized":"2024-03-19T02:26:37.643815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630617+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:26:37.630617+0000","last_peered":"2024-03-19T02:26:37.630617+0000","last_clean":"2024-03-19T02:26:37.630617+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:26:37.630617+0000","last_undegraded":"2024-03-19T02:26:37.630617+0000","last_fullsized":"2024-03-19T02:26:37.630617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.619443+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:26:37.619443+0000","last_peered":"2024-03-19T02:26:37.619443+0000","last_clean":"2024-03-19T02:26:37.619443+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:26:37.619443+0000","last_undegraded":"2024-03-19T02:26:37.619443+0000","last_fullsized":"2024-03-19T02:26:37.619443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636947+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:26:37.636947+0000","last_peered":"2024-03-19T02:26:37.636947+0000","last_clean":"2024-03-19T02:26:37.636947+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:26:37.636947+0000","last_undegraded":"2024-03-19T02:26:37.636947+0000","last_fullsized":"2024-03-19T02:26:37.636947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634368+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:26:37.634368+0000","last_peered":"2024-03-19T02:26:37.634368+0000","last_clean":"2024-03-19T02:26:37.634368+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:26:37.634368+0000","last_undegraded":"2024-03-19T02:26:37.634368+0000","last_fullsized":"2024-03-19T02:26:37.634368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636293+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:26:37.636293+0000","last_peered":"2024-03-19T02:26:37.636293+0000","last_clean":"2024-03-19T02:26:37.636293+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:26:37.636293+0000","last_undegraded":"2024-03-19T02:26:37.636293+0000","last_fullsized":"2024-03-19T02:26:37.636293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642534+0000","last_change":"2024-03-19T02:26:08.850490+0000","last_active":"2024-03-19T02:26:37.642534+0000","last_peered":"2024-03-19T02:26:37.642534+0000","last_clean":"2024-03-19T02:26:37.642534+0000","last_became_active":"2024-03-19T02:26:08.850087+0000","last_became_peered":"2024-03-19T02:26:08.850087+0000","last_unstale":"2024-03-19T02:26:37.642534+0000","last_undegraded":"2024-03-19T02:26:37.642534+0000","last_fullsized":"2024-03-19T02:26:37.642534+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636720+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:26:37.636720+0000","last_peered":"2024-03-19T02:26:37.636720+0000","last_clean":"2024-03-19T02:26:37.636720+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:26:37.636720+0000","last_undegraded":"2024-03-19T02:26:37.636720+0000","last_fullsized":"2024-03-19T02:26:37.636720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645253+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:26:37.645253+0000","last_peered":"2024-03-19T02:26:37.645253+0000","last_clean":"2024-03-19T02:26:37.645253+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:26:37.645253+0000","last_undegraded":"2024-03-19T02:26:37.645253+0000","last_fullsized":"2024-03-19T02:26:37.645253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.623229+0000","last_change":"2024-03-19T02:26:08.789057+0000","last_active":"2024-03-19T02:26:37.623229+0000","last_peered":"2024-03-19T02:26:37.623229+0000","last_clean":"2024-03-19T02:26:37.623229+0000","last_became_active":"2024-03-19T02:26:08.788689+0000","last_became_peered":"2024-03-19T02:26:08.788689+0000","last_unstale":"2024-03-19T02:26:37.623229+0000","last_undegraded":"2024-03-19T02:26:37.623229+0000","last_fullsized":"2024-03-19T02:26:37.623229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:08.295772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643445+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:26:37.643445+0000","last_peered":"2024-03-19T02:26:37.643445+0000","last_clean":"2024-03-19T02:26:37.643445+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:26:37.643445+0000","last_undegraded":"2024-03-19T02:26:37.643445+0000","last_fullsized":"2024-03-19T02:26:37.643445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.627588+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:26:37.627588+0000","last_peered":"2024-03-19T02:26:37.627588+0000","last_clean":"2024-03-19T02:26:37.627588+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:26:37.627588+0000","last_undegraded":"2024-03-19T02:26:37.627588+0000","last_fullsized":"2024-03-19T02:26:37.627588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.626405+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:26:37.626405+0000","last_peered":"2024-03-19T02:26:37.626405+0000","last_clean":"2024-03-19T02:26:37.626405+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:26:37.626405+0000","last_undegraded":"2024-03-19T02:26:37.626405+0000","last_fullsized":"2024-03-19T02:26:37.626405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.646720+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:26:37.646720+0000","last_peered":"2024-03-19T02:26:37.646720+0000","last_clean":"2024-03-19T02:26:37.646720+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:26:37.646720+0000","last_undegraded":"2024-03-19T02:26:37.646720+0000","last_fullsized":"2024-03-19T02:26:37.646720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647365+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:26:37.647365+0000","last_peered":"2024-03-19T02:26:37.647365+0000","last_clean":"2024-03-19T02:26:37.647365+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:26:37.647365+0000","last_undegraded":"2024-03-19T02:26:37.647365+0000","last_fullsized":"2024-03-19T02:26:37.647365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.624143+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:26:37.624143+0000","last_peered":"2024-03-19T02:26:37.624143+0000","last_clean":"2024-03-19T02:26:37.624143+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:26:37.624143+0000","last_undegraded":"2024-03-19T02:26:37.624143+0000","last_fullsized":"2024-03-19T02:26:37.624143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.640895+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:26:37.640895+0000","last_peered":"2024-03-19T02:26:37.640895+0000","last_clean":"2024-03-19T02:26:37.640895+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:26:37.640895+0000","last_undegraded":"2024-03-19T02:26:37.640895+0000","last_fullsized":"2024-03-19T02:26:37.640895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784660+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:26:08.784660+0000","last_peered":"2024-03-19T02:26:08.784660+0000","last_clean":"2024-03-19T02:26:08.784660+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:26:08.784660+0000","last_undegraded":"2024-03-19T02:26:08.784660+0000","last_fullsized":"2024-03-19T02:26:08.784660+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:47.758266+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:26:47.758266+0000","last_peered":"2024-03-19T02:26:47.758266+0000","last_clean":"2024-03-19T02:26:47.758266+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:26:47.758266+0000","last_undegraded":"2024-03-19T02:26:47.758266+0000","last_fullsized":"2024-03-19T02:26:47.758266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638692+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:26:37.638692+0000","last_peered":"2024-03-19T02:26:37.638692+0000","last_clean":"2024-03-19T02:26:37.638692+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:26:37.638692+0000","last_undegraded":"2024-03-19T02:26:37.638692+0000","last_fullsized":"2024-03-19T02:26:37.638692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848768+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:26:08.848768+0000","last_peered":"2024-03-19T02:26:08.848768+0000","last_clean":"2024-03-19T02:26:08.848768+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:26:08.848768+0000","last_undegraded":"2024-03-19T02:26:08.848768+0000","last_fullsized":"2024-03-19T02:26:08.848768+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788785+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:26:08.788785+0000","last_peered":"2024-03-19T02:26:08.788785+0000","last_clean":"2024-03-19T02:26:08.788785+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:26:08.788785+0000","last_undegraded":"2024-03-19T02:26:08.788785+0000","last_fullsized":"2024-03-19T02:26:08.788785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.631931+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:26:37.631931+0000","last_peered":"2024-03-19T02:26:37.631931+0000","last_clean":"2024-03-19T02:26:37.631931+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:26:37.631931+0000","last_undegraded":"2024-03-19T02:26:37.631931+0000","last_fullsized":"2024-03-19T02:26:37.631931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849957+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:26:08.849957+0000","last_peered":"2024-03-19T02:26:08.849957+0000","last_clean":"2024-03-19T02:26:08.849957+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:26:08.849957+0000","last_undegraded":"2024-03-19T02:26:08.849957+0000","last_fullsized":"2024-03-19T02:26:08.849957+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783299+0000","last_change":"2024-03-19T02:25:07.648715+0000","last_active":"2024-03-19T02:26:08.783299+0000","last_peered":"2024-03-19T02:26:08.783299+0000","last_clean":"2024-03-19T02:26:08.783299+0000","last_became_active":"2024-03-19T02:25:07.646184+0000","last_became_peered":"2024-03-19T02:25:07.646184+0000","last_unstale":"2024-03-19T02:26:08.783299+0000","last_undegraded":"2024-03-19T02:26:08.783299+0000","last_fullsized":"2024-03-19T02:26:08.783299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:04.444659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647017+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:26:37.647017+0000","last_peered":"2024-03-19T02:26:37.647017+0000","last_clean":"2024-03-19T02:26:37.647017+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:26:37.647017+0000","last_undegraded":"2024-03-19T02:26:37.647017+0000","last_fullsized":"2024-03-19T02:26:37.647017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848523+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:26:08.848523+0000","last_peered":"2024-03-19T02:26:08.848523+0000","last_clean":"2024-03-19T02:26:08.848523+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:26:08.848523+0000","last_undegraded":"2024-03-19T02:26:08.848523+0000","last_fullsized":"2024-03-19T02:26:08.848523+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783590+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:26:08.783590+0000","last_peered":"2024-03-19T02:26:08.783590+0000","last_clean":"2024-03-19T02:26:08.783590+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:26:08.783590+0000","last_undegraded":"2024-03-19T02:26:08.783590+0000","last_fullsized":"2024-03-19T02:26:08.783590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643998+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:26:37.643998+0000","last_peered":"2024-03-19T02:26:37.643998+0000","last_clean":"2024-03-19T02:26:37.643998+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:26:37.643998+0000","last_undegraded":"2024-03-19T02:26:37.643998+0000","last_fullsized":"2024-03-19T02:26:37.643998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784836+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:26:08.784836+0000","last_peered":"2024-03-19T02:26:08.784836+0000","last_clean":"2024-03-19T02:26:08.784836+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:26:08.784836+0000","last_undegraded":"2024-03-19T02:26:08.784836+0000","last_fullsized":"2024-03-19T02:26:08.784836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:48.755955+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:26:48.755955+0000","last_peered":"2024-03-19T02:26:48.755955+0000","last_clean":"2024-03-19T02:26:48.755955+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:26:48.755955+0000","last_undegraded":"2024-03-19T02:26:48.755955+0000","last_fullsized":"2024-03-19T02:26:48.755955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.624742+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:26:37.624742+0000","last_peered":"2024-03-19T02:26:37.624742+0000","last_clean":"2024-03-19T02:26:37.624742+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:26:37.624742+0000","last_undegraded":"2024-03-19T02:26:37.624742+0000","last_fullsized":"2024-03-19T02:26:37.624742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850745+0000","last_change":"2024-03-19T02:26:08.850745+0000","last_active":"2024-03-19T02:26:08.850745+0000","last_peered":"2024-03-19T02:26:08.850745+0000","last_clean":"2024-03-19T02:26:08.850745+0000","last_became_active":"2024-03-19T02:26:08.850285+0000","last_became_peered":"2024-03-19T02:26:08.850285+0000","last_unstale":"2024-03-19T02:26:08.850745+0000","last_undegraded":"2024-03-19T02:26:08.850745+0000","last_fullsized":"2024-03-19T02:26:08.850745+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:17.233944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782535+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:26:08.782535+0000","last_peered":"2024-03-19T02:26:08.782535+0000","last_clean":"2024-03-19T02:26:08.782535+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:26:08.782535+0000","last_undegraded":"2024-03-19T02:26:08.782535+0000","last_fullsized":"2024-03-19T02:26:08.782535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.632423+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:26:37.632423+0000","last_peered":"2024-03-19T02:26:37.632423+0000","last_clean":"2024-03-19T02:26:37.632423+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:26:37.632423+0000","last_undegraded":"2024-03-19T02:26:37.632423+0000","last_fullsized":"2024-03-19T02:26:37.632423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850073+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:26:08.850073+0000","last_peered":"2024-03-19T02:26:08.850073+0000","last_clean":"2024-03-19T02:26:08.850073+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:26:08.850073+0000","last_undegraded":"2024-03-19T02:26:08.850073+0000","last_fullsized":"2024-03-19T02:26:08.850073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787342+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:26:08.787342+0000","last_peered":"2024-03-19T02:26:08.787342+0000","last_clean":"2024-03-19T02:26:08.787342+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:26:08.787342+0000","last_undegraded":"2024-03-19T02:26:08.787342+0000","last_fullsized":"2024-03-19T02:26:08.787342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641978+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:26:37.641978+0000","last_peered":"2024-03-19T02:26:37.641978+0000","last_clean":"2024-03-19T02:26:37.641978+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:26:37.641978+0000","last_undegraded":"2024-03-19T02:26:37.641978+0000","last_fullsized":"2024-03-19T02:26:37.641978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784932+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:26:08.784932+0000","last_peered":"2024-03-19T02:26:08.784932+0000","last_clean":"2024-03-19T02:26:08.784932+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:26:08.784932+0000","last_undegraded":"2024-03-19T02:26:08.784932+0000","last_fullsized":"2024-03-19T02:26:08.784932+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848513+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:26:08.848513+0000","last_peered":"2024-03-19T02:26:08.848513+0000","last_clean":"2024-03-19T02:26:08.848513+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:26:08.848513+0000","last_undegraded":"2024-03-19T02:26:08.848513+0000","last_fullsized":"2024-03-19T02:26:08.848513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641892+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:26:37.641892+0000","last_peered":"2024-03-19T02:26:37.641892+0000","last_clean":"2024-03-19T02:26:37.641892+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:26:37.641892+0000","last_undegraded":"2024-03-19T02:26:37.641892+0000","last_fullsized":"2024-03-19T02:26:37.641892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786503+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:26:08.786503+0000","last_peered":"2024-03-19T02:26:08.786503+0000","last_clean":"2024-03-19T02:26:08.786503+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:26:08.786503+0000","last_undegraded":"2024-03-19T02:26:08.786503+0000","last_fullsized":"2024-03-19T02:26:08.786503+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":175,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:47.754612+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:26:47.754612+0000","last_peered":"2024-03-19T02:26:47.754612+0000","last_clean":"2024-03-19T02:26:47.754612+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:26:47.754612+0000","last_undegraded":"2024-03-19T02:26:47.754612+0000","last_fullsized":"2024-03-19T02:26:47.754612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.646981+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:26:37.646981+0000","last_peered":"2024-03-19T02:26:37.646981+0000","last_clean":"2024-03-19T02:26:37.646981+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:26:37.646981+0000","last_undegraded":"2024-03-19T02:26:37.646981+0000","last_fullsized":"2024-03-19T02:26:37.646981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.781931+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:26:08.781931+0000","last_peered":"2024-03-19T02:26:08.781931+0000","last_clean":"2024-03-19T02:26:08.781931+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:26:08.781931+0000","last_undegraded":"2024-03-19T02:26:08.781931+0000","last_fullsized":"2024-03-19T02:26:08.781931+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849105+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:26:08.849105+0000","last_peered":"2024-03-19T02:26:08.849105+0000","last_clean":"2024-03-19T02:26:08.849105+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:26:08.849105+0000","last_undegraded":"2024-03-19T02:26:08.849105+0000","last_fullsized":"2024-03-19T02:26:08.849105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636537+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:26:37.636537+0000","last_peered":"2024-03-19T02:26:37.636537+0000","last_clean":"2024-03-19T02:26:37.636537+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:26:37.636537+0000","last_undegraded":"2024-03-19T02:26:37.636537+0000","last_fullsized":"2024-03-19T02:26:37.636537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850194+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:26:08.850194+0000","last_peered":"2024-03-19T02:26:08.850194+0000","last_clean":"2024-03-19T02:26:08.850194+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:26:08.850194+0000","last_undegraded":"2024-03-19T02:26:08.850194+0000","last_fullsized":"2024-03-19T02:26:08.850194+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849001+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:26:08.849001+0000","last_peered":"2024-03-19T02:26:08.849001+0000","last_clean":"2024-03-19T02:26:08.849001+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:26:08.849001+0000","last_undegraded":"2024-03-19T02:26:08.849001+0000","last_fullsized":"2024-03-19T02:26:08.849001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647373+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:26:37.647373+0000","last_peered":"2024-03-19T02:26:37.647373+0000","last_clean":"2024-03-19T02:26:37.647373+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:26:37.647373+0000","last_undegraded":"2024-03-19T02:26:37.647373+0000","last_fullsized":"2024-03-19T02:26:37.647373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782070+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:26:08.782070+0000","last_peered":"2024-03-19T02:26:08.782070+0000","last_clean":"2024-03-19T02:26:08.782070+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:26:08.782070+0000","last_undegraded":"2024-03-19T02:26:08.782070+0000","last_fullsized":"2024-03-19T02:26:08.782070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:47.754665+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:26:47.754665+0000","last_peered":"2024-03-19T02:26:47.754665+0000","last_clean":"2024-03-19T02:26:47.754665+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:26:47.754665+0000","last_undegraded":"2024-03-19T02:26:47.754665+0000","last_fullsized":"2024-03-19T02:26:47.754665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62024,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":136,"num_write":38,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.628671+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:26:37.628671+0000","last_peered":"2024-03-19T02:26:37.628671+0000","last_clean":"2024-03-19T02:26:37.628671+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:26:37.628671+0000","last_undegraded":"2024-03-19T02:26:37.628671+0000","last_fullsized":"2024-03-19T02:26:37.628671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849227+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:26:08.849227+0000","last_peered":"2024-03-19T02:26:08.849227+0000","last_clean":"2024-03-19T02:26:08.849227+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:26:08.849227+0000","last_undegraded":"2024-03-19T02:26:08.849227+0000","last_fullsized":"2024-03-19T02:26:08.849227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784722+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:26:08.784722+0000","last_peered":"2024-03-19T02:26:08.784722+0000","last_clean":"2024-03-19T02:26:08.784722+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:26:08.784722+0000","last_undegraded":"2024-03-19T02:26:08.784722+0000","last_fullsized":"2024-03-19T02:26:08.784722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.643157+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:26:37.643157+0000","last_peered":"2024-03-19T02:26:37.643157+0000","last_clean":"2024-03-19T02:26:37.643157+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:26:37.643157+0000","last_undegraded":"2024-03-19T02:26:37.643157+0000","last_fullsized":"2024-03-19T02:26:37.643157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849704+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:26:08.849704+0000","last_peered":"2024-03-19T02:26:08.849704+0000","last_clean":"2024-03-19T02:26:08.849704+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:26:08.849704+0000","last_undegraded":"2024-03-19T02:26:08.849704+0000","last_fullsized":"2024-03-19T02:26:08.849704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849370+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:26:08.849370+0000","last_peered":"2024-03-19T02:26:08.849370+0000","last_clean":"2024-03-19T02:26:08.849370+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:26:08.849370+0000","last_undegraded":"2024-03-19T02:26:08.849370+0000","last_fullsized":"2024-03-19T02:26:08.849370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642815+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:26:37.642815+0000","last_peered":"2024-03-19T02:26:37.642815+0000","last_clean":"2024-03-19T02:26:37.642815+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:26:37.642815+0000","last_undegraded":"2024-03-19T02:26:37.642815+0000","last_fullsized":"2024-03-19T02:26:37.642815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.852290+0000","last_change":"2024-03-19T02:26:08.850986+0000","last_active":"2024-03-19T02:26:08.852290+0000","last_peered":"2024-03-19T02:26:08.852290+0000","last_clean":"2024-03-19T02:26:08.852290+0000","last_became_active":"2024-03-19T02:26:08.850619+0000","last_became_peered":"2024-03-19T02:26:08.850619+0000","last_unstale":"2024-03-19T02:26:08.852290+0000","last_undegraded":"2024-03-19T02:26:08.852290+0000","last_fullsized":"2024-03-19T02:26:08.852290+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:15.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:48.759709+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:26:48.759709+0000","last_peered":"2024-03-19T02:26:48.759709+0000","last_clean":"2024-03-19T02:26:48.759709+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:26:48.759709+0000","last_undegraded":"2024-03-19T02:26:48.759709+0000","last_fullsized":"2024-03-19T02:26:48.759709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.633138+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:26:37.633138+0000","last_peered":"2024-03-19T02:26:37.633138+0000","last_clean":"2024-03-19T02:26:37.633138+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:26:37.633138+0000","last_undegraded":"2024-03-19T02:26:37.633138+0000","last_fullsized":"2024-03-19T02:26:37.633138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787187+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:26:08.787187+0000","last_peered":"2024-03-19T02:26:08.787187+0000","last_clean":"2024-03-19T02:26:08.787187+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:26:08.787187+0000","last_undegraded":"2024-03-19T02:26:08.787187+0000","last_fullsized":"2024-03-19T02:26:08.787187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782479+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:26:08.782479+0000","last_peered":"2024-03-19T02:26:08.782479+0000","last_clean":"2024-03-19T02:26:08.782479+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:26:08.782479+0000","last_undegraded":"2024-03-19T02:26:08.782479+0000","last_fullsized":"2024-03-19T02:26:08.782479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630792+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:26:37.630792+0000","last_peered":"2024-03-19T02:26:37.630792+0000","last_clean":"2024-03-19T02:26:37.630792+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:26:37.630792+0000","last_undegraded":"2024-03-19T02:26:37.630792+0000","last_fullsized":"2024-03-19T02:26:37.630792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848652+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:26:08.848652+0000","last_peered":"2024-03-19T02:26:08.848652+0000","last_clean":"2024-03-19T02:26:08.848652+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:26:08.848652+0000","last_undegraded":"2024-03-19T02:26:08.848652+0000","last_fullsized":"2024-03-19T02:26:08.848652+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783432+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:26:08.783432+0000","last_peered":"2024-03-19T02:26:08.783432+0000","last_clean":"2024-03-19T02:26:08.783432+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:26:08.783432+0000","last_undegraded":"2024-03-19T02:26:08.783432+0000","last_fullsized":"2024-03-19T02:26:08.783432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.639006+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:26:37.639006+0000","last_peered":"2024-03-19T02:26:37.639006+0000","last_clean":"2024-03-19T02:26:37.639006+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:26:37.639006+0000","last_undegraded":"2024-03-19T02:26:37.639006+0000","last_fullsized":"2024-03-19T02:26:37.639006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782714+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:26:08.782714+0000","last_peered":"2024-03-19T02:26:08.782714+0000","last_clean":"2024-03-19T02:26:08.782714+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:26:08.782714+0000","last_undegraded":"2024-03-19T02:26:08.782714+0000","last_fullsized":"2024-03-19T02:26:08.782714+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.851073+0000","last_change":"2024-03-19T02:26:08.851073+0000","last_active":"2024-03-19T02:26:08.851073+0000","last_peered":"2024-03-19T02:26:08.851073+0000","last_clean":"2024-03-19T02:26:08.851073+0000","last_became_active":"2024-03-19T02:26:08.850702+0000","last_became_peered":"2024-03-19T02:26:08.850702+0000","last_unstale":"2024-03-19T02:26:08.851073+0000","last_undegraded":"2024-03-19T02:26:08.851073+0000","last_fullsized":"2024-03-19T02:26:08.851073+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:11.668203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642312+0000","last_change":"2024-03-19T02:26:08.788232+0000","last_active":"2024-03-19T02:26:37.642312+0000","last_peered":"2024-03-19T02:26:37.642312+0000","last_clean":"2024-03-19T02:26:37.642312+0000","last_became_active":"2024-03-19T02:26:08.787993+0000","last_became_peered":"2024-03-19T02:26:08.787993+0000","last_unstale":"2024-03-19T02:26:37.642312+0000","last_undegraded":"2024-03-19T02:26:37.642312+0000","last_fullsized":"2024-03-19T02:26:37.642312+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:40.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786844+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:26:08.786844+0000","last_peered":"2024-03-19T02:26:08.786844+0000","last_clean":"2024-03-19T02:26:08.786844+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:26:08.786844+0000","last_undegraded":"2024-03-19T02:26:08.786844+0000","last_fullsized":"2024-03-19T02:26:08.786844+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:45.756761+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:26:45.756761+0000","last_peered":"2024-03-19T02:26:45.756761+0000","last_clean":"2024-03-19T02:26:45.756761+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:26:45.756761+0000","last_undegraded":"2024-03-19T02:26:45.756761+0000","last_fullsized":"2024-03-19T02:26:45.756761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.636395+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:26:37.636395+0000","last_peered":"2024-03-19T02:26:37.636395+0000","last_clean":"2024-03-19T02:26:37.636395+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:26:37.636395+0000","last_undegraded":"2024-03-19T02:26:37.636395+0000","last_fullsized":"2024-03-19T02:26:37.636395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788849+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:26:08.788849+0000","last_peered":"2024-03-19T02:26:08.788849+0000","last_clean":"2024-03-19T02:26:08.788849+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:26:08.788849+0000","last_undegraded":"2024-03-19T02:26:08.788849+0000","last_fullsized":"2024-03-19T02:26:08.788849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630761+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:26:37.630761+0000","last_peered":"2024-03-19T02:26:37.630761+0000","last_clean":"2024-03-19T02:26:37.630761+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:26:37.630761+0000","last_undegraded":"2024-03-19T02:26:37.630761+0000","last_fullsized":"2024-03-19T02:26:37.630761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:49.760332+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:26:49.760332+0000","last_peered":"2024-03-19T02:26:49.760332+0000","last_clean":"2024-03-19T02:26:49.760332+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:26:49.760332+0000","last_undegraded":"2024-03-19T02:26:49.760332+0000","last_fullsized":"2024-03-19T02:26:49.760332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848454+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:26:08.848454+0000","last_peered":"2024-03-19T02:26:08.848454+0000","last_clean":"2024-03-19T02:26:08.848454+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:26:08.848454+0000","last_undegraded":"2024-03-19T02:26:08.848454+0000","last_fullsized":"2024-03-19T02:26:08.848454+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.644175+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:26:37.644175+0000","last_peered":"2024-03-19T02:26:37.644175+0000","last_clean":"2024-03-19T02:26:37.644175+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:26:37.644175+0000","last_undegraded":"2024-03-19T02:26:37.644175+0000","last_fullsized":"2024-03-19T02:26:37.644175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850130+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:26:08.850130+0000","last_peered":"2024-03-19T02:26:08.850130+0000","last_clean":"2024-03-19T02:26:08.850130+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:26:08.850130+0000","last_undegraded":"2024-03-19T02:26:08.850130+0000","last_fullsized":"2024-03-19T02:26:08.850130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":361,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848703+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:26:08.848703+0000","last_peered":"2024-03-19T02:26:08.848703+0000","last_clean":"2024-03-19T02:26:08.848703+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:26:08.848703+0000","last_undegraded":"2024-03-19T02:26:08.848703+0000","last_fullsized":"2024-03-19T02:26:08.848703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848937+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:26:08.848937+0000","last_peered":"2024-03-19T02:26:08.848937+0000","last_clean":"2024-03-19T02:26:08.848937+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:26:08.848937+0000","last_undegraded":"2024-03-19T02:26:08.848937+0000","last_fullsized":"2024-03-19T02:26:08.848937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642438+0000","last_change":"2024-03-19T02:26:08.788444+0000","last_active":"2024-03-19T02:26:37.642438+0000","last_peered":"2024-03-19T02:26:37.642438+0000","last_clean":"2024-03-19T02:26:37.642438+0000","last_became_active":"2024-03-19T02:26:08.788084+0000","last_became_peered":"2024-03-19T02:26:08.788084+0000","last_unstale":"2024-03-19T02:26:37.642438+0000","last_undegraded":"2024-03-19T02:26:37.642438+0000","last_fullsized":"2024-03-19T02:26:37.642438+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:18.879498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.848917+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:26:08.848917+0000","last_peered":"2024-03-19T02:26:08.848917+0000","last_clean":"2024-03-19T02:26:08.848917+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:26:08.848917+0000","last_undegraded":"2024-03-19T02:26:08.848917+0000","last_fullsized":"2024-03-19T02:26:08.848917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849421+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:26:08.849421+0000","last_peered":"2024-03-19T02:26:08.849421+0000","last_clean":"2024-03-19T02:26:08.849421+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:26:08.849421+0000","last_undegraded":"2024-03-19T02:26:08.849421+0000","last_fullsized":"2024-03-19T02:26:08.849421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.783039+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:26:08.783039+0000","last_peered":"2024-03-19T02:26:08.783039+0000","last_clean":"2024-03-19T02:26:08.783039+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:26:08.783039+0000","last_undegraded":"2024-03-19T02:26:08.783039+0000","last_fullsized":"2024-03-19T02:26:08.783039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642676+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:26:37.642676+0000","last_peered":"2024-03-19T02:26:37.642676+0000","last_clean":"2024-03-19T02:26:37.642676+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:26:37.642676+0000","last_undegraded":"2024-03-19T02:26:37.642676+0000","last_fullsized":"2024-03-19T02:26:37.642676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.788585+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:26:08.788585+0000","last_peered":"2024-03-19T02:26:08.788585+0000","last_clean":"2024-03-19T02:26:08.788585+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:26:08.788585+0000","last_undegraded":"2024-03-19T02:26:08.788585+0000","last_fullsized":"2024-03-19T02:26:08.788585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782703+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:26:08.782703+0000","last_peered":"2024-03-19T02:26:08.782703+0000","last_clean":"2024-03-19T02:26:08.782703+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:26:08.782703+0000","last_undegraded":"2024-03-19T02:26:08.782703+0000","last_fullsized":"2024-03-19T02:26:08.782703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641744+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:26:37.641744+0000","last_peered":"2024-03-19T02:26:37.641744+0000","last_clean":"2024-03-19T02:26:37.641744+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:26:37.641744+0000","last_undegraded":"2024-03-19T02:26:37.641744+0000","last_fullsized":"2024-03-19T02:26:37.641744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784197+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:26:08.784197+0000","last_peered":"2024-03-19T02:26:08.784197+0000","last_clean":"2024-03-19T02:26:08.784197+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:26:08.784197+0000","last_undegraded":"2024-03-19T02:26:08.784197+0000","last_fullsized":"2024-03-19T02:26:08.784197+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850898+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:26:08.850898+0000","last_peered":"2024-03-19T02:26:08.850898+0000","last_clean":"2024-03-19T02:26:08.850898+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:26:08.850898+0000","last_undegraded":"2024-03-19T02:26:08.850898+0000","last_fullsized":"2024-03-19T02:26:08.850898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.635563+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:26:37.635563+0000","last_peered":"2024-03-19T02:26:37.635563+0000","last_clean":"2024-03-19T02:26:37.635563+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:26:37.635563+0000","last_undegraded":"2024-03-19T02:26:37.635563+0000","last_fullsized":"2024-03-19T02:26:37.635563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850604+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:26:08.850604+0000","last_peered":"2024-03-19T02:26:08.850604+0000","last_clean":"2024-03-19T02:26:08.850604+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:26:08.850604+0000","last_undegraded":"2024-03-19T02:26:08.850604+0000","last_fullsized":"2024-03-19T02:26:08.850604+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786068+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:26:08.786068+0000","last_peered":"2024-03-19T02:26:08.786068+0000","last_clean":"2024-03-19T02:26:08.786068+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:26:08.786068+0000","last_undegraded":"2024-03-19T02:26:08.786068+0000","last_fullsized":"2024-03-19T02:26:08.786068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.597134+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:26:37.597134+0000","last_peered":"2024-03-19T02:26:37.597134+0000","last_clean":"2024-03-19T02:26:37.597134+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:26:37.597134+0000","last_undegraded":"2024-03-19T02:26:37.597134+0000","last_fullsized":"2024-03-19T02:26:37.597134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782948+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:26:08.782948+0000","last_peered":"2024-03-19T02:26:08.782948+0000","last_clean":"2024-03-19T02:26:08.782948+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:26:08.782948+0000","last_undegraded":"2024-03-19T02:26:08.782948+0000","last_fullsized":"2024-03-19T02:26:08.782948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.786609+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:26:08.786609+0000","last_peered":"2024-03-19T02:26:08.786609+0000","last_clean":"2024-03-19T02:26:08.786609+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:26:08.786609+0000","last_undegraded":"2024-03-19T02:26:08.786609+0000","last_fullsized":"2024-03-19T02:26:08.786609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638015+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:26:37.638015+0000","last_peered":"2024-03-19T02:26:37.638015+0000","last_clean":"2024-03-19T02:26:37.638015+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:26:37.638015+0000","last_undegraded":"2024-03-19T02:26:37.638015+0000","last_fullsized":"2024-03-19T02:26:37.638015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850823+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:26:08.850823+0000","last_peered":"2024-03-19T02:26:08.850823+0000","last_clean":"2024-03-19T02:26:08.850823+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:26:08.850823+0000","last_undegraded":"2024-03-19T02:26:08.850823+0000","last_fullsized":"2024-03-19T02:26:08.850823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.784484+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:26:08.784484+0000","last_peered":"2024-03-19T02:26:08.784484+0000","last_clean":"2024-03-19T02:26:08.784484+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:26:08.784484+0000","last_undegraded":"2024-03-19T02:26:08.784484+0000","last_fullsized":"2024-03-19T02:26:08.784484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.630482+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:26:37.630482+0000","last_peered":"2024-03-19T02:26:37.630482+0000","last_clean":"2024-03-19T02:26:37.630482+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:26:37.630482+0000","last_undegraded":"2024-03-19T02:26:37.630482+0000","last_fullsized":"2024-03-19T02:26:37.630482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.850460+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:26:08.850460+0000","last_peered":"2024-03-19T02:26:08.850460+0000","last_clean":"2024-03-19T02:26:08.850460+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:26:08.850460+0000","last_undegraded":"2024-03-19T02:26:08.850460+0000","last_fullsized":"2024-03-19T02:26:08.850460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.782712+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:26:08.782712+0000","last_peered":"2024-03-19T02:26:08.782712+0000","last_clean":"2024-03-19T02:26:08.782712+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:26:08.782712+0000","last_undegraded":"2024-03-19T02:26:08.782712+0000","last_fullsized":"2024-03-19T02:26:08.782712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645735+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:26:37.645735+0000","last_peered":"2024-03-19T02:26:37.645735+0000","last_clean":"2024-03-19T02:26:37.645735+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:26:37.645735+0000","last_undegraded":"2024-03-19T02:26:37.645735+0000","last_fullsized":"2024-03-19T02:26:37.645735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.785663+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:26:08.785663+0000","last_peered":"2024-03-19T02:26:08.785663+0000","last_clean":"2024-03-19T02:26:08.785663+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:26:08.785663+0000","last_undegraded":"2024-03-19T02:26:08.785663+0000","last_fullsized":"2024-03-19T02:26:08.785663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:48.759638+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:26:48.759638+0000","last_peered":"2024-03-19T02:26:48.759638+0000","last_clean":"2024-03-19T02:26:48.759638+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:26:48.759638+0000","last_undegraded":"2024-03-19T02:26:48.759638+0000","last_fullsized":"2024-03-19T02:26:48.759638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.645144+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:26:37.645144+0000","last_peered":"2024-03-19T02:26:37.645144+0000","last_clean":"2024-03-19T02:26:37.645144+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:26:37.645144+0000","last_undegraded":"2024-03-19T02:26:37.645144+0000","last_fullsized":"2024-03-19T02:26:37.645144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849926+0000","last_change":"2024-03-19T02:26:08.849926+0000","last_active":"2024-03-19T02:26:08.849926+0000","last_peered":"2024-03-19T02:26:08.849926+0000","last_clean":"2024-03-19T02:26:08.849926+0000","last_became_active":"2024-03-19T02:26:08.849548+0000","last_became_peered":"2024-03-19T02:26:08.849548+0000","last_unstale":"2024-03-19T02:26:08.849926+0000","last_undegraded":"2024-03-19T02:26:08.849926+0000","last_fullsized":"2024-03-19T02:26:08.849926+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:44:35.078313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.849115+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:26:08.849115+0000","last_peered":"2024-03-19T02:26:08.849115+0000","last_clean":"2024-03-19T02:26:08.849115+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:26:08.849115+0000","last_undegraded":"2024-03-19T02:26:08.849115+0000","last_fullsized":"2024-03-19T02:26:08.849115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.628214+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:26:37.628214+0000","last_peered":"2024-03-19T02:26:37.628214+0000","last_clean":"2024-03-19T02:26:37.628214+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:26:37.628214+0000","last_undegraded":"2024-03-19T02:26:37.628214+0000","last_fullsized":"2024-03-19T02:26:37.628214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.851525+0000","last_change":"2024-03-19T02:26:08.850968+0000","last_active":"2024-03-19T02:26:08.851525+0000","last_peered":"2024-03-19T02:26:08.851525+0000","last_clean":"2024-03-19T02:26:08.851525+0000","last_became_active":"2024-03-19T02:26:08.850571+0000","last_became_peered":"2024-03-19T02:26:08.850571+0000","last_unstale":"2024-03-19T02:26:08.851525+0000","last_undegraded":"2024-03-19T02:26:08.851525+0000","last_fullsized":"2024-03-19T02:26:08.851525+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:29.532645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:08.787300+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:26:08.787300+0000","last_peered":"2024-03-19T02:26:08.787300+0000","last_clean":"2024-03-19T02:26:08.787300+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:26:08.787300+0000","last_undegraded":"2024-03-19T02:26:08.787300+0000","last_fullsized":"2024-03-19T02:26:08.787300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634016+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:26:37.634016+0000","last_peered":"2024-03-19T02:26:37.634016+0000","last_clean":"2024-03-19T02:26:37.634016+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:26:37.634016+0000","last_undegraded":"2024-03-19T02:26:37.634016+0000","last_fullsized":"2024-03-19T02:26:37.634016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641108+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:26:37.641108+0000","last_peered":"2024-03-19T02:26:37.641108+0000","last_clean":"2024-03-19T02:26:37.641108+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:26:37.641108+0000","last_undegraded":"2024-03-19T02:26:37.641108+0000","last_fullsized":"2024-03-19T02:26:37.641108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.625170+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:26:37.625170+0000","last_peered":"2024-03-19T02:26:37.625170+0000","last_clean":"2024-03-19T02:26:37.625170+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:26:37.625170+0000","last_undegraded":"2024-03-19T02:26:37.625170+0000","last_fullsized":"2024-03-19T02:26:37.625170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.625408+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:26:37.625408+0000","last_peered":"2024-03-19T02:26:37.625408+0000","last_clean":"2024-03-19T02:26:37.625408+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:26:37.625408+0000","last_undegraded":"2024-03-19T02:26:37.625408+0000","last_fullsized":"2024-03-19T02:26:37.625408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.634739+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:26:37.634739+0000","last_peered":"2024-03-19T02:26:37.634739+0000","last_clean":"2024-03-19T02:26:37.634739+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:26:37.634739+0000","last_undegraded":"2024-03-19T02:26:37.634739+0000","last_fullsized":"2024-03-19T02:26:37.634739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.641270+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:26:37.641270+0000","last_peered":"2024-03-19T02:26:37.641270+0000","last_clean":"2024-03-19T02:26:37.641270+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:26:37.641270+0000","last_undegraded":"2024-03-19T02:26:37.641270+0000","last_fullsized":"2024-03-19T02:26:37.641270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638949+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:26:37.638949+0000","last_peered":"2024-03-19T02:26:37.638949+0000","last_clean":"2024-03-19T02:26:37.638949+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:26:37.638949+0000","last_undegraded":"2024-03-19T02:26:37.638949+0000","last_fullsized":"2024-03-19T02:26:37.638949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.617197+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:26:37.617197+0000","last_peered":"2024-03-19T02:26:37.617197+0000","last_clean":"2024-03-19T02:26:37.617197+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:26:37.617197+0000","last_undegraded":"2024-03-19T02:26:37.617197+0000","last_fullsized":"2024-03-19T02:26:37.617197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638115+0000","last_change":"2024-03-19T02:26:08.790951+0000","last_active":"2024-03-19T02:26:37.638115+0000","last_peered":"2024-03-19T02:26:37.638115+0000","last_clean":"2024-03-19T02:26:37.638115+0000","last_became_active":"2024-03-19T02:26:08.790587+0000","last_became_peered":"2024-03-19T02:26:08.790587+0000","last_unstale":"2024-03-19T02:26:37.638115+0000","last_undegraded":"2024-03-19T02:26:37.638115+0000","last_fullsized":"2024-03-19T02:26:37.638115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:11.410269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.629113+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:26:37.629113+0000","last_peered":"2024-03-19T02:26:37.629113+0000","last_clean":"2024-03-19T02:26:37.629113+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:26:37.629113+0000","last_undegraded":"2024-03-19T02:26:37.629113+0000","last_fullsized":"2024-03-19T02:26:37.629113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.642650+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:26:37.642650+0000","last_peered":"2024-03-19T02:26:37.642650+0000","last_clean":"2024-03-19T02:26:37.642650+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:26:37.642650+0000","last_undegraded":"2024-03-19T02:26:37.642650+0000","last_fullsized":"2024-03-19T02:26:37.642650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.647350+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:26:37.647350+0000","last_peered":"2024-03-19T02:26:37.647350+0000","last_clean":"2024-03-19T02:26:37.647350+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:26:37.647350+0000","last_undegraded":"2024-03-19T02:26:37.647350+0000","last_fullsized":"2024-03-19T02:26:37.647350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.644245+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:26:37.644245+0000","last_peered":"2024-03-19T02:26:37.644245+0000","last_clean":"2024-03-19T02:26:37.644245+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:26:37.644245+0000","last_undegraded":"2024-03-19T02:26:37.644245+0000","last_fullsized":"2024-03-19T02:26:37.644245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-19T02:26:37.638496+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:26:37.638496+0000","last_peered":"2024-03-19T02:26:37.638496+0000","last_clean":"2024-03-19T02:26:37.638496+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:26:37.638496+0000","last_undegraded":"2024-03-19T02:26:37.638496+0000","last_fullsized":"2024-03-19T02:26:37.638496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68446,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":766,"num_read_kb":458,"num_write":243,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":504492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1153332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":508518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":504800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1100906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":499926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":562980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":501382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1126400,"data_stored":563200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":499634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":504000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1096488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:26:53.461 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-19T02:26:53.461 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T02:26:53.461 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-19T02:26:53.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-19T02:26:53.709 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:26:53.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:53 smithi116 ceph-mon[32263]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:53 smithi154 ceph-mon[35693]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:53 smithi162 ceph-mon[35761]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:26:55.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:54 smithi116 ceph-mon[32263]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:55.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:54 smithi154 ceph-mon[35693]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:54 smithi162 ceph-mon[35761]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 -- 172.21.15.116:0/63980959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0390101860 msgr2=0x7f0390106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 --2- 172.21.15.116:0/63980959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0390101860 0x7f0390106fa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03800099c0 tx=0x7f0380009cd0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 -- 172.21.15.116:0/63980959 shutdown_connections 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 --2- 172.21.15.116:0/63980959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03901076d0 0x7f039010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 --2- 172.21.15.116:0/63980959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f0390101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 --2- 172.21.15.116:0/63980959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0390101860 0x7f0390106fa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.533+0000 7f03981c8700 1 -- 172.21.15.116:0/63980959 >> 172.21.15.116:0/63980959 conn(0x7f0390076550 msgr2=0x7f0390076960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- 172.21.15.116:0/63980959 shutdown_connections 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- 172.21.15.116:0/63980959 wait complete. 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 Processor -- start 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- start start 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 0x7f039019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03901076d0 0x7f0390198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03901124d0 con 0x7f03901076d0 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0390112350 con 0x7f0390101860 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.534+0000 7f03981c8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0390112650 con 0x7f0390100eb0 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:60528/0 (socket says 172.21.15.116:60528) 2024-03-19T02:26:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 -- 172.21.15.116:0/2827533786 learned_addr learned my addr 172.21.15.116:0/2827533786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395763700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 0x7f039019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0396765700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03901076d0 0x7f0390198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 -- 172.21.15.116:0/2827533786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 msgr2=0x7f039019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 0x7f039019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 -- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03901076d0 msgr2=0x7f0390198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03901076d0 0x7f0390198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 -- 172.21.15.116:0/2827533786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0390199070 con 0x7f0390100eb0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395763700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 0x7f039019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0395f64700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f038c00d980 tx=0x7f038c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c016070 con 0x7f0390100eb0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0390199360 con 0x7f0390100eb0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f038c004830 con 0x7f0390100eb0 2024-03-19T02:26:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c005530 con 0x7f0390100eb0 2024-03-19T02:26:55.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.535+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0390101d60 con 0x7f0390100eb0 2024-03-19T02:26:55.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.536+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039010a910 con 0x7f0390100eb0 2024-03-19T02:26:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.538+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f038c0049e0 con 0x7f0390100eb0 2024-03-19T02:26:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.538+0000 7f0386ffd700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 0x7f037c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.538+0000 7f0395763700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 0x7f037c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.538+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f038c0cefb0 con 0x7f0390100eb0 2024-03-19T02:26:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.539+0000 7f0395763700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 0x7f037c0a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0380009990 tx=0x7f0380005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.546+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f038c01f020 con 0x7f0390100eb0 2024-03-19T02:26:55.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.794+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0390199e40 con 0x7f0390100eb0 2024-03-19T02:26:55.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.795+0000 7f0386ffd700 1 -- 172.21.15.116:0/2827533786 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f038c09bcf0 con 0x7f0390100eb0 2024-03-19T02:26:55.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:55.796 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T02:26:55.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 msgr2=0x7f037c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:55.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 0x7f037c0a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0380009990 tx=0x7f0380005fb0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 msgr2=0x7f039019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:55.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f038c00d980 tx=0x7f038c00dc90 comp rx=0 tx=0).stop 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 shutdown_connections 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03901076d0 0x7f0390198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f037c0a2890 0x7f037c0a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0390100eb0 0x7f039019e6b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 --2- 172.21.15.116:0/2827533786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0390101860 0x7f039019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 >> 172.21.15.116:0/2827533786 conn(0x7f0390076550 msgr2=0x7f0390100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 shutdown_connections 2024-03-19T02:26:55.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:55.797+0000 7f03981c8700 1 -- 172.21.15.116:0/2827533786 wait complete. 2024-03-19T02:26:56.613 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-19T02:26:56.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:26:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- 172.21.15.116:0/2291196003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df01022f0 msgr2=0x7f1df0108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- 172.21.15.116:0/2291196003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df01022f0 0x7f1df0108b80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009ca0 tx=0x7f1de802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- 172.21.15.116:0/2291196003 shutdown_connections 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- 172.21.15.116:0/2291196003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df01022f0 0x7f1df0108b80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- 172.21.15.116:0/2291196003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0101930 0x7f1df0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- 172.21.15.116:0/2291196003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df0100740 0x7f1df0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- 172.21.15.116:0/2291196003 >> 172.21.15.116:0/2291196003 conn(0x7f1df00fbcb0 msgr2=0x7f1df00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- 172.21.15.116:0/2291196003 shutdown_connections 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- 172.21.15.116:0/2291196003 wait complete. 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 Processor -- start 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- start start 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0100740 0x7f1df00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:56.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df01022f0 0x7f1df019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010c270 con 0x7f1df0101930 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010c0f0 con 0x7f1df01022f0 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.723+0000 7f1df6c14640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010c3f0 con 0x7f1df0100740 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44816/0 (socket says 172.21.15.116:44816) 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 -- 172.21.15.116:0/3178947546 learned_addr learned my addr 172.21.15.116:0/3178947546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1df49b0640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0100740 0x7f1df00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1df51b1640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df01022f0 0x7f1df019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 -- 172.21.15.116:0/3178947546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0100740 msgr2=0x7f1df00694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0100740 0x7f1df00694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 -- 172.21.15.116:0/3178947546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df01022f0 msgr2=0x7f1df019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df01022f0 0x7f1df019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 -- 172.21.15.116:0/3178947546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df019f6b0 con 0x7f1df0101930 2024-03-19T02:26:56.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1deffff640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1de400e9f0 tx=0x7f1de400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de400cd80 con 0x7f1df0101930 2024-03-19T02:26:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1de40054d0 con 0x7f1df0101930 2024-03-19T02:26:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df019f940 con 0x7f1df0101930 2024-03-19T02:26:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4010490 con 0x7f1df0101930 2024-03-19T02:26:56.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.724+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df019fe90 con 0x7f1df0101930 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.725+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df0107a60 con 0x7f1df0101930 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.726+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1de4005020 con 0x7f1df0101930 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.727+0000 7f1dedffb640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 0x7f1dd80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.727+0000 7f1df49b0640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 0x7f1dd80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.727+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f1de40cf090 con 0x7f1df0101930 2024-03-19T02:26:56.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.728+0000 7f1df49b0640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 0x7f1dd80a4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1df0101790 tx=0x7f1de0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:56.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.732+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1de409bdd0 con 0x7f1df0101930 2024-03-19T02:26:56.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.909+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1df0062420 con 0x7f1df0101930 2024-03-19T02:26:56.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.911+0000 7f1dedffb640 1 -- 172.21.15.116:0/3178947546 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24665 (secure 0 0 0) 0x7f1de409b6d0 con 0x7f1df0101930 2024-03-19T02:26:56.911 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:56.912 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:26:08.779202+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:26:56.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 msgr2=0x7f1dd80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:56.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 0x7f1dd80a4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1df0101790 tx=0x7f1de0005f90 comp rx=0 tx=0).stop 2024-03-19T02:26:56.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 msgr2=0x7f1df019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1de400e9f0 tx=0x7f1de400edf0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 shutdown_connections 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1dd80a2880 0x7f1dd80a4d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df01022f0 0x7f1df019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df0101930 0x7f1df019c780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 --2- 172.21.15.116:0/3178947546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df0100740 0x7f1df00694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 >> 172.21.15.116:0/3178947546 conn(0x7f1df00fbcb0 msgr2=0x7f1df0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 shutdown_connections 2024-03-19T02:26:56.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:56.913+0000 7f1df6c14640 1 -- 172.21.15.116:0/3178947546 wait complete. 2024-03-19T02:26:56.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:56 smithi116 ceph-mon[32263]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:56.923 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:56 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2827533786' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:26:56.925 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:26:56.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:26:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:56 smithi154 ceph-mon[35693]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:57.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:56 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2827533786' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 -- 172.21.15.116:0/295895185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac1022f0 msgr2=0x7fe1ac108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/295895185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac108b80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009900 tx=0x7fe1a802bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 -- 172.21.15.116:0/295895185 shutdown_connections 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/295895185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac108b80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/295895185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac101930 0x7fe1ac101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/295895185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 -- 172.21.15.116:0/295895185 >> 172.21.15.116:0/295895185 conn(0x7fe1ac0fbcd0 msgr2=0x7fe1ac0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 -- 172.21.15.116:0/295895185 shutdown_connections 2024-03-19T02:26:57.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.030+0000 7fe1b45cd640 1 -- 172.21.15.116:0/295895185 wait complete. 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 Processor -- start 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 -- start start 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac10d380 con 0x7fe1ac101930 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac10d200 con 0x7fe1ac100740 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b45cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac10d500 con 0x7fe1ac1022f0 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44832/0 (socket says 172.21.15.116:44832) 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 -- 172.21.15.116:0/3908113919 learned_addr learned my addr 172.21.15.116:0/3908113919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b2369640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b2b6a640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 -- 172.21.15.116:0/3908113919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac1022f0 msgr2=0x7fe1ac19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 -- 172.21.15.116:0/3908113919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 msgr2=0x7fe1ac197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 -- 172.21.15.116:0/3908113919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac19f620 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b2369640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1b1b68640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00e9f0 tx=0x7fe19c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.031+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c00cd80 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.032+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe19c0054d0 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.032+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ac19f910 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.032+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c010490 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.032+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ac19fe10 con 0x7fe1ac101930 2024-03-19T02:26:57.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.032+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1ac107b80 con 0x7fe1ac101930 2024-03-19T02:26:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.034+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe19c005000 con 0x7fe1ac101930 2024-03-19T02:26:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.034+0000 7fe1a37fe640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.034+0000 7fe1b2369640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.034+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe19c0cf320 con 0x7fe1ac101930 2024-03-19T02:26:57.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.035+0000 7fe1b2369640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 0x7fe1980a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac101790 tx=0x7fe1a4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.038+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe19c053740 con 0x7fe1ac101930 2024-03-19T02:26:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:56 smithi162 ceph-mon[35761]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:26:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:56 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2827533786' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:26:57.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.210+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1ac02d400 con 0x7fe1ac101930 2024-03-19T02:26:57.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.210+0000 7fe1a37fe640 1 -- 172.21.15.116:0/3908113919 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fe19c0052f0 con 0x7fe1ac101930 2024-03-19T02:26:57.212 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:26:57.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 msgr2=0x7fe1980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 0x7fe1980a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac101790 tx=0x7fe1a4009500 comp rx=0 tx=0).stop 2024-03-19T02:26:57.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 msgr2=0x7fe1ac197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe19c00e9f0 tx=0x7fe19c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 shutdown_connections 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1ac1022f0 0x7fe1ac19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1ac101930 0x7fe1ac197f40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.212+0000 7fe1b45cd640 1 --2- 172.21.15.116:0/3908113919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1ac100740 0x7fe1ac197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.213+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 >> 172.21.15.116:0/3908113919 conn(0x7fe1ac0fbcd0 msgr2=0x7fe1ac104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.213+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 shutdown_connections 2024-03-19T02:26:57.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.213+0000 7fe1b45cd640 1 -- 172.21.15.116:0/3908113919 wait complete. 2024-03-19T02:26:57.224 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 -- 172.21.15.116:0/1050894556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa40101930 msgr2=0x7faa40101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 --2- 172.21.15.116:0/1050894556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa40101930 0x7faa40101db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faa34009900 tx=0x7faa3402bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 -- 172.21.15.116:0/1050894556 shutdown_connections 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 --2- 172.21.15.116:0/1050894556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa401022f0 0x7faa40108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 --2- 172.21.15.116:0/1050894556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa40101930 0x7faa40101db0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 --2- 172.21.15.116:0/1050894556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40100740 0x7faa40100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 -- 172.21.15.116:0/1050894556 >> 172.21.15.116:0/1050894556 conn(0x7faa400fbcd0 msgr2=0x7faa400fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 -- 172.21.15.116:0/1050894556 shutdown_connections 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.328+0000 7faa46913640 1 -- 172.21.15.116:0/1050894556 wait complete. 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 Processor -- start 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 -- start start 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40100740 0x7faa401969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40101930 0x7faa40196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa4010c270 con 0x7faa401022f0 2024-03-19T02:26:57.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa4010c0f0 con 0x7faa40100740 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa46913640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa4010c3f0 con 0x7faa40101930 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40100740 0x7faa401969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44848/0 (socket says 172.21.15.116:44848) 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 -- 172.21.15.116:0/2149441893 learned_addr learned my addr 172.21.15.116:0/2149441893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa3f7fe640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40101930 0x7faa40196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 -- 172.21.15.116:0/2149441893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40101930 msgr2=0x7faa40196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40101930 0x7faa40196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 -- 172.21.15.116:0/2149441893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40100740 msgr2=0x7faa401969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40100740 0x7faa401969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.329+0000 7faa44eb0640 1 -- 172.21.15.116:0/2149441893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4019e5e0 con 0x7faa401022f0 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa44eb0640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7faa3800eb10 tx=0x7faa3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa38004b20 con 0x7faa401022f0 2024-03-19T02:26:57.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa38004cc0 con 0x7faa401022f0 2024-03-19T02:26:57.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa38010690 con 0x7faa401022f0 2024-03-19T02:26:57.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4019e8d0 con 0x7faa401022f0 2024-03-19T02:26:57.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.330+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4019eda0 con 0x7faa401022f0 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.331+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa40107bc0 con 0x7faa401022f0 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.332+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7faa38005020 con 0x7faa401022f0 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.333+0000 7faa3d7fa640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 0x7faa280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.333+0000 7faa3ffff640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 0x7faa280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.333+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7faa380cf350 con 0x7faa401022f0 2024-03-19T02:26:57.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.333+0000 7faa3ffff640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 0x7faa280a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa40101790 tx=0x7faa30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.337+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa3809c090 con 0x7faa401022f0 2024-03-19T02:26:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.512+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faa40062420 con 0x7faa401022f0 2024-03-19T02:26:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.512+0000 7faa3d7fa640 1 -- 172.21.15.116:0/2149441893 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7faa3809b990 con 0x7faa401022f0 2024-03-19T02:26:57.513 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.514+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 msgr2=0x7faa280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.514+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 0x7faa280a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa40101790 tx=0x7faa30005f90 comp rx=0 tx=0).stop 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.514+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 msgr2=0x7faa4019df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.514+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7faa3800eb10 tx=0x7faa3800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 shutdown_connections 2024-03-19T02:26:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7faa280a2800 0x7faa280a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa401022f0 0x7faa4019df70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa40101930 0x7faa40196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 --2- 172.21.15.116:0/2149441893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40100740 0x7faa401969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 >> 172.21.15.116:0/2149441893 conn(0x7faa400fbcd0 msgr2=0x7faa40104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 shutdown_connections 2024-03-19T02:26:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.515+0000 7faa46913640 1 -- 172.21.15.116:0/2149441893 wait complete. 2024-03-19T02:26:57.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:26:57.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 -- 172.21.15.116:0/199304683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 msgr2=0x7fe1c4104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/199304683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4104730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009900 tx=0x7fe1c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 -- 172.21.15.116:0/199304683 shutdown_connections 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/199304683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c4109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/199304683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 0x7fe1c4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/199304683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4104730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 -- 172.21.15.116:0/199304683 >> 172.21.15.116:0/199304683 conn(0x7fe1c40fbcb0 msgr2=0x7fe1c40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 -- 172.21.15.116:0/199304683 shutdown_connections 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.630+0000 7fe1cbd87640 1 -- 172.21.15.116:0/199304683 wait complete. 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 Processor -- start 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 -- start start 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 0x7fe1c4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c410d340 con 0x7fe1c4102340 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c410d1c0 con 0x7fe1c41075a0 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1cbd87640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c410d4c0 con 0x7fe1c4104c70 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50268/0 (socket says 172.21.15.116:50268) 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 -- 172.21.15.116:0/1507497700 learned_addr learned my addr 172.21.15.116:0/1507497700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1ca324640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9322640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 0x7fe1c4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 -- 172.21.15.116:0/1507497700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 msgr2=0x7fe1c4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 0x7fe1c4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 -- 172.21.15.116:0/1507497700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 msgr2=0x7fe1c419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 -- 172.21.15.116:0/1507497700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c419f6f0 con 0x7fe1c4102340 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1ca324640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1c9b23640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0006010 tx=0x7fe1c0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.631+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0043020 con 0x7fe1c4102340 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.632+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1c0004440 con 0x7fe1c4102340 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.632+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0003bf0 con 0x7fe1c4102340 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.632+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c419f980 con 0x7fe1c4102340 2024-03-19T02:26:57.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.632+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c419fe80 con 0x7fe1c4102340 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.633+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c40fd7a0 con 0x7fe1c4102340 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.634+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe1c0046020 con 0x7fe1c4102340 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.634+0000 7fe1baffd640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 0x7fe1b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.634+0000 7fe1c9322640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 0x7fe1b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.634+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe1c00f10d0 con 0x7fe1c4102340 2024-03-19T02:26:57.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.635+0000 7fe1c9322640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 0x7fe1b00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4198fe0 tx=0x7fe1b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.639+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1c004d020 con 0x7fe1c4102340 2024-03-19T02:26:57.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.817+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe1c4062420 con 0x7fe1c4102340 2024-03-19T02:26:57.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.817+0000 7fe1baffd640 1 -- 172.21.15.116:0/1507497700 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fe1c00bde10 con 0x7fe1c4102340 2024-03-19T02:26:57.818 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 msgr2=0x7fe1b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 0x7fe1b00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4198fe0 tx=0x7fe1b400b410 comp rx=0 tx=0).stop 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 msgr2=0x7fe1c4197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0006010 tx=0x7fe1c0004040 comp rx=0 tx=0).stop 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 shutdown_connections 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fe1b00a2800 0x7fe1b00a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1c41075a0 0x7fe1c419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1c4104c70 0x7fe1c4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 --2- 172.21.15.116:0/1507497700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1c4102340 0x7fe1c4197ad0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 >> 172.21.15.116:0/1507497700 conn(0x7fe1c40fbcb0 msgr2=0x7fe1c41041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 shutdown_connections 2024-03-19T02:26:57.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.819+0000 7fe1cbd87640 1 -- 172.21.15.116:0/1507497700 wait complete. 2024-03-19T02:26:57.830 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:57 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3178947546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:26:57.830 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:57 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3908113919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:26:57.830 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:57 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2149441893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:26:57.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:26:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.934+0000 7f27cccff640 1 -- 172.21.15.116:0/1491006284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c8101910 msgr2=0x7f27c8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.934+0000 7f27cccff640 1 --2- 172.21.15.116:0/1491006284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c8101910 0x7f27c8101d90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f27b8009900 tx=0x7f27b802bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- 172.21.15.116:0/1491006284 shutdown_connections 2024-03-19T02:26:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- 172.21.15.116:0/1491006284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c81022d0 0x7f27c8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- 172.21.15.116:0/1491006284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c8101910 0x7f27c8101d90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- 172.21.15.116:0/1491006284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8100720 0x7f27c8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- 172.21.15.116:0/1491006284 >> 172.21.15.116:0/1491006284 conn(0x7f27c80fbcb0 msgr2=0x7f27c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- 172.21.15.116:0/1491006284 shutdown_connections 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- 172.21.15.116:0/1491006284 wait complete. 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 Processor -- start 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- start start 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c8100720 0x7f27c8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8101910 0x7f27c8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.935+0000 7f27cccff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810c250 con 0x7f27c81022d0 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27cccff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810c0d0 con 0x7f27c8100720 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27cccff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27c810c3d0 con 0x7f27c8101910 2024-03-19T02:26:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50286/0 (socket says 172.21.15.116:50286) 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 -- 172.21.15.116:0/226406137 learned_addr learned my addr 172.21.15.116:0/226406137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c659c640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c8100720 0x7f27c8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c5d9b640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8101910 0x7f27c8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 -- 172.21.15.116:0/226406137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8101910 msgr2=0x7f27c8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8101910 0x7f27c8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 -- 172.21.15.116:0/226406137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c8100720 msgr2=0x7f27c8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c8100720 0x7f27c8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 -- 172.21.15.116:0/226406137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c819f720 con 0x7f27c81022d0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27c6d9d640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009fd0 tx=0x7f27c000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0017070 con 0x7f27c81022d0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27c0003e20 con 0x7f27c81022d0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c819fa10 con 0x7f27c81022d0 2024-03-19T02:26:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0010460 con 0x7f27c81022d0 2024-03-19T02:26:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.936+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c819fee0 con 0x7f27c81022d0 2024-03-19T02:26:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.937+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c81a0110 con 0x7f27c81022d0 2024-03-19T02:26:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.938+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f27c0003970 con 0x7f27c81022d0 2024-03-19T02:26:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.939+0000 7f27b77fe640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 0x7f27b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.939+0000 7f27c659c640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 0x7f27b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.939+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f27c00ceb00 con 0x7f27c81022d0 2024-03-19T02:26:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.939+0000 7f27c659c640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 0x7f27b00a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27c8101770 tx=0x7f27bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:57.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:57.943+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27c009b1f0 con 0x7f27c81022d0 2024-03-19T02:26:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:57 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3178947546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:26:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:57 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3908113919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:26:58.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:57 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2149441893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:26:58.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.115+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f27c81a04a0 con 0x7f27c81022d0 2024-03-19T02:26:58.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.116+0000 7f27b77fe640 1 -- 172.21.15.116:0/226406137 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f27c00bb0c0 con 0x7f27c81022d0 2024-03-19T02:26:58.116 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 msgr2=0x7f27b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 0x7f27b00a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27c8101770 tx=0x7f27bc005f90 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 msgr2=0x7f27c819f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009fd0 tx=0x7f27c000b260 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 shutdown_connections 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f27b00a2800 0x7f27b00a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c81022d0 0x7f27c819f0b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27c8101910 0x7f27c8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 --2- 172.21.15.116:0/226406137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c8100720 0x7f27c8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 >> 172.21.15.116:0/226406137 conn(0x7f27c80fbcb0 msgr2=0x7f27c8068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 shutdown_connections 2024-03-19T02:26:58.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.118+0000 7f27cccff640 1 -- 172.21.15.116:0/226406137 wait complete. 2024-03-19T02:26:58.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:26:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:57 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3178947546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:26:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:57 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3908113919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:26:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:57 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2149441893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:26:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 -- 172.21.15.116:0/415332414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 msgr2=0x7f96f0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 --2- 172.21.15.116:0/415332414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0101db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009900 tx=0x7f96e402bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 -- 172.21.15.116:0/415332414 shutdown_connections 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 --2- 172.21.15.116:0/415332414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 --2- 172.21.15.116:0/415332414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0101db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 --2- 172.21.15.116:0/415332414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.233+0000 7f96f5b59640 1 -- 172.21.15.116:0/415332414 >> 172.21.15.116:0/415332414 conn(0x7f96f00fbcb0 msgr2=0x7f96f00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- 172.21.15.116:0/415332414 shutdown_connections 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- 172.21.15.116:0/415332414 wait complete. 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 Processor -- start 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- start start 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96f010c250 con 0x7f96f0100740 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96f010c0d0 con 0x7f96f0101930 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.234+0000 7f96f5b59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96f010c3d0 con 0x7f96f01022f0 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50306/0 (socket says 172.21.15.116:50306) 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 -- 172.21.15.116:0/2752515945 learned_addr learned my addr 172.21.15.116:0/2752515945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96effff640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96eeffd640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 -- 172.21.15.116:0/2752515945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 msgr2=0x7f96f0103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 -- 172.21.15.116:0/2752515945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 msgr2=0x7f96f0102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 -- 172.21.15.116:0/2752515945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f0103ae0 con 0x7f96f0100740 2024-03-19T02:26:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96effff640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ef7fe640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9c0 tx=0x7f96e000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e000cd80 con 0x7f96f0100740 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e00108e0 con 0x7f96f0100740 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f01a1c20 con 0x7f96f0100740 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e00188e0 con 0x7f96f0100740 2024-03-19T02:26:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.235+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f01a2140 con 0x7f96f0100740 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.237+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f01a2310 con 0x7f96f0100740 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.237+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f96e0010a90 con 0x7f96f0100740 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.238+0000 7f96ecff9640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 0x7f96d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.238+0000 7f96eeffd640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 0x7f96d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.238+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f96e00cf210 con 0x7f96f0100740 2024-03-19T02:26:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.238+0000 7f96eeffd640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 0x7f96d80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009230 tx=0x7f96e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.242+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96e0053740 con 0x7f96f0100740 2024-03-19T02:26:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.482+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96f01a24e0 con 0x7f96f0100740 2024-03-19T02:26:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.483+0000 7f96ecff9640 1 -- 172.21.15.116:0/2752515945 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f96e009bf50 con 0x7f96f0100740 2024-03-19T02:26:58.484 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:58.484 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:26:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 msgr2=0x7f96d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 0x7f96d80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009230 tx=0x7f96e4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 msgr2=0x7f96f01046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9c0 tx=0x7f96e000edc0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 shutdown_connections 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f96d80a2800 0x7f96d80a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96f01022f0 0x7f96f0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f0101930 0x7f96f0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 --2- 172.21.15.116:0/2752515945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96f0100740 0x7f96f01046d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 >> 172.21.15.116:0/2752515945 conn(0x7f96f00fbcb0 msgr2=0x7f96f00689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 shutdown_connections 2024-03-19T02:26:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.485+0000 7f96f5b59640 1 -- 172.21.15.116:0/2752515945 wait complete. 2024-03-19T02:26:58.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 -- 172.21.15.116:0/3909840167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7700100740 msgr2=0x7f7700100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 --2- 172.21.15.116:0/3909840167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7700100740 0x7f7700100b60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009900 tx=0x7f76f002bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 -- 172.21.15.116:0/3909840167 shutdown_connections 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 --2- 172.21.15.116:0/3909840167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77001022f0 0x7f7700108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 --2- 172.21.15.116:0/3909840167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 --2- 172.21.15.116:0/3909840167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7700100740 0x7f7700100b60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.603+0000 7f7704a27640 1 -- 172.21.15.116:0/3909840167 >> 172.21.15.116:0/3909840167 conn(0x7f77000fbcb0 msgr2=0x7f77000fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- 172.21.15.116:0/3909840167 shutdown_connections 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- 172.21.15.116:0/3909840167 wait complete. 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 Processor -- start 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- start start 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7700100740 0x7f7700197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c290 con 0x7f77001022f0 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c110 con 0x7f7700100740 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.604+0000 7f7704a27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c410 con 0x7f7700101930 2024-03-19T02:26:58.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50332/0 (socket says 172.21.15.116:50332) 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 -- 172.21.15.116:0/2094577160 learned_addr learned my addr 172.21.15.116:0/2094577160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fdd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fe59c640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7700100740 0x7f7700197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 -- 172.21.15.116:0/2094577160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 msgr2=0x7f7700198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 -- 172.21.15.116:0/2094577160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7700100740 msgr2=0x7f7700197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7700100740 0x7f7700197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 -- 172.21.15.116:0/2094577160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770019f700 con 0x7f77001022f0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fdd9b640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76fed9d640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f76f800efd0 tx=0x7f76f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f8003b70 con 0x7f77001022f0 2024-03-19T02:26:58.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76f8003d10 con 0x7f77001022f0 2024-03-19T02:26:58.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f8010980 con 0x7f77001022f0 2024-03-19T02:26:58.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770019f9f0 con 0x7f77001022f0 2024-03-19T02:26:58.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.605+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770019fef0 con 0x7f77001022f0 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.606+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77001a01b0 con 0x7f77001022f0 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.607+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f76f8004020 con 0x7f77001022f0 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.608+0000 7f76ef7fe640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 0x7f76e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.608+0000 7f76fe59c640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 0x7f76e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.608+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f76f80cf220 con 0x7f77001022f0 2024-03-19T02:26:58.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.609+0000 7f76fe59c640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 0x7f76e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76f002c240 tx=0x7f76f002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.613+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76f809bf60 con 0x7f77001022f0 2024-03-19T02:26:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.798+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77001a0380 con 0x7f77001022f0 2024-03-19T02:26:58.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.799+0000 7f76ef7fe640 1 -- 172.21.15.116:0/2094577160 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24665 (secure 0 0 0) 0x7f76f809b860 con 0x7f77001022f0 2024-03-19T02:26:58.800 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:26:58.800 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:26:08.779202+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:26:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 msgr2=0x7f76e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 0x7f76e80a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76f002c240 tx=0x7f76f002c7e0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 msgr2=0x7f770019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f76f800efd0 tx=0x7f76f800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 shutdown_connections 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f76e80a2800 0x7f76e80a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77001022f0 0x7f770019f090 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7700101930 0x7f7700198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 --2- 172.21.15.116:0/2094577160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7700100740 0x7f7700197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 >> 172.21.15.116:0/2094577160 conn(0x7f77000fbcb0 msgr2=0x7f7700068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 shutdown_connections 2024-03-19T02:26:58.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.802+0000 7f7704a27640 1 -- 172.21.15.116:0/2094577160 wait complete. 2024-03-19T02:26:58.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:26:58.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- 172.21.15.116:0/3204151114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908100720 msgr2=0x7f5908100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 --2- 172.21.15.116:0/3204151114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908100720 0x7f5908100b40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009900 tx=0x7f58f802bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- 172.21.15.116:0/3204151114 shutdown_connections 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 --2- 172.21.15.116:0/3204151114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 0x7f5908108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 --2- 172.21.15.116:0/3204151114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908101910 0x7f5908101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 --2- 172.21.15.116:0/3204151114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908100720 0x7f5908100b40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- 172.21.15.116:0/3204151114 >> 172.21.15.116:0/3204151114 conn(0x7f59080fbcb0 msgr2=0x7f59080fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- 172.21.15.116:0/3204151114 shutdown_connections 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- 172.21.15.116:0/3204151114 wait complete. 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 Processor -- start 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.922+0000 7f590e7bc640 1 -- start start 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 0x7f59081969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 0x7f590819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f590810c240 con 0x7f5908101910 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f590810c0c0 con 0x7f59081022d0 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590e7bc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f590810c3c0 con 0x7f5908100720 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f59077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 0x7f59081969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f59077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50354/0 (socket says 172.21.15.116:50354) 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f59077fe640 1 -- 172.21.15.116:0/569933412 learned_addr learned my addr 172.21.15.116:0/569933412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f590cd59640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 0x7f590819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 -- 172.21.15.116:0/569933412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 msgr2=0x7f590819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 0x7f590819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 -- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 msgr2=0x7f5908196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 -- 172.21.15.116:0/569933412 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590819e600 con 0x7f5908100720 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f59077fe640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.923+0000 7f5907fff640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 0x7f59081969e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036040 tx=0x7f58f8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.924+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8043020 con 0x7f5908100720 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.924+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590819e890 con 0x7f5908100720 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.924+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58f8004350 con 0x7f5908100720 2024-03-19T02:26:58.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.924+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8003ab0 con 0x7f5908100720 2024-03-19T02:26:58.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.924+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590819ee00 con 0x7f5908100720 2024-03-19T02:26:58.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.925+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5908107bb0 con 0x7f5908100720 2024-03-19T02:26:58.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.926+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f58f8046020 con 0x7f5908100720 2024-03-19T02:26:58.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.926+0000 7f59057fa640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 0x7f58f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:58.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.926+0000 7f59077fe640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 0x7f58f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:58.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.926+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f58f80f1500 con 0x7f5908100720 2024-03-19T02:26:58.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.927+0000 7f59077fe640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 0x7f58f00a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5908197ef0 tx=0x7f58fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:58.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:58.930+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f58f804d020 con 0x7f5908100720 2024-03-19T02:26:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:58 smithi154 ceph-mon[35693]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:58 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1507497700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:58 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/226406137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:58 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2752515945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:26:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:58 smithi116 ceph-mon[32263]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1507497700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/226406137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:58 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2752515945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:26:59.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.112+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f590802d310 con 0x7f5908100720 2024-03-19T02:26:59.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.112+0000 7f59057fa640 1 -- 172.21.15.116:0/569933412 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f590802d310 con 0x7f5908100720 2024-03-19T02:26:59.113 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 msgr2=0x7f58f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 0x7f58f00a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5908197ef0 tx=0x7f58fc005f90 comp rx=0 tx=0).stop 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 msgr2=0x7f59081969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 0x7f59081969e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036040 tx=0x7f58f8005dc0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 shutdown_connections 2024-03-19T02:26:59.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f58f00a2800 0x7f58f00a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59081022d0 0x7f590819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5908101910 0x7f5908196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 --2- 172.21.15.116:0/569933412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5908100720 0x7f59081969e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 >> 172.21.15.116:0/569933412 conn(0x7f59080fbcb0 msgr2=0x7f5908104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 shutdown_connections 2024-03-19T02:26:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.114+0000 7f590e7bc640 1 -- 172.21.15.116:0/569933412 wait complete. 2024-03-19T02:26:59.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:26:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:58 smithi162 ceph-mon[35761]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:26:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:58 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1507497700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:58 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/226406137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:26:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:58 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2752515945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:26:59.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 -- 172.21.15.116:0/1372395418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320101910 msgr2=0x7fb320101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1372395418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320101910 0x7fb320101d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb310009860 tx=0x7fb31002bb70 comp rx=0 tx=0).stop 2024-03-19T02:26:59.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 -- 172.21.15.116:0/1372395418 shutdown_connections 2024-03-19T02:26:59.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1372395418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1372395418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320101910 0x7fb320101d90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1372395418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320100720 0x7fb320100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 -- 172.21.15.116:0/1372395418 >> 172.21.15.116:0/1372395418 conn(0x7fb3200fbcb0 msgr2=0x7fb3200fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.232+0000 7fb325a0b640 1 -- 172.21.15.116:0/1372395418 shutdown_connections 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 -- 172.21.15.116:0/1372395418 wait complete. 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 Processor -- start 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 -- start start 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320100720 0x7fb3201046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb32010c230 con 0x7fb320101910 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb32010c0b0 con 0x7fb3201022d0 2024-03-19T02:26:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb325a0b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb32010c3b0 con 0x7fb320100720 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb31e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb31e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50378/0 (socket says 172.21.15.116:50378) 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.233+0000 7fb31e7fc640 1 -- 172.21.15.116:0/1042961211 learned_addr learned my addr 172.21.15.116:0/1042961211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31f7fe640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31effd640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320100720 0x7fb3201046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 -- 172.21.15.116:0/1042961211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320100720 msgr2=0x7fb3201046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320100720 0x7fb3201046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 -- 172.21.15.116:0/1042961211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 msgr2=0x7fb320103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 -- 172.21.15.116:0/1042961211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb320103ac0 con 0x7fb320101910 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31f7fe640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb31e7fc640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb310005940 tx=0x7fb310004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310041020 con 0x7fb320101910 2024-03-19T02:26:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3201a1ba0 con 0x7fb320101910 2024-03-19T02:26:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb31002c9c0 con 0x7fb320101910 2024-03-19T02:26:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310039040 con 0x7fb320101910 2024-03-19T02:26:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.234+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3201a2140 con 0x7fb320101910 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.235+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3201a2310 con 0x7fb320101910 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.236+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb310044020 con 0x7fb320101910 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.237+0000 7fb324a09640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 0x7fb3080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.237+0000 7fb31effd640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 0x7fb3080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.237+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb3100f06e0 con 0x7fb320101910 2024-03-19T02:26:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.237+0000 7fb31effd640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 0x7fb3080a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb3140095f0 tx=0x7fb314005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.241+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb31004b020 con 0x7fb320101910 2024-03-19T02:26:59.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.422+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb3201a2600 con 0x7fb320101910 2024-03-19T02:26:59.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.423+0000 7fb324a09640 1 -- 172.21.15.116:0/1042961211 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fb3100bd420 con 0x7fb320101910 2024-03-19T02:26:59.423 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.424+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 msgr2=0x7fb3080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.424+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 0x7fb3080a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb3140095f0 tx=0x7fb314005f90 comp rx=0 tx=0).stop 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.424+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 msgr2=0x7fb320102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.424+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb310005940 tx=0x7fb310004270 comp rx=0 tx=0).stop 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 shutdown_connections 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb3080a2800 0x7fb3080a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3201022d0 0x7fb320103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb320101910 0x7fb320102cf0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 --2- 172.21.15.116:0/1042961211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb320100720 0x7fb3201046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 >> 172.21.15.116:0/1042961211 conn(0x7fb3200fbcb0 msgr2=0x7fb320068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 shutdown_connections 2024-03-19T02:26:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.425+0000 7fb325a0b640 1 -- 172.21.15.116:0/1042961211 wait complete. 2024-03-19T02:26:59.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:26:59.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 -- 172.21.15.116:0/2105191682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c101850 msgr2=0x7fde2c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 --2- 172.21.15.116:0/2105191682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c101850 0x7fde2c101c70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009900 tx=0x7fde1c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:26:59.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 -- 172.21.15.116:0/2105191682 shutdown_connections 2024-03-19T02:26:59.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 --2- 172.21.15.116:0/2105191682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c103400 0x7fde2c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 --2- 172.21.15.116:0/2105191682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 --2- 172.21.15.116:0/2105191682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c101850 0x7fde2c101c70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 -- 172.21.15.116:0/2105191682 >> 172.21.15.116:0/2105191682 conn(0x7fde2c0fcdc0 msgr2=0x7fde2c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 -- 172.21.15.116:0/2105191682 shutdown_connections 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.540+0000 7fde32a27640 1 -- 172.21.15.116:0/2105191682 wait complete. 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 Processor -- start 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 -- start start 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c101850 0x7fde2c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10d350 con 0x7fde2c103400 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10d1d0 con 0x7fde2c102a40 2024-03-19T02:26:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde32a27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10d4d0 con 0x7fde2c101850 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde30fc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde30fc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50394/0 (socket says 172.21.15.116:50394) 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde2b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde30fc4640 1 -- 172.21.15.116:0/1248631654 learned_addr learned my addr 172.21.15.116:0/1248631654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.541+0000 7fde2bfff640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c101850 0x7fde2c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 -- 172.21.15.116:0/1248631654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c101850 msgr2=0x7fde2c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c101850 0x7fde2c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 -- 172.21.15.116:0/1248631654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 msgr2=0x7fde2c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 -- 172.21.15.116:0/1248631654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c19f700 con 0x7fde2c103400 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde2b7fe640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde30fc4640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fde2400efd0 tx=0x7fde2400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24003b70 con 0x7fde2c103400 2024-03-19T02:26:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde24003d10 con 0x7fde2c103400 2024-03-19T02:26:59.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c19f9f0 con 0x7fde2c103400 2024-03-19T02:26:59.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24010980 con 0x7fde2c103400 2024-03-19T02:26:59.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.542+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c19fef0 con 0x7fde2c103400 2024-03-19T02:26:59.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.543+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2c108ce0 con 0x7fde2c103400 2024-03-19T02:26:59.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.544+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fde24004b40 con 0x7fde2c103400 2024-03-19T02:26:59.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.545+0000 7fde297fa640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 0x7fde140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.545+0000 7fde2bfff640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 0x7fde140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.545+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fde240cf5a0 con 0x7fde2c103400 2024-03-19T02:26:59.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.545+0000 7fde2bfff640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 0x7fde140a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde1c005ab0 tx=0x7fde1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.548+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde2409c2e0 con 0x7fde2c103400 2024-03-19T02:26:59.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.729+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde2c062420 con 0x7fde2c103400 2024-03-19T02:26:59.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.730+0000 7fde297fa640 1 -- 172.21.15.116:0/1248631654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fde2409bbe0 con 0x7fde2c103400 2024-03-19T02:26:59.730 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 msgr2=0x7fde140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 0x7fde140a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde1c005ab0 tx=0x7fde1c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 msgr2=0x7fde2c19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fde2400efd0 tx=0x7fde2400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 shutdown_connections 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fde140a2800 0x7fde140a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde2c103400 0x7fde2c19f090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde2c102a40 0x7fde2c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 --2- 172.21.15.116:0/1248631654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde2c101850 0x7fde2c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 >> 172.21.15.116:0/1248631654 conn(0x7fde2c0fcdc0 msgr2=0x7fde2c105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 shutdown_connections 2024-03-19T02:26:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.732+0000 7fde32a27640 1 -- 172.21.15.116:0/1248631654 wait complete. 2024-03-19T02:26:59.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:26:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- 172.21.15.116:0/3974565263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d81022e0 msgr2=0x7ff1d8108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 --2- 172.21.15.116:0/3974565263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d81022e0 0x7ff1d8108b70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009ca0 tx=0x7ff1d002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- 172.21.15.116:0/3974565263 shutdown_connections 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 --2- 172.21.15.116:0/3974565263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d81022e0 0x7ff1d8108b70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 --2- 172.21.15.116:0/3974565263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8101920 0x7ff1d8101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 --2- 172.21.15.116:0/3974565263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d8100730 0x7ff1d8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- 172.21.15.116:0/3974565263 >> 172.21.15.116:0/3974565263 conn(0x7ff1d80fbcc0 msgr2=0x7ff1d80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- 172.21.15.116:0/3974565263 shutdown_connections 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- 172.21.15.116:0/3974565263 wait complete. 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 Processor -- start 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.849+0000 7ff1de861640 1 -- start start 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8100730 0x7ff1d8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d81022e0 0x7ff1d819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c260 con 0x7ff1d8101920 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c0e0 con 0x7ff1d81022e0 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c3e0 con 0x7ff1d8100730 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50418/0 (socket says 172.21.15.116:50418) 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 -- 172.21.15.116:0/143963245 learned_addr learned my addr 172.21.15.116:0/143963245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1dcdfe640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d81022e0 0x7ff1d819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d7fff640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8100730 0x7ff1d8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 -- 172.21.15.116:0/143963245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8100730 msgr2=0x7ff1d8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8100730 0x7ff1d8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 -- 172.21.15.116:0/143963245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d81022e0 msgr2=0x7ff1d819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:26:59.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d81022e0 0x7ff1d819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 -- 172.21.15.116:0/143963245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d819f670 con 0x7ff1d8101920 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d77fe640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e9f0 tx=0x7ff1c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800cd80 con 0x7ff1d8101920 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1c80054d0 con 0x7ff1d8101920 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8010490 con 0x7ff1d8101920 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.850+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d819f960 con 0x7ff1d8101920 2024-03-19T02:26:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.851+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d819fe60 con 0x7ff1d8101920 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.851+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d81a00f0 con 0x7ff1d8101920 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.853+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ff1c8005000 con 0x7ff1d8101920 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.853+0000 7ff1d57fa640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 0x7ff1c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.853+0000 7ff1d7fff640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 0x7ff1c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.853+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff1c80cf320 con 0x7ff1d8101920 2024-03-19T02:26:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.854+0000 7ff1d7fff640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 0x7ff1c00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8101780 tx=0x7ff1cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:26:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:26:59.857+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1c8053740 con 0x7ff1d8101920 2024-03-19T02:27:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:59 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2094577160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:59 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/569933412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:26:59 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1042961211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:59 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2094577160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:59 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/569933412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:26:59 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1042961211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.031+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff1d81a02c0 con 0x7ff1d8101920 2024-03-19T02:27:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.031+0000 7ff1d57fa640 1 -- 172.21.15.116:0/143963245 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7ff1c80052f0 con 0x7ff1d8101920 2024-03-19T02:27:00.032 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 msgr2=0x7ff1c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 0x7ff1c00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8101780 tx=0x7ff1cc005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 msgr2=0x7ff1d8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e9f0 tx=0x7ff1c800edf0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 shutdown_connections 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff1c00a2800 0x7ff1c00a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d81022e0 0x7ff1d819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1d8101920 0x7ff1d8197f90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 --2- 172.21.15.116:0/143963245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d8100730 0x7ff1d8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 >> 172.21.15.116:0/143963245 conn(0x7ff1d80fbcc0 msgr2=0x7ff1d80fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 shutdown_connections 2024-03-19T02:27:00.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.033+0000 7ff1de861640 1 -- 172.21.15.116:0/143963245 wait complete. 2024-03-19T02:27:00.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/1033747009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 msgr2=0x7f3fec101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/1033747009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec101db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009860 tx=0x7f3fe802bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/1033747009 shutdown_connections 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/1033747009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec1022f0 0x7f3fec108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/1033747009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec101db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/1033747009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100740 0x7f3fec100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/1033747009 >> 172.21.15.116:0/1033747009 conn(0x7f3fec0fbcb0 msgr2=0x7f3fec0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/1033747009 shutdown_connections 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/1033747009 wait complete. 2024-03-19T02:27:00.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 Processor -- start 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.146+0000 7f3ff3f1b640 1 -- start start 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 0x7f3fec06aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10c250 con 0x7f3fec1022f0 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10c0d0 con 0x7f3fec101930 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10c3d0 con 0x7f3fec100740 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff1cb7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 0x7f3fec06aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff1cb7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 0x7f3fec06aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36408/0 (socket says 172.21.15.116:36408) 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50434/0 (socket says 172.21.15.116:50434) 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff1cb7640 1 -- 172.21.15.116:0/2679079690 learned_addr learned my addr 172.21.15.116:0/2679079690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:00.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff14b6640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 -- 172.21.15.116:0/2679079690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 msgr2=0x7f3fec06aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 0x7f3fec06aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 -- 172.21.15.116:0/2679079690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 msgr2=0x7f3fec19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 -- 172.21.15.116:0/2679079690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fec19f690 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff14b6640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff24b8640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400efc0 tx=0x7f3fe400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4003970 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fec19f980 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fe4004020 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fec19fea0 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.147+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400b940 con 0x7f3fec1022f0 2024-03-19T02:27:00.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.148+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fec068a60 con 0x7f3fec1022f0 2024-03-19T02:27:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.149+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3fe401d020 con 0x7f3fec1022f0 2024-03-19T02:27:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.150+0000 7f3fdeffd640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 0x7f3fd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.150+0000 7f3ff1cb7640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 0x7f3fd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.150+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3fe40cec90 con 0x7f3fec1022f0 2024-03-19T02:27:00.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.151+0000 7f3ff1cb7640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 0x7f3fd80a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00095f0 tx=0x7f3fe0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.155+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fe409b9d0 con 0x7f3fec1022f0 2024-03-19T02:27:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:59 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2094577160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:59 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/569933412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:26:59 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1042961211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.368+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fec062420 con 0x7f3fec1022f0 2024-03-19T02:27:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.368+0000 7f3fdeffd640 1 -- 172.21.15.116:0/2679079690 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f3fe409b2d0 con 0x7f3fec1022f0 2024-03-19T02:27:00.369 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:00.369 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:27:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 msgr2=0x7f3fd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 0x7f3fd80a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00095f0 tx=0x7f3fe0005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 msgr2=0x7f3fec19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400efc0 tx=0x7f3fe400c800 comp rx=0 tx=0).stop 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 shutdown_connections 2024-03-19T02:27:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3fd80a2800 0x7f3fd80a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec1022f0 0x7f3fec19edd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3fec101930 0x7f3fec19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 --2- 172.21.15.116:0/2679079690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fec100740 0x7f3fec06aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.370+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 >> 172.21.15.116:0/2679079690 conn(0x7f3fec0fbcb0 msgr2=0x7f3fec104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.371+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 shutdown_connections 2024-03-19T02:27:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.371+0000 7f3ff3f1b640 1 -- 172.21.15.116:0/2679079690 wait complete. 2024-03-19T02:27:00.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 -- 172.21.15.116:0/3001147741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 msgr2=0x7f29f4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/3001147741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4100be0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009900 tx=0x7f29e402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 -- 172.21.15.116:0/3001147741 shutdown_connections 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/3001147741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 0x7f29f4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/3001147741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 0x7f29f4101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/3001147741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4100be0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 -- 172.21.15.116:0/3001147741 >> 172.21.15.116:0/3001147741 conn(0x7f29f40fbd50 msgr2=0x7f29f40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 -- 172.21.15.116:0/3001147741 shutdown_connections 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.483+0000 7f29f9fb2640 1 -- 172.21.15.116:0/3001147741 wait complete. 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 Processor -- start 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 -- start start 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 0x7f29f4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 0x7f29f419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410d3a0 con 0x7f29f41007c0 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410d220 con 0x7f29f41019b0 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f9fb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410d520 con 0x7f29f4102370 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50464/0 (socket says 172.21.15.116:50464) 2024-03-19T02:27:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 -- 172.21.15.116:0/1510789059 learned_addr learned my addr 172.21.15.116:0/1510789059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f3fff640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 0x7f29f419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f2ffd640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 0x7f29f4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 -- 172.21.15.116:0/1510789059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 msgr2=0x7f29f419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 0x7f29f419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 -- 172.21.15.116:0/1510789059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 msgr2=0x7f29f4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 0x7f29f4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 -- 172.21.15.116:0/1510789059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f419f6d0 con 0x7f29f41007c0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.484+0000 7f29f37fe640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f29e4000c00 tx=0x7f29e4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.485+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4043020 con 0x7f29f41007c0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.485+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f419f960 con 0x7f29f41007c0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.485+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29e4004350 con 0x7f29f41007c0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.485+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4003b80 con 0x7f29f41007c0 2024-03-19T02:27:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.485+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f419feb0 con 0x7f29f41007c0 2024-03-19T02:27:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.486+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f4068a60 con 0x7f29f41007c0 2024-03-19T02:27:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.487+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f29e4046020 con 0x7f29f41007c0 2024-03-19T02:27:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.487+0000 7f29f0ff9640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 0x7f29dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.487+0000 7f29f2ffd640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 0x7f29dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.487+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f29e40f0ee0 con 0x7f29f41007c0 2024-03-19T02:27:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.488+0000 7f29f2ffd640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 0x7f29dc0a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29e80095f0 tx=0x7f29e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.492+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29e404d020 con 0x7f29f41007c0 2024-03-19T02:27:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.668+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29f4062420 con 0x7f29f41007c0 2024-03-19T02:27:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.669+0000 7f29f0ff9640 1 -- 172.21.15.116:0/1510789059 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24665 (secure 0 0 0) 0x7f29e40bdc20 con 0x7f29f41007c0 2024-03-19T02:27:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:26:08.779202+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 msgr2=0x7f29dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 0x7f29dc0a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f29e80095f0 tx=0x7f29e8005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 msgr2=0x7f29f4197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f29e4000c00 tx=0x7f29e4005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 shutdown_connections 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f29dc0a2800 0x7f29dc0a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29f4102370 0x7f29f419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29f41019b0 0x7f29f4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 --2- 172.21.15.116:0/1510789059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29f41007c0 0x7f29f4197ab0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.671+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 >> 172.21.15.116:0/1510789059 conn(0x7f29f40fbd50 msgr2=0x7f29f4104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.672+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 shutdown_connections 2024-03-19T02:27:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.672+0000 7f29f9fb2640 1 -- 172.21.15.116:0/1510789059 wait complete. 2024-03-19T02:27:00.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 -- 172.21.15.116:0/2252243909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f4102ad0 msgr2=0x7f22f4102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 --2- 172.21.15.116:0/2252243909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f4102ad0 0x7f22f4102f50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009930 tx=0x7f22dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 -- 172.21.15.116:0/2252243909 shutdown_connections 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 --2- 172.21.15.116:0/2252243909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4103490 0x7f22f4109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 --2- 172.21.15.116:0/2252243909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f4102ad0 0x7f22f4102f50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 --2- 172.21.15.116:0/2252243909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f40ffb00 0x7f22f40fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 -- 172.21.15.116:0/2252243909 >> 172.21.15.116:0/2252243909 conn(0x7f22f40fb450 msgr2=0x7f22f40fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 -- 172.21.15.116:0/2252243909 shutdown_connections 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.786+0000 7f22fa109640 1 -- 172.21.15.116:0/2252243909 wait complete. 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 Processor -- start 2024-03-19T02:27:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 -- start start 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4102ad0 0x7f22f4100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4103490 0x7f22f4100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410d370 con 0x7f22f40ffb00 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410d1f0 con 0x7f22f4103490 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410d4f0 con 0x7f22f4102ad0 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50488/0 (socket says 172.21.15.116:50488) 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 -- 172.21.15.116:0/4269442664 learned_addr learned my addr 172.21.15.116:0/4269442664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f3fff640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4103490 0x7f22f4100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22ebfff640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4102ad0 0x7f22f4100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 -- 172.21.15.116:0/4269442664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4102ad0 msgr2=0x7f22f4100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4102ad0 0x7f22f4100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 -- 172.21.15.116:0/4269442664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4103490 msgr2=0x7f22f4100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4103490 0x7f22f4100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 -- 172.21.15.116:0/4269442664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f4101880 con 0x7f22f40ffb00 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f37fe640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f22e400e9c0 tx=0x7f22e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e400cd80 con 0x7f22f40ffb00 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22e40035f0 con 0x7f22f40ffb00 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4010690 con 0x7f22f40ffb00 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.787+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41a1c30 con 0x7f22f40ffb00 2024-03-19T02:27:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.788+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a2150 con 0x7f22f40ffb00 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.789+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4108d70 con 0x7f22f40ffb00 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.790+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f22e40037a0 con 0x7f22f40ffb00 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.790+0000 7f22f17fa640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 0x7f22e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.791+0000 7f22ebfff640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 0x7f22e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.791+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f22e40cf140 con 0x7f22f40ffb00 2024-03-19T02:27:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.791+0000 7f22ebfff640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 0x7f22e00a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009900 tx=0x7f22dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:00.795+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22e409be80 con 0x7f22f40ffb00 2024-03-19T02:27:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:00 smithi154 ceph-mon[35693]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1248631654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/143963245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2679079690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1510789059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.006+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22f402d1f0 con 0x7f22f40ffb00 2024-03-19T02:27:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.007+0000 7f22f17fa640 1 -- 172.21.15.116:0/4269442664 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f22e409b780 con 0x7f22f40ffb00 2024-03-19T02:27:01.008 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:01.008 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:27:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:27:01.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:00 smithi116 ceph-mon[32263]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:01.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1248631654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/143963245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2679079690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1510789059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.009+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 msgr2=0x7f22e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.009+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 0x7f22e00a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009900 tx=0x7f22dc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 msgr2=0x7f22f41023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f22e400e9c0 tx=0x7f22e400ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 shutdown_connections 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f22e00a2800 0x7f22e00a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4103490 0x7f22f4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f4102ad0 0x7f22f4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 --2- 172.21.15.116:0/4269442664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22f40ffb00 0x7f22f41023f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 >> 172.21.15.116:0/4269442664 conn(0x7f22f40fb450 msgr2=0x7f22f4105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 shutdown_connections 2024-03-19T02:27:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.010+0000 7f22fa109640 1 -- 172.21.15.116:0/4269442664 wait complete. 2024-03-19T02:27:01.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T02:25:08.015174+0000', 'modified': '2024-03-19T02:25:36.674122+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24494, 'mds_1': 24484, 'mds_2': 24470, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1891911264', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1891911264}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1891911264}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6841/3941624950', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 3941624950}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 3941624950}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6833/1010748955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6832', 'nonce': 1010748955}, {'type': 'v1', 'addr': '172.21.15.154:6833', 'nonce': 1010748955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6833/750284800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 750284800}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 750284800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6835/1464153745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 1464153745}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 1464153745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.162:6835/831287214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 831287214}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 831287214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.162:6837/3347737639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 3347737639}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 3347737639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.154:6837/455932973', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 455932973}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 455932973}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6839/1427360219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 1427360219}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 1427360219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/4009274564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 4009274564}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 4009274564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24463, 24470, 24484, 24494, 24451]} 2024-03-19T02:27:01.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T02:27:01.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 -- 172.21.15.116:0/1634966460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9484101060 msgr2=0x7f94841014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 --2- 172.21.15.116:0/1634966460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9484101060 0x7f94841014e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9474009900 tx=0x7f947402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 -- 172.21.15.116:0/1634966460 shutdown_connections 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 --2- 172.21.15.116:0/1634966460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 0x7f9484108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 --2- 172.21.15.116:0/1634966460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9484101060 0x7f94841014e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 --2- 172.21.15.116:0/1634966460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94840ffe80 0x7f94841002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 -- 172.21.15.116:0/1634966460 >> 172.21.15.116:0/1634966460 conn(0x7f94840fb420 msgr2=0x7f94840fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 -- 172.21.15.116:0/1634966460 shutdown_connections 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.124+0000 7f948c636640 1 -- 172.21.15.116:0/1634966460 wait complete. 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 Processor -- start 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 -- start start 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9484101060 0x7f9484197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 0x7f948419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f948410b180 con 0x7f94840ffe80 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f948410b000 con 0x7f9484101a20 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948c636640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f948410b300 con 0x7f9484101060 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50502/0 (socket says 172.21.15.116:50502) 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 -- 172.21.15.116:0/3344603046 learned_addr learned my addr 172.21.15.116:0/3344603046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f9489bd1640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9484101060 0x7f9484197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948abd3640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 0x7f948419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 -- 172.21.15.116:0/3344603046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9484101060 msgr2=0x7f9484197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9484101060 0x7f9484197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 -- 172.21.15.116:0/3344603046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 msgr2=0x7f948419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 0x7f948419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 -- 172.21.15.116:0/3344603046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948419f680 con 0x7f94840ffe80 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f94800058c0 tx=0x7f948000ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.125+0000 7f947b7fe640 1 -- 172.21.15.116:0/3344603046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948001b070 con 0x7f94840ffe80 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.126+0000 7f947b7fe640 1 -- 172.21.15.116:0/3344603046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f948000cda0 con 0x7f94840ffe80 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.126+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948419f970 con 0x7f94840ffe80 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.126+0000 7f947b7fe640 1 -- 172.21.15.116:0/3344603046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000b760 con 0x7f94840ffe80 2024-03-19T02:27:01.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.126+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948419fe90 con 0x7f94840ffe80 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f947b7fe640 1 -- 172.21.15.116:0/3344603046 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f948001e070 con 0x7f94840ffe80 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f948f21d740 1 Processor -- start 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f947b7fe640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 0x7f94700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f948abd3640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 0x7f94700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f948f21d740 1 -- start start 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.128+0000 7f947b7fe640 1 -- 172.21.15.116:0/3344603046 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f94800ce8f0 con 0x7f94840ffe80 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b10889e0 0x5556b108d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b108ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5556b0fecde0 con 0x5556b1087ff0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556b0f95c80 con 0x5556b108db50 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948f21d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556b0fe8e80 con 0x5556b10889e0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50508/0 (socket says 172.21.15.116:50508) 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 -- 172.21.15.116:0/383726168 learned_addr learned my addr 172.21.15.116:0/383726168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f9489bd1640 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b10889e0 0x5556b108d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948abd3640 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b108ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 -- 172.21.15.116:0/383726168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b10889e0 msgr2=0x5556b108d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b10889e0 0x5556b108d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 -- 172.21.15.116:0/383726168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 msgr2=0x5556b108ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b108ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948abd3640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 0x7f94700a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f947c005fd0 tx=0x7f947c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 -- 172.21.15.116:0/383726168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556b0fa7fb0 con 0x5556b1087ff0 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f948a3d2640 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f948009af70 tx=0x7f948009aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3996d0c2526c9e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f9478bfe640 1 -- 172.21.15.116:0/383726168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800992e0 con 0x5556b1087ff0 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f9478bfe640 1 -- 172.21.15.116:0/383726168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9480016b70 con 0x5556b1087ff0 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.129+0000 7f9478bfe640 1 -- 172.21.15.116:0/383726168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480096d30 con 0x5556b1087ff0 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- 172.21.15.116:0/383726168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 msgr2=0x5556b1088410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f948009af70 tx=0x7f948009aee0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- 172.21.15.116:0/383726168 shutdown_connections 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b108ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b10889e0 0x5556b108d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 --2- 172.21.15.116:0/383726168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b1087ff0 0x5556b1088410 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- 172.21.15.116:0/383726168 >> 172.21.15.116:0/383726168 conn(0x5556b1083560 msgr2=0x5556b1083960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- 172.21.15.116:0/383726168 shutdown_connections 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- 172.21.15.116:0/383726168 wait complete. 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.130+0000 7f948f21d740 1 -- start start 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 Processor -- start 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 0x5556b1092480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b108db50 0x5556b1099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5556b1090aa0 con 0x5556b10889e0 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556b0fa8fc0 con 0x5556b1087ff0 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556b1090c20 con 0x5556b108db50 2024-03-19T02:27:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948f21d740 10 client.0 _ll_register_callbacks cb 0x5556b0fe4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50514/0 (socket says 172.21.15.116:50514) 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 -- 172.21.15.116:0/3306028500 learned_addr learned my addr 172.21.15.116:0/3306028500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 0x5556b1092480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948abd3640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b108db50 0x5556b1099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 -- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b108db50 msgr2=0x5556b1099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b108db50 0x5556b1099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 -- 172.21.15.116:0/3306028500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 msgr2=0x5556b1092480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 0x5556b1092480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 -- 172.21.15.116:0/3306028500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556b0d012d0 con 0x5556b10889e0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 0x5556b1092480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f9489bd1640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9474009900 tx=0x7f9474004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b70583cebb309e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.131+0000 7f946dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474005030 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f94740051d0 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9480016b70 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9474052050 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 10 client.15222 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556b0dcb6b0 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9480016b70 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f94740035f0 con 0x5556b10889e0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 10 client.15222 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 10 client.15222 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 20 client.15222 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b10a0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 4 client.15222 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556b0ffa320 con 0x5556b109f2f0 2024-03-19T02:27:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.132+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b10a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.133+0000 7f948a3d2640 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b10a0760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7ecaa6e3d9fcb435 server_cookie=9e48175424c969d4 in_seq=0 out_seq=0 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.133+0000 7f946dffb640 10 client.15222 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.133+0000 7f946dffb640 1 -- 172.21.15.116:0/3306028500 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f9480079e50 con 0x5556b109f2f0 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.133+0000 7f946dffb640 10 client.15222 handle_command_reply: tid=0 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.133+0000 7f948f21d740 1 client.15222 shutdown 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 msgr2=0x5556b10929c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9474009900 tx=0x7f9474004270 comp rx=0 tx=0).stop 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 shutdown_connections 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b10a0760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b108db50 0x5556b1099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f946dffb640 0 client.15222 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b10929c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 --2- 172.21.15.116:0/3306028500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b1087ff0 0x5556b1092480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 >> 172.21.15.116:0/3306028500 conn(0x5556b1083560 msgr2=0x5556b10827e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 shutdown_connections 2024-03-19T02:27:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 1 -- 172.21.15.116:0/3306028500 wait complete. 2024-03-19T02:27:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.134+0000 7f948f21d740 20 client.15222 trim_cache size 0 max 16384 2024-03-19T02:27:01.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.151+0000 7f948f21d740 1 Processor -- start 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.151+0000 7f948f21d740 1 -- start start 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b108d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 0x5556b108ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5556b109eaa0 con 0x5556b108db50 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556b109f170 con 0x5556b10889e0 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948f21d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556b109f840 con 0x5556b1087ff0 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50530/0 (socket says 172.21.15.116:50530) 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948a3d2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b108d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 -- 172.21.15.116:0/1849409795 learned_addr learned my addr 172.21.15.116:0/1849409795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f9489bd1640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 0x5556b108ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 -- 172.21.15.116:0/1849409795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 msgr2=0x5556b108d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b108d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 -- 172.21.15.116:0/1849409795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 msgr2=0x5556b108ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 0x5556b108ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 -- 172.21.15.116:0/1849409795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556b108d190 con 0x5556b108db50 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948a3d2640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b108d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f9489bd1640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 0x5556b108ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f948abd3640 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f947c003c40 tx=0x7f947c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8155f99261bf8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f946ffff640 1 -- 172.21.15.116:0/1849409795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c022040 con 0x5556b108db50 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f946ffff640 1 -- 172.21.15.116:0/1849409795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f947c003db0 con 0x5556b108db50 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.152+0000 7f946ffff640 1 -- 172.21.15.116:0/1849409795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c01c4b0 con 0x5556b108db50 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- 172.21.15.116:0/1849409795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 msgr2=0x5556b1089ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f947c003c40 tx=0x7f947c01c000 comp rx=0 tx=0).stop 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- 172.21.15.116:0/1849409795 shutdown_connections 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b108db50 0x5556b1089ea0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b10889e0 0x5556b108ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 --2- 172.21.15.116:0/1849409795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b108d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- 172.21.15.116:0/1849409795 >> 172.21.15.116:0/1849409795 conn(0x5556b1083560 msgr2=0x5556b1083940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- 172.21.15.116:0/1849409795 shutdown_connections 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- 172.21.15.116:0/1849409795 wait complete. 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 -- start start 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.153+0000 7f948f21d740 1 Processor -- start 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b1103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b1107be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5556b10a05e0 con 0x5556b10889e0 2024-03-19T02:27:01.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556b109ff10 con 0x5556b108db50 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556b10a0cb0 con 0x5556b1087ff0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 10 client.0 _ll_register_callbacks cb 0x7f9474009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50538/0 (socket says 172.21.15.116:50538) 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 -- 172.21.15.116:0/1524642112 learned_addr learned my addr 172.21.15.116:0/1524642112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948abd3640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b1107be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948a3d2640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b1103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 -- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 msgr2=0x5556b1103050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b1103050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 -- 172.21.15.116:0/1524642112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 msgr2=0x5556b1107be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b1107be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 -- 172.21.15.116:0/1524642112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556b1108410 con 0x5556b10889e0 2024-03-19T02:27:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f9489bd1640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f947402c170 tx=0x7f9474003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efddf55deacf8e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f946dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947405a020 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f94740048c0 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c003db0 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9474062050 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 10 client.15228 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.154+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556b11097b0 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f947c003db0 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9474005440 con 0x5556b10889e0 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 10 client.15228 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 10 client.15228 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 20 client.15228 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b110d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 4 client.15228 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "status"}) -- 0x5556b0ff4670 con 0x5556b109f2f0 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948a3d2640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b110d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f946dffb640 10 client.15228 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948a3d2640 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b110d0f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=425aeaba3466526d server_cookie=c24c1be31c489068 in_seq=0 out_seq=0 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f946dffb640 1 -- 172.21.15.116:0/1524642112 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5556b0ff4670 con 0x5556b109f2f0 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f946dffb640 10 client.15228 handle_command_reply: tid=0 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.155+0000 7f948f21d740 1 client.15228 shutdown 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 msgr2=0x5556b11055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f947402c170 tx=0x7f9474003eb0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 shutdown_connections 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5556b109f2f0 0x5556b110d0f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556b108db50 0x5556b1107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f946dffb640 0 client.15228 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5556b10889e0 0x5556b11055a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 --2- 172.21.15.116:0/1524642112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556b1087ff0 0x5556b1103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 >> 172.21.15.116:0/1524642112 conn(0x5556b1083560 msgr2=0x5556b107e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 shutdown_connections 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 1 -- 172.21.15.116:0/1524642112 wait complete. 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948f21d740 20 client.15228 trim_cache size 0 max 16384 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24451, 2024-03-19T02:27:01.164 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 85.438441267000002, 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 92, 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 134.04034906699999 2024-03-19T02:27:01.165 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 msgr2=0x7f94700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 0x7f94700a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f947c005fd0 tx=0x7f947c009500 comp rx=0 tx=0).stop 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 msgr2=0x7f9484197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f94800058c0 tx=0x7f948000ef80 comp rx=0 tx=0).stop 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 shutdown_connections 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f94700a2800 0x7f94700a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9484101a20 0x7f948419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9484101060 0x7f9484197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 --2- 172.21.15.116:0/3344603046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94840ffe80 0x7f9484197a60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 >> 172.21.15.116:0/3344603046 conn(0x7f94840fb420 msgr2=0x7f9484106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 shutdown_connections 2024-03-19T02:27:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.156+0000 7f948c636640 1 -- 172.21.15.116:0/3344603046 wait complete. 2024-03-19T02:27:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:00 smithi162 ceph-mon[35761]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:01.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1248631654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/143963245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:01.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2679079690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1510789059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:01.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 -- 172.21.15.116:0/3189543157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c100730 msgr2=0x7efe4c100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 --2- 172.21.15.116:0/3189543157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c100730 0x7efe4c100b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009900 tx=0x7efe3c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 -- 172.21.15.116:0/3189543157 shutdown_connections 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 --2- 172.21.15.116:0/3189543157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c1022e0 0x7efe4c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 --2- 172.21.15.116:0/3189543157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c101920 0x7efe4c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 --2- 172.21.15.116:0/3189543157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c100730 0x7efe4c100b50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 -- 172.21.15.116:0/3189543157 >> 172.21.15.116:0/3189543157 conn(0x7efe4c0fbcc0 msgr2=0x7efe4c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 -- 172.21.15.116:0/3189543157 shutdown_connections 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.270+0000 7efe537f4640 1 -- 172.21.15.116:0/3189543157 wait complete. 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 Processor -- start 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 -- start start 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c101920 0x7efe4c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c1022e0 0x7efe4c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c10d340 con 0x7efe4c100730 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c10d1c0 con 0x7efe4c101920 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe4c10d4c0 con 0x7efe4c1022e0 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50560/0 (socket says 172.21.15.116:50560) 2024-03-19T02:27:01.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 -- 172.21.15.116:0/1573428170 learned_addr learned my addr 172.21.15.116:0/1573428170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c101920 0x7efe4c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51d91640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c1022e0 0x7efe4c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 -- 172.21.15.116:0/1573428170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c1022e0 msgr2=0x7efe4c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c1022e0 0x7efe4c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 -- 172.21.15.116:0/1573428170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c101920 msgr2=0x7efe4c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c101920 0x7efe4c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 -- 172.21.15.116:0/1573428170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe4c19f720 con 0x7efe4c100730 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe51590640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe427fc640 1 -- 172.21.15.116:0/1573428170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c043020 con 0x7efe4c100730 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe427fc640 1 -- 172.21.15.116:0/1573428170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3c004350 con 0x7efe4c100730 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe4c19f9b0 con 0x7efe4c100730 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.271+0000 7efe427fc640 1 -- 172.21.15.116:0/1573428170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3c003b10 con 0x7efe4c100730 2024-03-19T02:27:01.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.272+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe4c19ff00 con 0x7efe4c100730 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.274+0000 7efe427fc640 1 -- 172.21.15.116:0/1573428170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7efe3c046020 con 0x7efe4c100730 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.274+0000 7efe563e9740 1 Processor -- start 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 -- start start 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584784e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x56065847afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5606583d7de0 con 0x560658472ff0 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560658380c80 con 0x5606584739e0 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe563e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5606583d3e80 con 0x560658478b50 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe427fc640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50568/0 (socket says 172.21.15.116:50568) 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 -- 172.21.15.116:0/1941787417 learned_addr learned my addr 172.21.15.116:0/1941787417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x56065847afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51590640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 -- 172.21.15.116:0/1941787417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 msgr2=0x5606584784e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584784e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 -- 172.21.15.116:0/1941787417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 msgr2=0x560658473410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 -- 172.21.15.116:0/1941787417 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560658392fb0 con 0x560658478b50 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe427fc640 1 -- 172.21.15.116:0/1573428170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7efe3c0f0f20 con 0x7efe4c100730 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x56065847afe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe48009bf0 tx=0x7efe48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4c4c13ecfef9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.275+0000 7efe51d91640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x560658474240 tx=0x7efe44004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe377fe640 1 -- 172.21.15.116:0/1941787417 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48004050 con 0x560658478b50 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe377fe640 1 -- 172.21.15.116:0/1941787417 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe480054d0 con 0x560658478b50 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe377fe640 1 -- 172.21.15.116:0/1941787417 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48004cb0 con 0x560658478b50 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 -- 172.21.15.116:0/1941787417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 msgr2=0x56065847afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x56065847afe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe48009bf0 tx=0x7efe48011810 comp rx=0 tx=0).stop 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 -- 172.21.15.116:0/1941787417 shutdown_connections 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x56065847afe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 --2- 172.21.15.116:0/1941787417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658472ff0 0x560658473410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 -- 172.21.15.116:0/1941787417 >> 172.21.15.116:0/1941787417 conn(0x56065846e560 msgr2=0x56065846e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 -- 172.21.15.116:0/1941787417 shutdown_connections 2024-03-19T02:27:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.276+0000 7efe563e9740 1 -- 172.21.15.116:0/1941787417 wait complete. 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 -- start start 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 Processor -- start 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560658472ff0 0x56065847d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x560658484a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56065847bb00 con 0x5606584739e0 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560658393fc0 con 0x560658472ff0 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56065847bc80 con 0x560658478b50 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe563e9740 10 client.0 _ll_register_callbacks cb 0x5606583cfa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50570/0 (socket says 172.21.15.116:50570) 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 -- 172.21.15.116:0/1121988291 learned_addr learned my addr 172.21.15.116:0/1121988291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe51d91640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x560658484a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe51590640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560658472ff0 0x56065847d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 -- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 msgr2=0x560658484a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x560658484a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 -- 172.21.15.116:0/1121988291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560658472ff0 msgr2=0x56065847d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560658472ff0 0x56065847d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.277+0000 7efe50d8f640 1 -- 172.21.15.116:0/1121988291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606580ec2d0 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe4801b040 tx=0x7efe48009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b321e8de21f9af1b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48029070 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7efe48003a00 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe480054d0 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe48035050 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 10 client.15240 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5606581b66b0 con 0x5606584739e0 2024-03-19T02:27:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe480054d0 con 0x5606584739e0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efe48005100 con 0x5606584739e0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 10 client.15240 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 20 client.15240 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56065848a0c0 0x56065848b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 4 client.15240 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.278+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606583e5320 con 0x56065848a0c0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.279+0000 7efe51590640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56065848a0c0 0x56065848b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.279+0000 7efe51590640 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56065848a0c0 0x56065848b530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=99f7050e69004b2a server_cookie=d5d674c48dc1ba22 in_seq=0 out_seq=0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.279+0000 7efe34ff9640 10 client.15240 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe34ff9640 1 -- 172.21.15.116:0/1121988291 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5606583e5320 con 0x56065848a0c0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe34ff9640 10 client.15240 handle_command_reply: tid=0 2024-03-19T02:27:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 client.15240 shutdown 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 msgr2=0x56065847d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7efe4801b040 tx=0x7efe48009fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 shutdown_connections 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658478b50 0x560658484a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56065848a0c0 0x56065848b530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606584739e0 0x56065847d9e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 --2- 172.21.15.116:0/1121988291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560658472ff0 0x56065847d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe34ff9640 0 client.15240 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 >> 172.21.15.116:0/1121988291 conn(0x56065846e560 msgr2=0x56065846d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 shutdown_connections 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 1 -- 172.21.15.116:0/1121988291 wait complete. 2024-03-19T02:27:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.280+0000 7efe563e9740 20 client.15240 trim_cache size 0 max 16384 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 Processor -- start 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 -- start start 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x56065847c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560658489870 con 0x560658478b50 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560658489f40 con 0x5606584739e0 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe563e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56065848a610 con 0x560658472ff0 2024-03-19T02:27:01.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50572/0 (socket says 172.21.15.116:50572) 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 -- 172.21.15.116:0/1211302064 learned_addr learned my addr 172.21.15.116:0/1211302064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51590640 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x56065847c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 -- 172.21.15.116:0/1211302064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 msgr2=0x56065847c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x56065847c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 -- 172.21.15.116:0/1211302064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 msgr2=0x5606584849e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584849e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 -- 172.21.15.116:0/1211302064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560658474270 con 0x560658478b50 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe51d91640 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efe44013db0 tx=0x7efe44009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99804b8505bd4c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe36ffd640 1 -- 172.21.15.116:0/1211302064 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44013370 con 0x560658478b50 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe36ffd640 1 -- 172.21.15.116:0/1211302064 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe440109b0 con 0x560658478b50 2024-03-19T02:27:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.298+0000 7efe36ffd640 1 -- 172.21.15.116:0/1211302064 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44005560 con 0x560658478b50 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- 172.21.15.116:0/1211302064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 msgr2=0x560658488fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efe44013db0 tx=0x7efe44009bf0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- 172.21.15.116:0/1211302064 shutdown_connections 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x560658488fd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x5606584849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 --2- 172.21.15.116:0/1211302064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x56065847c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- 172.21.15.116:0/1211302064 >> 172.21.15.116:0/1211302064 conn(0x56065846e560 msgr2=0x56065846e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- 172.21.15.116:0/1211302064 shutdown_connections 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- 172.21.15.116:0/1211302064 wait complete. 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 -- start start 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.299+0000 7efe563e9740 1 Processor -- start 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x560658482c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x560658483160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56065848b3b0 con 0x560658478b50 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56065848ace0 con 0x5606584739e0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56065848ba80 con 0x560658472ff0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe563e9740 10 client.0 _ll_register_callbacks cb 0x7efe44009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50588/0 (socket says 172.21.15.116:50588) 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 -- 172.21.15.116:0/1526081092 learned_addr learned my addr 172.21.15.116:0/1526081092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe50d8f640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x560658483160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51590640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x560658482c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 -- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 msgr2=0x560658482c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x560658482c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 -- 172.21.15.116:0/1526081092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 msgr2=0x560658483160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x560658483160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 -- 172.21.15.116:0/1526081092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560658483ed0 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe51d91640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efe440137f0 tx=0x7efe44009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15c8885be9f1deb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.300+0000 7efe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44010df0 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7efe44004b20 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe440109b0 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efe4403f050 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 10 client.15246 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5606584f0010 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe440109b0 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7efe4402a020 con 0x560658478b50 2024-03-19T02:27:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 10 client.15246 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 10 client.15246 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 20 client.15246 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5606584f48f0 0x5606584f3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 4 client.15246 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "status"}) -- 0x5606583df670 con 0x5606584f48f0 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.301+0000 7efe51590640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5606584f48f0 0x5606584f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe51590640 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5606584f48f0 0x5606584f3600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea411ce4889f09f9 server_cookie=acb9257babb5c68f in_seq=0 out_seq=0 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe34ff9640 10 client.15246 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe34ff9640 1 -- 172.21.15.116:0/1526081092 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5606583df670 con 0x5606584f48f0 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe34ff9640 10 client.15246 handle_command_reply: tid=0 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 client.15246 shutdown 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 msgr2=0x5606584836a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efe440137f0 tx=0x7efe44009a70 comp rx=0 tx=0).stop 2024-03-19T02:27:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 shutdown_connections 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5606584f48f0 0x5606584f3600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560658478b50 0x5606584836a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5606584739e0 0x560658483160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe34ff9640 0 client.15246 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 --2- 172.21.15.116:0/1526081092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560658472ff0 0x560658482c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 >> 172.21.15.116:0/1526081092 conn(0x56065846e560 msgr2=0x560658469670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 shutdown_connections 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 1 -- 172.21.15.116:0/1526081092 wait complete. 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.302+0000 7efe563e9740 20 client.15246 trim_cache size 0 max 16384 2024-03-19T02:27:01.310 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24463, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 86.598450228999994, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 92, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 131.26431468300001 2024-03-19T02:27:01.311 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 msgr2=0x7efe380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 0x7efe380a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x560658474240 tx=0x7efe44004ab0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 msgr2=0x7efe4c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 shutdown_connections 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efe380a2800 0x7efe380a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe4c1022e0 0x7efe4c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe4c101920 0x7efe4c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 --2- 172.21.15.116:0/1573428170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe4c100730 0x7efe4c197b00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 >> 172.21.15.116:0/1573428170 conn(0x7efe4c0fbcc0 msgr2=0x7efe4c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 shutdown_connections 2024-03-19T02:27:01.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.303+0000 7efe537f4640 1 -- 172.21.15.116:0/1573428170 wait complete. 2024-03-19T02:27:01.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T02:27:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1203161012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4101950 msgr2=0x7f01e4101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1203161012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4101950 0x7f01e4101dd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009900 tx=0x7f01d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1203161012 shutdown_connections 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1203161012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 0x7f01e4108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1203161012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4101950 0x7f01e4101dd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1203161012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4100760 0x7f01e4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1203161012 >> 172.21.15.116:0/1203161012 conn(0x7f01e40fbcd0 msgr2=0x7f01e40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1203161012 shutdown_connections 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1203161012 wait complete. 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 Processor -- start 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.416+0000 7f01ec4e8640 1 -- start start 2024-03-19T02:27:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4101950 0x7f01e419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 0x7f01e419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01e410c290 con 0x7f01e4100760 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01e410c110 con 0x7f01e4102310 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f01e410c410 con 0x7f01e4101950 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50606/0 (socket says 172.21.15.116:50606) 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 -- 172.21.15.116:0/1173143965 learned_addr learned my addr 172.21.15.116:0/1173143965 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01e9a83640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4101950 0x7f01e419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01eaa85640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 0x7f01e419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 -- 172.21.15.116:0/1173143965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4101950 msgr2=0x7f01e419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4101950 0x7f01e419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 -- 172.21.15.116:0/1173143965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 msgr2=0x7f01e419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 0x7f01e419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 -- 172.21.15.116:0/1173143965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e419f740 con 0x7f01e4100760 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ea284640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01d400eb70 tx=0x7f01d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01e37fe640 1 -- 172.21.15.116:0/1173143965 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4003b70 con 0x7f01e4100760 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e419f9d0 con 0x7f01e4100760 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01e37fe640 1 -- 172.21.15.116:0/1173143965 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d400ee70 con 0x7f01e4100760 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01e37fe640 1 -- 172.21.15.116:0/1173143965 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4010650 con 0x7f01e4100760 2024-03-19T02:27:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.417+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e419fef0 con 0x7f01e4100760 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.420+0000 7f01e37fe640 1 -- 172.21.15.116:0/1173143965 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f01d4004020 con 0x7f01e4100760 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.420+0000 7f01ef1ad740 1 Processor -- start 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 -- start start 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7afe8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7afed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df7af48de0 con 0x55df7afe8df0 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df7aef1c80 con 0x55df7afe8400 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df7af44e80 con 0x55df7afedf60 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e37fe640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 0x7f01cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01ea284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7afed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01eaa85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7afe8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01eaa85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7afe8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51442/0 (socket says 172.21.15.116:51442) 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36570/0 (socket says 172.21.15.116:36570) 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01eaa85640 1 -- 172.21.15.116:0/3790116848 learned_addr learned my addr 172.21.15.116:0/3790116848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01eaa85640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 0x7f01cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e37fe640 1 -- 172.21.15.116:0/1173143965 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f01d40cf4a0 con 0x7f01e4100760 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 -- 172.21.15.116:0/3790116848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 msgr2=0x55df7afe8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7afe8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 -- 172.21.15.116:0/3790116848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 msgr2=0x55df7afed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7afed8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.421+0000 7f01e9a83640 1 -- 172.21.15.116:0/3790116848 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df7af03fb0 con 0x55df7afedf60 2024-03-19T02:27:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d793845c4d6305db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01eaa85640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 0x7f01cc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55df7afe9650 tx=0x7f01dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01e0bfe640 1 -- 172.21.15.116:0/3790116848 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8045020 con 0x55df7afedf60 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01e0bfe640 1 -- 172.21.15.116:0/3790116848 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d8004040 con 0x55df7afedf60 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01e0bfe640 1 -- 172.21.15.116:0/3790116848 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8005530 con 0x55df7afedf60 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3790116848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 msgr2=0x55df7aff03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8009560 comp rx=0 tx=0).stop 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3790116848 shutdown_connections 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff03f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7afed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3790116848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7afe8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.422+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3790116848 >> 172.21.15.116:0/3790116848 conn(0x55df7afdf560 msgr2=0x55df7afdf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3790116848 shutdown_connections 2024-03-19T02:27:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3790116848 wait complete. 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- start start 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 Processor -- start 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7aff28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df7aff0eb0 con 0x55df7afe8df0 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df7af04fc0 con 0x55df7afe8400 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df7aff1030 con 0x55df7afedf60 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ef1ad740 10 client.0 _ll_register_callbacks cb 0x55df7af40a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50616/0 (socket says 172.21.15.116:50616) 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01ea284640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7aff28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01e9a83640 1 -- 172.21.15.116:0/3914965375 learned_addr learned my addr 172.21.15.116:0/3914965375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01eaa85640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01e9a83640 1 -- 172.21.15.116:0/3914965375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 msgr2=0x55df7aff9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.423+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01e9a83640 1 -- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 msgr2=0x55df7aff28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7aff28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01e9a83640 1 -- 172.21.15.116:0/3914965375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df7ac5d2d0 con 0x55df7afe8df0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ea284640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7aff28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01d80054e0 tx=0x7f01d8005510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929521ea2f78ed8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8045070 con 0x55df7afe8df0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d8004040 con 0x55df7afe8df0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f01d80046c0 con 0x55df7afe8df0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01d805a050 con 0x55df7afe8df0 2024-03-19T02:27:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 10 client.15258 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df7ad276b0 con 0x55df7afe8df0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d8004040 con 0x55df7afe8df0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f01d8003920 con 0x55df7afe8df0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 10 client.15258 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 10 client.15258 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.424+0000 7f01ef1ad740 20 client.15258 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7afff2f0 0x55df7b000760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01ef1ad740 4 client.15258 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df7af56320 con 0x55df7afff2f0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01ea284640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7afff2f0 0x55df7b000760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01ea284640 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7afff2f0 0x55df7b000760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89d1bbca29baff4d server_cookie=aa04d6041eeca4be in_seq=0 out_seq=0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.425+0000 7f01c9ffb640 10 client.15258 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01c9ffb640 1 -- 172.21.15.116:0/3914965375 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df7af56320 con 0x55df7afff2f0 2024-03-19T02:27:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01c9ffb640 10 client.15258 handle_command_reply: tid=0 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 client.15258 shutdown 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 msgr2=0x55df7aff2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01d80054e0 tx=0x7f01d8005510 comp rx=0 tx=0).stop 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 shutdown_connections 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7afff2f0 0x55df7b000760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afedf60 0x55df7aff9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff2e00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01c9ffb640 0 client.15258 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3914965375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afe8400 0x55df7aff28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 >> 172.21.15.116:0/3914965375 conn(0x55df7afdf560 msgr2=0x55df7afde7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 shutdown_connections 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3914965375 wait complete. 2024-03-19T02:27:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.426+0000 7f01ef1ad740 20 client.15258 trim_cache size 0 max 16384 2024-03-19T02:27:01.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 Processor -- start 2024-03-19T02:27:01.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 -- start start 2024-03-19T02:27:01.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 0x55df7aff9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df7affeaa0 con 0x55df7afe8400 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df7afff170 con 0x55df7afedf60 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df7afff840 con 0x55df7afe8df0 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50622/0 (socket says 172.21.15.116:50622) 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 -- 172.21.15.116:0/3155918197 learned_addr learned my addr 172.21.15.116:0/3155918197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01eaa85640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 0x55df7aff9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 -- 172.21.15.116:0/3155918197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 msgr2=0x55df7aff9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 0x55df7aff9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 -- 172.21.15.116:0/3155918197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 msgr2=0x55df7aff2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 -- 172.21.15.116:0/3155918197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df7afe91d0 con 0x55df7afe8400 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01e9a83640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 0x55df7aff9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01ea284640 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f01d4098450 tx=0x7f01d406d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8315958befc5eed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.444+0000 7f01cbfff640 1 -- 172.21.15.116:0/3155918197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d409c180 con 0x55df7afe8400 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01cbfff640 1 -- 172.21.15.116:0/3155918197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d409ba80 con 0x55df7afe8400 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01cbfff640 1 -- 172.21.15.116:0/3155918197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d40980f0 con 0x55df7afe8400 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3155918197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 msgr2=0x55df7aff0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f01d4098450 tx=0x7f01d406d700 comp rx=0 tx=0).stop 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3155918197 shutdown_connections 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8df0 0x55df7aff9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/3155918197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8400 0x55df7aff0350 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3155918197 >> 172.21.15.116:0/3155918197 conn(0x55df7afdf560 msgr2=0x55df7aff13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3155918197 shutdown_connections 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- 172.21.15.116:0/3155918197 wait complete. 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 -- start start 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.445+0000 7f01ef1ad740 1 Processor -- start 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 0x55df7aff6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df7b0005e0 con 0x55df7afe8df0 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df7affff10 con 0x55df7afedf60 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55df7b000cb0 con 0x55df7afe8400 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 10 client.0 _ll_register_callbacks cb 0x7f01d4003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50632/0 (socket says 172.21.15.116:50632) 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 -- 172.21.15.116:0/1093713699 learned_addr learned my addr 172.21.15.116:0/1093713699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01eaa85640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ea284640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 0x55df7aff6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 -- 172.21.15.116:0/1093713699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 msgr2=0x55df7aff6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 0x55df7aff6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 -- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 msgr2=0x55df7aff8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 -- 172.21.15.116:0/1093713699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df7aff9480 con 0x55df7afe8df0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ea284640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 0x55df7aff6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01e9a83640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9103979ac1d8ad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8045040 con 0x55df7afe8df0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f01d80035f0 con 0x55df7afe8df0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d409ba80 con 0x55df7afe8df0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01d8004020 con 0x55df7afe8df0 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.446+0000 7f01ef1ad740 10 client.15264 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df7b065410 con 0x55df7afe8df0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01d409ba80 con 0x55df7afe8df0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f01d8041bb0 con 0x55df7afe8df0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 10 client.15264 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 10 client.15264 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 20 client.15264 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7b069e40 0x55df7b066950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 4 client.15264 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "status"}) -- 0x55df7af50670 con 0x55df7b069e40 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ea284640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7b069e40 0x55df7b066950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01c9ffb640 10 client.15264 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.447+0000 7f01ea284640 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7b069e40 0x55df7b066950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a0d243fa15a4b69 server_cookie=de1665c8a892bb66 in_seq=0 out_seq=0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01c9ffb640 1 -- 172.21.15.116:0/1093713699 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55df7af50670 con 0x55df7b069e40 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01c9ffb640 10 client.15264 handle_command_reply: tid=0 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 client.15264 shutdown 2024-03-19T02:27:01.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 msgr2=0x55df7aff6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 shutdown_connections 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55df7b069e40 0x55df7b066950 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df7afedf60 0x55df7aff8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df7afe8df0 0x55df7aff6570 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01c9ffb640 0 client.15264 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 --2- 172.21.15.116:0/1093713699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df7afe8400 0x55df7aff6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 >> 172.21.15.116:0/1093713699 conn(0x55df7afdf560 msgr2=0x55df7afdf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 shutdown_connections 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 1 -- 172.21.15.116:0/1093713699 wait complete. 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ef1ad740 20 client.15264 trim_cache size 0 max 16384 2024-03-19T02:27:01.455 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24470, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 87.750366873999994, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 92, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 128.424290418 2024-03-19T02:27:01.456 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:01.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 msgr2=0x7f01cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 0x7f01cc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55df7afe9650 tx=0x7f01dc005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:01.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 msgr2=0x7f01e406a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01d400eb70 tx=0x7f01d400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 shutdown_connections 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f01cc0a2800 0x7f01cc0a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01e4102310 0x7f01e419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01e4101950 0x7f01e419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 --2- 172.21.15.116:0/1173143965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e4100760 0x7f01e406a9a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 >> 172.21.15.116:0/1173143965 conn(0x7f01e40fbcd0 msgr2=0x7f01e4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 shutdown_connections 2024-03-19T02:27:01.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.448+0000 7f01ec4e8640 1 -- 172.21.15.116:0/1173143965 wait complete. 2024-03-19T02:27:01.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T02:27:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 -- 172.21.15.116:0/2727316985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84101830 msgr2=0x7fbb84101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 --2- 172.21.15.116:0/2727316985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84101830 0x7fbb84101c50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009900 tx=0x7fbb7402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 -- 172.21.15.116:0/2727316985 shutdown_connections 2024-03-19T02:27:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 --2- 172.21.15.116:0/2727316985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 0x7fbb84109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 --2- 172.21.15.116:0/2727316985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84102a20 0x7fbb84102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 --2- 172.21.15.116:0/2727316985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84101830 0x7fbb84101c50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 -- 172.21.15.116:0/2727316985 >> 172.21.15.116:0/2727316985 conn(0x7fbb840fcdc0 msgr2=0x7fbb840ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 -- 172.21.15.116:0/2727316985 shutdown_connections 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.562+0000 7fbb88f80640 1 -- 172.21.15.116:0/2727316985 wait complete. 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 Processor -- start 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 -- start start 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84102a20 0x7fbb84197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 0x7fbb8419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8410d360 con 0x7fbb84101830 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8410d1e0 con 0x7fbb841033e0 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb88f80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8410d4e0 con 0x7fbb84102a20 2024-03-19T02:27:01.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50642/0 (socket says 172.21.15.116:50642) 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 -- 172.21.15.116:0/683037655 learned_addr learned my addr 172.21.15.116:0/683037655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84102a20 0x7fbb84197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 0x7fbb8419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 -- 172.21.15.116:0/683037655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84102a20 msgr2=0x7fbb84197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84102a20 0x7fbb84197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 -- 172.21.15.116:0/683037655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 msgr2=0x7fbb8419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 0x7fbb8419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.563+0000 7fbb8259c640 1 -- 172.21.15.116:0/683037655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8419f6c0 con 0x7fbb84101830 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb8259c640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb74000c00 tx=0x7fbb74005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb737fe640 1 -- 172.21.15.116:0/683037655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74043020 con 0x7fbb84101830 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8419f950 con 0x7fbb84101830 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb737fe640 1 -- 172.21.15.116:0/683037655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb74004350 con 0x7fbb84101830 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb737fe640 1 -- 172.21.15.116:0/683037655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74003b80 con 0x7fbb84101830 2024-03-19T02:27:01.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.564+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb8419fef0 con 0x7fbb84101830 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.566+0000 7fbb737fe640 1 -- 172.21.15.116:0/683037655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbb74046020 con 0x7fbb84101830 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 Processor -- start 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 -- start start 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564ba444ede0 con 0x564ba44efb50 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ba43f7c80 con 0x564ba44ea9e0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ba444ae80 con 0x564ba44e9ff0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb737fe640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 0x7fbb6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb8259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50658/0 (socket says 172.21.15.116:50658) 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 -- 172.21.15.116:0/2632081005 learned_addr learned my addr 172.21.15.116:0/2632081005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 0x7fbb6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb737fe640 1 -- 172.21.15.116:0/683037655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbb740f0f10 con 0x7fbb84101830 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 -- 172.21.15.116:0/2632081005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 msgr2=0x564ba44ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 -- 172.21.15.116:0/2632081005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 msgr2=0x564ba44ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.567+0000 7fbb81d9b640 1 -- 172.21.15.116:0/2632081005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ba4409fb0 con 0x564ba44efb50 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.568+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 0x7fbb6c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x564ba44eb240 tx=0x7fbb7c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.568+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009c90 tx=0x7fbb78011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab3350dabf75357 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.568+0000 7fbb70ff9640 1 -- 172.21.15.116:0/2632081005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801e070 con 0x564ba44efb50 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.568+0000 7fbb70ff9640 1 -- 172.21.15.116:0/2632081005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb78009e40 con 0x564ba44efb50 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.568+0000 7fbb70ff9640 1 -- 172.21.15.116:0/2632081005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7800e8f0 con 0x564ba44efb50 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- 172.21.15.116:0/2632081005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 msgr2=0x564ba44f1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009c90 tx=0x7fbb78011810 comp rx=0 tx=0).stop 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- 172.21.15.116:0/2632081005 shutdown_connections 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44f1fe0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/2632081005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- 172.21.15.116:0/2632081005 >> 172.21.15.116:0/2632081005 conn(0x564ba44e5560 msgr2=0x564ba44e5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- 172.21.15.116:0/2632081005 shutdown_connections 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- 172.21.15.116:0/2632081005 wait complete. 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- start start 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 Processor -- start 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44ea9e0 0x564ba44f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564ba44f2b00 con 0x564ba44e9ff0 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ba440afc0 con 0x564ba44efb50 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ba44f2c80 con 0x564ba44ea9e0 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8bb68740 10 client.0 _ll_register_callbacks cb 0x564ba4446a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50660/0 (socket says 172.21.15.116:50660) 2024-03-19T02:27:01.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb8259c640 1 -- 172.21.15.116:0/1105510242 learned_addr learned my addr 172.21.15.116:0/1105510242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44ea9e0 0x564ba44f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.569+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 -- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44ea9e0 msgr2=0x564ba44f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44ea9e0 0x564ba44f4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 -- 172.21.15.116:0/1105510242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 msgr2=0x564ba44fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 -- 172.21.15.116:0/1105510242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ba41632d0 con 0x564ba44e9ff0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8259c640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbb7402c4e0 tx=0x7fbb7402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc50beea4144685 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb740bd090 con 0x564ba44e9ff0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbb740c1400 con 0x564ba44e9ff0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ba445c320 con 0x564ba44e9ff0 2024-03-19T02:27:01.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbb740bceb0 con 0x564ba44e9ff0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 10 client.15276 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ba422d6b0 con 0x564ba44e9ff0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564ba445c320 con 0x564ba44e9ff0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbb740c5860 con 0x564ba44e9ff0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 10 client.15276 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 20 client.15276 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba4501390 0x564ba4502800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 4 client.15276 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.570+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ba4456670 con 0x564ba4501390 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.571+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba4501390 0x564ba4502800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.571+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba4501390 0x564ba4502800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e8db2bfb76dfa6f server_cookie=c2038915e42ef836 in_seq=0 out_seq=0 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.571+0000 7fbb69ffb640 10 client.15276 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:01.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb69ffb640 1 -- 172.21.15.116:0/1105510242 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ba4456670 con 0x564ba4501390 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb69ffb640 10 client.15276 handle_command_reply: tid=0 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 client.15276 shutdown 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 msgr2=0x564ba44f4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbb7402c4e0 tx=0x7fbb7402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 shutdown_connections 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba4501390 0x564ba4502800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb69ffb640 0 client.15276 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44ea9e0 0x564ba44f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1105510242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44e9ff0 0x564ba44f4590 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 >> 172.21.15.116:0/1105510242 conn(0x564ba44e5560 msgr2=0x564ba44e47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 shutdown_connections 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1105510242 wait complete. 2024-03-19T02:27:01.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.572+0000 7fbb8bb68740 20 client.15276 trim_cache size 0 max 16384 2024-03-19T02:27:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 Processor -- start 2024-03-19T02:27:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 -- start start 2024-03-19T02:27:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44eb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44e5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564ba4500b40 con 0x564ba44ea9e0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ba4501210 con 0x564ba44efb50 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ba45018e0 con 0x564ba44e9ff0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50674/0 (socket says 172.21.15.116:50674) 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 -- 172.21.15.116:0/1709120510 learned_addr learned my addr 172.21.15.116:0/1709120510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb8259c640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 -- 172.21.15.116:0/1709120510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 msgr2=0x564ba44eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 -- 172.21.15.116:0/1709120510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 msgr2=0x564ba44e5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44e5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 -- 172.21.15.116:0/1709120510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ba44e0fc0 con 0x564ba44ea9e0 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:27:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x564ba44e64d0 tx=0x7fbb78009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b815938a40e6088 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb6bfff640 1 -- 172.21.15.116:0/1709120510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801e040 con 0x564ba44ea9e0 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb6bfff640 1 -- 172.21.15.116:0/1709120510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb780098c0 con 0x564ba44ea9e0 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.590+0000 7fbb6bfff640 1 -- 172.21.15.116:0/1709120510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb78003b40 con 0x564ba44ea9e0 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1709120510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 msgr2=0x564ba44fbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x564ba44e64d0 tx=0x7fbb78009850 comp rx=0 tx=0).stop 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1709120510 shutdown_connections 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44efb50 0x564ba44e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44ea9e0 0x564ba44fbaa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/1709120510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1709120510 >> 172.21.15.116:0/1709120510 conn(0x564ba44e5560 msgr2=0x564ba44ebee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1709120510 shutdown_connections 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- 172.21.15.116:0/1709120510 wait complete. 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 -- start start 2024-03-19T02:27:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.591+0000 7fbb8bb68740 1 Processor -- start 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44f7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44f7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564ba4502680 con 0x564ba44efb50 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ba4501fb0 con 0x564ba44ea9e0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ba4502d50 con 0x564ba44e9ff0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 10 client.0 _ll_register_callbacks cb 0x7fbb740057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50690/0 (socket says 172.21.15.116:50690) 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 -- 172.21.15.116:0/204134594 learned_addr learned my addr 172.21.15.116:0/204134594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb81d9b640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8259c640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 -- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 msgr2=0x564ba44f7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44f7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 -- 172.21.15.116:0/204134594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 msgr2=0x564ba44f7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44f7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 -- 172.21.15.116:0/204134594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ba44fade0 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb82d9d640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005fd0 tx=0x7fbb7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d01057de81c766b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c02e060 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbb7c020450 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ba43d4a00 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbb7c035050 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 10 client.15282 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.592+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ba45671d0 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564ba43d4a00 con 0x564ba44efb50 2024-03-19T02:27:01.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbb7c028070 con 0x564ba44efb50 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 10 client.15282 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 20 client.15282 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba456bec0 0x564ba456abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 4 client.15282 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "status"}) -- 0x564ba4420900 con 0x564ba456bec0 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8259c640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba456bec0 0x564ba456abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8259c640 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba456bec0 0x564ba456abd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a36af38df78ec5dd server_cookie=59a793542ad6367d in_seq=0 out_seq=0 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb69ffb640 10 client.15282 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb69ffb640 1 -- 172.21.15.116:0/204134594 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564ba4420900 con 0x564ba456bec0 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb69ffb640 10 client.15282 handle_command_reply: tid=0 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.593+0000 7fbb8bb68740 1 client.15282 shutdown 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 msgr2=0x564ba44fa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005fd0 tx=0x7fbb7c004a40 comp rx=0 tx=0).stop 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 shutdown_connections 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564ba456bec0 0x564ba456abd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564ba44efb50 0x564ba44fa7f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ba44ea9e0 0x564ba44f7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 --2- 172.21.15.116:0/204134594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ba44e9ff0 0x564ba44f7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb69ffb640 0 client.15282 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 >> 172.21.15.116:0/204134594 conn(0x564ba44e5560 msgr2=0x564ba44f41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 shutdown_connections 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 1 -- 172.21.15.116:0/204134594 wait complete. 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb8bb68740 20 client.15282 trim_cache size 0 max 16384 2024-03-19T02:27:01.602 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24484, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 88.907858305999994, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 92, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 125.77633873400001 2024-03-19T02:27:01.603 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 msgr2=0x7fbb6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 0x7fbb6c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x564ba44eb240 tx=0x7fbb7c009500 comp rx=0 tx=0).stop 2024-03-19T02:27:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 msgr2=0x7fbb84197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb74000c00 tx=0x7fbb74005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 shutdown_connections 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbb6c0a2800 0x7fbb6c0a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb841033e0 0x7fbb8419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb84102a20 0x7fbb84197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 --2- 172.21.15.116:0/683037655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb84101830 0x7fbb84197aa0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 >> 172.21.15.116:0/683037655 conn(0x7fbb840fcdc0 msgr2=0x7fbb84105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 shutdown_connections 2024-03-19T02:27:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.594+0000 7fbb88f80640 1 -- 172.21.15.116:0/683037655 wait complete. 2024-03-19T02:27:01.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- 172.21.15.116:0/3701529670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 msgr2=0x7f4cfc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/3701529670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc100b40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009900 tx=0x7f4cf002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- 172.21.15.116:0/3701529670 shutdown_connections 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/3701529670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/3701529670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/3701529670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc100b40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- 172.21.15.116:0/3701529670 >> 172.21.15.116:0/3701529670 conn(0x7f4cfc0fbcb0 msgr2=0x7f4cfc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- 172.21.15.116:0/3701529670 shutdown_connections 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- 172.21.15.116:0/3701529670 wait complete. 2024-03-19T02:27:01.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 Processor -- start 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- start start 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc10c2a0 con 0x7f4cfc101910 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc10c120 con 0x7f4cfc1022d0 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.710+0000 7f4d02aa0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc10c420 con 0x7f4cfc100720 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50694/0 (socket says 172.21.15.116:50694) 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4d0103d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 -- 172.21.15.116:0/1868273077 learned_addr learned my addr 172.21.15.116:0/1868273077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4d0083c640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 -- 172.21.15.116:0/1868273077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 msgr2=0x7f4cfc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 -- 172.21.15.116:0/1868273077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 msgr2=0x7f4cfc19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 -- 172.21.15.116:0/1868273077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc19e5c0 con 0x7f4cfc101910 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800ea60 tx=0x7f4ce800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cf9ffb640 1 -- 172.21.15.116:0/1868273077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce800cde0 con 0x7f4cfc101910 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc19e8b0 con 0x7f4cfc101910 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cf9ffb640 1 -- 172.21.15.116:0/1868273077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ce80035f0 con 0x7f4cfc101910 2024-03-19T02:27:01.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4cf9ffb640 1 -- 172.21.15.116:0/1868273077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8010690 con 0x7f4cfc101910 2024-03-19T02:27:01.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.711+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc19ee00 con 0x7f4cfc101910 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.714+0000 7f4cf9ffb640 1 -- 172.21.15.116:0/1868273077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4ce80037a0 con 0x7f4cfc101910 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.714+0000 7f4d05685740 1 Processor -- start 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.714+0000 7f4d05685740 1 -- start start 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.714+0000 7f4cf9ffb640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 0x7f4cec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.714+0000 7f4d05685740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d05685740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830ba780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d05685740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830c42f0 0x556a830c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d05685740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a8301ede0 con 0x556a830bb670 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 0x7f4cec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d05685740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a82fc7c80 con 0x556a830ba300 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d05685740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a8301ae80 con 0x556a830c42f0 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4cf9ffb640 1 -- 172.21.15.116:0/1868273077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f4ce80ceff0 con 0x7f4cfc101910 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50702/0 (socket says 172.21.15.116:50702) 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 -- 172.21.15.116:0/1070109253 learned_addr learned my addr 172.21.15.116:0/1070109253 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830ba780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 -- 172.21.15.116:0/1070109253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830c42f0 msgr2=0x556a830c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830c42f0 0x556a830c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 -- 172.21.15.116:0/1070109253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 msgr2=0x556a830ba780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830ba780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 -- 172.21.15.116:0/1070109253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a82fd9fb0 con 0x556a830bb670 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 0x7f4cec0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009fd0 tx=0x7f4cf400fb40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4d0083c640 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000c00 tx=0x7f4cf00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e64c0e1580f884 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4ce6ffd640 1 -- 172.21.15.116:0/1070109253 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0045020 con 0x556a830bb670 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4ce6ffd640 1 -- 172.21.15.116:0/1070109253 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cf0004040 con 0x556a830bb670 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.715+0000 7f4ce6ffd640 1 -- 172.21.15.116:0/1070109253 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0005560 con 0x556a830bb670 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- 172.21.15.116:0/1070109253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 msgr2=0x556a830b9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000c00 tx=0x7f4cf00095f0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- 172.21.15.116:0/1070109253 shutdown_connections 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830c42f0 0x556a830c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830ba780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 --2- 172.21.15.116:0/1070109253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830b9dc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- 172.21.15.116:0/1070109253 >> 172.21.15.116:0/1070109253 conn(0x556a830b5560 msgr2=0x556a830b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- 172.21.15.116:0/1070109253 shutdown_connections 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- 172.21.15.116:0/1070109253 wait complete. 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 -- start start 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.716+0000 7f4d05685740 1 Processor -- start 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830ba300 0x556a830c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830bb670 0x556a830c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a830c6ec0 con 0x556a830c42f0 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a82fdafc0 con 0x556a830bb670 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a830c7040 con 0x556a830ba300 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 10 client.0 _ll_register_callbacks cb 0x556a83016a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50714/0 (socket says 172.21.15.116:50714) 2024-03-19T02:27:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 -- 172.21.15.116:0/828371819 learned_addr learned my addr 172.21.15.116:0/828371819 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0083c640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830ba300 0x556a830c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830bb670 0x556a830c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 -- 172.21.15.116:0/828371819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830ba300 msgr2=0x556a830c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830ba300 0x556a830c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 -- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830bb670 msgr2=0x556a830c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830bb670 0x556a830c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 -- 172.21.15.116:0/828371819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a82d332d0 con 0x556a830c42f0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d0103d640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4003ba0 tx=0x7f4cf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643425f2a67fbea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4cdbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4017070 con 0x556a830c42f0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f4cf4026400 con 0x556a830c42f0 2024-03-19T02:27:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.717+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cf0004040 con 0x556a830c42f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4cf4026600 con 0x556a830c42f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 10 client.15294 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a82dfd6b0 con 0x556a830c42f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cf0004040 con 0x556a830c42f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4cf401e780 con 0x556a830c42f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 10 client.15294 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 10 client.15294 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 20 client.15294 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830d56f0 0x556a830d6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 4 client.15294 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a8302c320 con 0x556a830d56f0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.718+0000 7f4d0083c640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830d56f0 0x556a830d6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d0083c640 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830d56f0 0x556a830d6b60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e0cccd43c91a92 server_cookie=719f2f6ccd4a394d in_seq=0 out_seq=0 2024-03-19T02:27:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4cdbfff640 10 client.15294 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4cdbfff640 1 -- 172.21.15.116:0/828371819 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a8302c320 con 0x556a830d56f0 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4cdbfff640 10 client.15294 handle_command_reply: tid=0 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 client.15294 shutdown 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 msgr2=0x556a830cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4003ba0 tx=0x7f4cf4004a40 comp rx=0 tx=0).stop 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 shutdown_connections 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830d56f0 0x556a830d6b60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830c42f0 0x556a830cfdb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830bb670 0x556a830c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4cdbfff640 0 client.15294 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 --2- 172.21.15.116:0/828371819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830ba300 0x556a830c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.719+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 >> 172.21.15.116:0/828371819 conn(0x556a830b5560 msgr2=0x556a830b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.720+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 shutdown_connections 2024-03-19T02:27:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.720+0000 7f4d05685740 1 -- 172.21.15.116:0/828371819 wait complete. 2024-03-19T02:27:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.720+0000 7f4d05685740 20 client.15294 trim_cache size 0 max 16384 2024-03-19T02:27:01.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 Processor -- start 2024-03-19T02:27:01.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 -- start start 2024-03-19T02:27:01.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830b5560 0x556a830d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a830d4ea0 con 0x556a830bb670 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a830d5570 con 0x556a830ba300 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.737+0000 7f4d05685740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a830d5c40 con 0x556a830b5560 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50718/0 (socket says 172.21.15.116:50718) 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 -- 172.21.15.116:0/1892195532 learned_addr learned my addr 172.21.15.116:0/1892195532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0083c640 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830b5560 0x556a830d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 -- 172.21.15.116:0/1892195532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830b5560 msgr2=0x556a830d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830b5560 0x556a830d0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 -- 172.21.15.116:0/1892195532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 msgr2=0x556a830cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 -- 172.21.15.116:0/1892195532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a830b62d0 con 0x556a830bb670 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4d0103d640 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4011830 tx=0x7f4cf400f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e420257066d039 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4ce67fc640 1 -- 172.21.15.116:0/1892195532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4017040 con 0x556a830bb670 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4ce67fc640 1 -- 172.21.15.116:0/1892195532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cf4038070 con 0x556a830bb670 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.738+0000 7f4ce67fc640 1 -- 172.21.15.116:0/1892195532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf401ee60 con 0x556a830bb670 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- 172.21.15.116:0/1892195532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 msgr2=0x556a830c0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4011830 tx=0x7f4cf400f560 comp rx=0 tx=0).stop 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- 172.21.15.116:0/1892195532 shutdown_connections 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830bb670 0x556a830c0730 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830ba300 0x556a830cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- 172.21.15.116:0/1892195532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830b5560 0x556a830d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- 172.21.15.116:0/1892195532 >> 172.21.15.116:0/1892195532 conn(0x556a830b9f20 msgr2=0x556a830d44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- 172.21.15.116:0/1892195532 shutdown_connections 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- 172.21.15.116:0/1892195532 wait complete. 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- start start 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 Processor -- start 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830b5560 0x556a830cb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830bb670 0x556a830ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a830d69e0 con 0x556a830ba300 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a830d6310 con 0x556a830b5560 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a830d70b0 con 0x556a830bb670 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.739+0000 7f4d05685740 10 client.0 _ll_register_callbacks cb 0x7f4cf400fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50734/0 (socket says 172.21.15.116:50734) 2024-03-19T02:27:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 -- 172.21.15.116:0/4079397459 learned_addr learned my addr 172.21.15.116:0/4079397459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4d0083c640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830b5560 0x556a830cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4d0103d640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830bb670 0x556a830ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 -- 172.21.15.116:0/4079397459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830bb670 msgr2=0x556a830ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830bb670 0x556a830ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 -- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830b5560 msgr2=0x556a830cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830b5560 0x556a830cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 -- 172.21.15.116:0/4079397459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a830cf340 con 0x556a830ba300 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cfbfff640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80ccb00 tx=0x7f4ce8009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12959987ed04635 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cdbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80b8070 con 0x556a830ba300 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f4ce80b4250 con 0x556a830ba300 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cf4038070 con 0x556a830ba300 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ce8082050 con 0x556a830ba300 2024-03-19T02:27:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4d05685740 10 client.15300 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a8313b7d0 con 0x556a830ba300 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.740+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cf4038070 con 0x556a830ba300 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4ce8084bc0 con 0x556a830ba300 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 10 client.15300 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 20 client.15300 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830c42f0 0x556a8313f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 4 client.15300 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "status"}) -- 0x556a83026670 con 0x556a830c42f0 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d0083c640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830c42f0 0x556a8313f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d0083c640 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830c42f0 0x556a8313f0f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c403474057ebd8 server_cookie=ac9efe22c203dbba in_seq=0 out_seq=0 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4cdbfff640 10 client.15300 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4cdbfff640 1 -- 172.21.15.116:0/4079397459 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556a83026670 con 0x556a830c42f0 2024-03-19T02:27:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4cdbfff640 10 client.15300 handle_command_reply: tid=0 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 1 client.15300 shutdown 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 msgr2=0x556a830cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.741+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80ccb00 tx=0x7f4ce8009490 comp rx=0 tx=0).stop 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 shutdown_connections 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a830c42f0 0x556a8313f0f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a830bb670 0x556a830ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4cdbfff640 0 client.15300 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a830ba300 0x556a830cbd80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 --2- 172.21.15.116:0/4079397459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a830b5560 0x556a830cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 >> 172.21.15.116:0/4079397459 conn(0x556a830b9f20 msgr2=0x556a830c0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 shutdown_connections 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 1 -- 172.21.15.116:0/4079397459 wait complete. 2024-03-19T02:27:01.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d05685740 20 client.15300 trim_cache size 0 max 16384 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24494, 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 113.09174720999999, 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 92, 2024-03-19T02:27:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 117.336286124 2024-03-19T02:27:01.751 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 msgr2=0x7f4cec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 0x7f4cec0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009fd0 tx=0x7f4cf400fb40 comp rx=0 tx=0).stop 2024-03-19T02:27:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 msgr2=0x7f4cfc196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800ea60 tx=0x7f4ce800ef20 comp rx=0 tx=0).stop 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 shutdown_connections 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4cec0a2800 0x7f4cec0a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cfc1022d0 0x7f4cfc19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc101910 0x7f4cfc196ee0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 --2- 172.21.15.116:0/1868273077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cfc100720 0x7f4cfc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 >> 172.21.15.116:0/1868273077 conn(0x7f4cfc0fbcb0 msgr2=0x7f4cfc104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 shutdown_connections 2024-03-19T02:27:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:01.742+0000 7f4d02aa0640 1 -- 172.21.15.116:0/1868273077 wait complete. 2024-03-19T02:27:01.754 INFO:tasks.cephadm:Setup complete, yielding 2024-03-19T02:27:01.754 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T02:27:01.767 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:27:01.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-19T02:27:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:01 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4269442664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:01 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4269442664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:01 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4269442664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:02 smithi154 ceph-mon[35693]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:02 smithi116 ceph-mon[32263]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:02 smithi162 ceph-mon[35761]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:03.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 -- 172.21.15.116:0/2878482389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60103540 msgr2=0x7feb601039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:03.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 --2- 172.21.15.116:0/2878482389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60103540 0x7feb601039a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feb500099c0 tx=0x7feb50009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 -- 172.21.15.116:0/2878482389 shutdown_connections 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 --2- 172.21.15.116:0/2878482389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 --2- 172.21.15.116:0/2878482389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60102340 0x7feb60102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 --2- 172.21.15.116:0/2878482389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60103540 0x7feb601039a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.905+0000 7feb64b6f700 1 -- 172.21.15.116:0/2878482389 >> 172.21.15.116:0/2878482389 conn(0x7feb600fd8c0 msgr2=0x7feb600ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 -- 172.21.15.116:0/2878482389 shutdown_connections 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 -- 172.21.15.116:0/2878482389 wait complete. 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 Processor -- start 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 -- start start 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 0x7feb601989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.906+0000 7feb64b6f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010dde0 con 0x7feb60102340 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb64b6f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010dc60 con 0x7feb60103540 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb64b6f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010df60 con 0x7feb60103ee0 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50738/0 (socket says 172.21.15.116:50738) 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 -- 172.21.15.116:0/3616011819 learned_addr learned my addr 172.21.15.116:0/3616011819 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5dd9b700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 0x7feb601989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5ed9d700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 -- 172.21.15.116:0/3616011819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 msgr2=0x7feb6019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 -- 172.21.15.116:0/3616011819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 msgr2=0x7feb601989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 0x7feb601989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 -- 172.21.15.116:0/3616011819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb601a00e0 con 0x7feb60102340 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5dd9b700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 0x7feb601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5ed9d700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb5e59c700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feb5400e9a0 tx=0x7feb5400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54004220 con 0x7feb60102340 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb601a03d0 con 0x7feb60102340 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb540043c0 con 0x7feb60102340 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb54003ce0 con 0x7feb60102340 2024-03-19T02:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.907+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb601a0960 con 0x7feb60102340 2024-03-19T02:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.908+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb60068fd0 con 0x7feb60102340 2024-03-19T02:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.910+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7feb5400c810 con 0x7feb60102340 2024-03-19T02:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.910+0000 7feb4f7fe700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 0x7feb480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.910+0000 7feb5dd9b700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 0x7feb480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.910+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7feb540cf070 con 0x7feb60102340 2024-03-19T02:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.911+0000 7feb5dd9b700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 0x7feb480a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb50000c00 tx=0x7feb50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:03.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:03.914+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb5409be40 con 0x7feb60102340 2024-03-19T02:27:04.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.124+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7feb60062750 con 0x7feb480a2890 2024-03-19T02:27:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.126+0000 7feb4f7fe700 1 -- 172.21.15.116:0/3616011819 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7feb60062750 con 0x7feb480a2890 2024-03-19T02:27:04.126 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-19T02:27:04.126 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-19T02:27:04.126 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-19T02:27:04.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 msgr2=0x7feb480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:04.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 0x7feb480a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feb50000c00 tx=0x7feb50005fb0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 msgr2=0x7feb60198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feb5400e9a0 tx=0x7feb5400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 shutdown_connections 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7feb480a2890 0x7feb480a4d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb60103ee0 0x7feb6019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb60102340 0x7feb60198480 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 --2- 172.21.15.116:0/3616011819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb60103540 0x7feb601989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 >> 172.21.15.116:0/3616011819 conn(0x7feb600fd8c0 msgr2=0x7feb601068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 shutdown_connections 2024-03-19T02:27:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:04.128+0000 7feb64b6f700 1 -- 172.21.15.116:0/3616011819 wait complete. 2024-03-19T02:27:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:04 smithi116 ceph-mon[32263]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:27:04.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-19T02:27:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:04 smithi154 ceph-mon[35693]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:27:05.110 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:04 smithi162 ceph-mon[35761]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:27:05.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:05 smithi116 ceph-mon[32263]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:05 smithi154 ceph-mon[35693]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:05 smithi162 ceph-mon[35761]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 -- 172.21.15.116:0/3840034800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964102360 msgr2=0x7f9964102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 --2- 172.21.15.116:0/3840034800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964102360 0x7f9964102780 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f99540099c0 tx=0x7f9954009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 -- 172.21.15.116:0/3840034800 shutdown_connections 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 --2- 172.21.15.116:0/3840034800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103f00 0x7f996410a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 --2- 172.21.15.116:0/3840034800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964102360 0x7f9964102780 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 --2- 172.21.15.116:0/3840034800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964103560 0x7f99641039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 -- 172.21.15.116:0/3840034800 >> 172.21.15.116:0/3840034800 conn(0x7f99640fd920 msgr2=0x7f99640ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 -- 172.21.15.116:0/3840034800 shutdown_connections 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.883+0000 7f9968cd2700 1 -- 172.21.15.116:0/3840034800 wait complete. 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 Processor -- start 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 -- start start 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964102360 0x7f9964198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103560 0x7f99641989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f996410dde0 con 0x7f9964103f00 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f996410dc60 con 0x7f9964102360 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9968cd2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f996410df60 con 0x7f9964103560 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:06.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50756/0 (socket says 172.21.15.116:50756) 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 -- 172.21.15.116:0/58677876 learned_addr learned my addr 172.21.15.116:0/58677876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9961d9b700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103560 0x7f99641989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f996259c700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964102360 0x7f9964198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 -- 172.21.15.116:0/58677876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103560 msgr2=0x7f99641989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103560 0x7f99641989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 -- 172.21.15.116:0/58677876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964102360 msgr2=0x7f9964198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964102360 0x7f9964198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.884+0000 7f9962d9d700 1 -- 172.21.15.116:0/58677876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99641a00e0 con 0x7f9964103f00 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f9962d9d700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f995c00ef60 tx=0x7f995c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0092e0 con 0x7f9964103f00 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99641a0370 con 0x7f9964103f00 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f995c004bb0 con 0x7f9964103f00 2024-03-19T02:27:06.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c004020 con 0x7f9964103f00 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.885+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99641a08b0 con 0x7f9964103f00 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.886+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9964068fd0 con 0x7f9964103f00 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.887+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f995c01b030 con 0x7f9964103f00 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.887+0000 7f99537fe700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 0x7f994c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.888+0000 7f996259c700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 0x7f994c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.888+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f995c0ce7f0 con 0x7f9964103f00 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.888+0000 7f996259c700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 0x7f994c0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f995400b580 tx=0x7f9954005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:06.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:06.892+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f995c022020 con 0x7f9964103f00 2024-03-19T02:27:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:06 smithi154 ceph-mon[35693]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:06 smithi116 ceph-mon[32263]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:07.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.097+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f9964062750 con 0x7f994c0a2890 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.110+0000 7f99537fe700 1 -- 172.21.15.116:0/58677876 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f9964062750 con 0x7f994c0a2890 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:mds.a smithi116 running (2m) 21s ago 2m 16.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 1e618512a368 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:mds.b smithi154 running (2m) 56s ago 2m 26.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3f46e3f61195 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:mds.c smithi162 running (2m) 56s ago 2m 18.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 004f362370e8 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:mds.d smithi116 running (2m) 21s ago 2m 16.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 f620a04f2f70 2024-03-19T02:27:07.111 INFO:teuthology.orchestra.run.smithi116.stdout:mds.e smithi154 running (2m) 56s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 5b9c314a9099 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.f smithi162 running (2m) 56s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3c3ca9b06120 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.g smithi116 running (2m) 21s ago 2m 16.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 688e51c62030 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.h smithi154 running (2m) 56s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 fbebced225e9 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.i smithi162 running (2m) 56s ago 2m 18.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 e4de0e85a9bc 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.j smithi116 running (2m) 21s ago 2m 15.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 9b06d97a41fe 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.k smithi154 running (2m) 56s ago 2m 16.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3aef6bc0f01c 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mds.l smithi162 running (2m) 56s ago 2m 18.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 11fb4d09207d 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi116 *:9283,8765 running (9m) 21s ago 9m 514M - 19.0.0-2206-gb6ae2491 2312c8e42d00 32a03d90c716 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi154 *:8443,8765 running (7m) 56s ago 7m 452M - 19.0.0-2206-gb6ae2491 2312c8e42d00 c129d349b125 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.z smithi162 *:8443,8765 running (7m) 56s ago 7m 452M - 19.0.0-2206-gb6ae2491 2312c8e42d00 bd5ac1cbffd9 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (9m) 21s ago 10m 50.3M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 e99971b59171 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi154 running (8m) 56s ago 8m 38.6M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 f2d9dcebfc1f 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi162 running (8m) 56s ago 8m 38.5M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 ee7ec7f8e7aa 2024-03-19T02:27:07.112 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 21s ago 7m 46.8M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 ff044e02929c 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi154 running (7m) 56s ago 7m 55.9M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 71cb2d9bb61e 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi162 running (6m) 56s ago 6m 49.5M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 e47de26fe76a 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (6m) 21s ago 6m 45.1M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 c3e7c6272524 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi154 running (5m) 56s ago 5m 50.7M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 bbf5d75fa2ac 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi162 running (5m) 56s ago 5m 45.2M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 640d9be4a59e 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi116 running (5m) 21s ago 5m 50.8M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 888e55a5cf28 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi154 running (4m) 56s ago 4m 53.5M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 ea1e4126f04d 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 smithi162 running (4m) 56s ago 4m 43.4M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 c653484c3c0c 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 smithi116 running (3m) 21s ago 3m 50.8M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 a8f7a4612add 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 smithi154 running (3m) 56s ago 3m 43.3M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 fee7f806bcef 2024-03-19T02:27:07.113 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 smithi162 running (2m) 56s ago 2m 44.2M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 a7c180972453 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 msgr2=0x7f994c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 0x7f994c0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f995400b580 tx=0x7f9954005fb0 comp rx=0 tx=0).stop 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 msgr2=0x7f996419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f995c00ef60 tx=0x7f995c00ef90 comp rx=0 tx=0).stop 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 shutdown_connections 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9964103f00 0x7f996419fa40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9964102360 0x7f9964198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f994c0a2890 0x7f994c0a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 --2- 172.21.15.116:0/58677876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9964103560 0x7f99641989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.112+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 >> 172.21.15.116:0/58677876 conn(0x7f99640fd920 msgr2=0x7f9964106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.113+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 shutdown_connections 2024-03-19T02:27:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:07.113+0000 7f9968cd2700 1 -- 172.21.15.116:0/58677876 wait complete. 2024-03-19T02:27:07.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:06 smithi162 ceph-mon[35761]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:07.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-19T02:27:08.199 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:08 smithi154 ceph-mon[35693]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:08 smithi154 ceph-mon[35693]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:08 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T02:27:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:08 smithi116 ceph-mon[32263]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:08 smithi116 ceph-mon[32263]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:08 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T02:27:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:08 smithi162 ceph-mon[35761]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:08 smithi162 ceph-mon[35761]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:08 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T02:27:10.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:09 smithi116 ceph-mon[32263]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T02:27:10.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:09 smithi116 ceph-mon[32263]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:27:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:09 smithi154 ceph-mon[35693]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T02:27:10.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:09 smithi154 ceph-mon[35693]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:27:10.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/3563739544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4102330 msgr2=0x7fcdc4102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/3563739544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4102330 0x7fcdc4102750 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40099c0 tx=0x7fcdb4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/3563739544 shutdown_connections 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/3563739544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4103ed0 0x7fcdc410a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/3563739544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4102330 0x7fcdc4102750 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/3563739544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103530 0x7fcdc4103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/3563739544 >> 172.21.15.116:0/3563739544 conn(0x7fcdc40fd8f0 msgr2=0x7fcdc40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/3563739544 shutdown_connections 2024-03-19T02:27:10.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/3563739544 wait complete. 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.011+0000 7fcdc8a0a700 1 Processor -- start 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 -- start start 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 0x7fcdc4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103ed0 0x7fcdc419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410ddb0 con 0x7fcdc4103530 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410dc30 con 0x7fcdc4102330 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc8a0a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc410df30 con 0x7fcdc4103ed0 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:10.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc1d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58454/0 (socket says 172.21.15.116:58454) 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 0x7fcdc4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc1d9b700 1 -- 172.21.15.116:0/159050235 learned_addr learned my addr 172.21.15.116:0/159050235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc2d9d700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103ed0 0x7fcdc419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 -- 172.21.15.116:0/159050235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103ed0 msgr2=0x7fcdc419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103ed0 0x7fcdc419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 -- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 msgr2=0x7fcdc4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 -- 172.21.15.116:0/159050235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc41a00b0 con 0x7fcdc4102330 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc1d9b700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.012+0000 7fcdc259c700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 0x7fcdc4198450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40036f0 tx=0x7fcdb400f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4009e60 con 0x7fcdc4102330 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41a0340 con 0x7fcdc4102330 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdb4005940 con 0x7fcdc4102330 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4005450 con 0x7fcdc4102330 2024-03-19T02:27:10.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc41a0870 con 0x7fcdc4102330 2024-03-19T02:27:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.013+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc4068fd0 con 0x7fcdc4102330 2024-03-19T02:27:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.015+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fcdb4003b30 con 0x7fcdc4102330 2024-03-19T02:27:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.015+0000 7fcdb37fe700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 0x7fcdac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.015+0000 7fcdc1d9b700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 0x7fcdac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:10.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.015+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fcdb40cfab0 con 0x7fcdc4102330 2024-03-19T02:27:10.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.016+0000 7fcdc1d9b700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 0x7fcdac0a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8011550 tx=0x7fcdb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:10.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.019+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdb4013070 con 0x7fcdc4102330 2024-03-19T02:27:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:09 smithi162 ceph-mon[35761]: from='mgr.14152 172.21.15.116:0/4033843987' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T02:27:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:09 smithi162 ceph-mon[35761]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:27:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.223+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fcdc4062750 con 0x7fcdac0a2890 2024-03-19T02:27:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.229+0000 7fcdb37fe700 1 -- 172.21.15.116:0/159050235 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fcdc4062750 con 0x7fcdac0a2890 2024-03-19T02:27:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-19T02:27:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:mds.all 12/12 59s ago 2m smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T02:27:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 3/3 59s ago 7m smithi116=x;smithi154=y;smithi162=z;count:3 2024-03-19T02:27:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:mon 3/3 59s ago 8m smithi116:172.21.15.116=a;smithi154:172.21.15.154=b;smithi162:172.21.15.162=c;count:3 2024-03-19T02:27:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:osd 12 59s ago - 2024-03-19T02:27:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 msgr2=0x7fcdac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 0x7fcdac0a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8011550 tx=0x7fcdb8009260 comp rx=0 tx=0).stop 2024-03-19T02:27:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 msgr2=0x7fcdc4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 0x7fcdc4198450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40036f0 tx=0x7fcdb400f710 comp rx=0 tx=0).stop 2024-03-19T02:27:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 shutdown_connections 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fcdac0a2890 0x7fcdac0a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdc4103ed0 0x7fcdc419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdc4102330 0x7fcdc4198450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 --2- 172.21.15.116:0/159050235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc4103530 0x7fcdc4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 >> 172.21.15.116:0/159050235 conn(0x7fcdc40fd8f0 msgr2=0x7fcdc4106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 shutdown_connections 2024-03-19T02:27:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:10.231+0000 7fcdc8a0a700 1 -- 172.21.15.116:0/159050235 wait complete. 2024-03-19T02:27:10.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:10 smithi116 ceph-mon[32263]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:27:10.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:10 smithi116 ceph-mon[32263]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:27:10.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-19T02:27:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:10 smithi154 ceph-mon[35693]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:27:11.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:10 smithi154 ceph-mon[35693]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:27:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:10 smithi162 ceph-mon[35761]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:27:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:10 smithi162 ceph-mon[35761]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:27:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:11 smithi154 ceph-mon[35693]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:12.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:11 smithi116 ceph-mon[32263]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:11 smithi162 ceph-mon[35761]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:12.895 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:12 smithi116 ceph-mon[32263]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-19T02:27:13.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:12 smithi154 ceph-mon[35693]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-19T02:27:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 -- 172.21.15.116:0/2823177909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 msgr2=0x7f0b78100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/2823177909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78100a70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009990 tx=0x7f0b74009ca0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 -- 172.21.15.116:0/2823177909 shutdown_connections 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/2823177909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103fd0 0x7f0b7810a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/2823177909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78100a70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/2823177909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103630 0x7f0b78103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 -- 172.21.15.116:0/2823177909 >> 172.21.15.116:0/2823177909 conn(0x7f0b780fbfb0 msgr2=0x7f0b780fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 -- 172.21.15.116:0/2823177909 shutdown_connections 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.998+0000 7f0b7f66c700 1 -- 172.21.15.116:0/2823177909 wait complete. 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 Processor -- start 2024-03-19T02:27:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 -- start start 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103630 0x7f0b781989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103fd0 0x7f0b7819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810de20 con 0x7f0b78100650 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810dca0 con 0x7f0b78103fd0 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7f66c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7810dfa0 con 0x7f0b78103630 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58478/0 (socket says 172.21.15.116:58478) 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 -- 172.21.15.116:0/1509021216 learned_addr learned my addr 172.21.15.116:0/1509021216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7cc07700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103630 0x7f0b781989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7dc09700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103fd0 0x7f0b7819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 -- 172.21.15.116:0/1509021216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103630 msgr2=0x7f0b781989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103630 0x7f0b781989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 -- 172.21.15.116:0/1509021216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103fd0 msgr2=0x7f0b7819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103fd0 0x7f0b7819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 -- 172.21.15.116:0/1509021216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b781a00c0 con 0x7f0b78100650 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:12.999+0000 7f0b7d408700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b740098e0 tx=0x7f0b740057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.000+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74026070 con 0x7f0b78100650 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.000+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b74021410 con 0x7f0b78100650 2024-03-19T02:27:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.000+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b781a0350 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.000+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74017650 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.000+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b781a07a0 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.001+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b78068fd0 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.002+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0b74029070 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.002+0000 7f0b6e7fc700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 0x7f0b640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.002+0000 7f0b7cc07700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 0x7f0b640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.003+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f0b740cfe90 con 0x7f0b78100650 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.003+0000 7f0b7cc07700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 0x7f0b640a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009c60 tx=0x7f0b68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.006+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b74030080 con 0x7f0b78100650 2024-03-19T02:27:13.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:12 smithi162 ceph-mon[35761]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-19T02:27:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.207+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f0b78062750 con 0x7f0b640a2890 2024-03-19T02:27:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.209+0000 7f0b6e7fc700 1 -- 172.21.15.116:0/1509021216 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f0b78062750 con 0x7f0b640a2890 2024-03-19T02:27:13.209 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-19T02:27:13.209 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-19T02:27:13.210 INFO:teuthology.orchestra.run.smithi116.stdout:smithi154 172.21.15.154 2024-03-19T02:27:13.210 INFO:teuthology.orchestra.run.smithi116.stdout:smithi162 172.21.15.162 2024-03-19T02:27:13.210 INFO:teuthology.orchestra.run.smithi116.stdout:3 hosts in cluster 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 msgr2=0x7f0b640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 0x7f0b640a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009c60 tx=0x7f0b68009260 comp rx=0 tx=0).stop 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 msgr2=0x7f0b78198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b740098e0 tx=0x7f0b740057c0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 shutdown_connections 2024-03-19T02:27:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b78103fd0 0x7f0b7819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b78100650 0x7f0b78198460 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0b640a2890 0x7f0b640a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 --2- 172.21.15.116:0/1509021216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b78103630 0x7f0b781989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.211+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 >> 172.21.15.116:0/1509021216 conn(0x7f0b780fbfb0 msgr2=0x7f0b78106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.212+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 shutdown_connections 2024-03-19T02:27:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:13.212+0000 7f0b7f66c700 1 -- 172.21.15.116:0/1509021216 wait complete. 2024-03-19T02:27:13.996 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-19T02:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:14 smithi154 ceph-mon[35693]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:14 smithi154 ceph-mon[35693]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:14 smithi116 ceph-mon[32263]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:14 smithi116 ceph-mon[32263]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:14 smithi162 ceph-mon[35761]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:14 smithi162 ceph-mon[35761]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 -- 172.21.15.116:0/1850238490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88101240 msgr2=0x7f4d88101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/1850238490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88101240 0x7f4d88101660 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d780099c0 tx=0x7f4d78009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 -- 172.21.15.116:0/1850238490 shutdown_connections 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/1850238490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88102de0 0x7f4d88109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/1850238490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88101240 0x7f4d88101660 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/1850238490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102440 0x7f4d881028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 -- 172.21.15.116:0/1850238490 >> 172.21.15.116:0/1850238490 conn(0x7f4d880fc7e0 msgr2=0x7f4d880fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 -- 172.21.15.116:0/1850238490 shutdown_connections 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.948+0000 7f4d8d67a700 1 -- 172.21.15.116:0/1850238490 wait complete. 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 Processor -- start 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 -- start start 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101240 0x7f4d881973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 0x7f4d8819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810ccb0 con 0x7f4d88102440 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810cb30 con 0x7f4d88101240 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d8d67a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810ce30 con 0x7f4d88102de0 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58496/0 (socket says 172.21.15.116:58496) 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 0x7f4d8819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 -- 172.21.15.116:0/2233430092 learned_addr learned my addr 172.21.15.116:0/2233430092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d86ffd700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101240 0x7f4d881973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 -- 172.21.15.116:0/2233430092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 msgr2=0x7f4d8819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 0x7f4d8819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 -- 172.21.15.116:0/2233430092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101240 msgr2=0x7f4d881973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.949+0000 7f4d7e7fc700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101240 0x7f4d881973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:15.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d7e7fc700 1 -- 172.21.15.116:0/2233430092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8819f040 con 0x7f4d88102440 2024-03-19T02:27:15.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d877fe700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 0x7f4d8819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:15.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d7e7fc700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e9d0 tx=0x7f4d7000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:15.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7000cbd0 con 0x7f4d88102440 2024-03-19T02:27:15.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d7000ee70 con 0x7f4d88102440 2024-03-19T02:27:15.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8819f330 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70018680 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.950+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8819f870 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.951+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d88068fd0 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.952+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4d70010430 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.952+0000 7f4d84ff9700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 0x7f4d740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.953+0000 7f4d86ffd700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 0x7f4d740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.953+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f4d700cf030 con 0x7f4d88102440 2024-03-19T02:27:15.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.953+0000 7f4d86ffd700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 0x7f4d740a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4d78000c00 tx=0x7f4d78005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:15.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:15.957+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d7000cd70 con 0x7f4d88102440 2024-03-19T02:27:16.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.160+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4d88062750 con 0x7f4d740a2890 2024-03-19T02:27:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.162+0000 7f4d84ff9700 1 -- 172.21.15.116:0/2233430092 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4d88062750 con 0x7f4d740a2890 2024-03-19T02:27:16.163 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-19T02:27:16.163 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T02:27:16.163 INFO:teuthology.orchestra.run.smithi116.stdout:smithi154 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233003B400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T02:27:16.163 INFO:teuthology.orchestra.run.smithi116.stdout:smithi162 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 msgr2=0x7f4d740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 0x7f4d740a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4d78000c00 tx=0x7f4d78005fb0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 msgr2=0x7f4d88197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e9d0 tx=0x7f4d7000ece0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 shutdown_connections 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4d740a2890 0x7f4d740a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d88102de0 0x7f4d8819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101240 0x7f4d881973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 --2- 172.21.15.116:0/2233430092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d88102440 0x7f4d88197920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:16.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.164+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 >> 172.21.15.116:0/2233430092 conn(0x7f4d880fc7e0 msgr2=0x7f4d88105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:16.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.165+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 shutdown_connections 2024-03-19T02:27:16.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:16.165+0000 7f4d8d67a700 1 -- 172.21.15.116:0/2233430092 wait complete. 2024-03-19T02:27:16.835 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:16 smithi116 ceph-mon[32263]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:16.862 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T02:27:16.871 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:27:16.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T02:27:17.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:16 smithi154 ceph-mon[35693]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:17.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:16 smithi162 ceph-mon[35761]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:18.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:17 smithi154 ceph-mon[35693]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:17 smithi116 ceph-mon[32263]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:18.067 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:17 smithi162 ceph-mon[35761]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 -- 172.21.15.116:0/1301243441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103540 msgr2=0x7f0a581039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/1301243441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103540 0x7f0a581039a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0a400099c0 tx=0x7f0a40009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 -- 172.21.15.116:0/1301243441 shutdown_connections 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/1301243441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58103ee0 0x7f0a5810a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/1301243441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58102340 0x7f0a58102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/1301243441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103540 0x7f0a581039a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 -- 172.21.15.116:0/1301243441 >> 172.21.15.116:0/1301243441 conn(0x7f0a580fd8c0 msgr2=0x7f0a580ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 -- 172.21.15.116:0/1301243441 shutdown_connections 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.776+0000 7f0a5d81e700 1 -- 172.21.15.116:0/1301243441 wait complete. 2024-03-19T02:27:18.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 Processor -- start 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 -- start start 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 0x7f0a58198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 0x7f0a5819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5810ddb0 con 0x7f0a58103ee0 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5810dc30 con 0x7f0a58102340 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a5d81e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5810df30 con 0x7f0a58103540 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a4e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 0x7f0a58198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a577fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 0x7f0a5819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:44628/0 (socket says 172.21.15.116:44628) 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 -- 172.21.15.116:0/3792395012 learned_addr learned my addr 172.21.15.116:0/3792395012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 -- 172.21.15.116:0/3792395012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 msgr2=0x7f0a58198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 0x7f0a58198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 -- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 msgr2=0x7f0a5819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 0x7f0a5819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.777+0000 7f0a56ffd700 1 -- 172.21.15.116:0/3792395012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a581a00a0 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a4e7fc700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 0x7f0a58198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a577fe700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 0x7f0a5819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a56ffd700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e970 tx=0x7f0a4800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48004220 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a581a0390 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a480043c0 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a581a0920 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.778+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48003cc0 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.779+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a58068fd0 con 0x7f0a58102340 2024-03-19T02:27:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.780+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0a4800c7e0 con 0x7f0a58102340 2024-03-19T02:27:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.780+0000 7f0a5c81c700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 0x7f0a440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.780+0000 7f0a4e7fc700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 0x7f0a440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.780+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f0a480ceee0 con 0x7f0a58102340 2024-03-19T02:27:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.781+0000 7f0a4e7fc700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 0x7f0a440a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:18.786+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a4809bc00 con 0x7f0a58102340 2024-03-19T02:27:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:18 smithi116 ceph-mon[32263]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:19.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:18 smithi154 ceph-mon[35693]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:19.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.021+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f0a58062f70 con 0x7f0a58102340 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.022+0000 7f0a5c81c700 1 -- 172.21.15.116:0/3792395012 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3104 (secure 0 0 0) 0x7f0a4809b500 con 0x7f0a58102340 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:e27 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-19T02:27:19.023 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 27 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-19T02:25:08.015174+0000 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-19T02:25:36.674122+0000 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 300 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:27:19.024 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 5 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:in 0,1,2,3,4 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24494,1=24484,2=24470,3=24463,4=24451} 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3,4] 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:qdb_cluster leader: 24494 members: 24463,24470,24484,24494,24451 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b{0:24494} state up:active seq 3 addr [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.025 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.e{0:24440} state up:standby-replay seq 1 addr [v2:172.21.15.154:6834/1464153745,v1:172.21.15.154:6835/1464153745] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.i{1:24484} state up:active seq 11 addr [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.l{1:24418} state up:standby-replay seq 1 addr [v2:172.21.15.162:6832/750284800,v1:172.21.15.162:6833/750284800] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.h{2:24470} state up:active seq 12 addr [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.k{2:24410} state up:standby-replay seq 1 addr [v2:172.21.15.154:6832/1010748955,v1:172.21.15.154:6833/1010748955] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c{3:24463} state up:active seq 13 addr [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a{3:14646} state up:standby-replay seq 1 addr [v2:172.21.15.116:6840/3941624950,v1:172.21.15.116:6841/3941624950] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.f{4:24451} state up:active seq 14 addr [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.g{4:14634} state up:standby-replay seq 1 addr [v2:172.21.15.116:6838/1891911264,v1:172.21.15.116:6839/1891911264] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.j{-1:14550} state up:standby seq 1 addr [v2:172.21.15.116:6834/1165879311,v1:172.21.15.116:6835/1165879311] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.026 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.116:6836/261800782,v1:172.21.15.116:6837/261800782] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:27:19.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 msgr2=0x7f0a440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:19.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 0x7f0a440a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0a40000c00 tx=0x7f0a40005fb0 comp rx=0 tx=0).stop 2024-03-19T02:27:19.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 msgr2=0x7f0a58198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:19.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e970 tx=0x7f0a4800ec80 comp rx=0 tx=0).stop 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 shutdown_connections 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58103ee0 0x7f0a5819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a58102340 0x7f0a58198440 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0a440a2890 0x7f0a440a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 --2- 172.21.15.116:0/3792395012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a58103540 0x7f0a58198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 >> 172.21.15.116:0/3792395012 conn(0x7f0a580fd8c0 msgr2=0x7f0a58106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.024+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 shutdown_connections 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:19.025+0000 7f0a5d81e700 1 -- 172.21.15.116:0/3792395012 wait complete. 2024-03-19T02:27:19.028 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:27:19.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:18 smithi162 ceph-mon[35761]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-19T02:27:19.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-19T02:27:19.961 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:20.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:19 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3792395012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:27:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:19 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3792395012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:27:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:19 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3792395012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:27:21.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:20 smithi154 ceph-mon[35693]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:20 smithi116 ceph-mon[32263]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:21.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:20 smithi162 ceph-mon[35761]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:21.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 -- 172.21.15.116:0/1299440609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c401035e0 msgr2=0x7f5c40103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 --2- 172.21.15.116:0/1299440609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c401035e0 0x7f5c40103a40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009990 tx=0x7f5c30009ca0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 -- 172.21.15.116:0/1299440609 shutdown_connections 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 --2- 172.21.15.116:0/1299440609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c40103f80 0x7f5c4010a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 --2- 172.21.15.116:0/1299440609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40100600 0x7f5c40100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 --2- 172.21.15.116:0/1299440609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c401035e0 0x7f5c40103a40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 -- 172.21.15.116:0/1299440609 >> 172.21.15.116:0/1299440609 conn(0x7f5c400fbf80 msgr2=0x7f5c400fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 -- 172.21.15.116:0/1299440609 shutdown_connections 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.630+0000 7f5c47625700 1 -- 172.21.15.116:0/1299440609 wait complete. 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 Processor -- start 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 -- start start 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:21.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c401035e0 0x7f5c40198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40103f80 0x7f5c4019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4010ddd0 con 0x7f5c40100600 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4010dc50 con 0x7f5c401035e0 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c47625700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4010df50 con 0x7f5c40103f80 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44846/0 (socket says 172.21.15.116:44846) 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 -- 172.21.15.116:0/1032299637 learned_addr learned my addr 172.21.15.116:0/1032299637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c44bc0700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c401035e0 0x7f5c40198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c45bc2700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40103f80 0x7f5c4019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 -- 172.21.15.116:0/1032299637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40103f80 msgr2=0x7f5c4019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40103f80 0x7f5c4019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 -- 172.21.15.116:0/1032299637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c401035e0 msgr2=0x7f5c40198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c401035e0 0x7f5c40198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 -- 172.21.15.116:0/1032299637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c401a00a0 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c453c1700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009f20 tx=0x7f5c3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.631+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c003a00 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.632+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c401a0390 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.632+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c3c00ce70 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.632+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c004710 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.632+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c401a0880 con 0x7f5c40100600 2024-03-19T02:27:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.632+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c40068fd0 con 0x7f5c40100600 2024-03-19T02:27:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.634+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5c3c012050 con 0x7f5c40100600 2024-03-19T02:27:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.634+0000 7f5c367fc700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 0x7f5c2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.634+0000 7f5c44bc0700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 0x7f5c2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.634+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5c3c0ceec0 con 0x7f5c40100600 2024-03-19T02:27:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.635+0000 7f5c44bc0700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 0x7f5c2c0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000b710 tx=0x7f5c30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.638+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c3c010040 con 0x7f5c40100600 2024-03-19T02:27:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.841+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f5c40062f20 con 0x7f5c40100600 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.842+0000 7f5c367fc700 1 -- 172.21.15.116:0/1032299637 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5655 (secure 0 0 0) 0x7f5c3c0048b0 con 0x7f5c40100600 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 94 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:fsid a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-19T02:17:06.753416+0000 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-19T02:27:09.726326+0000 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:crush_version 27 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:full_ratio 0.95 2024-03-19T02:27:21.842 INFO:teuthology.orchestra.run.smithi116.stdout:backfillfull_ratio 0.9 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:nearfull_ratio 0.85 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:require_min_compat_client luminous 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:min_compat_client luminous 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:require_osd_release squid 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:stretch_mode_enabled false 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:max_osd 12 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6802/3168976854,v1:172.21.15.116:6803/3168976854] [v2:172.21.15.116:6804/3168976854,v1:172.21.15.116:6805/3168976854] exists,up 228d73fd-9b05-45b9-b2fb-93b859800b76 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6800/3251901849,v1:172.21.15.154:6801/3251901849] [v2:172.21.15.154:6802/3251901849,v1:172.21.15.154:6803/3251901849] exists,up fd497f40-8e3f-473a-8a69-450ad65aa6d9 2024-03-19T02:27:21.843 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 up in weight 1 up_from 18 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/2685600713,v1:172.21.15.162:6801/2685600713] [v2:172.21.15.162:6802/2685600713,v1:172.21.15.162:6803/2685600713] exists,up 89b45a09-6328-44d9-84ba-890c9225881f 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 up in weight 1 up_from 26 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6810/2748385188,v1:172.21.15.116:6811/2748385188] [v2:172.21.15.116:6812/2748385188,v1:172.21.15.116:6813/2748385188] exists,up 5d6c5e73-72bf-40e9-95b8-8473f091ea90 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 up in weight 1 up_from 34 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6808/719513760,v1:172.21.15.154:6809/719513760] [v2:172.21.15.154:6810/719513760,v1:172.21.15.154:6811/719513760] exists,up 4bf282e6-a231-437d-af9b-f8b7a11d67c2 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 up in weight 1 up_from 39 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/1110895518,v1:172.21.15.162:6809/1110895518] [v2:172.21.15.162:6810/1110895518,v1:172.21.15.162:6811/1110895518] exists,up 4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 up in weight 1 up_from 47 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/119271604,v1:172.21.15.116:6819/119271604] [v2:172.21.15.116:6820/119271604,v1:172.21.15.116:6821/119271604] exists,up e9047802-fa47-4d12-98d7-b7a23641af81 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 up in weight 1 up_from 52 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6816/2515714011,v1:172.21.15.154:6817/2515714011] [v2:172.21.15.154:6818/2515714011,v1:172.21.15.154:6819/2515714011] exists,up c2ab68c8-d137-4f5c-a8e1-e539b0819897 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 up in weight 1 up_from 60 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/2609002747,v1:172.21.15.162:6817/2609002747] [v2:172.21.15.162:6818/2609002747,v1:172.21.15.162:6819/2609002747] exists,up ba54ef24-b96d-4423-b606-6b742a134ca7 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6826/3348335330,v1:172.21.15.116:6827/3348335330] [v2:172.21.15.116:6828/3348335330,v1:172.21.15.116:6829/3348335330] exists,up 3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 up in weight 1 up_from 73 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6824/1964548863,v1:172.21.15.154:6825/1964548863] [v2:172.21.15.154:6826/1964548863,v1:172.21.15.154:6827/1964548863] exists,up b644847f-a734-4600-9861-c8574c3f0622 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 up in weight 1 up_from 81 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/3174333504,v1:172.21.15.162:6825/3174333504] [v2:172.21.15.162:6826/3174333504,v1:172.21.15.162:6827/3174333504] exists,up 883bfd81-b55d-441a-9a0c-2f5faef7f4a0 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.4 [2,11] 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.1d [8,11] 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.1 [4,9] 2024-03-19T02:27:21.844 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.5 [3,9] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.27 [4,8] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.34 [4,9] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.37 [5,6] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.8 [5,8] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.b [9,8] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.17 [9,8] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.1c [9,10] 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/1993666663 expires 2024-03-20T02:18:07.512230+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/707840070 expires 2024-03-20T02:18:07.512230+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/996139070 expires 2024-03-20T02:18:07.512230+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2583747917 expires 2024-03-20T02:17:41.784026+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3953610795 expires 2024-03-20T02:17:41.784026+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/1136807551 expires 2024-03-20T02:17:41.784026+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/647665947 expires 2024-03-20T02:18:07.512230+0000 2024-03-19T02:27:21.845 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/647665947 expires 2024-03-20T02:18:07.512230+0000 2024-03-19T02:27:21.846 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/1136807551 expires 2024-03-20T02:17:41.784026+0000 2024-03-19T02:27:21.846 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2020074950 expires 2024-03-20T02:17:41.784026+0000 2024-03-19T02:27:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 msgr2=0x7f5c2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 0x7f5c2c0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000b710 tx=0x7f5c30005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 msgr2=0x7f5c40198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009f20 tx=0x7f5c3c00c3a0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 shutdown_connections 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5c2c0a2890 0x7f5c2c0a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c40103f80 0x7f5c4019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c40100600 0x7f5c40198440 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 --2- 172.21.15.116:0/1032299637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c401035e0 0x7f5c40198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 >> 172.21.15.116:0/1032299637 conn(0x7f5c400fbf80 msgr2=0x7f5c40106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 shutdown_connections 2024-03-19T02:27:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:21.844+0000 7f5c47625700 1 -- 172.21.15.116:0/1032299637 wait complete. 2024-03-19T02:27:22.533 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-19T02:27:22.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 -- 172.21.15.116:0/583647901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f81022f0 msgr2=0x7f12f8108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 --2- 172.21.15.116:0/583647901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f81022f0 0x7f12f8108b80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009ca0 tx=0x7f12f002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 -- 172.21.15.116:0/583647901 shutdown_connections 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 --2- 172.21.15.116:0/583647901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f81022f0 0x7f12f8108b80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 --2- 172.21.15.116:0/583647901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f8101930 0x7f12f8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 --2- 172.21.15.116:0/583647901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8100740 0x7f12f8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 -- 172.21.15.116:0/583647901 >> 172.21.15.116:0/583647901 conn(0x7f12f80fbcb0 msgr2=0x7f12f80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 -- 172.21.15.116:0/583647901 shutdown_connections 2024-03-19T02:27:22.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.655+0000 7f12fecf1640 1 -- 172.21.15.116:0/583647901 wait complete. 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 Processor -- start 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 -- start start 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8101930 0x7f12f8196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f81022f0 0x7f12f819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12f810c270 con 0x7f12f8100740 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12f810c0f0 con 0x7f12f81022f0 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fecf1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12f810c3f0 con 0x7f12f8101930 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44860/0 (socket says 172.21.15.116:44860) 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 -- 172.21.15.116:0/1514159262 learned_addr learned my addr 172.21.15.116:0/1514159262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:22.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8101930 0x7f12f8196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fd28e640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f81022f0 0x7f12f819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 -- 172.21.15.116:0/1514159262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8101930 msgr2=0x7f12f8196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8101930 0x7f12f8196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 -- 172.21.15.116:0/1514159262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f81022f0 msgr2=0x7f12f819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f81022f0 0x7f12f819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.656+0000 7f12fca8d640 1 -- 172.21.15.116:0/1514159262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f819e500 con 0x7f12f8100740 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12fca8d640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00ead0 tx=0x7f12ec00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec01c070 con 0x7f12f8100740 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12ec00bbd0 con 0x7f12f8100740 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f8069ef0 con 0x7f12f8100740 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec003d00 con 0x7f12f8100740 2024-03-19T02:27:22.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.657+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f806a440 con 0x7f12f8100740 2024-03-19T02:27:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.658+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f8068a60 con 0x7f12f8100740 2024-03-19T02:27:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.659+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f12ec01f070 con 0x7f12f8100740 2024-03-19T02:27:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.659+0000 7f12f5ffb640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 0x7f12e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.659+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 0x7f12e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.660+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f12ec0d0360 con 0x7f12f8100740 2024-03-19T02:27:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.660+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 0x7f12e80a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f12f8197df0 tx=0x7f12e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:22.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.664+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12ec09cad0 con 0x7f12f8100740 2024-03-19T02:27:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.843+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12f819e6d0 con 0x7f12f8100740 2024-03-19T02:27:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.845+0000 7f12f5ffb640 1 -- 172.21.15.116:0/1514159262 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7f12ec0a0b70 con 0x7f12f8100740 2024-03-19T02:27:22.845 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:22.845 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:22.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 msgr2=0x7f12e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 0x7f12e80a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f12f8197df0 tx=0x7f12e4005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:22.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 msgr2=0x7f12f81968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00ead0 tx=0x7f12ec00ef90 comp rx=0 tx=0).stop 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 shutdown_connections 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f12e80a2880 0x7f12e80a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12f81022f0 0x7f12f819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12f8101930 0x7f12f8196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 --2- 172.21.15.116:0/1514159262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f8100740 0x7f12f81968e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 >> 172.21.15.116:0/1514159262 conn(0x7f12f80fbcb0 msgr2=0x7f12f8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 shutdown_connections 2024-03-19T02:27:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.847+0000 7f12fecf1640 1 -- 172.21.15.116:0/1514159262 wait complete. 2024-03-19T02:27:22.858 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:27:22.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:27:22.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 -- 172.21.15.116:0/2768458180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 msgr2=0x7f3e80100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2768458180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e80100b60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009900 tx=0x7f3e7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 -- 172.21.15.116:0/2768458180 shutdown_connections 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2768458180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 0x7f3e80108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2768458180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 0x7f3e80101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2768458180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e80100b60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 -- 172.21.15.116:0/2768458180 >> 172.21.15.116:0/2768458180 conn(0x7f3e800fbcd0 msgr2=0x7f3e800fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.964+0000 7f3e8680c640 1 -- 172.21.15.116:0/2768458180 shutdown_connections 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 -- 172.21.15.116:0/2768458180 wait complete. 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 Processor -- start 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 -- start start 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 0x7f3e8019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 0x7f3e8019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c2c0 con 0x7f3e80100740 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c140 con 0x7f3e80101930 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e8680c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8010c440 con 0x7f3e801022f0 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44888/0 (socket says 172.21.15.116:44888) 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.965+0000 7f3e7ffff640 1 -- 172.21.15.116:0/2276233898 learned_addr learned my addr 172.21.15.116:0/2276233898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e84da9640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 0x7f3e8019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7f7fe640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 0x7f3e8019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 -- 172.21.15.116:0/2276233898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 msgr2=0x7f3e8019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 0x7f3e8019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 -- 172.21.15.116:0/2276233898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 msgr2=0x7f3e8019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 0x7f3e8019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 -- 172.21.15.116:0/2276233898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8019f710 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7ffff640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000c00 tx=0x7f3e70005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70043020 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e700043d0 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8019f940 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70003b40 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.966+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8019fe90 con 0x7f3e80100740 2024-03-19T02:27:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.967+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e80107b80 con 0x7f3e80100740 2024-03-19T02:27:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.968+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3e70046020 con 0x7f3e80100740 2024-03-19T02:27:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.969+0000 7f3e7d7fa640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.969+0000 7f3e7f7fe640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.969+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3e700f0f20 con 0x7f3e80100740 2024-03-19T02:27:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.969+0000 7f3e7f7fe640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 0x7f3e680a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e8006a710 tx=0x7f3e74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:22.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:22.973+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e7004d020 con 0x7f3e80100740 2024-03-19T02:27:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:22 smithi154 ceph-mon[35693]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:22 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1032299637' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:27:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:22 smithi116 ceph-mon[32263]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:22 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1032299637' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:27:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.148+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e80062420 con 0x7f3e80100740 2024-03-19T02:27:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.149+0000 7f3e7d7fa640 1 -- 172.21.15.116:0/2276233898 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f3e700bdc40 con 0x7f3e80100740 2024-03-19T02:27:23.148 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:27:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 msgr2=0x7f3e680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 0x7f3e680a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e8006a710 tx=0x7f3e74005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 msgr2=0x7f3e8006a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000c00 tx=0x7f3e70005e60 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 shutdown_connections 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3e680a2800 0x7f3e680a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e801022f0 0x7f3e8019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e80101930 0x7f3e8019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 --2- 172.21.15.116:0/2276233898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e80100740 0x7f3e8006a9a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 >> 172.21.15.116:0/2276233898 conn(0x7f3e800fbcd0 msgr2=0x7f3e80104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 shutdown_connections 2024-03-19T02:27:23.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.151+0000 7f3e8680c640 1 -- 172.21.15.116:0/2276233898 wait complete. 2024-03-19T02:27:23.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:27:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:22 smithi162 ceph-mon[35761]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:22 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1032299637' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:27:23.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- 172.21.15.116:0/667263801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 msgr2=0x7f1a94100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/667263801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a94100b50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009900 tx=0x7f1a8402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- 172.21.15.116:0/667263801 shutdown_connections 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/667263801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a94108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/667263801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 0x7f1a94101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/667263801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a94100b50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- 172.21.15.116:0/667263801 >> 172.21.15.116:0/667263801 conn(0x7f1a940fbcc0 msgr2=0x7f1a940fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- 172.21.15.116:0/667263801 shutdown_connections 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- 172.21.15.116:0/667263801 wait complete. 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 Processor -- start 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.268+0000 7f1a99f7e640 1 -- start start 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 0x7f1a9419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a941a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410c230 con 0x7f1a94100730 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410c0b0 con 0x7f1a941022e0 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410c3b0 con 0x7f1a94101920 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44910/0 (socket says 172.21.15.116:44910) 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 -- 172.21.15.116:0/2569767334 learned_addr learned my addr 172.21.15.116:0/2569767334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 0x7f1a9419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a93fff640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a941a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 -- 172.21.15.116:0/2569767334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 msgr2=0x7f1a9419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 0x7f1a9419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 -- 172.21.15.116:0/2569767334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 msgr2=0x7f1a941a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a941a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 -- 172.21.15.116:0/2569767334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941a1950 con 0x7f1a94100730 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a93fff640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a941a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a937fe640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a84000c00 tx=0x7f1a84005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84043020 con 0x7f1a94100730 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a941a1b80 con 0x7f1a94100730 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a84004350 con 0x7f1a94100730 2024-03-19T02:27:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84003b80 con 0x7f1a94100730 2024-03-19T02:27:23.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.269+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941a20d0 con 0x7f1a94100730 2024-03-19T02:27:23.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.271+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a94107b70 con 0x7f1a94100730 2024-03-19T02:27:23.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.272+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1a84046020 con 0x7f1a94100730 2024-03-19T02:27:23.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.272+0000 7f1a917fa640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 0x7f1a800a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.272+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 0x7f1a800a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.272+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1a840f0fc0 con 0x7f1a94100730 2024-03-19T02:27:23.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.273+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 0x7f1a800a4cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1a9406a6f0 tx=0x7f1a7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.277+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a8404d020 con 0x7f1a94100730 2024-03-19T02:27:23.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.450+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a94062420 con 0x7f1a94100730 2024-03-19T02:27:23.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.451+0000 7f1a917fa640 1 -- 172.21.15.116:0/2569767334 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f1a840bdce0 con 0x7f1a94100730 2024-03-19T02:27:23.451 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:23.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 msgr2=0x7f1a800a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 0x7f1a800a4cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1a9406a6f0 tx=0x7f1a7c005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 msgr2=0x7f1a9406a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1a84000c00 tx=0x7f1a84005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 shutdown_connections 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a800a2830 0x7f1a800a4cf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a941022e0 0x7f1a941a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a94101920 0x7f1a9419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 --2- 172.21.15.116:0/2569767334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94100730 0x7f1a9406a980 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 >> 172.21.15.116:0/2569767334 conn(0x7f1a940fbcc0 msgr2=0x7f1a94104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 shutdown_connections 2024-03-19T02:27:23.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.453+0000 7f1a99f7e640 1 -- 172.21.15.116:0/2569767334 wait complete. 2024-03-19T02:27:23.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 -- 172.21.15.116:0/1236529305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c103490 msgr2=0x7fbf0c109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 --2- 172.21.15.116:0/1236529305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c103490 0x7fbf0c109d20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009cd0 tx=0x7fbf0402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 -- 172.21.15.116:0/1236529305 shutdown_connections 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 --2- 172.21.15.116:0/1236529305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c103490 0x7fbf0c109d20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 --2- 172.21.15.116:0/1236529305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 0x7fbf0c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 --2- 172.21.15.116:0/1236529305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 -- 172.21.15.116:0/1236529305 >> 172.21.15.116:0/1236529305 conn(0x7fbf0c0fb430 msgr2=0x7fbf0c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 -- 172.21.15.116:0/1236529305 shutdown_connections 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.569+0000 7fbf1424b640 1 -- 172.21.15.116:0/1236529305 wait complete. 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 Processor -- start 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 -- start start 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 0x7fbf0c195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c103490 0x7fbf0c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c10b1d0 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c10b050 con 0x7fbf0c102ad0 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf1424b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c10b350 con 0x7fbf0c103490 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44920/0 (socket says 172.21.15.116:44920) 2024-03-19T02:27:23.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 -- 172.21.15.116:0/804218843 learned_addr learned my addr 172.21.15.116:0/804218843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf127e8640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c103490 0x7fbf0c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf117e6640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 0x7fbf0c195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 -- 172.21.15.116:0/804218843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c103490 msgr2=0x7fbf0c19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c103490 0x7fbf0c19cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 -- 172.21.15.116:0/804218843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 msgr2=0x7fbf0c195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 0x7fbf0c195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.570+0000 7fbf11fe7640 1 -- 172.21.15.116:0/804218843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c19d430 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf11fe7640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009be0 tx=0x7fbf08007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08007c40 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf08007de0 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c19d720 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080193f0 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.571+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c19dc70 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.572+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c108cd0 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.573+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbf08020030 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.573+0000 7fbf02ffd640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 0x7fbef80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.573+0000 7fbf117e6640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 0x7fbef80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.574+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fbf080d25c0 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.574+0000 7fbf117e6640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 0x7fbef80a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0095f0 tx=0x7fbefc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.577+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf08027060 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.755+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbf0c062450 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.756+0000 7fbf02ffd640 1 -- 172.21.15.116:0/804218843 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fbf08016070 con 0x7fbf0c0ffb00 2024-03-19T02:27:23.755 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 msgr2=0x7fbef80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 0x7fbef80a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0095f0 tx=0x7fbefc005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 msgr2=0x7fbf0c195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009be0 tx=0x7fbf08007590 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 shutdown_connections 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fbef80a2800 0x7fbef80a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf0c103490 0x7fbf0c19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf0c102ad0 0x7fbf0c195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 --2- 172.21.15.116:0/804218843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0c0ffb00 0x7fbf0c195810 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 >> 172.21.15.116:0/804218843 conn(0x7fbf0c0fb430 msgr2=0x7fbf0c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 shutdown_connections 2024-03-19T02:27:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.758+0000 7fbf1424b640 1 -- 172.21.15.116:0/804218843 wait complete. 2024-03-19T02:27:23.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 -- 172.21.15.116:0/1266885689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a84101910 msgr2=0x7f1a84101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 --2- 172.21.15.116:0/1266885689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a84101910 0x7f1a84101d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009900 tx=0x7f1a7402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 -- 172.21.15.116:0/1266885689 shutdown_connections 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 --2- 172.21.15.116:0/1266885689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a841022d0 0x7f1a84108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 --2- 172.21.15.116:0/1266885689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a84101910 0x7f1a84101d90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 --2- 172.21.15.116:0/1266885689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84100720 0x7f1a84100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 -- 172.21.15.116:0/1266885689 >> 172.21.15.116:0/1266885689 conn(0x7f1a840fbcb0 msgr2=0x7f1a840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 -- 172.21.15.116:0/1266885689 shutdown_connections 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.874+0000 7f1a8c034640 1 -- 172.21.15.116:0/1266885689 wait complete. 2024-03-19T02:27:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 Processor -- start 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 -- start start 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a84100720 0x7f1a84197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84101910 0x7f1a84197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8410d370 con 0x7f1a841022d0 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8410d1f0 con 0x7f1a84100720 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8c034640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8410d4f0 con 0x7f1a84101910 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8a5d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8a5d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44934/0 (socket says 172.21.15.116:44934) 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a8a5d1640 1 -- 172.21.15.116:0/3050873688 learned_addr learned my addr 172.21.15.116:0/3050873688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a89dd0640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a84100720 0x7f1a84197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.875+0000 7f1a895cf640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84101910 0x7f1a84197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 -- 172.21.15.116:0/3050873688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84101910 msgr2=0x7f1a84197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84101910 0x7f1a84197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 -- 172.21.15.116:0/3050873688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a84100720 msgr2=0x7f1a84197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a84100720 0x7f1a84197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 -- 172.21.15.116:0/3050873688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8419f680 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8a5d1640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e970 tx=0x7f1a8000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000ccc0 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a8000ce60 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8419f970 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80010430 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.876+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8419fe40 con 0x7f1a841022d0 2024-03-19T02:27:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.877+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a84107b60 con 0x7f1a841022d0 2024-03-19T02:27:23.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.878+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1a80004ff0 con 0x7f1a841022d0 2024-03-19T02:27:23.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.878+0000 7f1a7affd640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 0x7f1a700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:23.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.878+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1a800cf1f0 con 0x7f1a841022d0 2024-03-19T02:27:23.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.878+0000 7f1a89dd0640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 0x7f1a700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:23.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.879+0000 7f1a89dd0640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 0x7f1a700a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a84101770 tx=0x7f1a7c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:23.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:23.883+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a8009bf10 con 0x7f1a841022d0 2024-03-19T02:27:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1514159262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2276233898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:23 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2569767334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.056+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1a8402d410 con 0x7f1a841022d0 2024-03-19T02:27:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.057+0000 7f1a7affd640 1 -- 172.21.15.116:0/3050873688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f1a8009b810 con 0x7f1a841022d0 2024-03-19T02:27:24.056 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 msgr2=0x7f1a700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 0x7f1a700a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1a84101770 tx=0x7f1a7c009500 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 msgr2=0x7f1a8419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e970 tx=0x7f1a8000ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 shutdown_connections 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1a700a2800 0x7f1a700a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a841022d0 0x7f1a8419f010 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a84101910 0x7f1a84197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 --2- 172.21.15.116:0/3050873688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a84100720 0x7f1a84197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 >> 172.21.15.116:0/3050873688 conn(0x7f1a840fbcb0 msgr2=0x7f1a84104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 shutdown_connections 2024-03-19T02:27:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.059+0000 7f1a8c034640 1 -- 172.21.15.116:0/3050873688 wait complete. 2024-03-19T02:27:24.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1514159262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2276233898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:23 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2569767334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 -- 172.21.15.116:0/1585775482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d64101930 msgr2=0x7f8d64101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 --2- 172.21.15.116:0/1585775482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d64101930 0x7f8d64101db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009900 tx=0x7f8d5802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 -- 172.21.15.116:0/1585775482 shutdown_connections 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 --2- 172.21.15.116:0/1585775482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d641022f0 0x7f8d64108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 --2- 172.21.15.116:0/1585775482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d64101930 0x7f8d64101db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 --2- 172.21.15.116:0/1585775482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64100740 0x7f8d64100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 -- 172.21.15.116:0/1585775482 >> 172.21.15.116:0/1585775482 conn(0x7f8d640fbcb0 msgr2=0x7f8d640fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 -- 172.21.15.116:0/1585775482 shutdown_connections 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.171+0000 7f8d692a1640 1 -- 172.21.15.116:0/1585775482 wait complete. 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 Processor -- start 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 -- start start 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 0x7f8d64198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 0x7f8d6419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6410c250 con 0x7f8d64100740 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6410c0d0 con 0x7f8d64101930 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d692a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6410c3d0 con 0x7f8d641022f0 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44956/0 (socket says 172.21.15.116:44956) 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d62ffd640 1 -- 172.21.15.116:0/3685848289 learned_addr learned my addr 172.21.15.116:0/3685848289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d627fc640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 0x7f8d64198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.172+0000 7f8d637fe640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 0x7f8d6419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 -- 172.21.15.116:0/3685848289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 msgr2=0x7f8d6419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 0x7f8d6419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 -- 172.21.15.116:0/3685848289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 msgr2=0x7f8d64198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 0x7f8d64198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 -- 172.21.15.116:0/3685848289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6419f6f0 con 0x7f8d64100740 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d627fc640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 0x7f8d64198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:24.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d637fe640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 0x7f8d6419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d62ffd640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400ebc0 tx=0x7f8d5400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d54003b70 con 0x7f8d64100740 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d6419f9e0 con 0x7f8d64100740 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d54003d10 con 0x7f8d64100740 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d54019820 con 0x7f8d64100740 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.173+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d6419ff50 con 0x7f8d64100740 2024-03-19T02:27:24.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.174+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d641a01e0 con 0x7f8d64100740 2024-03-19T02:27:24.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.175+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8d540199c0 con 0x7f8d64100740 2024-03-19T02:27:24.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.176+0000 7f8d4bfff640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 0x7f8d4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.176+0000 7f8d627fc640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 0x7f8d4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.176+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f8d540cf670 con 0x7f8d64100740 2024-03-19T02:27:24.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.176+0000 7f8d627fc640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 0x7f8d4c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009230 tx=0x7f8d58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.180+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d5409c390 con 0x7f8d64100740 2024-03-19T02:27:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1514159262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2276233898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:23 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2569767334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:24.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.397+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d641a03b0 con 0x7f8d64100740 2024-03-19T02:27:24.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.397+0000 7f8d4bfff640 1 -- 172.21.15.116:0/3685848289 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f8d5409bc90 con 0x7f8d64100740 2024-03-19T02:27:24.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:24.397 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:27:24.397 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:27:24.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 msgr2=0x7f8d4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 0x7f8d4c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009230 tx=0x7f8d58005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 msgr2=0x7f8d64197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400ebc0 tx=0x7f8d5400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 shutdown_connections 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8d4c0a2800 0x7f8d4c0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.399+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d641022f0 0x7f8d6419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.400+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d64101930 0x7f8d64198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.400+0000 7f8d692a1640 1 --2- 172.21.15.116:0/3685848289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d64100740 0x7f8d64197ad0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.400+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 >> 172.21.15.116:0/3685848289 conn(0x7f8d640fbcb0 msgr2=0x7f8d64068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.400+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 shutdown_connections 2024-03-19T02:27:24.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.400+0000 7f8d692a1640 1 -- 172.21.15.116:0/3685848289 wait complete. 2024-03-19T02:27:24.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- 172.21.15.116:0/4104890838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc103420 msgr2=0x7f0fdc109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/4104890838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc103420 0x7f0fdc109cb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009be0 tx=0x7f0fd402bef0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- 172.21.15.116:0/4104890838 shutdown_connections 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/4104890838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc103420 0x7f0fdc109cb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/4104890838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc102a60 0x7f0fdc102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/4104890838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- 172.21.15.116:0/4104890838 >> 172.21.15.116:0/4104890838 conn(0x7f0fdc0fcdc0 msgr2=0x7f0fdc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- 172.21.15.116:0/4104890838 shutdown_connections 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- 172.21.15.116:0/4104890838 wait complete. 2024-03-19T02:27:24.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 Processor -- start 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.513+0000 7f0fe41c6640 1 -- start start 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc102a60 0x7f0fdc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc103420 0x7f0fdc19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc10d380 con 0x7f0fdc101870 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc10d200 con 0x7f0fdc103420 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc10d500 con 0x7f0fdc102a60 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44970/0 (socket says 172.21.15.116:44970) 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 -- 172.21.15.116:0/2004016040 learned_addr learned my addr 172.21.15.116:0/2004016040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1761640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc102a60 0x7f0fdc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe2763640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc103420 0x7f0fdc19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 -- 172.21.15.116:0/2004016040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc102a60 msgr2=0x7f0fdc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc102a60 0x7f0fdc197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 -- 172.21.15.116:0/2004016040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc103420 msgr2=0x7f0fdc19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc103420 0x7f0fdc19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 -- 172.21.15.116:0/2004016040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc19f640 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe1f62640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00e9f0 tx=0x7f0fcc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc00ccc0 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fcc0035f0 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc00ce60 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc19f930 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.514+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc19fe80 con 0x7f0fdc101870 2024-03-19T02:27:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.515+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fdc068610 con 0x7f0fdc101870 2024-03-19T02:27:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.516+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0fcc0037a0 con 0x7f0fdc101870 2024-03-19T02:27:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.517+0000 7f0fd2ffd640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 0x7f0fc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.517+0000 7f0fe1761640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 0x7f0fc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.517+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f0fcc0cf030 con 0x7f0fdc101870 2024-03-19T02:27:24.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.518+0000 7f0fe1761640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 0x7f0fc80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc198f30 tx=0x7f0fd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.522+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0fcc09bd50 con 0x7f0fdc101870 2024-03-19T02:27:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.699+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fdc062420 con 0x7f0fdc101870 2024-03-19T02:27:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.700+0000 7f0fd2ffd640 1 -- 172.21.15.116:0/2004016040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7f0fcc09b650 con 0x7f0fdc101870 2024-03-19T02:27:24.700 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 msgr2=0x7f0fc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 0x7f0fc80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc198f30 tx=0x7f0fd8005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 msgr2=0x7f0fdc197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00e9f0 tx=0x7f0fcc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 shutdown_connections 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f0fc80a2800 0x7f0fc80a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fdc103420 0x7f0fdc19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fdc102a60 0x7f0fdc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 --2- 172.21.15.116:0/2004016040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fdc101870 0x7f0fdc197a20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 >> 172.21.15.116:0/2004016040 conn(0x7f0fdc0fcdc0 msgr2=0x7f0fdc105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 shutdown_connections 2024-03-19T02:27:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.702+0000 7f0fe41c6640 1 -- 172.21.15.116:0/2004016040 wait complete. 2024-03-19T02:27:24.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:27:24.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:27:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- 172.21.15.116:0/673508530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc102a40 msgr2=0x7f67cc102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 --2- 172.21.15.116:0/673508530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc102a40 0x7f67cc102ec0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009900 tx=0x7f67bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- 172.21.15.116:0/673508530 shutdown_connections 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 --2- 172.21.15.116:0/673508530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc103400 0x7f67cc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 --2- 172.21.15.116:0/673508530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc102a40 0x7f67cc102ec0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 --2- 172.21.15.116:0/673508530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 0x7f67cc101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- 172.21.15.116:0/673508530 >> 172.21.15.116:0/673508530 conn(0x7f67cc0fcdc0 msgr2=0x7f67cc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- 172.21.15.116:0/673508530 shutdown_connections 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- 172.21.15.116:0/673508530 wait complete. 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 Processor -- start 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.820+0000 7f67d13d4640 1 -- start start 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 0x7f67cc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc102a40 0x7f67cc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10d360 con 0x7f67cc103400 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10d1e0 con 0x7f67cc101850 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10d4e0 con 0x7f67cc102a40 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44992/0 (socket says 172.21.15.116:44992) 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 -- 172.21.15.116:0/2648445830 learned_addr learned my addr 172.21.15.116:0/2648445830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67caffd640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 0x7f67cc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67ca7fc640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc102a40 0x7f67cc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 -- 172.21.15.116:0/2648445830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc102a40 msgr2=0x7f67cc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc102a40 0x7f67cc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 -- 172.21.15.116:0/2648445830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 msgr2=0x7f67cc197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 0x7f67cc197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 -- 172.21.15.116:0/2648445830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc19f690 con 0x7f67cc103400 2024-03-19T02:27:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67c400eb10 tx=0x7f67c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4004b20 con 0x7f67cc103400 2024-03-19T02:27:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67c4004cc0 con 0x7f67cc103400 2024-03-19T02:27:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4010690 con 0x7f67cc103400 2024-03-19T02:27:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67cc19f980 con 0x7f67cc103400 2024-03-19T02:27:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.821+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67cc19fea0 con 0x7f67cc103400 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.823+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc108c60 con 0x7f67cc103400 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.823+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f67c4005020 con 0x7f67cc103400 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.824+0000 7f67b3fff640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.824+0000 7f67caffd640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.824+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f67c40cf2f0 con 0x7f67cc103400 2024-03-19T02:27:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.824+0000 7f67caffd640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 0x7f67b40a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67cc1028a0 tx=0x7f67c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:24.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:24.828+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67c409c010 con 0x7f67cc103400 2024-03-19T02:27:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.006+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67cc02d240 con 0x7f67cc103400 2024-03-19T02:27:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.006+0000 7f67b3fff640 1 -- 172.21.15.116:0/2648445830 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f67c409b910 con 0x7f67cc103400 2024-03-19T02:27:25.006 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 msgr2=0x7f67b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 0x7f67b40a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f67cc1028a0 tx=0x7f67c0005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 msgr2=0x7f67cc19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f67c400eb10 tx=0x7f67c400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 shutdown_connections 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f67b40a2880 0x7f67b40a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc103400 0x7f67cc19f020 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67cc102a40 0x7f67cc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 --2- 172.21.15.116:0/2648445830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67cc101850 0x7f67cc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.008+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 >> 172.21.15.116:0/2648445830 conn(0x7f67cc0fcdc0 msgr2=0x7f67cc105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:24 smithi116 ceph-mon[32263]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:25.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/804218843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3050873688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3685848289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2004016040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.009+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 shutdown_connections 2024-03-19T02:27:25.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.009+0000 7f67d13d4640 1 -- 172.21.15.116:0/2648445830 wait complete. 2024-03-19T02:27:25.019 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 -- 172.21.15.116:0/3386928901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc950100730 msgr2=0x7fc950100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 --2- 172.21.15.116:0/3386928901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc950100730 0x7fc950100b50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc948009ca0 tx=0x7fc94802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 -- 172.21.15.116:0/3386928901 shutdown_connections 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 --2- 172.21.15.116:0/3386928901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9501022e0 0x7fc950108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 --2- 172.21.15.116:0/3386928901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950101920 0x7fc950101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 --2- 172.21.15.116:0/3386928901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc950100730 0x7fc950100b50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 -- 172.21.15.116:0/3386928901 >> 172.21.15.116:0/3386928901 conn(0x7fc9500fbcc0 msgr2=0x7fc9500fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 -- 172.21.15.116:0/3386928901 shutdown_connections 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 -- 172.21.15.116:0/3386928901 wait complete. 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.125+0000 7fc955774640 1 Processor -- start 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- start start 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 0x7fc950197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc950101920 0x7fc950197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc95010c230 con 0x7fc9501022e0 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc95010c0b0 con 0x7fc950100730 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc95010c3b0 con 0x7fc950101920 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45002/0 (socket says 172.21.15.116:45002) 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 0x7fc950197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 -- 172.21.15.116:0/1261919448 learned_addr learned my addr 172.21.15.116:0/1261919448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94e7fc640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc950101920 0x7fc950197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 -- 172.21.15.116:0/1261919448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc950101920 msgr2=0x7fc950197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc950101920 0x7fc950197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 -- 172.21.15.116:0/1261919448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 msgr2=0x7fc950197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 0x7fc950197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 -- 172.21.15.116:0/1261919448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95019f6b0 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94effd640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 0x7fc950197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc94f7fe640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc94000e970 tx=0x7fc94000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94000ccc0 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc94000ce60 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95019f9a0 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940010430 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.126+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc95019fea0 con 0x7fc9501022e0 2024-03-19T02:27:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.127+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9501a00d0 con 0x7fc9501022e0 2024-03-19T02:27:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.128+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc940004ff0 con 0x7fc9501022e0 2024-03-19T02:27:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.129+0000 7fc947fff640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 0x7fc93c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.129+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc9400cf1f0 con 0x7fc9501022e0 2024-03-19T02:27:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.129+0000 7fc94effd640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 0x7fc93c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.129+0000 7fc94effd640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 0x7fc93c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc948032380 tx=0x7fc948000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.133+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc94009bf10 con 0x7fc9501022e0 2024-03-19T02:27:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:24 smithi162 ceph-mon[35761]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/804218843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3050873688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3685848289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2004016040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:24 smithi154 ceph-mon[35693]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T02:27:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/804218843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3050873688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3685848289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2004016040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.308+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9501a0340 con 0x7fc9501022e0 2024-03-19T02:27:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.308+0000 7fc947fff640 1 -- 172.21.15.116:0/1261919448 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc94009b810 con 0x7fc9501022e0 2024-03-19T02:27:25.308 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 msgr2=0x7fc93c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 0x7fc93c0a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc948032380 tx=0x7fc948000f40 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 msgr2=0x7fc95019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc94000e970 tx=0x7fc94000ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 shutdown_connections 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc93c0a2800 0x7fc93c0a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501022e0 0x7fc95019f040 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc950101920 0x7fc950197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 --2- 172.21.15.116:0/1261919448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc950100730 0x7fc950197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 >> 172.21.15.116:0/1261919448 conn(0x7fc9500fbcc0 msgr2=0x7fc950069d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 shutdown_connections 2024-03-19T02:27:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.310+0000 7fc955774640 1 -- 172.21.15.116:0/1261919448 wait complete. 2024-03-19T02:27:25.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.426+0000 7f4278af1640 1 -- 172.21.15.116:0/2372508842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274101910 msgr2=0x7f4274101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.426+0000 7f4278af1640 1 --2- 172.21.15.116:0/2372508842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274101910 0x7f4274101d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4268009900 tx=0x7f426802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- 172.21.15.116:0/2372508842 shutdown_connections 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- 172.21.15.116:0/2372508842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 0x7f4274108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- 172.21.15.116:0/2372508842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274101910 0x7f4274101d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- 172.21.15.116:0/2372508842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274100720 0x7f4274100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- 172.21.15.116:0/2372508842 >> 172.21.15.116:0/2372508842 conn(0x7f42740fbcb0 msgr2=0x7f42740fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- 172.21.15.116:0/2372508842 shutdown_connections 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- 172.21.15.116:0/2372508842 wait complete. 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 Processor -- start 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- start start 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 0x7f42741969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 0x7f427419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f427410c230 con 0x7f4274101910 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f427410c0b0 con 0x7f4274100720 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.427+0000 7f4278af1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f427410c3b0 con 0x7f42741022d0 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45014/0 (socket says 172.21.15.116:45014) 2024-03-19T02:27:25.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 -- 172.21.15.116:0/1510022921 learned_addr learned my addr 172.21.15.116:0/1510022921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f427259c640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 0x7f42741969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4272d9d640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 0x7f427419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 -- 172.21.15.116:0/1510022921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 msgr2=0x7f427419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 0x7f427419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 -- 172.21.15.116:0/1510022921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 msgr2=0x7f42741969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 0x7f42741969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 -- 172.21.15.116:0/1510022921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427419e5c0 con 0x7f4274101910 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f427259c640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 0x7f42741969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4271d9b640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4268006010 tx=0x7f4268005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268043020 con 0x7f4274101910 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427419e850 con 0x7f4274101910 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42680043d0 con 0x7f4274101910 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268003a80 con 0x7f4274101910 2024-03-19T02:27:25.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.428+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427419edf0 con 0x7f4274101910 2024-03-19T02:27:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.429+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4274107bb0 con 0x7f4274101910 2024-03-19T02:27:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.431+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4268046020 con 0x7f4274101910 2024-03-19T02:27:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.431+0000 7f42637fe640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 0x7f425c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.432+0000 7f427259c640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 0x7f425c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.432+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f42680f15f0 con 0x7f4274101910 2024-03-19T02:27:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.432+0000 7f427259c640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 0x7f425c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4274101770 tx=0x7f4264009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.435+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f426804d020 con 0x7f4274101910 2024-03-19T02:27:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.610+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f427402d1f0 con 0x7f4274101910 2024-03-19T02:27:25.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.610+0000 7f42637fe640 1 -- 172.21.15.116:0/1510022921 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f42680be310 con 0x7f4274101910 2024-03-19T02:27:25.610 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 msgr2=0x7f425c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 0x7f425c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4274101770 tx=0x7f4264009500 comp rx=0 tx=0).stop 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 msgr2=0x7f4274196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4268006010 tx=0x7f4268005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 shutdown_connections 2024-03-19T02:27:25.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f425c0a2800 0x7f425c0a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42741022d0 0x7f427419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4274101910 0x7f4274196ee0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 --2- 172.21.15.116:0/1510022921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4274100720 0x7f42741969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 >> 172.21.15.116:0/1510022921 conn(0x7f42740fbcb0 msgr2=0x7f4274104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 shutdown_connections 2024-03-19T02:27:25.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.612+0000 7f4278af1640 1 -- 172.21.15.116:0/1510022921 wait complete. 2024-03-19T02:27:25.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:27:25.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- 172.21.15.116:0/954793090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b0101830 msgr2=0x7fb8b0101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/954793090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b0101830 0x7fb8b0101c50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009ca0 tx=0x7fb8a802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- 172.21.15.116:0/954793090 shutdown_connections 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/954793090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b01033e0 0x7fb8b0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/954793090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0102a20 0x7fb8b0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/954793090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b0101830 0x7fb8b0101c50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- 172.21.15.116:0/954793090 >> 172.21.15.116:0/954793090 conn(0x7fb8b00fcdc0 msgr2=0x7fb8b00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- 172.21.15.116:0/954793090 shutdown_connections 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- 172.21.15.116:0/954793090 wait complete. 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 Processor -- start 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- start start 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 0x7fb8b0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b0102a20 0x7fb8b0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010d360 con 0x7fb8b01033e0 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010d1e0 con 0x7fb8b0101830 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.728+0000 7fb8b75f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010d4e0 con 0x7fb8b0102a20 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b538d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 0x7fb8b0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45038/0 (socket says 172.21.15.116:45038) 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 -- 172.21.15.116:0/4054085588 learned_addr learned my addr 172.21.15.116:0/4054085588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b4b8c640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b0102a20 0x7fb8b0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 -- 172.21.15.116:0/4054085588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b0102a20 msgr2=0x7fb8b0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b0102a20 0x7fb8b0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 -- 172.21.15.116:0/4054085588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 msgr2=0x7fb8b0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 0x7fb8b0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 -- 172.21.15.116:0/4054085588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b019f6c0 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b538d640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 0x7fb8b0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b5b8e640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000e970 tx=0x7fb8a000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a000ccc0 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8a000ce60 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b019f9b0 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0010430 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.729+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b019ff00 con 0x7fb8b01033e0 2024-03-19T02:27:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.730+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b0108c70 con 0x7fb8b01033e0 2024-03-19T02:27:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.731+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb8a0004f90 con 0x7fb8b01033e0 2024-03-19T02:27:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.732+0000 7fb8ae7fc640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 0x7fb8980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.732+0000 7fb8b538d640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 0x7fb8980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.732+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb8a00cf180 con 0x7fb8b01033e0 2024-03-19T02:27:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.732+0000 7fb8b538d640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 0x7fb8980a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8032380 tx=0x7fb8a8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.736+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb8a009bea0 con 0x7fb8b01033e0 2024-03-19T02:27:25.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.912+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb8b0062450 con 0x7fb8b01033e0 2024-03-19T02:27:25.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.912+0000 7fb8ae7fc640 1 -- 172.21.15.116:0/4054085588 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb8a009b7a0 con 0x7fb8b01033e0 2024-03-19T02:27:25.912 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 msgr2=0x7fb8980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 0x7fb8980a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8032380 tx=0x7fb8a8000f40 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 msgr2=0x7fb8b019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000e970 tx=0x7fb8a000ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 shutdown_connections 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb8980a2800 0x7fb8980a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b01033e0 0x7fb8b019f050 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b0102a20 0x7fb8b0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 --2- 172.21.15.116:0/4054085588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b0101830 0x7fb8b0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:25.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 >> 172.21.15.116:0/4054085588 conn(0x7fb8b00fcdc0 msgr2=0x7fb8b0105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:25.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 shutdown_connections 2024-03-19T02:27:25.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:25.914+0000 7fb8b75f1640 1 -- 172.21.15.116:0/4054085588 wait complete. 2024-03-19T02:27:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:25 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2648445830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:25 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1261919448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.924 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:25 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1510022921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:25.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 -- 172.21.15.116:0/3197296402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158102b00 msgr2=0x7f5158102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 --2- 172.21.15.116:0/3197296402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158102b00 0x7f5158102f80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5140009930 tx=0x7f514002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 -- 172.21.15.116:0/3197296402 shutdown_connections 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 --2- 172.21.15.116:0/3197296402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51581034c0 0x7f5158109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 --2- 172.21.15.116:0/3197296402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5158102b00 0x7f5158102f80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.026+0000 7f515d663640 1 --2- 172.21.15.116:0/3197296402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51580ffb30 0x7f51580fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- 172.21.15.116:0/3197296402 >> 172.21.15.116:0/3197296402 conn(0x7f51580fb460 msgr2=0x7f51580fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- 172.21.15.116:0/3197296402 shutdown_connections 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- 172.21.15.116:0/3197296402 wait complete. 2024-03-19T02:27:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 Processor -- start 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- start start 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5158102b00 0x7f5158198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f515810d3c0 con 0x7f51581034c0 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f515810d240 con 0x7f5158102b00 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.027+0000 7f515d663640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f515810d540 con 0x7f51580ffb30 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f51577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f51567fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5158102b00 0x7f5158198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34124/0 (socket says 172.21.15.116:34124) 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 -- 172.21.15.116:0/3692414565 learned_addr learned my addr 172.21.15.116:0/3692414565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f51577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45052/0 (socket says 172.21.15.116:45052) 2024-03-19T02:27:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 -- 172.21.15.116:0/3692414565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5158102b00 msgr2=0x7f5158198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5158102b00 0x7f5158198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 -- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 msgr2=0x7f515819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 -- 172.21.15.116:0/3692414565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515819f710 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f51577fe640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f5156ffd640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f515000ebc0 tx=0x7f515000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5150003b70 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f515819fa00 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5150003d10 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51500197c0 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.028+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515819ff20 con 0x7f51580ffb30 2024-03-19T02:27:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.029+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5158108d20 con 0x7f51580ffb30 2024-03-19T02:27:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.030+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5150004b60 con 0x7f51580ffb30 2024-03-19T02:27:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.031+0000 7f514ffff640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.031+0000 7f51567fc640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.031+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f51500cfa60 con 0x7f51580ffb30 2024-03-19T02:27:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.031+0000 7f51567fc640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 0x7f51440a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5140009900 tx=0x7f5140005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.039+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f515009c720 con 0x7f51580ffb30 2024-03-19T02:27:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:25 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2648445830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:25 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1261919448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:25 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1510022921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.251+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f515802d1f0 con 0x7f51580ffb30 2024-03-19T02:27:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.252+0000 7f514ffff640 1 -- 172.21.15.116:0/3692414565 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11950 (secure 0 0 0) 0x7f515009c020 con 0x7f51580ffb30 2024-03-19T02:27:26.252 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:26.252 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:25:36.674122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24463,24470,24484,24494,24451]},"id":1}]} 2024-03-19T02:27:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-03-19T02:27:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 msgr2=0x7f51440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 0x7f51440a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5140009900 tx=0x7f5140005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 msgr2=0x7f5158197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f515000ebc0 tx=0x7f515000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 shutdown_connections 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f51440a2800 0x7f51440a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51581034c0 0x7f515819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5158102b00 0x7f5158198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 --2- 172.21.15.116:0/3692414565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51580ffb30 0x7f5158197af0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 >> 172.21.15.116:0/3692414565 conn(0x7f51580fb460 msgr2=0x7f5158105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 shutdown_connections 2024-03-19T02:27:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.254+0000 7f515d663640 1 -- 172.21.15.116:0/3692414565 wait complete. 2024-03-19T02:27:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:25 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2648445830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:25 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1261919448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:25 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1510022921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:26.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.367+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2619930906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38103560 msgr2=0x7f3f38109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.367+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2619930906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38103560 0x7f3f38109df0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009930 tx=0x7f3f2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:26.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.367+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2619930906 shutdown_connections 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.367+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2619930906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38103560 0x7f3f38109df0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.367+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2619930906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38102ba0 0x7f3f38103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2619930906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 0x7f3f380fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2619930906 >> 172.21.15.116:0/2619930906 conn(0x7f3f380fb440 msgr2=0x7f3f380fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2619930906 shutdown_connections 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2619930906 wait complete. 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 Processor -- start 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- start start 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 0x7f3f38197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38103560 0x7f3f3819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3810d360 con 0x7f3f38102ba0 2024-03-19T02:27:26.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3810d1e0 con 0x7f3f380ffb30 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3f3b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3810d4e0 con 0x7f3f38103560 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3c94e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3c94e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45068/0 (socket says 172.21.15.116:45068) 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3c94e640 1 -- 172.21.15.116:0/2448006793 learned_addr learned my addr 172.21.15.116:0/2448006793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3d950640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38103560 0x7f3f3819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.368+0000 7f3f3d14f640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 0x7f3f38197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 -- 172.21.15.116:0/2448006793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38103560 msgr2=0x7f3f3819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38103560 0x7f3f3819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 -- 172.21.15.116:0/2448006793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 msgr2=0x7f3f38197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 0x7f3f38197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 -- 172.21.15.116:0/2448006793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3819f650 con 0x7f3f38102ba0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3c94e640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00ea60 tx=0x7f3f2c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00cde0 con 0x7f3f38102ba0 2024-03-19T02:27:26.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3819f940 con 0x7f3f38102ba0 2024-03-19T02:27:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f2c0035f0 con 0x7f3f38102ba0 2024-03-19T02:27:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c010690 con 0x7f3f38102ba0 2024-03-19T02:27:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.369+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3819fee0 con 0x7f3f38102ba0 2024-03-19T02:27:26.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.370+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f38108d70 con 0x7f3f38102ba0 2024-03-19T02:27:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.371+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3f2c0037a0 con 0x7f3f38102ba0 2024-03-19T02:27:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.371+0000 7f3f367fc640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 0x7f3f200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.372+0000 7f3f3d14f640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 0x7f3f200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.372+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3f2c0cf0e0 con 0x7f3f38102ba0 2024-03-19T02:27:26.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.372+0000 7f3f3d14f640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 0x7f3f200a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3f38069c10 tx=0x7f3f30009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.376+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f2c09be00 con 0x7f3f38102ba0 2024-03-19T02:27:26.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.551+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f38062420 con 0x7f3f38102ba0 2024-03-19T02:27:26.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.553+0000 7f3f367fc640 1 -- 172.21.15.116:0/2448006793 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7f3f2c09b700 con 0x7f3f38102ba0 2024-03-19T02:27:26.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:26.553 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 msgr2=0x7f3f200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 0x7f3f200a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3f38069c10 tx=0x7f3f30009500 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 msgr2=0x7f3f38197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00ea60 tx=0x7f3f2c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 shutdown_connections 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f3f200a2800 0x7f3f200a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f38103560 0x7f3f3819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f38102ba0 0x7f3f38197f70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 --2- 172.21.15.116:0/2448006793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f380ffb30 0x7f3f38197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 >> 172.21.15.116:0/2448006793 conn(0x7f3f380fb440 msgr2=0x7f3f38105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 shutdown_connections 2024-03-19T02:27:26.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.555+0000 7f3f3f3b3640 1 -- 172.21.15.116:0/2448006793 wait complete. 2024-03-19T02:27:26.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T02:25:08.015174+0000', 'modified': '2024-03-19T02:25:36.674122+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24494, 'mds_1': 24484, 'mds_2': 24470, 'mds_3': 24463, 'mds_4': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1891911264', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1891911264}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1891911264}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6841/3941624950', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 3941624950}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 3941624950}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6833/1010748955', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6832', 'nonce': 1010748955}, {'type': 'v1', 'addr': '172.21.15.154:6833', 'nonce': 1010748955}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6833/750284800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 750284800}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 750284800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6835/1464153745', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 1464153745}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 1464153745}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.162:6835/831287214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 831287214}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 831287214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.162:6837/3347737639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 3347737639}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 3347737639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.154:6837/455932973', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 455932973}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 455932973}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6839/1427360219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 1427360219}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 1427360219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/4009274564', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 4009274564}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 4009274564}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24463, 24470, 24484, 24494, 24451]} 2024-03-19T02:27:26.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T02:27:26.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-19T02:27:26.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- 172.21.15.116:0/2440359579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd01033e0 msgr2=0x7fdfd0109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/2440359579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd01033e0 0x7fdfd0109c70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009ca0 tx=0x7fdfc802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- 172.21.15.116:0/2440359579 shutdown_connections 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/2440359579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd01033e0 0x7fdfd0109c70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/2440359579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 0x7fdfd0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/2440359579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd0101830 0x7fdfd0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- 172.21.15.116:0/2440359579 >> 172.21.15.116:0/2440359579 conn(0x7fdfd00fcdc0 msgr2=0x7fdfd00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- 172.21.15.116:0/2440359579 shutdown_connections 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- 172.21.15.116:0/2440359579 wait complete. 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 Processor -- start 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 -- start start 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.670+0000 7fdfd73ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 0x7fdfd0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd01033e0 0x7fdfd019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd010d360 con 0x7fdfd0101830 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd010d1e0 con 0x7fdfd01033e0 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd010d4e0 con 0x7fdfd0102a20 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45080/0 (socket says 172.21.15.116:45080) 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 -- 172.21.15.116:0/1396663458 learned_addr learned my addr 172.21.15.116:0/1396663458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd4946640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 0x7fdfd0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5948640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd01033e0 0x7fdfd019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 -- 172.21.15.116:0/1396663458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 msgr2=0x7fdfd0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 0x7fdfd0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 -- 172.21.15.116:0/1396663458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd01033e0 msgr2=0x7fdfd019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd01033e0 0x7fdfd019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 -- 172.21.15.116:0/1396663458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd019f630 con 0x7fdfd0101830 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd5147640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e9c0 tx=0x7fdfcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1396663458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00cd80 con 0x7fdfd0101830 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd019f920 con 0x7fdfd0101830 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1396663458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfcc0035f0 con 0x7fdfd0101830 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1396663458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc010690 con 0x7fdfd0101830 2024-03-19T02:27:26.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.671+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd019fe70 con 0x7fdfd0101830 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfd9f8b740 1 Processor -- start 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1396663458 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fdfcc0037a0 con 0x7fdfd0101830 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfd9f8b740 1 -- start start 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f221f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221f9e0 0x5559f22244e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.674+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559f2183de0 con 0x5559f2224b50 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5559f212cc80 con 0x5559f221f9e0 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559f217fe80 con 0x5559f221eff0 2024-03-19T02:27:26.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45094/0 (socket says 172.21.15.116:45094) 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 -- 172.21.15.116:0/1243504100 learned_addr learned my addr 172.21.15.116:0/1243504100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd5948640 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f221f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd5147640 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221f9e0 0x5559f22244e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfc67fc640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 0x7fdfbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 -- 172.21.15.116:0/1243504100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 msgr2=0x5559f221f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f221f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 -- 172.21.15.116:0/1243504100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221f9e0 msgr2=0x5559f22244e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221f9e0 0x5559f22244e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 -- 172.21.15.116:0/1243504100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559f213efb0 con 0x5559f2224b50 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd5948640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 0x7fdfbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfc67fc640 1 -- 172.21.15.116:0/1396663458 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fdfcc0cf170 con 0x7fdfd0101830 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfd4946640 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009ca0 tx=0x7fdfc0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae14617fbfcb2d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfbb7fe640 1 -- 172.21.15.116:0/1243504100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc001e070 con 0x5559f2224b50 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfbb7fe640 1 -- 172.21.15.116:0/1243504100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc0019ae0 con 0x5559f2224b50 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.675+0000 7fdfbb7fe640 1 -- 172.21.15.116:0/1243504100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc000e970 con 0x5559f2224b50 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/1243504100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 msgr2=0x5559f2226fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd5948640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 0x7fdfbc0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8006010 tx=0x7fdfc802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009ca0 tx=0x7fdfc0011810 comp rx=0 tx=0).stop 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/1243504100 shutdown_connections 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f2226fe0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221f9e0 0x5559f22244e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/1243504100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f221f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/1243504100 >> 172.21.15.116:0/1243504100 conn(0x5559f221a560 msgr2=0x5559f221a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/1243504100 shutdown_connections 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/1243504100 wait complete. 2024-03-19T02:27:26.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.676+0000 7fdfd9f8b740 1 -- start start 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 Processor -- start 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f22294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f2230a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559f2227b00 con 0x5559f221f9e0 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5559f213ffc0 con 0x5559f2224b50 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559f2227c80 con 0x5559f221eff0 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd9f8b740 10 client.0 _ll_register_callbacks cb 0x5559f217ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45098/0 (socket says 172.21.15.116:45098) 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 -- 172.21.15.116:0/2809930018 learned_addr learned my addr 172.21.15.116:0/2809930018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd5948640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f2230a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd5147640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f22294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 -- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 msgr2=0x5559f22294c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f22294c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 -- 172.21.15.116:0/2809930018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 msgr2=0x5559f2230a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f2230a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.677+0000 7fdfd4946640 1 -- 172.21.15.116:0/2809930018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559f1e982d0 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd4946640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009c70 tx=0x7fdfc00099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec28f1893772b0fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0027020 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fdfc0005520 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc0019ae0 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdfc0032050 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 10 client.15432 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559f1f626b0 con 0x5559f221f9e0 2024-03-19T02:27:26.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc0019ae0 con 0x5559f221f9e0 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdfc000ea50 con 0x5559f221f9e0 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 10 client.15432 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 10 client.15432 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 20 client.15432 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f2236050 0x5559f22374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 4 client.15432 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.678+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559f2191320 con 0x5559f2236050 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfd5147640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f2236050 0x5559f22374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfb8ff9640 10 client.15432 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfd5147640 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f2236050 0x5559f22374c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a37522a630ae2ec server_cookie=5aeb6bc3c033d684 in_seq=0 out_seq=0 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/2809930018 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559f2191320 con 0x5559f2236050 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfb8ff9640 10 client.15432 handle_command_reply: tid=0 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.679+0000 7fdfd9f8b740 1 client.15432 shutdown 2024-03-19T02:27:26.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 msgr2=0x5559f2229a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009c70 tx=0x7fdfc00099f0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 shutdown_connections 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f2236050 0x5559f22374c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f2230a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfb8ff9640 0 client.15432 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221f9e0 0x5559f2229a00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/2809930018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221eff0 0x5559f22294c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 >> 172.21.15.116:0/2809930018 conn(0x5559f221a560 msgr2=0x5559f22197e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 shutdown_connections 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/2809930018 wait complete. 2024-03-19T02:27:26.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.680+0000 7fdfd9f8b740 20 client.15432 trim_cache size 0 max 16384 2024-03-19T02:27:26.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.697+0000 7fdfd9f8b740 1 Processor -- start 2024-03-19T02:27:26.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 -- start start 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221eff0 0x5559f2228050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2230a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559f2235800 con 0x5559f2224b50 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5559f2235ed0 con 0x5559f221eff0 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559f22365a0 con 0x5559f221f9e0 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45112/0 (socket says 172.21.15.116:45112) 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 -- 172.21.15.116:0/182911657 learned_addr learned my addr 172.21.15.116:0/182911657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5147640 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221eff0 0x5559f2228050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd4946640 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2230a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 -- 172.21.15.116:0/182911657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 msgr2=0x5559f2230a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2230a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 -- 172.21.15.116:0/182911657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221eff0 msgr2=0x5559f2228050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221eff0 0x5559f2228050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 -- 172.21.15.116:0/182911657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559f222a5f0 con 0x5559f2224b50 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfd5948640 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009ca0 tx=0x7fdfc8041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b9db630cfe62c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfbaffd640 1 -- 172.21.15.116:0/182911657 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8039040 con 0x5559f2224b50 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfbaffd640 1 -- 172.21.15.116:0/182911657 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8053070 con 0x5559f2224b50 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.698+0000 7fdfbaffd640 1 -- 172.21.15.116:0/182911657 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80416d0 con 0x5559f2224b50 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/182911657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 msgr2=0x5559f221e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009ca0 tx=0x7fdfc8041000 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/182911657 shutdown_connections 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f2224b50 0x5559f221e050 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2230a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/182911657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f221eff0 0x5559f2228050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/182911657 >> 172.21.15.116:0/182911657 conn(0x5559f221a560 msgr2=0x5559f2220480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/182911657 shutdown_connections 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/182911657 wait complete. 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 -- start start 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.699+0000 7fdfd9f8b740 1 Processor -- start 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2297da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f229a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559f2237340 con 0x5559f221eff0 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5559f2236c70 con 0x5559f2224b50 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559f2237a10 con 0x5559f221f9e0 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd9f8b740 10 client.0 _ll_register_callbacks cb 0x7fdfc001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45124/0 (socket says 172.21.15.116:45124) 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 -- 172.21.15.116:0/4170924128 learned_addr learned my addr 172.21.15.116:0/4170924128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd4946640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2297da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5948640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f229a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 -- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 msgr2=0x5559f2297da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2297da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 -- 172.21.15.116:0/4170924128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 msgr2=0x5559f229a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f229a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 -- 172.21.15.116:0/4170924128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559f229a880 con 0x5559f221eff0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfd5147640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc09b720 tx=0x7fdfcc0d2c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ed85c9b620cda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.700+0000 7fdfb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc09b560 con 0x5559f221eff0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fdfcc099a90 con 0x5559f221eff0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc8053070 con 0x5559f221eff0 2024-03-19T02:27:26.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdfcc09b0f0 con 0x5559f221eff0 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 10 client.15438 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559f229bc50 con 0x5559f221eff0 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc8053070 con 0x5559f221eff0 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdfcc097e00 con 0x5559f221eff0 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 10 client.15438 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 10 client.15438 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 20 client.15438 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f22a0800 0x5559f229f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 4 client.15438 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "status"}) -- 0x5559f218b670 con 0x5559f22a0800 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.701+0000 7fdfd4946640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f22a0800 0x5559f229f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfb8ff9640 10 client.15438 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:26.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd4946640 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f22a0800 0x5559f229f510 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5cc01102983ad99b server_cookie=cd346aff23eacdb0 in_seq=0 out_seq=0 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfb8ff9640 1 -- 172.21.15.116:0/4170924128 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5559f218b670 con 0x5559f22a0800 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfb8ff9640 10 client.15438 handle_command_reply: tid=0 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 client.15438 shutdown 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 msgr2=0x5559f222e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc09b720 tx=0x7fdfcc0d2c50 comp rx=0 tx=0).stop 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 shutdown_connections 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5559f22a0800 0x5559f229f510 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559f2224b50 0x5559f229a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559f221f9e0 0x5559f2297da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfb8ff9640 0 client.15438 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 --2- 172.21.15.116:0/4170924128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559f221eff0 0x5559f222e9f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 >> 172.21.15.116:0/4170924128 conn(0x5559f221a560 msgr2=0x5559f2228050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 shutdown_connections 2024-03-19T02:27:26.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 1 -- 172.21.15.116:0/4170924128 wait complete. 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.702+0000 7fdfd9f8b740 20 client.15438 trim_cache size 0 max 16384 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24451, 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:26.710 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 110.98377972999999, 2024-03-19T02:27:26.711 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:26.711 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 94, 2024-03-19T02:27:26.711 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:26.711 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 159.58568770100001 2024-03-19T02:27:26.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 msgr2=0x7fdfbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 0x7fdfbc0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8006010 tx=0x7fdfc802e040 comp rx=0 tx=0).stop 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 msgr2=0x7fdfd0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00e9c0 tx=0x7fdfcc00ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 shutdown_connections 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fdfbc0a2800 0x7fdfbc0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfd01033e0 0x7fdfd019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfd0102a20 0x7fdfd0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 --2- 172.21.15.116:0/1396663458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd0101830 0x7fdfd0197a10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 >> 172.21.15.116:0/1396663458 conn(0x7fdfd00fcdc0 msgr2=0x7fdfd0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 shutdown_connections 2024-03-19T02:27:26.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.703+0000 7fdfd73ab640 1 -- 172.21.15.116:0/1396663458 wait complete. 2024-03-19T02:27:26.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-19T02:27:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/2521508504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 msgr2=0x7f6fbc102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/2521508504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc102170 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009900 tx=0x7f6fac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/2521508504 shutdown_connections 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/2521508504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 0x7f6fbc108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/2521508504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 0x7f6fbc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/2521508504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc102170 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/2521508504 >> 172.21.15.116:0/2521508504 conn(0x7f6fbc0fb450 msgr2=0x7f6fbc0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/2521508504 shutdown_connections 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.819+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/2521508504 wait complete. 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 Processor -- start 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 -- start start 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 0x7f6fbc196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 0x7f6fbc196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc10c280 con 0x7f6fbc101d50 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc10c100 con 0x7f6fbc1009a0 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc10c400 con 0x7f6fbc101360 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45144/0 (socket says 172.21.15.116:45144) 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 -- 172.21.15.116:0/629640015 learned_addr learned my addr 172.21.15.116:0/629640015 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 0x7f6fbc196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fba59c640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 0x7f6fbc196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 -- 172.21.15.116:0/629640015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 msgr2=0x7f6fbc196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 0x7f6fbc196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 -- 172.21.15.116:0/629640015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 msgr2=0x7f6fbc196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 0x7f6fbc196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 -- 172.21.15.116:0/629640015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc19e5a0 con 0x7f6fbc101d50 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fbad9d640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400eb10 tx=0x7f6fb400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fab7fe640 1 -- 172.21.15.116:0/629640015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4004b20 con 0x7f6fbc101d50 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fab7fe640 1 -- 172.21.15.116:0/629640015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fb4004cc0 con 0x7f6fbc101d50 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.820+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fbc19e890 con 0x7f6fbc101d50 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.821+0000 7f6fab7fe640 1 -- 172.21.15.116:0/629640015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4010690 con 0x7f6fbc101d50 2024-03-19T02:27:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.821+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fbc19ee30 con 0x7f6fbc101d50 2024-03-19T02:27:26.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fab7fe640 1 -- 172.21.15.116:0/629640015 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6fb4005020 con 0x7f6fbc101d50 2024-03-19T02:27:26.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 Processor -- start 2024-03-19T02:27:26.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 -- start start 2024-03-19T02:27:26.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de1f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de26fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55971dd83de0 con 0x55971de1f9e0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971dd2cc80 con 0x55971de24b50 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55971dd7fe80 con 0x55971de1eff0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fab7fe640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 0x7f6fa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45148/0 (socket says 172.21.15.116:45148) 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fba59c640 1 -- 172.21.15.116:0/703669374 learned_addr learned my addr 172.21.15.116:0/703669374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de1f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.823+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de26fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fbad9d640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 0x7f6fa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 -- 172.21.15.116:0/703669374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 msgr2=0x55971de1f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de1f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 -- 172.21.15.116:0/703669374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 msgr2=0x55971de26fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de26fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 -- 172.21.15.116:0/703669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971dd3efb0 con 0x55971de1f9e0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de1f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fab7fe640 1 -- 172.21.15.116:0/629640015 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6fb40cf420 con 0x7f6fbc101d50 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fba59c640 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009900 tx=0x7f6fac009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346e6c44a9531261 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fbad9d640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 0x7f6fa40a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fb409b740 tx=0x7f6fb4086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fa8ff9640 1 -- 172.21.15.116:0/703669374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac045020 con 0x55971de1f9e0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fa8ff9640 1 -- 172.21.15.116:0/703669374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fac004040 con 0x55971de1f9e0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.824+0000 7f6fa8ff9640 1 -- 172.21.15.116:0/703669374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac005540 con 0x55971de1f9e0 2024-03-19T02:27:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- 172.21.15.116:0/703669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 msgr2=0x55971de244e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009900 tx=0x7f6fac009560 comp rx=0 tx=0).stop 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- 172.21.15.116:0/703669374 shutdown_connections 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de26fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de244e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/703669374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de1f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- 172.21.15.116:0/703669374 >> 172.21.15.116:0/703669374 conn(0x55971de1a560 msgr2=0x55971de1a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- 172.21.15.116:0/703669374 shutdown_connections 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- 172.21.15.116:0/703669374 wait complete. 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- start start 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 Processor -- start 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1f9e0 0x55971de299f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de30a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55971de27aa0 con 0x55971de1eff0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971dd3ffc0 con 0x55971de24b50 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55971de27c20 con 0x55971de1f9e0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.825+0000 7f6fc37d9740 10 client.0 _ll_register_callbacks cb 0x55971dd7ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45154/0 (socket says 172.21.15.116:45154) 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/1655302059 learned_addr learned my addr 172.21.15.116:0/1655302059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1f9e0 0x55971de299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de30a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1f9e0 msgr2=0x55971de299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1f9e0 0x55971de299f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/1655302059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 msgr2=0x55971de30a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de30a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/1655302059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971da982d0 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00043e0 tx=0x7f6fb0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35ca6f4423a2559 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0005220 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6fb00053c0 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fac004040 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6fb002a050 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fc37d9740 10 client.15450 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55971db626b0 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.826+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fac004040 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6fb0011880 con 0x55971de1eff0 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 10 client.15450 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:26.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 10 client.15450 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 20 client.15450 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971de362b0 0x55971de37720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 4 client.15450 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55971dd91320 con 0x55971de362b0 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971de362b0 0x55971de37720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fa1ffb640 10 client.15450 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.827+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971de362b0 0x55971de37720 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ad93bbc145cac806 server_cookie=6fa20a4f301f2296 in_seq=0 out_seq=0 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1655302059 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55971dd91320 con 0x55971de362b0 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fa1ffb640 10 client.15450 handle_command_reply: tid=0 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 client.15450 shutdown 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 msgr2=0x55971de294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00043e0 tx=0x7f6fb0005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 shutdown_connections 2024-03-19T02:27:26.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de30a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1f9e0 0x55971de299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971de362b0 0x55971de37720 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1655302059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de294b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fa1ffb640 0 client.15450 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 >> 172.21.15.116:0/1655302059 conn(0x55971de1a560 msgr2=0x55971de197e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 shutdown_connections 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1655302059 wait complete. 2024-03-19T02:27:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.828+0000 7f6fc37d9740 20 client.15450 trim_cache size 0 max 16384 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 Processor -- start 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 -- start start 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de1f9e0 0x55971de309c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de24b50 0x55971de20e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55971de35a60 con 0x55971de1eff0 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971de36130 con 0x55971de1f9e0 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.846+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55971de36800 con 0x55971de24b50 2024-03-19T02:27:26.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45158/0 (socket says 172.21.15.116:45158) 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/90886641 learned_addr learned my addr 172.21.15.116:0/90886641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fba59c640 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de1f9e0 0x55971de309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de24b50 0x55971de20e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/90886641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de24b50 msgr2=0x55971de20e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de24b50 0x55971de20e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/90886641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de1f9e0 msgr2=0x55971de309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de1f9e0 0x55971de309c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 -- 172.21.15.116:0/90886641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971de24130 con 0x55971de1eff0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00057c0 tx=0x7f6fb00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5aaee8e1b7efa4b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fa3fff640 1 -- 172.21.15.116:0/90886641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0011960 con 0x55971de1eff0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fa3fff640 1 -- 172.21.15.116:0/90886641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fb0011070 con 0x55971de1eff0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.847+0000 7f6fa3fff640 1 -- 172.21.15.116:0/90886641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0010430 con 0x55971de1eff0 2024-03-19T02:27:26.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- 172.21.15.116:0/90886641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 msgr2=0x55971de35100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00057c0 tx=0x7f6fb00092d0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- 172.21.15.116:0/90886641 shutdown_connections 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de24b50 0x55971de20e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de1f9e0 0x55971de309c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/90886641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1eff0 0x55971de35100 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- 172.21.15.116:0/90886641 >> 172.21.15.116:0/90886641 conn(0x55971de1a560 msgr2=0x55971de1a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- 172.21.15.116:0/90886641 shutdown_connections 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- 172.21.15.116:0/90886641 wait complete. 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- start start 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 Processor -- start 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de2eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de2f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55971de375a0 con 0x55971de1f9e0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971de36ed0 con 0x55971de24b50 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55971de37c70 con 0x55971de1eff0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fc37d9740 10 client.0 _ll_register_callbacks cb 0x7f6fb0004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45164/0 (socket says 172.21.15.116:45164) 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fba59c640 1 -- 172.21.15.116:0/1933645814 learned_addr learned my addr 172.21.15.116:0/1933645814 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.848+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de2eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de2f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 -- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 msgr2=0x55971de2eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de2eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 -- 172.21.15.116:0/1933645814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 msgr2=0x55971de2f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de2f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 -- 172.21.15.116:0/1933645814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971de2fe40 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fba59c640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6fac005ab0 tx=0x7f6fac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cc17b43b22b05d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac045040 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6fac02c8e0 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fb0011070 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6fac05f050 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fc37d9740 10 client.15456 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55971de9e180 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fb0011070 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.849+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6fac041a50 con 0x55971de1f9e0 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 10 client.15456 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 10 client.15456 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 20 client.15456 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:26.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971dea2990 0x55971dea16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 4 client.15456 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "status"}) -- 0x55971dd8b670 con 0x55971dea2990 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971dea2990 0x55971dea16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fb9d9b640 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971dea2990 0x55971dea16a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a9543a2f0c242a0 server_cookie=43e5b93aae31c6a7 in_seq=0 out_seq=0 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fa1ffb640 10 client.15456 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fa1ffb640 1 -- 172.21.15.116:0/1933645814 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55971dd8b670 con 0x55971dea2990 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fa1ffb640 10 client.15456 handle_command_reply: tid=0 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 client.15456 shutdown 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 msgr2=0x55971de2f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6fac005ab0 tx=0x7f6fac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 shutdown_connections 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971de24b50 0x55971de2f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55971dea2990 0x55971dea16a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55971de1f9e0 0x55971de2f0d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fa1ffb640 0 client.15456 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 --2- 172.21.15.116:0/1933645814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55971de1eff0 0x55971de2eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 >> 172.21.15.116:0/1933645814 conn(0x55971de1a560 msgr2=0x55971de35100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 shutdown_connections 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.850+0000 7f6fc37d9740 1 -- 172.21.15.116:0/1933645814 wait complete. 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc37d9740 20 client.15456 trim_cache size 0 max 16384 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-19T02:27:26.858 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24463, 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 112.145948072, 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 94, 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 156.811813844 2024-03-19T02:27:26.859 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 msgr2=0x7f6fa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 0x7f6fa40a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fb409b740 tx=0x7f6fb4086040 comp rx=0 tx=0).stop 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 msgr2=0x7f6fbc19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400eb10 tx=0x7f6fb400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 shutdown_connections 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6fa40a2880 0x7f6fa40a4d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fbc101d50 0x7f6fbc19df30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fbc101360 0x7f6fbc196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 --2- 172.21.15.116:0/629640015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1009a0 0x7f6fbc196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 >> 172.21.15.116:0/629640015 conn(0x7f6fbc0fb450 msgr2=0x7f6fbc104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 shutdown_connections 2024-03-19T02:27:26.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.851+0000 7f6fc0bf3640 1 -- 172.21.15.116:0/629640015 wait complete. 2024-03-19T02:27:26.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:26 smithi116 ceph-mon[32263]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:26.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:26 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4054085588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:26.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:26 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3692414565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:26.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:26 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2448006793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:26.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.964+0000 7ff753b48640 1 -- 172.21.15.116:0/282419294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 msgr2=0x7ff74c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.964+0000 7ff753b48640 1 --2- 172.21.15.116:0/282419294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c101d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009900 tx=0x7ff73c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- 172.21.15.116:0/282419294 shutdown_connections 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- 172.21.15.116:0/282419294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- 172.21.15.116:0/282419294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c101d90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- 172.21.15.116:0/282419294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- 172.21.15.116:0/282419294 >> 172.21.15.116:0/282419294 conn(0x7ff74c0fbcb0 msgr2=0x7ff74c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- 172.21.15.116:0/282419294 shutdown_connections 2024-03-19T02:27:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- 172.21.15.116:0/282419294 wait complete. 2024-03-19T02:27:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 Processor -- start 2024-03-19T02:27:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- start start 2024-03-19T02:27:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c10c230 con 0x7ff74c100720 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c10c0b0 con 0x7ff74c101910 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.965+0000 7ff753b48640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff74c10c3b0 con 0x7ff74c1022d0 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45174/0 (socket says 172.21.15.116:45174) 2024-03-19T02:27:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 -- 172.21.15.116:0/2321237823 learned_addr learned my addr 172.21.15.116:0/2321237823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7510e3640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7520e5640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 -- 172.21.15.116:0/2321237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 msgr2=0x7ff74c19df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c19df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 -- 172.21.15.116:0/2321237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 msgr2=0x7ff74c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 -- 172.21.15.116:0/2321237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c19e5e0 con 0x7ff74c100720 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7520e5640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff74800ead0 tx=0x7ff74800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff742ffd640 1 -- 172.21.15.116:0/2321237823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74800bbd0 con 0x7ff74c100720 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74c0689e0 con 0x7ff74c100720 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff742ffd640 1 -- 172.21.15.116:0/2321237823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74800bd70 con 0x7ff74c100720 2024-03-19T02:27:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff742ffd640 1 -- 172.21.15.116:0/2321237823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748017410 con 0x7ff74c100720 2024-03-19T02:27:26.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.966+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff74c068f00 con 0x7ff74c100720 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.968+0000 7ff742ffd640 1 -- 172.21.15.116:0/2321237823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ff74801e020 con 0x7ff74c100720 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 Processor -- start 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 -- start start 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff742ffd640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 0x7ff7380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99ee4b50 0x556b99ee6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556b99e43de0 con 0x556b99edeff0 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b99decc80 con 0x556b99edf9e0 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff75680c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b99e3fe80 con 0x556b99ee4b50 2024-03-19T02:27:26.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7520e5640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 0x7ff7380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff742ffd640 1 -- 172.21.15.116:0/2321237823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff7480d0500 con 0x7ff74c100720 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7518e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7518e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45188/0 (socket says 172.21.15.116:45188) 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7510e3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7518e4640 1 -- 172.21.15.116:0/2230014104 learned_addr learned my addr 172.21.15.116:0/2230014104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.969+0000 7ff7520e5640 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99ee4b50 0x556b99ee6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 -- 172.21.15.116:0/2230014104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99ee4b50 msgr2=0x556b99ee6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99ee4b50 0x556b99ee6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 -- 172.21.15.116:0/2230014104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 msgr2=0x556b99ee44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 -- 172.21.15.116:0/2230014104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b99dfefb0 con 0x556b99edeff0 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7510e3640 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:26.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7520e5640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 0x7ff7380a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff744005fd0 tx=0x7ff744009eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff7518e4640 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff748098f80 tx=0x7ff74809c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab6bd0f5b9557e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff737fff640 1 -- 172.21.15.116:0/2230014104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74809d220 con 0x556b99edeff0 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff737fff640 1 -- 172.21.15.116:0/2230014104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74809de50 con 0x556b99edeff0 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.970+0000 7ff737fff640 1 -- 172.21.15.116:0/2230014104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74809ae20 con 0x556b99edeff0 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- 172.21.15.116:0/2230014104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 msgr2=0x556b99edf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff748098f80 tx=0x7ff74809c480 comp rx=0 tx=0).stop 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- 172.21.15.116:0/2230014104 shutdown_connections 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99ee4b50 0x556b99ee6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- 172.21.15.116:0/2230014104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99edeff0 0x556b99edf410 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- 172.21.15.116:0/2230014104 >> 172.21.15.116:0/2230014104 conn(0x556b99eda560 msgr2=0x556b99eda960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- 172.21.15.116:0/2230014104 shutdown_connections 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- 172.21.15.116:0/2230014104 wait complete. 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- start start 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 Processor -- start 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edeff0 0x556b99ee9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556b99ee7b00 con 0x556b99ee4b50 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b99dfffc0 con 0x556b99edf9e0 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.971+0000 7ff75680c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b99ee7c80 con 0x556b99edeff0 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff75680c740 10 client.0 _ll_register_callbacks cb 0x556b99e3ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:26.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45194/0 (socket says 172.21.15.116:45194) 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 -- 172.21.15.116:0/88428424 learned_addr learned my addr 172.21.15.116:0/88428424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7518e4640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edeff0 0x556b99ee9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7510e3640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 -- 172.21.15.116:0/88428424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edeff0 msgr2=0x556b99ee9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edeff0 0x556b99ee9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 -- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 msgr2=0x556b99ee9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 -- 172.21.15.116:0/88428424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b99b582d0 con 0x556b99ee4b50 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7520e5640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff744003c20 tx=0x7ff74401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825ff39d496813a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7357fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744003e60 con 0x556b99ee4b50 2024-03-19T02:27:26.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74809de50 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff744020410 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff744020610 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff75680c740 10 client.15468 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b99c226b0 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.972+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74809de50 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff74401f050 con 0x556b99ee4b50 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 10 client.15468 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 20 client.15468 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99ef6390 0x556b99ef7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 4 client.15468 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b99e51320 con 0x556b99ef6390 2024-03-19T02:27:26.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff7518e4640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99ef6390 0x556b99ef7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff7518e4640 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99ef6390 0x556b99ef7800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69aa25809c614d46 server_cookie=34e61da98bb3090e in_seq=0 out_seq=0 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.973+0000 7ff7357fa640 10 client.15468 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff7357fa640 1 -- 172.21.15.116:0/88428424 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b99e51320 con 0x556b99ef6390 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff7357fa640 10 client.15468 handle_command_reply: tid=0 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 client.15468 shutdown 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 msgr2=0x556b99ef0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff744003c20 tx=0x7ff74401c000 comp rx=0 tx=0).stop 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 shutdown_connections 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99ef6390 0x556b99ef7800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0af0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edf9e0 0x556b99ee9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 --2- 172.21.15.116:0/88428424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edeff0 0x556b99ee9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff7357fa640 0 client.15468 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:26.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.974+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 >> 172.21.15.116:0/88428424 conn(0x556b99eda560 msgr2=0x556b99ed97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:26.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.975+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 shutdown_connections 2024-03-19T02:27:26.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.975+0000 7ff75680c740 1 -- 172.21.15.116:0/88428424 wait complete. 2024-03-19T02:27:26.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.975+0000 7ff75680c740 20 client.15468 trim_cache size 0 max 16384 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 Processor -- start 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 -- start start 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99ee0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99ee14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef5b40 con 0x556b99ee4b50 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef6210 con 0x556b99edeff0 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.992+0000 7ff75680c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef68e0 con 0x556b99edf9e0 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45200/0 (socket says 172.21.15.116:45200) 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 -- 172.21.15.116:0/588597515 learned_addr learned my addr 172.21.15.116:0/588597515 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7518e4640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99ee0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7510e3640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99ee14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 -- 172.21.15.116:0/588597515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 msgr2=0x556b99ee14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99ee14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 -- 172.21.15.116:0/588597515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 msgr2=0x556b99ee0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99ee0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 -- 172.21.15.116:0/588597515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b99edfdc0 con 0x556b99ee4b50 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7518e4640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99ee0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7520e5640 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff744009be0 tx=0x7ff744012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed90d177c713096 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7377fe640 1 -- 172.21.15.116:0/588597515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7440047d0 con 0x556b99ee4b50 2024-03-19T02:27:26.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7377fe640 1 -- 172.21.15.116:0/588597515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff744004a00 con 0x556b99ee4b50 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.993+0000 7ff7377fe640 1 -- 172.21.15.116:0/588597515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74402a650 con 0x556b99ee4b50 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- 172.21.15.116:0/588597515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 msgr2=0x556b99ef0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff744009be0 tx=0x7ff744012a30 comp rx=0 tx=0).stop 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- 172.21.15.116:0/588597515 shutdown_connections 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99ef0ab0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99ee14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- 172.21.15.116:0/588597515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99ee0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- 172.21.15.116:0/588597515 >> 172.21.15.116:0/588597515 conn(0x556b99eda560 msgr2=0x556b99ee9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- 172.21.15.116:0/588597515 shutdown_connections 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- 172.21.15.116:0/588597515 wait complete. 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- start start 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 Processor -- start 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99eecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99eed180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef7680 con 0x556b99ee4b50 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef6fb0 con 0x556b99edeff0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b99ef7d50 con 0x556b99edf9e0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.994+0000 7ff75680c740 10 client.0 _ll_register_callbacks cb 0x7ff744009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45214/0 (socket says 172.21.15.116:45214) 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 -- 172.21.15.116:0/3121496897 learned_addr learned my addr 172.21.15.116:0/3121496897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7510e3640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99eed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7518e4640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99eecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 -- 172.21.15.116:0/3121496897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 msgr2=0x556b99eed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99eed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 -- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 msgr2=0x556b99eecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99eecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 -- 172.21.15.116:0/3121496897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b99ef0300 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7520e5640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff74402a8b0 tx=0x7ff74402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2409cc82ee7c8d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7357fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7440207b0 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff744020950 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff744004a00 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff744049050 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff75680c740 10 client.15474 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b99f5c6c0 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.995+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff744004a00 con 0x556b99ee4b50 2024-03-19T02:27:27.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff744038900 con 0x556b99ee4b50 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 10 client.15474 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 10 client.15474 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 20 client.15474 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99f61050 0x556b99f5fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 4 client.15474 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "status"}) -- 0x556b99e4b670 con 0x556b99f61050 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7518e4640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99f61050 0x556b99f5fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7357fa640 10 client.15474 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7518e4640 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99f61050 0x556b99f5fd60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3813d758dcef878c server_cookie=ffd7b22c548a1eee in_seq=0 out_seq=0 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7357fa640 1 -- 172.21.15.116:0/3121496897 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556b99e4b670 con 0x556b99f61050 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff7357fa640 10 client.15474 handle_command_reply: tid=0 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 1 client.15474 shutdown 2024-03-19T02:27:27.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 msgr2=0x556b99eefd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.996+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff74402a8b0 tx=0x7ff74402a8e0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 shutdown_connections 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556b99f61050 0x556b99f5fd60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b99ee4b50 0x556b99eefd10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b99edf9e0 0x556b99eed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff7357fa640 0 client.15474 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 --2- 172.21.15.116:0/3121496897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556b99edeff0 0x556b99eecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 >> 172.21.15.116:0/3121496897 conn(0x556b99eda560 msgr2=0x556b99ee9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 shutdown_connections 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 1 -- 172.21.15.116:0/3121496897 wait complete. 2024-03-19T02:27:27.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff75680c740 20 client.15474 trim_cache size 0 max 16384 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24470, 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 113.298174514, 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 94, 2024-03-19T02:27:27.005 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:27.006 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 153.97209781699999 2024-03-19T02:27:27.006 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 msgr2=0x7ff7380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 0x7ff7380a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff744005fd0 tx=0x7ff744009eb0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 msgr2=0x7ff74c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff74800ead0 tx=0x7ff74800ef90 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 shutdown_connections 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff7380a2800 0x7ff7380a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff74c1022d0 0x7ff74c19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff74c101910 0x7ff74c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 --2- 172.21.15.116:0/2321237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff74c100720 0x7ff74c1969c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 >> 172.21.15.116:0/2321237823 conn(0x7ff74c0fbcb0 msgr2=0x7ff74c104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 shutdown_connections 2024-03-19T02:27:27.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:26.997+0000 7ff753b48640 1 -- 172.21.15.116:0/2321237823 wait complete. 2024-03-19T02:27:27.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 -- 172.21.15.116:0/1248010837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7300102a20 msgr2=0x7f7300102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 --2- 172.21.15.116:0/1248010837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7300102a20 0x7f7300102ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009860 tx=0x7f72f402bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 -- 172.21.15.116:0/1248010837 shutdown_connections 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 --2- 172.21.15.116:0/1248010837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73001033e0 0x7f7300109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 --2- 172.21.15.116:0/1248010837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7300102a20 0x7f7300102ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 --2- 172.21.15.116:0/1248010837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300101830 0x7f7300101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 -- 172.21.15.116:0/1248010837 >> 172.21.15.116:0/1248010837 conn(0x7f73000fcdc0 msgr2=0x7f73000ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 -- 172.21.15.116:0/1248010837 shutdown_connections 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 -- 172.21.15.116:0/1248010837 wait complete. 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.112+0000 7f730763f640 1 Processor -- start 2024-03-19T02:27:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- start start 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 0x7f7300073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73001033e0 0x7f7300074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d340 con 0x7f7300101830 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d1c0 con 0x7f73001033e0 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f730010d4c0 con 0x7f7300102a20 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45232/0 (socket says 172.21.15.116:45232) 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 -- 172.21.15.116:0/2026072682 learned_addr learned my addr 172.21.15.116:0/2026072682 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73001033e0 0x7f7300074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f7304bda640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 0x7f7300073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 -- 172.21.15.116:0/2026072682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 msgr2=0x7f7300073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 0x7f7300073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 -- 172.21.15.116:0/2026072682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73001033e0 msgr2=0x7f7300074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73001033e0 0x7f7300074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 -- 172.21.15.116:0/2026072682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73000749e0 con 0x7f7300101830 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f7304bda640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 0x7f7300073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f73053db640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f72f800ebc0 tx=0x7f72f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f72fe7fc640 1 -- 172.21.15.116:0/2026072682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8003b70 con 0x7f7300101830 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73001a1be0 con 0x7f7300101830 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f72fe7fc640 1 -- 172.21.15.116:0/2026072682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f8003d10 con 0x7f7300101830 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f72fe7fc640 1 -- 172.21.15.116:0/2026072682 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8019820 con 0x7f7300101830 2024-03-19T02:27:27.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.113+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73001a2150 con 0x7f7300101830 2024-03-19T02:27:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.116+0000 7f72fe7fc640 1 -- 172.21.15.116:0/2026072682 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f72f80199c0 con 0x7f7300101830 2024-03-19T02:27:27.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.116+0000 7f730a220740 1 Processor -- start 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 -- start start 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e174e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e19fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8d76de0 con 0x55b5f8e17b50 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8d1fc80 con 0x55b5f8e129e0 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f730a220740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8d72e80 con 0x55b5f8e11ff0 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e19fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e19fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45244/0 (socket says 172.21.15.116:45244) 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34308/0 (socket says 172.21.15.116:34308) 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7304bda640 1 -- 172.21.15.116:0/2100097701 learned_addr learned my addr 172.21.15.116:0/2100097701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f73053db640 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f72fe7fc640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 -- 172.21.15.116:0/2100097701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 msgr2=0x55b5f8e174e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e174e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 -- 172.21.15.116:0/2100097701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 msgr2=0x55b5f8e19fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e19fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 -- 172.21.15.116:0/2100097701 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f8d31fb0 con 0x55b5f8e11ff0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f72fe7fc640 1 -- 172.21.15.116:0/2026072682 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f72f80cf6e0 con 0x7f7300101830 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.117+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72f0005fd0 tx=0x7f72f0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d63afffc1e0f5f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f72e77fe640 1 -- 172.21.15.116:0/2100097701 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0017050 con 0x55b5f8e11ff0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f72e77fe640 1 -- 172.21.15.116:0/2100097701 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f00268f0 con 0x55b5f8e11ff0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f72e77fe640 1 -- 172.21.15.116:0/2100097701 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f000f6b0 con 0x55b5f8e11ff0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f7305bdc640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 0x7f72e80a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72f0023000 tx=0x7f72f0024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 -- 172.21.15.116:0/2100097701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 msgr2=0x55b5f8e12410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72f0005fd0 tx=0x7f72f0011810 comp rx=0 tx=0).stop 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 -- 172.21.15.116:0/2100097701 shutdown_connections 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e19fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e174e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 --2- 172.21.15.116:0/2100097701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e12410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 -- 172.21.15.116:0/2100097701 >> 172.21.15.116:0/2100097701 conn(0x55b5f8e0d560 msgr2=0x55b5f8e0d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 -- 172.21.15.116:0/2100097701 shutdown_connections 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.118+0000 7f730a220740 1 -- 172.21.15.116:0/2100097701 wait complete. 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 -- start start 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 Processor -- start 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e1c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 0x55b5f8e23a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e1ab00 con 0x55b5f8e129e0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8d32fc0 con 0x55b5f8e11ff0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e1ac80 con 0x55b5f8e17b50 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f730a220740 10 client.0 _ll_register_callbacks cb 0x55b5f8d6ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f73053db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e1c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45256/0 (socket says 172.21.15.116:45256) 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f7304bda640 1 -- 172.21.15.116:0/3063769931 learned_addr learned my addr 172.21.15.116:0/3063769931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.119+0000 7f7305bdc640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 0x55b5f8e23a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 -- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 msgr2=0x55b5f8e23a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 0x55b5f8e23a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 -- 172.21.15.116:0/3063769931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 msgr2=0x55b5f8e1c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e1c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 -- 172.21.15.116:0/3063769931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f8a8b2d0 con 0x55b5f8e129e0 2024-03-19T02:27:27.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f73053db640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e1c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7305bdc640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 0x55b5f8e23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f7304bda640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f72f402c180 tx=0x7f72f4005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd9d1a98343c97f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f72e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4039040 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f72f402c510 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72f00268f0 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f72f404e050 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f730a220740 10 client.15486 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5f8b556b0 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.120+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f00268f0 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f72f4003cc0 con 0x55b5f8e129e0 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 10 client.15486 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 10 client.15486 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:27:27.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 20 client.15486 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e29380 0x55b5f8e2a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 4 client.15486 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5f8d84320 con 0x55b5f8e29380 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.121+0000 7f73053db640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e29380 0x55b5f8e2a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f73053db640 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e29380 0x55b5f8e2a7f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=943023a7782b6877 server_cookie=e1eadd56fd2b3a05 in_seq=0 out_seq=0 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f72e4ff9640 10 client.15486 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f72e4ff9640 1 -- 172.21.15.116:0/3063769931 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5f8d84320 con 0x55b5f8e29380 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f72e4ff9640 10 client.15486 handle_command_reply: tid=0 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 client.15486 shutdown 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 msgr2=0x55b5f8e1ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f72f402c180 tx=0x7f72f4005b20 comp rx=0 tx=0).stop 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 shutdown_connections 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e29380 0x55b5f8e2a7f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e17b50 0x55b5f8e23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e129e0 0x55b5f8e1ca20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 --2- 172.21.15.116:0/3063769931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e1c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 >> 172.21.15.116:0/3063769931 conn(0x55b5f8e0d560 msgr2=0x55b5f8e0c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.122+0000 7f72e4ff9640 0 client.15486 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.123+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 shutdown_connections 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.123+0000 7f730a220740 1 -- 172.21.15.116:0/3063769931 wait complete. 2024-03-19T02:27:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.123+0000 7f730a220740 20 client.15486 trim_cache size 0 max 16384 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 Processor -- start 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 -- start start 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e129e0 0x55b5f8e23a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e17b50 0x55b5f8e1b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e28b30 con 0x55b5f8e11ff0 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e29200 con 0x55b5f8e17b50 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f730a220740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e298d0 con 0x55b5f8e129e0 2024-03-19T02:27:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45272/0 (socket says 172.21.15.116:45272) 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 -- 172.21.15.116:0/3610062848 learned_addr learned my addr 172.21.15.116:0/3610062848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f7304bda640 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e129e0 0x55b5f8e23a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f7305bdc640 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e17b50 0x55b5f8e1b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 -- 172.21.15.116:0/3610062848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e129e0 msgr2=0x55b5f8e23a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e129e0 0x55b5f8e23a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 -- 172.21.15.116:0/3610062848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e17b50 msgr2=0x55b5f8e1b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e17b50 0x55b5f8e1b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 -- 172.21.15.116:0/3610062848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f8e11c30 con 0x55b5f8e11ff0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f73053db640 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f72f800cfd0 tx=0x7f72f80980d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74180a3ddc326f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f72e6ffd640 1 -- 172.21.15.116:0/3610062848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f809c400 con 0x55b5f8e11ff0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f72e6ffd640 1 -- 172.21.15.116:0/3610062848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f80b2030 con 0x55b5f8e11ff0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.141+0000 7f72e6ffd640 1 -- 172.21.15.116:0/3610062848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8099e30 con 0x55b5f8e11ff0 2024-03-19T02:27:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- 172.21.15.116:0/3610062848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 msgr2=0x55b5f8e0d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f72f800cfd0 tx=0x7f72f80980d0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- 172.21.15.116:0/3610062848 shutdown_connections 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e17b50 0x55b5f8e1b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e129e0 0x55b5f8e23a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 --2- 172.21.15.116:0/3610062848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e11ff0 0x55b5f8e0d940 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- 172.21.15.116:0/3610062848 >> 172.21.15.116:0/3610062848 conn(0x55b5f8e0d560 msgr2=0x55b5f8e13400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- 172.21.15.116:0/3610062848 shutdown_connections 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- 172.21.15.116:0/3610062848 wait complete. 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 -- start start 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.142+0000 7f730a220740 1 Processor -- start 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e15f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e22010 0x55b5f8e22490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e2a670 con 0x55b5f8e17b50 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e29fa0 con 0x55b5f8e129e0 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8e2ad40 con 0x55b5f8e22010 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 10 client.0 _ll_register_callbacks cb 0x7f72f80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45280/0 (socket says 172.21.15.116:45280) 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 -- 172.21.15.116:0/1924464511 learned_addr learned my addr 172.21.15.116:0/1924464511 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f73053db640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e15f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7305bdc640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e22010 0x55b5f8e22490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 -- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e22010 msgr2=0x55b5f8e22490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e22010 0x55b5f8e22490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 -- 172.21.15.116:0/1924464511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 msgr2=0x55b5f8e15f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e15f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 -- 172.21.15.116:0/1924464511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f8e229d0 con 0x55b5f8e17b50 2024-03-19T02:27:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f7304bda640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f72f4005770 tx=0x7f72f4003870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447c73bc7b3436d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f72e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4056070 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f72f402c950 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.143+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72f80b2030 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f72f4039050 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 10 client.15492 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5f8e23d70 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f80b2030 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f72f403f450 con 0x55b5f8e17b50 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 10 client.15492 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 10 client.15492 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 20 client.15492 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e93b30 0x55b5f8e92840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 4 client.15492 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "status"}) -- 0x55b5f8d7e670 con 0x55b5f8e93b30 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.144+0000 7f73053db640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e93b30 0x55b5f8e92840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f72e4ff9640 10 client.15492 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f73053db640 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e93b30 0x55b5f8e92840 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2db8d373cba22dbb server_cookie=64a39e47514bc4ff in_seq=0 out_seq=0 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f72e4ff9640 1 -- 172.21.15.116:0/1924464511 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b5f8d7e670 con 0x55b5f8e93b30 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f72e4ff9640 10 client.15492 handle_command_reply: tid=0 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 client.15492 shutdown 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 msgr2=0x55b5f8e16470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f72f4005770 tx=0x7f72f4003870 comp rx=0 tx=0).stop 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 shutdown_connections 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b5f8e93b30 0x55b5f8e92840 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5f8e22010 0x55b5f8e22490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f8e17b50 0x55b5f8e16470 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 --2- 172.21.15.116:0/1924464511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5f8e129e0 0x55b5f8e15f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 >> 172.21.15.116:0/1924464511 conn(0x55b5f8e0d560 msgr2=0x55b5f8e0d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f72e4ff9640 0 client.15492 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 shutdown_connections 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 1 -- 172.21.15.116:0/1924464511 wait complete. 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730a220740 20 client.15492 trim_cache size 0 max 16384 2024-03-19T02:27:27.152 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24484, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 114.458095552, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 94, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 151.32657592000001 2024-03-19T02:27:27.153 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 msgr2=0x7f72e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.145+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 0x7f72e80a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72f0023000 tx=0x7f72f0024000 comp rx=0 tx=0).stop 2024-03-19T02:27:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 msgr2=0x7f73000755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f72f800ebc0 tx=0x7f72f800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 shutdown_connections 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73001033e0 0x7f7300074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7300102a20 0x7f7300073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 --2- 172.21.15.116:0/2026072682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7300101830 0x7f73000755d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 >> 172.21.15.116:0/2026072682 conn(0x7f73000fcdc0 msgr2=0x7f7300105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 shutdown_connections 2024-03-19T02:27:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.146+0000 7f730763f640 1 -- 172.21.15.116:0/2026072682 wait complete. 2024-03-19T02:27:27.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:27:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:26 smithi162 ceph-mon[35761]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:26 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4054085588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:26 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3692414565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:27.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:26 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2448006793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:26 smithi154 ceph-mon[35693]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T02:27:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:26 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4054085588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:26 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3692414565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:26 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2448006793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 -- 172.21.15.116:0/3700142971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 msgr2=0x7f9784101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3700142971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784101db0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9774009900 tx=0x7f977402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 -- 172.21.15.116:0/3700142971 shutdown_connections 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3700142971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f9784108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3700142971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784101db0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3700142971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 -- 172.21.15.116:0/3700142971 >> 172.21.15.116:0/3700142971 conn(0x7f97840fbcd0 msgr2=0x7f97840fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 -- 172.21.15.116:0/3700142971 shutdown_connections 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.259+0000 7f978b5cd640 1 -- 172.21.15.116:0/3700142971 wait complete. 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 Processor -- start 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 -- start start 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f978419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f978410d390 con 0x7f9784100740 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f978410d210 con 0x7f9784101930 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f978b5cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f978410d510 con 0x7f97841022f0 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45300/0 (socket says 172.21.15.116:45300) 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789369640 1 -- 172.21.15.116:0/3180219770 learned_addr learned my addr 172.21.15.116:0/3180219770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789b6a640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f978419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 -- 172.21.15.116:0/3180219770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 msgr2=0x7f978419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f978419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 -- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 msgr2=0x7f9784197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9788b68640 1 -- 172.21.15.116:0/3180219770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978419f690 con 0x7f9784101930 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789369640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.260+0000 7f9789b6a640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f978419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f9788b68640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784197fb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9774000c00 tx=0x7f9774005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f977a7fc640 1 -- 172.21.15.116:0/3180219770 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774043020 con 0x7f9784101930 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f977a7fc640 1 -- 172.21.15.116:0/3180219770 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9774004350 con 0x7f9784101930 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f977a7fc640 1 -- 172.21.15.116:0/3180219770 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774003b50 con 0x7f9784101930 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978419f920 con 0x7f9784101930 2024-03-19T02:27:27.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.261+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978419fe70 con 0x7f9784101930 2024-03-19T02:27:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f977a7fc640 1 -- 172.21.15.116:0/3180219770 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9774046020 con 0x7f9784101930 2024-03-19T02:27:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 Processor -- start 2024-03-19T02:27:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 -- start start 2024-03-19T02:27:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659ea5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eaa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c659e09de0 con 0x55c659eaab50 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.263+0000 7f978e1b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c659db2c80 con 0x55c659ea59e0 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f978e1b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c659e05e80 con 0x55c659ea4ff0 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9789369640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eaa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45302/0 (socket says 172.21.15.116:45302) 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 -- 172.21.15.116:0/3289134631 learned_addr learned my addr 172.21.15.116:0/3289134631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9789b6a640 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659ea5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f977a7fc640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 0x7f97700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 -- 172.21.15.116:0/3289134631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 msgr2=0x55c659ea5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659ea5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 -- 172.21.15.116:0/3289134631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 msgr2=0x55c659eaa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eaa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 -- 172.21.15.116:0/3289134631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c659dc4fb0 con 0x55c659eaab50 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9789369640 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eaa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9789b6a640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 0x7f97700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f977a7fc640 1 -- 172.21.15.116:0/3180219770 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f97740f0e40 con 0x7f9784101930 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9788b68640 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97740ebdf0 tx=0x7f97740e7eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745a4c34571aa445 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f976f7fe640 1 -- 172.21.15.116:0/3289134631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97740bcfa0 con 0x55c659eaab50 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f9789b6a640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 0x7f97700a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55c659ea6240 tx=0x7f977c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f976f7fe640 1 -- 172.21.15.116:0/3289134631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97740e8b40 con 0x55c659eaab50 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.264+0000 7f976f7fe640 1 -- 172.21.15.116:0/3289134631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97740c15b0 con 0x55c659eaab50 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- 172.21.15.116:0/3289134631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 msgr2=0x55c659eacfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97740ebdf0 tx=0x7f97740e7eb0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- 172.21.15.116:0/3289134631 shutdown_connections 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eacfe0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eaa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 --2- 172.21.15.116:0/3289134631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659ea5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- 172.21.15.116:0/3289134631 >> 172.21.15.116:0/3289134631 conn(0x55c659ea0560 msgr2=0x55c659ea0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- 172.21.15.116:0/3289134631 shutdown_connections 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- 172.21.15.116:0/3289134631 wait complete. 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.265+0000 7f978e1b8740 1 -- start start 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 Processor -- start 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eaf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659eaab50 0x55c659eb6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c659ead8c0 con 0x55c659ea59e0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c659dc5fc0 con 0x55c659eaab50 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c659eada40 con 0x55c659ea4ff0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f978e1b8740 10 client.0 _ll_register_callbacks cb 0x55c659e01a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45306/0 (socket says 172.21.15.116:45306) 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 -- 172.21.15.116:0/666491890 learned_addr learned my addr 172.21.15.116:0/666491890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9789b6a640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659eaab50 0x55c659eb6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9789369640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eaf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 -- 172.21.15.116:0/666491890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 msgr2=0x55c659eaf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eaf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 -- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659eaab50 msgr2=0x55c659eb6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659eaab50 0x55c659eb6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 -- 172.21.15.116:0/666491890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c659b1e2d0 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f9788b68640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97740ee610 tx=0x7f97740b90b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54756b667fe7dc5c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.266+0000 7f976cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97740c1810 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f97740054d0 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c659e17320 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97740bcdc0 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 10 client.15504 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c659be86b0 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c659e17320 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f97740f2020 con 0x55c659ea59e0 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 10 client.15504 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 10 client.15504 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:27:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 20 client.15504 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659ebbe20 0x55c659ebd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 4 client.15504 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c659e11670 con 0x55c659ebbe20 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.267+0000 7f9789369640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659ebbe20 0x55c659ebd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.268+0000 7f976cff9640 10 client.15504 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.268+0000 7f9789369640 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659ebbe20 0x55c659ebd290 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e36f3c13ac43b6 server_cookie=7cff3e3bbea279ce in_seq=0 out_seq=0 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f976cff9640 1 -- 172.21.15.116:0/666491890 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c659e11670 con 0x55c659ebbe20 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f976cff9640 10 client.15504 handle_command_reply: tid=0 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 client.15504 shutdown 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 msgr2=0x55c659eaf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97740ee610 tx=0x7f97740b90b0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 shutdown_connections 2024-03-19T02:27:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659ebbe20 0x55c659ebd290 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659eaab50 0x55c659eb6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea59e0 0x55c659eaf740 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f976cff9640 0 client.15504 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 --2- 172.21.15.116:0/666491890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eaf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 >> 172.21.15.116:0/666491890 conn(0x55c659ea0560 msgr2=0x55c659e9f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 shutdown_connections 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 1 -- 172.21.15.116:0/666491890 wait complete. 2024-03-19T02:27:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.269+0000 7f978e1b8740 20 client.15504 trim_cache size 0 max 16384 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 Processor -- start 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 -- start start 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659eaab50 0x55c659eaf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebb5d0 con 0x55c659ea4ff0 2024-03-19T02:27:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebbca0 con 0x55c659ea59e0 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f978e1b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebc370 con 0x55c659eaab50 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45314/0 (socket says 172.21.15.116:45314) 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 -- 172.21.15.116:0/2135864455 learned_addr learned my addr 172.21.15.116:0/2135864455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9788b68640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789b6a640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659eaab50 0x55c659eaf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 -- 172.21.15.116:0/2135864455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659eaab50 msgr2=0x55c659eaf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659eaab50 0x55c659eaf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 -- 172.21.15.116:0/2135864455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 msgr2=0x55c659eb6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 -- 172.21.15.116:0/2135864455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c659ea6270 con 0x55c659ea4ff0 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9788b68640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.287+0000 7f9789369640 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9780012620 tx=0x7f9780005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd61ece6fa8c12ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f976effd640 1 -- 172.21.15.116:0/2135864455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780024030 con 0x55c659ea4ff0 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f976effd640 1 -- 172.21.15.116:0/2135864455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97800054d0 con 0x55c659ea4ff0 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f976effd640 1 -- 172.21.15.116:0/2135864455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97800202f0 con 0x55c659ea4ff0 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- 172.21.15.116:0/2135864455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 msgr2=0x55c659eaee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9780012620 tx=0x7f9780005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- 172.21.15.116:0/2135864455 shutdown_connections 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659eaab50 0x55c659eaf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 --2- 172.21.15.116:0/2135864455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659ea4ff0 0x55c659eaee10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- 172.21.15.116:0/2135864455 >> 172.21.15.116:0/2135864455 conn(0x55c659ea0560 msgr2=0x55c659ea6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- 172.21.15.116:0/2135864455 shutdown_connections 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- 172.21.15.116:0/2135864455 wait complete. 2024-03-19T02:27:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.288+0000 7f978e1b8740 1 -- start start 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 Processor -- start 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebd110 con 0x55c659eaab50 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebca40 con 0x55c659ea59e0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c659ebd7e0 con 0x55c659ea4ff0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f978e1b8740 10 client.0 _ll_register_callbacks cb 0x7f9780009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789369640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45320/0 (socket says 172.21.15.116:45320) 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789369640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34414/0 (socket says 172.21.15.116:34414) 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9788b68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 -- 172.21.15.116:0/850310750 learned_addr learned my addr 172.21.15.116:0/850310750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 -- 172.21.15.116:0/850310750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 msgr2=0x55c659eb29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 -- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 msgr2=0x55c659eb2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789b6a640 1 -- 172.21.15.116:0/850310750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c659eb64f0 con 0x55c659eaab50 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9789369640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.289+0000 7f9788b68640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f9789b6a640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f977c011ed0 tx=0x7f977c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91af4ca550e73952 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c02f070 con 0x55c659eaab50 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f977c004b20 con 0x55c659eaab50 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97800054d0 con 0x55c659eaab50 2024-03-19T02:27:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f977c03a050 con 0x55c659eaab50 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 10 client.15510 fetch_fsmap learned FSMap version 27 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c659f22780 con 0x55c659eaab50 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97800054d0 con 0x55c659eaab50 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f977c0050b0 con 0x55c659eaab50 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 10 client.15510 fetch_fsmap finished waiting for FSMap version 27 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 10 client.15510 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 20 client.15510 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659f27440 0x55c659f26150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 4 client.15510 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "status"}) -- 0x55c659d8fa00 con 0x55c659f27440 2024-03-19T02:27:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.290+0000 7f9789369640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659f27440 0x55c659f26150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f9789369640 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659f27440 0x55c659f26150 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c27bd59196b32b server_cookie=3f4e0d5e3d67fd0d in_seq=0 out_seq=0 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f976cff9640 10 client.15510 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f976cff9640 1 -- 172.21.15.116:0/850310750 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c659d8fa00 con 0x55c659f27440 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f976cff9640 10 client.15510 handle_command_reply: tid=0 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 client.15510 shutdown 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 msgr2=0x55c659eb5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f977c011ed0 tx=0x7f977c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 shutdown_connections 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c659f27440 0x55c659f26150 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c659eaab50 0x55c659eb5f00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f976cff9640 0 client.15510 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c659ea59e0 0x55c659eb2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 --2- 172.21.15.116:0/850310750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c659ea4ff0 0x55c659eb29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 >> 172.21.15.116:0/850310750 conn(0x55c659ea0560 msgr2=0x55c659eaeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 shutdown_connections 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 1 -- 172.21.15.116:0/850310750 wait complete. 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.291+0000 7f978e1b8740 20 client.15510 trim_cache size 0 max 16384 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a1da9bbc-e596-11ee-95c9-87774f69a715", 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24494, 2024-03-19T02:27:27.299 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 138.64020153199999, 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 27, 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 94, 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 90, 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 142.88474584900001 2024-03-19T02:27:27.300 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 msgr2=0x7f97700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 0x7f97700a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55c659ea6240 tx=0x7f977c004a40 comp rx=0 tx=0).stop 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 msgr2=0x7f9784197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784197fb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9774000c00 tx=0x7f9774005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 shutdown_connections 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f97700a2800 0x7f97700a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97841022f0 0x7f978419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784101930 0x7f9784197fb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 --2- 172.21.15.116:0/3180219770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9784100740 0x7f9784197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 >> 172.21.15.116:0/3180219770 conn(0x7f97840fbcd0 msgr2=0x7f9784104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 shutdown_connections 2024-03-19T02:27:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:27.292+0000 7f978b5cd640 1 -- 172.21.15.116:0/3180219770 wait complete. 2024-03-19T02:27:27.303 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-19T02:27:27.314 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-19T02:27:27.315 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-19T02:27:27.315 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-19T02:27:27.315 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T02:27:27.315 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:27.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:27.315 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T02:27:27.331 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-19T02:27:27.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:27.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:27:27.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:27.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-19T02:27:27.498 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-19T02:27:27.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T02:27:27.742 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:27:28.890 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:28 smithi116 ceph-mon[32263]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:29.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:28 smithi162 ceph-mon[35761]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:28 smithi154 ceph-mon[35693]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 -- 172.21.15.116:0/786797546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524108340 msgr2=0x7f2524108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 --2- 172.21.15.116:0/786797546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524108340 0x7f2524108720 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f25200099c0 tx=0x7f2520009cd0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 -- 172.21.15.116:0/786797546 shutdown_connections 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 --2- 172.21.15.116:0/786797546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 --2- 172.21.15.116:0/786797546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524108340 0x7f2524108720 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 --2- 172.21.15.116:0/786797546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524102360 0x7f25241027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 -- 172.21.15.116:0/786797546 >> 172.21.15.116:0/786797546 conn(0x7f2524079440 msgr2=0x7f2524079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.434+0000 7f252c7ee700 1 -- 172.21.15.116:0/786797546 shutdown_connections 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.435+0000 7f252c7ee700 1 -- 172.21.15.116:0/786797546 wait complete. 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.435+0000 7f252c7ee700 1 Processor -- start 2024-03-19T02:27:29.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.435+0000 7f252c7ee700 1 -- start start 2024-03-19T02:27:29.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:29.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:29.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524108340 0x7f2524198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2524112470 con 0x7f2524102360 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25241122f0 con 0x7f2524108340 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252c7ee700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25241125f0 con 0x7f2524102d20 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36304/0 (socket says 172.21.15.116:36304) 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 -- 172.21.15.116:0/1581365810 learned_addr learned my addr 172.21.15.116:0/1581365810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f2529d89700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252ad8b700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524108340 0x7f2524198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 -- 172.21.15.116:0/1581365810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 msgr2=0x7f252419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 -- 172.21.15.116:0/1581365810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524108340 msgr2=0x7f2524198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524108340 0x7f2524198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 -- 172.21.15.116:0/1581365810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2524199070 con 0x7f2524102360 2024-03-19T02:27:29.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f2529d89700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.436+0000 7f252a58a700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2520006010 tx=0x7f2520004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.437+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520026070 con 0x7f2524102360 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.437+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2524199300 con 0x7f2524102360 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.437+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2520004630 con 0x7f2524102360 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.437+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252000fb20 con 0x7f2524102360 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.437+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25241a55f0 con 0x7f2524102360 2024-03-19T02:27:29.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.438+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25241039f0 con 0x7f2524102360 2024-03-19T02:27:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.439+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2520029070 con 0x7f2524102360 2024-03-19T02:27:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.439+0000 7f251b7fe700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 0x7f25100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.440+0000 7f2529d89700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 0x7f25100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.440+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f25200cf930 con 0x7f2524102360 2024-03-19T02:27:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.440+0000 7f2529d89700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 0x7f25100a4dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2524199f00 tx=0x7f2514009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.443+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2520030080 con 0x7f2524102360 2024-03-19T02:27:29.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.684+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2524199c70 con 0x7f2524102360 2024-03-19T02:27:29.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.685+0000 7f251b7fe700 1 -- 172.21.15.116:0/1581365810 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f2520013070 con 0x7f2524102360 2024-03-19T02:27:29.685 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T02:27:29.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 msgr2=0x7f25100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 0x7f25100a4dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2524199f00 tx=0x7f2514009260 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 msgr2=0x7f252419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2520006010 tx=0x7f2520004270 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 shutdown_connections 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2524108340 0x7f2524198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2524102360 0x7f252419e670 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f25100a2910 0x7f25100a4dd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 --2- 172.21.15.116:0/1581365810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2524102d20 0x7f252419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 >> 172.21.15.116:0/1581365810 conn(0x7f2524079440 msgr2=0x7f25241010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 shutdown_connections 2024-03-19T02:27:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:29.687+0000 7f252c7ee700 1 -- 172.21.15.116:0/1581365810 wait complete. 2024-03-19T02:27:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:29 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1581365810' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:27:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:29 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1581365810' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:27:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:29 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1581365810' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:30.400 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T02:27:30.401 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T02:27:30.401 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T02:27:30.401 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:27:30.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:30.401 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:27:30.417 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 40944sec preferred_lft 40944sec 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-19T02:27:30.418 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:27:30.418 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T02:27:30.419 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:27:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:27:30.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:30.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:30.571 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T02:27:30.597 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-19T02:27:30.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:30.599 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-19T02:27:30.655 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-19T02:27:30.655 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T02:27:30.655 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T02:27:30.655 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T02:27:30.655 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T02:27:30.656 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:27:30.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:27:30.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:30.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:30.796 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T02:27:30.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:30.810 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-19T02:27:30.865 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:27:30.866 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:27:30.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:30.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T02:27:30.866 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:27:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:27:30.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:30 smithi116 ceph-mon[32263]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:30.989 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T02:27:30.989 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:27:31.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:27:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:31.116 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:27:31.116 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T02:27:31.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-19T02:27:31.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T02:27:31.116 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T02:27:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:27:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:30 smithi162 ceph-mon[35761]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:31.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:31.198 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-19T02:27:31.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:27:31.198 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:31.213 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:27:31.213 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:27:31.213 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:30 smithi154 ceph-mon[35693]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-19T02:27:31.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-19T02:27:31.338 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:27:31.393 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-19T02:27:31.393 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-19T02:27:31.394 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/105c939f0328de1668168813dcbfde3e8eac64353870a865edc6e64881aafc43/merged 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/1ca6329da29ba9cc1ded52d1fda3fa7f4b282c5d6c2cc11a5dbe3497e0b413e8/merged 2024-03-19T02:27:31.395 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/73a69ff94c6f368cf169b44a6d3a6ad7a62ffe99dd12cd8959b91ac1a9f9a7bf/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/776d3185faea4a5790e60d649621a6c756987875b5f3e54f8f1a6d47dbb4ae2c/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/101d9845f1265f94cc82f49e12f8fa48bbddbce39c7a3fc6d104394b56d90024/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/8a251936db2ed6b341d5b7f1c962f8d2f4cdee7a8f13f657a58c0aa0db2c59fa/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6ce183691416a01ad903095c2847c36f5ff96371767d62f353ea226d8b796e7a/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/69b48faf2550c0d2579eb5fdd431271f2fd54aa710d23c4fb41af23559ce35f1/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/52d53c064a0f81dee0cede53b46458f5c9011d803f5831b430af148b5061e918/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d9e070467a714fd4702b0b56c587243707ab26dc972ca1ea99c3ce18b3d975f5/merged 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:31.396 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:31.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:31.397 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-19T02:27:31.448 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T02:27:31.448 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/479ec2b2-297d-461d-8f26-3eef92c9c7c9 --client_fs=cephfs) 2024-03-19T02:27:31.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-19T02:27:31.517 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-19T02:27:31.573 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:27:31.574 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/105c939f0328de1668168813dcbfde3e8eac64353870a865edc6e64881aafc43/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/1ca6329da29ba9cc1ded52d1fda3fa7f4b282c5d6c2cc11a5dbe3497e0b413e8/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/73a69ff94c6f368cf169b44a6d3a6ad7a62ffe99dd12cd8959b91ac1a9f9a7bf/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/776d3185faea4a5790e60d649621a6c756987875b5f3e54f8f1a6d47dbb4ae2c/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/101d9845f1265f94cc82f49e12f8fa48bbddbce39c7a3fc6d104394b56d90024/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/8a251936db2ed6b341d5b7f1c962f8d2f4cdee7a8f13f657a58c0aa0db2c59fa/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6ce183691416a01ad903095c2847c36f5ff96371767d62f353ea226d8b796e7a/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/69b48faf2550c0d2579eb5fdd431271f2fd54aa710d23c4fb41af23559ce35f1/merged 2024-03-19T02:27:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/52d53c064a0f81dee0cede53b46458f5c9011d803f5831b430af148b5061e918/merged 2024-03-19T02:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d9e070467a714fd4702b0b56c587243707ab26dc972ca1ea99c3ce18b3d975f5/merged 2024-03-19T02:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-19T02:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:31.576 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:31.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:31.576 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-19T02:27:31.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2024-03-19T02:27:31.576+0000 7f760f842480 -1 init, newargv = 0x5617d36c0290 newargc=15 2024-03-19T02:27:31.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[84163]: starting ceph client 2024-03-19T02:27:31.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[84163]: starting fuse 2024-03-19T02:27:32.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-19T02:27:32.727 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-19T02:27:32.786 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:27:32.787 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/105c939f0328de1668168813dcbfde3e8eac64353870a865edc6e64881aafc43/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/1ca6329da29ba9cc1ded52d1fda3fa7f4b282c5d6c2cc11a5dbe3497e0b413e8/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/73a69ff94c6f368cf169b44a6d3a6ad7a62ffe99dd12cd8959b91ac1a9f9a7bf/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/776d3185faea4a5790e60d649621a6c756987875b5f3e54f8f1a6d47dbb4ae2c/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/101d9845f1265f94cc82f49e12f8fa48bbddbce39c7a3fc6d104394b56d90024/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/8a251936db2ed6b341d5b7f1c962f8d2f4cdee7a8f13f657a58c0aa0db2c59fa/merged 2024-03-19T02:27:32.788 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6ce183691416a01ad903095c2847c36f5ff96371767d62f353ea226d8b796e7a/merged 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/69b48faf2550c0d2579eb5fdd431271f2fd54aa710d23c4fb41af23559ce35f1/merged 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/52d53c064a0f81dee0cede53b46458f5c9011d803f5831b430af148b5061e918/merged 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d9e070467a714fd4702b0b56c587243707ab26dc972ca1ea99c3ce18b3d975f5/merged 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:32.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:32.789 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-19T02:27:32.843 INFO:teuthology.orchestra.run.smithi116.stdout:119 2024-03-19T02:27:32.843 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-19T02:27:32.844 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-19T02:27:32.845 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T02:27:32.846 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-19T02:27:32.846 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2024-03-19T02:27:32.846 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-19T02:27:32.969 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.84163.asok 2024-03-19T02:27:32.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:27:32.979 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:32 smithi116 ceph-mon[32263]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:32.980 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84163.asok 2024-03-19T02:27:32.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:32.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84163.asok status 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2024-03-19T02:27:33.117 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "84163", 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/volumes/qa/sv_0/479ec2b2-297d-461d-8f26-3eef92c9c7c9" 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 15522, 2024-03-19T02:27:33.118 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 15522 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1712485974 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:27:33.119 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1712485974 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.15522 192.168.0.1:0/1712485974", 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1712485974", 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 5, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 27, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 94, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-19T02:27:33.120 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:33.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:33.126 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-19T02:27:33.126 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:33.141 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-19T02:27:33.141 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:33.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:33.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:32 smithi162 ceph-mon[35761]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:33.210 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T02:27:33.220 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:27:33.220 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:27:33.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-19T02:27:33.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:32 smithi154 ceph-mon[35693]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/747395293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4102310 msgr2=0x7fa1b4108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/747395293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4102310 0x7fa1b4108ba0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009ca0 tx=0x7fa1ac02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/747395293 shutdown_connections 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/747395293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4102310 0x7fa1b4108ba0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/747395293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4101950 0x7fa1b4101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/747395293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 0x7fa1b4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/747395293 >> 172.21.15.116:0/747395293 conn(0x7fa1b40fbcd0 msgr2=0x7fa1b40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/747395293 shutdown_connections 2024-03-19T02:27:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.352+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/747395293 wait complete. 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 Processor -- start 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 -- start start 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 0x7fa1b406ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4102310 0x7fa1b419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410c290 con 0x7fa1b4101950 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410c110 con 0x7fa1b4102310 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b9ba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410c410 con 0x7fa1b4100760 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36318/0 (socket says 172.21.15.116:36318) 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 -- 172.21.15.116:0/1528071283 learned_addr learned my addr 172.21.15.116:0/1528071283 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 0x7fa1b406ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b3fff640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4102310 0x7fa1b419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 -- 172.21.15.116:0/1528071283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 msgr2=0x7fa1b406ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 0x7fa1b406ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 -- 172.21.15.116:0/1528071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4102310 msgr2=0x7fa1b419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4102310 0x7fa1b419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.353+0000 7fa1b2ffd640 1 -- 172.21.15.116:0/1528071283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b419f5d0 con 0x7fa1b4101950 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b2ffd640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800e9f0 tx=0x7fa1a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a800cd80 con 0x7fa1b4101950 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b419f860 con 0x7fa1b4101950 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1a80108e0 con 0x7fa1b4101950 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8018920 con 0x7fa1b4101950 2024-03-19T02:27:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.354+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b419fdb0 con 0x7fa1b4101950 2024-03-19T02:27:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.355+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4107a20 con 0x7fa1b4101950 2024-03-19T02:27:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.356+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa1a8010a90 con 0x7fa1b4101950 2024-03-19T02:27:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.356+0000 7fa1b0ff9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 0x7fa19c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.357+0000 7fa1b37fe640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 0x7fa19c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.357+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fa1a80cf0f0 con 0x7fa1b4101950 2024-03-19T02:27:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.357+0000 7fa1b37fe640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 0x7fa19c0a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1b41017b0 tx=0x7fa1a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.361+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1a809b7c0 con 0x7fa1b4101950 2024-03-19T02:27:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.576+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa1b4062420 con 0x7fa1b4101950 2024-03-19T02:27:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.775+0000 7fa1b0ff9640 1 -- 172.21.15.116:0/1528071283 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7fa1a809f860 con 0x7fa1b4101950 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 msgr2=0x7fa19c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 0x7fa19c0a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1b41017b0 tx=0x7fa1a4005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 msgr2=0x7fa1b419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800e9f0 tx=0x7fa1a800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 shutdown_connections 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa19c0a28a0 0x7fa19c0a4d60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1b4102310 0x7fa1b419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b4101950 0x7fa1b419c6a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 --2- 172.21.15.116:0/1528071283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1b4100760 0x7fa1b406ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 >> 172.21.15.116:0/1528071283 conn(0x7fa1b40fbcd0 msgr2=0x7fa1b4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 shutdown_connections 2024-03-19T02:27:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.777+0000 7fa1b9ba9640 1 -- 172.21.15.116:0/1528071283 wait complete. 2024-03-19T02:27:33.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 -- 172.21.15.116:0/1648925339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4101950 msgr2=0x7effe4101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 --2- 172.21.15.116:0/1648925339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4101950 0x7effe4101dd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7effd4009900 tx=0x7effd402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 -- 172.21.15.116:0/1648925339 shutdown_connections 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 --2- 172.21.15.116:0/1648925339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 0x7effe4108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 --2- 172.21.15.116:0/1648925339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4101950 0x7effe4101dd0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 --2- 172.21.15.116:0/1648925339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4100760 0x7effe4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 -- 172.21.15.116:0/1648925339 >> 172.21.15.116:0/1648925339 conn(0x7effe40fbcd0 msgr2=0x7effe40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 -- 172.21.15.116:0/1648925339 shutdown_connections 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.892+0000 7effeb64b640 1 -- 172.21.15.116:0/1648925339 wait complete. 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 Processor -- start 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 -- start start 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 0x7effe419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 0x7effe419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7effe410c290 con 0x7effe4100760 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7effe410c110 con 0x7effe4101950 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effeb64b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7effe410c410 con 0x7effe4102310 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36338/0 (socket says 172.21.15.116:36338) 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe8be6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 0x7effe419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 -- 172.21.15.116:0/3257770192 learned_addr learned my addr 172.21.15.116:0/3257770192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:33.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe9be8640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 0x7effe419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 -- 172.21.15.116:0/3257770192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 msgr2=0x7effe419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 0x7effe419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 -- 172.21.15.116:0/3257770192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 msgr2=0x7effe419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 0x7effe419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 -- 172.21.15.116:0/3257770192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe419f620 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe8be6640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 0x7effe419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.893+0000 7effe93e7640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7effe000e9c0 tx=0x7effe000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.894+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe000cd80 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.894+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe419f910 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.894+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe00035f0 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.894+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0010690 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.894+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe419fe30 con 0x7effe4100760 2024-03-19T02:27:33.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.895+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe4107a20 con 0x7effe4100760 2024-03-19T02:27:33.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.896+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7effe00037a0 con 0x7effe4100760 2024-03-19T02:27:33.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.896+0000 7effda7fc640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 0x7effd00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:33.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.896+0000 7effe8be6640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 0x7effd00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:33.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.896+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7effe00cf170 con 0x7effe4100760 2024-03-19T02:27:33.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.897+0000 7effe8be6640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 0x7effd00a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7effd4006010 tx=0x7effd402c740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:33.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:33.901+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effe009be90 con 0x7effe4100760 2024-03-19T02:27:34.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.102+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7effe4061c60 con 0x7effd00a28a0 2024-03-19T02:27:34.108 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:33 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T02:27:34.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.117+0000 7effda7fc640 1 -- 172.21.15.116:0/3257770192 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7effe4061c60 con 0x7effd00a28a0 2024-03-19T02:27:34.117 INFO:teuthology.orchestra.run.smithi116.stdout:{} 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 msgr2=0x7effd00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 0x7effd00a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7effd4006010 tx=0x7effd402c740 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 msgr2=0x7effe406aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7effe000e9c0 tx=0x7effe000ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 shutdown_connections 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7effd00a28a0 0x7effd00a4d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7effe4102310 0x7effe419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe4101950 0x7effe419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 --2- 172.21.15.116:0/3257770192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effe4100760 0x7effe406aaf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 >> 172.21.15.116:0/3257770192 conn(0x7effe40fbcd0 msgr2=0x7effe4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 shutdown_connections 2024-03-19T02:27:34.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.119+0000 7effeb64b640 1 -- 172.21.15.116:0/3257770192 wait complete. 2024-03-19T02:27:34.130 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-19T02:27:34.142 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-19T02:27:34.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:33 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 -- 172.21.15.116:0/3814548081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 msgr2=0x7fa8ec0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3814548081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec0fff20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009930 tx=0x7fa8e402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 -- 172.21.15.116:0/3814548081 shutdown_connections 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3814548081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec103490 0x7fa8ec109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3814548081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec102ad0 0x7fa8ec102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3814548081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec0fff20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 -- 172.21.15.116:0/3814548081 >> 172.21.15.116:0/3814548081 conn(0x7fa8ec0fb450 msgr2=0x7fa8ec0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 -- 172.21.15.116:0/3814548081 shutdown_connections 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.255+0000 7fa8f083d640 1 -- 172.21.15.116:0/3814548081 wait complete. 2024-03-19T02:27:34.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 Processor -- start 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 -- start start 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec102ad0 0x7fa8ec198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec103490 0x7fa8ec19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ec10d380 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ec10d200 con 0x7fa8ec102ad0 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8f083d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ec10d500 con 0x7fa8ec103490 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36366/0 (socket says 172.21.15.116:36366) 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 -- 172.21.15.116:0/3233325988 learned_addr learned my addr 172.21.15.116:0/3233325988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8e9d9b640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec102ad0 0x7fa8ec198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ead9d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec103490 0x7fa8ec19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 -- 172.21.15.116:0/3233325988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec103490 msgr2=0x7fa8ec19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec103490 0x7fa8ec19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 -- 172.21.15.116:0/3233325988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec102ad0 msgr2=0x7fa8ec198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec102ad0 0x7fa8ec198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 -- 172.21.15.116:0/3233325988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ec19f6d0 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8ea59c640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4000c00 tx=0x7fa8e4005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.256+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4043020 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.257+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e40043d0 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.257+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8ec19f960 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.257+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4003b50 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.257+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8ec19feb0 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:33 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.258+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8ec108dc0 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.259+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa8e4046020 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.259+0000 7fa8db7fe640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 0x7fa8d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.259+0000 7fa8e9d9b640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 0x7fa8d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.259+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fa8e40f0f10 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.260+0000 7fa8e9d9b640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 0x7fa8d40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec198fc0 tx=0x7fa8e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.264+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8e404d020 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.440+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8ec062420 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.442+0000 7fa8db7fe640 1 -- 172.21.15.116:0/3233325988 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7fa8e40bdc30 con 0x7fa8ec0ffb00 2024-03-19T02:27:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:34.443 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 msgr2=0x7fa8d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 0x7fa8d40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec198fc0 tx=0x7fa8e0009500 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 msgr2=0x7fa8ec197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4000c00 tx=0x7fa8e4005e70 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 shutdown_connections 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fa8d40a2800 0x7fa8d40a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8ec103490 0x7fa8ec19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8ec102ad0 0x7fa8ec198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 --2- 172.21.15.116:0/3233325988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8ec0ffb00 0x7fa8ec197b00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 >> 172.21.15.116:0/3233325988 conn(0x7fa8ec0fb450 msgr2=0x7fa8ec105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 shutdown_connections 2024-03-19T02:27:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.444+0000 7fa8f083d640 1 -- 172.21.15.116:0/3233325988 wait complete. 2024-03-19T02:27:34.455 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:34.107+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:34.455 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:34.107+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:34.455 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:34.107+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:34.455 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:34.107+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:34.455 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:34.107+0000 7f5aef186700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:34.456 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:27:34.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:27:34.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- 172.21.15.116:0/515295282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4100730 msgr2=0x7f94a4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 --2- 172.21.15.116:0/515295282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4100730 0x7f94a4100b50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9494009900 tx=0x7f949402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:34.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- 172.21.15.116:0/515295282 shutdown_connections 2024-03-19T02:27:34.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 --2- 172.21.15.116:0/515295282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a41022e0 0x7f94a4108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 --2- 172.21.15.116:0/515295282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4101920 0x7f94a4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 --2- 172.21.15.116:0/515295282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a4100730 0x7f94a4100b50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- 172.21.15.116:0/515295282 >> 172.21.15.116:0/515295282 conn(0x7f94a40fbcc0 msgr2=0x7f94a40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- 172.21.15.116:0/515295282 shutdown_connections 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- 172.21.15.116:0/515295282 wait complete. 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 Processor -- start 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.561+0000 7f94a97df640 1 -- start start 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4100730 0x7f94a41969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 0x7f94a4196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410c250 con 0x7f94a41022e0 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410c0d0 con 0x7f94a4101920 2024-03-19T02:27:34.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410c3d0 con 0x7f94a4100730 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4100730 0x7f94a41969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a27fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 0x7f94a4196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36374/0 (socket says 172.21.15.116:36374) 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 -- 172.21.15.116:0/2249614918 learned_addr learned my addr 172.21.15.116:0/2249614918 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 -- 172.21.15.116:0/2249614918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4100730 msgr2=0x7f94a41969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4100730 0x7f94a41969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 -- 172.21.15.116:0/2249614918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 msgr2=0x7f94a4196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 0x7f94a4196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 -- 172.21.15.116:0/2249614918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a419e600 con 0x7f94a41022e0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a27fc640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 0x7f94a4196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a37fe640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f949c00eb10 tx=0x7f949c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c004b20 con 0x7f94a41022e0 2024-03-19T02:27:34.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949c004cc0 con 0x7f94a41022e0 2024-03-19T02:27:34.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a419e8f0 con 0x7f94a41022e0 2024-03-19T02:27:34.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c010690 con 0x7f94a41022e0 2024-03-19T02:27:34.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.562+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a419edf0 con 0x7f94a41022e0 2024-03-19T02:27:34.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.563+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a4107b10 con 0x7f94a41022e0 2024-03-19T02:27:34.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.565+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f949c005000 con 0x7f94a41022e0 2024-03-19T02:27:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.565+0000 7f948bfff640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 0x7f948c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.565+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 0x7f948c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.565+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f949c0cf570 con 0x7f94a41022e0 2024-03-19T02:27:34.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.565+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 0x7f948c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9494005ab0 tx=0x7f9494005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.568+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f949c053740 con 0x7f94a41022e0 2024-03-19T02:27:34.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.744+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94a402d1f0 con 0x7f94a41022e0 2024-03-19T02:27:34.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.745+0000 7f948bfff640 1 -- 172.21.15.116:0/2249614918 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f949c09c290 con 0x7f94a41022e0 2024-03-19T02:27:34.746 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:27:34.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 msgr2=0x7f948c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 0x7f948c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9494005ab0 tx=0x7f9494005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 msgr2=0x7f94a419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f949c00eb10 tx=0x7f949c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 shutdown_connections 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f948c0a2800 0x7f948c0a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94a41022e0 0x7f94a419df90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a4101920 0x7f94a4196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 --2- 172.21.15.116:0/2249614918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94a4100730 0x7f94a41969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 >> 172.21.15.116:0/2249614918 conn(0x7f94a40fbcc0 msgr2=0x7f94a4104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 shutdown_connections 2024-03-19T02:27:34.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.747+0000 7f94a97df640 1 -- 172.21.15.116:0/2249614918 wait complete. 2024-03-19T02:27:34.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- 172.21.15.116:0/3477096205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0101850 msgr2=0x7f11d0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/3477096205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0101850 0x7f11d0101c70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009ca0 tx=0x7f11c802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- 172.21.15.116:0/3477096205 shutdown_connections 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/3477096205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0103400 0x7f11d0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/3477096205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0102a40 0x7f11d0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/3477096205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0101850 0x7f11d0101c70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- 172.21.15.116:0/3477096205 >> 172.21.15.116:0/3477096205 conn(0x7f11d00fcdc0 msgr2=0x7f11d00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- 172.21.15.116:0/3477096205 shutdown_connections 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- 172.21.15.116:0/3477096205 wait complete. 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 Processor -- start 2024-03-19T02:27:34.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- start start 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0101850 0x7f11d0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0103400 0x7f11d019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d370 con 0x7f11d0102a40 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d1f0 con 0x7f11d0101850 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.864+0000 7f11d6c8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d4f0 con 0x7f11d0103400 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36394/0 (socket says 172.21.15.116:36394) 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 -- 172.21.15.116:0/276411742 learned_addr learned my addr 172.21.15.116:0/276411742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11d4a28640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0101850 0x7f11d0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11d5229640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0103400 0x7f11d019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 -- 172.21.15.116:0/276411742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0103400 msgr2=0x7f11d019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0103400 0x7f11d019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 -- 172.21.15.116:0/276411742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0101850 msgr2=0x7f11d0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0101850 0x7f11d0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 -- 172.21.15.116:0/276411742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d019f6e0 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cffff640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f11c400e9f0 tx=0x7f11c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c400cd80 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11c40054d0 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d019f9d0 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c4010490 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.865+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d019fed0 con 0x7f11d0102a40 2024-03-19T02:27:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.866+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d0108c30 con 0x7f11d0102a40 2024-03-19T02:27:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.867+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f11c4005020 con 0x7f11d0102a40 2024-03-19T02:27:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.868+0000 7f11cdffb640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 0x7f11b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.868+0000 7f11d4a28640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 0x7f11b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:34.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.868+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f11c40cf180 con 0x7f11d0102a40 2024-03-19T02:27:34.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.868+0000 7f11d4a28640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 0x7f11b80a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f11c8032380 tx=0x7f11c8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:34.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:34.872+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11c409bea0 con 0x7f11d0102a40 2024-03-19T02:27:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-mon[32263]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T02:27:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T02:27:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:27:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3233325988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:35.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2249614918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.047+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f11d0062420 con 0x7f11d0102a40 2024-03-19T02:27:35.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.047+0000 7f11cdffb640 1 -- 172.21.15.116:0/276411742 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f11c409b7a0 con 0x7f11d0102a40 2024-03-19T02:27:35.048 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 msgr2=0x7f11b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 0x7f11b80a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f11c8032380 tx=0x7f11c8000f40 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 msgr2=0x7f11d0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f11c400e9f0 tx=0x7f11c400edf0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 shutdown_connections 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f11b80a2800 0x7f11b80a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11d0103400 0x7f11d019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0102a40 0x7f11d0198000 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 --2- 172.21.15.116:0/276411742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11d0101850 0x7f11d0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 >> 172.21.15.116:0/276411742 conn(0x7f11d00fcdc0 msgr2=0x7f11d0105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 shutdown_connections 2024-03-19T02:27:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.049+0000 7f11d6c8c640 1 -- 172.21.15.116:0/276411742 wait complete. 2024-03-19T02:27:35.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:27:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.163+0000 7f86daabd640 1 -- 172.21.15.116:0/3454377934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 msgr2=0x7f86d4101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.163+0000 7f86daabd640 1 --2- 172.21.15.116:0/3454377934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4101da0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009ca0 tx=0x7f86c402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- 172.21.15.116:0/3454377934 shutdown_connections 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- 172.21.15.116:0/3454377934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 0x7f86d4108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- 172.21.15.116:0/3454377934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4101da0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- 172.21.15.116:0/3454377934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 0x7f86d4100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- 172.21.15.116:0/3454377934 >> 172.21.15.116:0/3454377934 conn(0x7f86d40fbcc0 msgr2=0x7f86d40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- 172.21.15.116:0/3454377934 shutdown_connections 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- 172.21.15.116:0/3454377934 wait complete. 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 Processor -- start 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- start start 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 0x7f86d4196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 0x7f86d419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410c240 con 0x7f86d4101920 2024-03-19T02:27:35.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410c0c0 con 0x7f86d41022e0 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86daabd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410c3c0 con 0x7f86d4100730 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36418/0 (socket says 172.21.15.116:36418) 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86d3fff640 1 -- 172.21.15.116:0/3776652042 learned_addr learned my addr 172.21.15.116:0/3776652042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86d8859640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 0x7f86d4196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.164+0000 7f86d905a640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 0x7f86d419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 -- 172.21.15.116:0/3776652042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 msgr2=0x7f86d4196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 0x7f86d4196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 -- 172.21.15.116:0/3776652042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 msgr2=0x7f86d419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 0x7f86d419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 -- 172.21.15.116:0/3776652042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d419e590 con 0x7f86d4101920 2024-03-19T02:27:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d3fff640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009dc0 tx=0x7f86c40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4043020 con 0x7f86d4101920 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d4069e10 con 0x7f86d4101920 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86c4003e20 con 0x7f86d4101920 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4039040 con 0x7f86d4101920 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.165+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d406a300 con 0x7f86d4101920 2024-03-19T02:27:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.166+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d4068a60 con 0x7f86d4101920 2024-03-19T02:27:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:34 smithi162 ceph-mon[35761]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T02:27:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T02:27:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:34 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:27:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3233325988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:35.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2249614918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.167+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f86c4046020 con 0x7f86d4101920 2024-03-19T02:27:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.168+0000 7f86d1ffb640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 0x7f86c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.168+0000 7f86d8859640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 0x7f86c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.168+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f86c40f3330 con 0x7f86d4101920 2024-03-19T02:27:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.168+0000 7f86d8859640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 0x7f86c00a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f86d4101780 tx=0x7f86cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.172+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86c404d020 con 0x7f86d4101920 2024-03-19T02:27:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:34 smithi154 ceph-mon[35693]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T02:27:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1528071283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T02:27:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:34 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:27:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3233325988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2249614918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.346+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f86d419e820 con 0x7f86d4101920 2024-03-19T02:27:35.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.347+0000 7f86d1ffb640 1 -- 172.21.15.116:0/3776652042 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f86c40c0050 con 0x7f86d4101920 2024-03-19T02:27:35.348 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 msgr2=0x7f86c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 0x7f86c00a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f86d4101780 tx=0x7f86cc005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 msgr2=0x7f86d4196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009dc0 tx=0x7f86c40328a0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 shutdown_connections 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f86c00a2800 0x7f86c00a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d41022e0 0x7f86d419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86d4101920 0x7f86d4196eb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 --2- 172.21.15.116:0/3776652042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86d4100730 0x7f86d4196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 >> 172.21.15.116:0/3776652042 conn(0x7f86d40fbcc0 msgr2=0x7f86d4104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 shutdown_connections 2024-03-19T02:27:35.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.349+0000 7f86daabd640 1 -- 172.21.15.116:0/3776652042 wait complete. 2024-03-19T02:27:35.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:27:35.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- 172.21.15.116:0/1619186703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6028101930 msgr2=0x7f6028101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 --2- 172.21.15.116:0/1619186703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6028101930 0x7f6028101db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6010009900 tx=0x7f601002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:35.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- 172.21.15.116:0/1619186703 shutdown_connections 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 --2- 172.21.15.116:0/1619186703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60281022f0 0x7f6028108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 --2- 172.21.15.116:0/1619186703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6028101930 0x7f6028101db0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 --2- 172.21.15.116:0/1619186703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028100740 0x7f6028100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- 172.21.15.116:0/1619186703 >> 172.21.15.116:0/1619186703 conn(0x7f60280fbcd0 msgr2=0x7f60280fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- 172.21.15.116:0/1619186703 shutdown_connections 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- 172.21.15.116:0/1619186703 wait complete. 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 Processor -- start 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.465+0000 7f602eb28640 1 -- start start 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6028100740 0x7f6028197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028101930 0x7f6028197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d390 con 0x7f60281022f0 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d210 con 0x7f6028101930 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f602810d510 con 0x7f6028100740 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36446/0 (socket says 172.21.15.116:36446) 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 -- 172.21.15.116:0/1891066892 learned_addr learned my addr 172.21.15.116:0/1891066892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602c8c4640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6028100740 0x7f6028197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f6027fff640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028101930 0x7f6028197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 -- 172.21.15.116:0/1891066892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6028100740 msgr2=0x7f6028197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6028100740 0x7f6028197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 -- 172.21.15.116:0/1891066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028101930 msgr2=0x7f6028197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028101930 0x7f6028197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 -- 172.21.15.116:0/1891066892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602819f6a0 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602d0c5640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f601800e970 tx=0x7f601800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601800ccc0 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f601800ce60 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602819f990 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018010430 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.466+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f602819feb0 con 0x7f60281022f0 2024-03-19T02:27:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.467+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6028107ae0 con 0x7f60281022f0 2024-03-19T02:27:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.468+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6018004ff0 con 0x7f60281022f0 2024-03-19T02:27:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.469+0000 7f6025ffb640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 0x7f60140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.469+0000 7f602c8c4640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 0x7f60140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.469+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f60180cf170 con 0x7f60281022f0 2024-03-19T02:27:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.469+0000 7f602c8c4640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 0x7f60140a4d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6028101790 tx=0x7f6020009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.473+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f601809be90 con 0x7f60281022f0 2024-03-19T02:27:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.650+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6028062420 con 0x7f60281022f0 2024-03-19T02:27:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.651+0000 7f6025ffb640 1 -- 172.21.15.116:0/1891066892 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f601809b790 con 0x7f60281022f0 2024-03-19T02:27:35.652 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 msgr2=0x7f60140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 0x7f60140a4d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6028101790 tx=0x7f6020009500 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 msgr2=0x7f602819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f601800e970 tx=0x7f601800ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 shutdown_connections 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f60140a2880 0x7f60140a4d40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60281022f0 0x7f602819f030 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028101930 0x7f6028197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 --2- 172.21.15.116:0/1891066892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6028100740 0x7f6028197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 >> 172.21.15.116:0/1891066892 conn(0x7f60280fbcd0 msgr2=0x7f6028104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 shutdown_connections 2024-03-19T02:27:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.653+0000 7f602eb28640 1 -- 172.21.15.116:0/1891066892 wait complete. 2024-03-19T02:27:35.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:35.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- 172.21.15.116:0/251614893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd820102360 msgr2=0x7fd820104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 --2- 172.21.15.116:0/251614893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd820102360 0x7fd820104750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd818009be0 tx=0x7fd81802bef0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- 172.21.15.116:0/251614893 shutdown_connections 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 --2- 172.21.15.116:0/251614893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8201075c0 0x7fd8201099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 --2- 172.21.15.116:0/251614893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820104c90 0x7fd820107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 --2- 172.21.15.116:0/251614893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd820102360 0x7fd820104750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- 172.21.15.116:0/251614893 >> 172.21.15.116:0/251614893 conn(0x7fd8200fbcd0 msgr2=0x7fd8200fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- 172.21.15.116:0/251614893 shutdown_connections 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- 172.21.15.116:0/251614893 wait complete. 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 Processor -- start 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.767+0000 7fd827a61640 1 -- start start 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd820104c90 0x7fd820197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8201075c0 0x7fd82019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010d360 con 0x7fd820102360 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010d1e0 con 0x7fd820104c90 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd82010d4e0 con 0x7fd8201075c0 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36452/0 (socket says 172.21.15.116:36452) 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 -- 172.21.15.116:0/3684779793 learned_addr learned my addr 172.21.15.116:0/3684779793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd825ffe640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8201075c0 0x7fd82019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd824ffc640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd820104c90 0x7fd820197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 -- 172.21.15.116:0/3684779793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8201075c0 msgr2=0x7fd82019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8201075c0 0x7fd82019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 -- 172.21.15.116:0/3684779793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd820104c90 msgr2=0x7fd820197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd820104c90 0x7fd820197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 -- 172.21.15.116:0/3684779793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82019f6b0 con 0x7fd820102360 2024-03-19T02:27:35.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd8257fd640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd818004af0 tx=0x7fd818033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818043020 con 0x7fd820102360 2024-03-19T02:27:35.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82019f940 con 0x7fd820102360 2024-03-19T02:27:35.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd81802ca80 con 0x7fd820102360 2024-03-19T02:27:35.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd818004860 con 0x7fd820102360 2024-03-19T02:27:35.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.768+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd82019fee0 con 0x7fd820102360 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.769+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd820068a60 con 0x7fd820102360 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.770+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd818046020 con 0x7fd820102360 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.771+0000 7fd81e7fc640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 0x7fd8080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.771+0000 7fd824ffc640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 0x7fd8080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.771+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd8180f1370 con 0x7fd820102360 2024-03-19T02:27:35.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.771+0000 7fd824ffc640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 0x7fd8080a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd820198fa0 tx=0x7fd814009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:35.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.775+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd81804d020 con 0x7fd820102360 2024-03-19T02:27:35.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.788+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7fd8180ba0a0 con 0x7fd820102360 2024-03-19T02:27:35.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.959+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd820062420 con 0x7fd820102360 2024-03-19T02:27:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.961+0000 7fd81e7fc640 1 -- 172.21.15.116:0/3684779793 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7fd8180be130 con 0x7fd820102360 2024-03-19T02:27:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:35.962 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 msgr2=0x7fd8080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 0x7fd8080a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd820198fa0 tx=0x7fd814009260 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 msgr2=0x7fd820197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd818004af0 tx=0x7fd818033c90 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 shutdown_connections 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8080a2800 0x7fd8080a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8201075c0 0x7fd82019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd820104c90 0x7fd820197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 --2- 172.21.15.116:0/3684779793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820102360 0x7fd820197a90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 >> 172.21.15.116:0/3684779793 conn(0x7fd8200fbcd0 msgr2=0x7fd820102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 shutdown_connections 2024-03-19T02:27:35.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:35.963+0000 7fd827a61640 1 -- 172.21.15.116:0/3684779793 wait complete. 2024-03-19T02:27:35.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:35 smithi116 ceph-mon[32263]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:35.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/276411742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3776652042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1891066892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:35.975 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:27:35.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:27:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 -- 172.21.15.116:0/2322056196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24101920 msgr2=0x7f8c24101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/2322056196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24101920 0x7f8c24101da0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009900 tx=0x7f8c1402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 -- 172.21.15.116:0/2322056196 shutdown_connections 2024-03-19T02:27:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/2322056196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 0x7f8c24108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/2322056196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24101920 0x7f8c24101da0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/2322056196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24100730 0x7f8c24100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 -- 172.21.15.116:0/2322056196 >> 172.21.15.116:0/2322056196 conn(0x7f8c240fbcc0 msgr2=0x7f8c240fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 -- 172.21.15.116:0/2322056196 shutdown_connections 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.079+0000 7f8c2bf44640 1 -- 172.21.15.116:0/2322056196 wait complete. 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 Processor -- start 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 -- start start 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24101920 0x7f8c2419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 0x7f8c241a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410c260 con 0x7f8c24100730 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410c0e0 con 0x7f8c24101920 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2bf44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2410c3e0 con 0x7f8c241022e0 2024-03-19T02:27:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36480/0 (socket says 172.21.15.116:36480) 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 -- 172.21.15.116:0/3483954179 learned_addr learned my addr 172.21.15.116:0/3483954179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c2a4e1640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 0x7f8c241a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c294df640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24101920 0x7f8c2419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 -- 172.21.15.116:0/3483954179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 msgr2=0x7f8c241a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 0x7f8c241a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 -- 172.21.15.116:0/3483954179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24101920 msgr2=0x7f8c2419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24101920 0x7f8c2419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.080+0000 7f8c29ce0640 1 -- 172.21.15.116:0/3483954179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c241a18f0 con 0x7f8c24100730 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c29ce0640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000e9c0 tx=0x7f8c2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000cd80 con 0x7f8c24100730 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c200035f0 con 0x7f8c24100730 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c241a1be0 con 0x7f8c24100730 2024-03-19T02:27:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20010690 con 0x7f8c24100730 2024-03-19T02:27:36.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.081+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c241a2100 con 0x7f8c24100730 2024-03-19T02:27:36.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.082+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c241a22d0 con 0x7f8c24100730 2024-03-19T02:27:36.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.083+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f8c200037a0 con 0x7f8c24100730 2024-03-19T02:27:36.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.083+0000 7f8c1affd640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 0x7f8c100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.083+0000 7f8c294df640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 0x7f8c100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.084+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f8c200cf2f0 con 0x7f8c24100730 2024-03-19T02:27:36.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.084+0000 7f8c294df640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 0x7f8c100a4d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009230 tx=0x7f8c14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.088+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c2009beb0 con 0x7f8c24100730 2024-03-19T02:27:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:35 smithi162 ceph-mon[35761]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/276411742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3776652042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1891066892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:35 smithi154 ceph-mon[35693]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/276411742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3776652042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1891066892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.265+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c241a24a0 con 0x7f8c24100730 2024-03-19T02:27:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.265+0000 7f8c1affd640 1 -- 172.21.15.116:0/3483954179 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f8c2009b7b0 con 0x7f8c24100730 2024-03-19T02:27:36.266 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 msgr2=0x7f8c100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 0x7f8c100a4d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009230 tx=0x7f8c14005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 msgr2=0x7f8c24199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000e9c0 tx=0x7f8c2000ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 shutdown_connections 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8c100a2890 0x7f8c100a4d50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c241022e0 0x7f8c241a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c24101920 0x7f8c2419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 --2- 172.21.15.116:0/3483954179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c24100730 0x7f8c24199cb0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 >> 172.21.15.116:0/3483954179 conn(0x7f8c240fbcc0 msgr2=0x7f8c24069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 shutdown_connections 2024-03-19T02:27:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.267+0000 7f8c2bf44640 1 -- 172.21.15.116:0/3483954179 wait complete. 2024-03-19T02:27:36.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:27:36.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 -- 172.21.15.116:0/3451275496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384103490 msgr2=0x7f8384109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 --2- 172.21.15.116:0/3451275496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384103490 0x7f8384109d20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f837c009be0 tx=0x7f837c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 -- 172.21.15.116:0/3451275496 shutdown_connections 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 --2- 172.21.15.116:0/3451275496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384103490 0x7f8384109d20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 --2- 172.21.15.116:0/3451275496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8384102ad0 0x7f8384102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 --2- 172.21.15.116:0/3451275496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83840ffb00 0x7f83840fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 -- 172.21.15.116:0/3451275496 >> 172.21.15.116:0/3451275496 conn(0x7f83840fb410 msgr2=0x7f83840fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 -- 172.21.15.116:0/3451275496 shutdown_connections 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 -- 172.21.15.116:0/3451275496 wait complete. 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.383+0000 7f8389fb2640 1 Processor -- start 2024-03-19T02:27:36.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 -- start start 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384102ad0 0x7f8384100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384103490 0x7f8384100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f838410d380 con 0x7f83840ffb00 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f838410d200 con 0x7f8384102ad0 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8389fb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f838410d500 con 0x7f8384103490 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8383fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384103490 0x7f8384100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36498/0 (socket says 172.21.15.116:36498) 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 -- 172.21.15.116:0/1347959320 learned_addr learned my addr 172.21.15.116:0/1347959320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f8382ffd640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384102ad0 0x7f8384100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 -- 172.21.15.116:0/1347959320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384103490 msgr2=0x7f8384100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384103490 0x7f8384100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 -- 172.21.15.116:0/1347959320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384102ad0 msgr2=0x7f8384100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384102ad0 0x7f8384100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 -- 172.21.15.116:0/1347959320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8384101850 con 0x7f83840ffb00 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.384+0000 7f83837fe640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f837400e9c0 tx=0x7f837400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.385+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837400cd80 con 0x7f83840ffb00 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.385+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83841a1af0 con 0x7f83840ffb00 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.385+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83740035f0 con 0x7f83840ffb00 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.385+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374010690 con 0x7f83840ffb00 2024-03-19T02:27:36.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.385+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83841a2040 con 0x7f83840ffb00 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.386+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8384108c00 con 0x7f83840ffb00 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.387+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f83740037a0 con 0x7f83840ffb00 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.387+0000 7f8380ff9640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 0x7f836c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.387+0000 7f8382ffd640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 0x7f836c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.387+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f83740cf4c0 con 0x7f83840ffb00 2024-03-19T02:27:36.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.388+0000 7f8382ffd640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 0x7f836c0a4d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f83841021b0 tx=0x7f8378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.391+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f837409c080 con 0x7f83840ffb00 2024-03-19T02:27:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.564+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f838402d3e0 con 0x7f83840ffb00 2024-03-19T02:27:36.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.564+0000 7f8380ff9640 1 -- 172.21.15.116:0/1347959320 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f837409b980 con 0x7f83840ffb00 2024-03-19T02:27:36.565 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 msgr2=0x7f836c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 0x7f836c0a4d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f83841021b0 tx=0x7f8378009260 comp rx=0 tx=0).stop 2024-03-19T02:27:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 msgr2=0x7f8384102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f837400e9c0 tx=0x7f837400ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:36.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 shutdown_connections 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f836c0a2890 0x7f836c0a4d50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8384103490 0x7f8384100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8384102ad0 0x7f8384100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 --2- 172.21.15.116:0/1347959320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83840ffb00 0x7f8384102440 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.566+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 >> 172.21.15.116:0/1347959320 conn(0x7f83840fb410 msgr2=0x7f8384105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.567+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 shutdown_connections 2024-03-19T02:27:36.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.567+0000 7f8389fb2640 1 -- 172.21.15.116:0/1347959320 wait complete. 2024-03-19T02:27:36.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 -- 172.21.15.116:0/3573840827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b741022f0 msgr2=0x7f5b74108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/3573840827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b741022f0 0x7f5b74108b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009900 tx=0x7f5b7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 -- 172.21.15.116:0/3573840827 shutdown_connections 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/3573840827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b741022f0 0x7f5b74108b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/3573840827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74101930 0x7f5b74101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/3573840827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 0x7f5b74100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 -- 172.21.15.116:0/3573840827 >> 172.21.15.116:0/3573840827 conn(0x7f5b740fbcd0 msgr2=0x7f5b740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 -- 172.21.15.116:0/3573840827 shutdown_connections 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 -- 172.21.15.116:0/3573840827 wait complete. 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.683+0000 7f5b7bd59640 1 Processor -- start 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 -- start start 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 0x7f5b7406a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b74101930 0x7f5b7419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410c250 con 0x7f5b741022f0 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410c0d0 con 0x7f5b74100740 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7410c3d0 con 0x7f5b74101930 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b79af5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 0x7f5b7406a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36522/0 (socket says 172.21.15.116:36522) 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 -- 172.21.15.116:0/54690131 learned_addr learned my addr 172.21.15.116:0/54690131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b792f4640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b74101930 0x7f5b7419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 -- 172.21.15.116:0/54690131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b74101930 msgr2=0x7f5b7419c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b74101930 0x7f5b7419c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 -- 172.21.15.116:0/54690131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 msgr2=0x7f5b7406a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 0x7f5b7406a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 -- 172.21.15.116:0/54690131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7419f620 con 0x7f5b741022f0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7a2f6640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009230 tx=0x7f5b70005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70043020 con 0x7f5b741022f0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7419f8b0 con 0x7f5b741022f0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b700043d0 con 0x7f5b741022f0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.684+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70003b40 con 0x7f5b741022f0 2024-03-19T02:27:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.685+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7419fe50 con 0x7f5b741022f0 2024-03-19T02:27:36.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.685+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b74107ae0 con 0x7f5b741022f0 2024-03-19T02:27:36.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.686+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f5b70046020 con 0x7f5b741022f0 2024-03-19T02:27:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.687+0000 7f5b6affd640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 0x7f5b600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.687+0000 7f5b79af5640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 0x7f5b600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.687+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5b700f1460 con 0x7f5b741022f0 2024-03-19T02:27:36.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.688+0000 7f5b79af5640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 0x7f5b600a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5b74101790 tx=0x7f5b6c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.691+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b7004d020 con 0x7f5b741022f0 2024-03-19T02:27:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.872+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5b7402d380 con 0x7f5b741022f0 2024-03-19T02:27:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.872+0000 7f5b6affd640 1 -- 172.21.15.116:0/54690131 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5b700be130 con 0x7f5b741022f0 2024-03-19T02:27:36.873 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:27:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 msgr2=0x7f5b600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 0x7f5b600a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5b74101790 tx=0x7f5b6c009500 comp rx=0 tx=0).stop 2024-03-19T02:27:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 msgr2=0x7f5b7419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009230 tx=0x7f5b70005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:36.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 shutdown_connections 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f5b600a2890 0x7f5b600a4d50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b741022f0 0x7f5b7419ed60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b74101930 0x7f5b7419c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 --2- 172.21.15.116:0/54690131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b74100740 0x7f5b7406a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 >> 172.21.15.116:0/54690131 conn(0x7f5b740fbcd0 msgr2=0x7f5b74104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 shutdown_connections 2024-03-19T02:27:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.874+0000 7f5b7bd59640 1 -- 172.21.15.116:0/54690131 wait complete. 2024-03-19T02:27:36.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:36 smithi116 ceph-mon[32263]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-19T02:27:36.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:36 smithi116 ceph-mon[32263]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:27:36.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3684779793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:36.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3483954179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.885 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1347959320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:36.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 -- 172.21.15.116:0/3111326641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc102e60 msgr2=0x7f72dc103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3111326641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc102e60 0x7f72dc103280 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009900 tx=0x7f72cc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 -- 172.21.15.116:0/3111326641 shutdown_connections 2024-03-19T02:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3111326641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 0x7f72dc109e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3111326641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc0ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3111326641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc102e60 0x7f72dc103280 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 -- 172.21.15.116:0/3111326641 >> 172.21.15.116:0/3111326641 conn(0x7f72dc0fb450 msgr2=0x7f72dc0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 -- 172.21.15.116:0/3111326641 shutdown_connections 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.989+0000 7f72e0a61640 1 -- 172.21.15.116:0/3111326641 wait complete. 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 Processor -- start 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 -- start start 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 0x7f72dc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc102e60 0x7f72dc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc10d360 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc10d1e0 con 0x7f72dc100290 2024-03-19T02:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72e0a61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc10d4e0 con 0x7f72dc102e60 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36540/0 (socket says 172.21.15.116:36540) 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 -- 172.21.15.116:0/3635079170 learned_addr learned my addr 172.21.15.116:0/3635079170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72d9d9b640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 0x7f72dc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72dad9d640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc102e60 0x7f72dc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 -- 172.21.15.116:0/3635079170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc102e60 msgr2=0x7f72dc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc102e60 0x7f72dc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 -- 172.21.15.116:0/3635079170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 msgr2=0x7f72dc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 0x7f72dc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.990+0000 7f72da59c640 1 -- 172.21.15.116:0/3635079170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc19f700 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72da59c640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72cc000c00 tx=0x7f72cc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc043020 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72cc004350 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc19f990 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc003b80 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.991+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc19ff30 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.992+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72dc108e40 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.993+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f72cc046020 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.993+0000 7f72cb7fe640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 0x7f72c40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:36.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.993+0000 7f72d9d9b640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 0x7f72c40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:36.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.993+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f72cc0f0f20 con 0x7f72dc0ff8d0 2024-03-19T02:27:36.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.994+0000 7f72d9d9b640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 0x7f72c40a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72dc198ff0 tx=0x7f72d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:36.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:36.998+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72cc04d020 con 0x7f72dc0ff8d0 2024-03-19T02:27:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:36 smithi162 ceph-mon[35761]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-19T02:27:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:36 smithi162 ceph-mon[35761]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:27:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3684779793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3483954179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1347959320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.171+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f72dc062420 con 0x7f72dc0ff8d0 2024-03-19T02:27:37.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.171+0000 7f72cb7fe640 1 -- 172.21.15.116:0/3635079170 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f72cc0bdbf0 con 0x7f72dc0ff8d0 2024-03-19T02:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 msgr2=0x7f72c40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 0x7f72c40a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f72dc198ff0 tx=0x7f72d0005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 msgr2=0x7f72dc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72cc000c00 tx=0x7f72cc005dc0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 shutdown_connections 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f72c40a2910 0x7f72c40a4dd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72dc102e60 0x7f72dc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72dc100290 0x7f72dc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 --2- 172.21.15.116:0/3635079170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72dc0ff8d0 0x7f72dc197ae0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 >> 172.21.15.116:0/3635079170 conn(0x7f72dc0fb450 msgr2=0x7f72dc106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 shutdown_connections 2024-03-19T02:27:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.173+0000 7f72e0a61640 1 -- 172.21.15.116:0/3635079170 wait complete. 2024-03-19T02:27:37.185 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-19T02:27:37.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:36 smithi154 ceph-mon[35693]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-19T02:27:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:36 smithi154 ceph-mon[35693]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:27:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3684779793' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3483954179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1347959320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 -- 172.21.15.116:0/2605709569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58101830 msgr2=0x7f2f58101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/2605709569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58101830 0x7f2f58101c50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009900 tx=0x7f2f5402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 -- 172.21.15.116:0/2605709569 shutdown_connections 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/2605709569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f581033e0 0x7f2f58109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/2605709569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f58102a20 0x7f2f58102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/2605709569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58101830 0x7f2f58101c50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 -- 172.21.15.116:0/2605709569 >> 172.21.15.116:0/2605709569 conn(0x7f2f580fce00 msgr2=0x7f2f580ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 -- 172.21.15.116:0/2605709569 shutdown_connections 2024-03-19T02:27:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.286+0000 7f2f5fb59640 1 -- 172.21.15.116:0/2605709569 wait complete. 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 Processor -- start 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 -- start start 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f58101830 0x7f2f58197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f581033e0 0x7f2f5819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5810d380 con 0x7f2f58102a20 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5810d200 con 0x7f2f581033e0 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5fb59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5810d500 con 0x7f2f58101830 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36560/0 (socket says 172.21.15.116:36560) 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 -- 172.21.15.116:0/1926924698 learned_addr learned my addr 172.21.15.116:0/1926924698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d8f5640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f58101830 0x7f2f58197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5e0f6640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f581033e0 0x7f2f5819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 -- 172.21.15.116:0/1926924698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f58101830 msgr2=0x7f2f58197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f58101830 0x7f2f58197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 -- 172.21.15.116:0/1926924698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f581033e0 msgr2=0x7f2f5819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f581033e0 0x7f2f5819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.287+0000 7f2f5d0f4640 1 -- 172.21.15.116:0/1926924698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5819f6d0 con 0x7f2f58102a20 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f5d0f4640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800e9f0 tx=0x7f2f4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800cd80 con 0x7f2f58102a20 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f480054d0 con 0x7f2f58102a20 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5819f9c0 con 0x7f2f58102a20 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48010490 con 0x7f2f58102a20 2024-03-19T02:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.288+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5819fec0 con 0x7f2f58102a20 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.289+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f58108b70 con 0x7f2f58102a20 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.290+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f2f48005020 con 0x7f2f58102a20 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.290+0000 7f2f4effd640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 0x7f2f440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.291+0000 7f2f5d8f5640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 0x7f2f440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.291+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f2f480cf2d0 con 0x7f2f58102a20 2024-03-19T02:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.291+0000 7f2f5d8f5640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 0x7f2f440a4d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f54005ab0 tx=0x7f2f54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.295+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f4809be90 con 0x7f2f58102a20 2024-03-19T02:27:37.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.510+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f5802d1f0 con 0x7f2f58102a20 2024-03-19T02:27:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.510+0000 7f2f4effd640 1 -- 172.21.15.116:0/1926924698 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11950 (secure 0 0 0) 0x7f2f4809b790 con 0x7f2f58102a20 2024-03-19T02:27:37.511 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:37.511 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:27:33.577980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:27:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 msgr2=0x7f2f440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 0x7f2f440a4d50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f54005ab0 tx=0x7f2f54005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 msgr2=0x7f2f58197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800e9f0 tx=0x7f2f4800edf0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 shutdown_connections 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2f440a2890 0x7f2f440a4d50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f581033e0 0x7f2f5819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f58102a20 0x7f2f58197ff0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 --2- 172.21.15.116:0/1926924698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f58101830 0x7f2f58197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 >> 172.21.15.116:0/1926924698 conn(0x7f2f580fce00 msgr2=0x7f2f58105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 shutdown_connections 2024-03-19T02:27:37.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.512+0000 7f2f5fb59640 1 -- 172.21.15.116:0/1926924698 wait complete. 2024-03-19T02:27:37.524 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-19T02:27:37.524 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-19T02:27:37.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3851382798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8101930 msgr2=0x7fc8d8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3851382798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8101930 0x7fc8d8101db0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009900 tx=0x7fc8c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3851382798 shutdown_connections 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3851382798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d81022f0 0x7fc8d8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3851382798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8101930 0x7fc8d8101db0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3851382798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d8100740 0x7fc8d8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3851382798 >> 172.21.15.116:0/3851382798 conn(0x7fc8d80fbcb0 msgr2=0x7fc8d80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3851382798 shutdown_connections 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.626+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3851382798 wait complete. 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 Processor -- start 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 -- start start 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d8101930 0x7fc8d8196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d81022f0 0x7fc8d819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d810c250 con 0x7fc8d8100740 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d810c0d0 con 0x7fc8d81022f0 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8e02bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d810c3d0 con 0x7fc8d8101930 2024-03-19T02:27:37.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58956/0 (socket says 172.21.15.116:58956) 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 -- 172.21.15.116:0/3629164303 learned_addr learned my addr 172.21.15.116:0/3629164303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8dd85a640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d8101930 0x7fc8d8196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de85c640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d81022f0 0x7fc8d819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 -- 172.21.15.116:0/3629164303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d8101930 msgr2=0x7fc8d8196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d8101930 0x7fc8d8196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 -- 172.21.15.116:0/3629164303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d81022f0 msgr2=0x7fc8d819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d81022f0 0x7fc8d819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 -- 172.21.15.116:0/3629164303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d819e5b0 con 0x7fc8d8100740 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8de05b640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400e9c0 tx=0x7fc8d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.627+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400cd80 con 0x7fc8d8100740 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.628+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8d40035f0 con 0x7fc8d8100740 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.628+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d819e8a0 con 0x7fc8d8100740 2024-03-19T02:27:37.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.628+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4010690 con 0x7fc8d8100740 2024-03-19T02:27:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.628+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d819edc0 con 0x7fc8d8100740 2024-03-19T02:27:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.629+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d8107b80 con 0x7fc8d8100740 2024-03-19T02:27:37.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.629+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7fc8d40037a0 con 0x7fc8d8100740 2024-03-19T02:27:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.630+0000 7fc8cf7fe640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 0x7fc8c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.630+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc8d40cf2e0 con 0x7fc8d8100740 2024-03-19T02:27:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.630+0000 7fc8dd85a640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 0x7fc8c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.631+0000 7fc8dd85a640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 0x7fc8c40a4d50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009230 tx=0x7fc8c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.634+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8d409bea0 con 0x7fc8d8100740 2024-03-19T02:27:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.812+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8d8061c60 con 0x7fc8c40a2890 2024-03-19T02:27:37.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.822+0000 7fc8cf7fe640 1 -- 172.21.15.116:0/3629164303 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7fc8d8061c60 con 0x7fc8c40a2890 2024-03-19T02:27:37.830 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:37.832 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:27:37.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 msgr2=0x7fc8c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 0x7fc8c40a4d50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009230 tx=0x7fc8c8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 msgr2=0x7fc8d8196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400e9c0 tx=0x7fc8d400ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 shutdown_connections 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fc8c40a2890 0x7fc8c40a4d50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8d81022f0 0x7fc8d819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8d8101930 0x7fc8d8196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 --2- 172.21.15.116:0/3629164303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d8100740 0x7fc8d8196990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 >> 172.21.15.116:0/3629164303 conn(0x7fc8d80fbcb0 msgr2=0x7fc8d8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 shutdown_connections 2024-03-19T02:27:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.825+0000 7fc8e02bf640 1 -- 172.21.15.116:0/3629164303 wait complete. 2024-03-19T02:27:37.837 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-03-19T02:27:37.636605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661904,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1361,"num_read_kb":806,"num_write":451,"num_write_kb":3097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372208,"kb_used_data":14728,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569616,"statfs":{"total":1151940427776,"available":1151559286784,"internally_reserved":0,"allocated":15081472,"data_stored":8245654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":61,"num_write":25,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735019+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:27:09.735019+0000","last_peered":"2024-03-19T02:27:09.735019+0000","last_clean":"2024-03-19T02:27:09.735019+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:27:09.735019+0000","last_undegraded":"2024-03-19T02:27:09.735019+0000","last_fullsized":"2024-03-19T02:27:09.735019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031274+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:27:10.031274+0000","last_peered":"2024-03-19T02:27:10.031274+0000","last_clean":"2024-03-19T02:27:10.031274+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:27:10.031274+0000","last_undegraded":"2024-03-19T02:27:10.031274+0000","last_fullsized":"2024-03-19T02:27:10.031274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030982+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:27:10.030982+0000","last_peered":"2024-03-19T02:27:10.030982+0000","last_clean":"2024-03-19T02:27:10.030982+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:27:10.030982+0000","last_undegraded":"2024-03-19T02:27:10.030982+0000","last_fullsized":"2024-03-19T02:27:10.030982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784827+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:27:09.784827+0000","last_peered":"2024-03-19T02:27:09.784827+0000","last_clean":"2024-03-19T02:27:09.784827+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:27:09.784827+0000","last_undegraded":"2024-03-19T02:27:09.784827+0000","last_fullsized":"2024-03-19T02:27:09.784827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736421+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:27:09.736421+0000","last_peered":"2024-03-19T02:27:09.736421+0000","last_clean":"2024-03-19T02:27:09.736421+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:27:09.736421+0000","last_undegraded":"2024-03-19T02:27:09.736421+0000","last_fullsized":"2024-03-19T02:27:09.736421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732549+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:27:09.732549+0000","last_peered":"2024-03-19T02:27:09.732549+0000","last_clean":"2024-03-19T02:27:09.732549+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:27:09.732549+0000","last_undegraded":"2024-03-19T02:27:09.732549+0000","last_fullsized":"2024-03-19T02:27:09.732549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736294+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:27:09.736294+0000","last_peered":"2024-03-19T02:27:09.736294+0000","last_clean":"2024-03-19T02:27:09.736294+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:27:09.736294+0000","last_undegraded":"2024-03-19T02:27:09.736294+0000","last_fullsized":"2024-03-19T02:27:09.736294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732129+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:27:09.732129+0000","last_peered":"2024-03-19T02:27:09.732129+0000","last_clean":"2024-03-19T02:27:09.732129+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:27:09.732129+0000","last_undegraded":"2024-03-19T02:27:09.732129+0000","last_fullsized":"2024-03-19T02:27:09.732129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031592+0000","last_change":"2024-03-19T02:26:08.850490+0000","last_active":"2024-03-19T02:27:10.031592+0000","last_peered":"2024-03-19T02:27:10.031592+0000","last_clean":"2024-03-19T02:27:10.031592+0000","last_became_active":"2024-03-19T02:26:08.850087+0000","last_became_peered":"2024-03-19T02:26:08.850087+0000","last_unstale":"2024-03-19T02:27:10.031592+0000","last_undegraded":"2024-03-19T02:27:10.031592+0000","last_fullsized":"2024-03-19T02:27:10.031592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732867+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:27:09.732867+0000","last_peered":"2024-03-19T02:27:09.732867+0000","last_clean":"2024-03-19T02:27:09.732867+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:27:09.732867+0000","last_undegraded":"2024-03-19T02:27:09.732867+0000","last_fullsized":"2024-03-19T02:27:09.732867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736754+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:27:09.736754+0000","last_peered":"2024-03-19T02:27:09.736754+0000","last_clean":"2024-03-19T02:27:09.736754+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:27:09.736754+0000","last_undegraded":"2024-03-19T02:27:09.736754+0000","last_fullsized":"2024-03-19T02:27:09.736754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784382+0000","last_change":"2024-03-19T02:26:08.789057+0000","last_active":"2024-03-19T02:27:09.784382+0000","last_peered":"2024-03-19T02:27:09.784382+0000","last_clean":"2024-03-19T02:27:09.784382+0000","last_became_active":"2024-03-19T02:26:08.788689+0000","last_became_peered":"2024-03-19T02:26:08.788689+0000","last_unstale":"2024-03-19T02:27:09.784382+0000","last_undegraded":"2024-03-19T02:27:09.784382+0000","last_fullsized":"2024-03-19T02:27:09.784382+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:08.295772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782778+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:27:09.782778+0000","last_peered":"2024-03-19T02:27:09.782778+0000","last_clean":"2024-03-19T02:27:09.782778+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:27:09.782778+0000","last_undegraded":"2024-03-19T02:27:09.782778+0000","last_fullsized":"2024-03-19T02:27:09.782778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030898+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:27:10.030898+0000","last_peered":"2024-03-19T02:27:10.030898+0000","last_clean":"2024-03-19T02:27:10.030898+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:27:10.030898+0000","last_undegraded":"2024-03-19T02:27:10.030898+0000","last_fullsized":"2024-03-19T02:27:10.030898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030955+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:27:10.030955+0000","last_peered":"2024-03-19T02:27:10.030955+0000","last_clean":"2024-03-19T02:27:10.030955+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:27:10.030955+0000","last_undegraded":"2024-03-19T02:27:10.030955+0000","last_fullsized":"2024-03-19T02:27:10.030955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031044+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:27:10.031044+0000","last_peered":"2024-03-19T02:27:10.031044+0000","last_clean":"2024-03-19T02:27:10.031044+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:27:10.031044+0000","last_undegraded":"2024-03-19T02:27:10.031044+0000","last_fullsized":"2024-03-19T02:27:10.031044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735240+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:27:09.735240+0000","last_peered":"2024-03-19T02:27:09.735240+0000","last_clean":"2024-03-19T02:27:09.735240+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:27:09.735240+0000","last_undegraded":"2024-03-19T02:27:09.735240+0000","last_fullsized":"2024-03-19T02:27:09.735240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784626+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:27:09.784626+0000","last_peered":"2024-03-19T02:27:09.784626+0000","last_clean":"2024-03-19T02:27:09.784626+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:27:09.784626+0000","last_undegraded":"2024-03-19T02:27:09.784626+0000","last_fullsized":"2024-03-19T02:27:09.784626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732894+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:27:09.732894+0000","last_peered":"2024-03-19T02:27:09.732894+0000","last_clean":"2024-03-19T02:27:09.732894+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:27:09.732894+0000","last_undegraded":"2024-03-19T02:27:09.732894+0000","last_fullsized":"2024-03-19T02:27:09.732894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732436+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:27:09.732436+0000","last_peered":"2024-03-19T02:27:09.732436+0000","last_clean":"2024-03-19T02:27:09.732436+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:27:09.732436+0000","last_undegraded":"2024-03-19T02:27:09.732436+0000","last_fullsized":"2024-03-19T02:27:09.732436+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.809123+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:27:32.809123+0000","last_peered":"2024-03-19T02:27:32.809123+0000","last_clean":"2024-03-19T02:27:32.809123+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:27:32.809123+0000","last_undegraded":"2024-03-19T02:27:32.809123+0000","last_fullsized":"2024-03-19T02:27:32.809123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732449+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:27:09.732449+0000","last_peered":"2024-03-19T02:27:09.732449+0000","last_clean":"2024-03-19T02:27:09.732449+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:27:09.732449+0000","last_undegraded":"2024-03-19T02:27:09.732449+0000","last_fullsized":"2024-03-19T02:27:09.732449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784240+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:27:09.784240+0000","last_peered":"2024-03-19T02:27:09.784240+0000","last_clean":"2024-03-19T02:27:09.784240+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:27:09.784240+0000","last_undegraded":"2024-03-19T02:27:09.784240+0000","last_fullsized":"2024-03-19T02:27:09.784240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732693+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:27:09.732693+0000","last_peered":"2024-03-19T02:27:09.732693+0000","last_clean":"2024-03-19T02:27:09.732693+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:27:09.732693+0000","last_undegraded":"2024-03-19T02:27:09.732693+0000","last_fullsized":"2024-03-19T02:27:09.732693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031294+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:27:10.031294+0000","last_peered":"2024-03-19T02:27:10.031294+0000","last_clean":"2024-03-19T02:27:10.031294+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:27:10.031294+0000","last_undegraded":"2024-03-19T02:27:10.031294+0000","last_fullsized":"2024-03-19T02:27:10.031294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784386+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:27:09.784386+0000","last_peered":"2024-03-19T02:27:09.784386+0000","last_clean":"2024-03-19T02:27:09.784386+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:27:09.784386+0000","last_undegraded":"2024-03-19T02:27:09.784386+0000","last_fullsized":"2024-03-19T02:27:09.784386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.746725+0000","last_change":"2024-03-19T02:27:09.746277+0000","last_active":"2024-03-19T02:27:09.746725+0000","last_peered":"2024-03-19T02:27:09.746725+0000","last_clean":"2024-03-19T02:27:09.746725+0000","last_became_active":"2024-03-19T02:27:09.737675+0000","last_became_peered":"2024-03-19T02:27:09.737675+0000","last_unstale":"2024-03-19T02:27:09.746725+0000","last_undegraded":"2024-03-19T02:27:09.746725+0000","last_fullsized":"2024-03-19T02:27:09.746725+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:10.829154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031180+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:27:10.031180+0000","last_peered":"2024-03-19T02:27:10.031180+0000","last_clean":"2024-03-19T02:27:10.031180+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:27:10.031180+0000","last_undegraded":"2024-03-19T02:27:10.031180+0000","last_fullsized":"2024-03-19T02:27:10.031180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784279+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:27:09.784279+0000","last_peered":"2024-03-19T02:27:09.784279+0000","last_clean":"2024-03-19T02:27:09.784279+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:27:09.784279+0000","last_undegraded":"2024-03-19T02:27:09.784279+0000","last_fullsized":"2024-03-19T02:27:09.784279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736489+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:27:09.736489+0000","last_peered":"2024-03-19T02:27:09.736489+0000","last_clean":"2024-03-19T02:27:09.736489+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:27:09.736489+0000","last_undegraded":"2024-03-19T02:27:09.736489+0000","last_fullsized":"2024-03-19T02:27:09.736489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780290+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:27:09.780290+0000","last_peered":"2024-03-19T02:27:09.780290+0000","last_clean":"2024-03-19T02:27:09.780290+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:27:09.780290+0000","last_undegraded":"2024-03-19T02:27:09.780290+0000","last_fullsized":"2024-03-19T02:27:09.780290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732749+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:27:09.732749+0000","last_peered":"2024-03-19T02:27:09.732749+0000","last_clean":"2024-03-19T02:27:09.732749+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:27:09.732749+0000","last_undegraded":"2024-03-19T02:27:09.732749+0000","last_fullsized":"2024-03-19T02:27:09.732749+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.806736+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:27:34.806736+0000","last_peered":"2024-03-19T02:27:34.806736+0000","last_clean":"2024-03-19T02:27:34.806736+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:27:34.806736+0000","last_undegraded":"2024-03-19T02:27:34.806736+0000","last_fullsized":"2024-03-19T02:27:34.806736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736376+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:27:09.736376+0000","last_peered":"2024-03-19T02:27:09.736376+0000","last_clean":"2024-03-19T02:27:09.736376+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:27:09.736376+0000","last_undegraded":"2024-03-19T02:27:09.736376+0000","last_fullsized":"2024-03-19T02:27:09.736376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732639+0000","last_change":"2024-03-19T02:26:08.850745+0000","last_active":"2024-03-19T02:27:09.732639+0000","last_peered":"2024-03-19T02:27:09.732639+0000","last_clean":"2024-03-19T02:27:09.732639+0000","last_became_active":"2024-03-19T02:26:08.850285+0000","last_became_peered":"2024-03-19T02:26:08.850285+0000","last_unstale":"2024-03-19T02:27:09.732639+0000","last_undegraded":"2024-03-19T02:27:09.732639+0000","last_fullsized":"2024-03-19T02:27:09.732639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:17.233944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732492+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:27:09.732492+0000","last_peered":"2024-03-19T02:27:09.732492+0000","last_clean":"2024-03-19T02:27:09.732492+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:27:09.732492+0000","last_undegraded":"2024-03-19T02:27:09.732492+0000","last_fullsized":"2024-03-19T02:27:09.732492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780207+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:27:09.780207+0000","last_peered":"2024-03-19T02:27:09.780207+0000","last_clean":"2024-03-19T02:27:09.780207+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:27:09.780207+0000","last_undegraded":"2024-03-19T02:27:09.780207+0000","last_fullsized":"2024-03-19T02:27:09.780207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734797+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:27:09.734797+0000","last_peered":"2024-03-19T02:27:09.734797+0000","last_clean":"2024-03-19T02:27:09.734797+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:27:09.734797+0000","last_undegraded":"2024-03-19T02:27:09.734797+0000","last_fullsized":"2024-03-19T02:27:09.734797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782652+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:27:09.782652+0000","last_peered":"2024-03-19T02:27:09.782652+0000","last_clean":"2024-03-19T02:27:09.782652+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:27:09.782652+0000","last_undegraded":"2024-03-19T02:27:09.782652+0000","last_fullsized":"2024-03-19T02:27:09.782652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732313+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:27:09.732313+0000","last_peered":"2024-03-19T02:27:09.732313+0000","last_clean":"2024-03-19T02:27:09.732313+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:27:09.732313+0000","last_undegraded":"2024-03-19T02:27:09.732313+0000","last_fullsized":"2024-03-19T02:27:09.732313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031213+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:27:10.031213+0000","last_peered":"2024-03-19T02:27:10.031213+0000","last_clean":"2024-03-19T02:27:10.031213+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:27:10.031213+0000","last_undegraded":"2024-03-19T02:27:10.031213+0000","last_fullsized":"2024-03-19T02:27:10.031213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779904+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:27:09.779904+0000","last_peered":"2024-03-19T02:27:09.779904+0000","last_clean":"2024-03-19T02:27:09.779904+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:27:09.779904+0000","last_undegraded":"2024-03-19T02:27:09.779904+0000","last_fullsized":"2024-03-19T02:27:09.779904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784266+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:27:09.784266+0000","last_peered":"2024-03-19T02:27:09.784266+0000","last_clean":"2024-03-19T02:27:09.784266+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:27:09.784266+0000","last_undegraded":"2024-03-19T02:27:09.784266+0000","last_fullsized":"2024-03-19T02:27:09.784266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732230+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:27:09.732230+0000","last_peered":"2024-03-19T02:27:09.732230+0000","last_clean":"2024-03-19T02:27:09.732230+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:27:09.732230+0000","last_undegraded":"2024-03-19T02:27:09.732230+0000","last_fullsized":"2024-03-19T02:27:09.732230+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.808510+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:27:32.808510+0000","last_peered":"2024-03-19T02:27:32.808510+0000","last_clean":"2024-03-19T02:27:32.808510+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:27:32.808510+0000","last_undegraded":"2024-03-19T02:27:32.808510+0000","last_fullsized":"2024-03-19T02:27:32.808510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031462+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:27:10.031462+0000","last_peered":"2024-03-19T02:27:10.031462+0000","last_clean":"2024-03-19T02:27:10.031462+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:27:10.031462+0000","last_undegraded":"2024-03-19T02:27:10.031462+0000","last_fullsized":"2024-03-19T02:27:10.031462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784117+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:27:09.784117+0000","last_peered":"2024-03-19T02:27:09.784117+0000","last_clean":"2024-03-19T02:27:09.784117+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:27:09.784117+0000","last_undegraded":"2024-03-19T02:27:09.784117+0000","last_fullsized":"2024-03-19T02:27:09.784117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736846+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:27:09.736846+0000","last_peered":"2024-03-19T02:27:09.736846+0000","last_clean":"2024-03-19T02:27:09.736846+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:27:09.736846+0000","last_undegraded":"2024-03-19T02:27:09.736846+0000","last_fullsized":"2024-03-19T02:27:09.736846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031299+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:27:10.031299+0000","last_peered":"2024-03-19T02:27:10.031299+0000","last_clean":"2024-03-19T02:27:10.031299+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:27:10.031299+0000","last_undegraded":"2024-03-19T02:27:10.031299+0000","last_fullsized":"2024-03-19T02:27:10.031299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784581+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:27:09.784581+0000","last_peered":"2024-03-19T02:27:09.784581+0000","last_clean":"2024-03-19T02:27:09.784581+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:27:09.784581+0000","last_undegraded":"2024-03-19T02:27:09.784581+0000","last_fullsized":"2024-03-19T02:27:09.784581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780340+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:27:09.780340+0000","last_peered":"2024-03-19T02:27:09.780340+0000","last_clean":"2024-03-19T02:27:09.780340+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:27:09.780340+0000","last_undegraded":"2024-03-19T02:27:09.780340+0000","last_fullsized":"2024-03-19T02:27:09.780340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031379+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:27:10.031379+0000","last_peered":"2024-03-19T02:27:10.031379+0000","last_clean":"2024-03-19T02:27:10.031379+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:27:10.031379+0000","last_undegraded":"2024-03-19T02:27:10.031379+0000","last_fullsized":"2024-03-19T02:27:10.031379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784258+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:27:09.784258+0000","last_peered":"2024-03-19T02:27:09.784258+0000","last_clean":"2024-03-19T02:27:09.784258+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:27:09.784258+0000","last_undegraded":"2024-03-19T02:27:09.784258+0000","last_fullsized":"2024-03-19T02:27:09.784258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":312,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.803913+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:27:32.803913+0000","last_peered":"2024-03-19T02:27:32.803913+0000","last_clean":"2024-03-19T02:27:32.803913+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:27:32.803913+0000","last_undegraded":"2024-03-19T02:27:32.803913+0000","last_fullsized":"2024-03-19T02:27:32.803913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62517,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":182,"num_write":40,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736974+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:27:09.736974+0000","last_peered":"2024-03-19T02:27:09.736974+0000","last_clean":"2024-03-19T02:27:09.736974+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:27:09.736974+0000","last_undegraded":"2024-03-19T02:27:09.736974+0000","last_fullsized":"2024-03-19T02:27:09.736974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780524+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:27:09.780524+0000","last_peered":"2024-03-19T02:27:09.780524+0000","last_clean":"2024-03-19T02:27:09.780524+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:27:09.780524+0000","last_undegraded":"2024-03-19T02:27:09.780524+0000","last_fullsized":"2024-03-19T02:27:09.780524+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732713+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:27:09.732713+0000","last_peered":"2024-03-19T02:27:09.732713+0000","last_clean":"2024-03-19T02:27:09.732713+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:27:09.732713+0000","last_undegraded":"2024-03-19T02:27:09.732713+0000","last_fullsized":"2024-03-19T02:27:09.732713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782845+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:27:09.782845+0000","last_peered":"2024-03-19T02:27:09.782845+0000","last_clean":"2024-03-19T02:27:09.782845+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:27:09.782845+0000","last_undegraded":"2024-03-19T02:27:09.782845+0000","last_fullsized":"2024-03-19T02:27:09.782845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734425+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:27:09.734425+0000","last_peered":"2024-03-19T02:27:09.734425+0000","last_clean":"2024-03-19T02:27:09.734425+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:27:09.734425+0000","last_undegraded":"2024-03-19T02:27:09.734425+0000","last_fullsized":"2024-03-19T02:27:09.734425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780119+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:27:09.780119+0000","last_peered":"2024-03-19T02:27:09.780119+0000","last_clean":"2024-03-19T02:27:09.780119+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:27:09.780119+0000","last_undegraded":"2024-03-19T02:27:09.780119+0000","last_fullsized":"2024-03-19T02:27:09.780119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736687+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:27:09.736687+0000","last_peered":"2024-03-19T02:27:09.736687+0000","last_clean":"2024-03-19T02:27:09.736687+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:27:09.736687+0000","last_undegraded":"2024-03-19T02:27:09.736687+0000","last_fullsized":"2024-03-19T02:27:09.736687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736910+0000","last_change":"2024-03-19T02:26:08.850986+0000","last_active":"2024-03-19T02:27:09.736910+0000","last_peered":"2024-03-19T02:27:09.736910+0000","last_clean":"2024-03-19T02:27:09.736910+0000","last_became_active":"2024-03-19T02:26:08.850619+0000","last_became_peered":"2024-03-19T02:26:08.850619+0000","last_unstale":"2024-03-19T02:27:09.736910+0000","last_undegraded":"2024-03-19T02:27:09.736910+0000","last_fullsized":"2024-03-19T02:27:09.736910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:15.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810904+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:27:33.810904+0000","last_peered":"2024-03-19T02:27:33.810904+0000","last_clean":"2024-03-19T02:27:33.810904+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:27:33.810904+0000","last_undegraded":"2024-03-19T02:27:33.810904+0000","last_fullsized":"2024-03-19T02:27:33.810904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736368+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:27:09.736368+0000","last_peered":"2024-03-19T02:27:09.736368+0000","last_clean":"2024-03-19T02:27:09.736368+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:27:09.736368+0000","last_undegraded":"2024-03-19T02:27:09.736368+0000","last_fullsized":"2024-03-19T02:27:09.736368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782900+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:27:09.782900+0000","last_peered":"2024-03-19T02:27:09.782900+0000","last_clean":"2024-03-19T02:27:09.782900+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:27:09.782900+0000","last_undegraded":"2024-03-19T02:27:09.782900+0000","last_fullsized":"2024-03-19T02:27:09.782900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736140+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:27:09.736140+0000","last_peered":"2024-03-19T02:27:09.736140+0000","last_clean":"2024-03-19T02:27:09.736140+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:27:09.736140+0000","last_undegraded":"2024-03-19T02:27:09.736140+0000","last_fullsized":"2024-03-19T02:27:09.736140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779991+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:27:09.779991+0000","last_peered":"2024-03-19T02:27:09.779991+0000","last_clean":"2024-03-19T02:27:09.779991+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:27:09.779991+0000","last_undegraded":"2024-03-19T02:27:09.779991+0000","last_fullsized":"2024-03-19T02:27:09.779991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779948+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:27:09.779948+0000","last_peered":"2024-03-19T02:27:09.779948+0000","last_clean":"2024-03-19T02:27:09.779948+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:27:09.779948+0000","last_undegraded":"2024-03-19T02:27:09.779948+0000","last_fullsized":"2024-03-19T02:27:09.779948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:21.805893+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:27:21.805893+0000","last_peered":"2024-03-19T02:27:21.805893+0000","last_clean":"2024-03-19T02:27:21.805893+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:27:21.805893+0000","last_undegraded":"2024-03-19T02:27:21.805893+0000","last_fullsized":"2024-03-19T02:27:21.805893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734692+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:27:09.734692+0000","last_peered":"2024-03-19T02:27:09.734692+0000","last_clean":"2024-03-19T02:27:09.734692+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:27:09.734692+0000","last_undegraded":"2024-03-19T02:27:09.734692+0000","last_fullsized":"2024-03-19T02:27:09.734692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736264+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:27:09.736264+0000","last_peered":"2024-03-19T02:27:09.736264+0000","last_clean":"2024-03-19T02:27:09.736264+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:27:09.736264+0000","last_undegraded":"2024-03-19T02:27:09.736264+0000","last_fullsized":"2024-03-19T02:27:09.736264+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784488+0000","last_change":"2024-03-19T02:26:08.851073+0000","last_active":"2024-03-19T02:27:09.784488+0000","last_peered":"2024-03-19T02:27:09.784488+0000","last_clean":"2024-03-19T02:27:09.784488+0000","last_became_active":"2024-03-19T02:26:08.850702+0000","last_became_peered":"2024-03-19T02:26:08.850702+0000","last_unstale":"2024-03-19T02:27:09.784488+0000","last_undegraded":"2024-03-19T02:27:09.784488+0000","last_fullsized":"2024-03-19T02:27:09.784488+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:11.668203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732945+0000","last_change":"2024-03-19T02:26:08.788232+0000","last_active":"2024-03-19T02:27:09.732945+0000","last_peered":"2024-03-19T02:27:09.732945+0000","last_clean":"2024-03-19T02:27:09.732945+0000","last_became_active":"2024-03-19T02:26:08.787993+0000","last_became_peered":"2024-03-19T02:26:08.787993+0000","last_unstale":"2024-03-19T02:27:09.732945+0000","last_undegraded":"2024-03-19T02:27:09.732945+0000","last_fullsized":"2024-03-19T02:27:09.732945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:40.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782988+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:27:09.782988+0000","last_peered":"2024-03-19T02:27:09.782988+0000","last_clean":"2024-03-19T02:27:09.782988+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:27:09.782988+0000","last_undegraded":"2024-03-19T02:27:09.782988+0000","last_fullsized":"2024-03-19T02:27:09.782988+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:35.812648+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:27:35.812648+0000","last_peered":"2024-03-19T02:27:35.812648+0000","last_clean":"2024-03-19T02:27:35.812648+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:27:35.812648+0000","last_undegraded":"2024-03-19T02:27:35.812648+0000","last_fullsized":"2024-03-19T02:27:35.812648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732734+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:27:09.732734+0000","last_peered":"2024-03-19T02:27:09.732734+0000","last_clean":"2024-03-19T02:27:09.732734+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:27:09.732734+0000","last_undegraded":"2024-03-19T02:27:09.732734+0000","last_fullsized":"2024-03-19T02:27:09.732734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782965+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:27:09.782965+0000","last_peered":"2024-03-19T02:27:09.782965+0000","last_clean":"2024-03-19T02:27:09.782965+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:27:09.782965+0000","last_undegraded":"2024-03-19T02:27:09.782965+0000","last_fullsized":"2024-03-19T02:27:09.782965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780052+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:27:09.780052+0000","last_peered":"2024-03-19T02:27:09.780052+0000","last_clean":"2024-03-19T02:27:09.780052+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:27:09.780052+0000","last_undegraded":"2024-03-19T02:27:09.780052+0000","last_fullsized":"2024-03-19T02:27:09.780052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.811532+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:27:34.811532+0000","last_peered":"2024-03-19T02:27:34.811532+0000","last_clean":"2024-03-19T02:27:34.811532+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:27:34.811532+0000","last_undegraded":"2024-03-19T02:27:34.811532+0000","last_fullsized":"2024-03-19T02:27:34.811532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736630+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:27:09.736630+0000","last_peered":"2024-03-19T02:27:09.736630+0000","last_clean":"2024-03-19T02:27:09.736630+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:27:09.736630+0000","last_undegraded":"2024-03-19T02:27:09.736630+0000","last_fullsized":"2024-03-19T02:27:09.736630+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732519+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:27:09.732519+0000","last_peered":"2024-03-19T02:27:09.732519+0000","last_clean":"2024-03-19T02:27:09.732519+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:27:09.732519+0000","last_undegraded":"2024-03-19T02:27:09.732519+0000","last_fullsized":"2024-03-19T02:27:09.732519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784470+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:27:09.784470+0000","last_peered":"2024-03-19T02:27:09.784470+0000","last_clean":"2024-03-19T02:27:09.784470+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:27:09.784470+0000","last_undegraded":"2024-03-19T02:27:09.784470+0000","last_fullsized":"2024-03-19T02:27:09.784470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736160+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:27:09.736160+0000","last_peered":"2024-03-19T02:27:09.736160+0000","last_clean":"2024-03-19T02:27:09.736160+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:27:09.736160+0000","last_undegraded":"2024-03-19T02:27:09.736160+0000","last_fullsized":"2024-03-19T02:27:09.736160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780528+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:27:09.780528+0000","last_peered":"2024-03-19T02:27:09.780528+0000","last_clean":"2024-03-19T02:27:09.780528+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:27:09.780528+0000","last_undegraded":"2024-03-19T02:27:09.780528+0000","last_fullsized":"2024-03-19T02:27:09.780528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782541+0000","last_change":"2024-03-19T02:26:08.788444+0000","last_active":"2024-03-19T02:27:09.782541+0000","last_peered":"2024-03-19T02:27:09.782541+0000","last_clean":"2024-03-19T02:27:09.782541+0000","last_became_active":"2024-03-19T02:26:08.788084+0000","last_became_peered":"2024-03-19T02:26:08.788084+0000","last_unstale":"2024-03-19T02:27:09.782541+0000","last_undegraded":"2024-03-19T02:27:09.782541+0000","last_fullsized":"2024-03-19T02:27:09.782541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:18.879498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736300+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:27:09.736300+0000","last_peered":"2024-03-19T02:27:09.736300+0000","last_clean":"2024-03-19T02:27:09.736300+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:27:09.736300+0000","last_undegraded":"2024-03-19T02:27:09.736300+0000","last_fullsized":"2024-03-19T02:27:09.736300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734318+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:27:09.734318+0000","last_peered":"2024-03-19T02:27:09.734318+0000","last_clean":"2024-03-19T02:27:09.734318+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:27:09.734318+0000","last_undegraded":"2024-03-19T02:27:09.734318+0000","last_fullsized":"2024-03-19T02:27:09.734318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.737005+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:27:09.737005+0000","last_peered":"2024-03-19T02:27:09.737005+0000","last_clean":"2024-03-19T02:27:09.737005+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:27:09.737005+0000","last_undegraded":"2024-03-19T02:27:09.737005+0000","last_fullsized":"2024-03-19T02:27:09.737005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783071+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:27:09.783071+0000","last_peered":"2024-03-19T02:27:09.783071+0000","last_clean":"2024-03-19T02:27:09.783071+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:27:09.783071+0000","last_undegraded":"2024-03-19T02:27:09.783071+0000","last_fullsized":"2024-03-19T02:27:09.783071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732209+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:27:09.732209+0000","last_peered":"2024-03-19T02:27:09.732209+0000","last_clean":"2024-03-19T02:27:09.732209+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:27:09.732209+0000","last_undegraded":"2024-03-19T02:27:09.732209+0000","last_fullsized":"2024-03-19T02:27:09.732209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.667708+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:27:33.667708+0000","last_peered":"2024-03-19T02:27:33.667708+0000","last_clean":"2024-03-19T02:27:33.667708+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:27:33.667708+0000","last_undegraded":"2024-03-19T02:27:33.667708+0000","last_fullsized":"2024-03-19T02:27:33.667708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732653+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:27:09.732653+0000","last_peered":"2024-03-19T02:27:09.732653+0000","last_clean":"2024-03-19T02:27:09.732653+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:27:09.732653+0000","last_undegraded":"2024-03-19T02:27:09.732653+0000","last_fullsized":"2024-03-19T02:27:09.732653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732378+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:27:09.732378+0000","last_peered":"2024-03-19T02:27:09.732378+0000","last_clean":"2024-03-19T02:27:09.732378+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:27:09.732378+0000","last_undegraded":"2024-03-19T02:27:09.732378+0000","last_fullsized":"2024-03-19T02:27:09.732378+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784698+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:27:09.784698+0000","last_peered":"2024-03-19T02:27:09.784698+0000","last_clean":"2024-03-19T02:27:09.784698+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:27:09.784698+0000","last_undegraded":"2024-03-19T02:27:09.784698+0000","last_fullsized":"2024-03-19T02:27:09.784698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031322+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:27:10.031322+0000","last_peered":"2024-03-19T02:27:10.031322+0000","last_clean":"2024-03-19T02:27:10.031322+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:27:10.031322+0000","last_undegraded":"2024-03-19T02:27:10.031322+0000","last_fullsized":"2024-03-19T02:27:10.031322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784756+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:27:09.784756+0000","last_peered":"2024-03-19T02:27:09.784756+0000","last_clean":"2024-03-19T02:27:09.784756+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:27:09.784756+0000","last_undegraded":"2024-03-19T02:27:09.784756+0000","last_fullsized":"2024-03-19T02:27:09.784756+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031094+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:27:10.031094+0000","last_peered":"2024-03-19T02:27:10.031094+0000","last_clean":"2024-03-19T02:27:10.031094+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:27:10.031094+0000","last_undegraded":"2024-03-19T02:27:10.031094+0000","last_fullsized":"2024-03-19T02:27:10.031094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736139+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:27:09.736139+0000","last_peered":"2024-03-19T02:27:09.736139+0000","last_clean":"2024-03-19T02:27:09.736139+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:27:09.736139+0000","last_undegraded":"2024-03-19T02:27:09.736139+0000","last_fullsized":"2024-03-19T02:27:09.736139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784635+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:27:09.784635+0000","last_peered":"2024-03-19T02:27:09.784635+0000","last_clean":"2024-03-19T02:27:09.784635+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:27:09.784635+0000","last_undegraded":"2024-03-19T02:27:09.784635+0000","last_fullsized":"2024-03-19T02:27:09.784635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732299+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:27:09.732299+0000","last_peered":"2024-03-19T02:27:09.732299+0000","last_clean":"2024-03-19T02:27:09.732299+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:27:09.732299+0000","last_undegraded":"2024-03-19T02:27:09.732299+0000","last_fullsized":"2024-03-19T02:27:09.732299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734210+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:27:09.734210+0000","last_peered":"2024-03-19T02:27:09.734210+0000","last_clean":"2024-03-19T02:27:09.734210+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:27:09.734210+0000","last_undegraded":"2024-03-19T02:27:09.734210+0000","last_fullsized":"2024-03-19T02:27:09.734210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784665+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:27:09.784665+0000","last_peered":"2024-03-19T02:27:09.784665+0000","last_clean":"2024-03-19T02:27:09.784665+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:27:09.784665+0000","last_undegraded":"2024-03-19T02:27:09.784665+0000","last_fullsized":"2024-03-19T02:27:09.784665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:31.716710+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:27:31.716710+0000","last_peered":"2024-03-19T02:27:31.716710+0000","last_clean":"2024-03-19T02:27:31.716710+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:27:31.716710+0000","last_undegraded":"2024-03-19T02:27:31.716710+0000","last_fullsized":"2024-03-19T02:27:31.716710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732367+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:27:09.732367+0000","last_peered":"2024-03-19T02:27:09.732367+0000","last_clean":"2024-03-19T02:27:09.732367+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:27:09.732367+0000","last_undegraded":"2024-03-19T02:27:09.732367+0000","last_fullsized":"2024-03-19T02:27:09.732367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784597+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:27:09.784597+0000","last_peered":"2024-03-19T02:27:09.784597+0000","last_clean":"2024-03-19T02:27:09.784597+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:27:09.784597+0000","last_undegraded":"2024-03-19T02:27:09.784597+0000","last_fullsized":"2024-03-19T02:27:09.784597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736496+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:27:09.736496+0000","last_peered":"2024-03-19T02:27:09.736496+0000","last_clean":"2024-03-19T02:27:09.736496+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:27:09.736496+0000","last_undegraded":"2024-03-19T02:27:09.736496+0000","last_fullsized":"2024-03-19T02:27:09.736496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736549+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:27:09.736549+0000","last_peered":"2024-03-19T02:27:09.736549+0000","last_clean":"2024-03-19T02:27:09.736549+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:27:09.736549+0000","last_undegraded":"2024-03-19T02:27:09.736549+0000","last_fullsized":"2024-03-19T02:27:09.736549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031127+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:27:10.031127+0000","last_peered":"2024-03-19T02:27:10.031127+0000","last_clean":"2024-03-19T02:27:10.031127+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:27:10.031127+0000","last_undegraded":"2024-03-19T02:27:10.031127+0000","last_fullsized":"2024-03-19T02:27:10.031127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810079+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:27:33.810079+0000","last_peered":"2024-03-19T02:27:33.810079+0000","last_clean":"2024-03-19T02:27:33.810079+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:27:33.810079+0000","last_undegraded":"2024-03-19T02:27:33.810079+0000","last_fullsized":"2024-03-19T02:27:33.810079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736802+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:27:09.736802+0000","last_peered":"2024-03-19T02:27:09.736802+0000","last_clean":"2024-03-19T02:27:09.736802+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:27:09.736802+0000","last_undegraded":"2024-03-19T02:27:09.736802+0000","last_fullsized":"2024-03-19T02:27:09.736802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031053+0000","last_change":"2024-03-19T02:26:08.849926+0000","last_active":"2024-03-19T02:27:10.031053+0000","last_peered":"2024-03-19T02:27:10.031053+0000","last_clean":"2024-03-19T02:27:10.031053+0000","last_became_active":"2024-03-19T02:26:08.849548+0000","last_became_peered":"2024-03-19T02:26:08.849548+0000","last_unstale":"2024-03-19T02:27:10.031053+0000","last_undegraded":"2024-03-19T02:27:10.031053+0000","last_fullsized":"2024-03-19T02:27:10.031053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:44:35.078313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780463+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:27:09.780463+0000","last_peered":"2024-03-19T02:27:09.780463+0000","last_clean":"2024-03-19T02:27:09.780463+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:27:09.780463+0000","last_undegraded":"2024-03-19T02:27:09.780463+0000","last_fullsized":"2024-03-19T02:27:09.780463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736591+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:27:09.736591+0000","last_peered":"2024-03-19T02:27:09.736591+0000","last_clean":"2024-03-19T02:27:09.736591+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:27:09.736591+0000","last_undegraded":"2024-03-19T02:27:09.736591+0000","last_fullsized":"2024-03-19T02:27:09.736591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736069+0000","last_change":"2024-03-19T02:26:08.850968+0000","last_active":"2024-03-19T02:27:09.736069+0000","last_peered":"2024-03-19T02:27:09.736069+0000","last_clean":"2024-03-19T02:27:09.736069+0000","last_became_active":"2024-03-19T02:26:08.850571+0000","last_became_peered":"2024-03-19T02:26:08.850571+0000","last_unstale":"2024-03-19T02:27:09.736069+0000","last_undegraded":"2024-03-19T02:27:09.736069+0000","last_fullsized":"2024-03-19T02:27:09.736069+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:29.532645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783058+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:27:09.783058+0000","last_peered":"2024-03-19T02:27:09.783058+0000","last_clean":"2024-03-19T02:27:09.783058+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:27:09.783058+0000","last_undegraded":"2024-03-19T02:27:09.783058+0000","last_fullsized":"2024-03-19T02:27:09.783058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736961+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:27:09.736961+0000","last_peered":"2024-03-19T02:27:09.736961+0000","last_clean":"2024-03-19T02:27:09.736961+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:27:09.736961+0000","last_undegraded":"2024-03-19T02:27:09.736961+0000","last_fullsized":"2024-03-19T02:27:09.736961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734276+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:27:09.734276+0000","last_peered":"2024-03-19T02:27:09.734276+0000","last_clean":"2024-03-19T02:27:09.734276+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:27:09.734276+0000","last_undegraded":"2024-03-19T02:27:09.734276+0000","last_fullsized":"2024-03-19T02:27:09.734276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732682+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:27:09.732682+0000","last_peered":"2024-03-19T02:27:09.732682+0000","last_clean":"2024-03-19T02:27:09.732682+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:27:09.732682+0000","last_undegraded":"2024-03-19T02:27:09.732682+0000","last_fullsized":"2024-03-19T02:27:09.732682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732568+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:27:09.732568+0000","last_peered":"2024-03-19T02:27:09.732568+0000","last_clean":"2024-03-19T02:27:09.732568+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:27:09.732568+0000","last_undegraded":"2024-03-19T02:27:09.732568+0000","last_fullsized":"2024-03-19T02:27:09.732568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031210+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:27:10.031210+0000","last_peered":"2024-03-19T02:27:10.031210+0000","last_clean":"2024-03-19T02:27:10.031210+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:27:10.031210+0000","last_undegraded":"2024-03-19T02:27:10.031210+0000","last_fullsized":"2024-03-19T02:27:10.031210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783084+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:27:09.783084+0000","last_peered":"2024-03-19T02:27:09.783084+0000","last_clean":"2024-03-19T02:27:09.783084+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:27:09.783084+0000","last_undegraded":"2024-03-19T02:27:09.783084+0000","last_fullsized":"2024-03-19T02:27:09.783084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734859+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:27:09.734859+0000","last_peered":"2024-03-19T02:27:09.734859+0000","last_clean":"2024-03-19T02:27:09.734859+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:27:09.734859+0000","last_undegraded":"2024-03-19T02:27:09.734859+0000","last_fullsized":"2024-03-19T02:27:09.734859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732975+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:27:09.732975+0000","last_peered":"2024-03-19T02:27:09.732975+0000","last_clean":"2024-03-19T02:27:09.732975+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:27:09.732975+0000","last_undegraded":"2024-03-19T02:27:09.732975+0000","last_fullsized":"2024-03-19T02:27:09.732975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732768+0000","last_change":"2024-03-19T02:26:08.790951+0000","last_active":"2024-03-19T02:27:09.732768+0000","last_peered":"2024-03-19T02:27:09.732768+0000","last_clean":"2024-03-19T02:27:09.732768+0000","last_became_active":"2024-03-19T02:26:08.790587+0000","last_became_peered":"2024-03-19T02:26:08.790587+0000","last_unstale":"2024-03-19T02:27:09.732768+0000","last_undegraded":"2024-03-19T02:27:09.732768+0000","last_fullsized":"2024-03-19T02:27:09.732768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:11.410269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779952+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:27:09.779952+0000","last_peered":"2024-03-19T02:27:09.779952+0000","last_clean":"2024-03-19T02:27:09.779952+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:27:09.779952+0000","last_undegraded":"2024-03-19T02:27:09.779952+0000","last_fullsized":"2024-03-19T02:27:09.779952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732410+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:27:09.732410+0000","last_peered":"2024-03-19T02:27:09.732410+0000","last_clean":"2024-03-19T02:27:09.732410+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:27:09.732410+0000","last_undegraded":"2024-03-19T02:27:09.732410+0000","last_fullsized":"2024-03-19T02:27:09.732410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031305+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:27:10.031305+0000","last_peered":"2024-03-19T02:27:10.031305+0000","last_clean":"2024-03-19T02:27:10.031305+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:27:10.031305+0000","last_undegraded":"2024-03-19T02:27:10.031305+0000","last_fullsized":"2024-03-19T02:27:10.031305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782801+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:27:09.782801+0000","last_peered":"2024-03-19T02:27:09.782801+0000","last_clean":"2024-03-19T02:27:09.782801+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:27:09.782801+0000","last_undegraded":"2024-03-19T02:27:09.782801+0000","last_fullsized":"2024-03-19T02:27:09.782801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734543+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:27:09.734543+0000","last_peered":"2024-03-19T02:27:09.734543+0000","last_clean":"2024-03-19T02:27:09.734543+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:27:09.734543+0000","last_undegraded":"2024-03-19T02:27:09.734543+0000","last_fullsized":"2024-03-19T02:27:09.734543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71302,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1225,"num_read_kb":687,"num_write":268,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":227186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1769472,"data_stored":1172997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":527197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":523479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299456,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1691648,"data_stored":1121455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":519098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708696,"statfs":{"total":95995035648,"available":95957704704,"internally_reserved":0,"allocated":1146880,"data_stored":589285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":520554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":590389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":518313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":523172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1699840,"data_stored":1116051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:27:37.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:27:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 -- 172.21.15.116:0/4233932980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 msgr2=0x7efc80109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 --2- 172.21.15.116:0/4233932980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80109c90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efc78009ca0 tx=0x7efc7802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 -- 172.21.15.116:0/4233932980 shutdown_connections 2024-03-19T02:27:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 --2- 172.21.15.116:0/4233932980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80109c90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 --2- 172.21.15.116:0/4233932980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 0x7efc80102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 --2- 172.21.15.116:0/4233932980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 0x7efc80101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 -- 172.21.15.116:0/4233932980 >> 172.21.15.116:0/4233932980 conn(0x7efc800fcdc0 msgr2=0x7efc800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 -- 172.21.15.116:0/4233932980 shutdown_connections 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.978+0000 7efc8735b640 1 -- 172.21.15.116:0/4233932980 wait complete. 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 Processor -- start 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 -- start start 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 0x7efc800755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 0x7efc80073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc8010d360 con 0x7efc80103400 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc8010d1e0 con 0x7efc80101850 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc8010d4e0 con 0x7efc80102a40 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58966/0 (socket says 172.21.15.116:58966) 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 -- 172.21.15.116:0/2610996343 learned_addr learned my addr 172.21.15.116:0/2610996343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc850f7640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 0x7efc800755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc848f6640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 0x7efc80073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 -- 172.21.15.116:0/2610996343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 msgr2=0x7efc80073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 0x7efc80073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 -- 172.21.15.116:0/2610996343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 msgr2=0x7efc800755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 0x7efc800755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 -- 172.21.15.116:0/2610996343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc800749e0 con 0x7efc80103400 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc858f8640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efc7802c700 tx=0x7efc780328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc78041020 con 0x7efc80103400 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc801a1af0 con 0x7efc80103400 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc78003e20 con 0x7efc80103400 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.979+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc78039040 con 0x7efc80103400 2024-03-19T02:27:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.980+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc801a2040 con 0x7efc80103400 2024-03-19T02:27:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.981+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc80108c10 con 0x7efc80103400 2024-03-19T02:27:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.982+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7efc78044020 con 0x7efc80103400 2024-03-19T02:27:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.982+0000 7efc767fc640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 0x7efc6c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.982+0000 7efc850f7640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 0x7efc6c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:37.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.982+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7efc780f0fd0 con 0x7efc80103400 2024-03-19T02:27:37.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.983+0000 7efc850f7640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 0x7efc6c0a4d50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc801028a0 tx=0x7efc7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:37.987+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc7804b020 con 0x7efc80103400 2024-03-19T02:27:38.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.157+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc80061c60 con 0x7efc6c0a2890 2024-03-19T02:27:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.166+0000 7efc767fc640 1 -- 172.21.15.116:0/2610996343 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7efc80061c60 con 0x7efc6c0a2890 2024-03-19T02:27:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/54690131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3635079170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1926924698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:27:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 msgr2=0x7efc6c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 0x7efc6c0a4d50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc801028a0 tx=0x7efc7c005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 msgr2=0x7efc80074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efc7802c700 tx=0x7efc780328a0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 shutdown_connections 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7efc6c0a2890 0x7efc6c0a4d50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc80103400 0x7efc80074150 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc80102a40 0x7efc80073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 --2- 172.21.15.116:0/2610996343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc80101850 0x7efc800755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 >> 172.21.15.116:0/2610996343 conn(0x7efc800fcdc0 msgr2=0x7efc80105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 shutdown_connections 2024-03-19T02:27:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.170+0000 7efc8735b640 1 -- 172.21.15.116:0/2610996343 wait complete. 2024-03-19T02:27:38.181 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/54690131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.181 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3635079170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.181 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1926924698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:38.182 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-03-19T02:27:37.636605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661904,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1361,"num_read_kb":806,"num_write":451,"num_write_kb":3097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372208,"kb_used_data":14728,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569616,"statfs":{"total":1151940427776,"available":1151559286784,"internally_reserved":0,"allocated":15081472,"data_stored":8245654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":61,"num_write":25,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735019+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:27:09.735019+0000","last_peered":"2024-03-19T02:27:09.735019+0000","last_clean":"2024-03-19T02:27:09.735019+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:27:09.735019+0000","last_undegraded":"2024-03-19T02:27:09.735019+0000","last_fullsized":"2024-03-19T02:27:09.735019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031274+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:27:10.031274+0000","last_peered":"2024-03-19T02:27:10.031274+0000","last_clean":"2024-03-19T02:27:10.031274+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:27:10.031274+0000","last_undegraded":"2024-03-19T02:27:10.031274+0000","last_fullsized":"2024-03-19T02:27:10.031274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030982+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:27:10.030982+0000","last_peered":"2024-03-19T02:27:10.030982+0000","last_clean":"2024-03-19T02:27:10.030982+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:27:10.030982+0000","last_undegraded":"2024-03-19T02:27:10.030982+0000","last_fullsized":"2024-03-19T02:27:10.030982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784827+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:27:09.784827+0000","last_peered":"2024-03-19T02:27:09.784827+0000","last_clean":"2024-03-19T02:27:09.784827+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:27:09.784827+0000","last_undegraded":"2024-03-19T02:27:09.784827+0000","last_fullsized":"2024-03-19T02:27:09.784827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736421+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:27:09.736421+0000","last_peered":"2024-03-19T02:27:09.736421+0000","last_clean":"2024-03-19T02:27:09.736421+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:27:09.736421+0000","last_undegraded":"2024-03-19T02:27:09.736421+0000","last_fullsized":"2024-03-19T02:27:09.736421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732549+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:27:09.732549+0000","last_peered":"2024-03-19T02:27:09.732549+0000","last_clean":"2024-03-19T02:27:09.732549+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:27:09.732549+0000","last_undegraded":"2024-03-19T02:27:09.732549+0000","last_fullsized":"2024-03-19T02:27:09.732549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736294+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:27:09.736294+0000","last_peered":"2024-03-19T02:27:09.736294+0000","last_clean":"2024-03-19T02:27:09.736294+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:27:09.736294+0000","last_undegraded":"2024-03-19T02:27:09.736294+0000","last_fullsized":"2024-03-19T02:27:09.736294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732129+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:27:09.732129+0000","last_peered":"2024-03-19T02:27:09.732129+0000","last_clean":"2024-03-19T02:27:09.732129+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:27:09.732129+0000","last_undegraded":"2024-03-19T02:27:09.732129+0000","last_fullsized":"2024-03-19T02:27:09.732129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031592+0000","last_change":"2024-03-19T02:26:08.850490+0000","last_active":"2024-03-19T02:27:10.031592+0000","last_peered":"2024-03-19T02:27:10.031592+0000","last_clean":"2024-03-19T02:27:10.031592+0000","last_became_active":"2024-03-19T02:26:08.850087+0000","last_became_peered":"2024-03-19T02:26:08.850087+0000","last_unstale":"2024-03-19T02:27:10.031592+0000","last_undegraded":"2024-03-19T02:27:10.031592+0000","last_fullsized":"2024-03-19T02:27:10.031592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732867+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:27:09.732867+0000","last_peered":"2024-03-19T02:27:09.732867+0000","last_clean":"2024-03-19T02:27:09.732867+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:27:09.732867+0000","last_undegraded":"2024-03-19T02:27:09.732867+0000","last_fullsized":"2024-03-19T02:27:09.732867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736754+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:27:09.736754+0000","last_peered":"2024-03-19T02:27:09.736754+0000","last_clean":"2024-03-19T02:27:09.736754+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:27:09.736754+0000","last_undegraded":"2024-03-19T02:27:09.736754+0000","last_fullsized":"2024-03-19T02:27:09.736754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784382+0000","last_change":"2024-03-19T02:26:08.789057+0000","last_active":"2024-03-19T02:27:09.784382+0000","last_peered":"2024-03-19T02:27:09.784382+0000","last_clean":"2024-03-19T02:27:09.784382+0000","last_became_active":"2024-03-19T02:26:08.788689+0000","last_became_peered":"2024-03-19T02:26:08.788689+0000","last_unstale":"2024-03-19T02:27:09.784382+0000","last_undegraded":"2024-03-19T02:27:09.784382+0000","last_fullsized":"2024-03-19T02:27:09.784382+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:08.295772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782778+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:27:09.782778+0000","last_peered":"2024-03-19T02:27:09.782778+0000","last_clean":"2024-03-19T02:27:09.782778+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:27:09.782778+0000","last_undegraded":"2024-03-19T02:27:09.782778+0000","last_fullsized":"2024-03-19T02:27:09.782778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030898+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:27:10.030898+0000","last_peered":"2024-03-19T02:27:10.030898+0000","last_clean":"2024-03-19T02:27:10.030898+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:27:10.030898+0000","last_undegraded":"2024-03-19T02:27:10.030898+0000","last_fullsized":"2024-03-19T02:27:10.030898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030955+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:27:10.030955+0000","last_peered":"2024-03-19T02:27:10.030955+0000","last_clean":"2024-03-19T02:27:10.030955+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:27:10.030955+0000","last_undegraded":"2024-03-19T02:27:10.030955+0000","last_fullsized":"2024-03-19T02:27:10.030955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031044+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:27:10.031044+0000","last_peered":"2024-03-19T02:27:10.031044+0000","last_clean":"2024-03-19T02:27:10.031044+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:27:10.031044+0000","last_undegraded":"2024-03-19T02:27:10.031044+0000","last_fullsized":"2024-03-19T02:27:10.031044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735240+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:27:09.735240+0000","last_peered":"2024-03-19T02:27:09.735240+0000","last_clean":"2024-03-19T02:27:09.735240+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:27:09.735240+0000","last_undegraded":"2024-03-19T02:27:09.735240+0000","last_fullsized":"2024-03-19T02:27:09.735240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784626+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:27:09.784626+0000","last_peered":"2024-03-19T02:27:09.784626+0000","last_clean":"2024-03-19T02:27:09.784626+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:27:09.784626+0000","last_undegraded":"2024-03-19T02:27:09.784626+0000","last_fullsized":"2024-03-19T02:27:09.784626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732894+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:27:09.732894+0000","last_peered":"2024-03-19T02:27:09.732894+0000","last_clean":"2024-03-19T02:27:09.732894+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:27:09.732894+0000","last_undegraded":"2024-03-19T02:27:09.732894+0000","last_fullsized":"2024-03-19T02:27:09.732894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732436+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:27:09.732436+0000","last_peered":"2024-03-19T02:27:09.732436+0000","last_clean":"2024-03-19T02:27:09.732436+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:27:09.732436+0000","last_undegraded":"2024-03-19T02:27:09.732436+0000","last_fullsized":"2024-03-19T02:27:09.732436+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.809123+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:27:32.809123+0000","last_peered":"2024-03-19T02:27:32.809123+0000","last_clean":"2024-03-19T02:27:32.809123+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:27:32.809123+0000","last_undegraded":"2024-03-19T02:27:32.809123+0000","last_fullsized":"2024-03-19T02:27:32.809123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732449+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:27:09.732449+0000","last_peered":"2024-03-19T02:27:09.732449+0000","last_clean":"2024-03-19T02:27:09.732449+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:27:09.732449+0000","last_undegraded":"2024-03-19T02:27:09.732449+0000","last_fullsized":"2024-03-19T02:27:09.732449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784240+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:27:09.784240+0000","last_peered":"2024-03-19T02:27:09.784240+0000","last_clean":"2024-03-19T02:27:09.784240+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:27:09.784240+0000","last_undegraded":"2024-03-19T02:27:09.784240+0000","last_fullsized":"2024-03-19T02:27:09.784240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732693+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:27:09.732693+0000","last_peered":"2024-03-19T02:27:09.732693+0000","last_clean":"2024-03-19T02:27:09.732693+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:27:09.732693+0000","last_undegraded":"2024-03-19T02:27:09.732693+0000","last_fullsized":"2024-03-19T02:27:09.732693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031294+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:27:10.031294+0000","last_peered":"2024-03-19T02:27:10.031294+0000","last_clean":"2024-03-19T02:27:10.031294+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:27:10.031294+0000","last_undegraded":"2024-03-19T02:27:10.031294+0000","last_fullsized":"2024-03-19T02:27:10.031294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784386+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:27:09.784386+0000","last_peered":"2024-03-19T02:27:09.784386+0000","last_clean":"2024-03-19T02:27:09.784386+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:27:09.784386+0000","last_undegraded":"2024-03-19T02:27:09.784386+0000","last_fullsized":"2024-03-19T02:27:09.784386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.746725+0000","last_change":"2024-03-19T02:27:09.746277+0000","last_active":"2024-03-19T02:27:09.746725+0000","last_peered":"2024-03-19T02:27:09.746725+0000","last_clean":"2024-03-19T02:27:09.746725+0000","last_became_active":"2024-03-19T02:27:09.737675+0000","last_became_peered":"2024-03-19T02:27:09.737675+0000","last_unstale":"2024-03-19T02:27:09.746725+0000","last_undegraded":"2024-03-19T02:27:09.746725+0000","last_fullsized":"2024-03-19T02:27:09.746725+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:10.829154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031180+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:27:10.031180+0000","last_peered":"2024-03-19T02:27:10.031180+0000","last_clean":"2024-03-19T02:27:10.031180+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:27:10.031180+0000","last_undegraded":"2024-03-19T02:27:10.031180+0000","last_fullsized":"2024-03-19T02:27:10.031180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784279+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:27:09.784279+0000","last_peered":"2024-03-19T02:27:09.784279+0000","last_clean":"2024-03-19T02:27:09.784279+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:27:09.784279+0000","last_undegraded":"2024-03-19T02:27:09.784279+0000","last_fullsized":"2024-03-19T02:27:09.784279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736489+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:27:09.736489+0000","last_peered":"2024-03-19T02:27:09.736489+0000","last_clean":"2024-03-19T02:27:09.736489+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:27:09.736489+0000","last_undegraded":"2024-03-19T02:27:09.736489+0000","last_fullsized":"2024-03-19T02:27:09.736489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780290+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:27:09.780290+0000","last_peered":"2024-03-19T02:27:09.780290+0000","last_clean":"2024-03-19T02:27:09.780290+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:27:09.780290+0000","last_undegraded":"2024-03-19T02:27:09.780290+0000","last_fullsized":"2024-03-19T02:27:09.780290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732749+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:27:09.732749+0000","last_peered":"2024-03-19T02:27:09.732749+0000","last_clean":"2024-03-19T02:27:09.732749+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:27:09.732749+0000","last_undegraded":"2024-03-19T02:27:09.732749+0000","last_fullsized":"2024-03-19T02:27:09.732749+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.806736+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:27:34.806736+0000","last_peered":"2024-03-19T02:27:34.806736+0000","last_clean":"2024-03-19T02:27:34.806736+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:27:34.806736+0000","last_undegraded":"2024-03-19T02:27:34.806736+0000","last_fullsized":"2024-03-19T02:27:34.806736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736376+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:27:09.736376+0000","last_peered":"2024-03-19T02:27:09.736376+0000","last_clean":"2024-03-19T02:27:09.736376+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:27:09.736376+0000","last_undegraded":"2024-03-19T02:27:09.736376+0000","last_fullsized":"2024-03-19T02:27:09.736376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732639+0000","last_change":"2024-03-19T02:26:08.850745+0000","last_active":"2024-03-19T02:27:09.732639+0000","last_peered":"2024-03-19T02:27:09.732639+0000","last_clean":"2024-03-19T02:27:09.732639+0000","last_became_active":"2024-03-19T02:26:08.850285+0000","last_became_peered":"2024-03-19T02:26:08.850285+0000","last_unstale":"2024-03-19T02:27:09.732639+0000","last_undegraded":"2024-03-19T02:27:09.732639+0000","last_fullsized":"2024-03-19T02:27:09.732639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:17.233944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732492+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:27:09.732492+0000","last_peered":"2024-03-19T02:27:09.732492+0000","last_clean":"2024-03-19T02:27:09.732492+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:27:09.732492+0000","last_undegraded":"2024-03-19T02:27:09.732492+0000","last_fullsized":"2024-03-19T02:27:09.732492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780207+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:27:09.780207+0000","last_peered":"2024-03-19T02:27:09.780207+0000","last_clean":"2024-03-19T02:27:09.780207+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:27:09.780207+0000","last_undegraded":"2024-03-19T02:27:09.780207+0000","last_fullsized":"2024-03-19T02:27:09.780207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734797+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:27:09.734797+0000","last_peered":"2024-03-19T02:27:09.734797+0000","last_clean":"2024-03-19T02:27:09.734797+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:27:09.734797+0000","last_undegraded":"2024-03-19T02:27:09.734797+0000","last_fullsized":"2024-03-19T02:27:09.734797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782652+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:27:09.782652+0000","last_peered":"2024-03-19T02:27:09.782652+0000","last_clean":"2024-03-19T02:27:09.782652+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:27:09.782652+0000","last_undegraded":"2024-03-19T02:27:09.782652+0000","last_fullsized":"2024-03-19T02:27:09.782652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732313+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:27:09.732313+0000","last_peered":"2024-03-19T02:27:09.732313+0000","last_clean":"2024-03-19T02:27:09.732313+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:27:09.732313+0000","last_undegraded":"2024-03-19T02:27:09.732313+0000","last_fullsized":"2024-03-19T02:27:09.732313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031213+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:27:10.031213+0000","last_peered":"2024-03-19T02:27:10.031213+0000","last_clean":"2024-03-19T02:27:10.031213+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:27:10.031213+0000","last_undegraded":"2024-03-19T02:27:10.031213+0000","last_fullsized":"2024-03-19T02:27:10.031213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779904+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:27:09.779904+0000","last_peered":"2024-03-19T02:27:09.779904+0000","last_clean":"2024-03-19T02:27:09.779904+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:27:09.779904+0000","last_undegraded":"2024-03-19T02:27:09.779904+0000","last_fullsized":"2024-03-19T02:27:09.779904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784266+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:27:09.784266+0000","last_peered":"2024-03-19T02:27:09.784266+0000","last_clean":"2024-03-19T02:27:09.784266+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:27:09.784266+0000","last_undegraded":"2024-03-19T02:27:09.784266+0000","last_fullsized":"2024-03-19T02:27:09.784266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732230+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:27:09.732230+0000","last_peered":"2024-03-19T02:27:09.732230+0000","last_clean":"2024-03-19T02:27:09.732230+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:27:09.732230+0000","last_undegraded":"2024-03-19T02:27:09.732230+0000","last_fullsized":"2024-03-19T02:27:09.732230+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.808510+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:27:32.808510+0000","last_peered":"2024-03-19T02:27:32.808510+0000","last_clean":"2024-03-19T02:27:32.808510+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:27:32.808510+0000","last_undegraded":"2024-03-19T02:27:32.808510+0000","last_fullsized":"2024-03-19T02:27:32.808510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031462+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:27:10.031462+0000","last_peered":"2024-03-19T02:27:10.031462+0000","last_clean":"2024-03-19T02:27:10.031462+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:27:10.031462+0000","last_undegraded":"2024-03-19T02:27:10.031462+0000","last_fullsized":"2024-03-19T02:27:10.031462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784117+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:27:09.784117+0000","last_peered":"2024-03-19T02:27:09.784117+0000","last_clean":"2024-03-19T02:27:09.784117+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:27:09.784117+0000","last_undegraded":"2024-03-19T02:27:09.784117+0000","last_fullsized":"2024-03-19T02:27:09.784117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736846+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:27:09.736846+0000","last_peered":"2024-03-19T02:27:09.736846+0000","last_clean":"2024-03-19T02:27:09.736846+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:27:09.736846+0000","last_undegraded":"2024-03-19T02:27:09.736846+0000","last_fullsized":"2024-03-19T02:27:09.736846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031299+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:27:10.031299+0000","last_peered":"2024-03-19T02:27:10.031299+0000","last_clean":"2024-03-19T02:27:10.031299+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:27:10.031299+0000","last_undegraded":"2024-03-19T02:27:10.031299+0000","last_fullsized":"2024-03-19T02:27:10.031299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784581+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:27:09.784581+0000","last_peered":"2024-03-19T02:27:09.784581+0000","last_clean":"2024-03-19T02:27:09.784581+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:27:09.784581+0000","last_undegraded":"2024-03-19T02:27:09.784581+0000","last_fullsized":"2024-03-19T02:27:09.784581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780340+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:27:09.780340+0000","last_peered":"2024-03-19T02:27:09.780340+0000","last_clean":"2024-03-19T02:27:09.780340+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:27:09.780340+0000","last_undegraded":"2024-03-19T02:27:09.780340+0000","last_fullsized":"2024-03-19T02:27:09.780340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031379+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:27:10.031379+0000","last_peered":"2024-03-19T02:27:10.031379+0000","last_clean":"2024-03-19T02:27:10.031379+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:27:10.031379+0000","last_undegraded":"2024-03-19T02:27:10.031379+0000","last_fullsized":"2024-03-19T02:27:10.031379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784258+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:27:09.784258+0000","last_peered":"2024-03-19T02:27:09.784258+0000","last_clean":"2024-03-19T02:27:09.784258+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:27:09.784258+0000","last_undegraded":"2024-03-19T02:27:09.784258+0000","last_fullsized":"2024-03-19T02:27:09.784258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":312,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.803913+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:27:32.803913+0000","last_peered":"2024-03-19T02:27:32.803913+0000","last_clean":"2024-03-19T02:27:32.803913+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:27:32.803913+0000","last_undegraded":"2024-03-19T02:27:32.803913+0000","last_fullsized":"2024-03-19T02:27:32.803913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62517,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":182,"num_write":40,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736974+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:27:09.736974+0000","last_peered":"2024-03-19T02:27:09.736974+0000","last_clean":"2024-03-19T02:27:09.736974+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:27:09.736974+0000","last_undegraded":"2024-03-19T02:27:09.736974+0000","last_fullsized":"2024-03-19T02:27:09.736974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780524+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:27:09.780524+0000","last_peered":"2024-03-19T02:27:09.780524+0000","last_clean":"2024-03-19T02:27:09.780524+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:27:09.780524+0000","last_undegraded":"2024-03-19T02:27:09.780524+0000","last_fullsized":"2024-03-19T02:27:09.780524+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732713+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:27:09.732713+0000","last_peered":"2024-03-19T02:27:09.732713+0000","last_clean":"2024-03-19T02:27:09.732713+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:27:09.732713+0000","last_undegraded":"2024-03-19T02:27:09.732713+0000","last_fullsized":"2024-03-19T02:27:09.732713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782845+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:27:09.782845+0000","last_peered":"2024-03-19T02:27:09.782845+0000","last_clean":"2024-03-19T02:27:09.782845+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:27:09.782845+0000","last_undegraded":"2024-03-19T02:27:09.782845+0000","last_fullsized":"2024-03-19T02:27:09.782845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734425+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:27:09.734425+0000","last_peered":"2024-03-19T02:27:09.734425+0000","last_clean":"2024-03-19T02:27:09.734425+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:27:09.734425+0000","last_undegraded":"2024-03-19T02:27:09.734425+0000","last_fullsized":"2024-03-19T02:27:09.734425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780119+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:27:09.780119+0000","last_peered":"2024-03-19T02:27:09.780119+0000","last_clean":"2024-03-19T02:27:09.780119+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:27:09.780119+0000","last_undegraded":"2024-03-19T02:27:09.780119+0000","last_fullsized":"2024-03-19T02:27:09.780119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736687+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:27:09.736687+0000","last_peered":"2024-03-19T02:27:09.736687+0000","last_clean":"2024-03-19T02:27:09.736687+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:27:09.736687+0000","last_undegraded":"2024-03-19T02:27:09.736687+0000","last_fullsized":"2024-03-19T02:27:09.736687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736910+0000","last_change":"2024-03-19T02:26:08.850986+0000","last_active":"2024-03-19T02:27:09.736910+0000","last_peered":"2024-03-19T02:27:09.736910+0000","last_clean":"2024-03-19T02:27:09.736910+0000","last_became_active":"2024-03-19T02:26:08.850619+0000","last_became_peered":"2024-03-19T02:26:08.850619+0000","last_unstale":"2024-03-19T02:27:09.736910+0000","last_undegraded":"2024-03-19T02:27:09.736910+0000","last_fullsized":"2024-03-19T02:27:09.736910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:15.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810904+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:27:33.810904+0000","last_peered":"2024-03-19T02:27:33.810904+0000","last_clean":"2024-03-19T02:27:33.810904+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:27:33.810904+0000","last_undegraded":"2024-03-19T02:27:33.810904+0000","last_fullsized":"2024-03-19T02:27:33.810904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736368+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:27:09.736368+0000","last_peered":"2024-03-19T02:27:09.736368+0000","last_clean":"2024-03-19T02:27:09.736368+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:27:09.736368+0000","last_undegraded":"2024-03-19T02:27:09.736368+0000","last_fullsized":"2024-03-19T02:27:09.736368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782900+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:27:09.782900+0000","last_peered":"2024-03-19T02:27:09.782900+0000","last_clean":"2024-03-19T02:27:09.782900+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:27:09.782900+0000","last_undegraded":"2024-03-19T02:27:09.782900+0000","last_fullsized":"2024-03-19T02:27:09.782900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736140+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:27:09.736140+0000","last_peered":"2024-03-19T02:27:09.736140+0000","last_clean":"2024-03-19T02:27:09.736140+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:27:09.736140+0000","last_undegraded":"2024-03-19T02:27:09.736140+0000","last_fullsized":"2024-03-19T02:27:09.736140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779991+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:27:09.779991+0000","last_peered":"2024-03-19T02:27:09.779991+0000","last_clean":"2024-03-19T02:27:09.779991+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:27:09.779991+0000","last_undegraded":"2024-03-19T02:27:09.779991+0000","last_fullsized":"2024-03-19T02:27:09.779991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779948+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:27:09.779948+0000","last_peered":"2024-03-19T02:27:09.779948+0000","last_clean":"2024-03-19T02:27:09.779948+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:27:09.779948+0000","last_undegraded":"2024-03-19T02:27:09.779948+0000","last_fullsized":"2024-03-19T02:27:09.779948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:21.805893+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:27:21.805893+0000","last_peered":"2024-03-19T02:27:21.805893+0000","last_clean":"2024-03-19T02:27:21.805893+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:27:21.805893+0000","last_undegraded":"2024-03-19T02:27:21.805893+0000","last_fullsized":"2024-03-19T02:27:21.805893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734692+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:27:09.734692+0000","last_peered":"2024-03-19T02:27:09.734692+0000","last_clean":"2024-03-19T02:27:09.734692+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:27:09.734692+0000","last_undegraded":"2024-03-19T02:27:09.734692+0000","last_fullsized":"2024-03-19T02:27:09.734692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736264+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:27:09.736264+0000","last_peered":"2024-03-19T02:27:09.736264+0000","last_clean":"2024-03-19T02:27:09.736264+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:27:09.736264+0000","last_undegraded":"2024-03-19T02:27:09.736264+0000","last_fullsized":"2024-03-19T02:27:09.736264+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784488+0000","last_change":"2024-03-19T02:26:08.851073+0000","last_active":"2024-03-19T02:27:09.784488+0000","last_peered":"2024-03-19T02:27:09.784488+0000","last_clean":"2024-03-19T02:27:09.784488+0000","last_became_active":"2024-03-19T02:26:08.850702+0000","last_became_peered":"2024-03-19T02:26:08.850702+0000","last_unstale":"2024-03-19T02:27:09.784488+0000","last_undegraded":"2024-03-19T02:27:09.784488+0000","last_fullsized":"2024-03-19T02:27:09.784488+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:11.668203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732945+0000","last_change":"2024-03-19T02:26:08.788232+0000","last_active":"2024-03-19T02:27:09.732945+0000","last_peered":"2024-03-19T02:27:09.732945+0000","last_clean":"2024-03-19T02:27:09.732945+0000","last_became_active":"2024-03-19T02:26:08.787993+0000","last_became_peered":"2024-03-19T02:26:08.787993+0000","last_unstale":"2024-03-19T02:27:09.732945+0000","last_undegraded":"2024-03-19T02:27:09.732945+0000","last_fullsized":"2024-03-19T02:27:09.732945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:40.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782988+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:27:09.782988+0000","last_peered":"2024-03-19T02:27:09.782988+0000","last_clean":"2024-03-19T02:27:09.782988+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:27:09.782988+0000","last_undegraded":"2024-03-19T02:27:09.782988+0000","last_fullsized":"2024-03-19T02:27:09.782988+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:35.812648+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:27:35.812648+0000","last_peered":"2024-03-19T02:27:35.812648+0000","last_clean":"2024-03-19T02:27:35.812648+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:27:35.812648+0000","last_undegraded":"2024-03-19T02:27:35.812648+0000","last_fullsized":"2024-03-19T02:27:35.812648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732734+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:27:09.732734+0000","last_peered":"2024-03-19T02:27:09.732734+0000","last_clean":"2024-03-19T02:27:09.732734+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:27:09.732734+0000","last_undegraded":"2024-03-19T02:27:09.732734+0000","last_fullsized":"2024-03-19T02:27:09.732734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782965+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:27:09.782965+0000","last_peered":"2024-03-19T02:27:09.782965+0000","last_clean":"2024-03-19T02:27:09.782965+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:27:09.782965+0000","last_undegraded":"2024-03-19T02:27:09.782965+0000","last_fullsized":"2024-03-19T02:27:09.782965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780052+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:27:09.780052+0000","last_peered":"2024-03-19T02:27:09.780052+0000","last_clean":"2024-03-19T02:27:09.780052+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:27:09.780052+0000","last_undegraded":"2024-03-19T02:27:09.780052+0000","last_fullsized":"2024-03-19T02:27:09.780052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.811532+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:27:34.811532+0000","last_peered":"2024-03-19T02:27:34.811532+0000","last_clean":"2024-03-19T02:27:34.811532+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:27:34.811532+0000","last_undegraded":"2024-03-19T02:27:34.811532+0000","last_fullsized":"2024-03-19T02:27:34.811532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736630+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:27:09.736630+0000","last_peered":"2024-03-19T02:27:09.736630+0000","last_clean":"2024-03-19T02:27:09.736630+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:27:09.736630+0000","last_undegraded":"2024-03-19T02:27:09.736630+0000","last_fullsized":"2024-03-19T02:27:09.736630+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732519+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:27:09.732519+0000","last_peered":"2024-03-19T02:27:09.732519+0000","last_clean":"2024-03-19T02:27:09.732519+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:27:09.732519+0000","last_undegraded":"2024-03-19T02:27:09.732519+0000","last_fullsized":"2024-03-19T02:27:09.732519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784470+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:27:09.784470+0000","last_peered":"2024-03-19T02:27:09.784470+0000","last_clean":"2024-03-19T02:27:09.784470+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:27:09.784470+0000","last_undegraded":"2024-03-19T02:27:09.784470+0000","last_fullsized":"2024-03-19T02:27:09.784470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736160+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:27:09.736160+0000","last_peered":"2024-03-19T02:27:09.736160+0000","last_clean":"2024-03-19T02:27:09.736160+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:27:09.736160+0000","last_undegraded":"2024-03-19T02:27:09.736160+0000","last_fullsized":"2024-03-19T02:27:09.736160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780528+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:27:09.780528+0000","last_peered":"2024-03-19T02:27:09.780528+0000","last_clean":"2024-03-19T02:27:09.780528+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:27:09.780528+0000","last_undegraded":"2024-03-19T02:27:09.780528+0000","last_fullsized":"2024-03-19T02:27:09.780528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782541+0000","last_change":"2024-03-19T02:26:08.788444+0000","last_active":"2024-03-19T02:27:09.782541+0000","last_peered":"2024-03-19T02:27:09.782541+0000","last_clean":"2024-03-19T02:27:09.782541+0000","last_became_active":"2024-03-19T02:26:08.788084+0000","last_became_peered":"2024-03-19T02:26:08.788084+0000","last_unstale":"2024-03-19T02:27:09.782541+0000","last_undegraded":"2024-03-19T02:27:09.782541+0000","last_fullsized":"2024-03-19T02:27:09.782541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:18.879498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736300+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:27:09.736300+0000","last_peered":"2024-03-19T02:27:09.736300+0000","last_clean":"2024-03-19T02:27:09.736300+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:27:09.736300+0000","last_undegraded":"2024-03-19T02:27:09.736300+0000","last_fullsized":"2024-03-19T02:27:09.736300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734318+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:27:09.734318+0000","last_peered":"2024-03-19T02:27:09.734318+0000","last_clean":"2024-03-19T02:27:09.734318+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:27:09.734318+0000","last_undegraded":"2024-03-19T02:27:09.734318+0000","last_fullsized":"2024-03-19T02:27:09.734318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.737005+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:27:09.737005+0000","last_peered":"2024-03-19T02:27:09.737005+0000","last_clean":"2024-03-19T02:27:09.737005+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:27:09.737005+0000","last_undegraded":"2024-03-19T02:27:09.737005+0000","last_fullsized":"2024-03-19T02:27:09.737005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783071+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:27:09.783071+0000","last_peered":"2024-03-19T02:27:09.783071+0000","last_clean":"2024-03-19T02:27:09.783071+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:27:09.783071+0000","last_undegraded":"2024-03-19T02:27:09.783071+0000","last_fullsized":"2024-03-19T02:27:09.783071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732209+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:27:09.732209+0000","last_peered":"2024-03-19T02:27:09.732209+0000","last_clean":"2024-03-19T02:27:09.732209+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:27:09.732209+0000","last_undegraded":"2024-03-19T02:27:09.732209+0000","last_fullsized":"2024-03-19T02:27:09.732209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.667708+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:27:33.667708+0000","last_peered":"2024-03-19T02:27:33.667708+0000","last_clean":"2024-03-19T02:27:33.667708+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:27:33.667708+0000","last_undegraded":"2024-03-19T02:27:33.667708+0000","last_fullsized":"2024-03-19T02:27:33.667708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732653+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:27:09.732653+0000","last_peered":"2024-03-19T02:27:09.732653+0000","last_clean":"2024-03-19T02:27:09.732653+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:27:09.732653+0000","last_undegraded":"2024-03-19T02:27:09.732653+0000","last_fullsized":"2024-03-19T02:27:09.732653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732378+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:27:09.732378+0000","last_peered":"2024-03-19T02:27:09.732378+0000","last_clean":"2024-03-19T02:27:09.732378+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:27:09.732378+0000","last_undegraded":"2024-03-19T02:27:09.732378+0000","last_fullsized":"2024-03-19T02:27:09.732378+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784698+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:27:09.784698+0000","last_peered":"2024-03-19T02:27:09.784698+0000","last_clean":"2024-03-19T02:27:09.784698+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:27:09.784698+0000","last_undegraded":"2024-03-19T02:27:09.784698+0000","last_fullsized":"2024-03-19T02:27:09.784698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031322+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:27:10.031322+0000","last_peered":"2024-03-19T02:27:10.031322+0000","last_clean":"2024-03-19T02:27:10.031322+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:27:10.031322+0000","last_undegraded":"2024-03-19T02:27:10.031322+0000","last_fullsized":"2024-03-19T02:27:10.031322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784756+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:27:09.784756+0000","last_peered":"2024-03-19T02:27:09.784756+0000","last_clean":"2024-03-19T02:27:09.784756+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:27:09.784756+0000","last_undegraded":"2024-03-19T02:27:09.784756+0000","last_fullsized":"2024-03-19T02:27:09.784756+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031094+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:27:10.031094+0000","last_peered":"2024-03-19T02:27:10.031094+0000","last_clean":"2024-03-19T02:27:10.031094+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:27:10.031094+0000","last_undegraded":"2024-03-19T02:27:10.031094+0000","last_fullsized":"2024-03-19T02:27:10.031094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736139+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:27:09.736139+0000","last_peered":"2024-03-19T02:27:09.736139+0000","last_clean":"2024-03-19T02:27:09.736139+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:27:09.736139+0000","last_undegraded":"2024-03-19T02:27:09.736139+0000","last_fullsized":"2024-03-19T02:27:09.736139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784635+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:27:09.784635+0000","last_peered":"2024-03-19T02:27:09.784635+0000","last_clean":"2024-03-19T02:27:09.784635+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:27:09.784635+0000","last_undegraded":"2024-03-19T02:27:09.784635+0000","last_fullsized":"2024-03-19T02:27:09.784635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732299+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:27:09.732299+0000","last_peered":"2024-03-19T02:27:09.732299+0000","last_clean":"2024-03-19T02:27:09.732299+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:27:09.732299+0000","last_undegraded":"2024-03-19T02:27:09.732299+0000","last_fullsized":"2024-03-19T02:27:09.732299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734210+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:27:09.734210+0000","last_peered":"2024-03-19T02:27:09.734210+0000","last_clean":"2024-03-19T02:27:09.734210+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:27:09.734210+0000","last_undegraded":"2024-03-19T02:27:09.734210+0000","last_fullsized":"2024-03-19T02:27:09.734210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784665+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:27:09.784665+0000","last_peered":"2024-03-19T02:27:09.784665+0000","last_clean":"2024-03-19T02:27:09.784665+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:27:09.784665+0000","last_undegraded":"2024-03-19T02:27:09.784665+0000","last_fullsized":"2024-03-19T02:27:09.784665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:31.716710+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:27:31.716710+0000","last_peered":"2024-03-19T02:27:31.716710+0000","last_clean":"2024-03-19T02:27:31.716710+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:27:31.716710+0000","last_undegraded":"2024-03-19T02:27:31.716710+0000","last_fullsized":"2024-03-19T02:27:31.716710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732367+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:27:09.732367+0000","last_peered":"2024-03-19T02:27:09.732367+0000","last_clean":"2024-03-19T02:27:09.732367+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:27:09.732367+0000","last_undegraded":"2024-03-19T02:27:09.732367+0000","last_fullsized":"2024-03-19T02:27:09.732367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784597+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:27:09.784597+0000","last_peered":"2024-03-19T02:27:09.784597+0000","last_clean":"2024-03-19T02:27:09.784597+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:27:09.784597+0000","last_undegraded":"2024-03-19T02:27:09.784597+0000","last_fullsized":"2024-03-19T02:27:09.784597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736496+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:27:09.736496+0000","last_peered":"2024-03-19T02:27:09.736496+0000","last_clean":"2024-03-19T02:27:09.736496+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:27:09.736496+0000","last_undegraded":"2024-03-19T02:27:09.736496+0000","last_fullsized":"2024-03-19T02:27:09.736496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736549+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:27:09.736549+0000","last_peered":"2024-03-19T02:27:09.736549+0000","last_clean":"2024-03-19T02:27:09.736549+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:27:09.736549+0000","last_undegraded":"2024-03-19T02:27:09.736549+0000","last_fullsized":"2024-03-19T02:27:09.736549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031127+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:27:10.031127+0000","last_peered":"2024-03-19T02:27:10.031127+0000","last_clean":"2024-03-19T02:27:10.031127+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:27:10.031127+0000","last_undegraded":"2024-03-19T02:27:10.031127+0000","last_fullsized":"2024-03-19T02:27:10.031127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810079+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:27:33.810079+0000","last_peered":"2024-03-19T02:27:33.810079+0000","last_clean":"2024-03-19T02:27:33.810079+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:27:33.810079+0000","last_undegraded":"2024-03-19T02:27:33.810079+0000","last_fullsized":"2024-03-19T02:27:33.810079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736802+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:27:09.736802+0000","last_peered":"2024-03-19T02:27:09.736802+0000","last_clean":"2024-03-19T02:27:09.736802+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:27:09.736802+0000","last_undegraded":"2024-03-19T02:27:09.736802+0000","last_fullsized":"2024-03-19T02:27:09.736802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031053+0000","last_change":"2024-03-19T02:26:08.849926+0000","last_active":"2024-03-19T02:27:10.031053+0000","last_peered":"2024-03-19T02:27:10.031053+0000","last_clean":"2024-03-19T02:27:10.031053+0000","last_became_active":"2024-03-19T02:26:08.849548+0000","last_became_peered":"2024-03-19T02:26:08.849548+0000","last_unstale":"2024-03-19T02:27:10.031053+0000","last_undegraded":"2024-03-19T02:27:10.031053+0000","last_fullsized":"2024-03-19T02:27:10.031053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:44:35.078313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780463+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:27:09.780463+0000","last_peered":"2024-03-19T02:27:09.780463+0000","last_clean":"2024-03-19T02:27:09.780463+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:27:09.780463+0000","last_undegraded":"2024-03-19T02:27:09.780463+0000","last_fullsized":"2024-03-19T02:27:09.780463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736591+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:27:09.736591+0000","last_peered":"2024-03-19T02:27:09.736591+0000","last_clean":"2024-03-19T02:27:09.736591+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:27:09.736591+0000","last_undegraded":"2024-03-19T02:27:09.736591+0000","last_fullsized":"2024-03-19T02:27:09.736591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736069+0000","last_change":"2024-03-19T02:26:08.850968+0000","last_active":"2024-03-19T02:27:09.736069+0000","last_peered":"2024-03-19T02:27:09.736069+0000","last_clean":"2024-03-19T02:27:09.736069+0000","last_became_active":"2024-03-19T02:26:08.850571+0000","last_became_peered":"2024-03-19T02:26:08.850571+0000","last_unstale":"2024-03-19T02:27:09.736069+0000","last_undegraded":"2024-03-19T02:27:09.736069+0000","last_fullsized":"2024-03-19T02:27:09.736069+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:29.532645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783058+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:27:09.783058+0000","last_peered":"2024-03-19T02:27:09.783058+0000","last_clean":"2024-03-19T02:27:09.783058+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:27:09.783058+0000","last_undegraded":"2024-03-19T02:27:09.783058+0000","last_fullsized":"2024-03-19T02:27:09.783058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736961+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:27:09.736961+0000","last_peered":"2024-03-19T02:27:09.736961+0000","last_clean":"2024-03-19T02:27:09.736961+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:27:09.736961+0000","last_undegraded":"2024-03-19T02:27:09.736961+0000","last_fullsized":"2024-03-19T02:27:09.736961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734276+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:27:09.734276+0000","last_peered":"2024-03-19T02:27:09.734276+0000","last_clean":"2024-03-19T02:27:09.734276+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:27:09.734276+0000","last_undegraded":"2024-03-19T02:27:09.734276+0000","last_fullsized":"2024-03-19T02:27:09.734276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732682+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:27:09.732682+0000","last_peered":"2024-03-19T02:27:09.732682+0000","last_clean":"2024-03-19T02:27:09.732682+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:27:09.732682+0000","last_undegraded":"2024-03-19T02:27:09.732682+0000","last_fullsized":"2024-03-19T02:27:09.732682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732568+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:27:09.732568+0000","last_peered":"2024-03-19T02:27:09.732568+0000","last_clean":"2024-03-19T02:27:09.732568+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:27:09.732568+0000","last_undegraded":"2024-03-19T02:27:09.732568+0000","last_fullsized":"2024-03-19T02:27:09.732568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031210+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:27:10.031210+0000","last_peered":"2024-03-19T02:27:10.031210+0000","last_clean":"2024-03-19T02:27:10.031210+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:27:10.031210+0000","last_undegraded":"2024-03-19T02:27:10.031210+0000","last_fullsized":"2024-03-19T02:27:10.031210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783084+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:27:09.783084+0000","last_peered":"2024-03-19T02:27:09.783084+0000","last_clean":"2024-03-19T02:27:09.783084+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:27:09.783084+0000","last_undegraded":"2024-03-19T02:27:09.783084+0000","last_fullsized":"2024-03-19T02:27:09.783084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734859+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:27:09.734859+0000","last_peered":"2024-03-19T02:27:09.734859+0000","last_clean":"2024-03-19T02:27:09.734859+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:27:09.734859+0000","last_undegraded":"2024-03-19T02:27:09.734859+0000","last_fullsized":"2024-03-19T02:27:09.734859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732975+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:27:09.732975+0000","last_peered":"2024-03-19T02:27:09.732975+0000","last_clean":"2024-03-19T02:27:09.732975+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:27:09.732975+0000","last_undegraded":"2024-03-19T02:27:09.732975+0000","last_fullsized":"2024-03-19T02:27:09.732975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732768+0000","last_change":"2024-03-19T02:26:08.790951+0000","last_active":"2024-03-19T02:27:09.732768+0000","last_peered":"2024-03-19T02:27:09.732768+0000","last_clean":"2024-03-19T02:27:09.732768+0000","last_became_active":"2024-03-19T02:26:08.790587+0000","last_became_peered":"2024-03-19T02:26:08.790587+0000","last_unstale":"2024-03-19T02:27:09.732768+0000","last_undegraded":"2024-03-19T02:27:09.732768+0000","last_fullsized":"2024-03-19T02:27:09.732768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:11.410269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779952+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:27:09.779952+0000","last_peered":"2024-03-19T02:27:09.779952+0000","last_clean":"2024-03-19T02:27:09.779952+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:27:09.779952+0000","last_undegraded":"2024-03-19T02:27:09.779952+0000","last_fullsized":"2024-03-19T02:27:09.779952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732410+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:27:09.732410+0000","last_peered":"2024-03-19T02:27:09.732410+0000","last_clean":"2024-03-19T02:27:09.732410+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:27:09.732410+0000","last_undegraded":"2024-03-19T02:27:09.732410+0000","last_fullsized":"2024-03-19T02:27:09.732410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031305+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:27:10.031305+0000","last_peered":"2024-03-19T02:27:10.031305+0000","last_clean":"2024-03-19T02:27:10.031305+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:27:10.031305+0000","last_undegraded":"2024-03-19T02:27:10.031305+0000","last_fullsized":"2024-03-19T02:27:10.031305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782801+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:27:09.782801+0000","last_peered":"2024-03-19T02:27:09.782801+0000","last_clean":"2024-03-19T02:27:09.782801+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:27:09.782801+0000","last_undegraded":"2024-03-19T02:27:09.782801+0000","last_fullsized":"2024-03-19T02:27:09.782801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734543+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:27:09.734543+0000","last_peered":"2024-03-19T02:27:09.734543+0000","last_clean":"2024-03-19T02:27:09.734543+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:27:09.734543+0000","last_undegraded":"2024-03-19T02:27:09.734543+0000","last_fullsized":"2024-03-19T02:27:09.734543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71302,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1225,"num_read_kb":687,"num_write":268,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":227186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1769472,"data_stored":1172997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":527197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":523479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299456,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1691648,"data_stored":1121455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":519098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708696,"statfs":{"total":95995035648,"available":95957704704,"internally_reserved":0,"allocated":1146880,"data_stored":589285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":520554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":590389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":518313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":523172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1699840,"data_stored":1116051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:27:38.186 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-19T02:27:38.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:27:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/54690131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3635079170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1926924698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 -- 172.21.15.116:0/1750332659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578101930 msgr2=0x7f4578101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 --2- 172.21.15.116:0/1750332659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578101930 0x7f4578101db0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f456c009900 tx=0x7f456c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 -- 172.21.15.116:0/1750332659 shutdown_connections 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 --2- 172.21.15.116:0/1750332659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45781022f0 0x7f4578108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 --2- 172.21.15.116:0/1750332659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578101930 0x7f4578101db0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 --2- 172.21.15.116:0/1750332659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.323+0000 7f45803cd640 1 -- 172.21.15.116:0/1750332659 >> 172.21.15.116:0/1750332659 conn(0x7f45780fbcb0 msgr2=0x7f45780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- 172.21.15.116:0/1750332659 shutdown_connections 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- 172.21.15.116:0/1750332659 wait complete. 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 Processor -- start 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- start start 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4578101930 0x7f4578197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f457810d390 con 0x7f45781022f0 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f457810d210 con 0x7f4578100740 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f45803cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f457810d510 con 0x7f4578101930 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457e96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457e169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457e96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58978/0 (socket says 172.21.15.116:58978) 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457e169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41708/0 (socket says 172.21.15.116:41708) 2024-03-19T02:27:38.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457d968640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4578101930 0x7f4578197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.324+0000 7f457e96a640 1 -- 172.21.15.116:0/2616821131 learned_addr learned my addr 172.21.15.116:0/2616821131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 -- 172.21.15.116:0/2616821131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4578101930 msgr2=0x7f4578197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4578101930 0x7f4578197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 -- 172.21.15.116:0/2616821131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 msgr2=0x7f4578197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 -- 172.21.15.116:0/2616821131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457819f670 con 0x7f45781022f0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f457e96a640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f457000eb10 tx=0x7f457000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570004b20 con 0x7f45781022f0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4570004cc0 con 0x7f45781022f0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457819f960 con 0x7f45781022f0 2024-03-19T02:27:38.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570010690 con 0x7f45781022f0 2024-03-19T02:27:38.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.325+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457819fe30 con 0x7f45781022f0 2024-03-19T02:27:38.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.326+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4578107ae0 con 0x7f45781022f0 2024-03-19T02:27:38.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.327+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f4570005020 con 0x7f45781022f0 2024-03-19T02:27:38.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.328+0000 7f45777fe640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 0x7f45600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.328+0000 7f457e169640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 0x7f45600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.328+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f45700cf500 con 0x7f45781022f0 2024-03-19T02:27:38.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.328+0000 7f457e169640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 0x7f45600a4d50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45680095f0 tx=0x7f4568005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.332+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f457009c0c0 con 0x7f45781022f0 2024-03-19T02:27:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.506+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4578061c60 con 0x7f45600a2890 2024-03-19T02:27:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.515+0000 7f45777fe640 1 -- 172.21.15.116:0/2616821131 <== mgr.14152 v2:172.21.15.116:6800/3573130182 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7f4578061c60 con 0x7f45600a2890 2024-03-19T02:27:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:38.521 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T02:27:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 msgr2=0x7f45600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 0x7f45600a4d50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45680095f0 tx=0x7f4568005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 msgr2=0x7f457819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f457000eb10 tx=0x7f457000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 shutdown_connections 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f45600a2890 0x7f45600a4d50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45781022f0 0x7f457819f000 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4578101930 0x7f4578197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 --2- 172.21.15.116:0/2616821131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4578100740 0x7f4578197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 >> 172.21.15.116:0/2616821131 conn(0x7f45780fbcb0 msgr2=0x7f4578104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 shutdown_connections 2024-03-19T02:27:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.519+0000 7f45803cd640 1 -- 172.21.15.116:0/2616821131 wait complete. 2024-03-19T02:27:38.530 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-03-19T02:27:37.636605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661904,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1361,"num_read_kb":806,"num_write":451,"num_write_kb":3097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372208,"kb_used_data":14728,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569616,"statfs":{"total":1151940427776,"available":1151559286784,"internally_reserved":0,"allocated":15081472,"data_stored":8245654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":61,"num_write":25,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735019+0000","last_change":"2024-03-19T02:25:08.207912+0000","last_active":"2024-03-19T02:27:09.735019+0000","last_peered":"2024-03-19T02:27:09.735019+0000","last_clean":"2024-03-19T02:27:09.735019+0000","last_became_active":"2024-03-19T02:25:08.207140+0000","last_became_peered":"2024-03-19T02:25:08.207140+0000","last_unstale":"2024-03-19T02:27:09.735019+0000","last_undegraded":"2024-03-19T02:27:09.735019+0000","last_fullsized":"2024-03-19T02:27:09.735019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:53.266705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031274+0000","last_change":"2024-03-19T02:25:08.210303+0000","last_active":"2024-03-19T02:27:10.031274+0000","last_peered":"2024-03-19T02:27:10.031274+0000","last_clean":"2024-03-19T02:27:10.031274+0000","last_became_active":"2024-03-19T02:25:08.209987+0000","last_became_peered":"2024-03-19T02:25:08.209987+0000","last_unstale":"2024-03-19T02:27:10.031274+0000","last_undegraded":"2024-03-19T02:27:10.031274+0000","last_fullsized":"2024-03-19T02:27:10.031274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:37.995119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030982+0000","last_change":"2024-03-19T02:25:08.210532+0000","last_active":"2024-03-19T02:27:10.030982+0000","last_peered":"2024-03-19T02:27:10.030982+0000","last_clean":"2024-03-19T02:27:10.030982+0000","last_became_active":"2024-03-19T02:25:08.210191+0000","last_became_peered":"2024-03-19T02:25:08.210191+0000","last_unstale":"2024-03-19T02:27:10.030982+0000","last_undegraded":"2024-03-19T02:27:10.030982+0000","last_fullsized":"2024-03-19T02:27:10.030982+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:22.713770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784827+0000","last_change":"2024-03-19T02:25:08.210420+0000","last_active":"2024-03-19T02:27:09.784827+0000","last_peered":"2024-03-19T02:27:09.784827+0000","last_clean":"2024-03-19T02:27:09.784827+0000","last_became_active":"2024-03-19T02:25:08.210112+0000","last_became_peered":"2024-03-19T02:25:08.210112+0000","last_unstale":"2024-03-19T02:27:09.784827+0000","last_undegraded":"2024-03-19T02:27:09.784827+0000","last_fullsized":"2024-03-19T02:27:09.784827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:53.706309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736421+0000","last_change":"2024-03-19T02:25:08.206048+0000","last_active":"2024-03-19T02:27:09.736421+0000","last_peered":"2024-03-19T02:27:09.736421+0000","last_clean":"2024-03-19T02:27:09.736421+0000","last_became_active":"2024-03-19T02:25:08.205747+0000","last_became_peered":"2024-03-19T02:25:08.205747+0000","last_unstale":"2024-03-19T02:27:09.736421+0000","last_undegraded":"2024-03-19T02:27:09.736421+0000","last_fullsized":"2024-03-19T02:27:09.736421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:17.685657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732549+0000","last_change":"2024-03-19T02:25:08.211338+0000","last_active":"2024-03-19T02:27:09.732549+0000","last_peered":"2024-03-19T02:27:09.732549+0000","last_clean":"2024-03-19T02:27:09.732549+0000","last_became_active":"2024-03-19T02:25:08.210980+0000","last_became_peered":"2024-03-19T02:25:08.210980+0000","last_unstale":"2024-03-19T02:27:09.732549+0000","last_undegraded":"2024-03-19T02:27:09.732549+0000","last_fullsized":"2024-03-19T02:27:09.732549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:05.592536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736294+0000","last_change":"2024-03-19T02:25:08.210864+0000","last_active":"2024-03-19T02:27:09.736294+0000","last_peered":"2024-03-19T02:27:09.736294+0000","last_clean":"2024-03-19T02:27:09.736294+0000","last_became_active":"2024-03-19T02:25:08.210568+0000","last_became_peered":"2024-03-19T02:25:08.210568+0000","last_unstale":"2024-03-19T02:27:09.736294+0000","last_undegraded":"2024-03-19T02:27:09.736294+0000","last_fullsized":"2024-03-19T02:27:09.736294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:11.964245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732129+0000","last_change":"2024-03-19T02:25:08.212603+0000","last_active":"2024-03-19T02:27:09.732129+0000","last_peered":"2024-03-19T02:27:09.732129+0000","last_clean":"2024-03-19T02:27:09.732129+0000","last_became_active":"2024-03-19T02:25:08.212295+0000","last_became_peered":"2024-03-19T02:25:08.212295+0000","last_unstale":"2024-03-19T02:27:09.732129+0000","last_undegraded":"2024-03-19T02:27:09.732129+0000","last_fullsized":"2024-03-19T02:27:09.732129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:02:58.038250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031592+0000","last_change":"2024-03-19T02:26:08.850490+0000","last_active":"2024-03-19T02:27:10.031592+0000","last_peered":"2024-03-19T02:27:10.031592+0000","last_clean":"2024-03-19T02:27:10.031592+0000","last_became_active":"2024-03-19T02:26:08.850087+0000","last_became_peered":"2024-03-19T02:26:08.850087+0000","last_unstale":"2024-03-19T02:27:10.031592+0000","last_undegraded":"2024-03-19T02:27:10.031592+0000","last_fullsized":"2024-03-19T02:27:10.031592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:01.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732867+0000","last_change":"2024-03-19T02:25:08.213587+0000","last_active":"2024-03-19T02:27:09.732867+0000","last_peered":"2024-03-19T02:27:09.732867+0000","last_clean":"2024-03-19T02:27:09.732867+0000","last_became_active":"2024-03-19T02:25:08.213217+0000","last_became_peered":"2024-03-19T02:25:08.213217+0000","last_unstale":"2024-03-19T02:27:09.732867+0000","last_undegraded":"2024-03-19T02:27:09.732867+0000","last_fullsized":"2024-03-19T02:27:09.732867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:54.737139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736754+0000","last_change":"2024-03-19T02:25:08.217453+0000","last_active":"2024-03-19T02:27:09.736754+0000","last_peered":"2024-03-19T02:27:09.736754+0000","last_clean":"2024-03-19T02:27:09.736754+0000","last_became_active":"2024-03-19T02:25:08.217162+0000","last_became_peered":"2024-03-19T02:25:08.217162+0000","last_unstale":"2024-03-19T02:27:09.736754+0000","last_undegraded":"2024-03-19T02:27:09.736754+0000","last_fullsized":"2024-03-19T02:27:09.736754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:46.500347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784382+0000","last_change":"2024-03-19T02:26:08.789057+0000","last_active":"2024-03-19T02:27:09.784382+0000","last_peered":"2024-03-19T02:27:09.784382+0000","last_clean":"2024-03-19T02:27:09.784382+0000","last_became_active":"2024-03-19T02:26:08.788689+0000","last_became_peered":"2024-03-19T02:26:08.788689+0000","last_unstale":"2024-03-19T02:27:09.784382+0000","last_undegraded":"2024-03-19T02:27:09.784382+0000","last_fullsized":"2024-03-19T02:27:09.784382+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:36:08.295772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782778+0000","last_change":"2024-03-19T02:25:08.203663+0000","last_active":"2024-03-19T02:27:09.782778+0000","last_peered":"2024-03-19T02:27:09.782778+0000","last_clean":"2024-03-19T02:27:09.782778+0000","last_became_active":"2024-03-19T02:25:08.203358+0000","last_became_peered":"2024-03-19T02:25:08.203358+0000","last_unstale":"2024-03-19T02:27:09.782778+0000","last_undegraded":"2024-03-19T02:27:09.782778+0000","last_fullsized":"2024-03-19T02:27:09.782778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:33.017837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030898+0000","last_change":"2024-03-19T02:25:08.205318+0000","last_active":"2024-03-19T02:27:10.030898+0000","last_peered":"2024-03-19T02:27:10.030898+0000","last_clean":"2024-03-19T02:27:10.030898+0000","last_became_active":"2024-03-19T02:25:08.203818+0000","last_became_peered":"2024-03-19T02:25:08.203818+0000","last_unstale":"2024-03-19T02:27:10.030898+0000","last_undegraded":"2024-03-19T02:27:10.030898+0000","last_fullsized":"2024-03-19T02:27:10.030898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:17.006860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.030955+0000","last_change":"2024-03-19T02:25:08.211716+0000","last_active":"2024-03-19T02:27:10.030955+0000","last_peered":"2024-03-19T02:27:10.030955+0000","last_clean":"2024-03-19T02:27:10.030955+0000","last_became_active":"2024-03-19T02:25:08.211114+0000","last_became_peered":"2024-03-19T02:25:08.211114+0000","last_unstale":"2024-03-19T02:27:10.030955+0000","last_undegraded":"2024-03-19T02:27:10.030955+0000","last_fullsized":"2024-03-19T02:27:10.030955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:39.519643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031044+0000","last_change":"2024-03-19T02:25:08.216958+0000","last_active":"2024-03-19T02:27:10.031044+0000","last_peered":"2024-03-19T02:27:10.031044+0000","last_clean":"2024-03-19T02:27:10.031044+0000","last_became_active":"2024-03-19T02:25:08.216655+0000","last_became_peered":"2024-03-19T02:25:08.216655+0000","last_unstale":"2024-03-19T02:27:10.031044+0000","last_undegraded":"2024-03-19T02:27:10.031044+0000","last_fullsized":"2024-03-19T02:27:10.031044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:40.104763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.735240+0000","last_change":"2024-03-19T02:25:08.208215+0000","last_active":"2024-03-19T02:27:09.735240+0000","last_peered":"2024-03-19T02:27:09.735240+0000","last_clean":"2024-03-19T02:27:09.735240+0000","last_became_active":"2024-03-19T02:25:08.207570+0000","last_became_peered":"2024-03-19T02:25:08.207570+0000","last_unstale":"2024-03-19T02:27:09.735240+0000","last_undegraded":"2024-03-19T02:27:09.735240+0000","last_fullsized":"2024-03-19T02:27:09.735240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:34:50.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784626+0000","last_change":"2024-03-19T02:25:08.205949+0000","last_active":"2024-03-19T02:27:09.784626+0000","last_peered":"2024-03-19T02:27:09.784626+0000","last_clean":"2024-03-19T02:27:09.784626+0000","last_became_active":"2024-03-19T02:25:08.205437+0000","last_became_peered":"2024-03-19T02:25:08.205437+0000","last_unstale":"2024-03-19T02:27:09.784626+0000","last_undegraded":"2024-03-19T02:27:09.784626+0000","last_fullsized":"2024-03-19T02:27:09.784626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:10:22.621802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732894+0000","last_change":"2024-03-19T02:25:08.199180+0000","last_active":"2024-03-19T02:27:09.732894+0000","last_peered":"2024-03-19T02:27:09.732894+0000","last_clean":"2024-03-19T02:27:09.732894+0000","last_became_active":"2024-03-19T02:25:08.198871+0000","last_became_peered":"2024-03-19T02:25:08.198871+0000","last_unstale":"2024-03-19T02:27:09.732894+0000","last_undegraded":"2024-03-19T02:27:09.732894+0000","last_fullsized":"2024-03-19T02:27:09.732894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.568304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732436+0000","last_change":"2024-03-19T02:25:12.221363+0000","last_active":"2024-03-19T02:27:09.732436+0000","last_peered":"2024-03-19T02:27:09.732436+0000","last_clean":"2024-03-19T02:27:09.732436+0000","last_became_active":"2024-03-19T02:25:12.220857+0000","last_became_peered":"2024-03-19T02:25:12.220857+0000","last_unstale":"2024-03-19T02:27:09.732436+0000","last_undegraded":"2024-03-19T02:27:09.732436+0000","last_fullsized":"2024-03-19T02:27:09.732436+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.835240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.809123+0000","last_change":"2024-03-19T02:25:07.644837+0000","last_active":"2024-03-19T02:27:32.809123+0000","last_peered":"2024-03-19T02:27:32.809123+0000","last_clean":"2024-03-19T02:27:32.809123+0000","last_became_active":"2024-03-19T02:25:07.644546+0000","last_became_peered":"2024-03-19T02:25:07.644546+0000","last_unstale":"2024-03-19T02:27:32.809123+0000","last_undegraded":"2024-03-19T02:27:32.809123+0000","last_fullsized":"2024-03-19T02:27:32.809123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:57:54.366225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732449+0000","last_change":"2024-03-19T02:25:08.211664+0000","last_active":"2024-03-19T02:27:09.732449+0000","last_peered":"2024-03-19T02:27:09.732449+0000","last_clean":"2024-03-19T02:27:09.732449+0000","last_became_active":"2024-03-19T02:25:08.211324+0000","last_became_peered":"2024-03-19T02:25:08.211324+0000","last_unstale":"2024-03-19T02:27:09.732449+0000","last_undegraded":"2024-03-19T02:27:09.732449+0000","last_fullsized":"2024-03-19T02:27:09.732449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:49.975119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784240+0000","last_change":"2024-03-19T02:25:12.230026+0000","last_active":"2024-03-19T02:27:09.784240+0000","last_peered":"2024-03-19T02:27:09.784240+0000","last_clean":"2024-03-19T02:27:09.784240+0000","last_became_active":"2024-03-19T02:25:12.229602+0000","last_became_peered":"2024-03-19T02:25:12.229602+0000","last_unstale":"2024-03-19T02:27:09.784240+0000","last_undegraded":"2024-03-19T02:27:09.784240+0000","last_fullsized":"2024-03-19T02:27:09.784240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:58.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732693+0000","last_change":"2024-03-19T02:25:07.638014+0000","last_active":"2024-03-19T02:27:09.732693+0000","last_peered":"2024-03-19T02:27:09.732693+0000","last_clean":"2024-03-19T02:27:09.732693+0000","last_became_active":"2024-03-19T02:25:07.636981+0000","last_became_peered":"2024-03-19T02:25:07.636981+0000","last_unstale":"2024-03-19T02:27:09.732693+0000","last_undegraded":"2024-03-19T02:27:09.732693+0000","last_fullsized":"2024-03-19T02:27:09.732693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:00:05.441950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031294+0000","last_change":"2024-03-19T02:25:08.205399+0000","last_active":"2024-03-19T02:27:10.031294+0000","last_peered":"2024-03-19T02:27:10.031294+0000","last_clean":"2024-03-19T02:27:10.031294+0000","last_became_active":"2024-03-19T02:25:08.205086+0000","last_became_peered":"2024-03-19T02:25:08.205086+0000","last_unstale":"2024-03-19T02:27:10.031294+0000","last_undegraded":"2024-03-19T02:27:10.031294+0000","last_fullsized":"2024-03-19T02:27:10.031294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:09:09.819537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784386+0000","last_change":"2024-03-19T02:25:12.228003+0000","last_active":"2024-03-19T02:27:09.784386+0000","last_peered":"2024-03-19T02:27:09.784386+0000","last_clean":"2024-03-19T02:27:09.784386+0000","last_became_active":"2024-03-19T02:25:12.227383+0000","last_became_peered":"2024-03-19T02:25:12.227383+0000","last_unstale":"2024-03-19T02:27:09.784386+0000","last_undegraded":"2024-03-19T02:27:09.784386+0000","last_fullsized":"2024-03-19T02:27:09.784386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:11.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.746725+0000","last_change":"2024-03-19T02:27:09.746277+0000","last_active":"2024-03-19T02:27:09.746725+0000","last_peered":"2024-03-19T02:27:09.746725+0000","last_clean":"2024-03-19T02:27:09.746725+0000","last_became_active":"2024-03-19T02:27:09.737675+0000","last_became_peered":"2024-03-19T02:27:09.737675+0000","last_unstale":"2024-03-19T02:27:09.746725+0000","last_undegraded":"2024-03-19T02:27:09.746725+0000","last_fullsized":"2024-03-19T02:27:09.746725+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:12:10.829154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031180+0000","last_change":"2024-03-19T02:25:08.202622+0000","last_active":"2024-03-19T02:27:10.031180+0000","last_peered":"2024-03-19T02:27:10.031180+0000","last_clean":"2024-03-19T02:27:10.031180+0000","last_became_active":"2024-03-19T02:25:08.202308+0000","last_became_peered":"2024-03-19T02:25:08.202308+0000","last_unstale":"2024-03-19T02:27:10.031180+0000","last_undegraded":"2024-03-19T02:27:10.031180+0000","last_fullsized":"2024-03-19T02:27:10.031180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:30.515680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784279+0000","last_change":"2024-03-19T02:25:12.219017+0000","last_active":"2024-03-19T02:27:09.784279+0000","last_peered":"2024-03-19T02:27:09.784279+0000","last_clean":"2024-03-19T02:27:09.784279+0000","last_became_active":"2024-03-19T02:25:12.218657+0000","last_became_peered":"2024-03-19T02:25:12.218657+0000","last_unstale":"2024-03-19T02:27:09.784279+0000","last_undegraded":"2024-03-19T02:27:09.784279+0000","last_fullsized":"2024-03-19T02:27:09.784279+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.367077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736489+0000","last_change":"2024-03-19T02:25:07.644174+0000","last_active":"2024-03-19T02:27:09.736489+0000","last_peered":"2024-03-19T02:27:09.736489+0000","last_clean":"2024-03-19T02:27:09.736489+0000","last_became_active":"2024-03-19T02:25:07.643434+0000","last_became_peered":"2024-03-19T02:25:07.643434+0000","last_unstale":"2024-03-19T02:27:09.736489+0000","last_undegraded":"2024-03-19T02:27:09.736489+0000","last_fullsized":"2024-03-19T02:27:09.736489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:49:20.873233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780290+0000","last_change":"2024-03-19T02:25:08.208705+0000","last_active":"2024-03-19T02:27:09.780290+0000","last_peered":"2024-03-19T02:27:09.780290+0000","last_clean":"2024-03-19T02:27:09.780290+0000","last_became_active":"2024-03-19T02:25:08.208399+0000","last_became_peered":"2024-03-19T02:25:08.208399+0000","last_unstale":"2024-03-19T02:27:09.780290+0000","last_undegraded":"2024-03-19T02:27:09.780290+0000","last_fullsized":"2024-03-19T02:27:09.780290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:12.895226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732749+0000","last_change":"2024-03-19T02:25:12.222199+0000","last_active":"2024-03-19T02:27:09.732749+0000","last_peered":"2024-03-19T02:27:09.732749+0000","last_clean":"2024-03-19T02:27:09.732749+0000","last_became_active":"2024-03-19T02:25:12.221734+0000","last_became_peered":"2024-03-19T02:25:12.221734+0000","last_unstale":"2024-03-19T02:27:09.732749+0000","last_undegraded":"2024-03-19T02:27:09.732749+0000","last_fullsized":"2024-03-19T02:27:09.732749+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:57:17.209753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.806736+0000","last_change":"2024-03-19T02:25:07.644287+0000","last_active":"2024-03-19T02:27:34.806736+0000","last_peered":"2024-03-19T02:27:34.806736+0000","last_clean":"2024-03-19T02:27:34.806736+0000","last_became_active":"2024-03-19T02:25:07.643966+0000","last_became_peered":"2024-03-19T02:25:07.643966+0000","last_unstale":"2024-03-19T02:27:34.806736+0000","last_undegraded":"2024-03-19T02:27:34.806736+0000","last_fullsized":"2024-03-19T02:27:34.806736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:39:54.425045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736376+0000","last_change":"2024-03-19T02:25:08.206236+0000","last_active":"2024-03-19T02:27:09.736376+0000","last_peered":"2024-03-19T02:27:09.736376+0000","last_clean":"2024-03-19T02:27:09.736376+0000","last_became_active":"2024-03-19T02:25:08.205295+0000","last_became_peered":"2024-03-19T02:25:08.205295+0000","last_unstale":"2024-03-19T02:27:09.736376+0000","last_undegraded":"2024-03-19T02:27:09.736376+0000","last_fullsized":"2024-03-19T02:27:09.736376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:46.662669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732639+0000","last_change":"2024-03-19T02:26:08.850745+0000","last_active":"2024-03-19T02:27:09.732639+0000","last_peered":"2024-03-19T02:27:09.732639+0000","last_clean":"2024-03-19T02:27:09.732639+0000","last_became_active":"2024-03-19T02:26:08.850285+0000","last_became_peered":"2024-03-19T02:26:08.850285+0000","last_unstale":"2024-03-19T02:27:09.732639+0000","last_undegraded":"2024-03-19T02:27:09.732639+0000","last_fullsized":"2024-03-19T02:27:09.732639+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:17.233944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732492+0000","last_change":"2024-03-19T02:25:07.640751+0000","last_active":"2024-03-19T02:27:09.732492+0000","last_peered":"2024-03-19T02:27:09.732492+0000","last_clean":"2024-03-19T02:27:09.732492+0000","last_became_active":"2024-03-19T02:25:07.640401+0000","last_became_peered":"2024-03-19T02:25:07.640401+0000","last_unstale":"2024-03-19T02:27:09.732492+0000","last_undegraded":"2024-03-19T02:27:09.732492+0000","last_fullsized":"2024-03-19T02:27:09.732492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:12.035691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780207+0000","last_change":"2024-03-19T02:25:08.215402+0000","last_active":"2024-03-19T02:27:09.780207+0000","last_peered":"2024-03-19T02:27:09.780207+0000","last_clean":"2024-03-19T02:27:09.780207+0000","last_became_active":"2024-03-19T02:25:08.215073+0000","last_became_peered":"2024-03-19T02:25:08.215073+0000","last_unstale":"2024-03-19T02:27:09.780207+0000","last_undegraded":"2024-03-19T02:27:09.780207+0000","last_fullsized":"2024-03-19T02:27:09.780207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:06:31.794420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734797+0000","last_change":"2024-03-19T02:25:12.227682+0000","last_active":"2024-03-19T02:27:09.734797+0000","last_peered":"2024-03-19T02:27:09.734797+0000","last_clean":"2024-03-19T02:27:09.734797+0000","last_became_active":"2024-03-19T02:25:12.227134+0000","last_became_peered":"2024-03-19T02:25:12.227134+0000","last_unstale":"2024-03-19T02:27:09.734797+0000","last_undegraded":"2024-03-19T02:27:09.734797+0000","last_fullsized":"2024-03-19T02:27:09.734797+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:20.718660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782652+0000","last_change":"2024-03-19T02:25:07.640948+0000","last_active":"2024-03-19T02:27:09.782652+0000","last_peered":"2024-03-19T02:27:09.782652+0000","last_clean":"2024-03-19T02:27:09.782652+0000","last_became_active":"2024-03-19T02:25:07.640620+0000","last_became_peered":"2024-03-19T02:25:07.640620+0000","last_unstale":"2024-03-19T02:27:09.782652+0000","last_undegraded":"2024-03-19T02:27:09.782652+0000","last_fullsized":"2024-03-19T02:27:09.782652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:28.628299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732313+0000","last_change":"2024-03-19T02:25:08.204032+0000","last_active":"2024-03-19T02:27:09.732313+0000","last_peered":"2024-03-19T02:27:09.732313+0000","last_clean":"2024-03-19T02:27:09.732313+0000","last_became_active":"2024-03-19T02:25:08.203718+0000","last_became_peered":"2024-03-19T02:25:08.203718+0000","last_unstale":"2024-03-19T02:27:09.732313+0000","last_undegraded":"2024-03-19T02:27:09.732313+0000","last_fullsized":"2024-03-19T02:27:09.732313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:38.864112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031213+0000","last_change":"2024-03-19T02:25:12.217058+0000","last_active":"2024-03-19T02:27:10.031213+0000","last_peered":"2024-03-19T02:27:10.031213+0000","last_clean":"2024-03-19T02:27:10.031213+0000","last_became_active":"2024-03-19T02:25:12.216638+0000","last_became_peered":"2024-03-19T02:25:12.216638+0000","last_unstale":"2024-03-19T02:27:10.031213+0000","last_undegraded":"2024-03-19T02:27:10.031213+0000","last_fullsized":"2024-03-19T02:27:10.031213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:31.561286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779904+0000","last_change":"2024-03-19T02:25:07.644897+0000","last_active":"2024-03-19T02:27:09.779904+0000","last_peered":"2024-03-19T02:27:09.779904+0000","last_clean":"2024-03-19T02:27:09.779904+0000","last_became_active":"2024-03-19T02:25:07.644576+0000","last_became_peered":"2024-03-19T02:25:07.644576+0000","last_unstale":"2024-03-19T02:27:09.779904+0000","last_undegraded":"2024-03-19T02:27:09.779904+0000","last_fullsized":"2024-03-19T02:27:09.779904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:04.138808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784266+0000","last_change":"2024-03-19T02:25:08.207169+0000","last_active":"2024-03-19T02:27:09.784266+0000","last_peered":"2024-03-19T02:27:09.784266+0000","last_clean":"2024-03-19T02:27:09.784266+0000","last_became_active":"2024-03-19T02:25:08.206330+0000","last_became_peered":"2024-03-19T02:25:08.206330+0000","last_unstale":"2024-03-19T02:27:09.784266+0000","last_undegraded":"2024-03-19T02:27:09.784266+0000","last_fullsized":"2024-03-19T02:27:09.784266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:55:21.379153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732230+0000","last_change":"2024-03-19T02:25:12.223436+0000","last_active":"2024-03-19T02:27:09.732230+0000","last_peered":"2024-03-19T02:27:09.732230+0000","last_clean":"2024-03-19T02:27:09.732230+0000","last_became_active":"2024-03-19T02:25:12.222857+0000","last_became_peered":"2024-03-19T02:25:12.222857+0000","last_unstale":"2024-03-19T02:27:09.732230+0000","last_undegraded":"2024-03-19T02:27:09.732230+0000","last_fullsized":"2024-03-19T02:27:09.732230+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:24.742150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"94'6","reported_seq":272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.808510+0000","last_change":"2024-03-19T02:25:07.635992+0000","last_active":"2024-03-19T02:27:32.808510+0000","last_peered":"2024-03-19T02:27:32.808510+0000","last_clean":"2024-03-19T02:27:32.808510+0000","last_became_active":"2024-03-19T02:25:07.634924+0000","last_became_peered":"2024-03-19T02:25:07.634924+0000","last_unstale":"2024-03-19T02:27:32.808510+0000","last_undegraded":"2024-03-19T02:27:32.808510+0000","last_fullsized":"2024-03-19T02:27:32.808510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:28.474125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031462+0000","last_change":"2024-03-19T02:25:08.203636+0000","last_active":"2024-03-19T02:27:10.031462+0000","last_peered":"2024-03-19T02:27:10.031462+0000","last_clean":"2024-03-19T02:27:10.031462+0000","last_became_active":"2024-03-19T02:25:08.203320+0000","last_became_peered":"2024-03-19T02:25:08.203320+0000","last_unstale":"2024-03-19T02:27:10.031462+0000","last_undegraded":"2024-03-19T02:27:10.031462+0000","last_fullsized":"2024-03-19T02:27:10.031462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:06:21.628663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784117+0000","last_change":"2024-03-19T02:25:12.227722+0000","last_active":"2024-03-19T02:27:09.784117+0000","last_peered":"2024-03-19T02:27:09.784117+0000","last_clean":"2024-03-19T02:27:09.784117+0000","last_became_active":"2024-03-19T02:25:12.227331+0000","last_became_peered":"2024-03-19T02:25:12.227331+0000","last_unstale":"2024-03-19T02:27:09.784117+0000","last_undegraded":"2024-03-19T02:27:09.784117+0000","last_fullsized":"2024-03-19T02:27:09.784117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:01:58.250881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736846+0000","last_change":"2024-03-19T02:25:07.644527+0000","last_active":"2024-03-19T02:27:09.736846+0000","last_peered":"2024-03-19T02:27:09.736846+0000","last_clean":"2024-03-19T02:27:09.736846+0000","last_became_active":"2024-03-19T02:25:07.643654+0000","last_became_peered":"2024-03-19T02:25:07.643654+0000","last_unstale":"2024-03-19T02:27:09.736846+0000","last_undegraded":"2024-03-19T02:27:09.736846+0000","last_fullsized":"2024-03-19T02:27:09.736846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:38.221846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031299+0000","last_change":"2024-03-19T02:25:08.211518+0000","last_active":"2024-03-19T02:27:10.031299+0000","last_peered":"2024-03-19T02:27:10.031299+0000","last_clean":"2024-03-19T02:27:10.031299+0000","last_became_active":"2024-03-19T02:25:08.211200+0000","last_became_peered":"2024-03-19T02:25:08.211200+0000","last_unstale":"2024-03-19T02:27:10.031299+0000","last_undegraded":"2024-03-19T02:27:10.031299+0000","last_fullsized":"2024-03-19T02:27:10.031299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:44.181073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784581+0000","last_change":"2024-03-19T02:25:12.227959+0000","last_active":"2024-03-19T02:27:09.784581+0000","last_peered":"2024-03-19T02:27:09.784581+0000","last_clean":"2024-03-19T02:27:09.784581+0000","last_became_active":"2024-03-19T02:25:12.227541+0000","last_became_peered":"2024-03-19T02:25:12.227541+0000","last_unstale":"2024-03-19T02:27:09.784581+0000","last_undegraded":"2024-03-19T02:27:09.784581+0000","last_fullsized":"2024-03-19T02:27:09.784581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:13:16.236217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780340+0000","last_change":"2024-03-19T02:25:07.639436+0000","last_active":"2024-03-19T02:27:09.780340+0000","last_peered":"2024-03-19T02:27:09.780340+0000","last_clean":"2024-03-19T02:27:09.780340+0000","last_became_active":"2024-03-19T02:25:07.638282+0000","last_became_peered":"2024-03-19T02:25:07.638282+0000","last_unstale":"2024-03-19T02:27:09.780340+0000","last_undegraded":"2024-03-19T02:27:09.780340+0000","last_fullsized":"2024-03-19T02:27:09.780340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:05:45.940899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031379+0000","last_change":"2024-03-19T02:25:08.205605+0000","last_active":"2024-03-19T02:27:10.031379+0000","last_peered":"2024-03-19T02:27:10.031379+0000","last_clean":"2024-03-19T02:27:10.031379+0000","last_became_active":"2024-03-19T02:25:08.205258+0000","last_became_peered":"2024-03-19T02:25:08.205258+0000","last_unstale":"2024-03-19T02:27:10.031379+0000","last_undegraded":"2024-03-19T02:27:10.031379+0000","last_fullsized":"2024-03-19T02:27:10.031379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:22.153545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784258+0000","last_change":"2024-03-19T02:25:12.217815+0000","last_active":"2024-03-19T02:27:09.784258+0000","last_peered":"2024-03-19T02:27:09.784258+0000","last_clean":"2024-03-19T02:27:09.784258+0000","last_became_active":"2024-03-19T02:25:12.217233+0000","last_became_peered":"2024-03-19T02:25:12.217233+0000","last_unstale":"2024-03-19T02:27:09.784258+0000","last_undegraded":"2024-03-19T02:27:09.784258+0000","last_fullsized":"2024-03-19T02:27:09.784258+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:06.884159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":312,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:32.803913+0000","last_change":"2024-03-19T02:25:07.642222+0000","last_active":"2024-03-19T02:27:32.803913+0000","last_peered":"2024-03-19T02:27:32.803913+0000","last_clean":"2024-03-19T02:27:32.803913+0000","last_became_active":"2024-03-19T02:25:07.641869+0000","last_became_peered":"2024-03-19T02:25:07.641869+0000","last_unstale":"2024-03-19T02:27:32.803913+0000","last_undegraded":"2024-03-19T02:27:32.803913+0000","last_fullsized":"2024-03-19T02:27:32.803913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:17.702817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62517,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":182,"num_write":40,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736974+0000","last_change":"2024-03-19T02:25:08.206927+0000","last_active":"2024-03-19T02:27:09.736974+0000","last_peered":"2024-03-19T02:27:09.736974+0000","last_clean":"2024-03-19T02:27:09.736974+0000","last_became_active":"2024-03-19T02:25:08.206620+0000","last_became_peered":"2024-03-19T02:25:08.206620+0000","last_unstale":"2024-03-19T02:27:09.736974+0000","last_undegraded":"2024-03-19T02:27:09.736974+0000","last_fullsized":"2024-03-19T02:27:09.736974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:41.148386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780524+0000","last_change":"2024-03-19T02:25:12.219480+0000","last_active":"2024-03-19T02:27:09.780524+0000","last_peered":"2024-03-19T02:27:09.780524+0000","last_clean":"2024-03-19T02:27:09.780524+0000","last_became_active":"2024-03-19T02:25:12.219071+0000","last_became_peered":"2024-03-19T02:25:12.219071+0000","last_unstale":"2024-03-19T02:27:09.780524+0000","last_undegraded":"2024-03-19T02:27:09.780524+0000","last_fullsized":"2024-03-19T02:27:09.780524+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:16.894379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732713+0000","last_change":"2024-03-19T02:25:07.641703+0000","last_active":"2024-03-19T02:27:09.732713+0000","last_peered":"2024-03-19T02:27:09.732713+0000","last_clean":"2024-03-19T02:27:09.732713+0000","last_became_active":"2024-03-19T02:25:07.641360+0000","last_became_peered":"2024-03-19T02:25:07.641360+0000","last_unstale":"2024-03-19T02:27:09.732713+0000","last_undegraded":"2024-03-19T02:27:09.732713+0000","last_fullsized":"2024-03-19T02:27:09.732713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:29.906985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782845+0000","last_change":"2024-03-19T02:25:08.207029+0000","last_active":"2024-03-19T02:27:09.782845+0000","last_peered":"2024-03-19T02:27:09.782845+0000","last_clean":"2024-03-19T02:27:09.782845+0000","last_became_active":"2024-03-19T02:25:08.206696+0000","last_became_peered":"2024-03-19T02:25:08.206696+0000","last_unstale":"2024-03-19T02:27:09.782845+0000","last_undegraded":"2024-03-19T02:27:09.782845+0000","last_fullsized":"2024-03-19T02:27:09.782845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:31.994493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734425+0000","last_change":"2024-03-19T02:25:12.224395+0000","last_active":"2024-03-19T02:27:09.734425+0000","last_peered":"2024-03-19T02:27:09.734425+0000","last_clean":"2024-03-19T02:27:09.734425+0000","last_became_active":"2024-03-19T02:25:12.223909+0000","last_became_peered":"2024-03-19T02:25:12.223909+0000","last_unstale":"2024-03-19T02:27:09.734425+0000","last_undegraded":"2024-03-19T02:27:09.734425+0000","last_fullsized":"2024-03-19T02:27:09.734425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:16.291031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780119+0000","last_change":"2024-03-19T02:25:07.645048+0000","last_active":"2024-03-19T02:27:09.780119+0000","last_peered":"2024-03-19T02:27:09.780119+0000","last_clean":"2024-03-19T02:27:09.780119+0000","last_became_active":"2024-03-19T02:25:07.644678+0000","last_became_peered":"2024-03-19T02:25:07.644678+0000","last_unstale":"2024-03-19T02:27:09.780119+0000","last_undegraded":"2024-03-19T02:27:09.780119+0000","last_fullsized":"2024-03-19T02:27:09.780119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:01:58.154575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736687+0000","last_change":"2024-03-19T02:25:08.215678+0000","last_active":"2024-03-19T02:27:09.736687+0000","last_peered":"2024-03-19T02:27:09.736687+0000","last_clean":"2024-03-19T02:27:09.736687+0000","last_became_active":"2024-03-19T02:25:08.215343+0000","last_became_peered":"2024-03-19T02:25:08.215343+0000","last_unstale":"2024-03-19T02:27:09.736687+0000","last_undegraded":"2024-03-19T02:27:09.736687+0000","last_fullsized":"2024-03-19T02:27:09.736687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:08.710215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736910+0000","last_change":"2024-03-19T02:26:08.850986+0000","last_active":"2024-03-19T02:27:09.736910+0000","last_peered":"2024-03-19T02:27:09.736910+0000","last_clean":"2024-03-19T02:27:09.736910+0000","last_became_active":"2024-03-19T02:26:08.850619+0000","last_became_peered":"2024-03-19T02:26:08.850619+0000","last_unstale":"2024-03-19T02:27:09.736910+0000","last_undegraded":"2024-03-19T02:27:09.736910+0000","last_fullsized":"2024-03-19T02:27:09.736910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:15.795959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810904+0000","last_change":"2024-03-19T02:25:07.643985+0000","last_active":"2024-03-19T02:27:33.810904+0000","last_peered":"2024-03-19T02:27:33.810904+0000","last_clean":"2024-03-19T02:27:33.810904+0000","last_became_active":"2024-03-19T02:25:07.643640+0000","last_became_peered":"2024-03-19T02:25:07.643640+0000","last_unstale":"2024-03-19T02:27:33.810904+0000","last_undegraded":"2024-03-19T02:27:33.810904+0000","last_fullsized":"2024-03-19T02:27:33.810904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:51.385142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736368+0000","last_change":"2024-03-19T02:25:08.217852+0000","last_active":"2024-03-19T02:27:09.736368+0000","last_peered":"2024-03-19T02:27:09.736368+0000","last_clean":"2024-03-19T02:27:09.736368+0000","last_became_active":"2024-03-19T02:25:08.217514+0000","last_became_peered":"2024-03-19T02:25:08.217514+0000","last_unstale":"2024-03-19T02:27:09.736368+0000","last_undegraded":"2024-03-19T02:27:09.736368+0000","last_fullsized":"2024-03-19T02:27:09.736368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:59.571962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782900+0000","last_change":"2024-03-19T02:25:12.224498+0000","last_active":"2024-03-19T02:27:09.782900+0000","last_peered":"2024-03-19T02:27:09.782900+0000","last_clean":"2024-03-19T02:27:09.782900+0000","last_became_active":"2024-03-19T02:25:12.224097+0000","last_became_peered":"2024-03-19T02:25:12.224097+0000","last_unstale":"2024-03-19T02:27:09.782900+0000","last_undegraded":"2024-03-19T02:27:09.782900+0000","last_fullsized":"2024-03-19T02:27:09.782900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:03.474587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736140+0000","last_change":"2024-03-19T02:25:07.647263+0000","last_active":"2024-03-19T02:27:09.736140+0000","last_peered":"2024-03-19T02:27:09.736140+0000","last_clean":"2024-03-19T02:27:09.736140+0000","last_became_active":"2024-03-19T02:25:07.646784+0000","last_became_peered":"2024-03-19T02:25:07.646784+0000","last_unstale":"2024-03-19T02:27:09.736140+0000","last_undegraded":"2024-03-19T02:27:09.736140+0000","last_fullsized":"2024-03-19T02:27:09.736140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:09.053603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779991+0000","last_change":"2024-03-19T02:25:08.202656+0000","last_active":"2024-03-19T02:27:09.779991+0000","last_peered":"2024-03-19T02:27:09.779991+0000","last_clean":"2024-03-19T02:27:09.779991+0000","last_became_active":"2024-03-19T02:25:08.202366+0000","last_became_peered":"2024-03-19T02:25:08.202366+0000","last_unstale":"2024-03-19T02:27:09.779991+0000","last_undegraded":"2024-03-19T02:27:09.779991+0000","last_fullsized":"2024-03-19T02:27:09.779991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:09.521418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779948+0000","last_change":"2024-03-19T02:25:12.222033+0000","last_active":"2024-03-19T02:27:09.779948+0000","last_peered":"2024-03-19T02:27:09.779948+0000","last_clean":"2024-03-19T02:27:09.779948+0000","last_became_active":"2024-03-19T02:25:12.221413+0000","last_became_peered":"2024-03-19T02:25:12.221413+0000","last_unstale":"2024-03-19T02:27:09.779948+0000","last_undegraded":"2024-03-19T02:27:09.779948+0000","last_fullsized":"2024-03-19T02:27:09.779948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:50:44.711292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:21.805893+0000","last_change":"2024-03-19T02:25:07.647288+0000","last_active":"2024-03-19T02:27:21.805893+0000","last_peered":"2024-03-19T02:27:21.805893+0000","last_clean":"2024-03-19T02:27:21.805893+0000","last_became_active":"2024-03-19T02:25:07.646394+0000","last_became_peered":"2024-03-19T02:25:07.646394+0000","last_unstale":"2024-03-19T02:27:21.805893+0000","last_undegraded":"2024-03-19T02:27:21.805893+0000","last_fullsized":"2024-03-19T02:27:21.805893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:09.309184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734692+0000","last_change":"2024-03-19T02:25:08.204389+0000","last_active":"2024-03-19T02:27:09.734692+0000","last_peered":"2024-03-19T02:27:09.734692+0000","last_clean":"2024-03-19T02:27:09.734692+0000","last_became_active":"2024-03-19T02:25:08.204073+0000","last_became_peered":"2024-03-19T02:25:08.204073+0000","last_unstale":"2024-03-19T02:27:09.734692+0000","last_undegraded":"2024-03-19T02:27:09.734692+0000","last_fullsized":"2024-03-19T02:27:09.734692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:01.537331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736264+0000","last_change":"2024-03-19T02:25:12.338291+0000","last_active":"2024-03-19T02:27:09.736264+0000","last_peered":"2024-03-19T02:27:09.736264+0000","last_clean":"2024-03-19T02:27:09.736264+0000","last_became_active":"2024-03-19T02:25:12.337714+0000","last_became_peered":"2024-03-19T02:25:12.337714+0000","last_unstale":"2024-03-19T02:27:09.736264+0000","last_undegraded":"2024-03-19T02:27:09.736264+0000","last_fullsized":"2024-03-19T02:27:09.736264+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:04.231975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784488+0000","last_change":"2024-03-19T02:26:08.851073+0000","last_active":"2024-03-19T02:27:09.784488+0000","last_peered":"2024-03-19T02:27:09.784488+0000","last_clean":"2024-03-19T02:27:09.784488+0000","last_became_active":"2024-03-19T02:26:08.850702+0000","last_became_peered":"2024-03-19T02:26:08.850702+0000","last_unstale":"2024-03-19T02:27:09.784488+0000","last_undegraded":"2024-03-19T02:27:09.784488+0000","last_fullsized":"2024-03-19T02:27:09.784488+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:11.668203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732945+0000","last_change":"2024-03-19T02:26:08.788232+0000","last_active":"2024-03-19T02:27:09.732945+0000","last_peered":"2024-03-19T02:27:09.732945+0000","last_clean":"2024-03-19T02:27:09.732945+0000","last_became_active":"2024-03-19T02:26:08.787993+0000","last_became_peered":"2024-03-19T02:26:08.787993+0000","last_unstale":"2024-03-19T02:27:09.732945+0000","last_undegraded":"2024-03-19T02:27:09.732945+0000","last_fullsized":"2024-03-19T02:27:09.732945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:40.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782988+0000","last_change":"2024-03-19T02:25:12.220963+0000","last_active":"2024-03-19T02:27:09.782988+0000","last_peered":"2024-03-19T02:27:09.782988+0000","last_clean":"2024-03-19T02:27:09.782988+0000","last_became_active":"2024-03-19T02:25:12.219962+0000","last_became_peered":"2024-03-19T02:25:12.219962+0000","last_unstale":"2024-03-19T02:27:09.782988+0000","last_undegraded":"2024-03-19T02:27:09.782988+0000","last_fullsized":"2024-03-19T02:27:09.782988+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:11:29.134773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'6","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:35.812648+0000","last_change":"2024-03-19T02:25:07.651060+0000","last_active":"2024-03-19T02:27:35.812648+0000","last_peered":"2024-03-19T02:27:35.812648+0000","last_clean":"2024-03-19T02:27:35.812648+0000","last_became_active":"2024-03-19T02:25:07.650707+0000","last_became_peered":"2024-03-19T02:25:07.650707+0000","last_unstale":"2024-03-19T02:27:35.812648+0000","last_undegraded":"2024-03-19T02:27:35.812648+0000","last_fullsized":"2024-03-19T02:27:35.812648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:18:24.344378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732734+0000","last_change":"2024-03-19T02:25:08.209506+0000","last_active":"2024-03-19T02:27:09.732734+0000","last_peered":"2024-03-19T02:27:09.732734+0000","last_clean":"2024-03-19T02:27:09.732734+0000","last_became_active":"2024-03-19T02:25:08.209016+0000","last_became_peered":"2024-03-19T02:25:08.209016+0000","last_unstale":"2024-03-19T02:27:09.732734+0000","last_undegraded":"2024-03-19T02:27:09.732734+0000","last_fullsized":"2024-03-19T02:27:09.732734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:29.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782965+0000","last_change":"2024-03-19T02:25:12.338518+0000","last_active":"2024-03-19T02:27:09.782965+0000","last_peered":"2024-03-19T02:27:09.782965+0000","last_clean":"2024-03-19T02:27:09.782965+0000","last_became_active":"2024-03-19T02:25:12.337587+0000","last_became_peered":"2024-03-19T02:25:12.337587+0000","last_unstale":"2024-03-19T02:27:09.782965+0000","last_undegraded":"2024-03-19T02:27:09.782965+0000","last_fullsized":"2024-03-19T02:27:09.782965+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:55.063370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780052+0000","last_change":"2024-03-19T02:25:08.210909+0000","last_active":"2024-03-19T02:27:09.780052+0000","last_peered":"2024-03-19T02:27:09.780052+0000","last_clean":"2024-03-19T02:27:09.780052+0000","last_became_active":"2024-03-19T02:25:08.210575+0000","last_became_peered":"2024-03-19T02:25:08.210575+0000","last_unstale":"2024-03-19T02:27:09.780052+0000","last_undegraded":"2024-03-19T02:27:09.780052+0000","last_fullsized":"2024-03-19T02:27:09.780052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:02.840649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:34.811532+0000","last_change":"2024-03-19T02:25:07.640676+0000","last_active":"2024-03-19T02:27:34.811532+0000","last_peered":"2024-03-19T02:27:34.811532+0000","last_clean":"2024-03-19T02:27:34.811532+0000","last_became_active":"2024-03-19T02:25:07.640328+0000","last_became_peered":"2024-03-19T02:25:07.640328+0000","last_unstale":"2024-03-19T02:27:34.811532+0000","last_undegraded":"2024-03-19T02:27:34.811532+0000","last_fullsized":"2024-03-19T02:27:34.811532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.635786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736630+0000","last_change":"2024-03-19T02:25:12.216078+0000","last_active":"2024-03-19T02:27:09.736630+0000","last_peered":"2024-03-19T02:27:09.736630+0000","last_clean":"2024-03-19T02:27:09.736630+0000","last_became_active":"2024-03-19T02:25:12.215720+0000","last_became_peered":"2024-03-19T02:25:12.215720+0000","last_unstale":"2024-03-19T02:27:09.736630+0000","last_undegraded":"2024-03-19T02:27:09.736630+0000","last_fullsized":"2024-03-19T02:27:09.736630+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:33:36.987846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732519+0000","last_change":"2024-03-19T02:25:08.216179+0000","last_active":"2024-03-19T02:27:09.732519+0000","last_peered":"2024-03-19T02:27:09.732519+0000","last_clean":"2024-03-19T02:27:09.732519+0000","last_became_active":"2024-03-19T02:25:08.215170+0000","last_became_peered":"2024-03-19T02:25:08.215170+0000","last_unstale":"2024-03-19T02:27:09.732519+0000","last_undegraded":"2024-03-19T02:27:09.732519+0000","last_fullsized":"2024-03-19T02:27:09.732519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:19:21.947619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784470+0000","last_change":"2024-03-19T02:25:07.634897+0000","last_active":"2024-03-19T02:27:09.784470+0000","last_peered":"2024-03-19T02:27:09.784470+0000","last_clean":"2024-03-19T02:27:09.784470+0000","last_became_active":"2024-03-19T02:25:07.634559+0000","last_became_peered":"2024-03-19T02:25:07.634559+0000","last_unstale":"2024-03-19T02:27:09.784470+0000","last_undegraded":"2024-03-19T02:27:09.784470+0000","last_fullsized":"2024-03-19T02:27:09.784470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:42:21.608081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736160+0000","last_change":"2024-03-19T02:23:57.663554+0000","last_active":"2024-03-19T02:27:09.736160+0000","last_peered":"2024-03-19T02:27:09.736160+0000","last_clean":"2024-03-19T02:27:09.736160+0000","last_became_active":"2024-03-19T02:23:57.663113+0000","last_became_peered":"2024-03-19T02:23:57.663113+0000","last_unstale":"2024-03-19T02:27:09.736160+0000","last_undegraded":"2024-03-19T02:27:09.736160+0000","last_fullsized":"2024-03-19T02:27:09.736160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:20:31.998024+0000","last_clean_scrub_stamp":"2024-03-19T02:20:31.998024+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:48:11.248856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780528+0000","last_change":"2024-03-19T02:25:12.338585+0000","last_active":"2024-03-19T02:27:09.780528+0000","last_peered":"2024-03-19T02:27:09.780528+0000","last_clean":"2024-03-19T02:27:09.780528+0000","last_became_active":"2024-03-19T02:25:12.337861+0000","last_became_peered":"2024-03-19T02:25:12.337861+0000","last_unstale":"2024-03-19T02:27:09.780528+0000","last_undegraded":"2024-03-19T02:27:09.780528+0000","last_fullsized":"2024-03-19T02:27:09.780528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:47.060051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782541+0000","last_change":"2024-03-19T02:26:08.788444+0000","last_active":"2024-03-19T02:27:09.782541+0000","last_peered":"2024-03-19T02:27:09.782541+0000","last_clean":"2024-03-19T02:27:09.782541+0000","last_became_active":"2024-03-19T02:26:08.788084+0000","last_became_peered":"2024-03-19T02:26:08.788084+0000","last_unstale":"2024-03-19T02:27:09.782541+0000","last_undegraded":"2024-03-19T02:27:09.782541+0000","last_fullsized":"2024-03-19T02:27:09.782541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:18.879498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736300+0000","last_change":"2024-03-19T02:25:07.649612+0000","last_active":"2024-03-19T02:27:09.736300+0000","last_peered":"2024-03-19T02:27:09.736300+0000","last_clean":"2024-03-19T02:27:09.736300+0000","last_became_active":"2024-03-19T02:25:07.649338+0000","last_became_peered":"2024-03-19T02:25:07.649338+0000","last_unstale":"2024-03-19T02:27:09.736300+0000","last_undegraded":"2024-03-19T02:27:09.736300+0000","last_fullsized":"2024-03-19T02:27:09.736300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:56.647024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734318+0000","last_change":"2024-03-19T02:25:12.224978+0000","last_active":"2024-03-19T02:27:09.734318+0000","last_peered":"2024-03-19T02:27:09.734318+0000","last_clean":"2024-03-19T02:27:09.734318+0000","last_became_active":"2024-03-19T02:25:12.224457+0000","last_became_peered":"2024-03-19T02:25:12.224457+0000","last_unstale":"2024-03-19T02:27:09.734318+0000","last_undegraded":"2024-03-19T02:27:09.734318+0000","last_fullsized":"2024-03-19T02:27:09.734318+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:00.502491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.737005+0000","last_change":"2024-03-19T02:25:07.640672+0000","last_active":"2024-03-19T02:27:09.737005+0000","last_peered":"2024-03-19T02:27:09.737005+0000","last_clean":"2024-03-19T02:27:09.737005+0000","last_became_active":"2024-03-19T02:25:07.640308+0000","last_became_peered":"2024-03-19T02:25:07.640308+0000","last_unstale":"2024-03-19T02:27:09.737005+0000","last_undegraded":"2024-03-19T02:27:09.737005+0000","last_fullsized":"2024-03-19T02:27:09.737005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:27.510585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783071+0000","last_change":"2024-03-19T02:25:08.199171+0000","last_active":"2024-03-19T02:27:09.783071+0000","last_peered":"2024-03-19T02:27:09.783071+0000","last_clean":"2024-03-19T02:27:09.783071+0000","last_became_active":"2024-03-19T02:25:08.198715+0000","last_became_peered":"2024-03-19T02:25:08.198715+0000","last_unstale":"2024-03-19T02:27:09.783071+0000","last_undegraded":"2024-03-19T02:27:09.783071+0000","last_fullsized":"2024-03-19T02:27:09.783071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:26.198117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732209+0000","last_change":"2024-03-19T02:25:12.337856+0000","last_active":"2024-03-19T02:27:09.732209+0000","last_peered":"2024-03-19T02:27:09.732209+0000","last_clean":"2024-03-19T02:27:09.732209+0000","last_became_active":"2024-03-19T02:25:12.337248+0000","last_became_peered":"2024-03-19T02:25:12.337248+0000","last_unstale":"2024-03-19T02:27:09.732209+0000","last_undegraded":"2024-03-19T02:27:09.732209+0000","last_fullsized":"2024-03-19T02:27:09.732209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:16.109833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.667708+0000","last_change":"2024-03-19T02:25:07.640264+0000","last_active":"2024-03-19T02:27:33.667708+0000","last_peered":"2024-03-19T02:27:33.667708+0000","last_clean":"2024-03-19T02:27:33.667708+0000","last_became_active":"2024-03-19T02:25:07.639899+0000","last_became_peered":"2024-03-19T02:25:07.639899+0000","last_unstale":"2024-03-19T02:27:33.667708+0000","last_undegraded":"2024-03-19T02:27:33.667708+0000","last_fullsized":"2024-03-19T02:27:33.667708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:01.849030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732653+0000","last_change":"2024-03-19T02:25:08.214913+0000","last_active":"2024-03-19T02:27:09.732653+0000","last_peered":"2024-03-19T02:27:09.732653+0000","last_clean":"2024-03-19T02:27:09.732653+0000","last_became_active":"2024-03-19T02:25:08.214579+0000","last_became_peered":"2024-03-19T02:25:08.214579+0000","last_unstale":"2024-03-19T02:27:09.732653+0000","last_undegraded":"2024-03-19T02:27:09.732653+0000","last_fullsized":"2024-03-19T02:27:09.732653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.486677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732378+0000","last_change":"2024-03-19T02:25:12.213294+0000","last_active":"2024-03-19T02:27:09.732378+0000","last_peered":"2024-03-19T02:27:09.732378+0000","last_clean":"2024-03-19T02:27:09.732378+0000","last_became_active":"2024-03-19T02:25:12.212790+0000","last_became_peered":"2024-03-19T02:25:12.212790+0000","last_unstale":"2024-03-19T02:27:09.732378+0000","last_undegraded":"2024-03-19T02:27:09.732378+0000","last_fullsized":"2024-03-19T02:27:09.732378+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:04:03.039393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784698+0000","last_change":"2024-03-19T02:25:07.652441+0000","last_active":"2024-03-19T02:27:09.784698+0000","last_peered":"2024-03-19T02:27:09.784698+0000","last_clean":"2024-03-19T02:27:09.784698+0000","last_became_active":"2024-03-19T02:25:07.652136+0000","last_became_peered":"2024-03-19T02:25:07.652136+0000","last_unstale":"2024-03-19T02:27:09.784698+0000","last_undegraded":"2024-03-19T02:27:09.784698+0000","last_fullsized":"2024-03-19T02:27:09.784698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:17.527978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031322+0000","last_change":"2024-03-19T02:25:08.209430+0000","last_active":"2024-03-19T02:27:10.031322+0000","last_peered":"2024-03-19T02:27:10.031322+0000","last_clean":"2024-03-19T02:27:10.031322+0000","last_became_active":"2024-03-19T02:25:08.209033+0000","last_became_peered":"2024-03-19T02:25:08.209033+0000","last_unstale":"2024-03-19T02:27:10.031322+0000","last_undegraded":"2024-03-19T02:27:10.031322+0000","last_fullsized":"2024-03-19T02:27:10.031322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:52:49.644805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784756+0000","last_change":"2024-03-19T02:25:12.337831+0000","last_active":"2024-03-19T02:27:09.784756+0000","last_peered":"2024-03-19T02:27:09.784756+0000","last_clean":"2024-03-19T02:27:09.784756+0000","last_became_active":"2024-03-19T02:25:12.337430+0000","last_became_peered":"2024-03-19T02:25:12.337430+0000","last_unstale":"2024-03-19T02:27:09.784756+0000","last_undegraded":"2024-03-19T02:27:09.784756+0000","last_fullsized":"2024-03-19T02:27:09.784756+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:41.117747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031094+0000","last_change":"2024-03-19T02:25:07.641846+0000","last_active":"2024-03-19T02:27:10.031094+0000","last_peered":"2024-03-19T02:27:10.031094+0000","last_clean":"2024-03-19T02:27:10.031094+0000","last_became_active":"2024-03-19T02:25:07.641518+0000","last_became_peered":"2024-03-19T02:25:07.641518+0000","last_unstale":"2024-03-19T02:27:10.031094+0000","last_undegraded":"2024-03-19T02:27:10.031094+0000","last_fullsized":"2024-03-19T02:27:10.031094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:42.580121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736139+0000","last_change":"2024-03-19T02:25:08.207612+0000","last_active":"2024-03-19T02:27:09.736139+0000","last_peered":"2024-03-19T02:27:09.736139+0000","last_clean":"2024-03-19T02:27:09.736139+0000","last_became_active":"2024-03-19T02:25:08.207310+0000","last_became_peered":"2024-03-19T02:25:08.207310+0000","last_unstale":"2024-03-19T02:27:09.736139+0000","last_undegraded":"2024-03-19T02:27:09.736139+0000","last_fullsized":"2024-03-19T02:27:09.736139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:46.094369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784635+0000","last_change":"2024-03-19T02:25:12.230038+0000","last_active":"2024-03-19T02:27:09.784635+0000","last_peered":"2024-03-19T02:27:09.784635+0000","last_clean":"2024-03-19T02:27:09.784635+0000","last_became_active":"2024-03-19T02:25:12.228278+0000","last_became_peered":"2024-03-19T02:25:12.228278+0000","last_unstale":"2024-03-19T02:27:09.784635+0000","last_undegraded":"2024-03-19T02:27:09.784635+0000","last_fullsized":"2024-03-19T02:27:09.784635+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:23.456397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732299+0000","last_change":"2024-03-19T02:25:07.636715+0000","last_active":"2024-03-19T02:27:09.732299+0000","last_peered":"2024-03-19T02:27:09.732299+0000","last_clean":"2024-03-19T02:27:09.732299+0000","last_became_active":"2024-03-19T02:25:07.635892+0000","last_became_peered":"2024-03-19T02:25:07.635892+0000","last_unstale":"2024-03-19T02:27:09.732299+0000","last_undegraded":"2024-03-19T02:27:09.732299+0000","last_fullsized":"2024-03-19T02:27:09.732299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:36.309143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734210+0000","last_change":"2024-03-19T02:25:08.207397+0000","last_active":"2024-03-19T02:27:09.734210+0000","last_peered":"2024-03-19T02:27:09.734210+0000","last_clean":"2024-03-19T02:27:09.734210+0000","last_became_active":"2024-03-19T02:25:08.207075+0000","last_became_peered":"2024-03-19T02:25:08.207075+0000","last_unstale":"2024-03-19T02:27:09.734210+0000","last_undegraded":"2024-03-19T02:27:09.734210+0000","last_fullsized":"2024-03-19T02:27:09.734210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:10.037254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784665+0000","last_change":"2024-03-19T02:25:12.228748+0000","last_active":"2024-03-19T02:27:09.784665+0000","last_peered":"2024-03-19T02:27:09.784665+0000","last_clean":"2024-03-19T02:27:09.784665+0000","last_became_active":"2024-03-19T02:25:12.228336+0000","last_became_peered":"2024-03-19T02:25:12.228336+0000","last_unstale":"2024-03-19T02:27:09.784665+0000","last_undegraded":"2024-03-19T02:27:09.784665+0000","last_fullsized":"2024-03-19T02:27:09.784665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:26.823892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:31.716710+0000","last_change":"2024-03-19T02:25:07.643297+0000","last_active":"2024-03-19T02:27:31.716710+0000","last_peered":"2024-03-19T02:27:31.716710+0000","last_clean":"2024-03-19T02:27:31.716710+0000","last_became_active":"2024-03-19T02:25:07.642891+0000","last_became_peered":"2024-03-19T02:25:07.642891+0000","last_unstale":"2024-03-19T02:27:31.716710+0000","last_undegraded":"2024-03-19T02:27:31.716710+0000","last_fullsized":"2024-03-19T02:27:31.716710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:13.599013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732367+0000","last_change":"2024-03-19T02:25:08.209412+0000","last_active":"2024-03-19T02:27:09.732367+0000","last_peered":"2024-03-19T02:27:09.732367+0000","last_clean":"2024-03-19T02:27:09.732367+0000","last_became_active":"2024-03-19T02:25:08.209074+0000","last_became_peered":"2024-03-19T02:25:08.209074+0000","last_unstale":"2024-03-19T02:27:09.732367+0000","last_undegraded":"2024-03-19T02:27:09.732367+0000","last_fullsized":"2024-03-19T02:27:09.732367+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:52.426388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.784597+0000","last_change":"2024-03-19T02:25:12.337860+0000","last_active":"2024-03-19T02:27:09.784597+0000","last_peered":"2024-03-19T02:27:09.784597+0000","last_clean":"2024-03-19T02:27:09.784597+0000","last_became_active":"2024-03-19T02:25:12.337428+0000","last_became_peered":"2024-03-19T02:25:12.337428+0000","last_unstale":"2024-03-19T02:27:09.784597+0000","last_undegraded":"2024-03-19T02:27:09.784597+0000","last_fullsized":"2024-03-19T02:27:09.784597+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:53:33.290980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736496+0000","last_change":"2024-03-19T02:25:07.647033+0000","last_active":"2024-03-19T02:27:09.736496+0000","last_peered":"2024-03-19T02:27:09.736496+0000","last_clean":"2024-03-19T02:27:09.736496+0000","last_became_active":"2024-03-19T02:25:07.646729+0000","last_became_peered":"2024-03-19T02:25:07.646729+0000","last_unstale":"2024-03-19T02:27:09.736496+0000","last_undegraded":"2024-03-19T02:27:09.736496+0000","last_fullsized":"2024-03-19T02:27:09.736496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:14:28.145629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736549+0000","last_change":"2024-03-19T02:25:08.217021+0000","last_active":"2024-03-19T02:27:09.736549+0000","last_peered":"2024-03-19T02:27:09.736549+0000","last_clean":"2024-03-19T02:27:09.736549+0000","last_became_active":"2024-03-19T02:25:08.216520+0000","last_became_peered":"2024-03-19T02:25:08.216520+0000","last_unstale":"2024-03-19T02:27:09.736549+0000","last_undegraded":"2024-03-19T02:27:09.736549+0000","last_fullsized":"2024-03-19T02:27:09.736549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:36.017552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031127+0000","last_change":"2024-03-19T02:25:12.216360+0000","last_active":"2024-03-19T02:27:10.031127+0000","last_peered":"2024-03-19T02:27:10.031127+0000","last_clean":"2024-03-19T02:27:10.031127+0000","last_became_active":"2024-03-19T02:25:12.215923+0000","last_became_peered":"2024-03-19T02:25:12.215923+0000","last_unstale":"2024-03-19T02:27:10.031127+0000","last_undegraded":"2024-03-19T02:27:10.031127+0000","last_fullsized":"2024-03-19T02:27:10.031127+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:05:25.162481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'6","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:33.810079+0000","last_change":"2024-03-19T02:25:07.652689+0000","last_active":"2024-03-19T02:27:33.810079+0000","last_peered":"2024-03-19T02:27:33.810079+0000","last_clean":"2024-03-19T02:27:33.810079+0000","last_became_active":"2024-03-19T02:25:07.652366+0000","last_became_peered":"2024-03-19T02:25:07.652366+0000","last_unstale":"2024-03-19T02:27:33.810079+0000","last_undegraded":"2024-03-19T02:27:33.810079+0000","last_fullsized":"2024-03-19T02:27:33.810079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:56:23.120246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736802+0000","last_change":"2024-03-19T02:25:08.216675+0000","last_active":"2024-03-19T02:27:09.736802+0000","last_peered":"2024-03-19T02:27:09.736802+0000","last_clean":"2024-03-19T02:27:09.736802+0000","last_became_active":"2024-03-19T02:25:08.216302+0000","last_became_peered":"2024-03-19T02:25:08.216302+0000","last_unstale":"2024-03-19T02:27:09.736802+0000","last_undegraded":"2024-03-19T02:27:09.736802+0000","last_fullsized":"2024-03-19T02:27:09.736802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:59.478129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031053+0000","last_change":"2024-03-19T02:26:08.849926+0000","last_active":"2024-03-19T02:27:10.031053+0000","last_peered":"2024-03-19T02:27:10.031053+0000","last_clean":"2024-03-19T02:27:10.031053+0000","last_became_active":"2024-03-19T02:26:08.849548+0000","last_became_peered":"2024-03-19T02:26:08.849548+0000","last_unstale":"2024-03-19T02:27:10.031053+0000","last_undegraded":"2024-03-19T02:27:10.031053+0000","last_fullsized":"2024-03-19T02:27:10.031053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:44:35.078313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.780463+0000","last_change":"2024-03-19T02:25:07.640506+0000","last_active":"2024-03-19T02:27:09.780463+0000","last_peered":"2024-03-19T02:27:09.780463+0000","last_clean":"2024-03-19T02:27:09.780463+0000","last_became_active":"2024-03-19T02:25:07.640170+0000","last_became_peered":"2024-03-19T02:25:07.640170+0000","last_unstale":"2024-03-19T02:27:09.780463+0000","last_undegraded":"2024-03-19T02:27:09.780463+0000","last_fullsized":"2024-03-19T02:27:09.780463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:26.241695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736591+0000","last_change":"2024-03-19T02:25:08.205816+0000","last_active":"2024-03-19T02:27:09.736591+0000","last_peered":"2024-03-19T02:27:09.736591+0000","last_clean":"2024-03-19T02:27:09.736591+0000","last_became_active":"2024-03-19T02:25:08.205508+0000","last_became_peered":"2024-03-19T02:25:08.205508+0000","last_unstale":"2024-03-19T02:27:09.736591+0000","last_undegraded":"2024-03-19T02:27:09.736591+0000","last_fullsized":"2024-03-19T02:27:09.736591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:28.468896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736069+0000","last_change":"2024-03-19T02:26:08.850968+0000","last_active":"2024-03-19T02:27:09.736069+0000","last_peered":"2024-03-19T02:27:09.736069+0000","last_clean":"2024-03-19T02:27:09.736069+0000","last_became_active":"2024-03-19T02:26:08.850571+0000","last_became_peered":"2024-03-19T02:26:08.850571+0000","last_unstale":"2024-03-19T02:27:09.736069+0000","last_undegraded":"2024-03-19T02:27:09.736069+0000","last_fullsized":"2024-03-19T02:27:09.736069+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:11.190959+0000","last_clean_scrub_stamp":"2024-03-19T02:25:11.190959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:29.532645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783058+0000","last_change":"2024-03-19T02:25:07.649438+0000","last_active":"2024-03-19T02:27:09.783058+0000","last_peered":"2024-03-19T02:27:09.783058+0000","last_clean":"2024-03-19T02:27:09.783058+0000","last_became_active":"2024-03-19T02:25:07.649090+0000","last_became_peered":"2024-03-19T02:25:07.649090+0000","last_unstale":"2024-03-19T02:27:09.783058+0000","last_undegraded":"2024-03-19T02:27:09.783058+0000","last_fullsized":"2024-03-19T02:27:09.783058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:06.609616+0000","last_clean_scrub_stamp":"2024-03-19T02:25:06.609616+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:37:34.857006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.736961+0000","last_change":"2024-03-19T02:25:08.218046+0000","last_active":"2024-03-19T02:27:09.736961+0000","last_peered":"2024-03-19T02:27:09.736961+0000","last_clean":"2024-03-19T02:27:09.736961+0000","last_became_active":"2024-03-19T02:25:08.217134+0000","last_became_peered":"2024-03-19T02:25:08.217134+0000","last_unstale":"2024-03-19T02:27:09.736961+0000","last_undegraded":"2024-03-19T02:27:09.736961+0000","last_fullsized":"2024-03-19T02:27:09.736961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:52:50.832184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734276+0000","last_change":"2024-03-19T02:25:08.212344+0000","last_active":"2024-03-19T02:27:09.734276+0000","last_peered":"2024-03-19T02:27:09.734276+0000","last_clean":"2024-03-19T02:27:09.734276+0000","last_became_active":"2024-03-19T02:25:08.211962+0000","last_became_peered":"2024-03-19T02:25:08.211962+0000","last_unstale":"2024-03-19T02:27:09.734276+0000","last_undegraded":"2024-03-19T02:27:09.734276+0000","last_fullsized":"2024-03-19T02:27:09.734276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:13.719071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732682+0000","last_change":"2024-03-19T02:25:08.205041+0000","last_active":"2024-03-19T02:27:09.732682+0000","last_peered":"2024-03-19T02:27:09.732682+0000","last_clean":"2024-03-19T02:27:09.732682+0000","last_became_active":"2024-03-19T02:25:08.203790+0000","last_became_peered":"2024-03-19T02:25:08.203790+0000","last_unstale":"2024-03-19T02:27:09.732682+0000","last_undegraded":"2024-03-19T02:27:09.732682+0000","last_fullsized":"2024-03-19T02:27:09.732682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:09.521976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732568+0000","last_change":"2024-03-19T02:25:08.203112+0000","last_active":"2024-03-19T02:27:09.732568+0000","last_peered":"2024-03-19T02:27:09.732568+0000","last_clean":"2024-03-19T02:27:09.732568+0000","last_became_active":"2024-03-19T02:25:08.202677+0000","last_became_peered":"2024-03-19T02:25:08.202677+0000","last_unstale":"2024-03-19T02:27:09.732568+0000","last_undegraded":"2024-03-19T02:27:09.732568+0000","last_fullsized":"2024-03-19T02:27:09.732568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:06.747629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031210+0000","last_change":"2024-03-19T02:25:08.211591+0000","last_active":"2024-03-19T02:27:10.031210+0000","last_peered":"2024-03-19T02:27:10.031210+0000","last_clean":"2024-03-19T02:27:10.031210+0000","last_became_active":"2024-03-19T02:25:08.211272+0000","last_became_peered":"2024-03-19T02:25:08.211272+0000","last_unstale":"2024-03-19T02:27:10.031210+0000","last_undegraded":"2024-03-19T02:27:10.031210+0000","last_fullsized":"2024-03-19T02:27:10.031210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:59.412425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.783084+0000","last_change":"2024-03-19T02:25:08.200970+0000","last_active":"2024-03-19T02:27:09.783084+0000","last_peered":"2024-03-19T02:27:09.783084+0000","last_clean":"2024-03-19T02:27:09.783084+0000","last_became_active":"2024-03-19T02:25:08.200673+0000","last_became_peered":"2024-03-19T02:25:08.200673+0000","last_unstale":"2024-03-19T02:27:09.783084+0000","last_undegraded":"2024-03-19T02:27:09.783084+0000","last_fullsized":"2024-03-19T02:27:09.783084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:20.285789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734859+0000","last_change":"2024-03-19T02:25:08.203583+0000","last_active":"2024-03-19T02:27:09.734859+0000","last_peered":"2024-03-19T02:27:09.734859+0000","last_clean":"2024-03-19T02:27:09.734859+0000","last_became_active":"2024-03-19T02:25:08.203274+0000","last_became_peered":"2024-03-19T02:25:08.203274+0000","last_unstale":"2024-03-19T02:27:09.734859+0000","last_undegraded":"2024-03-19T02:27:09.734859+0000","last_fullsized":"2024-03-19T02:27:09.734859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:40.188244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732975+0000","last_change":"2024-03-19T02:25:08.200672+0000","last_active":"2024-03-19T02:27:09.732975+0000","last_peered":"2024-03-19T02:27:09.732975+0000","last_clean":"2024-03-19T02:27:09.732975+0000","last_became_active":"2024-03-19T02:25:08.200311+0000","last_became_peered":"2024-03-19T02:25:08.200311+0000","last_unstale":"2024-03-19T02:27:09.732975+0000","last_undegraded":"2024-03-19T02:27:09.732975+0000","last_fullsized":"2024-03-19T02:27:09.732975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:38.309084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732768+0000","last_change":"2024-03-19T02:26:08.790951+0000","last_active":"2024-03-19T02:27:09.732768+0000","last_peered":"2024-03-19T02:27:09.732768+0000","last_clean":"2024-03-19T02:27:09.732768+0000","last_became_active":"2024-03-19T02:26:08.790587+0000","last_became_peered":"2024-03-19T02:26:08.790587+0000","last_unstale":"2024-03-19T02:27:09.732768+0000","last_undegraded":"2024-03-19T02:27:09.732768+0000","last_fullsized":"2024-03-19T02:27:09.732768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:11.410269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.779952+0000","last_change":"2024-03-19T02:25:08.215436+0000","last_active":"2024-03-19T02:27:09.779952+0000","last_peered":"2024-03-19T02:27:09.779952+0000","last_clean":"2024-03-19T02:27:09.779952+0000","last_became_active":"2024-03-19T02:25:08.215167+0000","last_became_peered":"2024-03-19T02:25:08.215167+0000","last_unstale":"2024-03-19T02:27:09.779952+0000","last_undegraded":"2024-03-19T02:27:09.779952+0000","last_fullsized":"2024-03-19T02:27:09.779952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:58.158256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.732410+0000","last_change":"2024-03-19T02:25:08.206863+0000","last_active":"2024-03-19T02:27:09.732410+0000","last_peered":"2024-03-19T02:27:09.732410+0000","last_clean":"2024-03-19T02:27:09.732410+0000","last_became_active":"2024-03-19T02:25:08.206564+0000","last_became_peered":"2024-03-19T02:25:08.206564+0000","last_unstale":"2024-03-19T02:27:09.732410+0000","last_undegraded":"2024-03-19T02:27:09.732410+0000","last_fullsized":"2024-03-19T02:27:09.732410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:26.961931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:10.031305+0000","last_change":"2024-03-19T02:25:08.210311+0000","last_active":"2024-03-19T02:27:10.031305+0000","last_peered":"2024-03-19T02:27:10.031305+0000","last_clean":"2024-03-19T02:27:10.031305+0000","last_became_active":"2024-03-19T02:25:08.209965+0000","last_became_peered":"2024-03-19T02:25:08.209965+0000","last_unstale":"2024-03-19T02:27:10.031305+0000","last_undegraded":"2024-03-19T02:27:10.031305+0000","last_fullsized":"2024-03-19T02:27:10.031305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:24.883716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.782801+0000","last_change":"2024-03-19T02:25:08.207069+0000","last_active":"2024-03-19T02:27:09.782801+0000","last_peered":"2024-03-19T02:27:09.782801+0000","last_clean":"2024-03-19T02:27:09.782801+0000","last_became_active":"2024-03-19T02:25:08.206762+0000","last_became_peered":"2024-03-19T02:25:08.206762+0000","last_unstale":"2024-03-19T02:27:09.782801+0000","last_undegraded":"2024-03-19T02:27:09.782801+0000","last_fullsized":"2024-03-19T02:27:09.782801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:58.357291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T02:27:09.734543+0000","last_change":"2024-03-19T02:25:08.201699+0000","last_active":"2024-03-19T02:27:09.734543+0000","last_peered":"2024-03-19T02:27:09.734543+0000","last_clean":"2024-03-19T02:27:09.734543+0000","last_became_active":"2024-03-19T02:25:08.201397+0000","last_became_peered":"2024-03-19T02:25:08.201397+0000","last_unstale":"2024-03-19T02:27:09.734543+0000","last_undegraded":"2024-03-19T02:27:09.734543+0000","last_fullsized":"2024-03-19T02:27:09.734543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:25:07.621348+0000","last_clean_scrub_stamp":"2024-03-19T02:25:07.621348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:28.939369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71302,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1225,"num_read_kb":687,"num_write":268,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":227186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1769472,"data_stored":1172997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":527197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":523479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299456,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1691648,"data_stored":1121455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":519098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36456,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708696,"statfs":{"total":95995035648,"available":95957704704,"internally_reserved":0,"allocated":1146880,"data_stored":589285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":520554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":590389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":518313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":523172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1699840,"data_stored":1116051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:27:38.535 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 -- 172.21.15.116:0/2751512060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 msgr2=0x7f2a841002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2751512060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841002a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009860 tx=0x7f2a7402bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 -- 172.21.15.116:0/2751512060 shutdown_connections 2024-03-19T02:27:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2751512060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 0x7f2a84108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2751512060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 0x7f2a841014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2751512060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841002a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 -- 172.21.15.116:0/2751512060 >> 172.21.15.116:0/2751512060 conn(0x7f2a840fb440 msgr2=0x7f2a840fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 -- 172.21.15.116:0/2751512060 shutdown_connections 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.671+0000 7f2a8903f640 1 -- 172.21.15.116:0/2751512060 wait complete. 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 Processor -- start 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 -- start start 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 0x7f2a84195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 0x7f2a8419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8410b1a0 con 0x7f2a840ffe80 2024-03-19T02:27:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8410b020 con 0x7f2a84101a20 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8903f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8410b320 con 0x7f2a84101060 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58998/0 (socket says 172.21.15.116:58998) 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 -- 172.21.15.116:0/2981117489 learned_addr learned my addr 172.21.15.116:0/2981117489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 0x7f2a84195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a8359e640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 0x7f2a8419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 -- 172.21.15.116:0/2981117489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 msgr2=0x7f2a84195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 0x7f2a84195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 -- 172.21.15.116:0/2981117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 msgr2=0x7f2a8419ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 0x7f2a8419ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.672+0000 7f2a82d9d640 1 -- 172.21.15.116:0/2981117489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8419d4c0 con 0x7f2a840ffe80 2024-03-19T02:27:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a82d9d640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2a74005940 tx=0x7f2a740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74041020 con 0x7f2a840ffe80 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a740035f0 con 0x7f2a840ffe80 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8419d750 con 0x7f2a840ffe80 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74039040 con 0x7f2a840ffe80 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.673+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8419dc40 con 0x7f2a840ffe80 2024-03-19T02:27:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.674+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a841072a0 con 0x7f2a840ffe80 2024-03-19T02:27:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.675+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f2a74044020 con 0x7f2a840ffe80 2024-03-19T02:27:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.676+0000 7f2a6bfff640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.676+0000 7f2a8259c640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.676+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f2a740f0c40 con 0x7f2a840ffe80 2024-03-19T02:27:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.676+0000 7f2a8259c640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2a84196db0 tx=0x7f2a78011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.680+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a7404b020 con 0x7f2a840ffe80 2024-03-19T02:27:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.858+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a8402d3c0 con 0x7f2a840ffe80 2024-03-19T02:27:38.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.859+0000 7f2a6bfff640 1 -- 172.21.15.116:0/2981117489 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7f2a740bda30 con 0x7f2a840ffe80 2024-03-19T02:27:38.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:38.860 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 msgr2=0x7f2a6c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2a84196db0 tx=0x7f2a78011040 comp rx=0 tx=0).stop 2024-03-19T02:27:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 msgr2=0x7f2a841958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2a74005940 tx=0x7f2a740042f0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 shutdown_connections 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.861+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f2a6c0a2890 0x7f2a6c0a4d50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a84101a20 0x7f2a8419ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a84101060 0x7f2a84195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 --2- 172.21.15.116:0/2981117489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a840ffe80 0x7f2a841958a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 >> 172.21.15.116:0/2981117489 conn(0x7f2a840fb440 msgr2=0x7f2a84104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 shutdown_connections 2024-03-19T02:27:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.862+0000 7f2a8903f640 1 -- 172.21.15.116:0/2981117489 wait complete. 2024-03-19T02:27:38.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:27:38.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 -- 172.21.15.116:0/3509481538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 msgr2=0x7ff380101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 --2- 172.21.15.116:0/3509481538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380101d90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009900 tx=0x7ff37c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 -- 172.21.15.116:0/3509481538 shutdown_connections 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 --2- 172.21.15.116:0/3509481538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 0x7ff380108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 --2- 172.21.15.116:0/3509481538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380101d90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 --2- 172.21.15.116:0/3509481538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 0x7ff380100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 -- 172.21.15.116:0/3509481538 >> 172.21.15.116:0/3509481538 conn(0x7ff3800fbcb0 msgr2=0x7ff3800fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 -- 172.21.15.116:0/3509481538 shutdown_connections 2024-03-19T02:27:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.978+0000 7ff387b60640 1 -- 172.21.15.116:0/3509481538 wait complete. 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 Processor -- start 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 -- start start 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 0x7ff380197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 0x7ff38019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff38010d370 con 0x7ff380101910 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff38010d1f0 con 0x7ff3801022d0 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff387b60640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff38010d4f0 con 0x7ff380100720 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59004/0 (socket says 172.21.15.116:59004) 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 -- 172.21.15.116:0/2605258805 learned_addr learned my addr 172.21.15.116:0/2605258805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3858fc640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 0x7ff380197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3860fd640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 0x7ff38019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 -- 172.21.15.116:0/2605258805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 msgr2=0x7ff380197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 0x7ff380197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 -- 172.21.15.116:0/2605258805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 msgr2=0x7ff38019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 0x7ff38019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 -- 172.21.15.116:0/2605258805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38019f6a0 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff3850fb640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006010 tx=0x7ff37c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.979+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c043020 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.980+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff37c0043d0 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.980+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38019f930 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.980+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c003b80 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.980+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38019fe80 con 0x7ff380101910 2024-03-19T02:27:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.980+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff380107b60 con 0x7ff380101910 2024-03-19T02:27:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.981+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7ff37c046020 con 0x7ff380101910 2024-03-19T02:27:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.982+0000 7ff376ffd640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 0x7ff36c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.982+0000 7ff3858fc640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 0x7ff36c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.982+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff37c0f1050 con 0x7ff380101910 2024-03-19T02:27:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.983+0000 7ff3858fc640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 0x7ff36c0a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff380101770 tx=0x7ff370005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:38.986+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff37c04d020 con 0x7ff380101910 2024-03-19T02:27:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.161+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff38002d3d0 con 0x7ff380101910 2024-03-19T02:27:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.161+0000 7ff376ffd640 1 -- 172.21.15.116:0/2605258805 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7ff37c0bdd20 con 0x7ff380101910 2024-03-19T02:27:39.162 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 msgr2=0x7ff36c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 0x7ff36c0a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff380101770 tx=0x7ff370005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 msgr2=0x7ff380197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006010 tx=0x7ff37c005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 shutdown_connections 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7ff36c0a2890 0x7ff36c0a4d50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3801022d0 0x7ff38019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff380101910 0x7ff380197fc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 --2- 172.21.15.116:0/2605258805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff380100720 0x7ff380197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 >> 172.21.15.116:0/2605258805 conn(0x7ff3800fbcb0 msgr2=0x7ff380104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 shutdown_connections 2024-03-19T02:27:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.163+0000 7ff387b60640 1 -- 172.21.15.116:0/2605258805 wait complete. 2024-03-19T02:27:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:38 smithi162 ceph-mon[35761]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:27:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:38 smithi162 ceph-mon[35761]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:38 smithi162 ceph-mon[35761]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:38 smithi162 ceph-mon[35761]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:38 smithi116 ceph-mon[32263]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:27:39.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:38 smithi116 ceph-mon[32263]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:38 smithi116 ceph-mon[32263]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.174 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:38 smithi116 ceph-mon[32263]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:27:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:38 smithi154 ceph-mon[35693]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:27:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:38 smithi154 ceph-mon[35693]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:38 smithi154 ceph-mon[35693]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:38 smithi154 ceph-mon[35693]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:27:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 -- 172.21.15.116:0/766956081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc41022f0 msgr2=0x7f1fc4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 --2- 172.21.15.116:0/766956081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc41022f0 0x7f1fc4108b80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009ca0 tx=0x7f1fbc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 -- 172.21.15.116:0/766956081 shutdown_connections 2024-03-19T02:27:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 --2- 172.21.15.116:0/766956081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc41022f0 0x7f1fc4108b80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 --2- 172.21.15.116:0/766956081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4101930 0x7f1fc4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 --2- 172.21.15.116:0/766956081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc4100740 0x7f1fc4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 -- 172.21.15.116:0/766956081 >> 172.21.15.116:0/766956081 conn(0x7f1fc40fbcd0 msgr2=0x7f1fc40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 -- 172.21.15.116:0/766956081 shutdown_connections 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 -- 172.21.15.116:0/766956081 wait complete. 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.279+0000 7f1fc8843640 1 Processor -- start 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 -- start start 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 0x7f1fc4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc41022f0 0x7f1fc419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc410d390 con 0x7f1fc4101930 2024-03-19T02:27:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc410d210 con 0x7f1fc4100740 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc8843640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc410d510 con 0x7f1fc41022f0 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59018/0 (socket says 172.21.15.116:59018) 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 -- 172.21.15.116:0/343195952 learned_addr learned my addr 172.21.15.116:0/343195952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc259c640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 0x7f1fc4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc2d9d640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc41022f0 0x7f1fc419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 -- 172.21.15.116:0/343195952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc41022f0 msgr2=0x7f1fc419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc41022f0 0x7f1fc419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 -- 172.21.15.116:0/343195952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 msgr2=0x7f1fc4197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 0x7f1fc4197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 -- 172.21.15.116:0/343195952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc419f660 con 0x7f1fc4101930 2024-03-19T02:27:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fc259c640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 0x7f1fc4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fb9d9b640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00e9f0 tx=0x7f1fac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.280+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac00cd80 con 0x7f1fc4101930 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.281+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fac0108e0 con 0x7f1fc4101930 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.281+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc419f950 con 0x7f1fc4101930 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.281+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac018920 con 0x7f1fc4101930 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.281+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc419fea0 con 0x7f1fc4101930 2024-03-19T02:27:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.281+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc4107ab0 con 0x7f1fc4101930 2024-03-19T02:27:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.282+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f1fac010a90 con 0x7f1fc4101930 2024-03-19T02:27:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.283+0000 7f1fbbfff640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 0x7f1fa40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.283+0000 7f1fc259c640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 0x7f1fa40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.283+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1fac0cf260 con 0x7f1fc4101930 2024-03-19T02:27:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.284+0000 7f1fc259c640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 0x7f1fa40a4d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4101790 tx=0x7f1fb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.288+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fac09b7d0 con 0x7f1fc4101930 2024-03-19T02:27:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.462+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1fc402d3d0 con 0x7f1fc4101930 2024-03-19T02:27:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.462+0000 7f1fbbfff640 1 -- 172.21.15.116:0/343195952 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f1fac09f870 con 0x7f1fc4101930 2024-03-19T02:27:39.463 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 msgr2=0x7f1fa40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 0x7f1fa40a4d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4101790 tx=0x7f1fb4005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 msgr2=0x7f1fc4197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00e9f0 tx=0x7f1fac00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 shutdown_connections 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1fa40a2890 0x7f1fa40a4d50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fc41022f0 0x7f1fc419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fc4101930 0x7f1fc4197f80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 --2- 172.21.15.116:0/343195952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc4100740 0x7f1fc4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.464+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 >> 172.21.15.116:0/343195952 conn(0x7f1fc40fbcd0 msgr2=0x7f1fc4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.465+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 shutdown_connections 2024-03-19T02:27:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.465+0000 7f1fc8843640 1 -- 172.21.15.116:0/343195952 wait complete. 2024-03-19T02:27:39.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- 172.21.15.116:0/4205004507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac101830 msgr2=0x7fd8ac101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/4205004507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac101830 0x7fd8ac101c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009860 tx=0x7fd8a002bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- 172.21.15.116:0/4205004507 shutdown_connections 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/4205004507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac1033e0 0x7fd8ac109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/4205004507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac102a20 0x7fd8ac102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/4205004507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac101830 0x7fd8ac101c50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- 172.21.15.116:0/4205004507 >> 172.21.15.116:0/4205004507 conn(0x7fd8ac0fcdc0 msgr2=0x7fd8ac0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- 172.21.15.116:0/4205004507 shutdown_connections 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- 172.21.15.116:0/4205004507 wait complete. 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 Processor -- start 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.581+0000 7fd8b1a12640 1 -- start start 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 0x7fd8ac197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac1033e0 0x7fd8ac19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac10d330 con 0x7fd8ac102a20 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac10d1b0 con 0x7fd8ac101830 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac10d4b0 con 0x7fd8ac1033e0 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59044/0 (socket says 172.21.15.116:59044) 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 -- 172.21.15.116:0/1794528625 learned_addr learned my addr 172.21.15.116:0/1794528625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8ab7fe640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac1033e0 0x7fd8ac19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aaffd640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 0x7fd8ac197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 -- 172.21.15.116:0/1794528625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac1033e0 msgr2=0x7fd8ac19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac1033e0 0x7fd8ac19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 -- 172.21.15.116:0/1794528625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 msgr2=0x7fd8ac197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 0x7fd8ac197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 -- 172.21.15.116:0/1794528625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac19f6e0 con 0x7fd8ac102a20 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aaffd640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 0x7fd8ac197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8aa7fc640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00e9f0 tx=0x7fd89c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c00cd80 con 0x7fd8ac102a20 2024-03-19T02:27:39.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ac19f9d0 con 0x7fd8ac102a20 2024-03-19T02:27:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd89c0108e0 con 0x7fd8ac102a20 2024-03-19T02:27:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89c018920 con 0x7fd8ac102a20 2024-03-19T02:27:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.582+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ac19ff20 con 0x7fd8ac102a20 2024-03-19T02:27:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.583+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ac108cc0 con 0x7fd8ac102a20 2024-03-19T02:27:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.584+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7fd89c010a90 con 0x7fd8ac102a20 2024-03-19T02:27:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.585+0000 7fd8b0a10640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 0x7fd8940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.585+0000 7fd8aaffd640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 0x7fd8940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.585+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd89c0cf260 con 0x7fd8ac102a20 2024-03-19T02:27:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.585+0000 7fd8aaffd640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 0x7fd8940a4d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd8a002c180 tx=0x7fd8a00349a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.589+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd89c09b7d0 con 0x7fd8ac102a20 2024-03-19T02:27:39.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.767+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd8ac02d440 con 0x7fd8ac102a20 2024-03-19T02:27:39.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.768+0000 7fd8b0a10640 1 -- 172.21.15.116:0/1794528625 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fd89c09f870 con 0x7fd8ac102a20 2024-03-19T02:27:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 msgr2=0x7fd8940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 0x7fd8940a4d50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd8a002c180 tx=0x7fd8a00349a0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 msgr2=0x7fd8ac198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00e9f0 tx=0x7fd89c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 shutdown_connections 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fd8940a2890 0x7fd8940a4d50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8ac1033e0 0x7fd8ac19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8ac102a20 0x7fd8ac198000 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 --2- 172.21.15.116:0/1794528625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8ac101830 0x7fd8ac197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 >> 172.21.15.116:0/1794528625 conn(0x7fd8ac0fcdc0 msgr2=0x7fd8ac105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 shutdown_connections 2024-03-19T02:27:39.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.770+0000 7fd8b1a12640 1 -- 172.21.15.116:0/1794528625 wait complete. 2024-03-19T02:27:39.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- 172.21.15.116:0/363027051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec100720 msgr2=0x7f10ec100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 --2- 172.21.15.116:0/363027051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec100720 0x7f10ec100b40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009900 tx=0x7f10dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- 172.21.15.116:0/363027051 shutdown_connections 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 --2- 172.21.15.116:0/363027051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec1022d0 0x7f10ec108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 --2- 172.21.15.116:0/363027051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 0x7f10ec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 --2- 172.21.15.116:0/363027051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec100720 0x7f10ec100b40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- 172.21.15.116:0/363027051 >> 172.21.15.116:0/363027051 conn(0x7f10ec0fbcb0 msgr2=0x7f10ec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- 172.21.15.116:0/363027051 shutdown_connections 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- 172.21.15.116:0/363027051 wait complete. 2024-03-19T02:27:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 Processor -- start 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.886+0000 7f10f24a7640 1 -- start start 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec100720 0x7f10ec197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 0x7f10ec198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec10c240 con 0x7f10ec1022d0 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec10c0c0 con 0x7f10ec100720 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec10c3c0 con 0x7f10ec101910 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59062/0 (socket says 172.21.15.116:59062) 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 -- 172.21.15.116:0/2163598342 learned_addr learned my addr 172.21.15.116:0/2163598342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10eb7fe640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 0x7f10ec198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10ebfff640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec100720 0x7f10ec197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 -- 172.21.15.116:0/2163598342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 msgr2=0x7f10ec198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 0x7f10ec198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 -- 172.21.15.116:0/2163598342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec100720 msgr2=0x7f10ec197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec100720 0x7f10ec197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 -- 172.21.15.116:0/2163598342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec19f6f0 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f0a44640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f10e400eb10 tx=0x7f10e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4004b20 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ec19f9e0 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10e4004cc0 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4010690 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.887+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ec19ff30 con 0x7f10ec1022d0 2024-03-19T02:27:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.888+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec1a01c0 con 0x7f10ec1022d0 2024-03-19T02:27:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.889+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f10e4005020 con 0x7f10ec1022d0 2024-03-19T02:27:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.890+0000 7f10e97fa640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 0x7f10d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.890+0000 7f10ebfff640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 0x7f10d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.890+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f10e40cf4e0 con 0x7f10ec1022d0 2024-03-19T02:27:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.890+0000 7f10ebfff640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 0x7f10d40a4d50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005ab0 tx=0x7f10dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:39.894+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10e409c0a0 con 0x7f10ec1022d0 2024-03-19T02:27:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.066+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f10ec1a0390 con 0x7f10ec1022d0 2024-03-19T02:27:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.066+0000 7f10e97fa640 1 -- 172.21.15.116:0/2163598342 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f10e409b9a0 con 0x7f10ec1022d0 2024-03-19T02:27:40.067 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:27:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 msgr2=0x7f10d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 0x7f10d40a4d50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005ab0 tx=0x7f10dc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 msgr2=0x7f10ec19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f10e400eb10 tx=0x7f10e400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 shutdown_connections 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f10d40a2890 0x7f10d40a4d50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10ec1022d0 0x7f10ec19f080 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10ec101910 0x7f10ec198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 --2- 172.21.15.116:0/2163598342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10ec100720 0x7f10ec197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.068+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 >> 172.21.15.116:0/2163598342 conn(0x7f10ec0fbcb0 msgr2=0x7f10ec068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.069+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 shutdown_connections 2024-03-19T02:27:40.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.069+0000 7f10f24a7640 1 -- 172.21.15.116:0/2163598342 wait complete. 2024-03-19T02:27:40.079 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:39 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2981117489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:40.079 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:39 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2605258805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.079 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:39 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/343195952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.079 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:39 smithi116 ceph-mon[32263]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-19T02:27:40.079 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:39 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1794528625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:27:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:39 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2981117489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:39 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2605258805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:39 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/343195952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:39 smithi162 ceph-mon[35761]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-19T02:27:40.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:39 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1794528625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.181+0000 7f8d0893b640 1 -- 172.21.15.116:0/4186940004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04101940 msgr2=0x7f8d04101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.181+0000 7f8d0893b640 1 --2- 172.21.15.116:0/4186940004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04101940 0x7f8d04101dc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009860 tx=0x7f8cec02bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- 172.21.15.116:0/4186940004 shutdown_connections 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- 172.21.15.116:0/4186940004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04102300 0x7f8d04108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- 172.21.15.116:0/4186940004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04101940 0x7f8d04101dc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- 172.21.15.116:0/4186940004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04100750 0x7f8d04100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- 172.21.15.116:0/4186940004 >> 172.21.15.116:0/4186940004 conn(0x7f8d040fbd00 msgr2=0x7f8d040fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- 172.21.15.116:0/4186940004 shutdown_connections 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- 172.21.15.116:0/4186940004 wait complete. 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 Processor -- start 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- start start 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04100750 0x7f8d04197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04101940 0x7f8d04197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0410d390 con 0x7f8d04102300 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0410d210 con 0x7f8d04100750 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.182+0000 7f8d0893b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0410d510 con 0x7f8d04101940 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59084/0 (socket says 172.21.15.116:59084) 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 -- 172.21.15.116:0/3578630838 learned_addr learned my addr 172.21.15.116:0/3578630838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d01d9b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04101940 0x7f8d04197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d0259c640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04100750 0x7f8d04197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 -- 172.21.15.116:0/3578630838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04101940 msgr2=0x7f8d04197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04101940 0x7f8d04197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 -- 172.21.15.116:0/3578630838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04100750 msgr2=0x7f8d04197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04100750 0x7f8d04197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 -- 172.21.15.116:0/3578630838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0419f6a0 con 0x7f8d04102300 2024-03-19T02:27:40.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d02d9d640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400e970 tx=0x7f8cf400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400ccc0 con 0x7f8d04102300 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cf400ce60 con 0x7f8d04102300 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0419f990 con 0x7f8d04102300 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4010430 con 0x7f8d04102300 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.183+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0419fe60 con 0x7f8d04102300 2024-03-19T02:27:40.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.184+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d04107a70 con 0x7f8d04102300 2024-03-19T02:27:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.185+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f8cf4004ff0 con 0x7f8d04102300 2024-03-19T02:27:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.186+0000 7f8cfb7fe640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 0x7f8cf00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.186+0000 7f8d0259c640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 0x7f8cf00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.186+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f8cf40cf380 con 0x7f8d04102300 2024-03-19T02:27:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.186+0000 7f8d0259c640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 0x7f8cf00a4d50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8d041017a0 tx=0x7f8cfc010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.190+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8cf409bf40 con 0x7f8d04102300 2024-03-19T02:27:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:39 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2981117489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:39 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2605258805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:39 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/343195952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:39 smithi154 ceph-mon[35693]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-19T02:27:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:39 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1794528625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:27:40.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.403+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d0402d1f0 con 0x7f8d04102300 2024-03-19T02:27:40.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.404+0000 7f8cfb7fe640 1 -- 172.21.15.116:0/3578630838 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11950 (secure 0 0 0) 0x7f8cf409b840 con 0x7f8d04102300 2024-03-19T02:27:40.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:40.405 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:27:33.577980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:27:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 msgr2=0x7f8cf00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 0x7f8cf00a4d50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8d041017a0 tx=0x7f8cfc010040 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 msgr2=0x7f8d0419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400e970 tx=0x7f8cf400ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 shutdown_connections 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f8cf00a2890 0x7f8cf00a4d50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d04102300 0x7f8d0419f030 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d04101940 0x7f8d04197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 --2- 172.21.15.116:0/3578630838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d04100750 0x7f8d04197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 >> 172.21.15.116:0/3578630838 conn(0x7f8d040fbd00 msgr2=0x7f8d04104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 shutdown_connections 2024-03-19T02:27:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.406+0000 7f8d0893b640 1 -- 172.21.15.116:0/3578630838 wait complete. 2024-03-19T02:27:40.418 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.517+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/727538981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a040ffb30 msgr2=0x7f4a040fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.517+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/727538981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a040ffb30 0x7f4a040fff50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009890 tx=0x7f49fc02bb70 comp rx=0 tx=0).stop 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/727538981 shutdown_connections 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/727538981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04103560 0x7f4a04109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/727538981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 0x7f4a04103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/727538981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a040ffb30 0x7f4a040fff50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/727538981 >> 172.21.15.116:0/727538981 conn(0x7f4a040fb440 msgr2=0x7f4a040fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/727538981 shutdown_connections 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/727538981 wait complete. 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 Processor -- start 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- start start 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040ffb30 0x7f4a04197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 0x7f4a04198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0410d390 con 0x7f4a04103560 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0410d210 con 0x7f4a040ffb30 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0c5e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0410d510 con 0x7f4a04102ba0 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.518+0000 7f4a0ab83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0a382640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040ffb30 0x7f4a04197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59092/0 (socket says 172.21.15.116:59092) 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 -- 172.21.15.116:0/1271091146 learned_addr learned my addr 172.21.15.116:0/1271091146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a09b81640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 0x7f4a04198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 -- 172.21.15.116:0/1271091146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 msgr2=0x7f4a04198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 0x7f4a04198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 -- 172.21.15.116:0/1271091146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040ffb30 msgr2=0x7f4a04197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040ffb30 0x7f4a04197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 -- 172.21.15.116:0/1271091146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0419f710 con 0x7f4a04103560 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0ab83640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009fd0 tx=0x7f4a0000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00004b20 con 0x7f4a04103560 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0419fa00 con 0x7f4a04103560 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a00003e20 con 0x7f4a04103560 2024-03-19T02:27:40.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00004820 con 0x7f4a04103560 2024-03-19T02:27:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.519+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0419ff00 con 0x7f4a04103560 2024-03-19T02:27:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.520+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a04108e10 con 0x7f4a04103560 2024-03-19T02:27:40.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.521+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f4a0001d020 con 0x7f4a04103560 2024-03-19T02:27:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.522+0000 7f49fb7fe640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 0x7f49f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.522+0000 7f4a0a382640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 0x7f49f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.522+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f4a000ceef0 con 0x7f4a04103560 2024-03-19T02:27:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.522+0000 7f4a0a382640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 0x7f49f00a4d50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0097b0 tx=0x7f49fc0349a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.526+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a0009bab0 con 0x7f4a04103560 2024-03-19T02:27:40.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.703+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a0402d440 con 0x7f4a04103560 2024-03-19T02:27:40.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.705+0000 7f49fb7fe640 1 -- 172.21.15.116:0/1271091146 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24769 (secure 0 0 0) 0x7f4a0009b3b0 con 0x7f4a04103560 2024-03-19T02:27:40.706 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:27:40.706 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:27:09.726326+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 msgr2=0x7f49f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 0x7f49f00a4d50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0097b0 tx=0x7f49fc0349a0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 msgr2=0x7f4a0419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009fd0 tx=0x7f4a0000c800 comp rx=0 tx=0).stop 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 shutdown_connections 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f49f00a2890 0x7f49f00a4d50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a04103560 0x7f4a0419f0a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a04102ba0 0x7f4a04198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 --2- 172.21.15.116:0/1271091146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040ffb30 0x7f4a04197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 >> 172.21.15.116:0/1271091146 conn(0x7f4a040fb440 msgr2=0x7f4a04105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 shutdown_connections 2024-03-19T02:27:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.707+0000 7f4a0c5e6640 1 -- 172.21.15.116:0/1271091146 wait complete. 2024-03-19T02:27:40.719 DEBUG:tasks.fwd_scrub:Yielding 2024-03-19T02:27:40.719 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T02:27:40.729 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:27:40.729 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:27:40.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-19T02:27:40.729 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-19T02:27:40.729 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:27:40.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:27:40.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:27:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 -- 172.21.15.116:0/4293905190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c1075f0 msgr2=0x7f1f7c1099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 --2- 172.21.15.116:0/4293905190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c1099e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009be0 tx=0x7f1f7402bef0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 -- 172.21.15.116:0/4293905190 shutdown_connections 2024-03-19T02:27:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 --2- 172.21.15.116:0/4293905190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c1099e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 --2- 172.21.15.116:0/4293905190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c104cc0 0x7f1f7c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 --2- 172.21.15.116:0/4293905190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c102390 0x7f1f7c104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 -- 172.21.15.116:0/4293905190 >> 172.21.15.116:0/4293905190 conn(0x7f1f7c0fbcc0 msgr2=0x7f1f7c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 -- 172.21.15.116:0/4293905190 shutdown_connections 2024-03-19T02:27:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.835+0000 7f1f8144d640 1 -- 172.21.15.116:0/4293905190 wait complete. 2024-03-19T02:27:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 Processor -- start 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 -- start start 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c104cc0 0x7f1f7c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c10d3a0 con 0x7f1f7c102390 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c10d220 con 0x7f1f7c1075f0 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f8144d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c10d520 con 0x7f1f7c104cc0 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.836+0000 7f1f7affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c104cc0 0x7f1f7c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59118/0 (socket says 172.21.15.116:59118) 2024-03-19T02:27:40.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7b7fe640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 -- 172.21.15.116:0/2643113296 learned_addr learned my addr 172.21.15.116:0/2643113296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 -- 172.21.15.116:0/2643113296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c104cc0 msgr2=0x7f1f7c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c104cc0 0x7f1f7c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 -- 172.21.15.116:0/2643113296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c1075f0 msgr2=0x7f1f7c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 -- 172.21.15.116:0/2643113296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c19f5b0 con 0x7f1f7c102390 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f7affd640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e9c0 tx=0x7f1f6c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00cde0 con 0x7f1f7c102390 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1f6c0035f0 con 0x7f1f7c102390 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c19f8a0 con 0x7f1f7c102390 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c010690 con 0x7f1f7c102390 2024-03-19T02:27:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c19fdf0 con 0x7f1f7c102390 2024-03-19T02:27:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7fb237bb9640 1 -- 172.21.15.116:0/2074939130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230102a40 msgr2=0x7fb230102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.837+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2074939130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230102a40 0x7fb230102ec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb220009900 tx=0x7fb22002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- 172.21.15.116:0/2074939130 shutdown_connections 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2074939130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230103400 0x7fb230109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2074939130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230102a40 0x7fb230102ec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2074939130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb230101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- 172.21.15.116:0/2074939130 >> 172.21.15.116:0/2074939130 conn(0x7fb2300fcdc0 msgr2=0x7fb2300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- 172.21.15.116:0/2074939130 shutdown_connections 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- 172.21.15.116:0/2074939130 wait complete. 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 Processor -- start 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- start start 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230103400 0x7fb230073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c0fd7b0 con 0x7f1f7c102390 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230074150 0x7fb230074600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010d3d0 con 0x7fb230074150 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010d250 con 0x7fb230103400 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.838+0000 7fb237bb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb23010d550 con 0x7fb230101850 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52912/0 (socket says 172.21.15.116:52912) 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb236156640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230074150 0x7fb230074600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 -- 172.21.15.116:0/2845630591 learned_addr learned my addr 172.21.15.116:0/2845630591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235154640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230103400 0x7fb230073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 -- 172.21.15.116:0/2845630591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230103400 msgr2=0x7fb230073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230103400 0x7fb230073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 -- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230074150 msgr2=0x7fb230074600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230074150 0x7fb230074600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 -- 172.21.15.116:0/2845630591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301a1f00 con 0x7fb230101850 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb235955640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00e9c0 tx=0x7fb22c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb226ffd640 1 -- 172.21.15.116:0/2845630591 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00cd80 con 0x7fb230101850 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301a21f0 con 0x7fb230101850 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb226ffd640 1 -- 172.21.15.116:0/2845630591 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb22c0035f0 con 0x7fb230101850 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb226ffd640 1 -- 172.21.15.116:0/2845630591 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c010690 con 0x7fb230101850 2024-03-19T02:27:40.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2301a2710 con 0x7fb230101850 2024-03-19T02:27:40.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.839+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f1f6c0037a0 con 0x7f1f7c102390 2024-03-19T02:27:40.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.840+0000 7f1f63fff640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 0x7f1f640a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.840+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1f6c0cf290 con 0x7f1f7c102390 2024-03-19T02:27:40.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.840+0000 7f1f7a7fc640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 0x7f1f640a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.841+0000 7f1f7a7fc640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 0x7f1f640a4e80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1f700095f0 tx=0x7f1f70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.841+0000 7fb226ffd640 1 -- 172.21.15.116:0/2845630591 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7fb22c0037a0 con 0x7fb230101850 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.842+0000 7fb226ffd640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 0x7fb21c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.842+0000 7fb235154640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 0x7fb21c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.842+0000 7fb226ffd640 1 -- 172.21.15.116:0/2845630591 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb22c0cf4c0 con 0x7fb230101850 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.842+0000 7fb235154640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 0x7fb21c0a4d50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb220036040 tx=0x7fb22002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.843+0000 7fb23a7a2740 1 Processor -- start 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.843+0000 7fb23a7a2740 1 -- start start 2024-03-19T02:27:40.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x5628558fc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x562855904390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59146/0 (socket says 172.21.15.116:59146) 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 -- 172.21.15.116:0/1732603818 learned_addr learned my addr 172.21.15.116:0/1732603818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562855861160 con 0x5628558fcd90 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562855809e10 con 0x5628558fc3a0 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56285585d200 con 0x562855901f00 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235955640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x5628558fc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb236156640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x562855904390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 -- 172.21.15.116:0/1732603818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 msgr2=0x562855904390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x562855904390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 -- 172.21.15.116:0/1732603818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 msgr2=0x5628558fc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x5628558fc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 -- 172.21.15.116:0/1732603818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56285563d7e0 con 0x5628558fcd90 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235955640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x5628558fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.844+0000 7fb235154640 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b00528d587f16d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.847+0000 7fb21bfff640 1 -- 172.21.15.116:0/1732603818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22002cb20 con 0x5628558fcd90 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.847+0000 7fb21bfff640 1 -- 172.21.15.116:0/1732603818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb220050070 con 0x5628558fcd90 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.847+0000 7fb21bfff640 1 -- 172.21.15.116:0/1732603818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220004cc0 con 0x5628558fcd90 2024-03-19T02:27:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.847+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f6c009510 con 0x7f1f7c102390 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 msgr2=0x562855901890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22003e000 comp rx=0 tx=0).stop 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 shutdown_connections 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x562855904390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x562855901890 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1732603818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x5628558fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 >> 172.21.15.116:0/1732603818 conn(0x5628558f7910 msgr2=0x5628558f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 shutdown_connections 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1732603818 wait complete. 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 -- start start 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.848+0000 7fb23a7a2740 1 Processor -- start 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fcd90 0x562855906ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562855904e50 con 0x5628558fc3a0 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56285581d1a0 con 0x5628558fcd90 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562855904fd0 con 0x562855901f00 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb23a7a2740 10 client.0 _ll_register_callbacks cb 0x562855574c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59150/0 (socket says 172.21.15.116:59150) 2024-03-19T02:27:40.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 -- 172.21.15.116:0/3345264565 learned_addr learned my addr 172.21.15.116:0/3345264565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb236156640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235154640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fcd90 0x562855906ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 -- 172.21.15.116:0/3345264565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 msgr2=0x56285590dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 -- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fcd90 msgr2=0x562855906ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fcd90 0x562855906ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 -- 172.21.15.116:0/3345264565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56285569df70 con 0x5628558fc3a0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb235955640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009fd0 tx=0x7fb22c003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a9377cf0525b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb2197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.849+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0b4060 con 0x5628558fc3a0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb22c0860d0 con 0x5628558fc3a0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb220050070 con 0x5628558fc3a0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb22c09fad0 con 0x5628558fc3a0 2024-03-19T02:27:40.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 10 client.15684 fetch_fsmap learned FSMap version 28 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562855535170 con 0x5628558fc3a0 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb220050070 con 0x5628558fc3a0 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb22c097760 con 0x5628558fc3a0 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 10 client.15684 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 10 client.15684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 20 client.15684 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855912510 0x562855914980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 4 client.15684 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56285586e6a0 con 0x562855912510 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.850+0000 7fb235154640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855912510 0x562855914980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb235154640 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855912510 0x562855914980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe6797308c859fc server_cookie=a907134f7aeffe0d in_seq=0 out_seq=0 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb2197fa640 10 client.15684 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:40.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb2197fa640 1 -- 172.21.15.116:0/3345264565 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56285586e6a0 con 0x562855912510 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb2197fa640 10 client.15684 handle_command_reply: tid=0 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb23a7a2740 1 client.15684 shutdown 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 msgr2=0x562855906760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.851+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009fd0 tx=0x7fb22c003c50 comp rx=0 tx=0).stop 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 shutdown_connections 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855912510 0x562855914980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fcd90 0x562855906ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/3345264565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fc3a0 0x562855906760 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb2197fa640 0 client.15684 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 >> 172.21.15.116:0/3345264565 conn(0x5628558f7910 msgr2=0x5628558f6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 shutdown_connections 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 1 -- 172.21.15.116:0/3345264565 wait complete. 2024-03-19T02:27:40.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.852+0000 7fb23a7a2740 20 client.15684 trim_cache size 0 max 16384 2024-03-19T02:27:40.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 Processor -- start 2024-03-19T02:27:40.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 -- start start 2024-03-19T02:27:40.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628558fc3a0 0x562855906a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562855901f00 0x5628558f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301aca70 con 0x5628558fcd90 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301ae5b0 con 0x562855901f00 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301ad140 con 0x5628558fc3a0 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59152/0 (socket says 172.21.15.116:59152) 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb235154640 1 -- 172.21.15.116:0/328122904 learned_addr learned my addr 172.21.15.116:0/328122904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb235955640 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628558fc3a0 0x562855906a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.869+0000 7fb236156640 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562855901f00 0x5628558f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 -- 172.21.15.116:0/328122904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628558fc3a0 msgr2=0x562855906a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628558fc3a0 0x562855906a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 -- 172.21.15.116:0/328122904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562855901f00 msgr2=0x5628558f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562855901f00 0x5628558f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 -- 172.21.15.116:0/328122904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628558f8680 con 0x5628558fcd90 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb235154640 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb22002c6a0 tx=0x7fb220042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d4a2751e9b3a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb21b7fe640 1 -- 172.21.15.116:0/328122904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22002cc20 con 0x5628558fcd90 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb21b7fe640 1 -- 172.21.15.116:0/328122904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb220050030 con 0x5628558fcd90 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.870+0000 7fb21b7fe640 1 -- 172.21.15.116:0/328122904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220004920 con 0x5628558fcd90 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- 172.21.15.116:0/328122904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 msgr2=0x56285590dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb22002c6a0 tx=0x7fb220042000 comp rx=0 tx=0).stop 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- 172.21.15.116:0/328122904 shutdown_connections 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562855901f00 0x5628558f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590dca0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/328122904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628558fc3a0 0x562855906a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- 172.21.15.116:0/328122904 >> 172.21.15.116:0/328122904 conn(0x5628558f7910 msgr2=0x562855906370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- 172.21.15.116:0/328122904 shutdown_connections 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- 172.21.15.116:0/328122904 wait complete. 2024-03-19T02:27:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- start start 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 Processor -- start 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x56285590b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301adee0 con 0x5628558fcd90 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301ad810 con 0x5628558fc3a0 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301aaa50 con 0x562855901f00 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb23a7a2740 10 client.0 _ll_register_callbacks cb 0x7fb22c003b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.871+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59166/0 (socket says 172.21.15.116:59166) 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 -- 172.21.15.116:0/1709487311 learned_addr learned my addr 172.21.15.116:0/1709487311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb236156640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235955640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x56285590b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 -- 172.21.15.116:0/1709487311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 msgr2=0x56285590ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 -- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 msgr2=0x56285590b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x56285590b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 -- 172.21.15.116:0/1709487311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56285590d430 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb235154640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb22002bc70 tx=0x7fb22003a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1b14bccf7178a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb2197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22002cca0 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb220060ab0 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb220050030 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb220060cb0 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb23a7a2740 10 client.15690 fetch_fsmap learned FSMap version 28 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562855978da0 con 0x5628558fcd90 2024-03-19T02:27:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.872+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb220050030 con 0x5628558fcd90 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb220070030 con 0x5628558fcd90 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 10 client.15690 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 20 client.15690 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855979390 0x56285597b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 4 client.15690 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5628558689f0 con 0x562855979390 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb235955640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855979390 0x56285597b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb235955640 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855979390 0x56285597b780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb77b82753db3596 server_cookie=e863d69dc3b26289 in_seq=0 out_seq=0 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.873+0000 7fb2197fa640 10 client.15690 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb2197fa640 1 -- 172.21.15.116:0/1709487311 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5628558689f0 con 0x562855979390 2024-03-19T02:27:40.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb2197fa640 10 client.15690 handle_command_reply: tid=0 2024-03-19T02:27:40.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 client.15690 shutdown 2024-03-19T02:27:40.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 msgr2=0x56285590be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb22002bc70 tx=0x7fb22003a110 comp rx=0 tx=0).stop 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 shutdown_connections 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562855979390 0x56285597b780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562855901f00 0x56285590ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628558fcd90 0x56285590be30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb2197fa640 0 client.15690 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 --2- 172.21.15.116:0/1709487311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5628558fc3a0 0x56285590b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 >> 172.21.15.116:0/1709487311 conn(0x5628558f7910 msgr2=0x5628558fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 shutdown_connections 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.874+0000 7fb23a7a2740 1 -- 172.21.15.116:0/1709487311 wait complete. 2024-03-19T02:27:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb23a7a2740 20 client.15690 trim_cache size 0 max 16384 2024-03-19T02:27:40.882 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:40.882 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:27:40.882 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "428d65db-d4b3-45fe-9409-00a75e7d56d4", 2024-03-19T02:27:40.882 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:27:40.882 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 msgr2=0x7fb21c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 0x7fb21c0a4d50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb220036040 tx=0x7fb22002c7e0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 msgr2=0x7fb2300755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00e9c0 tx=0x7fb22c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 shutdown_connections 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7fb21c0a2890 0x7fb21c0a4d50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230074150 0x7fb230074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb230103400 0x7fb230073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 --2- 172.21.15.116:0/2845630591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb230101850 0x7fb2300755d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 >> 172.21.15.116:0/2845630591 conn(0x7fb2300fcdc0 msgr2=0x7fb230105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 shutdown_connections 2024-03-19T02:27:40.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.875+0000 7fb237bb9640 1 -- 172.21.15.116:0/2845630591 wait complete. 2024-03-19T02:27:40.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:27:40.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- 172.21.15.116:0/1560876046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 msgr2=0x7f6f2c109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- 172.21.15.116:0/1560876046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c109c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009900 tx=0x7f6f2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- 172.21.15.116:0/1560876046 shutdown_connections 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- 172.21.15.116:0/1560876046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c109c90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- 172.21.15.116:0/1560876046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 0x7f6f2c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- 172.21.15.116:0/1560876046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 0x7f6f2c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- 172.21.15.116:0/1560876046 >> 172.21.15.116:0/1560876046 conn(0x7f6f2c0fcdc0 msgr2=0x7f6f2c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- 172.21.15.116:0/1560876046 shutdown_connections 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- 172.21.15.116:0/1560876046 wait complete. 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 Processor -- start 2024-03-19T02:27:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- start start 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 0x7f6f2c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 0x7f6f2c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10d360 con 0x7f6f2c103400 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10d1e0 con 0x7f6f2c102a40 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.992+0000 7f6f33753640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10d4e0 con 0x7f6f2c101850 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59188/0 (socket says 172.21.15.116:59188) 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 0x7f6f2c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 -- 172.21.15.116:0/2288927658 learned_addr learned my addr 172.21.15.116:0/2288927658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 0x7f6f2c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 -- 172.21.15.116:0/2288927658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 msgr2=0x7f6f2c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 0x7f6f2c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 -- 172.21.15.116:0/2288927658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 msgr2=0x7f6f2c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 0x7f6f2c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 -- 172.21.15.116:0/2288927658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c19f650 con 0x7f6f2c103400 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f31cf0640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009230 tx=0x7f6f28005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f227fc640 1 -- 172.21.15.116:0/2288927658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28043020 con 0x7f6f2c103400 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f227fc640 1 -- 172.21.15.116:0/2288927658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f280043d0 con 0x7f6f2c103400 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2c19f8e0 con 0x7f6f2c103400 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f227fc640 1 -- 172.21.15.116:0/2288927658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f28003b40 con 0x7f6f2c103400 2024-03-19T02:27:40.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.993+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2c19fe30 con 0x7f6f2c103400 2024-03-19T02:27:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.995+0000 7f6f227fc640 1 -- 172.21.15.116:0/2288927658 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 137342+0+0 (secure 0 0 0) 0x7f6f28046020 con 0x7f6f2c103400 2024-03-19T02:27:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 Processor -- start 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 -- start start 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f70be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f713a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177f670500 con 0x56177f70c440 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56177f618be0 con 0x56177f7115b0 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f3641d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177f66c5a0 con 0x56177f70ba50 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f227fc640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 0x7f6f180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59204/0 (socket says 172.21.15.116:59204) 2024-03-19T02:27:41.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f30cee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f70be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f314ef640 1 -- 172.21.15.116:0/1018091713 learned_addr learned my addr 172.21.15.116:0/1018091713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f31cf0640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 0x7f6f180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.996+0000 7f6f30cee640 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f713a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f227fc640 1 -- 172.21.15.116:0/2288927658 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6f280f0fe0 con 0x7f6f2c103400 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 -- 172.21.15.116:0/1018091713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 msgr2=0x56177f70be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f70be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 -- 172.21.15.116:0/1018091713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 msgr2=0x56177f713a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f713a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 -- 172.21.15.116:0/1018091713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177f63a000 con 0x56177f70c440 2024-03-19T02:27:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f30cee640 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f70be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f314ef640 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f24011470 tx=0x7f6f24011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139e5af334513aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f31cf0640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 0x7f6f180a4d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6f28006010 tx=0x7f6f2808f230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f177fe640 1 -- 172.21.15.116:0/1018091713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24003970 con 0x56177f70c440 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f177fe640 1 -- 172.21.15.116:0/1018091713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f24004020 con 0x56177f70c440 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.997+0000 7f6f177fe640 1 -- 172.21.15.116:0/1018091713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400f860 con 0x56177f70c440 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- 172.21.15.116:0/1018091713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 msgr2=0x56177f710f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f24011470 tx=0x7f6f24011930 comp rx=0 tx=0).stop 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- 172.21.15.116:0/1018091713 shutdown_connections 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f713a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f710f40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 --2- 172.21.15.116:0/1018091713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f70be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- 172.21.15.116:0/1018091713 >> 172.21.15.116:0/1018091713 conn(0x56177f706fc0 msgr2=0x56177f7073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- 172.21.15.116:0/1018091713 shutdown_connections 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- 172.21.15.116:0/1018091713 wait complete. 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 -- start start 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.998+0000 7f6f3641d740 1 Processor -- start 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f70c440 0x56177f7163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f7115b0 0x56177f71d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177f714560 con 0x56177f70ba50 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56177f62c160 con 0x56177f70c440 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177f7146e0 con 0x56177f7115b0 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 10 client.0 _ll_register_callbacks cb 0x56177f668a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:41.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59210/0 (socket says 172.21.15.116:59210) 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 -- 172.21.15.116:0/2677877302 learned_addr learned my addr 172.21.15.116:0/2677877302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f70c440 0x56177f7163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f7115b0 0x56177f71d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 -- 172.21.15.116:0/2677877302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f7115b0 msgr2=0x56177f71d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f7115b0 0x56177f71d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 -- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f70c440 msgr2=0x56177f7163e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f70c440 0x56177f7163e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 -- 172.21.15.116:0/2677877302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177f44e7d0 con 0x56177f70ba50 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c0037a0 tx=0x7f6f1c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b321ee0024af0e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c011b50 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6f1c011cf0 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f24004020 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f1c02b050 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 10 client.15702 fetch_fsmap learned FSMap version 28 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:40.999+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56177f3843f0 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f24004020 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6f1c01e3c0 con 0x56177f70ba50 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 10 client.15702 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 20 client.15702 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:41.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f724110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 4 client.15702 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56177f67da40 con 0x56177f721c70 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f724110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f724110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32e8773bac69a95 server_cookie=8ef3cdaf14bc5fc3 in_seq=0 out_seq=0 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.000+0000 7f6f14ff9640 10 client.15702 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2677877302 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56177f67da40 con 0x56177f721c70 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f14ff9640 10 client.15702 handle_command_reply: tid=0 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 client.15702 shutdown 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 msgr2=0x56177f715ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c0037a0 tx=0x7f6f1c005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 shutdown_connections 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f724110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f7115b0 0x56177f71d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f14ff9640 0 client.15702 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f70c440 0x56177f7163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2677877302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70ba50 0x56177f715ea0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 >> 172.21.15.116:0/2677877302 conn(0x56177f706fc0 msgr2=0x56177f7061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 shutdown_connections 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 1 -- 172.21.15.116:0/2677877302 wait complete. 2024-03-19T02:27:41.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.001+0000 7f6f3641d740 20 client.15702 trim_cache size 0 max 16384 2024-03-19T02:27:41.012 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:40 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2163598342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:41.013 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:40 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3578630838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.013 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:40 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1271091146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.018+0000 7f6f3641d740 1 Processor -- start 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.018+0000 7f6f3641d740 1 -- start start 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f716120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f705e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1aa340 con 0x56177f70c440 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1abe80 con 0x56177f7115b0 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f3641d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1aaa10 con 0x56177f70ba50 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59216/0 (socket says 172.21.15.116:59216) 2024-03-19T02:27:41.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f30cee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f716120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 -- 172.21.15.116:0/43858231 learned_addr learned my addr 172.21.15.116:0/43858231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f30cee640 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f705e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 -- 172.21.15.116:0/43858231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 msgr2=0x56177f716120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f716120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 -- 172.21.15.116:0/43858231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 msgr2=0x56177f705e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f705e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 -- 172.21.15.116:0/43858231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177f3440f0 con 0x56177f70c440 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f314ef640 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6f24014040 tx=0x7f6f24003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2a7ef5614669d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f16ffd640 1 -- 172.21.15.116:0/43858231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24003c80 con 0x56177f70c440 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f16ffd640 1 -- 172.21.15.116:0/43858231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f24003e20 con 0x56177f70c440 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.019+0000 7f6f16ffd640 1 -- 172.21.15.116:0/43858231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24005040 con 0x56177f70c440 2024-03-19T02:27:41.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- 172.21.15.116:0/43858231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 msgr2=0x56177f705950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6f24014040 tx=0x7f6f24003b90 comp rx=0 tx=0).stop 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- 172.21.15.116:0/43858231 shutdown_connections 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f705e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f705950 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 --2- 172.21.15.116:0/43858231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f716120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- 172.21.15.116:0/43858231 >> 172.21.15.116:0/43858231 conn(0x56177f706fc0 msgr2=0x56177f715370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- 172.21.15.116:0/43858231 shutdown_connections 2024-03-19T02:27:41.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- 172.21.15.116:0/43858231 wait complete. 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 -- start start 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.020+0000 7f6f3641d740 1 Processor -- start 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f71a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f71b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1ab7b0 con 0x56177f70c440 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1ab0e0 con 0x56177f7115b0 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c1a8320 con 0x56177f70ba50 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f3641d740 10 client.0 _ll_register_callbacks cb 0x7f6f1c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59220/0 (socket says 172.21.15.116:59220) 2024-03-19T02:27:41.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 -- 172.21.15.116:0/2024626134 learned_addr learned my addr 172.21.15.116:0/2024626134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f71a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f71b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 -- 172.21.15.116:0/2024626134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 msgr2=0x56177f71a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f71a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 -- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 msgr2=0x56177f71b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f71b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 -- 172.21.15.116:0/2024626134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177f71be40 con 0x56177f70c440 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f71a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f314ef640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6f2401b040 tx=0x7f6f24005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29b3e4bf1c45ee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.021+0000 7f6f14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:27:41.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24003c80 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6f24003fd0 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f24003e20 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f24041050 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 10 client.15708 fetch_fsmap learned FSMap version 28 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56177f71d150 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f24003e20 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6f2402d020 con 0x56177f70c440 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 10 client.15708 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 20 client.15708 populate_metadata read hostname 'smithi116' 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f78a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 4 client.15708 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56177f677d90 con 0x56177f721c70 2024-03-19T02:27:41.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.022+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f78a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f14ff9640 10 client.15708 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f30cee640 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f78a2c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c305c0c7d15328b3 server_cookie=352c99b1318afba7 in_seq=0 out_seq=0 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f14ff9640 1 -- 172.21.15.116:0/2024626134 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56177f677d90 con 0x56177f721c70 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f14ff9640 10 client.15708 handle_command_reply: tid=0 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 client.15708 shutdown 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 msgr2=0x56177f71a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6f2401b040 tx=0x7f6f24005fc0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 shutdown_connections 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56177f721c70 0x56177f78a2c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56177f7115b0 0x56177f71b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177f70c440 0x56177f71a840 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 --2- 172.21.15.116:0/2024626134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177f70ba50 0x56177f71a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 >> 172.21.15.116:0/2024626134 conn(0x56177f706fc0 msgr2=0x56177f715370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f14ff9640 0 client.15708 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 shutdown_connections 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 1 -- 172.21.15.116:0/2024626134 wait complete. 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.023+0000 7f6f3641d740 20 client.15708 trim_cache size 0 max 16384 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stdout: "428d65db-d4b3-45fe-9409-00a75e7d56d4": { 2024-03-19T02:27:41.033 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:27:41.034 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "428d65db-d4b3-45fe-9409-00a75e7d56d4", 2024-03-19T02:27:41.034 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:27:41.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:27:41.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:27:41.034 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:27:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 msgr2=0x7f6f180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 0x7f6f180a4d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6f28006010 tx=0x7f6f2808f230 comp rx=0 tx=0).stop 2024-03-19T02:27:41.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 msgr2=0x7f6f2c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6f28009230 tx=0x7f6f28005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 shutdown_connections 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f6f180a2890 0x7f6f180a4d50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c103400 0x7f6f2c19efe0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c102a40 0x7f6f2c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 --2- 172.21.15.116:0/2288927658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f2c101850 0x7f6f2c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 >> 172.21.15.116:0/2288927658 conn(0x7f6f2c0fcdc0 msgr2=0x7f6f2c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 shutdown_connections 2024-03-19T02:27:41.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.024+0000 7f6f33753640 1 -- 172.21.15.116:0/2288927658 wait complete. 2024-03-19T02:27:41.037 INFO:tasks.cephfs.filesystem:scrub status for tag:428d65db-d4b3-45fe-9409-00a75e7d56d4 - {'path': '/', 'tag': '428d65db-d4b3-45fe-9409-00a75e7d56d4', 'options': 'recursive,force'} 2024-03-19T02:27:41.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.085+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f1f7c02d1f0 con 0x7f1f7c102390 2024-03-19T02:27:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:40 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2163598342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:40 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3578630838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:40 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1271091146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:40 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2163598342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:27:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:40 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3578630838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:40 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1271091146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:27:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.810+0000 7f1f63fff640 1 -- 172.21.15.116:0/2643113296 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) ==== 92+0+0 (secure 0 0 0) 0x7f1f6c09bdf0 con 0x7f1f7c102390 2024-03-19T02:27:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 msgr2=0x7f1f640a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 0x7f1f640a4e80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1f700095f0 tx=0x7f1f70005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 msgr2=0x7f1f7c197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e9c0 tx=0x7f1f6c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 shutdown_connections 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f1f640a29c0 0x7f1f640a4e80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f7c1075f0 0x7f1f7c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f7c104cc0 0x7f1f7c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 --2- 172.21.15.116:0/2643113296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f7c102390 0x7f1f7c197990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.812+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 >> 172.21.15.116:0/2643113296 conn(0x7f1f7c0fbcc0 msgr2=0x7f1f7c102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.813+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 shutdown_connections 2024-03-19T02:27:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.813+0000 7f1f8144d640 1 -- 172.21.15.116:0/2643113296 wait complete. 2024-03-19T02:27:41.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-19T02:27:41.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- 172.21.15.116:0/1122239429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 msgr2=0x7f2750101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- 172.21.15.116:0/1122239429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f2750101db0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2740009900 tx=0x7f274002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- 172.21.15.116:0/1122239429 shutdown_connections 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- 172.21.15.116:0/1122239429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27501022f0 0x7f2750108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- 172.21.15.116:0/1122239429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f2750101db0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- 172.21.15.116:0/1122239429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2750100740 0x7f2750100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- 172.21.15.116:0/1122239429 >> 172.21.15.116:0/1122239429 conn(0x7f27500fbcb0 msgr2=0x7f27500fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- 172.21.15.116:0/1122239429 shutdown_connections 2024-03-19T02:27:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- 172.21.15.116:0/1122239429 wait complete. 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 Processor -- start 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- start start 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750100740 0x7f2750069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27501022f0 0x7f275019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f275010c270 con 0x7f2750101930 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f275010c0f0 con 0x7f27501022f0 2024-03-19T02:27:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.940+0000 7f2757e4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f275010c3f0 con 0x7f2750100740 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59266/0 (socket says 172.21.15.116:59266) 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 -- 172.21.15.116:0/2233979700 learned_addr learned my addr 172.21.15.116:0/2233979700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2755be7640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750100740 0x7f2750069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27563e8640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27501022f0 0x7f275019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 -- 172.21.15.116:0/2233979700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750100740 msgr2=0x7f2750069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750100740 0x7f2750069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 -- 172.21.15.116:0/2233979700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27501022f0 msgr2=0x7f275019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27501022f0 0x7f275019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 -- 172.21.15.116:0/2233979700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275019f660 con 0x7f2750101930 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f27553e6640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2740006010 tx=0x7f2740005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740043020 con 0x7f2750101930 2024-03-19T02:27:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27400043d0 con 0x7f2750101930 2024-03-19T02:27:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275019f890 con 0x7f2750101930 2024-03-19T02:27:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740003b80 con 0x7f2750101930 2024-03-19T02:27:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.941+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275019fde0 con 0x7f2750101930 2024-03-19T02:27:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.942+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2750107a80 con 0x7f2750101930 2024-03-19T02:27:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.943+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 137359+0+0 (secure 0 0 0) 0x7f2740046020 con 0x7f2750101930 2024-03-19T02:27:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.944+0000 7f2746ffd640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 0x7f273c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.944+0000 7f2755be7640 1 -- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 msgr2=0x7f273c0a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.944+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f27400f10d0 con 0x7f2750101930 2024-03-19T02:27:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.944+0000 7f2755be7640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 0x7f273c0a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:27:41.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:41.948+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f274004d020 con 0x7f2750101930 2024-03-19T02:27:42.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setuser ceph since I am not root 2024-03-19T02:27:42.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: ignoring --setgroup ceph since I am not root 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:27:42.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:41 smithi116 ceph-mon[32263]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-19T02:27:42.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.131+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f275002d1f0 con 0x7f2750101930 2024-03-19T02:27:42.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.134+0000 7f2746ffd640 1 -- 172.21.15.116:0/2233979700 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f27400bdda0 con 0x7f2750101930 2024-03-19T02:27:42.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 msgr2=0x7f273c0a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:27:42.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 0x7f273c0a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 msgr2=0x7f275019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2740006010 tx=0x7f2740005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 shutdown_connections 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f273c0a2990 0x7f273c0a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27501022f0 0x7f275019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2750101930 0x7f275019c730 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 --2- 172.21.15.116:0/2233979700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2750100740 0x7f2750069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 >> 172.21.15.116:0/2233979700 conn(0x7f27500fbcb0 msgr2=0x7f2750104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 shutdown_connections 2024-03-19T02:27:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.137+0000 7f2757e4b640 1 -- 172.21.15.116:0/2233979700 wait complete. 2024-03-19T02:27:42.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-19T02:27:42.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: ignoring --setuser ceph since I am not root 2024-03-19T02:27:42.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: ignoring --setgroup ceph since I am not root 2024-03-19T02:27:42.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mgr[37603]: -- 172.21.15.162:0/3712185891 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574069345a0 con 0x557406908c00 2024-03-19T02:27:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:27:42.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:27:42.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:27:42.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:41 smithi162 ceph-mon[35761]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 -- 172.21.15.116:0/923223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 msgr2=0x7f3544101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 --2- 172.21.15.116:0/923223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544101db0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3534009900 tx=0x7f353402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 -- 172.21.15.116:0/923223567 shutdown_connections 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 --2- 172.21.15.116:0/923223567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35441022f0 0x7f3544108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 --2- 172.21.15.116:0/923223567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544101db0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 --2- 172.21.15.116:0/923223567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3544100740 0x7f3544100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 -- 172.21.15.116:0/923223567 >> 172.21.15.116:0/923223567 conn(0x7f35440fbcb0 msgr2=0x7f35440fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 -- 172.21.15.116:0/923223567 shutdown_connections 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 -- 172.21.15.116:0/923223567 wait complete. 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.252+0000 7f354b2e8640 1 Processor -- start 2024-03-19T02:27:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- start start 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3544100740 0x7f35441979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35441022f0 0x7f354419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f354410c250 con 0x7f3544101930 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f354410c0d0 con 0x7f3544100740 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f354410c3d0 con 0x7f35441022f0 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59274/0 (socket says 172.21.15.116:59274) 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 -- 172.21.15.116:0/2673680773 learned_addr learned my addr 172.21.15.116:0/2673680773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3549084640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3544100740 0x7f35441979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3549885640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35441022f0 0x7f354419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 -- 172.21.15.116:0/2673680773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35441022f0 msgr2=0x7f354419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35441022f0 0x7f354419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 -- 172.21.15.116:0/2673680773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3544100740 msgr2=0x7f35441979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3544100740 0x7f35441979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 -- 172.21.15.116:0/2673680773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354419f5e0 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f3548883640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534043020 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354419f870 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35340043d0 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534003b80 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.253+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354419fe10 con 0x7f3544101930 2024-03-19T02:27:42.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.254+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354419ffe0 con 0x7f3544101930 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: ignoring --setuser ceph since I am not root 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: ignoring --setgroup ceph since I am not root 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mgr[37095]: -- 172.21.15.154:0/3839918023 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606930845a0 con 0x560693058c00 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:27:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:27:42.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:41 smithi154 ceph-mon[35693]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-19T02:27:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.256+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 137359+0+0 (secure 0 0 0) 0x7f3534046020 con 0x7f3544101930 2024-03-19T02:27:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.256+0000 7f353a7fc640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 0x7f35300a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.256+0000 7f3549084640 1 -- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 msgr2=0x7f35300a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.256+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f35340f16d0 con 0x7f3544101930 2024-03-19T02:27:42.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.256+0000 7f3549084640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 0x7f35300a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:27:42.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.260+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f353404d020 con 0x7f3544101930 2024-03-19T02:27:42.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.444+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f35441a0370 con 0x7f3544101930 2024-03-19T02:27:42.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.448+0000 7f353a7fc640 1 -- 172.21.15.116:0/2673680773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f35340be3a0 con 0x7f3544101930 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 msgr2=0x7f35300a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 0x7f35300a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 msgr2=0x7f3544197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005e40 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 shutdown_connections 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f35300a2990 0x7f35300a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35441022f0 0x7f354419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3544101930 0x7f3544197f00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 --2- 172.21.15.116:0/2673680773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3544100740 0x7f35441979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 >> 172.21.15.116:0/2673680773 conn(0x7f35440fbcb0 msgr2=0x7f35440fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 shutdown_connections 2024-03-19T02:27:42.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.451+0000 7f354b2e8640 1 -- 172.21.15.116:0/2673680773 wait complete. 2024-03-19T02:27:42.467 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/1262004770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4102a60 msgr2=0x7f4db4102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/1262004770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4102a60 0x7f4db4102ee0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009900 tx=0x7f4db002bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/1262004770 shutdown_connections 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/1262004770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db4103420 0x7f4db4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/1262004770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4102a60 0x7f4db4102ee0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/1262004770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4db4101870 0x7f4db4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.573+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/1262004770 >> 172.21.15.116:0/1262004770 conn(0x7f4db40fcde0 msgr2=0x7f4db40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/1262004770 shutdown_connections 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/1262004770 wait complete. 2024-03-19T02:27:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 Processor -- start 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- start start 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 0x7f4db4197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4db4102a60 0x7f4db4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db4103420 0x7f4db419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d3a0 con 0x7f4db4101870 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d220 con 0x7f4db4103420 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dbbfb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410d520 con 0x7f4db4102a60 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4db9d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 0x7f4db4197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4db9d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 0x7f4db4197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59288/0 (socket says 172.21.15.116:59288) 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4db9d4e640 1 -- 172.21.15.116:0/3124886540 learned_addr learned my addr 172.21.15.116:0/3124886540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4db4102a60 0x7f4db4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.574+0000 7f4dba54f640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db4103420 0x7f4db419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4db4102a60 msgr2=0x7f4db4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4db4102a60 0x7f4db4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db4103420 msgr2=0x7f4db419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db4103420 0x7f4db419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db419f630 con 0x7f4db4101870 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4db9d4e640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 0x7f4db4197a10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4da400e9c0 tx=0x7f4da400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da400cd80 con 0x7f4db4101870 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db419f920 con 0x7f4db4101870 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4da40035f0 con 0x7f4db4101870 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4010690 con 0x7f4db4101870 2024-03-19T02:27:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.575+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db419fe40 con 0x7f4db4101870 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.576+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db404f580 con 0x7f4db4101870 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.577+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 137359+0+0 (secure 0 0 0) 0x7f4da40037a0 con 0x7f4db4101870 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.578+0000 7f4daaffd640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.578+0000 7f4db954d640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.578+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f4da40cf490 con 0x7f4db4101870 2024-03-19T02:27:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.578+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:27:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.582+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4da409c050 con 0x7f4db4101870 2024-03-19T02:27:42.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:42.401+0000 7fddada5a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:27:42.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:42.504+0000 7fddada5a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:27:42.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.779+0000 7f4db954d640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:42.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.779+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:27:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:42.786+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4db4061c60 con 0x7f4da00a2990 2024-03-19T02:27:42.812 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:42.531+0000 7ffa5f5ba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:27:42.812 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:42.646+0000 7ffa5f5ba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:27:42.812 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:42.509+0000 7fc585294200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:27:42.812 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:42.604+0000 7fc585294200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:27:43.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:27:43.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-mon[32263]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-19T02:27:43.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2233979700' entity='client.admin' 2024-03-19T02:27:43.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2673680773' entity='client.admin' 2024-03-19T02:27:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:27:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-mon[35761]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-19T02:27:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2233979700' entity='client.admin' 2024-03-19T02:27:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2673680773' entity='client.admin' 2024-03-19T02:27:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:43.180+0000 7f4db954d640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:43.180+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:27:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2643113296' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:27:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-mon[35693]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-19T02:27:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2233979700' entity='client.admin' 2024-03-19T02:27:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2673680773' entity='client.admin' 2024-03-19T02:27:43.483 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.151+0000 7fddada5a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:27:43.483 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.245+0000 7fddada5a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:27:43.483 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.387+0000 7fddada5a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:27:43.483 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.482+0000 7fddada5a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:27:43.648 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:43.370+0000 7fc585294200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:27:43.648 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:43.464+0000 7fc585294200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:27:43.667 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:43.414+0000 7ffa5f5ba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:27:43.667 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:43.511+0000 7ffa5f5ba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:27:43.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.567+0000 7fddada5a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:27:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:43.981+0000 7f4db954d640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:43.981+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:27:44.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:43.647+0000 7fc585294200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:27:44.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:43.744+0000 7fc585294200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:27:44.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:43.829+0000 7fc585294200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:27:44.121 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.830+0000 7fddada5a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:27:44.121 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:43.939+0000 7fddada5a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:44.121 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:44.026+0000 7fddada5a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:27:44.121 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:44.120+0000 7fddada5a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:27:44.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:43.710+0000 7ffa5f5ba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:27:44.167 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:43.838+0000 7ffa5f5ba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:27:44.168 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:43.923+0000 7ffa5f5ba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:27:44.447 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:44.097+0000 7fc585294200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:27:44.447 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:44.209+0000 7fc585294200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:44.447 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:44.339+0000 7fc585294200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:27:44.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:44.298+0000 7fddada5a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:27:44.584 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:44.212+0000 7ffa5f5ba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:27:44.584 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:44.356+0000 7ffa5f5ba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:44.584 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:44.462+0000 7ffa5f5ba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:27:44.584 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:44.583+0000 7ffa5f5ba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:27:44.756 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:44.446+0000 7fc585294200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:27:44.756 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:44.660+0000 7fc585294200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:27:44.917 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:44.763+0000 7ffa5f5ba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:27:45.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:45.257+0000 7fddada5a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:27:45.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:45.388+0000 7fddada5a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:27:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:45.582+0000 7f4db954d640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3573130182 2024-03-19T02:27:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:45.582+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:27:46.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:45.690+0000 7fc585294200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:27:46.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:45.824+0000 7fc585294200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:27:46.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:45.724+0000 7fddada5a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:27:46.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:45.815+0000 7fddada5a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:27:46.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:45.921+0000 7ffa5f5ba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:27:46.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:46.112+0000 7ffa5f5ba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:27:46.476 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:46.169+0000 7fc585294200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:27:46.476 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:46.260+0000 7fc585294200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:27:46.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:46.014+0000 7fddada5a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:46.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:46.201+0000 7fddada5a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:27:46.749 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:46.487+0000 7fddada5a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:46.755 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:46.474+0000 7fc585294200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:46.756 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:46.718+0000 7fc585294200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:27:46.917 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:46.526+0000 7ffa5f5ba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:27:46.917 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:46.660+0000 7ffa5f5ba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:27:47.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:46.748+0000 7fddada5a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:47.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:46.989+0000 7fddada5a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:27:47.392 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:47.070+0000 7fc585294200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:47.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:46.947+0000 7ffa5f5ba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:47.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:47.167+0000 7ffa5f5ba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:27:47.700 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:47.391+0000 7fc585294200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:47.845 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:47.555+0000 7ffa5f5ba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:27:47.845 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:47.845+0000 7ffa5f5ba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:27:48.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:47.699+0000 7fc585294200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:27:48.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:47.797+0000 7fddada5a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:27:48.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:48.101+0000 7ffa5f5ba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:27:48.690 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.370+0000 7fddada5a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:27:48.690 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.470+0000 7fddada5a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:27:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:48.713+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 21) ==== 136975+0+0 (secure 0 0 0) 0x7f4da4097fc0 con 0x7f4db4101870 2024-03-19T02:27:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:48.714+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 msgr2=0x7f4da00a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:27:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:48.714+0000 7f4daaffd640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/3573130182,v1:172.21.15.116:6801/3573130182] conn(0x7f4da00a2990 0x7f4da00a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:48.949 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:48.672+0000 7fc585294200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:27:48.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: Active manager daemon x restarted 2024-03-19T02:27:48.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: Activating manager daemon x 2024-03-19T02:27:48.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: mgrmap e21: x(active, starting, since 0.0113492s), standbys: z, y 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:27:48.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:27:48.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:27:48.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:48 smithi154 ceph-mon[35693]: Manager daemon x is now available 2024-03-19T02:27:48.959 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.689+0000 7fddada5a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:27:48.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: Active manager daemon x restarted 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: Activating manager daemon x 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: mgrmap e21: x(active, starting, since 0.0113492s), standbys: z, y 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:27:48.960 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:27:48.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:27:48.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:27:48.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:27:48.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-mon[32263]: Manager daemon x is now available 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: Active manager daemon x restarted 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: Activating manager daemon x 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: mgrmap e21: x(active, starting, since 0.0113492s), standbys: z, y 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:27:49.077 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:27:49.078 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:27:49.079 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:48 smithi162 ceph-mon[35761]: Manager daemon x is now available 2024-03-19T02:27:49.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd104b2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd104b2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd104b2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd104b2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd104b2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd0d4ac700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd0d4ac700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd0d4ac700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd0d4ac700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.259 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:27:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:27:48.961+0000 7fdd0d4ac700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:27:49.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:49.076+0000 7ffa5f5ba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:27:49.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.719+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 22) ==== 137102+0+0 (secure 0 0 0) 0x7f4da409bbb0 con 0x7f4db4101870 2024-03-19T02:27:49.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.719+0000 7f4daaffd640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4da0051790 0x7f4da0051b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:49.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.719+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4da003cae0 con 0x7f4da0051790 2024-03-19T02:27:49.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.721+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4da0051790 0x7f4da0051b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:49.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.723+0000 7f4db954d640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4da0051790 0x7f4da0051b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009230 tx=0x7f4db0005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:49.756 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:49.380+0000 7fc585294200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:27:49.756 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:49.492+0000 7fc585294200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:27:49.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.795+0000 7f4daaffd640 1 -- 172.21.15.116:0/3124886540 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4da003cae0 con 0x7f4da0051790 2024-03-19T02:27:49.796 INFO:teuthology.orchestra.run.smithi116.stdout:Schedule set for path / 2024-03-19T02:27:49.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4da0051790 msgr2=0x7f4da0051b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4da0051790 0x7f4da0051b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009230 tx=0x7f4db0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 msgr2=0x7f4db4197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 --2- 172.21.15.116:0/3124886540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db4101870 0x7f4db4197a10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4da400e9c0 tx=0x7f4da400ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dba54f640 1 -- 172.21.15.116:0/3124886540 reap_dead start 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 shutdown_connections 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 >> 172.21.15.116:0/3124886540 conn(0x7f4db40fcde0 msgr2=0x7f4db4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 shutdown_connections 2024-03-19T02:27:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.798+0000 7f4dbbfb2640 1 -- 172.21.15.116:0/3124886540 wait complete. 2024-03-19T02:27:49.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 -- 172.21.15.116:0/1650458264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7724101d50 msgr2=0x7f7724102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 --2- 172.21.15.116:0/1650458264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7724101d50 0x7f7724102170 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7714009930 tx=0x7f771402bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 -- 172.21.15.116:0/1650458264 shutdown_connections 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 --2- 172.21.15.116:0/1650458264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77241013b0 0x7f7724108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 --2- 172.21.15.116:0/1650458264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 0x7f7724100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 --2- 172.21.15.116:0/1650458264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7724101d50 0x7f7724102170 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 -- 172.21.15.116:0/1650458264 >> 172.21.15.116:0/1650458264 conn(0x7f77240fb440 msgr2=0x7f77240fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 -- 172.21.15.116:0/1650458264 shutdown_connections 2024-03-19T02:27:49.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.916+0000 7f7728d87640 1 -- 172.21.15.116:0/1650458264 wait complete. 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 Processor -- start 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 -- start start 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 0x7f7724197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7724101d50 0x7f772419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f772410d3e0 con 0x7f77241013b0 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f772410d260 con 0x7f7724101d50 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7728d87640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f772410d560 con 0x7f77241009f0 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:49.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53830/0 (socket says 172.21.15.116:53830) 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 -- 172.21.15.116:0/2668457656 learned_addr learned my addr 172.21.15.116:0/2668457656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f772259c640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 0x7f7724197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7722d9d640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7724101d50 0x7f772419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 -- 172.21.15.116:0/2668457656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 msgr2=0x7f7724197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 0x7f7724197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 -- 172.21.15.116:0/2668457656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7724101d50 msgr2=0x7f772419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7724101d50 0x7f772419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 -- 172.21.15.116:0/2668457656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772419f660 con 0x7f77241013b0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.917+0000 7f7721d9b640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f771800e9f0 tx=0x7f771800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.918+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771800cd80 con 0x7f77241013b0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.918+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77180054d0 con 0x7f77241013b0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.918+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772419f950 con 0x7f77241013b0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.918+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718010490 con 0x7f77241013b0 2024-03-19T02:27:49.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.918+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772419fe50 con 0x7f77241013b0 2024-03-19T02:27:49.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.919+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7724107bd0 con 0x7f77241013b0 2024-03-19T02:27:49.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.920+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 137102+0+0 (secure 0 0 0) 0x7f7718005020 con 0x7f77241013b0 2024-03-19T02:27:49.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.920+0000 7f77137fe640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 0x7f770c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:49.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.920+0000 7f772259c640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 0x7f770c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:49.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.920+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f77180cee60 con 0x7f77241013b0 2024-03-19T02:27:49.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.921+0000 7f772259c640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 0x7f770c0a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7714000c00 tx=0x7f7714005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:49.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:49.925+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f771809bd50 con 0x7f77241013b0 2024-03-19T02:27:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:49 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:49 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:49 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:27:50.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:49 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:27:50.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:49 smithi116 ceph-mon[32263]: mgrmap e22: x(active, since 1.01888s), standbys: z, y 2024-03-19T02:27:50.091 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y[37091]: 2024-03-19T02:27:49.810+0000 7fc585294200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:27:50.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:27:50.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:27:50.092 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:49 smithi154 ceph-mon[35693]: mgrmap e22: x(active, since 1.01888s), standbys: z, y 2024-03-19T02:27:50.131 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.131 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:50.131 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:27:50.131 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:27:50.131 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-mon[35761]: mgrmap e22: x(active, since 1.01888s), standbys: z, y 2024-03-19T02:27:50.132 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:49.779+0000 7ffa5f5ba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:27:50.132 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:49.932+0000 7ffa5f5ba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:27:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.132+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f7724061d00 con 0x7f770c0a26b0 2024-03-19T02:27:50.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.189+0000 7f77137fe640 1 -- 172.21.15.116:0/2668457656 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f7724061d00 con 0x7f770c0a26b0 2024-03-19T02:27:50.190 INFO:teuthology.orchestra.run.smithi116.stdout:Retention added to path / 2024-03-19T02:27:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 msgr2=0x7f770c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 0x7f770c0a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7714000c00 tx=0x7f7714005fd0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 msgr2=0x7f7724197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f771800e9f0 tx=0x7f771800edf0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 shutdown_connections 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f770c0a26b0 0x7f770c0a4b70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7724101d50 0x7f772419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77241013b0 0x7f7724197fd0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 --2- 172.21.15.116:0/2668457656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77241009f0 0x7f7724197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.191+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 >> 172.21.15.116:0/2668457656 conn(0x7f77240fb440 msgr2=0x7f7724104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.192+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 shutdown_connections 2024-03-19T02:27:50.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.192+0000 7f7728d87640 1 -- 172.21.15.116:0/2668457656 wait complete. 2024-03-19T02:27:50.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-19T02:27:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 -- 172.21.15.116:0/3254661602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b0101930 msgr2=0x7fd0b0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 --2- 172.21.15.116:0/3254661602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b0101930 0x7fd0b0101db0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd098009900 tx=0x7fd09802bc30 comp rx=0 tx=0).stop 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 -- 172.21.15.116:0/3254661602 shutdown_connections 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 --2- 172.21.15.116:0/3254661602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01022f0 0x7fd0b0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 --2- 172.21.15.116:0/3254661602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b0101930 0x7fd0b0101db0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 --2- 172.21.15.116:0/3254661602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 0x7fd0b0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.306+0000 7fd0b584b640 1 -- 172.21.15.116:0/3254661602 >> 172.21.15.116:0/3254661602 conn(0x7fd0b00fbcb0 msgr2=0x7fd0b00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.307+0000 7fd0b584b640 1 -- 172.21.15.116:0/3254661602 shutdown_connections 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.307+0000 7fd0b584b640 1 -- 172.21.15.116:0/3254661602 wait complete. 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 Processor -- start 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 -- start start 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 0x7fd0b01046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b0101930 0x7fd0b0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b010c250 con 0x7fd0b01022f0 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b010c0d0 con 0x7fd0b0100740 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0b584b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b010c3d0 con 0x7fd0b0101930 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0aeffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 0x7fd0b01046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0ae7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b0101930 0x7fd0b0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53854/0 (socket says 172.21.15.116:53854) 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 -- 172.21.15.116:0/843084285 learned_addr learned my addr 172.21.15.116:0/843084285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 -- 172.21.15.116:0/843084285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b0101930 msgr2=0x7fd0b0102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b0101930 0x7fd0b0102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 -- 172.21.15.116:0/843084285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 msgr2=0x7fd0b01046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 0x7fd0b01046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.308+0000 7fd0af7fe640 1 -- 172.21.15.116:0/843084285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b0103ae0 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0af7fe640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800e970 tx=0x7fd0a800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800ccc0 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0a800ce60 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b01a1ae0 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8010430 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.309+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b01a1fb0 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.310+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b004f580 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.311+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 137102+0+0 (secure 0 0 0) 0x7fd0a8004ff0 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.311+0000 7fd0b4849640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 0x7fd09c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.312+0000 7fd0aeffd640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 0x7fd09c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.312+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd0a80cef50 con 0x7fd0b01022f0 2024-03-19T02:27:50.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.312+0000 7fd0aeffd640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 0x7fd09c0a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0101790 tx=0x7fd0a0009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:50.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.316+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd0a809be40 con 0x7fd0b01022f0 2024-03-19T02:27:50.417 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z[37599]: 2024-03-19T02:27:50.221+0000 7ffa5f5ba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:27:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.523+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd0b00fde60 con 0x7fd09c0a26b0 2024-03-19T02:27:50.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.532+0000 7fd0b4849640 1 -- 172.21.15.116:0/843084285 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fd0b00fde60 con 0x7fd09c0a26b0 2024-03-19T02:27:50.532 INFO:teuthology.orchestra.run.smithi116.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-19T00:00:00", "created": "2024-03-19T02:27:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-19T02:27:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 msgr2=0x7fd09c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 0x7fd09c0a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0101790 tx=0x7fd0a0009500 comp rx=0 tx=0).stop 2024-03-19T02:27:50.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 msgr2=0x7fd0b0103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800e970 tx=0x7fd0a800ed70 comp rx=0 tx=0).stop 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 shutdown_connections 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd09c0a26b0 0x7fd09c0a4b70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b01022f0 0x7fd0b0103250 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b0101930 0x7fd0b0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.534+0000 7fd0b584b640 1 --2- 172.21.15.116:0/843084285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0b0100740 0x7fd0b01046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.535+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 >> 172.21.15.116:0/843084285 conn(0x7fd0b00fbcb0 msgr2=0x7fd0b00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.535+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 shutdown_connections 2024-03-19T02:27:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.535+0000 7fd0b584b640 1 -- 172.21.15.116:0/843084285 wait complete. 2024-03-19T02:27:50.547 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-19T02:27:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 -- 172.21.15.116:0/874041369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b81022f0 msgr2=0x7f19b8108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 --2- 172.21.15.116:0/874041369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b81022f0 0x7f19b8108b80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009ca0 tx=0x7f19b002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 -- 172.21.15.116:0/874041369 shutdown_connections 2024-03-19T02:27:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 --2- 172.21.15.116:0/874041369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b81022f0 0x7f19b8108b80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 --2- 172.21.15.116:0/874041369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 0x7f19b8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 --2- 172.21.15.116:0/874041369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b8100740 0x7f19b8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 -- 172.21.15.116:0/874041369 >> 172.21.15.116:0/874041369 conn(0x7f19b80fbcb0 msgr2=0x7f19b80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 -- 172.21.15.116:0/874041369 shutdown_connections 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.662+0000 7f19be168640 1 -- 172.21.15.116:0/874041369 wait complete. 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 Processor -- start 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 -- start start 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 0x7f19b8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b81022f0 0x7f19b8103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19b810c250 con 0x7f19b8100740 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19b810c0d0 con 0x7f19b8101930 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19be168640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19b810c3d0 con 0x7f19b81022f0 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.663+0000 7f19b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53870/0 (socket says 172.21.15.116:53870) 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 -- 172.21.15.116:0/4226713747 learned_addr learned my addr 172.21.15.116:0/4226713747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b6ffd640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 0x7f19b8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b7fff640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b81022f0 0x7f19b8103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 -- 172.21.15.116:0/4226713747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b81022f0 msgr2=0x7f19b8103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b81022f0 0x7f19b8103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 -- 172.21.15.116:0/4226713747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 msgr2=0x7f19b8102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 0x7f19b8102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 -- 172.21.15.116:0/4226713747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b8103b10 con 0x7f19b8100740 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b77fe640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f19a800e9c0 tx=0x7f19a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a800cd80 con 0x7f19b8100740 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b81a1ae0 con 0x7f19b8100740 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19a80035f0 con 0x7f19b8100740 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8010690 con 0x7f19b8100740 2024-03-19T02:27:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.664+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b81a2000 con 0x7f19b8100740 2024-03-19T02:27:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.665+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b81a2230 con 0x7f19b8100740 2024-03-19T02:27:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.667+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 137102+0+0 (secure 0 0 0) 0x7f19a80037a0 con 0x7f19b8100740 2024-03-19T02:27:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.668+0000 7f19b4ff9640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 0x7f19a00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:27:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.668+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f19a80cf000 con 0x7f19b8100740 2024-03-19T02:27:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.668+0000 7f19b6ffd640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 0x7f19a00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:27:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.668+0000 7f19b6ffd640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 0x7f19a00a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19b81043f0 tx=0x7f19ac005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:27:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.672+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f19a809bef0 con 0x7f19b8100740 2024-03-19T02:27:50.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.781+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 23) ==== 137411+0+0 (secure 0 0 0) 0x7f19a8097e60 con 0x7f19b8100740 2024-03-19T02:27:50.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.928+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f19b81a2400 con 0x7f19a00a26b0 2024-03-19T02:27:50.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.935+0000 7f19b4ff9640 1 -- 172.21.15.116:0/4226713747 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f19b81a2400 con 0x7f19a00a26b0 2024-03-19T02:27:50.936 INFO:teuthology.orchestra.run.smithi116.stdout:/ 1m 6m3h 2024-03-19T02:27:50.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 msgr2=0x7f19a00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 0x7f19a00a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19b81043f0 tx=0x7f19ac005f90 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 msgr2=0x7f19b8104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f19a800e9c0 tx=0x7f19a800ee80 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 shutdown_connections 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a00a26b0 0x7f19a00a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19b81022f0 0x7f19b8103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19b8101930 0x7f19b8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 --2- 172.21.15.116:0/4226713747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b8100740 0x7f19b8104680 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 >> 172.21.15.116:0/4226713747 conn(0x7f19b80fbcb0 msgr2=0x7f19b80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 shutdown_connections 2024-03-19T02:27:50.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:27:50.938+0000 7f19be168640 1 -- 172.21.15.116:0/4226713747 wait complete. 2024-03-19T02:27:50.949 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: [19/Mar/2024:02:27:49] ENGINE Bus STARTING 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: [19/Mar/2024:02:27:49] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: Standby manager daemon y restarted 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: Standby manager daemon y started 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: Standby manager daemon z restarted 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: Standby manager daemon z started 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:27:50.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:50.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:27:50.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:50 smithi116 ceph-mon[32263]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:50.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-19T02:27:50.979 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-19T02:27:50.991 INFO:teuthology.run_tasks:Running task workunit... 2024-03-19T02:27:51.016 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:27:51.016 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-19T02:27:51.016 INFO:tasks.workunit:timeout=3h 2024-03-19T02:27:51.016 INFO:tasks.workunit:cleanup=True 2024-03-19T02:27:51.017 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:51.034 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 02:25:36.071823905 +0000 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 02:25:36.071823905 +0000 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 02:27:33.208142492 +0000 2024-03-19T02:27:51.035 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-03-19T02:27:51.035 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-19T02:27:51.035 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-19T02:27:51.110 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:27:51.166 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-19T02:27:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: [19/Mar/2024:02:27:49] ENGINE Bus STARTING 2024-03-19T02:27:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: [19/Mar/2024:02:27:49] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: Standby manager daemon y restarted 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: Standby manager daemon y started 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: Standby manager daemon z restarted 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: Standby manager daemon z started 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:27:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:50 smithi162 ceph-mon[35761]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: [19/Mar/2024:02:27:49] ENGINE Bus STARTING 2024-03-19T02:27:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: [19/Mar/2024:02:27:49] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: Standby manager daemon y restarted 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: Standby manager daemon y started 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.154:0/3564332096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: Standby manager daemon z restarted 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: Standby manager daemon z started 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:27:51.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:50 smithi154 ceph-mon[35693]: from='mgr.? 172.21.15.162:0/4043350223' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:27:52.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: [19/Mar/2024:02:27:49] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T02:27:52.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: [19/Mar/2024:02:27:49] ENGINE Bus STARTED 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.158 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:51 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: [19/Mar/2024:02:27:49] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: [19/Mar/2024:02:27:49] ENGINE Bus STARTED 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:51 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: [19/Mar/2024:02:27:49] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T02:27:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: [19/Mar/2024:02:27:49] ENGINE Bus STARTED 2024-03-19T02:27:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:52.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:51 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:53.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:52 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:53.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:52 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:53.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:52 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:54.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:54.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:27:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:27:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:27:54.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:27:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:54 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:54 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:27:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:27:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: Updating smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:27:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:54 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:27:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:55 smithi162 ceph-mon[35761]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-19T02:27:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:55 smithi154 ceph-mon[35693]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-19T02:27:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:55 smithi116 ceph-mon[32263]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-19T02:27:58.060 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:57 smithi116 ceph-mon[32263]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-19T02:27:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:57 smithi162 ceph-mon[35761]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-19T02:27:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:57 smithi154 ceph-mon[35693]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-19T02:28:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:27:59 smithi162 ceph-mon[35761]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 102 KiB/s wr, 62 op/s 2024-03-19T02:28:00.199 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:27:59 smithi116 ceph-mon[32263]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 102 KiB/s wr, 62 op/s 2024-03-19T02:28:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:27:59 smithi154 ceph-mon[35693]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 102 KiB/s wr, 62 op/s 2024-03-19T02:28:00.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:28:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:28:00.197+0000 7fdcee02e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:28:00.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:28:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:28:00.197+0000 7fdcee02e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:28:00.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:28:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:28:00.197+0000 7fdcee02e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:28:00.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:28:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:28:00.197+0000 7fdcee02e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:28:00.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:28:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:28:00.197+0000 7fdcee02e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:28:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:01 smithi162 ceph-mon[35761]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-19T02:28:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:01 smithi162 ceph-mon[35761]: mgrmap e25: x(active, since 12s), standbys: z, y 2024-03-19T02:28:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:01 smithi154 ceph-mon[35693]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-19T02:28:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:01 smithi154 ceph-mon[35693]: mgrmap e25: x(active, since 12s), standbys: z, y 2024-03-19T02:28:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:01 smithi116 ceph-mon[32263]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-19T02:28:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:01 smithi116 ceph-mon[32263]: mgrmap e25: x(active, since 12s), standbys: z, y 2024-03-19T02:28:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:03 smithi162 ceph-mon[35761]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-19T02:28:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:03 smithi154 ceph-mon[35693]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-19T02:28:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:03 smithi116 ceph-mon[32263]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-19T02:28:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:05 smithi162 ceph-mon[35761]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-19T02:28:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:05 smithi154 ceph-mon[35693]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-19T02:28:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:05 smithi116 ceph-mon[32263]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-19T02:28:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:07 smithi162 ceph-mon[35761]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-19T02:28:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:07 smithi154 ceph-mon[35693]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-19T02:28:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:07 smithi116 ceph-mon[32263]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-19T02:28:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:09 smithi162 ceph-mon[35761]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:09 smithi154 ceph-mon[35693]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:09 smithi116 ceph-mon[32263]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:11.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:28:11.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 -- 172.21.15.116:0/1819996535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f68100680 msgr2=0x7f7f6810e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1819996535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f68100680 0x7f7f6810e640 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009ca0 tx=0x7f7f6002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 -- 172.21.15.116:0/1819996535 shutdown_connections 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1819996535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f68100680 0x7f7f6810e640 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1819996535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f680ffcc0 0x7f7f68100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1819996535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f68107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 -- 172.21.15.116:0/1819996535 >> 172.21.15.116:0/1819996535 conn(0x7f7f68076c40 msgr2=0x7f7f680fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 -- 172.21.15.116:0/1819996535 shutdown_connections 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.147+0000 7f7f6f261640 1 -- 172.21.15.116:0/1819996535 wait complete. 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 Processor -- start 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 -- start start 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f68100680 0x7f7f6819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f6819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68111a60 con 0x7f7f680ffcc0 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681118e0 con 0x7f7f681077c0 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6f261640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68111be0 con 0x7f7f68100680 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34996/0 (socket says 172.21.15.116:34996) 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 -- 172.21.15.116:0/1956058086 learned_addr learned my addr 172.21.15.116:0/1956058086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f6819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f67fff640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f68100680 0x7f7f6819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 -- 172.21.15.116:0/1956058086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f68100680 msgr2=0x7f7f6819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f68100680 0x7f7f6819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 -- 172.21.15.116:0/1956058086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 msgr2=0x7f7f6819ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f6819ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 -- 172.21.15.116:0/1956058086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6819f1c0 con 0x7f7f680ffcc0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f6819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.148+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9f0 tx=0x7f7f5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.149+0000 7f7f65ffb640 1 -- 172.21.15.116:0/1956058086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00cd80 con 0x7f7f680ffcc0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.149+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6819f4b0 con 0x7f7f680ffcc0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.149+0000 7f7f65ffb640 1 -- 172.21.15.116:0/1956058086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c0108e0 con 0x7f7f680ffcc0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.149+0000 7f7f65ffb640 1 -- 172.21.15.116:0/1956058086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c018920 con 0x7f7f680ffcc0 2024-03-19T02:28:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.149+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f681a6d40 con 0x7f7f680ffcc0 2024-03-19T02:28:11.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.151+0000 7f7f71e51740 1 Processor -- start 2024-03-19T02:28:11.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.151+0000 7f7f71e51740 1 -- start start 2024-03-19T02:28:11.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.151+0000 7f7f65ffb640 1 -- 172.21.15.116:0/1956058086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f7f5c010a90 con 0x7f7f680ffcc0 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a5120099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a511f64500 con 0x55a512009f30 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a511f0cbe0 con 0x55a512000a30 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f71e51740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a511f605a0 con 0x55a512001350 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f6d7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35000/0 (socket says 172.21.15.116:35000) 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f6cffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a5120099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f6d7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36486/0 (socket says 172.21.15.116:36486) 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 -- 172.21.15.116:0/3839886499 learned_addr learned my addr 172.21.15.116:0/3839886499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 -- 172.21.15.116:0/3839886499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 msgr2=0x55a5120099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a5120099f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 -- 172.21.15.116:0/3839886499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 msgr2=0x55a512000e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 -- 172.21.15.116:0/3839886499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a511f2e000 con 0x55a512009f30 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f65ffb640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 0x7f7f580a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f67fff640 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ca0 tx=0x7f7f54011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737bab19aa4f4092 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 0x7f7f580a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f65ffb640 1 -- 172.21.15.116:0/1956058086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7f5c0cf750 con 0x7f7f680ffcc0 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.152+0000 7f7f52ffd640 1 -- 172.21.15.116:0/3839886499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5401e070 con 0x55a512009f30 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f52ffd640 1 -- 172.21.15.116:0/3839886499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f54019ae0 con 0x55a512009f30 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f52ffd640 1 -- 172.21.15.116:0/3839886499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400e950 con 0x55a512009f30 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 -- 172.21.15.116:0/3839886499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 msgr2=0x55a51200c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 0x7f7f580a4f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f60004af0 tx=0x7f7f60005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ca0 tx=0x7f7f54011810 comp rx=0 tx=0).stop 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 -- 172.21.15.116:0/3839886499 shutdown_connections 2024-03-19T02:28:11.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a51200c320 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a5120099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 --2- 172.21.15.116:0/3839886499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a512000e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 -- 172.21.15.116:0/3839886499 >> 172.21.15.116:0/3839886499 conn(0x55a511ffb160 msgr2=0x55a511ffb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 -- 172.21.15.116:0/3839886499 shutdown_connections 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.153+0000 7f7f71e51740 1 -- 172.21.15.116:0/3839886499 wait complete. 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 -- start start 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 Processor -- start 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512000a30 0x55a512006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 0x55a512007910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a51200cd20 con 0x55a512001350 2024-03-19T02:28:11.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a511f20160 con 0x55a512009f30 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a51200cea0 con 0x55a512000a30 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f71e51740 10 client.0 _ll_register_callbacks cb 0x55a511f5ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f6cffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512000a30 0x55a512006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35002/0 (socket says 172.21.15.116:35002) 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f67fff640 1 -- 172.21.15.116:0/4203902372 learned_addr learned my addr 172.21.15.116:0/4203902372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.154+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 0x55a512007910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 -- 172.21.15.116:0/4203902372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512000a30 msgr2=0x55a512006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512000a30 0x55a512006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 -- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 msgr2=0x55a512007910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 0x55a512007910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 -- 172.21.15.116:0/4203902372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a511d427d0 con 0x55a512001350 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 0x55a512007910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f67fff640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009c70 tx=0x7f7f54019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74297cffcef6e166 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f47fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54027030 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7f54019850 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f54019ae0 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f54031050 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f71e51740 10 client.15822 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a511c783f0 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.155+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f54019ae0 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7f540046c0 con 0x55a512001350 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 10 client.15822 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 20 client.15822 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51201a1f0 0x55a51201c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 4 client.15822 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a511f71a40 con 0x55a51201a1f0 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51201a1f0 0x55a51201c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f47fff640 10 client.15822 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.156+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51201a1f0 0x55a51201c5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58ea48f4970f95fa server_cookie=85bb59998703c797 in_seq=0 out_seq=0 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f47fff640 1 -- 172.21.15.116:0/4203902372 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a511f71a40 con 0x55a51201a1f0 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f47fff640 10 client.15822 handle_command_reply: tid=0 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 client.15822 shutdown 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 msgr2=0x55a5120073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009c70 tx=0x7f7f54019660 comp rx=0 tx=0).stop 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 shutdown_connections 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51201a1f0 0x55a51201c5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512009f30 0x55a512007910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a5120073d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 --2- 172.21.15.116:0/4203902372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512000a30 0x55a512006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f47fff640 0 client.15822 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 >> 172.21.15.116:0/4203902372 conn(0x55a511ffb160 msgr2=0x55a511ffceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 shutdown_connections 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 1 -- 172.21.15.116:0/4203902372 wait complete. 2024-03-19T02:28:11.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.157+0000 7f7f71e51740 20 client.15822 trim_cache size 0 max 16384 2024-03-19T02:28:11.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 Processor -- start 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 -- start start 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a511ffb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a512015af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681b0cf0 con 0x55a512009f30 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681b2830 con 0x55a512000a30 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f71e51740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681b13c0 con 0x55a512001350 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f6d7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a512015af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f6d7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35010/0 (socket says 172.21.15.116:35010) 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f6d7fe640 1 -- 172.21.15.116:0/10111355 learned_addr learned my addr 172.21.15.116:0/10111355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.187+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a511ffb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 -- 172.21.15.116:0/10111355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 msgr2=0x55a512015af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a512015af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 -- 172.21.15.116:0/10111355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 msgr2=0x55a511ffb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a511ffb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 -- 172.21.15.116:0/10111355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a511c380f0 con 0x55a512009f30 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f67fff640 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a512015af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7f60036000 tx=0x7f7f60041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590ef4f065ab40f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f527fc640 1 -- 172.21.15.116:0/10111355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60039040 con 0x55a512009f30 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f527fc640 1 -- 172.21.15.116:0/10111355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f600044a0 con 0x55a512009f30 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.188+0000 7f7f527fc640 1 -- 172.21.15.116:0/10111355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6004db60 con 0x55a512009f30 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- 172.21.15.116:0/10111355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 msgr2=0x55a512021e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7f60036000 tx=0x7f7f60041000 comp rx=0 tx=0).stop 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- 172.21.15.116:0/10111355 shutdown_connections 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512009f30 0x55a512021e00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512001350 0x55a512015af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 --2- 172.21.15.116:0/10111355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a511ffb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- 172.21.15.116:0/10111355 >> 172.21.15.116:0/10111355 conn(0x55a511ffb160 msgr2=0x55a512005760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- 172.21.15.116:0/10111355 shutdown_connections 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- 172.21.15.116:0/10111355 wait complete. 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 -- start start 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.189+0000 7f7f71e51740 1 Processor -- start 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a51200d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512009f30 0x55a51200db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681b2160 con 0x55a512001350 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681b1a90 con 0x55a512000a30 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681aecd0 con 0x55a512009f30 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 10 client.0 _ll_register_callbacks cb 0x7f7f5401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35014/0 (socket says 172.21.15.116:35014) 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 -- 172.21.15.116:0/2402972848 learned_addr learned my addr 172.21.15.116:0/2402972848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a51200d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f6d7fe640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512009f30 0x55a51200db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 -- 172.21.15.116:0/2402972848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512009f30 msgr2=0x55a51200db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512009f30 0x55a51200db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 -- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 msgr2=0x55a51200d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a51200d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 -- 172.21.15.116:0/2402972848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a51207e580 con 0x55a512001350 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f67fff640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ca0 tx=0x7f7f54005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f18f3f2c212a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f47fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54027040 con 0x55a512001350 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7f54004840 con 0x55a512001350 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f600044a0 con 0x55a512001350 2024-03-19T02:28:11.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.190+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f54042050 con 0x55a512001350 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 10 client.15828 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a51207f890 con 0x55a512001350 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f600044a0 con 0x55a512001350 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7f5403d030 con 0x55a512001350 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 10 client.15828 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 20 client.15828 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51207ff70 0x55a512082390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 4 client.15828 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a511f6bd90 con 0x55a51207ff70 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51207ff70 0x55a512082390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f6cffd640 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51207ff70 0x55a512082390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e2122a18a81895 server_cookie=904bdbb7875ec919 in_seq=0 out_seq=0 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.191+0000 7f7f47fff640 10 client.15828 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:11.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f47fff640 1 -- 172.21.15.116:0/2402972848 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a511f6bd90 con 0x55a51207ff70 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f47fff640 10 client.15828 handle_command_reply: tid=0 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 client.15828 shutdown 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 msgr2=0x55a51200d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ca0 tx=0x7f7f54005710 comp rx=0 tx=0).stop 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 shutdown_connections 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a51207ff70 0x55a512082390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a512009f30 0x55a51200db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f47fff640 0 client.15828 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a512001350 0x55a51200d620 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 --2- 172.21.15.116:0/2402972848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a512000a30 0x55a51200d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 >> 172.21.15.116:0/2402972848 conn(0x55a511ffb160 msgr2=0x55a512006e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 shutdown_connections 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 1 -- 172.21.15.116:0/2402972848 wait complete. 2024-03-19T02:28:11.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.192+0000 7f7f71e51740 20 client.15828 trim_cache size 0 max 16384 2024-03-19T02:28:11.200 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:11.200 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:28:11.200 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:28:11.200 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 msgr2=0x7f7f580a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 0x7f7f580a4f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f60004af0 tx=0x7f7f60005c50 comp rx=0 tx=0).stop 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 msgr2=0x7f7f6819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9f0 tx=0x7f7f5c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 shutdown_connections 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f580a2a90 0x7f7f580a4f50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f681077c0 0x7f7f6819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f68100680 0x7f7f6819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 --2- 172.21.15.116:0/1956058086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f680ffcc0 0x7f7f6819a1d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 >> 172.21.15.116:0/1956058086 conn(0x7f7f68076c40 msgr2=0x7f7f6810d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 shutdown_connections 2024-03-19T02:28:11.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.193+0000 7f7f6f261640 1 -- 172.21.15.116:0/1956058086 wait complete. 2024-03-19T02:28:11.205 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:28:11.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 -- 172.21.15.116:0/3792829468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4180102a90 msgr2=0x7f4180102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 --2- 172.21.15.116:0/3792829468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4180102a90 0x7f4180102e70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4170009860 tx=0x7f417002bb70 comp rx=0 tx=0).stop 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 -- 172.21.15.116:0/3792829468 shutdown_connections 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 --2- 172.21.15.116:0/3792829468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f418010be60 0x7f418010e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 --2- 172.21.15.116:0/3792829468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41801033b0 0x7f418010b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 --2- 172.21.15.116:0/3792829468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4180102a90 0x7f4180102e70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 -- 172.21.15.116:0/3792829468 >> 172.21.15.116:0/3792829468 conn(0x7f4180077820 msgr2=0x7f4180077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 -- 172.21.15.116:0/3792829468 shutdown_connections 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 -- 172.21.15.116:0/3792829468 wait complete. 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.318+0000 7f4185019640 1 Processor -- start 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 -- start start 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 0x7f4180198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 0x7f41801985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111b10 con 0x7f418010be60 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111990 con 0x7f41801033b0 2024-03-19T02:28:11.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f4185019640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111c90 con 0x7f4180102a90 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 0x7f4180198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 0x7f41801985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35028/0 (socket says 172.21.15.116:35028) 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 -- 172.21.15.116:0/379106414 learned_addr learned my addr 172.21.15.116:0/379106414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 -- 172.21.15.116:0/379106414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 msgr2=0x7f4180198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 0x7f4180198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 -- 172.21.15.116:0/379106414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 msgr2=0x7f41801985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 0x7f41801985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417f59e640 1 -- 172.21.15.116:0/379106414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418019d010 con 0x7f418010be60 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417ed9d640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 0x7f4180198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.319+0000 7f417e59c640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 0x7f41801985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f417f59e640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f417800eb10 tx=0x7f417800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417800ce20 con 0x7f418010be60 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f418019d300 con 0x7f418010be60 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4178005020 con 0x7f418010be60 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178010690 con 0x7f418010be60 2024-03-19T02:28:11.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.320+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41801a4b90 con 0x7f418010be60 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.321+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4180068610 con 0x7f418010be60 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.322+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f41780051d0 con 0x7f418010be60 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.323+0000 7f4167fff640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 0x7f41680a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.323+0000 7f417ed9d640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 0x7f41680a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.323+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f41780cfc40 con 0x7f418010be60 2024-03-19T02:28:11.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.323+0000 7f417ed9d640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 0x7f41680a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f417002c180 tx=0x7f41700056d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.326+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4178053740 con 0x7f418010be60 2024-03-19T02:28:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.575+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4180106f50 con 0x7f418010be60 2024-03-19T02:28:11.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.576+0000 7f4167fff640 1 -- 172.21.15.116:0/379106414 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11950 (secure 0 0 0) 0x7f417809c3d0 con 0x7f418010be60 2024-03-19T02:28:11.576 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:28:11.577 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:27:33.577980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:28:11.577 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 msgr2=0x7f41680a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 0x7f41680a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f417002c180 tx=0x7f41700056d0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 msgr2=0x7f418019c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f417800eb10 tx=0x7f417800ef10 comp rx=0 tx=0).stop 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 shutdown_connections 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41680a2a90 0x7f41680a4f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f418010be60 0x7f418019c8e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41801033b0 0x7f41801985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 --2- 172.21.15.116:0/379106414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4180102a90 0x7f4180198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 >> 172.21.15.116:0/379106414 conn(0x7f4180077820 msgr2=0x7f41800fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 shutdown_connections 2024-03-19T02:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.578+0000 7f4185019640 1 -- 172.21.15.116:0/379106414 wait complete. 2024-03-19T02:28:11.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3908142141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1006a0 msgr2=0x7f1c9c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3908142141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c100b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009900 tx=0x7f1c9002bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3908142141 shutdown_connections 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3908142141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3908142141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c100b20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3908142141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1066b0 0x7f1c9c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3908142141 >> 172.21.15.116:0/3908142141 conn(0x7f1c9c0778d0 msgr2=0x7f1c9c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3908142141 shutdown_connections 2024-03-19T02:28:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.698+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3908142141 wait complete. 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 Processor -- start 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 -- start start 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c1973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1066b0 0x7f1c9c19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c110930 con 0x7f1c9c1006a0 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1107b0 con 0x7f1c9c1066b0 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1ca0a0b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c110ab0 con 0x7f1c9c101060 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35054/0 (socket says 172.21.15.116:35054) 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3365167255 learned_addr learned my addr 172.21.15.116:0/3365167255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c1973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1066b0 0x7f1c9c19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3365167255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 msgr2=0x7f1c9c1973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c1973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3365167255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1066b0 msgr2=0x7f1c9c19b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1066b0 0x7f1c9c19b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3365167255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c19be60 con 0x7f1c9c1006a0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.699+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c1973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00e9f0 tx=0x7f1c8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1c8b7fe640 1 -- 172.21.15.116:0/3365167255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c00cd80 con 0x7f1c9c1006a0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c19c150 con 0x7f1c9c1006a0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1c8b7fe640 1 -- 172.21.15.116:0/3365167255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c8c0108e0 con 0x7f1c9c1006a0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1c8b7fe640 1 -- 172.21.15.116:0/3365167255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c018920 con 0x7f1c9c1006a0 2024-03-19T02:28:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.700+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c1a39e0 con 0x7f1c9c1006a0 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.702+0000 7f1c8b7fe640 1 -- 172.21.15.116:0/3365167255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f1c8c010a90 con 0x7f1c9c1006a0 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c8b7fe640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 0x7f1c840a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 Processor -- start 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 0x7f1c840a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c8b7fe640 1 -- 172.21.15.116:0/3365167255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f1c8c0cf750 con 0x7f1c9c1006a0 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 -- start start 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a31290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a33bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629b598b4b0 con 0x5629b5a28bf0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5629b5933d70 con 0x5629b5a282d0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629b5987550 con 0x5629b5a317d0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a31290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36546/0 (socket says 172.21.15.116:36546) 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3329432640 learned_addr learned my addr 172.21.15.116:0/3329432640 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 0x7f1c840a4f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1c90006010 tx=0x7f1c9002c6c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.703+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a33bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3329432640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 msgr2=0x5629b5a33bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a33bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3329432640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 msgr2=0x5629b5a31290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a31290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 -- 172.21.15.116:0/3329432640 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b5956210 con 0x5629b5a282d0 2024-03-19T02:28:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a31290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00c6a0 tx=0x7f1c8c010ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ab179e268e6a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c88ff9640 1 -- 172.21.15.116:0/3329432640 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0a3f80 con 0x5629b5a282d0 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c88ff9640 1 -- 172.21.15.116:0/3329432640 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c8c097f00 con 0x5629b5a282d0 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.704+0000 7f1c88ff9640 1 -- 172.21.15.116:0/3329432640 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c0afdc0 con 0x5629b5a282d0 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3329432640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 msgr2=0x5629b5a286b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00c6a0 tx=0x7f1c8c010ea0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3329432640 shutdown_connections 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a33bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a31290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3329432640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a286b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3329432640 >> 172.21.15.116:0/3329432640 conn(0x5629b5a22a00 msgr2=0x5629b5a22de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3329432640 shutdown_connections 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3329432640 wait complete. 2024-03-19T02:28:11.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.705+0000 7f1ca35fa740 1 -- start start 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 Processor -- start 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a2e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a2ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a2f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629b5a345c0 con 0x5629b5a28bf0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5629b5947310 con 0x5629b5a317d0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629b5a34740 con 0x5629b5a282d0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1ca35fa740 10 client.0 _ll_register_callbacks cb 0x5629b593b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a2e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a2ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a2f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a2e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38390/0 (socket says 172.21.15.116:38390) 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9a59c640 1 -- 172.21.15.116:0/301137638 learned_addr learned my addr 172.21.15.116:0/301137638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 -- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 msgr2=0x5629b5a2e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a2e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 -- 172.21.15.116:0/301137638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 msgr2=0x5629b5a2ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a2ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c9ad9d640 1 -- 172.21.15.116:0/301137638 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b569f3f0 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.706+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a2ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a2f270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005fd0 tx=0x7f1c9400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43fc146b2551f443 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9400fe60 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c097f00 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f1c94004b20 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c94004d20 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 10 client.25676 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5629b57697d0 con 0x5629b5a317d0 2024-03-19T02:28:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c8c097f00 con 0x5629b5a317d0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c94019070 con 0x5629b5a317d0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 10 client.25676 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 10 client.25676 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 20 client.25676 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5a440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 4 client.25676 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.707+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629b59e93e0 con 0x5629b5a41cb0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.708+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5a440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.708+0000 7f1c81ffb640 10 client.25676 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.708+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5a440d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9b25ec0740eb0d0 server_cookie=4c8ed33937608e9a in_seq=0 out_seq=0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1c81ffb640 1 -- 172.21.15.116:0/301137638 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629b59e93e0 con 0x5629b5a41cb0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1c81ffb640 10 client.25676 handle_command_reply: tid=0 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 client.25676 shutdown 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 msgr2=0x5629b5a2f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a2f270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005fd0 tx=0x7f1c9400f790 comp rx=0 tx=0).stop 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 shutdown_connections 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5a440d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a2f270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a2ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1c81ffb640 0 client.25676 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/301137638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a2e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 >> 172.21.15.116:0/301137638 conn(0x5629b5a22a00 msgr2=0x5629b5a24750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 shutdown_connections 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 1 -- 172.21.15.116:0/301137638 wait complete. 2024-03-19T02:28:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.709+0000 7f1ca35fa740 20 client.25676 trim_cache size 0 max 16384 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.725+0000 7f1ca35fa740 1 Processor -- start 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 -- start start 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1ada20 con 0x5629b5a28bf0 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1af560 con 0x5629b5a282d0 2024-03-19T02:28:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1ae0f0 con 0x5629b5a317d0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35076/0 (socket says 172.21.15.116:35076) 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 -- 172.21.15.116:0/1019691030 learned_addr learned my addr 172.21.15.116:0/1019691030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 -- 172.21.15.116:0/1019691030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 msgr2=0x5629b5a399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 -- 172.21.15.116:0/1019691030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 msgr2=0x5629b5a307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 -- 172.21.15.116:0/1019691030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b5a2cd40 con 0x5629b5a28bf0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009900 tx=0x7f1c9003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249a45dc06586e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c83fff640 1 -- 172.21.15.116:0/1019691030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90004450 con 0x5629b5a28bf0 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c83fff640 1 -- 172.21.15.116:0/1019691030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c90046070 con 0x5629b5a28bf0 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.726+0000 7f1c83fff640 1 -- 172.21.15.116:0/1019691030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9003f9b0 con 0x5629b5a28bf0 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- 172.21.15.116:0/1019691030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 msgr2=0x5629b5a49980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009900 tx=0x7f1c9003f000 comp rx=0 tx=0).stop 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- 172.21.15.116:0/1019691030 shutdown_connections 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a317d0 0x5629b5a399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a49980 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/1019691030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a282d0 0x5629b5a307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- 172.21.15.116:0/1019691030 >> 172.21.15.116:0/1019691030 conn(0x5629b5a22a00 msgr2=0x5629b5a21310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- 172.21.15.116:0/1019691030 shutdown_connections 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- 172.21.15.116:0/1019691030 wait complete. 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.727+0000 7f1ca35fa740 1 -- start start 2024-03-19T02:28:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 Processor -- start 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a35450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a35ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1aee90 con 0x5629b5a28bf0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1ae7c0 con 0x5629b5a317d0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1aba00 con 0x5629b5a282d0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1ca35fa740 10 client.0 _ll_register_callbacks cb 0x7f1c94009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35086/0 (socket says 172.21.15.116:35086) 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 -- 172.21.15.116:0/3296158 learned_addr learned my addr 172.21.15.116:0/3296158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c9ad9d640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a35ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a35450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 -- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 msgr2=0x5629b5a35450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a35450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 -- 172.21.15.116:0/3296158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 msgr2=0x5629b5a35ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a35ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 -- 172.21.15.116:0/3296158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b5aa7180 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c99d9b640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005e40 tx=0x7f1c90005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f367bbac710c650 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.728+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90003a00 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f1c90003ba0 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c90046070 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c9006a050 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 10 client.15852 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5629b5aa8440 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c90046070 con 0x5629b5a28bf0 2024-03-19T02:28:11.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c900346a0 con 0x5629b5a28bf0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 10 client.15852 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 10 client.15852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 20 client.15852 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5aaac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 4 client.15852 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629b59989f0 con 0x5629b5a41cb0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.729+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5aaac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1c81ffb640 10 client.15852 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1c9a59c640 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5aaac20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ebf05ca832e9cd0 server_cookie=bc9b0d96dcc9014d in_seq=0 out_seq=0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1c81ffb640 1 -- 172.21.15.116:0/3296158 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629b59989f0 con 0x5629b5a41cb0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1c81ffb640 10 client.15852 handle_command_reply: tid=0 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 client.15852 shutdown 2024-03-19T02:28:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 msgr2=0x5629b5a35990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005e40 tx=0x7f1c90005e70 comp rx=0 tx=0).stop 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 shutdown_connections 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5629b5a41cb0 0x5629b5aaac20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629b5a317d0 0x5629b5a35ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1c81ffb640 0 client.15852 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629b5a28bf0 0x5629b5a35990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 --2- 172.21.15.116:0/3296158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629b5a282d0 0x5629b5a35450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 >> 172.21.15.116:0/3296158 conn(0x5629b5a22a00 msgr2=0x5629b5a23d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 shutdown_connections 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 1 -- 172.21.15.116:0/3296158 wait complete. 2024-03-19T02:28:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.730+0000 7f1ca35fa740 20 client.15852 trim_cache size 0 max 16384 2024-03-19T02:28:11.739 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:11.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 msgr2=0x7f1c840a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 0x7f1c840a4f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1c90006010 tx=0x7f1c9002c6c0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 msgr2=0x7f1c9c196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00e9f0 tx=0x7f1c8c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 shutdown_connections 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1c840a2a90 0x7f1c840a4f50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c9c1066b0 0x7f1c9c19b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c9c101060 0x7f1c9c1973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 --2- 172.21.15.116:0/3365167255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c9c1006a0 0x7f1c9c196e70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 >> 172.21.15.116:0/3365167255 conn(0x7f1c9c0778d0 msgr2=0x7f1c9c102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 shutdown_connections 2024-03-19T02:28:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.731+0000 7f1ca0a0b640 1 -- 172.21.15.116:0/3365167255 wait complete. 2024-03-19T02:28:11.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 -- 172.21.15.116:0/934645463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1066b0 msgr2=0x7f884c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 --2- 172.21.15.116:0/934645463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1066b0 0x7f884c106a90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8848009930 tx=0x7f884802bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 -- 172.21.15.116:0/934645463 shutdown_connections 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 --2- 172.21.15.116:0/934645463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c101060 0x7f884c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 --2- 172.21.15.116:0/934645463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c1006a0 0x7f884c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 --2- 172.21.15.116:0/934645463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1066b0 0x7f884c106a90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 -- 172.21.15.116:0/934645463 >> 172.21.15.116:0/934645463 conn(0x7f884c0778d0 msgr2=0x7f884c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 -- 172.21.15.116:0/934645463 shutdown_connections 2024-03-19T02:28:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.853+0000 7f8853977640 1 -- 172.21.15.116:0/934645463 wait complete. 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 Processor -- start 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 -- start start 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 0x7f884c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c1066b0 0x7f884c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1109c0 con 0x7f884c1006a0 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f884c110840 con 0x7f884c101060 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.854+0000 7f8853977640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f884c110b40 con 0x7f884c1066b0 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35104/0 (socket says 172.21.15.116:35104) 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 -- 172.21.15.116:0/1877246840 learned_addr learned my addr 172.21.15.116:0/1877246840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8850f12640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 0x7f884c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851f14640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c1066b0 0x7f884c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 -- 172.21.15.116:0/1877246840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c1066b0 msgr2=0x7f884c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c1066b0 0x7f884c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 -- 172.21.15.116:0/1877246840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 msgr2=0x7f884c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 0x7f884c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 -- 172.21.15.116:0/1877246840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c1a6400 con 0x7f884c1006a0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8850f12640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 0x7f884c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8851713640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8848000c00 tx=0x7f8848004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f88427fc640 1 -- 172.21.15.116:0/1877246840 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848043020 con 0x7f884c1006a0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f88427fc640 1 -- 172.21.15.116:0/1877246840 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f884802cb30 con 0x7f884c1006a0 2024-03-19T02:28:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884c1a6690 con 0x7f884c1006a0 2024-03-19T02:28:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f88427fc640 1 -- 172.21.15.116:0/1877246840 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848003a10 con 0x7f884c1006a0 2024-03-19T02:28:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.855+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884c1a6c30 con 0x7f884c1006a0 2024-03-19T02:28:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.857+0000 7f88427fc640 1 -- 172.21.15.116:0/1877246840 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f8848046020 con 0x7f884c1006a0 2024-03-19T02:28:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 Processor -- start 2024-03-19T02:28:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 -- start start 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88427fc640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 0x7f88380a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x55593917e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x555939189bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559390e14b0 con 0x55593917ebf0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555939089d70 con 0x5559391877d0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88561fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559390dd550 con 0x55593917e2d0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f8851f14640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 0x7f88380a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f88427fc640 1 -- 172.21.15.116:0/1877246840 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f88480f1420 con 0x7f884c1006a0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f8850f12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.858+0000 7f8850f12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35116/0 (socket says 172.21.15.116:35116) 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 -- 172.21.15.116:0/964526178 learned_addr learned my addr 172.21.15.116:0/964526178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8851713640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x55593917e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8851f14640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x555939189bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 -- 172.21.15.116:0/964526178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 msgr2=0x55593917e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x55593917e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 -- 172.21.15.116:0/964526178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 msgr2=0x555939189bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x555939189bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 -- 172.21.15.116:0/964526178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559390ac210 con 0x55593917ebf0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8851713640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x55593917e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8851f14640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 0x7f88380a5020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8844005fd0 tx=0x7f8844011b40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f8850f12640 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f883c0114f0 tx=0x7f883c0118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172b71e0edd56775 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f88377fe640 1 -- 172.21.15.116:0/964526178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c01e070 con 0x55593917ebf0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f88377fe640 1 -- 172.21.15.116:0/964526178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f883c00f930 con 0x55593917ebf0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.859+0000 7f88377fe640 1 -- 172.21.15.116:0/964526178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c0199e0 con 0x55593917ebf0 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- 172.21.15.116:0/964526178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 msgr2=0x555939187290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f883c0114f0 tx=0x7f883c0118f0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- 172.21.15.116:0/964526178 shutdown_connections 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x555939189bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x555939187290 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 --2- 172.21.15.116:0/964526178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x55593917e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- 172.21.15.116:0/964526178 >> 172.21.15.116:0/964526178 conn(0x555939178a00 msgr2=0x555939178de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- 172.21.15.116:0/964526178 shutdown_connections 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- 172.21.15.116:0/964526178 wait complete. 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 -- start start 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.860+0000 7f88561fd740 1 Processor -- start 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 0x555939184d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 0x5559391852b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55593918a5c0 con 0x55593917e2d0 2024-03-19T02:28:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55593909d310 con 0x55593917ebf0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55593918a740 con 0x5559391877d0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f88561fd740 10 client.0 _ll_register_callbacks cb 0x5559390912d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35118/0 (socket says 172.21.15.116:35118) 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 -- 172.21.15.116:0/2585563256 learned_addr learned my addr 172.21.15.116:0/2585563256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8850f12640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 0x555939184d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851f14640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 0x5559391852b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 -- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 msgr2=0x5559391852b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 0x5559391852b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 -- 172.21.15.116:0/2585563256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 msgr2=0x555939184d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 0x555939184d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 -- 172.21.15.116:0/2585563256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555938df53f0 con 0x55593917e2d0 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8850f12640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 0x555939184d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851f14640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 0x5559391852b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:28:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8851713640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f88480b9f90 tx=0x7f88480bd450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b438f5413e7cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.861+0000 7f8834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88480d6060 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f88480d6200 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f883c00f930 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88480bd5c0 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 10 client.15864 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555938ebf7d0 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f883c00f930 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f88480bd820 con 0x55593917e2d0 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 10 client.15864 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 10 client.15864 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 20 client.15864 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555939197c20 0x55593919a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 4 client.15864 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55593913f3e0 con 0x555939197c20 2024-03-19T02:28:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.862+0000 7f8850f12640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555939197c20 0x55593919a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.863+0000 7f8834ff9640 10 client.15864 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:11.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.863+0000 7f8850f12640 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555939197c20 0x55593919a060 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=92d3a92cc0c30e9 server_cookie=a1afe9a0c1695cef in_seq=0 out_seq=0 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.863+0000 7f8834ff9640 1 -- 172.21.15.116:0/2585563256 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55593913f3e0 con 0x555939197c20 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.863+0000 7f8834ff9640 10 client.15864 handle_command_reply: tid=0 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.863+0000 7f88561fd740 1 client.15864 shutdown 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 msgr2=0x555939184830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f88480b9f90 tx=0x7f88480bd450 comp rx=0 tx=0).stop 2024-03-19T02:28:11.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 shutdown_connections 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559391877d0 0x5559391852b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917ebf0 0x555939184d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555939197c20 0x55593919a060 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 --2- 172.21.15.116:0/2585563256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917e2d0 0x555939184830 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 >> 172.21.15.116:0/2585563256 conn(0x555939178a00 msgr2=0x55593917a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f8834ff9640 0 client.15864 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 shutdown_connections 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 1 -- 172.21.15.116:0/2585563256 wait complete. 2024-03-19T02:28:11.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.864+0000 7f88561fd740 20 client.15864 trim_cache size 0 max 16384 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 Processor -- start 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 -- start start 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x555939183100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x5559391850d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1b0c50 con 0x55593917ebf0 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1b2790 con 0x5559391877d0 2024-03-19T02:28:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.878+0000 7f88561fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1b1320 con 0x55593917e2d0 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35134/0 (socket says 172.21.15.116:35134) 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 -- 172.21.15.116:0/1276267867 learned_addr learned my addr 172.21.15.116:0/1276267867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8851713640 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x555939183100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8851f14640 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x5559391850d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 -- 172.21.15.116:0/1276267867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 msgr2=0x555939183100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x555939183100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 -- 172.21.15.116:0/1276267867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 msgr2=0x5559391850d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x5559391850d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 -- 172.21.15.116:0/1276267867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559391837e0 con 0x55593917ebf0 2024-03-19T02:28:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8850f12640 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f883c0195f0 tx=0x7f883c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ab5375453ba7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8836ffd640 1 -- 172.21.15.116:0/1276267867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c01e040 con 0x55593917ebf0 2024-03-19T02:28:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8836ffd640 1 -- 172.21.15.116:0/1276267867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f883c0296c0 con 0x55593917ebf0 2024-03-19T02:28:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.879+0000 7f8836ffd640 1 -- 172.21.15.116:0/1276267867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c00f740 con 0x55593917ebf0 2024-03-19T02:28:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- 172.21.15.116:0/1276267867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 msgr2=0x5559391933c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f883c0195f0 tx=0x7f883c009de0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- 172.21.15.116:0/1276267867 shutdown_connections 2024-03-19T02:28:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559391877d0 0x5559391850d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55593917ebf0 0x5559391933c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 --2- 172.21.15.116:0/1276267867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917e2d0 0x555939183100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- 172.21.15.116:0/1276267867 >> 172.21.15.116:0/1276267867 conn(0x555939178a00 msgr2=0x555939177ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- 172.21.15.116:0/1276267867 shutdown_connections 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- 172.21.15.116:0/1276267867 wait complete. 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 -- start start 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.880+0000 7f88561fd740 1 Processor -- start 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917e2d0 0x555939181390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917ebf0 0x5559391818d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1b20c0 con 0x5559391877d0 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1b19f0 con 0x55593917e2d0 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f884c1aec30 con 0x55593917ebf0 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f88561fd740 10 client.0 _ll_register_callbacks cb 0x7f8848009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:11.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35140/0 (socket says 172.21.15.116:35140) 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 -- 172.21.15.116:0/634902668 learned_addr learned my addr 172.21.15.116:0/634902668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851713640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917e2d0 0x555939181390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8850f12640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917ebf0 0x5559391818d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 -- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917ebf0 msgr2=0x5559391818d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917ebf0 0x5559391818d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 -- 172.21.15.116:0/634902668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917e2d0 msgr2=0x555939181390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917e2d0 0x555939181390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 -- 172.21.15.116:0/634902668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559391fcfa0 con 0x5559391877d0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8851f14640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f884400ff70 tx=0x7f8844004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397e7fdcd9ffcb28 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.881+0000 7f8834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884401a040 con 0x5559391877d0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8844009040 con 0x5559391877d0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f883c0296c0 con 0x5559391877d0 2024-03-19T02:28:11.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8844004760 con 0x5559391877d0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 10 client.15870 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5559391fe250 con 0x5559391877d0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f883c0296c0 con 0x5559391877d0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f884401d030 con 0x5559391877d0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 10 client.15870 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 10 client.15870 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 20 client.15870 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5559391fea00 0x555939200e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 4 client.15870 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.882+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559390ee9f0 con 0x5559391fea00 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8851713640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5559391fea00 0x555939200e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8834ff9640 10 client.15870 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8851713640 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5559391fea00 0x555939200e20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50c4edaa1e8c0a96 server_cookie=1bf9fe4d83658940 in_seq=0 out_seq=0 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8834ff9640 1 -- 172.21.15.116:0/634902668 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559390ee9f0 con 0x5559391fea00 2024-03-19T02:28:11.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8834ff9640 10 client.15870 handle_command_reply: tid=0 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 client.15870 shutdown 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 msgr2=0x555939181e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f884400ff70 tx=0x7f8844004ab0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 shutdown_connections 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5559391fea00 0x555939200e20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559391877d0 0x555939181e10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55593917ebf0 0x5559391818d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f8834ff9640 0 client.15870 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 --2- 172.21.15.116:0/634902668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55593917e2d0 0x555939181390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.883+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 >> 172.21.15.116:0/634902668 conn(0x555939178a00 msgr2=0x555939177ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 shutdown_connections 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f88561fd740 1 -- 172.21.15.116:0/634902668 wait complete. 2024-03-19T02:28:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f88561fd740 20 client.15870 trim_cache size 0 max 16384 2024-03-19T02:28:11.891 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 msgr2=0x7f88380a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 0x7f88380a5020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8844005fd0 tx=0x7f8844011b40 comp rx=0 tx=0).stop 2024-03-19T02:28:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 msgr2=0x7f884c196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8848000c00 tx=0x7f8848004530 comp rx=0 tx=0).stop 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 shutdown_connections 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88380a2b60 0x7f88380a5020 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f884c1066b0 0x7f884c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f884c101060 0x7f884c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 --2- 172.21.15.116:0/1877246840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f884c1006a0 0x7f884c196bf0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 >> 172.21.15.116:0/1877246840 conn(0x7f884c0778d0 msgr2=0x7f884c102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 shutdown_connections 2024-03-19T02:28:11.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:11.884+0000 7f8853977640 1 -- 172.21.15.116:0/1877246840 wait complete. 2024-03-19T02:28:11.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:11 smithi116 ceph-mon[32263]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:11.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:11 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/379106414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:11.896 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:28:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.002+0000 7fad5a377640 1 -- 172.21.15.116:0/422206574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 msgr2=0x7fad5410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.002+0000 7fad5a377640 1 --2- 172.21.15.116:0/422206574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad5410e220 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009ca0 tx=0x7fad4c02c180 comp rx=0 tx=0).stop 2024-03-19T02:28:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 -- 172.21.15.116:0/422206574 shutdown_connections 2024-03-19T02:28:12.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- 172.21.15.116:0/422206574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad5410e220 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- 172.21.15.116:0/422206574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54103360 0x7fad5410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- 172.21.15.116:0/422206574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54102a40 0x7fad54102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 -- 172.21.15.116:0/422206574 >> 172.21.15.116:0/422206574 conn(0x7fad540778d0 msgr2=0x7fad54077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 -- 172.21.15.116:0/422206574 shutdown_connections 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 -- 172.21.15.116:0/422206574 wait complete. 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 Processor -- start 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 -- start start 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 0x7fad54100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 0x7fad54100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.003+0000 7fad5a377640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad5a377640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad54111a30 con 0x7fad5410bd80 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad5a377640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad541118b0 con 0x7fad54102a40 2024-03-19T02:28:12.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad5a377640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad54111bb0 con 0x7fad54103360 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35158/0 (socket says 172.21.15.116:35158) 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 -- 172.21.15.116:0/3917177509 learned_addr learned my addr 172.21.15.116:0/3917177509 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 0x7fad54100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad53fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 0x7fad54100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 -- 172.21.15.116:0/3917177509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 msgr2=0x7fad54100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 0x7fad54100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 -- 172.21.15.116:0/3917177509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 msgr2=0x7fad54100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 0x7fad54100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 -- 172.21.15.116:0/3917177509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad541a6430 con 0x7fad5410bd80 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad537fe640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 0x7fad54100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad53fff640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 0x7fad54100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad58914640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fad4c02c790 tx=0x7fad4c035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad517fa640 1 -- 172.21.15.116:0/3917177509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c039040 con 0x7fad5410bd80 2024-03-19T02:28:12.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad517fa640 1 -- 172.21.15.116:0/3917177509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad4c044070 con 0x7fad5410bd80 2024-03-19T02:28:12.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad541a66c0 con 0x7fad5410bd80 2024-03-19T02:28:12.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad517fa640 1 -- 172.21.15.116:0/3917177509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c032850 con 0x7fad5410bd80 2024-03-19T02:28:12.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.004+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad541a6bb0 con 0x7fad5410bd80 2024-03-19T02:28:12.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad517fa640 1 -- 172.21.15.116:0/3917177509 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7fad4c0329f0 con 0x7fad5410bd80 2024-03-19T02:28:12.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad517fa640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 0x7fad3c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad5d039740 1 Processor -- start 2024-03-19T02:28:12.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad5d039740 1 -- start start 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad53fff640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 0x7fad3c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad517fa640 1 -- 172.21.15.116:0/3917177509 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad4c0f1710 con 0x7fad5410bd80 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad5d039740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.007+0000 7fad5d039740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad5d039740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a6bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad5d039740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7049c34b0 con 0x55d704a602d0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad5d039740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d70496bd70 con 0x55d704a60bf0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad5d039740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d7049bf550 con 0x55d704a697d0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a606b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35172/0 (socket says 172.21.15.116:35172) 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36638/0 (socket says 172.21.15.116:36638) 2024-03-19T02:28:12.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad53fff640 1 -- 172.21.15.116:0/3030278148 learned_addr learned my addr 172.21.15.116:0/3030278148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad58914640 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a6bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 -- 172.21.15.116:0/3030278148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 msgr2=0x55d704a6bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a6bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 -- 172.21.15.116:0/3030278148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 msgr2=0x55d704a606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad53fff640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 0x7fad3c0a4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad44013920 tx=0x7fad44013d20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 -- 172.21.15.116:0/3030278148 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70498e210 con 0x55d704a60bf0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad58914640 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a6bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad537fe640 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fad48009f20 tx=0x7fad48011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6444fc1948f35952 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad3a7fc640 1 -- 172.21.15.116:0/3030278148 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48004230 con 0x55d704a60bf0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad3a7fc640 1 -- 172.21.15.116:0/3030278148 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad480043d0 con 0x55d704a60bf0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.008+0000 7fad3a7fc640 1 -- 172.21.15.116:0/3030278148 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480035f0 con 0x55d704a60bf0 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 -- 172.21.15.116:0/3030278148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 msgr2=0x55d704a69290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fad48009f20 tx=0x7fad48011810 comp rx=0 tx=0).stop 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 -- 172.21.15.116:0/3030278148 shutdown_connections 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a6bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a60bf0 0x55d704a69290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 --2- 172.21.15.116:0/3030278148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 -- 172.21.15.116:0/3030278148 >> 172.21.15.116:0/3030278148 conn(0x55d704a5aa00 msgr2=0x55d704a5ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 -- 172.21.15.116:0/3030278148 shutdown_connections 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.009+0000 7fad5d039740 1 -- 172.21.15.116:0/3030278148 wait complete. 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 -- start start 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 Processor -- start 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 0x55d704a66850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d704a6c5c0 con 0x55d704a60bf0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d70497f310 con 0x55d704a697d0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d704a6c740 con 0x55d704a602d0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad5d039740 10 client.0 _ll_register_callbacks cb 0x55d7049732d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35180/0 (socket says 172.21.15.116:35180) 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad53fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 0x55d704a66850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad537fe640 1 -- 172.21.15.116:0/3970478091 learned_addr learned my addr 172.21.15.116:0/3970478091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.010+0000 7fad58914640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 -- 172.21.15.116:0/3970478091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 msgr2=0x55d704a66850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 0x55d704a66850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 -- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 msgr2=0x55d704a672d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a672d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 -- 172.21.15.116:0/3970478091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7046d73f0 con 0x55d704a60bf0 2024-03-19T02:28:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad53fff640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 0x55d704a66850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad58914640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a672d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad537fe640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fad48005390 tx=0x7fad480053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e650b347df28219 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad337fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48028020 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad4800e470 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad480043d0 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad48033050 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad5d039740 10 client.15882 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d7047a17d0 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.011+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad480043d0 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fad48019430 con 0x55d704a60bf0 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 10 client.15882 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 10 client.15882 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:28:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 20 client.15882 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704a79b40 0x55d704a7bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 4 client.15882 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d704a213e0 con 0x55d704a79b40 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad53fff640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704a79b40 0x55d704a7bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad53fff640 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704a79b40 0x55d704a7bf30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e42279e09658d88a server_cookie=2b72ad71bde7e72c in_seq=0 out_seq=0 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.012+0000 7fad337fe640 10 client.15882 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad337fe640 1 -- 172.21.15.116:0/3970478091 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d704a213e0 con 0x55d704a79b40 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad337fe640 10 client.15882 handle_command_reply: tid=0 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 client.15882 shutdown 2024-03-19T02:28:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 msgr2=0x55d704a66d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fad48005390 tx=0x7fad480053c0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 shutdown_connections 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704a79b40 0x55d704a7bf30 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a672d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad337fe640 0 client.15882 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a66d90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 --2- 172.21.15.116:0/3970478091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a602d0 0x55d704a66850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 >> 172.21.15.116:0/3970478091 conn(0x55d704a5aa00 msgr2=0x55d704a5c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 shutdown_connections 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 1 -- 172.21.15.116:0/3970478091 wait complete. 2024-03-19T02:28:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.013+0000 7fad5d039740 20 client.15882 trim_cache size 0 max 16384 2024-03-19T02:28:12.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 Processor -- start 2024-03-19T02:28:12.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 -- start start 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a60bf0 0x55d704a68680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad541b0cf0 con 0x55d704a602d0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad541b2830 con 0x55d704a697d0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad5d039740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad541b13c0 con 0x55d704a60bf0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35190/0 (socket says 172.21.15.116:35190) 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad53fff640 1 -- 172.21.15.116:0/2568292035 learned_addr learned my addr 172.21.15.116:0/2568292035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad58914640 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.039+0000 7fad537fe640 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a60bf0 0x55d704a68680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 -- 172.21.15.116:0/2568292035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a60bf0 msgr2=0x55d704a68680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a60bf0 0x55d704a68680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 -- 172.21.15.116:0/2568292035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 msgr2=0x55d704a816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a816a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 -- 172.21.15.116:0/2568292035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d704a5b5d0 con 0x55d704a602d0 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad53fff640 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fad44009c60 tx=0x7fad44020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedaa3540cca44e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad39ffb640 1 -- 172.21.15.116:0/2568292035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44033070 con 0x55d704a602d0 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad39ffb640 1 -- 172.21.15.116:0/2568292035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad440120d0 con 0x55d704a602d0 2024-03-19T02:28:12.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.040+0000 7fad39ffb640 1 -- 172.21.15.116:0/2568292035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4402d600 con 0x55d704a602d0 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 -- 172.21.15.116:0/2568292035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 msgr2=0x55d704a75360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fad44009c60 tx=0x7fad44020000 comp rx=0 tx=0).stop 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 -- 172.21.15.116:0/2568292035 shutdown_connections 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a697d0 0x55d704a816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a60bf0 0x55d704a68680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 --2- 172.21.15.116:0/2568292035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a602d0 0x55d704a75360 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 -- 172.21.15.116:0/2568292035 >> 172.21.15.116:0/2568292035 conn(0x55d704a5aa00 msgr2=0x55d704a67f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 -- 172.21.15.116:0/2568292035 shutdown_connections 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.041+0000 7fad5d039740 1 -- 172.21.15.116:0/2568292035 wait complete. 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 -- start start 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 Processor -- start 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a602d0 0x55d704a63460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a63ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad541b2160 con 0x55d704a60bf0 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad541b1a90 con 0x55d704a602d0 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad541aecd0 con 0x55d704a697d0 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad5d039740 10 client.0 _ll_register_callbacks cb 0x7fad44009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35194/0 (socket says 172.21.15.116:35194) 2024-03-19T02:28:12.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.042+0000 7fad537fe640 1 -- 172.21.15.116:0/564858473 learned_addr learned my addr 172.21.15.116:0/564858473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad53fff640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a602d0 0x55d704a63460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad58914640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a63ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 -- 172.21.15.116:0/564858473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 msgr2=0x55d704a63ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a63ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 -- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a602d0 msgr2=0x55d704a63460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a602d0 0x55d704a63460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 -- 172.21.15.116:0/564858473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d704adecd0 con 0x55d704a60bf0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad537fe640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55d704a5b7d0 tx=0x7fad48011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c160311421975a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad337fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4803c060 con 0x55d704a60bf0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad48019c70 con 0x55d704a60bf0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad440120d0 con 0x55d704a60bf0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad48028040 con 0x55d704a60bf0 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad5d039740 10 client.15888 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d704adff10 con 0x55d704a60bf0 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.043+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad440120d0 con 0x55d704a60bf0 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fad48004040 con 0x55d704a60bf0 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 10 client.15888 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 10 client.15888 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 20 client.15888 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704ae0630 0x55d704ae2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 4 client.15888 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7049d09f0 con 0x55d704ae0630 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad53fff640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704ae0630 0x55d704ae2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad53fff640 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704ae0630 0x55d704ae2a50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5c60650607bcc6a server_cookie=6e6bcec115f8404c in_seq=0 out_seq=0 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad337fe640 10 client.15888 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad337fe640 1 -- 172.21.15.116:0/564858473 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7049d09f0 con 0x55d704ae0630 2024-03-19T02:28:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad337fe640 10 client.15888 handle_command_reply: tid=0 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.044+0000 7fad5d039740 1 client.15888 shutdown 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 msgr2=0x55d704a639a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55d704a5b7d0 tx=0x7fad48011d90 comp rx=0 tx=0).stop 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 shutdown_connections 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d704ae0630 0x55d704ae2a50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d704a697d0 0x55d704a63ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d704a60bf0 0x55d704a639a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad337fe640 0 client.15888 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 --2- 172.21.15.116:0/564858473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d704a602d0 0x55d704a63460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 >> 172.21.15.116:0/564858473 conn(0x55d704a5aa00 msgr2=0x55d704a5c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 shutdown_connections 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 1 -- 172.21.15.116:0/564858473 wait complete. 2024-03-19T02:28:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5d039740 20 client.15888 trim_cache size 0 max 16384 2024-03-19T02:28:12.052 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:12.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 msgr2=0x7fad3c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 0x7fad3c0a4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad44013920 tx=0x7fad44013d20 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 msgr2=0x7fad54101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.045+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fad4c02c790 tx=0x7fad4c035db0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 shutdown_connections 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad3c0a2a90 0x7fad3c0a4f50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5410bd80 0x7fad54101110 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad54103360 0x7fad54100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 --2- 172.21.15.116:0/3917177509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad54102a40 0x7fad54100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 >> 172.21.15.116:0/3917177509 conn(0x7fad540778d0 msgr2=0x7fad540fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 shutdown_connections 2024-03-19T02:28:12.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.046+0000 7fad5a377640 1 -- 172.21.15.116:0/3917177509 wait complete. 2024-03-19T02:28:12.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:28:12.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 -- 172.21.15.116:0/2328998493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d0101060 msgr2=0x7f71d010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/2328998493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d0101060 0x7f71d010d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009ca0 tx=0x7f71c802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 -- 172.21.15.116:0/2328998493 shutdown_connections 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/2328998493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d0101060 0x7f71d010d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/2328998493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/2328998493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d01066b0 0x7f71d0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 -- 172.21.15.116:0/2328998493 >> 172.21.15.116:0/2328998493 conn(0x7f71d00778d0 msgr2=0x7f71d0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 -- 172.21.15.116:0/2328998493 shutdown_connections 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.166+0000 7f71d7b5b640 1 -- 172.21.15.116:0/2328998493 wait complete. 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 Processor -- start 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 -- start start 2024-03-19T02:28:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d0101060 0x7f71d01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d01066b0 0x7f71d01a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0110970 con 0x7f71d0101060 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01107f0 con 0x7f71d01066b0 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d7b5b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0110af0 con 0x7f71d01006a0 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d0101060 0x7f71d01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d60f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d01066b0 0x7f71d01a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38556/0 (socket says 172.21.15.116:38556) 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 -- 172.21.15.116:0/250262018 learned_addr learned my addr 172.21.15.116:0/250262018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 -- 172.21.15.116:0/250262018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d01066b0 msgr2=0x7f71d01a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d01066b0 0x7f71d01a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 -- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d0101060 msgr2=0x7f71d01992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d0101060 0x7f71d01992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.167+0000 7f71d58f7640 1 -- 172.21.15.116:0/250262018 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d01a4350 con 0x7f71d01006a0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71d58f7640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00ea60 tx=0x7f71cc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71c6ffd640 1 -- 172.21.15.116:0/250262018 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00cde0 con 0x7f71d01006a0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d01a4520 con 0x7f71d01006a0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71c6ffd640 1 -- 172.21.15.116:0/250262018 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71cc0035f0 con 0x7f71d01006a0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71c6ffd640 1 -- 172.21.15.116:0/250262018 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc010690 con 0x7f71d01006a0 2024-03-19T02:28:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.168+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d01a4a90 con 0x7f71d01006a0 2024-03-19T02:28:12.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:11 smithi162 ceph-mon[35761]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:12.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:11 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/379106414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71c6ffd640 1 -- 172.21.15.116:0/250262018 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f71cc0037a0 con 0x7f71d01006a0 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 Processor -- start 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 -- start start 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 0x562a9f3516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f35a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a9f2b44b0 con 0x562a9f351bf0 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a9f25cd70 con 0x562a9f35a7d0 2024-03-19T02:28:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.171+0000 7f71da825740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a9f2b0550 con 0x562a9f3512d0 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f35a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d60f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 0x562a9f3516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36684/0 (socket says 172.21.15.116:36684) 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d60f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 0x562a9f3516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38568/0 (socket says 172.21.15.116:38568) 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 -- 172.21.15.116:0/2363057228 learned_addr learned my addr 172.21.15.116:0/2363057228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f35a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35214/0 (socket says 172.21.15.116:35214) 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71c6ffd640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 0x7f71bc0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 -- 172.21.15.116:0/2363057228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 msgr2=0x562a9f3516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 0x562a9f3516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 -- 172.21.15.116:0/2363057228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 msgr2=0x562a9f35a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f35a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 -- 172.21.15.116:0/2363057228 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9f27f210 con 0x562a9f35a7d0 2024-03-19T02:28:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d60f8640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 0x7f71bc0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71c6ffd640 1 -- 172.21.15.116:0/250262018 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71cc0cf690 con 0x7f71d01006a0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71d50f6640 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009ca0 tx=0x7f71c0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed38e78130edb3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71bbfff640 1 -- 172.21.15.116:0/2363057228 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c001e070 con 0x562a9f35a7d0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71bbfff640 1 -- 172.21.15.116:0/2363057228 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c0019ae0 con 0x562a9f35a7d0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.172+0000 7f71bbfff640 1 -- 172.21.15.116:0/2363057228 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c000e9b0 con 0x562a9f35a7d0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- 172.21.15.116:0/2363057228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 msgr2=0x562a9f35cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009ca0 tx=0x7f71c0011810 comp rx=0 tx=0).stop 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71d60f8640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 0x7f71bc0a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71c8004af0 tx=0x7f71c802e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- 172.21.15.116:0/2363057228 shutdown_connections 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f35cbc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f35a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 --2- 172.21.15.116:0/2363057228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f3512d0 0x562a9f3516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- 172.21.15.116:0/2363057228 >> 172.21.15.116:0/2363057228 conn(0x562a9f34ba00 msgr2=0x562a9f34bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- 172.21.15.116:0/2363057228 shutdown_connections 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- 172.21.15.116:0/2363057228 wait complete. 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.173+0000 7f71da825740 1 -- start start 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 Processor -- start 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f3577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 0x562a9f358250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a9f35d5c0 con 0x562a9f351bf0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a9f270310 con 0x562a9f3512d0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a9f35d740 con 0x562a9f35a7d0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71da825740 10 client.0 _ll_register_callbacks cb 0x562a9f2642d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d60f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 0x562a9f358250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35230/0 (socket says 172.21.15.116:35230) 2024-03-19T02:28:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f3577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 -- 172.21.15.116:0/1228766247 learned_addr learned my addr 172.21.15.116:0/1228766247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 -- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 msgr2=0x562a9f358250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 0x562a9f358250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 -- 172.21.15.116:0/1228766247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 msgr2=0x562a9f3577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.174+0000 7f71d50f6640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f3577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71d50f6640 1 -- 172.21.15.116:0/1228766247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9efc83f0 con 0x562a9f351bf0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71d60f8640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 0x562a9f358250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71d50f6640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009f50 tx=0x7f71c0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b590d55d9c9533 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0027020 con 0x562a9f351bf0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71c0005030 con 0x562a9f351bf0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71c0019ae0 con 0x562a9f351bf0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71c0032050 con 0x562a9f351bf0 2024-03-19T02:28:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 10 client.15900 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a9f0927d0 con 0x562a9f351bf0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c0019ae0 con 0x562a9f351bf0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71c0004bd0 con 0x562a9f351bf0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 10 client.15900 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 10 client.15900 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 20 client.15900 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f36aab0 0x562a9f36cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 4 client.15900 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.175+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a9f3123e0 con 0x562a9f36aab0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.176+0000 7f71d58f7640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f36aab0 0x562a9f36cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.176+0000 7f71b97fa640 10 client.15900 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.176+0000 7f71d58f7640 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f36aab0 0x562a9f36cea0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a3009ae86dc851 server_cookie=35c54ec8825a19e9 in_seq=0 out_seq=0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71b97fa640 1 -- 172.21.15.116:0/1228766247 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a9f3123e0 con 0x562a9f36aab0 2024-03-19T02:28:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71b97fa640 10 client.15900 handle_command_reply: tid=0 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 client.15900 shutdown 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 msgr2=0x562a9f357d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009f50 tx=0x7f71c0019660 comp rx=0 tx=0).stop 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 shutdown_connections 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f36aab0 0x562a9f36cea0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f35a7d0 0x562a9f358250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f351bf0 0x562a9f357d10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71b97fa640 0 client.15900 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 --2- 172.21.15.116:0/1228766247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f3577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 >> 172.21.15.116:0/1228766247 conn(0x562a9f34ba00 msgr2=0x562a9f34d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 shutdown_connections 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 1 -- 172.21.15.116:0/1228766247 wait complete. 2024-03-19T02:28:12.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.177+0000 7f71da825740 20 client.15900 trim_cache size 0 max 16384 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 Processor -- start 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 -- start start 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f366360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f372630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01aeb50 con 0x562a9f3512d0 2024-03-19T02:28:12.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b0690 con 0x562a9f35a7d0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71da825740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01af220 con 0x562a9f351bf0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35240/0 (socket says 172.21.15.116:35240) 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 -- 172.21.15.116:0/1503240307 learned_addr learned my addr 172.21.15.116:0/1503240307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d60f8640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f372630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d50f6640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f366360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 -- 172.21.15.116:0/1503240307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 msgr2=0x562a9f366360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f366360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 -- 172.21.15.116:0/1503240307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 msgr2=0x562a9f372630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.192+0000 7f71d58f7640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f372630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71d58f7640 1 -- 172.21.15.116:0/1503240307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9f321880 con 0x562a9f3512d0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71d60f8640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f372630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71d58f7640 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009920 tx=0x7f71cc0d5510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556603026308846b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71bb7fe640 1 -- 172.21.15.116:0/1503240307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc09b4f0 con 0x562a9f3512d0 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71bb7fe640 1 -- 172.21.15.116:0/1503240307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71cc003b30 con 0x562a9f3512d0 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.193+0000 7f71bb7fe640 1 -- 172.21.15.116:0/1503240307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc09b080 con 0x562a9f3512d0 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- 172.21.15.116:0/1503240307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 msgr2=0x562a9f359f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009920 tx=0x7f71cc0d5510 comp rx=0 tx=0).stop 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- 172.21.15.116:0/1503240307 shutdown_connections 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f35a7d0 0x562a9f372630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f366360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- 172.21.15.116:0/1503240307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f3512d0 0x562a9f359f00 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- 172.21.15.116:0/1503240307 >> 172.21.15.116:0/1503240307 conn(0x562a9f34ba00 msgr2=0x562a9f34aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- 172.21.15.116:0/1503240307 shutdown_connections 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- 172.21.15.116:0/1503240307 wait complete. 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- start start 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 Processor -- start 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f35e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f35a7d0 0x562a9f35ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01affc0 con 0x562a9f35a7d0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01af8f0 con 0x562a9f3512d0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.194+0000 7f71da825740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01acb30 con 0x562a9f351bf0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71da825740 10 client.0 _ll_register_callbacks cb 0x7f71c001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d60f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f35a7d0 0x562a9f35ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f35e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d58f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f35e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36712/0 (socket says 172.21.15.116:36712) 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38582/0 (socket says 172.21.15.116:38582) 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d58f7640 1 -- 172.21.15.116:0/709545185 learned_addr learned my addr 172.21.15.116:0/709545185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 -- 172.21.15.116:0/709545185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 msgr2=0x562a9f35e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f35e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 -- 172.21.15.116:0/709545185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f35a7d0 msgr2=0x562a9f35ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f35a7d0 0x562a9f35ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 -- 172.21.15.116:0/709545185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9f3cfc60 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71d50f6640 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009bc0 tx=0x7f71c0011d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4296fa5a81fb283 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0027070 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71c000ecd0 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a9f2c19f0 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71c001e040 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71da825740 10 client.25663 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.195+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a9f3d0f00 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562a9f2c19f0 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71c0019760 con 0x562a9f351bf0 2024-03-19T02:28:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 10 client.25663 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 10 client.25663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 20 client.25663 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f3d15e0 0x562a9f3d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 4 client.25663 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a9f2bbd40 con 0x562a9f3d15e0 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71d58f7640 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f3d15e0 0x562a9f3d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71d58f7640 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f3d15e0 0x562a9f3d3a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4152002857c9cd29 server_cookie=6fdace944cf76f68 in_seq=0 out_seq=0 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.196+0000 7f71b97fa640 10 client.25663 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71b97fa640 1 -- 172.21.15.116:0/709545185 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a9f2bbd40 con 0x562a9f3d15e0 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71b97fa640 10 client.25663 handle_command_reply: tid=0 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 client.25663 shutdown 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 msgr2=0x562a9f35e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009bc0 tx=0x7f71c0011d10 comp rx=0 tx=0).stop 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 shutdown_connections 2024-03-19T02:28:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a9f3d15e0 0x562a9f3d3a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a9f35a7d0 0x562a9f35ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a9f351bf0 0x562a9f35e7c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 --2- 172.21.15.116:0/709545185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a9f3512d0 0x562a9f35e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71b97fa640 0 client.25663 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 >> 172.21.15.116:0/709545185 conn(0x562a9f34ba00 msgr2=0x562a9f34a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 shutdown_connections 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 1 -- 172.21.15.116:0/709545185 wait complete. 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71da825740 20 client.25663 trim_cache size 0 max 16384 2024-03-19T02:28:12.204 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 msgr2=0x7f71bc0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 0x7f71bc0a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71c8004af0 tx=0x7f71c802e040 comp rx=0 tx=0).stop 2024-03-19T02:28:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 msgr2=0x7f71d0198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00ea60 tx=0x7f71cc00ef20 comp rx=0 tx=0).stop 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 shutdown_connections 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71bc0a2a90 0x7f71bc0a4f50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d01066b0 0x7f71d01a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71d0101060 0x7f71d01992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 --2- 172.21.15.116:0/250262018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01006a0 0x7f71d0198d90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.197+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 >> 172.21.15.116:0/250262018 conn(0x7f71d00778d0 msgr2=0x7f71d0102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.198+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 shutdown_connections 2024-03-19T02:28:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.198+0000 7f71d7b5b640 1 -- 172.21.15.116:0/250262018 wait complete. 2024-03-19T02:28:12.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:28:12.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:11 smithi154 ceph-mon[35693]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:11 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/379106414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 -- 172.21.15.116:0/400578441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc101060 msgr2=0x7f7efc10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 --2- 172.21.15.116:0/400578441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc101060 0x7f7efc10d570 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009c10 tx=0x7f7ef402bef0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 -- 172.21.15.116:0/400578441 shutdown_connections 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 --2- 172.21.15.116:0/400578441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc101060 0x7f7efc10d570 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 --2- 172.21.15.116:0/400578441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 --2- 172.21.15.116:0/400578441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc1066b0 0x7f7efc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 -- 172.21.15.116:0/400578441 >> 172.21.15.116:0/400578441 conn(0x7f7efc0778d0 msgr2=0x7f7efc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 -- 172.21.15.116:0/400578441 shutdown_connections 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 -- 172.21.15.116:0/400578441 wait complete. 2024-03-19T02:28:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.321+0000 7f7f020a1640 1 Processor -- start 2024-03-19T02:28:12.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 -- start start 2024-03-19T02:28:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc1066b0 0x7f7efc103780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc110930 con 0x7f7efc1066b0 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1107b0 con 0x7f7efc101060 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7f020a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc110ab0 con 0x7f7efc1006a0 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36734/0 (socket says 172.21.15.116:36734) 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 -- 172.21.15.116:0/902234955 learned_addr learned my addr 172.21.15.116:0/902234955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38606/0 (socket says 172.21.15.116:38606) 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efbfff640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc1066b0 0x7f7efc103780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 -- 172.21.15.116:0/902234955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 msgr2=0x7f7efc102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 -- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc1066b0 msgr2=0x7f7efc103780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc1066b0 0x7f7efc103780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.322+0000 7f7efaffd640 1 -- 172.21.15.116:0/902234955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc1a6420 con 0x7f7efc101060 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7efaffd640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00e9c0 tx=0x7f7eec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7ef8ff9640 1 -- 172.21.15.116:0/902234955 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec00ccc0 con 0x7f7efc101060 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7efc1a6710 con 0x7f7efc101060 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7ef8ff9640 1 -- 172.21.15.116:0/902234955 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7eec0054d0 con 0x7f7efc101060 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7ef8ff9640 1 -- 172.21.15.116:0/902234955 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec010430 con 0x7f7efc101060 2024-03-19T02:28:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.323+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7efc1a6cb0 con 0x7f7efc101060 2024-03-19T02:28:12.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.325+0000 7f7ef8ff9640 1 -- 172.21.15.116:0/902234955 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f7eec0046c0 con 0x7f7efc101060 2024-03-19T02:28:12.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.325+0000 7f7f04d6b740 1 Processor -- start 2024-03-19T02:28:12.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.325+0000 7f7f04d6b740 1 -- start start 2024-03-19T02:28:12.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7ef8ff9640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 0x7f7ee40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557fe9a014b0 con 0x557fe9a9e2d0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557fe99a9d70 con 0x557fe9a9ebf0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fe99fd550 con 0x557fe9aa77d0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efbfff640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 0x7f7ee40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7ef8ff9640 1 -- 172.21.15.116:0/902234955 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7eec0cf700 con 0x7f7efc101060 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35290/0 (socket says 172.21.15.116:35290) 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 -- 172.21.15.116:0/1147468322 learned_addr learned my addr 172.21.15.116:0/1147468322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efbfff640 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efaffd640 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 -- 172.21.15.116:0/1147468322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 msgr2=0x557fe9aa9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 -- 172.21.15.116:0/1147468322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9ebf0 msgr2=0x557fe9aa7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 -- 172.21.15.116:0/1147468322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe99cc210 con 0x557fe9a9e2d0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efbfff640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 0x7f7ee40a4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4004af0 tx=0x7f7ef40333e0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009c70 tx=0x7f7ef0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1516e428d04c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.326+0000 7f7ee1ffb640 1 -- 172.21.15.116:0/1147468322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0004230 con 0x557fe9a9e2d0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7ee1ffb640 1 -- 172.21.15.116:0/1147468322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ef00043d0 con 0x557fe9a9e2d0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7ee1ffb640 1 -- 172.21.15.116:0/1147468322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00196d0 con 0x557fe9a9e2d0 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1147468322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 msgr2=0x557fe9a9e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009c70 tx=0x7f7ef0011810 comp rx=0 tx=0).stop 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1147468322 shutdown_connections 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1147468322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9e2d0 0x557fe9a9e6b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1147468322 >> 172.21.15.116:0/1147468322 conn(0x557fe9a98a00 msgr2=0x557fe9a98de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1147468322 shutdown_connections 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.327+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1147468322 wait complete. 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 -- start start 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 Processor -- start 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557fe9aaa5c0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557fe99bd310 con 0x557fe9a9e2d0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fe9aaa740 con 0x557fe9aa77d0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7f04d6b740 10 client.0 _ll_register_callbacks cb 0x557fe99b12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36752/0 (socket says 172.21.15.116:36752) 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35306/0 (socket says 172.21.15.116:35306) 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efb7fe640 1 -- 172.21.15.116:0/709818392 learned_addr learned my addr 172.21.15.116:0/709818392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 -- 172.21.15.116:0/709818392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 msgr2=0x557fe9aa51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 -- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 msgr2=0x557fe9aa4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.328+0000 7f7efaffd640 1 -- 172.21.15.116:0/709818392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe97153f0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7efaffd640 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0cc950 tx=0x7f7eec09b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87403e287863077c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:12.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec04a170 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7eec0cf340 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef00043d0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7eec0cf540 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 10 client.15918 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557fe97df7d0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ef00043d0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7eec0a3aa0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 10 client.15918 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 20 client.15918 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9ab7a60 0x557fe9ab9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 4 client.15918 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.329+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fe9a5f3e0 con 0x557fe9ab7a60 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.330+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9ab7a60 0x557fe9ab9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.330+0000 7f7edaffd640 10 client.15918 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.330+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9ab7a60 0x557fe9ab9e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=568226fa9b92d33e server_cookie=4753f2109323ba08 in_seq=0 out_seq=0 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7edaffd640 1 -- 172.21.15.116:0/709818392 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557fe9a5f3e0 con 0x557fe9ab7a60 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7edaffd640 10 client.15918 handle_command_reply: tid=0 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 client.15918 shutdown 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 msgr2=0x557fe9aa4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0cc950 tx=0x7f7eec09b1b0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 shutdown_connections 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9ab7a60 0x557fe9ab9e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aa51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9aa4cb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/709818392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7edaffd640 0 client.15918 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:12.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 >> 172.21.15.116:0/709818392 conn(0x557fe9a98a00 msgr2=0x557fe9a9a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 shutdown_connections 2024-03-19T02:28:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 1 -- 172.21.15.116:0/709818392 wait complete. 2024-03-19T02:28:12.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.331+0000 7f7f04d6b740 20 client.15918 trim_cache size 0 max 16384 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 Processor -- start 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 -- start start 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9a93a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1b0cc0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1b2800 con 0x557fe9a9e2d0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.344+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1b1390 con 0x557fe9aa77d0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35320/0 (socket says 172.21.15.116:35320) 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 -- 172.21.15.116:0/1588138119 learned_addr learned my addr 172.21.15.116:0/1588138119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efbfff640 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9a93a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 -- 172.21.15.116:0/1588138119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 msgr2=0x557fe9a93a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9a93a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 -- 172.21.15.116:0/1588138119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 msgr2=0x557fe9aa6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 -- 172.21.15.116:0/1588138119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe9aa3720 con 0x557fe9a9ebf0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7efaffd640 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0cca60 tx=0x7f7eec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351a68eadd534eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7ee17fa640 1 -- 172.21.15.116:0/1588138119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec09b2d0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7ee17fa640 1 -- 172.21.15.116:0/1588138119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7eec0a3f30 con 0x557fe9a9ebf0 2024-03-19T02:28:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.345+0000 7f7ee17fa640 1 -- 172.21.15.116:0/1588138119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec09f8e0 con 0x557fe9a9ebf0 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1588138119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 msgr2=0x557fe9ab3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0cca60 tx=0x7f7eec005880 comp rx=0 tx=0).stop 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1588138119 shutdown_connections 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9a93a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9ab3360 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/1588138119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9aa6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1588138119 >> 172.21.15.116:0/1588138119 conn(0x557fe9a98a00 msgr2=0x557fe9a97ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1588138119 shutdown_connections 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- 172.21.15.116:0/1588138119 wait complete. 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 -- start start 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.346+0000 7f7f04d6b740 1 Processor -- start 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9b170a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9b175e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aaba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1b2130 con 0x557fe9a9ebf0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1b1a60 con 0x557fe9a9e2d0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc1aeca0 con 0x557fe9aa77d0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7f04d6b740 10 client.0 _ll_register_callbacks cb 0x7f7eec09b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9b170a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9b175e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9b170a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36776/0 (socket says 172.21.15.116:36776) 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9b175e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35322/0 (socket says 172.21.15.116:35322) 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aaba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efb7fe640 1 -- 172.21.15.116:0/4211036089 learned_addr learned my addr 172.21.15.116:0/4211036089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 -- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 msgr2=0x557fe9b170a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9b170a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 -- 172.21.15.116:0/4211036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 msgr2=0x557fe9b175e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9b175e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 -- 172.21.15.116:0/4211036089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe9aac0f0 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7efbfff640 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aaba30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ef402c430 tx=0x7f7ef402bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de88715025706786 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7edaffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.347+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4054070 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7ef403a040 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7eec0a3f30 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ef402ce20 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 10 client.25681 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557fe9b1e7d0 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7eec0a3f30 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7ef404d070 con 0x557fe9aa77d0 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 10 client.25681 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 10 client.25681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 20 client.25681 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9b1ef20 0x557fe9b21340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 4 client.25681 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fe9a0e9f0 con 0x557fe9b1ef20 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.348+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9b1ef20 0x557fe9b21340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7edaffd640 10 client.25681 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7efb7fe640 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9b1ef20 0x557fe9b21340 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a9dccc0d179883 server_cookie=a036f3df6eb04cfd in_seq=0 out_seq=0 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7edaffd640 1 -- 172.21.15.116:0/4211036089 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fe9a0e9f0 con 0x557fe9b1ef20 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7edaffd640 10 client.25681 handle_command_reply: tid=0 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 client.25681 shutdown 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 msgr2=0x557fe9aaba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aaba30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ef402c430 tx=0x7f7ef402bef0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 shutdown_connections 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557fe9b1ef20 0x557fe9b21340 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fe9aa77d0 0x557fe9aaba30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7edaffd640 0 client.25681 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fe9a9ebf0 0x557fe9b175e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 --2- 172.21.15.116:0/4211036089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fe9a9e2d0 0x557fe9b170a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 >> 172.21.15.116:0/4211036089 conn(0x557fe9a98a00 msgr2=0x557fe9aa4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 shutdown_connections 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 1 -- 172.21.15.116:0/4211036089 wait complete. 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.349+0000 7f7f04d6b740 20 client.25681 trim_cache size 0 max 16384 2024-03-19T02:28:12.356 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 msgr2=0x7f7ee40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 0x7f7ee40a4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4004af0 tx=0x7f7ef40333e0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 msgr2=0x7f7efc103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00e9c0 tx=0x7f7eec00ee80 comp rx=0 tx=0).stop 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 shutdown_connections 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee40a2a90 0x7f7ee40a4f50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7efc1066b0 0x7f7efc103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7efc101060 0x7f7efc103240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 --2- 172.21.15.116:0/902234955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7efc1006a0 0x7f7efc102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 >> 172.21.15.116:0/902234955 conn(0x7f7efc0778d0 msgr2=0x7f7efc068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 shutdown_connections 2024-03-19T02:28:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:12.350+0000 7f7f020a1640 1 -- 172.21.15.116:0/902234955 wait complete. 2024-03-19T02:28:13.363 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:28:13.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:28:13.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 -- 172.21.15.116:0/3761866373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c069d90 msgr2=0x7ff63c06a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 --2- 172.21.15.116:0/3761866373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c069d90 0x7ff63c06a1d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff628009930 tx=0x7ff62802bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 -- 172.21.15.116:0/3761866373 shutdown_connections 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 --2- 172.21.15.116:0/3761866373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c10e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 --2- 172.21.15.116:0/3761866373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c069d90 0x7ff63c06a1d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 --2- 172.21.15.116:0/3761866373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c1077e0 0x7ff63c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 -- 172.21.15.116:0/3761866373 >> 172.21.15.116:0/3761866373 conn(0x7ff63c077700 msgr2=0x7ff63c077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 -- 172.21.15.116:0/3761866373 shutdown_connections 2024-03-19T02:28:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.496+0000 7ff642ec6640 1 -- 172.21.15.116:0/3761866373 wait complete. 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 Processor -- start 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 -- start start 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 0x7ff63c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c111a90 con 0x7ff63c1077e0 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c111910 con 0x7ff63c06a710 2024-03-19T02:28:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff642ec6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c111c10 con 0x7ff63c069d90 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff640c62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 0x7ff63c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff640c62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38666/0 (socket says 172.21.15.116:38666) 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff640c62640 1 -- 172.21.15.116:0/2569479911 learned_addr learned my addr 172.21.15.116:0/2569479911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 -- 172.21.15.116:0/2569479911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 msgr2=0x7ff63c198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 -- 172.21.15.116:0/2569479911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 msgr2=0x7ff63c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff641463640 1 -- 172.21.15.116:0/2569479911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c19d060 con 0x7ff63c1077e0 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff63bfff640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.497+0000 7ff640c62640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff641463640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 0x7ff63c19c930 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff634009fd0 tx=0x7ff63400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff639ffb640 1 -- 172.21.15.116:0/2569479911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63400ce20 con 0x7ff63c1077e0 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff639ffb640 1 -- 172.21.15.116:0/2569479911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff634005020 con 0x7ff63c1077e0 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63c19d350 con 0x7ff63c1077e0 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff639ffb640 1 -- 172.21.15.116:0/2569479911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634010690 con 0x7ff63c1077e0 2024-03-19T02:28:13.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.498+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63c1a4be0 con 0x7ff63c1077e0 2024-03-19T02:28:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 Processor -- start 2024-03-19T02:28:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff639ffb640 1 -- 172.21.15.116:0/2569479911 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7ff6340051d0 con 0x7ff63c1077e0 2024-03-19T02:28:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 -- start start 2024-03-19T02:28:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 0x563ed01c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ed0119160 con 0x563ed01b5ca0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ed00c1e10 con 0x563ed01b5380 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.501+0000 7ff645b8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ed0115200 con 0x563ed01be880 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35360/0 (socket says 172.21.15.116:35360) 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 -- 172.21.15.116:0/3041715040 learned_addr learned my addr 172.21.15.116:0/3041715040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff639ffb640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 0x7ff62c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 0x563ed01c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff639ffb640 1 -- 172.21.15.116:0/2569479911 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff6340cfba0 con 0x7ff63c1077e0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff641463640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 0x7ff62c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 -- 172.21.15.116:0/3041715040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 msgr2=0x563ed01c0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 0x563ed01c0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 -- 172.21.15.116:0/3041715040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 msgr2=0x563ed01b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 -- 172.21.15.116:0/3041715040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ecfef57e0 con 0x563ed01b5ca0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 0x563ed01c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.502+0000 7ff640c62640 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff6300115f0 tx=0x7ff630011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73a9c3f54e92687 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff641463640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 0x7ff62c0a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff6340cce50 tx=0x7ff63400c890 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff626ffd640 1 -- 172.21.15.116:0/3041715040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001e070 con 0x563ed01b5ca0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff626ffd640 1 -- 172.21.15.116:0/3041715040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63000fda0 con 0x563ed01b5ca0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff626ffd640 1 -- 172.21.15.116:0/3041715040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630019e50 con 0x563ed01b5ca0 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 -- 172.21.15.116:0/3041715040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 msgr2=0x563ed01be340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff6300115f0 tx=0x7ff630011ab0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 -- 172.21.15.116:0/3041715040 shutdown_connections 2024-03-19T02:28:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01be880 0x563ed01c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01be340 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3041715040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.503+0000 7ff645b8e740 1 -- 172.21.15.116:0/3041715040 >> 172.21.15.116:0/3041715040 conn(0x563ed01afab0 msgr2=0x563ed01afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.504+0000 7ff645b8e740 1 -- 172.21.15.116:0/3041715040 shutdown_connections 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.504+0000 7ff645b8e740 1 -- 172.21.15.116:0/3041715040 wait complete. 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.504+0000 7ff645b8e740 1 -- start start 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.504+0000 7ff645b8e740 1 Processor -- start 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ed01c1670 con 0x563ed01be880 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ed00d51a0 con 0x563ed01b5380 2024-03-19T02:28:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ed01c17f0 con 0x563ed01b5ca0 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff645b8e740 10 client.0 _ll_register_callbacks cb 0x563ecfe2cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff640c62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff640c62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38688/0 (socket says 172.21.15.116:38688) 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35376/0 (socket says 172.21.15.116:35376) 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff640c62640 1 -- 172.21.15.116:0/528718789 learned_addr learned my addr 172.21.15.116:0/528718789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 -- 172.21.15.116:0/528718789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 msgr2=0x563ed01bbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 -- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 msgr2=0x563ed01bb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff63bfff640 1 -- 172.21.15.116:0/528718789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ecff55f70 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.505+0000 7ff640c62640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff63bfff640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff628009230 tx=0x7ff62802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5884fc45a3bb5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628003800 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff6280039a0 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63000fda0 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff628057050 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff645b8e740 10 client.15936 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ecfded170 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63000fda0 con 0x563ed01be880 2024-03-19T02:28:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.506+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff628043940 con 0x563ed01be880 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 10 client.15936 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 20 client.15936 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed01cea70 0x563ed01d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 4 client.15936 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ed01266a0 con 0x563ed01cea70 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff640c62640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed01cea70 0x563ed01d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff61bfff640 10 client.15936 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.507+0000 7ff640c62640 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed01cea70 0x563ed01d0ea0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64e5165be4aaf8af server_cookie=eac55d38dbc4e06e in_seq=0 out_seq=0 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff61bfff640 1 -- 172.21.15.116:0/528718789 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ed01266a0 con 0x563ed01cea70 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff61bfff640 10 client.15936 handle_command_reply: tid=0 2024-03-19T02:28:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 client.15936 shutdown 2024-03-19T02:28:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 msgr2=0x563ed01bc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff628009230 tx=0x7ff62802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 shutdown_connections 2024-03-19T02:28:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed01cea70 0x563ed01d0ea0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01be880 0x563ed01bc280 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5ca0 0x563ed01bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 --2- 172.21.15.116:0/528718789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 >> 172.21.15.116:0/528718789 conn(0x563ed01afab0 msgr2=0x563ed01b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff61bfff640 0 client.15936 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 shutdown_connections 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 1 -- 172.21.15.116:0/528718789 wait complete. 2024-03-19T02:28:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.508+0000 7ff645b8e740 20 client.15936 trim_cache size 0 max 16384 2024-03-19T02:28:13.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.525+0000 7ff645b8e740 1 Processor -- start 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 -- start start 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b7130 0x563ed01ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1aeb30 con 0x563ed01b5ca0 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1b0670 con 0x563ed01b5380 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff645b8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1af200 con 0x563ed01b7130 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36822/0 (socket says 172.21.15.116:36822) 2024-03-19T02:28:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35392/0 (socket says 172.21.15.116:35392) 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 -- 172.21.15.116:0/289665877 learned_addr learned my addr 172.21.15.116:0/289665877 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b7130 0x563ed01ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 -- 172.21.15.116:0/289665877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b7130 msgr2=0x563ed01ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b7130 0x563ed01ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 -- 172.21.15.116:0/289665877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 msgr2=0x563ed01ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff63bfff640 1 -- 172.21.15.116:0/289665877 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ed01ab160 con 0x563ed01b5380 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.526+0000 7ff640c62640 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff63bfff640 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff62802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5855c4caec8024e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff6267fc640 1 -- 172.21.15.116:0/289665877 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628043450 con 0x563ed01b5380 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff6267fc640 1 -- 172.21.15.116:0/289665877 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff628043aa0 con 0x563ed01b5380 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff6267fc640 1 -- 172.21.15.116:0/289665877 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628003ee0 con 0x563ed01b5380 2024-03-19T02:28:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 -- 172.21.15.116:0/289665877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 msgr2=0x563ed01bdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff62802bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 -- 172.21.15.116:0/289665877 shutdown_connections 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b7130 0x563ed01ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 --2- 172.21.15.116:0/289665877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b5380 0x563ed01bdf30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.527+0000 7ff645b8e740 1 -- 172.21.15.116:0/289665877 >> 172.21.15.116:0/289665877 conn(0x563ed01afab0 msgr2=0x563ed01ae2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- 172.21.15.116:0/289665877 shutdown_connections 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- 172.21.15.116:0/289665877 wait complete. 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- start start 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 Processor -- start 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5380 0x563ed01c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 0x563ed0233da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1affa0 con 0x563ed01b5ca0 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1af8d0 con 0x563ed01b7130 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1acb10 con 0x563ed01b5380 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff645b8e740 10 client.0 _ll_register_callbacks cb 0x7ff628009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff640c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35398/0 (socket says 172.21.15.116:35398) 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff640c62640 1 -- 172.21.15.116:0/3738806312 learned_addr learned my addr 172.21.15.116:0/3738806312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5380 0x563ed01c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.528+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 0x563ed0233da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 -- 172.21.15.116:0/3738806312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5380 msgr2=0x563ed01c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5380 0x563ed01c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 -- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 msgr2=0x563ed0233da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 0x563ed0233da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 -- 172.21.15.116:0/3738806312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ed01c34f0 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 0x563ed0233da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff640c62640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff63001afd0 tx=0x7ff63000f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e129a0476329c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001e040 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff630019b20 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff628043aa0 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff630019d20 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff645b8e740 10 client.15942 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ed0239800 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff628043aa0 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.529+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff63000f6c0 con 0x563ed01b5ca0 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 10 client.15942 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 20 client.15942 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed0239df0 0x563ed023c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 4 client.15942 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563ed01209f0 con 0x563ed0239df0 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed0239df0 0x563ed023c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff61bfff640 10 client.15942 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.530+0000 7ff63bfff640 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed0239df0 0x563ed023c210 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5d28f6d2e48cc9 server_cookie=3deb90e7f13f5de0 in_seq=0 out_seq=0 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff61bfff640 1 -- 172.21.15.116:0/3738806312 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563ed01209f0 con 0x563ed0239df0 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff61bfff640 10 client.15942 handle_command_reply: tid=0 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 client.15942 shutdown 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 msgr2=0x563ed01c2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff63001afd0 tx=0x7ff63000f460 comp rx=0 tx=0).stop 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 shutdown_connections 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ed0239df0 0x563ed023c210 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff61bfff640 0 client.15942 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ed01b7130 0x563ed0233da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ed01b5ca0 0x563ed01c2bf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 --2- 172.21.15.116:0/3738806312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ed01b5380 0x563ed01c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.531+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 >> 172.21.15.116:0/3738806312 conn(0x563ed01afab0 msgr2=0x563ed01bc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.532+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 shutdown_connections 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.532+0000 7ff645b8e740 1 -- 172.21.15.116:0/3738806312 wait complete. 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.532+0000 7ff645b8e740 20 client.15942 trim_cache size 0 max 16384 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:28:13.539 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "fe009754-cac1-4643-b23e-6fe866323602", 2024-03-19T02:28:13.540 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:28:13.540 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 msgr2=0x7ff62c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 0x7ff62c0a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff6340cce50 tx=0x7ff63400c890 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 msgr2=0x7ff63c19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 0x7ff63c19c930 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff634009fd0 tx=0x7ff63400ef10 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 shutdown_connections 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff62c0a2a90 0x7ff62c0a4f50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff63c1077e0 0x7ff63c19c930 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff63c06a710 0x7ff63c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 --2- 172.21.15.116:0/2569479911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff63c069d90 0x7ff63c198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 >> 172.21.15.116:0/2569479911 conn(0x7ff63c077700 msgr2=0x7ff63c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 shutdown_connections 2024-03-19T02:28:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.533+0000 7ff642ec6640 1 -- 172.21.15.116:0/2569479911 wait complete. 2024-03-19T02:28:13.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:28:13.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 -- 172.21.15.116:0/3730612117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8107870 msgr2=0x7f05a8107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 --2- 172.21.15.116:0/3730612117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8107870 0x7f05a8107c50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f059c009890 tx=0x7f059c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 -- 172.21.15.116:0/3730612117 shutdown_connections 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 --2- 172.21.15.116:0/3730612117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a80693e0 0x7f05a810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 --2- 172.21.15.116:0/3730612117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a8068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 --2- 172.21.15.116:0/3730612117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8107870 0x7f05a8107c50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 -- 172.21.15.116:0/3730612117 >> 172.21.15.116:0/3730612117 conn(0x7f05a8077820 msgr2=0x7f05a8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 -- 172.21.15.116:0/3730612117 shutdown_connections 2024-03-19T02:28:13.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.656+0000 7f05aee44640 1 -- 172.21.15.116:0/3730612117 wait complete. 2024-03-19T02:28:13.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 Processor -- start 2024-03-19T02:28:13.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 -- start start 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 0x7f05a819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 0x7f05a819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8102100 con 0x7f05a80693e0 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8101f80 con 0x7f05a8068a20 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05aee44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8102280 con 0x7f05a8107870 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 0x7f05a819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05acbe0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05ad3e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 0x7f05a819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05acbe0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36846/0 (socket says 172.21.15.116:36846) 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 0x7f05a819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35412/0 (socket says 172.21.15.116:35412) 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05acbe0640 1 -- 172.21.15.116:0/2971395885 learned_addr learned my addr 172.21.15.116:0/2971395885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05ad3e1640 1 -- 172.21.15.116:0/2971395885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 msgr2=0x7f05a819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05ad3e1640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.657+0000 7f05ad3e1640 1 -- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 msgr2=0x7f05a819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05ad3e1640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 0x7f05a819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05ad3e1640 1 -- 172.21.15.116:0/2971395885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a819f1c0 con 0x7f05a8107870 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05ad3e1640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 0x7f05a819ea90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05a000ef00 tx=0x7f05a000c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05a5ffb640 1 -- 172.21.15.116:0/2971395885 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0004b20 con 0x7f05a8107870 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a819f4b0 con 0x7f05a8107870 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05a5ffb640 1 -- 172.21.15.116:0/2971395885 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a0003e20 con 0x7f05a8107870 2024-03-19T02:28:13.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05a5ffb640 1 -- 172.21.15.116:0/2971395885 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a000b570 con 0x7f05a8107870 2024-03-19T02:28:13.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.658+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a81a6d40 con 0x7f05a8107870 2024-03-19T02:28:13.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.661+0000 7f05a5ffb640 1 -- 172.21.15.116:0/2971395885 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f05a001d020 con 0x7f05a8107870 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.661+0000 7f05b1a2c740 1 Processor -- start 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.661+0000 7f05b1a2c740 1 -- start start 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.661+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.661+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb0c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5619cfa64500 con 0x5619cfb00a30 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619cfa0cbe0 con 0x5619cfb09f30 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5619cfa605a0 con 0x5619cfb01350 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35414/0 (socket says 172.21.15.116:35414) 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 -- 172.21.15.116:0/4165030373 learned_addr learned my addr 172.21.15.116:0/4165030373 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05acbe0640 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a5ffb640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 0x7f05980a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 -- 172.21.15.116:0/4165030373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 msgr2=0x5619cfb099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb099f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 -- 172.21.15.116:0/4165030373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 msgr2=0x5619cfb0c320 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb0c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05ad3e1640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 0x7f05980a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 -- 172.21.15.116:0/4165030373 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619cfa2e000 con 0x5619cfb00a30 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05acbe0640 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a5ffb640 1 -- 172.21.15.116:0/2971395885 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f05a00cf350 con 0x7f05a8107870 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.662+0000 7f05a7fff640 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f05940058c0 tx=0x7f0594011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22cba7be45f2664 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.663+0000 7f05ad3e1640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 0x7f05980a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05a00cc5a0 tx=0x7f05a001ba10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.663+0000 7f0592ffd640 1 -- 172.21.15.116:0/4165030373 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059401e070 con 0x5619cfb00a30 2024-03-19T02:28:13.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.663+0000 7f0592ffd640 1 -- 172.21.15.116:0/4165030373 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0594009df0 con 0x5619cfb00a30 2024-03-19T02:28:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.663+0000 7f0592ffd640 1 -- 172.21.15.116:0/4165030373 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059400f720 con 0x5619cfb00a30 2024-03-19T02:28:13.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- 172.21.15.116:0/4165030373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 msgr2=0x5619cfb00e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f05940058c0 tx=0x7f0594011810 comp rx=0 tx=0).stop 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- 172.21.15.116:0/4165030373 shutdown_connections 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb0c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/4165030373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb00a30 0x5619cfb00e10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- 172.21.15.116:0/4165030373 >> 172.21.15.116:0/4165030373 conn(0x5619cfafb160 msgr2=0x5619cfafb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- 172.21.15.116:0/4165030373 shutdown_connections 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- 172.21.15.116:0/4165030373 wait complete. 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.664+0000 7f05b1a2c740 1 -- start start 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 Processor -- start 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb07640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb16190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5619cfb0cd20 con 0x5619cfb07b80 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619cfa20160 con 0x5619cfb09f30 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5619cfb0cea0 con 0x5619cfb01350 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.665+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05acbe0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb07640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38740/0 (socket says 172.21.15.116:38740) 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 -- 172.21.15.116:0/1451588254 learned_addr learned my addr 172.21.15.116:0/1451588254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb16190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 -- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 msgr2=0x5619cfb07640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb07640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 -- 172.21.15.116:0/1451588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 msgr2=0x5619cfb16190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb16190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 -- 172.21.15.116:0/1451588254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619cf8427d0 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05b1a2c740 10 client.0 _ll_register_callbacks cb 0x5619cfa5ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0594014040 tx=0x7f059400f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bedbbbd79ca6a567 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f0587fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0594009df0 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f0587fff640 1 -- 172.21.15.116:0/1451588254 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059401e040 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f0587fff640 1 -- 172.21.15.116:0/1451588254 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f0594004020 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f0587fff640 1 -- 172.21.15.116:0/1451588254 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0594004260 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05b1a2c740 10 client.25711 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.666+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5619cf7783f0 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f0587fff640 1 -- 172.21.15.116:0/1451588254 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0594024070 con 0x5619cfb01350 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 10 client.25711 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 10 client.25711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:13.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 20 client.25711 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 4 client.25711 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619cfa71a40 con 0x5619cfb1a590 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05acbe0640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb1c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f0587fff640 10 client.25711 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.667+0000 7f05acbe0640 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb1c980 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa708b69fbe35daa server_cookie=a762e8917f9c6772 in_seq=0 out_seq=0 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f0587fff640 1 -- 172.21.15.116:0/1451588254 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5619cfa71a40 con 0x5619cfb1a590 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f0587fff640 10 client.25711 handle_command_reply: tid=0 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 client.25711 shutdown 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 msgr2=0x5619cfb07100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0594014040 tx=0x7f059400f3c0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 shutdown_connections 2024-03-19T02:28:13.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb1c980 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb16190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f0587fff640 0 client.25711 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb07640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1451588254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb07100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.668+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 >> 172.21.15.116:0/1451588254 conn(0x5619cfafb160 msgr2=0x5619cfafceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.669+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 shutdown_connections 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.669+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1451588254 wait complete. 2024-03-19T02:28:13.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.669+0000 7f05b1a2c740 20 client.25711 trim_cache size 0 max 16384 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 Processor -- start 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 -- start start 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb07b80 0x5619cfb22110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb09f30 0x5619cfb120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81b0ca0 con 0x5619cfb09f30 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81b27e0 con 0x5619cfb01350 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81b1370 con 0x5619cfb07b80 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05acbe0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb07b80 0x5619cfb22110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:36870/0 (socket says 172.21.15.116:36870) 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 -- 172.21.15.116:0/1885490053 learned_addr learned my addr 172.21.15.116:0/1885490053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb09f30 0x5619cfb120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 -- 172.21.15.116:0/1885490053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb07b80 msgr2=0x5619cfb22110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb07b80 0x5619cfb22110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 -- 172.21.15.116:0/1885490053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb09f30 msgr2=0x5619cfb120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.703+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb09f30 0x5619cfb120e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.704+0000 7f05a7fff640 1 -- 172.21.15.116:0/1885490053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619cf7380f0 con 0x5619cfb01350 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.704+0000 7f05a7fff640 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05940097e0 tx=0x7f059401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4679cc69b2d061e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.704+0000 7f05927fc640 1 -- 172.21.15.116:0/1885490053 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05940054c0 con 0x5619cfb01350 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.704+0000 7f05927fc640 1 -- 172.21.15.116:0/1885490053 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0594009df0 con 0x5619cfb01350 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.704+0000 7f05927fc640 1 -- 172.21.15.116:0/1885490053 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05940273f0 con 0x5619cfb01350 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1885490053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 msgr2=0x5619cfb023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05940097e0 tx=0x7f059401f000 comp rx=0 tx=0).stop 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1885490053 shutdown_connections 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb09f30 0x5619cfb120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb07b80 0x5619cfb22110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/1885490053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb01350 0x5619cfb023f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.705+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1885490053 >> 172.21.15.116:0/1885490053 conn(0x5619cfafb160 msgr2=0x5619cfafc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.706+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1885490053 shutdown_connections 2024-03-19T02:28:13.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.706+0000 7f05b1a2c740 1 -- 172.21.15.116:0/1885490053 wait complete. 2024-03-19T02:28:13.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.706+0000 7f05b1a2c740 1 -- start start 2024-03-19T02:28:13.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.706+0000 7f05b1a2c740 1 Processor -- start 2024-03-19T02:28:13.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb03990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb03ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81b2110 con 0x5619cfb07b80 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81b1a40 con 0x5619cfb09f30 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38766/0 (socket says 172.21.15.116:38766) 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05a7fff640 1 -- 172.21.15.116:0/2095007448 learned_addr learned my addr 172.21.15.116:0/2095007448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05b1a2c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81aec80 con 0x5619cfb01350 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05acbe0640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb03990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05a7fff640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb03ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.707+0000 7f05a7fff640 1 -- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 msgr2=0x5619cfb03ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05a7fff640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb03ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05a7fff640 1 -- 172.21.15.116:0/2095007448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 msgr2=0x5619cfb03990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05a7fff640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb03990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05a7fff640 1 -- 172.21.15.116:0/2095007448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619cfb044e0 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05b1a2c740 10 client.0 _ll_register_callbacks cb 0x7f059401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05a7fff640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0594009400 tx=0x7f0594031b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55277357cd98cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0594009df0 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f0587fff640 10 client.25717 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f0587fff640 1 -- 172.21.15.116:0/2095007448 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594045060 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f0587fff640 1 -- 172.21.15.116:0/2095007448 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f0594040440 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f0587fff640 1 -- 172.21.15.116:0/2095007448 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0594040680 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05b1a2c740 10 client.25717 fetch_fsmap learned FSMap version 28 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5619cfb802f0 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f0587fff640 1 -- 172.21.15.116:0/2095007448 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0594024070 con 0x5619cfb01350 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.708+0000 7f05b1a2c740 10 client.25717 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T02:28:13.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 20 client.25717 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb82910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 4 client.25717 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5619cfb079d0 con 0x5619cfb1a590 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05acbe0640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb82910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f0587fff640 10 client.25717 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05acbe0640 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb82910 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5282b334bdba2b server_cookie=68747898cb6298f8 in_seq=0 out_seq=0 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f0587fff640 1 -- 172.21.15.116:0/2095007448 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5619cfb079d0 con 0x5619cfb1a590 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f0587fff640 10 client.25717 handle_command_reply: tid=0 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.709+0000 7f05b1a2c740 1 client.25717 shutdown 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 msgr2=0x5619cfb122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0594009400 tx=0x7f0594031b70 comp rx=0 tx=0).stop 2024-03-19T02:28:13.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 shutdown_connections 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5619cfb1a590 0x5619cfb82910 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619cfb09f30 0x5619cfb03ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f0587fff640 0 client.25717 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619cfb07b80 0x5619cfb03990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 --2- 172.21.15.116:0/2095007448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619cfb01350 0x5619cfb122b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 >> 172.21.15.116:0/2095007448 conn(0x5619cfafb160 msgr2=0x5619cfafccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 shutdown_connections 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 1 -- 172.21.15.116:0/2095007448 wait complete. 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.710+0000 7f05b1a2c740 20 client.25717 trim_cache size 0 max 16384 2024-03-19T02:28:13.716 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "fe009754-cac1-4643-b23e-6fe866323602": { 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "fe009754-cac1-4643-b23e-6fe866323602", 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:28:13.717 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 msgr2=0x7f05980a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 0x7f05980a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f05a00cc5a0 tx=0x7f05a001ba10 comp rx=0 tx=0).stop 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 msgr2=0x7f05a819ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 0x7f05a819ea90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05a000ef00 tx=0x7f05a000c740 comp rx=0 tx=0).stop 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 shutdown_connections 2024-03-19T02:28:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2a90 0x7f05980a4f50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05a8107870 0x7f05a819ea90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a80693e0 0x7f05a819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 --2- 172.21.15.116:0/2971395885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05a8068a20 0x7f05a819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 >> 172.21.15.116:0/2971395885 conn(0x7f05a8077820 msgr2=0x7f05a80fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 shutdown_connections 2024-03-19T02:28:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:13.711+0000 7f05aee44640 1 -- 172.21.15.116:0/2971395885 wait complete. 2024-03-19T02:28:13.726 INFO:tasks.cephfs.filesystem:scrub status for tag:fe009754-cac1-4643-b23e-6fe866323602 - {'path': '/', 'tag': 'fe009754-cac1-4643-b23e-6fe866323602', 'options': 'recursive,force'} 2024-03-19T02:28:14.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:13 smithi116 ceph-mon[32263]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:13 smithi162 ceph-mon[35761]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:13 smithi154 ceph-mon[35693]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-19T02:28:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:14 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:28:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:14 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:14 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:28:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:14 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:14 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:28:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:14 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:28:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:14 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:14 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:28:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:14 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:14 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:28:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:14 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:28:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:14 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:14 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:28:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:14 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:14 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:28:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:15 smithi162 ceph-mon[35761]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:15 smithi154 ceph-mon[35693]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:15 smithi116 ceph-mon[32263]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:17 smithi162 ceph-mon[35761]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:17 smithi154 ceph-mon[35693]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:17 smithi116 ceph-mon[32263]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-19T02:28:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:19 smithi162 ceph-mon[35761]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-19T02:28:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:19 smithi154 ceph-mon[35693]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-19T02:28:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:19 smithi116 ceph-mon[32263]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-19T02:28:22.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:21 smithi154 ceph-mon[35693]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-19T02:28:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:21 smithi162 ceph-mon[35761]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-19T02:28:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:21 smithi116 ceph-mon[32263]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-19T02:28:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:23 smithi162 ceph-mon[35761]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-19T02:28:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:23 smithi154 ceph-mon[35693]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-19T02:28:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:23 smithi116 ceph-mon[32263]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-19T02:28:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:25 smithi162 ceph-mon[35761]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-03-19T02:28:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:25 smithi154 ceph-mon[35693]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-03-19T02:28:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:25 smithi116 ceph-mon[32263]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-03-19T02:28:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:27 smithi162 ceph-mon[35761]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:27 smithi154 ceph-mon[35693]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:27 smithi116 ceph-mon[32263]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:29 smithi162 ceph-mon[35761]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:29 smithi154 ceph-mon[35693]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:29 smithi116 ceph-mon[32263]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:31 smithi162 ceph-mon[35761]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:31 smithi154 ceph-mon[35693]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:31 smithi116 ceph-mon[32263]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:33 smithi162 ceph-mon[35761]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:33 smithi154 ceph-mon[35693]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:33 smithi116 ceph-mon[32263]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:35 smithi162 ceph-mon[35761]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:35 smithi154 ceph-mon[35693]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:35 smithi116 ceph-mon[32263]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:28:36.616 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T02:28:36.616 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T02:28:36.618 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-19T02:28:36.618 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T02:28:36.621 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-19T02:28:36.647 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T02:28:36.649 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:28:36.649 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T02:28:37.388 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T02:28:37.456 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T02:28:37.497 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:28:37.498 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:28:37.498 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T02:28:37.538 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:28:37.541 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:28:37.541 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-19T02:28:37.597 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-19T02:28:37.598 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-19T02:28:37.599 DEBUG:teuthology.orchestra.run.smithi116:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-19T02:28:37.662 INFO:tasks.workunit.client.0.smithi116.stderr:+ dbench 1 2024-03-19T02:28:37.668 INFO:tasks.workunit.client.0.smithi116.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-19T02:28:37.668 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:28:37.669 INFO:tasks.workunit.client.0.smithi116.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-19T02:28:37.677 INFO:tasks.workunit.client.0.smithi116.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-19T02:28:37.778 INFO:tasks.workunit.client.0.smithi116.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-19T02:28:37.779 INFO:tasks.workunit.client.0.smithi116.stdout:releasing clients 2024-03-19T02:28:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:37 smithi162 ceph-mon[35761]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:37 smithi154 ceph-mon[35693]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:37 smithi116 ceph-mon[32263]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:38.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 926 21.78 MB/sec warmup 1 sec latency 13.840 ms 2024-03-19T02:28:39.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 1606 13.08 MB/sec warmup 2 sec latency 19.710 ms 2024-03-19T02:28:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:39 smithi162 ceph-mon[35761]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:39 smithi154 ceph-mon[35693]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:39 smithi116 ceph-mon[32263]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:28:40.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 2337 9.28 MB/sec warmup 3 sec latency 16.568 ms 2024-03-19T02:28:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:40 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:28:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:40 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:28:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:40 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:28:41.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 3171 8.53 MB/sec warmup 4 sec latency 16.751 ms 2024-03-19T02:28:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:41 smithi162 ceph-mon[35761]: pgmap v29: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 32 op/s 2024-03-19T02:28:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:41 smithi154 ceph-mon[35693]: pgmap v29: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 32 op/s 2024-03-19T02:28:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:41 smithi116 ceph-mon[32263]: pgmap v29: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.8 MiB/s wr, 32 op/s 2024-03-19T02:28:42.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 3650 7.73 MB/sec warmup 5 sec latency 15.400 ms 2024-03-19T02:28:43.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:28:43.727 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:28:43.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 4106 6.68 MB/sec warmup 6 sec latency 19.825 ms 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 -- 172.21.15.116:0/2809739868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b41000e0 msgr2=0x7f15b4100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 --2- 172.21.15.116:0/2809739868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b41000e0 0x7f15b4100560 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f159c009930 tx=0x7f159c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 -- 172.21.15.116:0/2809739868 shutdown_connections 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 --2- 172.21.15.116:0/2809739868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4100aa0 0x7f15b410e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 --2- 172.21.15.116:0/2809739868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b41000e0 0x7f15b4100560 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 --2- 172.21.15.116:0/2809739868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b4107a20 0x7f15b4107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 -- 172.21.15.116:0/2809739868 >> 172.21.15.116:0/2809739868 conn(0x7f15b4075a60 msgr2=0x7f15b4075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 -- 172.21.15.116:0/2809739868 shutdown_connections 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 -- 172.21.15.116:0/2809739868 wait complete. 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.840+0000 7f15b8976640 1 Processor -- start 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 -- start start 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b41000e0 0x7f15b4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4107a20 0x7f15b419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4111b30 con 0x7f15b4100aa0 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41119b0 con 0x7f15b41000e0 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b8976640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4111cb0 con 0x7f15b4107a20 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36362/0 (socket says 172.21.15.116:36362) 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 -- 172.21.15.116:0/989460392 learned_addr learned my addr 172.21.15.116:0/989460392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b259c640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b41000e0 0x7f15b4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4107a20 0x7f15b419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 -- 172.21.15.116:0/989460392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4107a20 msgr2=0x7f15b419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4107a20 0x7f15b419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 -- 172.21.15.116:0/989460392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b41000e0 msgr2=0x7f15b4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b41000e0 0x7f15b4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.841+0000 7f15a9d9b640 1 -- 172.21.15.116:0/989460392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b419d040 con 0x7f15b4100aa0 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f159c009230 tx=0x7f159c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15abfff640 1 -- 172.21.15.116:0/989460392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c043020 con 0x7f15b4100aa0 2024-03-19T02:28:43.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15abfff640 1 -- 172.21.15.116:0/989460392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f159c02ca70 con 0x7f15b4100aa0 2024-03-19T02:28:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b419d2d0 con 0x7f15b4100aa0 2024-03-19T02:28:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15abfff640 1 -- 172.21.15.116:0/989460392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c003a50 con 0x7f15b4100aa0 2024-03-19T02:28:43.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.842+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41a4bc0 con 0x7f15b4100aa0 2024-03-19T02:28:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.844+0000 7f15abfff640 1 -- 172.21.15.116:0/989460392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f159c046020 con 0x7f15b4100aa0 2024-03-19T02:28:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 Processor -- start 2024-03-19T02:28:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 -- start start 2024-03-19T02:28:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15abfff640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 0x7f15a00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1bb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1be320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac116500 con 0x55d5ac1b3350 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac0bebe0 con 0x55d5ac1bbf30 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15bb634740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac1125a0 con 0x55d5ac1b2a30 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 0x7f15a00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1bb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55906/0 (socket says 172.21.15.116:55906) 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 -- 172.21.15.116:0/3361094633 learned_addr learned my addr 172.21.15.116:0/3361094633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15abfff640 1 -- 172.21.15.116:0/989460392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f159c0f1380 con 0x7f15b4100aa0 2024-03-19T02:28:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 -- 172.21.15.116:0/3361094633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 msgr2=0x55d5ac1be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1be320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.845+0000 7f15b259c640 1 -- 172.21.15.116:0/3361094633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 msgr2=0x55d5ac1bb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15b259c640 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1bb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15b259c640 1 -- 172.21.15.116:0/3361094633 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5ac0e0000 con 0x55d5ac1b2a30 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 0x7f15a00a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009fd0 tx=0x7f15ac011be0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15b259c640 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009cf0 tx=0x7f15a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecab13d5f49e02c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15a8d99640 1 -- 172.21.15.116:0/3361094633 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4004220 con 0x55d5ac1b2a30 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15a8d99640 1 -- 172.21.15.116:0/3361094633 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15a40043c0 con 0x55d5ac1b2a30 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.846+0000 7f15a8d99640 1 -- 172.21.15.116:0/3361094633 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4018840 con 0x55d5ac1b2a30 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- 172.21.15.116:0/3361094633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 msgr2=0x55d5ac1b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009cf0 tx=0x7f15a4011810 comp rx=0 tx=0).stop 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- 172.21.15.116:0/3361094633 shutdown_connections 2024-03-19T02:28:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- 172.21.15.116:0/3361094633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b2e10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- 172.21.15.116:0/3361094633 >> 172.21.15.116:0/3361094633 conn(0x55d5ac1ad160 msgr2=0x55d5ac1ad540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- 172.21.15.116:0/3361094633 shutdown_connections 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- 172.21.15.116:0/3361094633 wait complete. 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- start start 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 Processor -- start 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac1bed20 con 0x55d5ac1b3350 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.847+0000 7f15bb634740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac0d2160 con 0x55d5ac1b2a30 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15bb634740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac1beea0 con 0x55d5ac1bbf30 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15bb634740 10 client.0 _ll_register_callbacks cb 0x55d5ac10ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15b259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36378/0 (socket says 172.21.15.116:36378) 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3831762416 learned_addr learned my addr 172.21.15.116:0/3831762416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3831762416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 msgr2=0x55d5ac1b98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1b98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b2a30 msgr2=0x55d5ac1b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3831762416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5abef47d0 con 0x55d5ac1b3350 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f159c0bd5d0 tx=0x7f159c0bd2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9111f0471d2629df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f1599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c09aba0 con 0x55d5ac1b3350 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f159c0b9cb0 con 0x55d5ac1b3350 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a40043c0 con 0x55d5ac1b3350 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f159c0bdd40 con 0x55d5ac1b3350 2024-03-19T02:28:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15bb634740 10 client.15972 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.848+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5abe2a3f0 con 0x55d5ac1b3350 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.851+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15a40043c0 con 0x55d5ac1b3350 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.851+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f159c0053d0 con 0x55d5ac1b3350 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 10 client.15972 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 20 client.15972 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac1cc190 0x55d5ac1ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 4 client.15972 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5ac123a40 con 0x55d5ac1cc190 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15b259c640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac1cc190 0x55d5ac1ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f15b259c640 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac1cc190 0x55d5ac1ce620 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4187875c4b727e68 server_cookie=317c9d689292afc4 in_seq=0 out_seq=0 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.852+0000 7f1599ffb640 10 client.15972 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f1599ffb640 1 -- 172.21.15.116:0/3831762416 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5ac123a40 con 0x55d5ac1cc190 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f1599ffb640 10 client.15972 handle_command_reply: tid=0 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 client.15972 shutdown 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 msgr2=0x55d5ac1b9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f159c0bd5d0 tx=0x7f159c0bd2b0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 shutdown_connections 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac1cc190 0x55d5ac1ce620 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b9390 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 --2- 172.21.15.116:0/3831762416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f1599ffb640 0 client.15972 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 >> 172.21.15.116:0/3831762416 conn(0x55d5ac1ad160 msgr2=0x55d5ac1aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.854+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 shutdown_connections 2024-03-19T02:28:43.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.855+0000 7f15bb634740 1 -- 172.21.15.116:0/3831762416 wait complete. 2024-03-19T02:28:43.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.855+0000 7f15bb634740 20 client.15972 trim_cache size 0 max 16384 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.874+0000 7f15bb634740 1 Processor -- start 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.874+0000 7f15bb634740 1 -- start start 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.874+0000 7f15bb634740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.874+0000 7f15bb634740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15bb634740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1d3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15bb634740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41aebd0 con 0x55d5ac1b3350 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15bb634740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b0740 con 0x55d5ac1bbf30 2024-03-19T02:28:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15bb634740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41af2a0 con 0x55d5ac1b2a30 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36392/0 (socket says 172.21.15.116:36392) 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3446368514 learned_addr learned my addr 172.21.15.116:0/3446368514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15b259c640 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1d3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3446368514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 msgr2=0x55d5ac1b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3446368514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 msgr2=0x55d5ac1d3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1d3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 -- 172.21.15.116:0/3446368514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5abdea0f0 con 0x55d5ac1b3350 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f159c000c00 tx=0x7f159c0bbb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5f2b4550a8cb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f159bfff640 1 -- 172.21.15.116:0/3446368514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c0ee620 con 0x55d5ac1b3350 2024-03-19T02:28:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f159bfff640 1 -- 172.21.15.116:0/3446368514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f159c0bd640 con 0x55d5ac1b3350 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.875+0000 7f159bfff640 1 -- 172.21.15.116:0/3446368514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c0c5950 con 0x55d5ac1b3350 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- 172.21.15.116:0/3446368514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 msgr2=0x55d5ac1c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f159c000c00 tx=0x7f159c0bbb50 comp rx=0 tx=0).stop 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- 172.21.15.116:0/3446368514 shutdown_connections 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1bbf30 0x55d5ac1d3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b3350 0x55d5ac1c7ac0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 --2- 172.21.15.116:0/3446368514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- 172.21.15.116:0/3446368514 >> 172.21.15.116:0/3446368514 conn(0x55d5ac1ad160 msgr2=0x55d5ac1ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- 172.21.15.116:0/3446368514 shutdown_connections 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- 172.21.15.116:0/3446368514 wait complete. 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 -- start start 2024-03-19T02:28:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.876+0000 7f15bb634740 1 Processor -- start 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac231510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41b0040 con 0x55d5ac1b2a30 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41af970 con 0x55d5ac1b3350 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f15b41acbb0 con 0x55d5ac1bbf30 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 10 client.0 _ll_register_callbacks cb 0x7f159c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36394/0 (socket says 172.21.15.116:36394) 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 -- 172.21.15.116:0/861928208 learned_addr learned my addr 172.21.15.116:0/861928208 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b2d9d640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac231510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 -- 172.21.15.116:0/861928208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 msgr2=0x55d5ac231510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac231510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 -- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b3350 msgr2=0x55d5ac1b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 -- 172.21.15.116:0/861928208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5ac231a50 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15b259c640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009cf0 tx=0x7f15a4009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5505796860e1c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f1599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4005030 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f15a40051d0 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f159c0bd640 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f15a40053d0 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.877+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5ac232c60 con 0x55d5ac1b2a30 2024-03-19T02:28:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f159c0bd640 con 0x55d5ac1b2a30 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f15a4026800 con 0x55d5ac1b2a30 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 20 client.15978 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac233270 0x55d5ac235690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 4 client.15978 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d5ac11dd90 con 0x55d5ac233270 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac233270 0x55d5ac235690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f15a9d9b640 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac233270 0x55d5ac235690 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e9a1f4d969412b8 server_cookie=283349785d6c5138 in_seq=0 out_seq=0 2024-03-19T02:28:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.878+0000 7f1599ffb640 10 client.15978 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f1599ffb640 1 -- 172.21.15.116:0/861928208 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d5ac11dd90 con 0x55d5ac233270 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f1599ffb640 10 client.15978 handle_command_reply: tid=0 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 client.15978 shutdown 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 msgr2=0x55d5ac1b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009cf0 tx=0x7f15a4009330 comp rx=0 tx=0).stop 2024-03-19T02:28:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 shutdown_connections 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d5ac233270 0x55d5ac235690 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5ac1bbf30 0x55d5ac231510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f1599ffb640 0 client.15978 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5ac1b3350 0x55d5ac1b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 --2- 172.21.15.116:0/861928208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5ac1b2a30 0x55d5ac1b5ff0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 >> 172.21.15.116:0/861928208 conn(0x55d5ac1ad160 msgr2=0x55d5ac1b9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 shutdown_connections 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 1 -- 172.21.15.116:0/861928208 wait complete. 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.880+0000 7f15bb634740 20 client.15978 trim_cache size 0 max 16384 2024-03-19T02:28:43.888 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:28:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:28:43.889 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 msgr2=0x7f15a00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 0x7f15a00a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009fd0 tx=0x7f15ac011be0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 msgr2=0x7f15b4198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f159c009230 tx=0x7f159c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 shutdown_connections 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f15a00a2a90 0x7f15a00a4f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15b4107a20 0x7f15b419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b4100aa0 0x7f15b4198590 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 --2- 172.21.15.116:0/989460392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15b41000e0 0x7f15b4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 >> 172.21.15.116:0/989460392 conn(0x7f15b4075a60 msgr2=0x7f15b40fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 shutdown_connections 2024-03-19T02:28:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:43.881+0000 7f15b8976640 1 -- 172.21.15.116:0/989460392 wait complete. 2024-03-19T02:28:43.893 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:28:43.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:28:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3580363374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 msgr2=0x7f9ae8100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3580363374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8100b30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009890 tx=0x7f9ad802bb70 comp rx=0 tx=0).stop 2024-03-19T02:28:44.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3580363374 shutdown_connections 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3580363374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae8101070 0x7f9ae810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3580363374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8100b30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3580363374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae81066c0 0x7f9ae8106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3580363374 >> 172.21.15.116:0/3580363374 conn(0x7f9ae80777f0 msgr2=0x7f9ae8077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3580363374 shutdown_connections 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.003+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3580363374 wait complete. 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 Processor -- start 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 -- start start 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae81066c0 0x7f9ae8103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8110940 con 0x7f9ae8101070 2024-03-19T02:28:44.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae81107c0 con 0x7f9ae81066c0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aefbe1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae8110ac0 con 0x7f9ae81006b0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36418/0 (socket says 172.21.15.116:36418) 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 -- 172.21.15.116:0/3740323119 learned_addr learned my addr 172.21.15.116:0/3740323119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed97d640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aee17e640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae81066c0 0x7f9ae8103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 -- 172.21.15.116:0/3740323119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 msgr2=0x7f9ae8102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 -- 172.21.15.116:0/3740323119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae81066c0 msgr2=0x7f9ae8103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae81066c0 0x7f9ae8103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.004+0000 7f9aed17c640 1 -- 172.21.15.116:0/3740323119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae81a6510 con 0x7f9ae8101070 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9aed17c640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8000c00 tx=0x7f9ad8035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8041020 con 0x7f9ae8101070 2024-03-19T02:28:44.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae81a67a0 con 0x7f9ae8101070 2024-03-19T02:28:44.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ad802c9c0 con 0x7f9ae8101070 2024-03-19T02:28:44.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8039040 con 0x7f9ae8101070 2024-03-19T02:28:44.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.005+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae81a6ce0 con 0x7f9ae8101070 2024-03-19T02:28:44.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.006+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae8101df0 con 0x7f9ae8101070 2024-03-19T02:28:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:43 smithi116 ceph-mon[32263]: pgmap v30: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.8 MiB/s wr, 28 op/s 2024-03-19T02:28:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.008+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f9ad8044020 con 0x7f9ae8101070 2024-03-19T02:28:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.008+0000 7f9adeffd640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 0x7f9ad40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.008+0000 7f9aed97d640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 0x7f9ad40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.009+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9ad80f0f70 con 0x7f9ae8101070 2024-03-19T02:28:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.009+0000 7f9aed97d640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 0x7f9ad40a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40096c0 tx=0x7f9ae4009260 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.013+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ad804b020 con 0x7f9ae8101070 2024-03-19T02:28:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:43 smithi162 ceph-mon[35761]: pgmap v30: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.8 MiB/s wr, 28 op/s 2024-03-19T02:28:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:43 smithi154 ceph-mon[35693]: pgmap v30: 129 pgs: 129 active+clean; 22 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.8 MiB/s wr, 28 op/s 2024-03-19T02:28:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.264+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ae8109c50 con 0x7f9ae8101070 2024-03-19T02:28:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.264+0000 7f9adeffd640 1 -- 172.21.15.116:0/3740323119 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11951 (secure 0 0 0) 0x7f9ad80bd810 con 0x7f9ae8101070 2024-03-19T02:28:44.265 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:28:44.265 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:28:39.411853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:28:44.266 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-19T02:28:44.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.266+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 msgr2=0x7f9ad40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.266+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 0x7f9ad40a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40096c0 tx=0x7f9ae4009260 comp rx=0 tx=0).stop 2024-03-19T02:28:44.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.266+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 msgr2=0x7f9ae8103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.266+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8000c00 tx=0x7f9ad8035990 comp rx=0 tx=0).stop 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 shutdown_connections 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ad40a2a90 0x7f9ad40a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ae81066c0 0x7f9ae8103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ae8101070 0x7f9ae8103210 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 --2- 172.21.15.116:0/3740323119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ae81006b0 0x7f9ae8102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 >> 172.21.15.116:0/3740323119 conn(0x7f9ae80777f0 msgr2=0x7f9ae806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 shutdown_connections 2024-03-19T02:28:44.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.267+0000 7f9aefbe1640 1 -- 172.21.15.116:0/3740323119 wait complete. 2024-03-19T02:28:44.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:28:44.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.393+0000 7f36cf169640 1 -- 172.21.15.116:0/709945029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c8101060 msgr2=0x7f36c810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.393+0000 7f36cf169640 1 --2- 172.21.15.116:0/709945029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c8101060 0x7f36c810d570 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009cd0 tx=0x7f36c002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- 172.21.15.116:0/709945029 shutdown_connections 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- 172.21.15.116:0/709945029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c8101060 0x7f36c810d570 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- 172.21.15.116:0/709945029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 0x7f36c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- 172.21.15.116:0/709945029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c81066b0 0x7f36c8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- 172.21.15.116:0/709945029 >> 172.21.15.116:0/709945029 conn(0x7f36c80778d0 msgr2=0x7f36c8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- 172.21.15.116:0/709945029 shutdown_connections 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- 172.21.15.116:0/709945029 wait complete. 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 Processor -- start 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- start start 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 0x7f36c8196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c8101060 0x7f36c8197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36c8110970 con 0x7f36c81066b0 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81107f0 con 0x7f36c8101060 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.394+0000 7f36cf169640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c8110af0 con 0x7f36c81006a0 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c8101060 0x7f36c8197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 0x7f36c8196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36436/0 (socket says 172.21.15.116:36436) 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 -- 172.21.15.116:0/1192794071 learned_addr learned my addr 172.21.15.116:0/1192794071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 -- 172.21.15.116:0/1192794071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 msgr2=0x7f36c8196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 0x7f36c8196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 -- 172.21.15.116:0/1192794071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c8101060 msgr2=0x7f36c8197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c8101060 0x7f36c8197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 -- 172.21.15.116:0/1192794071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c819be20 con 0x7f36c81066b0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cd706640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f36c0004af0 tx=0x7f36c0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36c5ffb640 1 -- 172.21.15.116:0/1192794071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0039040 con 0x7f36c81066b0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36c5ffb640 1 -- 172.21.15.116:0/1192794071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36c0044070 con 0x7f36c81066b0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c819c0b0 con 0x7f36c81066b0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36c5ffb640 1 -- 172.21.15.116:0/1192794071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c00042a0 con 0x7f36c81066b0 2024-03-19T02:28:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.395+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c81a39a0 con 0x7f36c81066b0 2024-03-19T02:28:44.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c5ffb640 1 -- 172.21.15.116:0/1192794071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f36c0004440 con 0x7f36c81066b0 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 Processor -- start 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 -- start start 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 0x56156f9774a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f979dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56156f8cf4b0 con 0x56156f9779e0 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56156f877d70 con 0x56156f96bf90 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36d1e32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56156f8cb550 con 0x56156f96c8b0 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c5ffb640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 0x7f36b80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 0x56156f9774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53024/0 (socket says 172.21.15.116:53024) 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c7fff640 1 -- 172.21.15.116:0/1695762626 learned_addr learned my addr 172.21.15.116:0/1695762626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36c7fff640 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f979dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.398+0000 7f36cd706640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 0x7f36b80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 -- 172.21.15.116:0/1695762626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 msgr2=0x56156f96c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 -- 172.21.15.116:0/1695762626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 msgr2=0x56156f979dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f979dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 -- 172.21.15.116:0/1695762626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56156f89a210 con 0x56156f96c8b0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36c7fff640 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36c5ffb640 1 -- 172.21.15.116:0/1192794071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f36c00f18a0 con 0x7f36c81066b0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 0x56156f9774a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009c20 tx=0x7f36bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2e3659d72d3787 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36cd706640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 0x7f36b80a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f36c00eec60 tx=0x7f36c00e8490 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36b2ffd640 1 -- 172.21.15.116:0/1695762626 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc004050 con 0x56156f96c8b0 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36b2ffd640 1 -- 172.21.15.116:0/1695762626 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36bc0054d0 con 0x56156f96c8b0 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.399+0000 7f36b2ffd640 1 -- 172.21.15.116:0/1695762626 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc004bc0 con 0x56156f96c8b0 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- 172.21.15.116:0/1695762626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 msgr2=0x56156f9774a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 0x56156f9774a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009c20 tx=0x7f36bc011810 comp rx=0 tx=0).stop 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- 172.21.15.116:0/1695762626 shutdown_connections 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f979dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96c8b0 0x56156f9774a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1695762626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f96c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- 172.21.15.116:0/1695762626 >> 172.21.15.116:0/1695762626 conn(0x56156f966a00 msgr2=0x56156f966de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- 172.21.15.116:0/1695762626 shutdown_connections 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- 172.21.15.116:0/1695762626 wait complete. 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 -- start start 2024-03-19T02:28:44.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.400+0000 7f36d1e32740 1 Processor -- start 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f974f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f975450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56156f97a7d0 con 0x56156f96bf90 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56156f88b310 con 0x56156f96c8b0 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56156f97a950 con 0x56156f9779e0 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36d1e32740 10 client.0 _ll_register_callbacks cb 0x56156f87f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36460/0 (socket says 172.21.15.116:36460) 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f974f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 -- 172.21.15.116:0/3975837702 learned_addr learned my addr 172.21.15.116:0/3975837702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f975450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 -- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 msgr2=0x56156f975450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f975450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 -- 172.21.15.116:0/3975837702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 msgr2=0x56156f974f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f974f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 -- 172.21.15.116:0/3975837702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56156f5e33f0 con 0x56156f96bf90 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36ccf05640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f974f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36c7fff640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f36b4013c40 tx=0x7f36b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8223b6d364e32306 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b400ddd0 con 0x56156f96bf90 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.401+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f36b4013040 con 0x56156f96bf90 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36bc0054d0 con 0x56156f96bf90 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f36b4013240 con 0x56156f96bf90 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56156f6ad7d0 con 0x56156f96bf90 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36bc0054d0 con 0x56156f96bf90 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f36b40053c0 con 0x56156f96bf90 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 10 client.15996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 20 client.15996 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f987d30 0x56156f98a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 4 client.15996 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56156f92d3e0 con 0x56156f987d30 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.402+0000 7f36ccf05640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f987d30 0x56156f98a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.403+0000 7f36a7fff640 10 client.15996 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.403+0000 7f36ccf05640 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f987d30 0x56156f98a140 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3bcc9b69fcaceb47 server_cookie=74e29af23ecca1bd in_seq=0 out_seq=0 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.403+0000 7f36a7fff640 1 -- 172.21.15.116:0/3975837702 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56156f92d3e0 con 0x56156f987d30 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.403+0000 7f36a7fff640 10 client.15996 handle_command_reply: tid=0 2024-03-19T02:28:44.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 client.15996 shutdown 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 msgr2=0x56156f9749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f36b4013c40 tx=0x7f36b4004730 comp rx=0 tx=0).stop 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 shutdown_connections 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f987d30 0x56156f98a140 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f975450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f974f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 --2- 172.21.15.116:0/3975837702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96bf90 0x56156f9749d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 >> 172.21.15.116:0/3975837702 conn(0x56156f966a00 msgr2=0x56156f968750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36a7fff640 0 client.15996 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 shutdown_connections 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 1 -- 172.21.15.116:0/3975837702 wait complete. 2024-03-19T02:28:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.404+0000 7f36d1e32740 20 client.15996 trim_cache size 0 max 16384 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 Processor -- start 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 -- start start 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f977100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f9835e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81ac940 con 0x56156f96c8b0 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81ae4b0 con 0x56156f96bf90 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36d1e32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81ad010 con 0x56156f9779e0 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36468/0 (socket says 172.21.15.116:36468) 2024-03-19T02:28:44.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.419+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f9835e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 -- 172.21.15.116:0/1032490101 learned_addr learned my addr 172.21.15.116:0/1032490101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36c7fff640 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f977100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 -- 172.21.15.116:0/1032490101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 msgr2=0x56156f9835e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f9835e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 -- 172.21.15.116:0/1032490101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 msgr2=0x56156f977100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f977100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 -- 172.21.15.116:0/1032490101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56156f96f7b0 con 0x56156f96c8b0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36ccf05640 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009e90 tx=0x7f36bc009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cc7f5d9a4d5b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36b27fc640 1 -- 172.21.15.116:0/1032490101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc004360 con 0x56156f96c8b0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36b27fc640 1 -- 172.21.15.116:0/1032490101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36bc02e080 con 0x56156f96c8b0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.420+0000 7f36b27fc640 1 -- 172.21.15.116:0/1032490101 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc003850 con 0x56156f96c8b0 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- 172.21.15.116:0/1032490101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 msgr2=0x56156f9742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009e90 tx=0x7f36bc009680 comp rx=0 tx=0).stop 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- 172.21.15.116:0/1032490101 shutdown_connections 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f9779e0 0x56156f9835e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f96c8b0 0x56156f9742b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 --2- 172.21.15.116:0/1032490101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96bf90 0x56156f977100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- 172.21.15.116:0/1032490101 >> 172.21.15.116:0/1032490101 conn(0x56156f966a00 msgr2=0x56156f965ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- 172.21.15.116:0/1032490101 shutdown_connections 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- 172.21.15.116:0/1032490101 wait complete. 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 -- start start 2024-03-19T02:28:44.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.421+0000 7f36d1e32740 1 Processor -- start 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 0x56156f97abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f9ec030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81addb0 con 0x56156f9779e0 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81ad6e0 con 0x56156f96c8b0 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36c81aa920 con 0x56156f96bf90 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 10 client.0 _ll_register_callbacks cb 0x7f36b4013a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f9ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 0x56156f97abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 0x56156f97abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56018/0 (socket says 172.21.15.116:56018) 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36ccf05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53054/0 (socket says 172.21.15.116:53054) 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 -- 172.21.15.116:0/4119038470 learned_addr learned my addr 172.21.15.116:0/4119038470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 -- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 msgr2=0x56156f97abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 0x56156f97abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 -- 172.21.15.116:0/4119038470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 msgr2=0x56156f97b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 -- 172.21.15.116:0/4119038470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56156f97ba10 con 0x56156f9779e0 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36ccf05640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36c7fff640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f9ec030 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x56156f990260 tx=0x7f36b400bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7d16c472d2e911 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4005850 con 0x56156f9779e0 2024-03-19T02:28:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f36b4012260 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.422+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36bc02e080 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f36b403f050 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56156f9ed6a0 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36bc02e080 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f36b4012720 con 0x56156f9779e0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 10 client.16002 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 20 client.16002 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f9edd00 0x56156f9f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 4 client.16002 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56156f8dc9f0 con 0x56156f9edd00 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.423+0000 7f36ccf05640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f9edd00 0x56156f9f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36ccf05640 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f9edd00 0x56156f9f0120 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cc403909e7c3fce4 server_cookie=11a6a3b4dff14514 in_seq=0 out_seq=0 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36a7fff640 10 client.16002 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36a7fff640 1 -- 172.21.15.116:0/4119038470 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56156f8dc9f0 con 0x56156f9edd00 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36a7fff640 10 client.16002 handle_command_reply: tid=0 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 client.16002 shutdown 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 msgr2=0x56156f9ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f9ec030 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x56156f990260 tx=0x7f36b400bbf0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 shutdown_connections 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56156f9edd00 0x56156f9f0120 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36a7fff640 0 client.16002 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56156f9779e0 0x56156f9ec030 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56156f96c8b0 0x56156f97b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 --2- 172.21.15.116:0/4119038470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56156f96bf90 0x56156f97abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 >> 172.21.15.116:0/4119038470 conn(0x56156f966a00 msgr2=0x56156f965310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 shutdown_connections 2024-03-19T02:28:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 1 -- 172.21.15.116:0/4119038470 wait complete. 2024-03-19T02:28:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.424+0000 7f36d1e32740 20 client.16002 trim_cache size 0 max 16384 2024-03-19T02:28:44.432 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 msgr2=0x7f36b80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 0x7f36b80a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f36c00eec60 tx=0x7f36c00e8490 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 msgr2=0x7f36c819b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f36c0004af0 tx=0x7f36c0035ce0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 shutdown_connections 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36b80a2a90 0x7f36b80a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36c81066b0 0x7f36c819b6f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36c8101060 0x7f36c8197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 --2- 172.21.15.116:0/1192794071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36c81006a0 0x7f36c8196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 >> 172.21.15.116:0/1192794071 conn(0x7f36c80778d0 msgr2=0x7f36c8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 shutdown_connections 2024-03-19T02:28:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.425+0000 7f36cf169640 1 -- 172.21.15.116:0/1192794071 wait complete. 2024-03-19T02:28:44.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:28:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 -- 172.21.15.116:0/2006772960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 msgr2=0x7f790c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 --2- 172.21.15.116:0/2006772960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c10d570 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7904009cd0 tx=0x7f790402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 -- 172.21.15.116:0/2006772960 shutdown_connections 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 --2- 172.21.15.116:0/2006772960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c10d570 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 --2- 172.21.15.116:0/2006772960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1006a0 0x7f790c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 --2- 172.21.15.116:0/2006772960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1066b0 0x7f790c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 -- 172.21.15.116:0/2006772960 >> 172.21.15.116:0/2006772960 conn(0x7f790c0778d0 msgr2=0x7f790c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 -- 172.21.15.116:0/2006772960 shutdown_connections 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 -- 172.21.15.116:0/2006772960 wait complete. 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.549+0000 7f7911d1b640 1 Processor -- start 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 -- start start 2024-03-19T02:28:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1006a0 0x7f790c196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1066b0 0x7f790c19b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f790c110970 con 0x7f790c101060 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1107f0 con 0x7f790c1066b0 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7911d1b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f790c110af0 con 0x7f790c1006a0 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36506/0 (socket says 172.21.15.116:36506) 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1006a0 0x7f790c196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 -- 172.21.15.116:0/1458142892 learned_addr learned my addr 172.21.15.116:0/1458142892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790bfff640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1066b0 0x7f790c19b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 -- 172.21.15.116:0/1458142892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1006a0 msgr2=0x7f790c196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1006a0 0x7f790c196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 -- 172.21.15.116:0/1458142892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1066b0 msgr2=0x7f790c19b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1066b0 0x7f790c19b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 -- 172.21.15.116:0/1458142892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c19be30 con 0x7f790c101060 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f790affd640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f790000e9f0 tx=0x7f790000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.550+0000 7f7908ff9640 1 -- 172.21.15.116:0/1458142892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790000cd80 con 0x7f790c101060 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.551+0000 7f7908ff9640 1 -- 172.21.15.116:0/1458142892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79000108e0 con 0x7f790c101060 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.551+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790c19c120 con 0x7f790c101060 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.551+0000 7f7908ff9640 1 -- 172.21.15.116:0/1458142892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900018920 con 0x7f790c101060 2024-03-19T02:28:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.551+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790c1a39b0 con 0x7f790c101060 2024-03-19T02:28:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f7908ff9640 1 -- 172.21.15.116:0/1458142892 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f7900010a90 con 0x7f790c101060 2024-03-19T02:28:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 Processor -- start 2024-03-19T02:28:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 -- start start 2024-03-19T02:28:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee62478c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 0x562ee6252dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ee61a84b0 con 0x562ee62509e0 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562ee6150d70 con 0x562ee62474e0 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f79149e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ee61a4550 con 0x562ee6247e00 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56042/0 (socket says 172.21.15.116:56042) 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f790b7fe640 1 -- 172.21.15.116:0/3224558431 learned_addr learned my addr 172.21.15.116:0/3224558431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f790bfff640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee62478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.554+0000 7f7908ff9640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 0x7f78f40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790bfff640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 0x7f78f40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 -- 172.21.15.116:0/3224558431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 msgr2=0x562ee62478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee62478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f7908ff9640 1 -- 172.21.15.116:0/1458142892 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f79000cf7c0 con 0x7f790c101060 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 -- 172.21.15.116:0/3224558431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 msgr2=0x562ee6252dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790affd640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 0x562ee6252dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 0x562ee6252dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 -- 172.21.15.116:0/3224558431 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee6173210 con 0x562ee6247e00 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790affd640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 0x562ee6252dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790b7fe640 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009ca0 tx=0x7f78fc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15f570ddf1b49259 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f790bfff640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 0x7f78f40a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7904005cc0 tx=0x7f790402c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f78f1ffb640 1 -- 172.21.15.116:0/3224558431 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc01e070 con 0x562ee6247e00 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f78f1ffb640 1 -- 172.21.15.116:0/3224558431 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78fc019ae0 con 0x562ee6247e00 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.555+0000 7f78f1ffb640 1 -- 172.21.15.116:0/3224558431 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00e990 con 0x562ee6247e00 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 -- 172.21.15.116:0/3224558431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 msgr2=0x562ee62504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009ca0 tx=0x7f78fc011810 comp rx=0 tx=0).stop 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 -- 172.21.15.116:0/3224558431 shutdown_connections 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62509e0 0x562ee6252dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee6247e00 0x562ee62504a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 --2- 172.21.15.116:0/3224558431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee62478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 -- 172.21.15.116:0/3224558431 >> 172.21.15.116:0/3224558431 conn(0x562ee6241c10 msgr2=0x562ee6241ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 -- 172.21.15.116:0/3224558431 shutdown_connections 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.556+0000 7f79149e1740 1 -- 172.21.15.116:0/3224558431 wait complete. 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 -- start start 2024-03-19T02:28:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 Processor -- start 2024-03-19T02:28:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee624df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee624e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ee62537d0 con 0x562ee62474e0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562ee6164310 con 0x562ee6247e00 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ee6253950 con 0x562ee62509e0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f79149e1740 10 client.0 _ll_register_callbacks cb 0x562ee61582d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f790bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee624e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36514/0 (socket says 172.21.15.116:36514) 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f790b7fe640 1 -- 172.21.15.116:0/1366685655 learned_addr learned my addr 172.21.15.116:0/1366685655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.557+0000 7f790affd640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee624df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 -- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 msgr2=0x562ee624e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee624e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 -- 172.21.15.116:0/1366685655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 msgr2=0x562ee624df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee624df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 -- 172.21.15.116:0/1366685655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee5ebc3f0 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790bfff640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee624e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790affd640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee624df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f78fc005890 tx=0x7f78fc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e384890899260e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f78eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc027020 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f78fc005250 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78fc019ae0 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78fc032050 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f79149e1740 10 client.16011 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.558+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ee5f867d0 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78fc019ae0 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f78fc00e6d0 con 0x562ee62474e0 2024-03-19T02:28:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 10 client.16011 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 10 client.16011 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 20 client.16011 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee6260c70 0x562ee6263060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 4 client.16011 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ee62063e0 con 0x562ee6260c70 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f790affd640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee6260c70 0x562ee6263060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f78eaffd640 10 client.16011 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.559+0000 7f790affd640 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee6260c70 0x562ee6263060 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b71bdb94af6d8116 server_cookie=ccaa9c02f72a044b in_seq=0 out_seq=0 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.560+0000 7f78eaffd640 1 -- 172.21.15.116:0/1366685655 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ee62063e0 con 0x562ee6260c70 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.560+0000 7f78eaffd640 10 client.16011 handle_command_reply: tid=0 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.560+0000 7f79149e1740 1 client.16011 shutdown 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 msgr2=0x562ee624da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f78fc005890 tx=0x7f78fc019660 comp rx=0 tx=0).stop 2024-03-19T02:28:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 shutdown_connections 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee624e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee624df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee6260c70 0x562ee6263060 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 --2- 172.21.15.116:0/1366685655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee624da10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f78eaffd640 0 client.16011 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 >> 172.21.15.116:0/1366685655 conn(0x562ee6241c10 msgr2=0x562ee62430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 shutdown_connections 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 1 -- 172.21.15.116:0/1366685655 wait complete. 2024-03-19T02:28:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.561+0000 7f79149e1740 20 client.16011 trim_cache size 0 max 16384 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 Processor -- start 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 -- start start 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee6250140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee625c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1ac970 con 0x562ee6247e00 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1ae4e0 con 0x562ee62474e0 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f79149e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1ad040 con 0x562ee62509e0 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36526/0 (socket says 172.21.15.116:36526) 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 -- 172.21.15.116:0/278210389 learned_addr learned my addr 172.21.15.116:0/278210389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790b7fe640 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee6250140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790bfff640 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee625c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 -- 172.21.15.116:0/278210389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 msgr2=0x562ee625c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee625c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 -- 172.21.15.116:0/278210389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 msgr2=0x562ee6250140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee6250140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.576+0000 7f790affd640 1 -- 172.21.15.116:0/278210389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee6247c30 con 0x562ee6247e00 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f790affd640 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7900097d60 tx=0x7f7900009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba87a0765253d247 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f78f17fa640 1 -- 172.21.15.116:0/278210389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900099c00 con 0x562ee6247e00 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f78f17fa640 1 -- 172.21.15.116:0/278210389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f790009bb60 con 0x562ee6247e00 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f78f17fa640 1 -- 172.21.15.116:0/278210389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790009cb80 con 0x562ee6247e00 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f79149e1740 1 -- 172.21.15.116:0/278210389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 msgr2=0x562ee6248440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.577+0000 7f79149e1740 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7900097d60 tx=0x7f7900009310 comp rx=0 tx=0).stop 2024-03-19T02:28:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 -- 172.21.15.116:0/278210389 shutdown_connections 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee625c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee6247e00 0x562ee6248440 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 --2- 172.21.15.116:0/278210389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee62474e0 0x562ee6250140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 -- 172.21.15.116:0/278210389 >> 172.21.15.116:0/278210389 conn(0x562ee6241c10 msgr2=0x562ee6241ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 -- 172.21.15.116:0/278210389 shutdown_connections 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 -- 172.21.15.116:0/278210389 wait complete. 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 -- start start 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 Processor -- start 2024-03-19T02:28:44.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.578+0000 7f79149e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee6259000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee6254170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1adde0 con 0x562ee62474e0 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1ad710 con 0x562ee6247e00 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f790c1aa950 con 0x562ee62509e0 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 10 client.0 _ll_register_callbacks cb 0x7f78fc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee6254170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36530/0 (socket says 172.21.15.116:36530) 2024-03-19T02:28:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 -- 172.21.15.116:0/1708283181 learned_addr learned my addr 172.21.15.116:0/1708283181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790affd640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee6259000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 -- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 msgr2=0x562ee6254170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee6254170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 -- 172.21.15.116:0/1708283181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 msgr2=0x562ee6259000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee6259000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 -- 172.21.15.116:0/1708283181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee6254970 con 0x562ee62474e0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f790b7fe640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00ea70 tx=0x7f78fc0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df203597fabfbb4c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f78eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc027040 con 0x562ee62474e0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f78fc019a60 con 0x562ee62474e0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.579+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f790009bb60 con 0x562ee62474e0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78fc01e070 con 0x562ee62474e0 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 10 client.16017 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ee62c6b00 con 0x562ee62474e0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f790009bb60 con 0x562ee62474e0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f78fc019c60 con 0x562ee62474e0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 10 client.16017 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 10 client.16017 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 20 client.16017 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee62c72d0 0x562ee62c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 4 client.16017 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ee61b59f0 con 0x562ee62c72d0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.580+0000 7f790affd640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee62c72d0 0x562ee62c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f790affd640 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee62c72d0 0x562ee62c96f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca6aed0cba394b3b server_cookie=81d148161e646910 in_seq=0 out_seq=0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f78eaffd640 10 client.16017 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f78eaffd640 1 -- 172.21.15.116:0/1708283181 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ee61b59f0 con 0x562ee62c72d0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f78eaffd640 10 client.16017 handle_command_reply: tid=0 2024-03-19T02:28:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 client.16017 shutdown 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 msgr2=0x562ee6258ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00ea70 tx=0x7f78fc0099b0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 shutdown_connections 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ee62509e0 0x562ee6254170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ee6247e00 0x562ee6259000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562ee62c72d0 0x562ee62c96f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 --2- 172.21.15.116:0/1708283181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ee62474e0 0x562ee6258ac0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f78eaffd640 0 client.16017 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 >> 172.21.15.116:0/1708283181 conn(0x562ee6241c10 msgr2=0x562ee623e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 shutdown_connections 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 1 -- 172.21.15.116:0/1708283181 wait complete. 2024-03-19T02:28:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.581+0000 7f79149e1740 20 client.16017 trim_cache size 0 max 16384 2024-03-19T02:28:44.590 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 msgr2=0x7f78f40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 0x7f78f40a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7904005cc0 tx=0x7f790402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 msgr2=0x7f790c1973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f790000e9f0 tx=0x7f790000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 shutdown_connections 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78f40a2a90 0x7f78f40a4f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790c1066b0 0x7f790c19b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f790c101060 0x7f790c1973d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 --2- 172.21.15.116:0/1458142892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f790c1006a0 0x7f790c196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 >> 172.21.15.116:0/1458142892 conn(0x7f790c0778d0 msgr2=0x7f790c102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 shutdown_connections 2024-03-19T02:28:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.582+0000 7f7911d1b640 1 -- 172.21.15.116:0/1458142892 wait complete. 2024-03-19T02:28:44.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:28:44.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 -- 172.21.15.116:0/3732493711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32801066b0 msgr2=0x7f3280106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 --2- 172.21.15.116:0/3732493711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32801066b0 0x7f3280106a90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f327c009930 tx=0x7f327c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:44.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 -- 172.21.15.116:0/3732493711 shutdown_connections 2024-03-19T02:28:44.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 --2- 172.21.15.116:0/3732493711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3280101060 0x7f328010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 --2- 172.21.15.116:0/3732493711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 0x7f3280100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 --2- 172.21.15.116:0/3732493711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32801066b0 0x7f3280106a90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 -- 172.21.15.116:0/3732493711 >> 172.21.15.116:0/3732493711 conn(0x7f32800778d0 msgr2=0x7f3280077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 -- 172.21.15.116:0/3732493711 shutdown_connections 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.708+0000 7f32885cd640 1 -- 172.21.15.116:0/3732493711 wait complete. 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 Processor -- start 2024-03-19T02:28:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 -- start start 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 0x7f3280198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32801066b0 0x7f32801a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3280110970 con 0x7f3280101060 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32801107f0 con 0x7f32801006a0 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f32885cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3280110af0 con 0x7f32801066b0 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f3285b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f3285b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36560/0 (socket says 172.21.15.116:36560) 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f3285b68640 1 -- 172.21.15.116:0/1625667468 learned_addr learned my addr 172.21.15.116:0/1625667468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32801066b0 0x7f32801a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.709+0000 7f3286369640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 0x7f3280198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 -- 172.21.15.116:0/1625667468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32801066b0 msgr2=0x7f32801a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32801066b0 0x7f32801a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 -- 172.21.15.116:0/1625667468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 msgr2=0x7f3280198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 0x7f3280198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 -- 172.21.15.116:0/1625667468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3280196be0 con 0x7f3280101060 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f3285b68640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f327000e9f0 tx=0x7f327000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f32777fe640 1 -- 172.21.15.116:0/1625667468 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327000cd80 con 0x7f3280101060 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f32777fe640 1 -- 172.21.15.116:0/1625667468 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32700108e0 con 0x7f3280101060 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f32777fe640 1 -- 172.21.15.116:0/1625667468 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270018920 con 0x7f3280101060 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3280196ed0 con 0x7f3280101060 2024-03-19T02:28:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.710+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32801973d0 con 0x7f3280101060 2024-03-19T02:28:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.712+0000 7f32777fe640 1 -- 172.21.15.116:0/1625667468 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f3270010a90 con 0x7f3280101060 2024-03-19T02:28:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 Processor -- start 2024-03-19T02:28:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 -- start start 2024-03-19T02:28:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f32777fe640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 0x7f326c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 0x56479a78c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56479a6e64b0 con 0x56479a7832d0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56479a68ed70 con 0x56479a78c7d0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f328b1bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56479a6e2550 con 0x56479a783bf0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.713+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 0x7f326c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f32777fe640 1 -- 172.21.15.116:0/1625667468 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f32700cf7a0 con 0x7f3280101060 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3285b68640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 0x56479a78c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286b6a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36562/0 (socket says 172.21.15.116:36562) 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 -- 172.21.15.116:0/814832163 learned_addr learned my addr 172.21.15.116:0/814832163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 -- 172.21.15.116:0/814832163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 msgr2=0x56479a78c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 0x56479a78c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 -- 172.21.15.116:0/814832163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 msgr2=0x56479a78ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 -- 172.21.15.116:0/814832163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479a6b1210 con 0x56479a7832d0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3285b68640 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 0x56479a78c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 0x7f326c0a4f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3278009fd0 tx=0x7f3278009d80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3286369640 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f327c036040 tx=0x7f327c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d32562692a4f701 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3274bfe640 1 -- 172.21.15.116:0/814832163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c045020 con 0x56479a7832d0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3274bfe640 1 -- 172.21.15.116:0/814832163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f327c02c7e0 con 0x56479a7832d0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.714+0000 7f3274bfe640 1 -- 172.21.15.116:0/814832163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c005420 con 0x56479a7832d0 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- 172.21.15.116:0/814832163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 msgr2=0x56479a7836b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f327c036040 tx=0x7f327c009560 comp rx=0 tx=0).stop 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- 172.21.15.116:0/814832163 shutdown_connections 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a783bf0 0x56479a78c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 --2- 172.21.15.116:0/814832163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a7836b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- 172.21.15.116:0/814832163 >> 172.21.15.116:0/814832163 conn(0x56479a77da00 msgr2=0x56479a77dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- 172.21.15.116:0/814832163 shutdown_connections 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- 172.21.15.116:0/814832163 wait complete. 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.715+0000 7f328b1bc740 1 -- start start 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 Processor -- start 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 0x56479a7897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a789cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56479a78f560 con 0x56479a783bf0 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56479a6a2310 con 0x56479a78c7d0 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56479a78f6e0 con 0x56479a7832d0 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f328b1bc740 10 client.0 _ll_register_callbacks cb 0x56479a6962d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286369640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 0x56479a7897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3285b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a789cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53154/0 (socket says 172.21.15.116:53154) 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 -- 172.21.15.116:0/1666392760 learned_addr learned my addr 172.21.15.116:0/1666392760 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286369640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 0x56479a7897b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56094/0 (socket says 172.21.15.116:56094) 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 -- 172.21.15.116:0/1666392760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 msgr2=0x56479a7897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 0x56479a7897b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 -- 172.21.15.116:0/1666392760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 msgr2=0x56479a789cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a789cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.716+0000 7f3286b6a640 1 -- 172.21.15.116:0/1666392760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479a3fa3f0 con 0x56479a78c7d0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3286b6a640 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f327800fe10 tx=0x7f327801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb2df97da1f8659d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278013330 con 0x56479a78c7d0 2024-03-19T02:28:44.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f327c02c7e0 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3278003e00 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3278032050 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 10 client.25859 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56479a4c47d0 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f327c02c7e0 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f3278020050 con 0x56479a78c7d0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 10 client.25859 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 10 client.25859 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 20 client.25859 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a79eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 4 client.25859 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.717+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56479a7443e0 con 0x56479a79ca90 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.718+0000 7f3286369640 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a79eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.718+0000 7f3269ffb640 10 client.25859 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.718+0000 7f3286369640 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a79eee0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a899a756dd0e5ca2 server_cookie=3d3884e31e57965c in_seq=0 out_seq=0 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f3269ffb640 1 -- 172.21.15.116:0/1666392760 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56479a7443e0 con 0x56479a79ca90 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f3269ffb640 10 client.25859 handle_command_reply: tid=0 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 client.25859 shutdown 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 msgr2=0x56479a78a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f327800fe10 tx=0x7f327801d000 comp rx=0 tx=0).stop 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 shutdown_connections 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a79eee0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a78c7d0 0x56479a78a230 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f3269ffb640 0 client.25859 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a789cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 --2- 172.21.15.116:0/1666392760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a7832d0 0x56479a7897b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 >> 172.21.15.116:0/1666392760 conn(0x56479a77da00 msgr2=0x56479a77f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 shutdown_connections 2024-03-19T02:28:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 1 -- 172.21.15.116:0/1666392760 wait complete. 2024-03-19T02:28:44.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.719+0000 7f328b1bc740 20 client.25859 trim_cache size 0 max 16384 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 Processor -- start 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 -- start start 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a7832d0 0x56479a78bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a798300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b0cd0 con 0x56479a783bf0 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b2840 con 0x56479a7832d0 2024-03-19T02:28:44.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.742+0000 7f328b1bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b13a0 con 0x56479a78c7d0 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56104/0 (socket says 172.21.15.116:56104) 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 -- 172.21.15.116:0/2124271177 learned_addr learned my addr 172.21.15.116:0/2124271177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286369640 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a7832d0 0x56479a78bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 -- 172.21.15.116:0/2124271177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a7832d0 msgr2=0x56479a78bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a7832d0 0x56479a78bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 -- 172.21.15.116:0/2124271177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 msgr2=0x56479a798300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a798300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.743+0000 7f3286b6a640 1 -- 172.21.15.116:0/2124271177 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479a788760 con 0x56479a78c7d0 2024-03-19T02:28:44.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.744+0000 7f3286b6a640 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f32780096b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c6bb6385562fb3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.744+0000 7f326bfff640 1 -- 172.21.15.116:0/2124271177 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278003e00 con 0x56479a78c7d0 2024-03-19T02:28:44.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.744+0000 7f326bfff640 1 -- 172.21.15.116:0/2124271177 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3278013330 con 0x56479a78c7d0 2024-03-19T02:28:44.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.744+0000 7f326bfff640 1 -- 172.21.15.116:0/2124271177 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278005570 con 0x56479a78c7d0 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 -- 172.21.15.116:0/2124271177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 msgr2=0x56479a77cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f32780096b0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 -- 172.21.15.116:0/2124271177 shutdown_connections 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a77cba0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a783bf0 0x56479a798300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2124271177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a7832d0 0x56479a78bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 -- 172.21.15.116:0/2124271177 >> 172.21.15.116:0/2124271177 conn(0x56479a77da00 msgr2=0x56479a77dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.745+0000 7f328b1bc740 1 -- 172.21.15.116:0/2124271177 shutdown_connections 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 -- 172.21.15.116:0/2124271177 wait complete. 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 -- start start 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 Processor -- start 2024-03-19T02:28:44.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a80a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b2140 con 0x56479a7832d0 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b1a70 con 0x56479a783bf0 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32801aecb0 con 0x56479a78c7d0 2024-03-19T02:28:44.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.746+0000 7f328b1bc740 10 client.0 _ll_register_callbacks cb 0x7f3278013690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3286369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3286369640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36588/0 (socket says 172.21.15.116:36588) 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3286369640 1 -- 172.21.15.116:0/2202987138 learned_addr learned my addr 172.21.15.116:0/2202987138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53164/0 (socket says 172.21.15.116:53164) 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3286b6a640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a80a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 -- 172.21.15.116:0/2202987138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 msgr2=0x56479a80a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a80a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 -- 172.21.15.116:0/2202987138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 msgr2=0x56479a790560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3285b68640 1 -- 172.21.15.116:0/2202987138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479a791340 con 0x56479a783bf0 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.747+0000 7f3286369640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.748+0000 7f3285b68640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32700cc9f0 tx=0x7f3270099df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=622027695177cc3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.748+0000 7f3269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.748+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32700b4060 con 0x56479a783bf0 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.748+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f327001dc70 con 0x56479a783bf0 2024-03-19T02:28:44.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.749+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3278013330 con 0x56479a783bf0 2024-03-19T02:28:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.750+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32700b9050 con 0x56479a783bf0 2024-03-19T02:28:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.750+0000 7f328b1bc740 10 client.25862 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.750+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56479a80bc60 con 0x56479a783bf0 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.750+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3278013330 con 0x56479a783bf0 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f327009bb40 con 0x56479a783bf0 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 10 client.25862 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 10 client.25862 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 20 client.25862 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a80e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 4 client.25862 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56479a6f39f0 con 0x56479a79ca90 2024-03-19T02:28:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.751+0000 7f3286369640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a80e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.752+0000 7f3269ffb640 10 client.25862 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:44.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.752+0000 7f3286369640 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a80e3d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=879773acb1fb0bfd server_cookie=7bcbe8d22116cb94 in_seq=0 out_seq=0 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f3269ffb640 1 -- 172.21.15.116:0/2202987138 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56479a6f39f0 con 0x56479a79ca90 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f3269ffb640 10 client.25862 handle_command_reply: tid=0 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 client.25862 shutdown 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 msgr2=0x56479a790aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32700cc9f0 tx=0x7f3270099df0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 shutdown_connections 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56479a79ca90 0x56479a80e3d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56479a78c7d0 0x56479a80a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56479a783bf0 0x56479a790aa0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 --2- 172.21.15.116:0/2202987138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56479a7832d0 0x56479a790560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 >> 172.21.15.116:0/2202987138 conn(0x56479a77da00 msgr2=0x56479a78a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f3269ffb640 0 client.25862 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 shutdown_connections 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.753+0000 7f328b1bc740 1 -- 172.21.15.116:0/2202987138 wait complete. 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f328b1bc740 20 client.25862 trim_cache size 0 max 16384 2024-03-19T02:28:44.755 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 msgr2=0x7f326c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 0x7f326c0a4f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3278009fd0 tx=0x7f3278009d80 comp rx=0 tx=0).stop 2024-03-19T02:28:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 msgr2=0x7f3280199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f327000e9f0 tx=0x7f327000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 shutdown_connections 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f326c0a2a90 0x7f326c0a4f50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32801066b0 0x7f32801a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3280101060 0x7f3280199330 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 --2- 172.21.15.116:0/1625667468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32801006a0 0x7f3280198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 >> 172.21.15.116:0/1625667468 conn(0x7f32800778d0 msgr2=0x7f3280102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 shutdown_connections 2024-03-19T02:28:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.754+0000 7f32885cd640 1 -- 172.21.15.116:0/1625667468 wait complete. 2024-03-19T02:28:44.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:28:44.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 4667 6.01 MB/sec warmup 7 sec latency 38.689 ms 2024-03-19T02:28:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- 172.21.15.116:0/1612689374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01010a0 msgr2=0x7fdda010d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 --2- 172.21.15.116:0/1612689374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01010a0 0x7fdda010d5b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009cd0 tx=0x7fdd9802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- 172.21.15.116:0/1612689374 shutdown_connections 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 --2- 172.21.15.116:0/1612689374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01010a0 0x7fdda010d5b0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 --2- 172.21.15.116:0/1612689374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006e0 0x7fdda0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 --2- 172.21.15.116:0/1612689374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01066f0 0x7fdda0106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- 172.21.15.116:0/1612689374 >> 172.21.15.116:0/1612689374 conn(0x7fdda0077830 msgr2=0x7fdda0077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- 172.21.15.116:0/1612689374 shutdown_connections 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- 172.21.15.116:0/1612689374 wait complete. 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 Processor -- start 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.882+0000 7fdda83a9640 1 -- start start 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01010a0 0x7fdda0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda0110970 con 0x7fdda01066f0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01107f0 con 0x7fdda01006e0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda83a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda0110af0 con 0x7fdda01010a0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6145640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36596/0 (socket says 172.21.15.116:36596) 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6145640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53188/0 (socket says 172.21.15.116:53188) 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 -- 172.21.15.116:0/3667250555 learned_addr learned my addr 172.21.15.116:0/3667250555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda5944640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01010a0 0x7fdda0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 -- 172.21.15.116:0/3667250555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01010a0 msgr2=0x7fdda0199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01010a0 0x7fdda0199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 -- 172.21.15.116:0/3667250555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 msgr2=0x7fdda0198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 -- 172.21.15.116:0/3667250555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda01a43b0 con 0x7fdda01066f0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6145640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.883+0000 7fdda6946640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004af0 tx=0x7fdd98035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.884+0000 7fdd977fe640 1 -- 172.21.15.116:0/3667250555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98039040 con 0x7fdda01066f0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.884+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda01a4580 con 0x7fdda01066f0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.884+0000 7fdd977fe640 1 -- 172.21.15.116:0/3667250555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd98044070 con 0x7fdda01066f0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.884+0000 7fdd977fe640 1 -- 172.21.15.116:0/3667250555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd980042a0 con 0x7fdda01066f0 2024-03-19T02:28:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.884+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda01a49f0 con 0x7fdda01066f0 2024-03-19T02:28:44.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.886+0000 7fdd977fe640 1 -- 172.21.15.116:0/3667250555 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7fdd98003970 con 0x7fdda01066f0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 Processor -- start 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 -- start start 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 0x555c7c017290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c0177d0 0x555c7c019bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555c7bf714b0 con 0x555c7c00ebf0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555c7bf19d70 con 0x555c7c0177d0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fddaaf92740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555c7bf6d550 con 0x555c7c00e2d0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda6145640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 0x555c7c017290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdd977fe640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 0x7fdd8c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda6145640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 0x555c7c017290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36610/0 (socket says 172.21.15.116:36610) 2024-03-19T02:28:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56142/0 (socket says 172.21.15.116:56142) 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda6145640 1 -- 172.21.15.116:0/645081490 learned_addr learned my addr 172.21.15.116:0/645081490 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c0177d0 0x555c7c019bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda6946640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 0x7fdd8c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 -- 172.21.15.116:0/645081490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c0177d0 msgr2=0x555c7c019bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c0177d0 0x555c7c019bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 -- 172.21.15.116:0/645081490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 msgr2=0x555c7c017290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 0x555c7c017290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.887+0000 7fdda5944640 1 -- 172.21.15.116:0/645081490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c7bf3c210 con 0x555c7c00e2d0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdd977fe640 1 -- 172.21.15.116:0/3667250555 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd980f17a0 con 0x7fdda01066f0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdda5944640 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009730 tx=0x7fdd900096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8049f8b26fa74896 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdda6946640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 0x7fdd8c0a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdd980eeb60 tx=0x7fdd980a8040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdd94bfe640 1 -- 172.21.15.116:0/645081490 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90004050 con 0x555c7c00e2d0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdd94bfe640 1 -- 172.21.15.116:0/645081490 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd900054d0 con 0x555c7c00e2d0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.888+0000 7fdd94bfe640 1 -- 172.21.15.116:0/645081490 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90004c70 con 0x555c7c00e2d0 2024-03-19T02:28:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- 172.21.15.116:0/645081490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 msgr2=0x555c7c00e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009730 tx=0x7fdd900096c0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- 172.21.15.116:0/645081490 shutdown_connections 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c0177d0 0x555c7c019bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00ebf0 0x555c7c017290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 --2- 172.21.15.116:0/645081490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00e2d0 0x555c7c00e6b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- 172.21.15.116:0/645081490 >> 172.21.15.116:0/645081490 conn(0x555c7c008a00 msgr2=0x555c7c008de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- 172.21.15.116:0/645081490 shutdown_connections 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- 172.21.15.116:0/645081490 wait complete. 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 -- start start 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.889+0000 7fddaaf92740 1 Processor -- start 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c014820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c014d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555c7c01a5c0 con 0x555c7c0177d0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555c7bf2d310 con 0x555c7c00e2d0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555c7c01a740 con 0x555c7c00ebf0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 10 client.0 _ll_register_callbacks cb 0x555c7bf212d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36616/0 (socket says 172.21.15.116:36616) 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 -- 172.21.15.116:0/3301646344 learned_addr learned my addr 172.21.15.116:0/3301646344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda6145640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c014d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c014820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 -- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 msgr2=0x555c7c014d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c014d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 -- 172.21.15.116:0/3301646344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 msgr2=0x555c7c014820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c014820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 -- 172.21.15.116:0/3301646344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c7bc853f0 con 0x555c7c0177d0 2024-03-19T02:28:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdda5944640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fdd90004850 tx=0x7fdd90004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7358e11fd6363e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdd89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90005110 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd900052b0 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.890+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd900054d0 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd90011020 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 10 client.16044 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c7bd4f7d0 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd900054d0 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fdd9002f430 con 0x555c7c0177d0 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 10 client.16044 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 10 client.16044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 20 client.16044 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c027a00 0x555c7c029df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 4 client.16044 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c7bfcf3e0 con 0x555c7c027a00 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.891+0000 7fdda6145640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c027a00 0x555c7c029df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.892+0000 7fdda6145640 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c027a00 0x555c7c029df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=510ab5a1e80bc08e server_cookie=75bc2ecc5459eda0 in_seq=0 out_seq=0 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.892+0000 7fdd89ffb640 10 client.16044 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fdd89ffb640 1 -- 172.21.15.116:0/3301646344 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555c7bfcf3e0 con 0x555c7c027a00 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fdd89ffb640 10 client.16044 handle_command_reply: tid=0 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 client.16044 shutdown 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 msgr2=0x555c7c0152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fdd90004850 tx=0x7fdd90004950 comp rx=0 tx=0).stop 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 shutdown_connections 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c027a00 0x555c7c029df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c0152a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c014d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 --2- 172.21.15.116:0/3301646344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c014820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 >> 172.21.15.116:0/3301646344 conn(0x555c7c008a00 msgr2=0x555c7c00a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fdd89ffb640 0 client.16044 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 shutdown_connections 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 1 -- 172.21.15.116:0/3301646344 wait complete. 2024-03-19T02:28:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.893+0000 7fddaaf92740 20 client.16044 trim_cache size 0 max 16384 2024-03-19T02:28:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.908+0000 7fddaaf92740 1 Processor -- start 2024-03-19T02:28:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 -- start start 2024-03-19T02:28:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c016f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c023360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01aeae0 con 0x555c7c0177d0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b0650 con 0x555c7c00e2d0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fddaaf92740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01af1b0 con 0x555c7c00ebf0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36622/0 (socket says 172.21.15.116:36622) 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda6145640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c023360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 -- 172.21.15.116:0/4243686017 learned_addr learned my addr 172.21.15.116:0/4243686017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c016f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 -- 172.21.15.116:0/4243686017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 msgr2=0x555c7c023360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c023360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 -- 172.21.15.116:0/4243686017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 msgr2=0x555c7c016f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c016f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 -- 172.21.15.116:0/4243686017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c7c0137d0 con 0x555c7c0177d0 2024-03-19T02:28:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdda5944640 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009fd0 tx=0x7fdd90005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9719820fbb24c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.909+0000 7fdd8bfff640 1 -- 172.21.15.116:0/4243686017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90009bd0 con 0x555c7c0177d0 2024-03-19T02:28:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fdd8bfff640 1 -- 172.21.15.116:0/4243686017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd900111c0 con 0x555c7c0177d0 2024-03-19T02:28:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fdd8bfff640 1 -- 172.21.15.116:0/4243686017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9002f430 con 0x555c7c0177d0 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 -- 172.21.15.116:0/4243686017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 msgr2=0x555c7c007ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009fd0 tx=0x7fdd90005970 comp rx=0 tx=0).stop 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 -- 172.21.15.116:0/4243686017 shutdown_connections 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c0177d0 0x555c7c007ba0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c00ebf0 0x555c7c023360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 --2- 172.21.15.116:0/4243686017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00e2d0 0x555c7c016f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 -- 172.21.15.116:0/4243686017 >> 172.21.15.116:0/4243686017 conn(0x555c7c008a00 msgr2=0x555c7c008de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 -- 172.21.15.116:0/4243686017 shutdown_connections 2024-03-19T02:28:44.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.910+0000 7fddaaf92740 1 -- 172.21.15.116:0/4243686017 wait complete. 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 -- start start 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 Processor -- start 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 0x555c7c0111b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c0177d0 0x555c7c011730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01aff50 con 0x555c7c00e2d0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01af880 con 0x555c7c00ebf0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01acac0 con 0x555c7c0177d0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fddaaf92740 10 client.0 _ll_register_callbacks cb 0x7fdd9c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda5944640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36630/0 (socket says 172.21.15.116:36630) 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda5944640 1 -- 172.21.15.116:0/1794786132 learned_addr learned my addr 172.21.15.116:0/1794786132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 0x555c7c0111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda5944640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c0177d0 0x555c7c011730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 -- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c0177d0 msgr2=0x555c7c011730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c0177d0 0x555c7c011730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 -- 172.21.15.116:0/1794786132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 msgr2=0x555c7c01fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda6145640 1 -- 172.21.15.116:0/1794786132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c7c011f60 con 0x555c7c00ebf0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.911+0000 7fdda5944640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdda6145640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 0x555c7c0111b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x555c7c02a570 tx=0x7fdd9c01f830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66dfd034c43ebe71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c005220 con 0x555c7c00ebf0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd900111c0 con 0x555c7c00ebf0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd9c0053c0 con 0x555c7c00ebf0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd9c0055c0 con 0x555c7c00ebf0 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 10 client.25880 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:44.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c7c08dd30 con 0x555c7c00ebf0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd900111c0 con 0x555c7c00ebf0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fdd9c026070 con 0x555c7c00ebf0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 10 client.25880 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 10 client.25880 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 20 client.25880 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.912+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c08e4b0 0x555c7c0908d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 4 client.25880 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c7bf7e9f0 con 0x555c7c08e4b0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdda5944640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c08e4b0 0x555c7c0908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdda5944640 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c08e4b0 0x555c7c0908d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7713c16987e3f12 server_cookie=6ae5d6970341028b in_seq=0 out_seq=0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdd89ffb640 10 client.25880 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdd89ffb640 1 -- 172.21.15.116:0/1794786132 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c7bf7e9f0 con 0x555c7c08e4b0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdd89ffb640 10 client.25880 handle_command_reply: tid=0 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 client.25880 shutdown 2024-03-19T02:28:44.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 msgr2=0x555c7c0111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 0x555c7c0111b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x555c7c02a570 tx=0x7fdd9c01f830 comp rx=0 tx=0).stop 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 shutdown_connections 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x555c7c08e4b0 0x555c7c0908d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c7c0177d0 0x555c7c011730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fdd89ffb640 0 client.25880 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c7c00ebf0 0x555c7c0111b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 --2- 172.21.15.116:0/1794786132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c7c00e2d0 0x555c7c01fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 >> 172.21.15.116:0/1794786132 conn(0x555c7c008a00 msgr2=0x555c7c007ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 shutdown_connections 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.913+0000 7fddaaf92740 1 -- 172.21.15.116:0/1794786132 wait complete. 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fddaaf92740 20 client.25880 trim_cache size 0 max 16384 2024-03-19T02:28:44.920 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 msgr2=0x7fdd8c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 0x7fdd8c0a4f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdd980eeb60 tx=0x7fdd980a8040 comp rx=0 tx=0).stop 2024-03-19T02:28:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 msgr2=0x7fdda01a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004af0 tx=0x7fdd98035ce0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 shutdown_connections 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2a90 0x7fdd8c0a4f50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066f0 0x7fdda01a3e70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01010a0 0x7fdda0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 --2- 172.21.15.116:0/3667250555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01006e0 0x7fdda0198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 >> 172.21.15.116:0/3667250555 conn(0x7fdda0077830 msgr2=0x7fdda01032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 shutdown_connections 2024-03-19T02:28:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:44.914+0000 7fdda83a9640 1 -- 172.21.15.116:0/3667250555 wait complete. 2024-03-19T02:28:44.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.081+0000 7f73fac85640 1 -- 172.21.15.116:0/2804463293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41006a0 msgr2=0x7f73f4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.081+0000 7f73fac85640 1 --2- 172.21.15.116:0/2804463293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41006a0 0x7f73f4100b20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009930 tx=0x7f73e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- 172.21.15.116:0/2804463293 shutdown_connections 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- 172.21.15.116:0/2804463293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f4101060 0x7f73f410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- 172.21.15.116:0/2804463293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41006a0 0x7f73f4100b20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- 172.21.15.116:0/2804463293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f41066b0 0x7f73f4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- 172.21.15.116:0/2804463293 >> 172.21.15.116:0/2804463293 conn(0x7f73f40778d0 msgr2=0x7f73f4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- 172.21.15.116:0/2804463293 shutdown_connections 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- 172.21.15.116:0/2804463293 wait complete. 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 Processor -- start 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- start start 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f41006a0 0x7f73f4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4101060 0x7f73f4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4110970 con 0x7f73f41066b0 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41107f0 con 0x7f73f41006a0 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73fac85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4110af0 con 0x7f73f4101060 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.082+0000 7f73f9222640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4101060 0x7f73f4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f8a21640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f41006a0 0x7f73f4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 -- 172.21.15.116:0/1245966214 learned_addr learned my addr 172.21.15.116:0/1245966214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 -- 172.21.15.116:0/1245966214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4101060 msgr2=0x7f73f4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4101060 0x7f73f4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 -- 172.21.15.116:0/1245966214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f41006a0 msgr2=0x7f73f4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f41006a0 0x7f73f4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 -- 172.21.15.116:0/1245966214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f419cfd0 con 0x7f73f41066b0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f9222640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00eb10 tx=0x7f73ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f1ffb640 1 -- 172.21.15.116:0/1245966214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec004b20 con 0x7f73f41066b0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f1ffb640 1 -- 172.21.15.116:0/1245966214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ec004cc0 con 0x7f73f41066b0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f419d2c0 con 0x7f73f41066b0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73f1ffb640 1 -- 172.21.15.116:0/1245966214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec010690 con 0x7f73f41066b0 2024-03-19T02:28:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.083+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41a4b50 con 0x7f73f41066b0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.085+0000 7f73f1ffb640 1 -- 172.21.15.116:0/1245966214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f73ec005020 con 0x7f73f41066b0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f1ffb640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 0x7f73e40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 Processor -- start 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f1ffb640 1 -- 172.21.15.116:0/1245966214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f73ec0cfa20 con 0x7f73f41066b0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 -- start start 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 0x7f73e40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7a996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7aa2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e0d79fc4b0 con 0x55e0d7a992d0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e0d79a4d70 con 0x55e0d7a99bf0 2024-03-19T02:28:45.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73fd875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0d79f8550 con 0x55e0d7aa27d0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7aa2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f9222640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f9222640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56182/0 (socket says 172.21.15.116:56182) 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f9222640 1 -- 172.21.15.116:0/534103537 learned_addr learned my addr 172.21.15.116:0/534103537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f8a21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7a996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.086+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7aa2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53248/0 (socket says 172.21.15.116:53248) 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 0x7f73e40a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009ba0 tx=0x7f73e8011440 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 -- 172.21.15.116:0/534103537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 msgr2=0x55e0d7aa2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7aa2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 -- 172.21.15.116:0/534103537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 msgr2=0x55e0d7a996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7a996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 -- 172.21.15.116:0/534103537 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d79c7210 con 0x55e0d7aa27d0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73f9222640 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0cfd40 tx=0x7f73ec09b5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a187c1f7a9826aec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73deffd640 1 -- 172.21.15.116:0/534103537 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec09bab0 con 0x55e0d7aa27d0 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73deffd640 1 -- 172.21.15.116:0/534103537 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ec09fc00 con 0x55e0d7aa27d0 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.087+0000 7f73deffd640 1 -- 172.21.15.116:0/534103537 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0a3fc0 con 0x55e0d7aa27d0 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- 172.21.15.116:0/534103537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 msgr2=0x55e0d7aa4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0cfd40 tx=0x7f73ec09b5f0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- 172.21.15.116:0/534103537 shutdown_connections 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa4bc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7aa2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 --2- 172.21.15.116:0/534103537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7a996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- 172.21.15.116:0/534103537 >> 172.21.15.116:0/534103537 conn(0x55e0d7a93a00 msgr2=0x55e0d7a93de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- 172.21.15.116:0/534103537 shutdown_connections 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- 172.21.15.116:0/534103537 wait complete. 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 -- start start 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.088+0000 7f73fd875740 1 Processor -- start 2024-03-19T02:28:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e0d7aa5560 con 0x55e0d7a99bf0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e0d79b8310 con 0x55e0d7aa27d0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0d7aa56e0 con 0x55e0d7a992d0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73fd875740 10 client.0 _ll_register_callbacks cb 0x55e0d79ac2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f8a21640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36666/0 (socket says 172.21.15.116:36666) 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 -- 172.21.15.116:0/1394106039 learned_addr learned my addr 172.21.15.116:0/1394106039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f9222640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 -- 172.21.15.116:0/1394106039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 msgr2=0x55e0d7a9f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 -- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7aa27d0 msgr2=0x55e0d7aa0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 -- 172.21.15.116:0/1394106039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d77103f0 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73f3fff640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f73e0036040 tx=0x7f73e002c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4160be8567766b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.089+0000 7f73d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e002c5d0 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f73e0005030 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73ec09fc00 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73e0005230 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e0d77da7d0 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73ec09fc00 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f73e00035f0 con 0x55e0d7a99bf0 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:45.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 20 client.16062 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7ab2b40 0x55e0d7ab4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 4 client.16062 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0d7a5a3e0 con 0x55e0d7ab2b40 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.090+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7ab2b40 0x55e0d7ab4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.091+0000 7f73d3fff640 10 client.16062 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:45.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.091+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7ab2b40 0x55e0d7ab4f30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3def69756a6381c server_cookie=dd24aab7aff8f484 in_seq=0 out_seq=0 2024-03-19T02:28:45.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73d3fff640 1 -- 172.21.15.116:0/1394106039 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e0d7a5a3e0 con 0x55e0d7ab2b40 2024-03-19T02:28:45.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73d3fff640 10 client.16062 handle_command_reply: tid=0 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 client.16062 shutdown 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 msgr2=0x55e0d7a9fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f73e0036040 tx=0x7f73e002c9f0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 shutdown_connections 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7ab2b40 0x55e0d7ab4f30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aa0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9fd00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73d3fff640 0 client.16062 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 --2- 172.21.15.116:0/1394106039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 >> 172.21.15.116:0/1394106039 conn(0x55e0d7a93a00 msgr2=0x55e0d7a95750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 shutdown_connections 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.092+0000 7f73fd875740 1 -- 172.21.15.116:0/1394106039 wait complete. 2024-03-19T02:28:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.093+0000 7f73fd875740 20 client.16062 trim_cache size 0 max 16384 2024-03-19T02:28:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 Processor -- start 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 -- start start 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41aebf0 con 0x55e0d7a992d0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41b0760 con 0x55e0d7a99bf0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73fd875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41af2c0 con 0x55e0d7aa27d0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f8a21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f8a21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36672/0 (socket says 172.21.15.116:36672) 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f8a21640 1 -- 172.21.15.116:0/3089277046 learned_addr learned my addr 172.21.15.116:0/3089277046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f9222640 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f3fff640 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.107+0000 7f73f8a21640 1 -- 172.21.15.116:0/3089277046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 msgr2=0x55e0d7aae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73f8a21640 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73f8a21640 1 -- 172.21.15.116:0/3089277046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 msgr2=0x55e0d7a9e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73f8a21640 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73f8a21640 1 -- 172.21.15.116:0/3089277046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d7ab5590 con 0x55e0d7a992d0 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73f8a21640 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x55e0d7ab57c0 tx=0x7f73e801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8410dc58cdddc5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73de7fc640 1 -- 172.21.15.116:0/3089277046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8017040 con 0x55e0d7a992d0 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73de7fc640 1 -- 172.21.15.116:0/3089277046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73e8019050 con 0x55e0d7a992d0 2024-03-19T02:28:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73de7fc640 1 -- 172.21.15.116:0/3089277046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8005410 con 0x55e0d7a992d0 2024-03-19T02:28:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.108+0000 7f73fd875740 1 -- 172.21.15.116:0/3089277046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 msgr2=0x55e0d7aa1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x55e0d7ab57c0 tx=0x7f73e801c000 comp rx=0 tx=0).stop 2024-03-19T02:28:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- 172.21.15.116:0/3089277046 shutdown_connections 2024-03-19T02:28:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7aa27d0 0x55e0d7aae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- 172.21.15.116:0/3089277046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7a992d0 0x55e0d7aa1ef0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- 172.21.15.116:0/3089277046 >> 172.21.15.116:0/3089277046 conn(0x55e0d7a93a00 msgr2=0x55e0d7a94e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- 172.21.15.116:0/3089277046 shutdown_connections 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- 172.21.15.116:0/3089277046 wait complete. 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- start start 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 Processor -- start 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41b0060 con 0x55e0d7aa27d0 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41af990 con 0x55e0d7a99bf0 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41acbd0 con 0x55e0d7a992d0 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73fd875740 10 client.0 _ll_register_callbacks cb 0x7f73e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.109+0000 7f73f9222640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36684/0 (socket says 172.21.15.116:36684) 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 -- 172.21.15.116:0/1439091495 learned_addr learned my addr 172.21.15.116:0/1439091495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f3fff640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 -- 172.21.15.116:0/1439091495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 msgr2=0x55e0d7a9ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 -- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 msgr2=0x55e0d7a9cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 -- 172.21.15.116:0/1439091495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d7a8f0a0 con 0x55e0d7aa27d0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f3fff640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73f9222640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f73ec099d60 tx=0x7f73ec09c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7729694722d985ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0a3fc0 con 0x55e0d7aa27d0 2024-03-19T02:28:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f73ec0ccf50 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73e8019050 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73ec0a4250 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73fd875740 10 client.16068 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e0d7b19430 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.110+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73e8019050 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f73ec005310 con 0x55e0d7aa27d0 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 10 client.16068 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 20 client.16068 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7b19790 0x55e0d7b1bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 4 client.16068 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0d7a099f0 con 0x55e0d7b19790 2024-03-19T02:28:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7b19790 0x55e0d7b1bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73d3fff640 10 client.16068 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.111+0000 7f73f8a21640 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7b19790 0x55e0d7b1bb80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95af2af767a17720 server_cookie=aaff8b4b32286145 in_seq=0 out_seq=0 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73d3fff640 1 -- 172.21.15.116:0/1439091495 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0d7a099f0 con 0x55e0d7b19790 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73d3fff640 10 client.16068 handle_command_reply: tid=0 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 client.16068 shutdown 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 msgr2=0x55e0d7b17e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f73ec099d60 tx=0x7f73ec09c270 comp rx=0 tx=0).stop 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 shutdown_connections 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0d7b19790 0x55e0d7b1bb80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0d7aa27d0 0x55e0d7b17e50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0d7a99bf0 0x55e0d7a9cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73d3fff640 0 client.16068 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 --2- 172.21.15.116:0/1439091495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0d7a992d0 0x55e0d7a9ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 >> 172.21.15.116:0/1439091495 conn(0x55e0d7a93a00 msgr2=0x55e0d7a956a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 shutdown_connections 2024-03-19T02:28:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.112+0000 7f73fd875740 1 -- 172.21.15.116:0/1439091495 wait complete. 2024-03-19T02:28:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fd875740 20 client.16068 trim_cache size 0 max 16384 2024-03-19T02:28:45.119 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:28:45.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 msgr2=0x7f73e40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 0x7f73e40a4f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009ba0 tx=0x7f73e8011440 comp rx=0 tx=0).stop 2024-03-19T02:28:45.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 msgr2=0x7f73f419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:45.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00eb10 tx=0x7f73ec00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 shutdown_connections 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f73e40a2a90 0x7f73e40a4f50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73f41066b0 0x7f73f419c8a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73f4101060 0x7f73f4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 --2- 172.21.15.116:0/1245966214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f41006a0 0x7f73f4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 >> 172.21.15.116:0/1245966214 conn(0x7f73f40778d0 msgr2=0x7f73f4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 shutdown_connections 2024-03-19T02:28:45.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:45.113+0000 7f73fac85640 1 -- 172.21.15.116:0/1245966214 wait complete. 2024-03-19T02:28:45.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3740323119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3740323119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3740323119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:28:45.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 5386 5.68 MB/sec warmup 8 sec latency 25.721 ms 2024-03-19T02:28:46.126 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:28:46.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:28:46.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:28:46.151 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:45 smithi116 ceph-mon[32263]: pgmap v31: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.6 MiB/s wr, 66 op/s 2024-03-19T02:28:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:45 smithi162 ceph-mon[35761]: pgmap v31: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.6 MiB/s wr, 66 op/s 2024-03-19T02:28:46.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 -- 172.21.15.116:0/631063313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d41059f0 msgr2=0x7f33d410d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 --2- 172.21.15.116:0/631063313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d41059f0 0x7f33d410d600 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009cd0 tx=0x7f33cc02c140 comp rx=0 tx=0).stop 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 -- 172.21.15.116:0/631063313 shutdown_connections 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 --2- 172.21.15.116:0/631063313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d41059f0 0x7f33d410d600 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 --2- 172.21.15.116:0/631063313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41019f0 0x7f33d41052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 --2- 172.21.15.116:0/631063313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d4101040 0x7f33d4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 -- 172.21.15.116:0/631063313 >> 172.21.15.116:0/631063313 conn(0x7f33d4076c40 msgr2=0x7f33d40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 -- 172.21.15.116:0/631063313 shutdown_connections 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.236+0000 7f33db2ff640 1 -- 172.21.15.116:0/631063313 wait complete. 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 Processor -- start 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 -- start start 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41059f0 0x7f33d40fc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d40fcb10 0x7f33d41a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f33d4110960 con 0x7f33d4101040 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41107e0 con 0x7f33d41059f0 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33db2ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f33d4110ae0 con 0x7f33d40fcb10 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33d909b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33d909b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36698/0 (socket says 172.21.15.116:36698) 2024-03-19T02:28:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33d909b640 1 -- 172.21.15.116:0/3603507914 learned_addr learned my addr 172.21.15.116:0/3603507914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33d889a640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41059f0 0x7f33d40fc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.237+0000 7f33d989c640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d40fcb10 0x7f33d41a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 -- 172.21.15.116:0/3603507914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d40fcb10 msgr2=0x7f33d41a6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d40fcb10 0x7f33d41a6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 -- 172.21.15.116:0/3603507914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41059f0 msgr2=0x7f33d40fc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41059f0 0x7f33d40fc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 -- 172.21.15.116:0/3603507914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d41a6870 con 0x7f33d4101040 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33d909b640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33ca7fc640 1 -- 172.21.15.116:0/3603507914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d000cd80 con 0x7f33d4101040 2024-03-19T02:28:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33ca7fc640 1 -- 172.21.15.116:0/3603507914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d00054d0 con 0x7f33d4101040 2024-03-19T02:28:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d41a6b60 con 0x7f33d4101040 2024-03-19T02:28:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33ca7fc640 1 -- 172.21.15.116:0/3603507914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0010490 con 0x7f33d4101040 2024-03-19T02:28:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.238+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d41a7060 con 0x7f33d4101040 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.240+0000 7f33ca7fc640 1 -- 172.21.15.116:0/3603507914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f33d0005020 con 0x7f33d4101040 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 Processor -- start 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 -- start start 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ca7fc640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 0x7f33c00a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed1d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed1fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56010ec78160 con 0x56010ed1d880 2024-03-19T02:28:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56010ec20e10 con 0x56010ed14380 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56010ec74200 con 0x56010ed14ca0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33d989c640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 0x7f33c00a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33ca7fc640 1 -- 172.21.15.116:0/3603507914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f33d00cf7e0 con 0x7f33d4101040 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33d909b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33d909b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53308/0 (socket says 172.21.15.116:53308) 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.241+0000 7f33d909b640 1 -- 172.21.15.116:0/3415100334 learned_addr learned my addr 172.21.15.116:0/3415100334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d889a640 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed1d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d989c640 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed1fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 -- 172.21.15.116:0/3415100334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 msgr2=0x56010ed1d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed1d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d989c640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 0x7f33c00a4f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005cc0 tx=0x7f33cc032e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 -- 172.21.15.116:0/3415100334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 msgr2=0x56010ed1fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed1fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 -- 172.21.15.116:0/3415100334 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010ea547e0 con 0x56010ed14380 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33d909b640 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33d0014040 tx=0x7f33d009b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438e9cca0175fc49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.242+0000 7f33bf7fe640 1 -- 172.21.15.116:0/3415100334 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0005680 con 0x56010ed14380 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33bf7fe640 1 -- 172.21.15.116:0/3415100334 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33bf7fe640 1 -- 172.21.15.116:0/3415100334 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0099c00 con 0x56010ed14380 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3415100334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 msgr2=0x56010ed14760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33d0014040 tx=0x7f33d009b3b0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3415100334 shutdown_connections 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed1fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed1d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3415100334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed14760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.243+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3415100334 >> 172.21.15.116:0/3415100334 conn(0x56010ed0eab0 msgr2=0x56010ed0ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3415100334 shutdown_connections 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3415100334 wait complete. 2024-03-19T02:28:46.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 -- start start 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 Processor -- start 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 0x56010ed1acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.244+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed1d880 0x56010ed1b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56010ed20670 con 0x56010ed14ca0 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56010ec341a0 con 0x56010ed14380 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56010ed207f0 con 0x56010ed1d880 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33ddfc8740 10 client.0 _ll_register_callbacks cb 0x56010e98bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d889a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 0x56010ed1acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d889a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 0x56010ed1acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36708/0 (socket says 172.21.15.116:36708) 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d889a640 1 -- 172.21.15.116:0/3531284246 learned_addr learned my addr 172.21.15.116:0/3531284246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d989c640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed1d880 0x56010ed1b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 -- 172.21.15.116:0/3531284246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed1d880 msgr2=0x56010ed1b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed1d880 0x56010ed1b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 -- 172.21.15.116:0/3531284246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 msgr2=0x56010ed1acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 0x56010ed1acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.245+0000 7f33d909b640 1 -- 172.21.15.116:0/3531284246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010eab4f70 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33d909b640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1a790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f33d0098240 tx=0x7f33d009b250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=448307d25ae20593 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00b8020 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f33d00860c0 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33d00ad050 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33ddfc8740 10 client.25910 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.246+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56010e94c170 con 0x56010ed14380 2024-03-19T02:28:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f33d00a3b80 con 0x56010ed14380 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 10 client.25910 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 10 client.25910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 20 client.25910 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed2d980 0x56010ed2fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 4 client.25910 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.247+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56010ec856a0 con 0x56010ed2d980 2024-03-19T02:28:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.248+0000 7f33d889a640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed2d980 0x56010ed2fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.249+0000 7f33bcff9640 10 client.25910 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.249+0000 7f33d889a640 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed2d980 0x56010ed2fd70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac8e02bc8c75fe2d server_cookie=9ae432b10be9a496 in_seq=0 out_seq=0 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.251+0000 7f33bcff9640 1 -- 172.21.15.116:0/3531284246 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56010ec856a0 con 0x56010ed2d980 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.251+0000 7f33bcff9640 10 client.25910 handle_command_reply: tid=0 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.251+0000 7f33ddfc8740 1 client.25910 shutdown 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.251+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 msgr2=0x56010ed1a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.251+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1a790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f33d0098240 tx=0x7f33d009b250 comp rx=0 tx=0).stop 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 shutdown_connections 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed2d980 0x56010ed2fd70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed1d880 0x56010ed1b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33bcff9640 0 client.25910 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14ca0 0x56010ed1acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3531284246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1a790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 >> 172.21.15.116:0/3531284246 conn(0x56010ed0eab0 msgr2=0x56010ed10800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 shutdown_connections 2024-03-19T02:28:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3531284246 wait complete. 2024-03-19T02:28:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.252+0000 7f33ddfc8740 20 client.25910 trim_cache size 0 max 16384 2024-03-19T02:28:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:45 smithi154 ceph-mon[35693]: pgmap v31: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.6 MiB/s wr, 66 op/s 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 Processor -- start 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 -- start start 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41b0dc0 con 0x56010ed1d880 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41b2930 con 0x56010ed14380 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41b1490 con 0x56010ed14ca0 2024-03-19T02:28:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d989c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d889a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d989c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36720/0 (socket says 172.21.15.116:36720) 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d889a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56254/0 (socket says 172.21.15.116:56254) 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d989c640 1 -- 172.21.15.116:0/3283855369 learned_addr learned my addr 172.21.15.116:0/3283855369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 -- 172.21.15.116:0/3283855369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 msgr2=0x56010ed29410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 -- 172.21.15.116:0/3283855369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 msgr2=0x56010ed19640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d909b640 1 -- 172.21.15.116:0/3283855369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010ed300e0 con 0x56010ed14380 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d889a640 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.270+0000 7f33d989c640 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.271+0000 7f33d909b640 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1cec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f33d0097cd0 tx=0x7f33d0099b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c900fa8e9a82b75c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.271+0000 7f33beffd640 1 -- 172.21.15.116:0/3283855369 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00b8070 con 0x56010ed14380 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.271+0000 7f33beffd640 1 -- 172.21.15.116:0/3283855369 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.271+0000 7f33beffd640 1 -- 172.21.15.116:0/3283855369 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00ccd60 con 0x56010ed14380 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3283855369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 msgr2=0x56010ed1cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1cec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f33d0097cd0 tx=0x7f33d0099b70 comp rx=0 tx=0).stop 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3283855369 shutdown_connections 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed1d880 0x56010ed19640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed29410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/3283855369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed14380 0x56010ed1cec0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3283855369 >> 172.21.15.116:0/3283855369 conn(0x56010ed0eab0 msgr2=0x56010ed18f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3283855369 shutdown_connections 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- 172.21.15.116:0/3283855369 wait complete. 2024-03-19T02:28:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- start start 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 Processor -- start 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed17810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41b2230 con 0x56010ed14380 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41b1b60 con 0x56010ed1d880 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.272+0000 7f33ddfc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f33d41aeda0 con 0x56010ed14ca0 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33ddfc8740 10 client.0 _ll_register_callbacks cb 0x7f33d009b640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d989c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36728/0 (socket says 172.21.15.116:36728) 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d889a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed17810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 -- 172.21.15.116:0/2341384567 learned_addr learned my addr 172.21.15.116:0/2341384567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d989c640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53330/0 (socket says 172.21.15.116:53330) 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 -- 172.21.15.116:0/2341384567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 msgr2=0x56010ed17810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed17810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 -- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 msgr2=0x56010ed17d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 -- 172.21.15.116:0/2341384567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010ed14ad0 con 0x56010ed14380 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d889a640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed17810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d989c640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33d909b640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f33d0004a10 tx=0x7f33d009b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623174eeb1238568 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00b8050 con 0x56010ed14380 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f33d009aec0 con 0x56010ed14380 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.273+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33d0093050 con 0x56010ed14380 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 10 client.16086 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56010ed14ad0 con 0x56010ed14380 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d009bf70 con 0x56010ed14380 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f33d00975c0 con 0x56010ed14380 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 10 client.16086 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 20 client.16086 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed93d60 0x56010ed96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 4 client.16086 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56010ec7f9f0 con 0x56010ed93d60 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33d889a640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed93d60 0x56010ed96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.275+0000 7f33bcff9640 10 client.16086 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.274+0000 7f33d889a640 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed93d60 0x56010ed96180 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389ccf0a1b46b442 server_cookie=1912a723caede36 in_seq=0 out_seq=0 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33bcff9640 1 -- 172.21.15.116:0/2341384567 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56010ec7f9f0 con 0x56010ed93d60 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33bcff9640 10 client.16086 handle_command_reply: tid=0 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 client.16086 shutdown 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 msgr2=0x56010ed172d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f33d0004a10 tx=0x7f33d009b2c0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 shutdown_connections 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56010ed93d60 0x56010ed96180 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010ed1d880 0x56010ed17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010ed14ca0 0x56010ed17810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 --2- 172.21.15.116:0/2341384567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010ed14380 0x56010ed172d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 >> 172.21.15.116:0/2341384567 conn(0x56010ed0eab0 msgr2=0x56010ed1afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33bcff9640 0 client.16086 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 shutdown_connections 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 1 -- 172.21.15.116:0/2341384567 wait complete. 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.276+0000 7f33ddfc8740 20 client.16086 trim_cache size 0 max 16384 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "38c709b9-f3b0-4812-bc29-e1cdf836ed5c", 2024-03-19T02:28:46.284 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:28:46.285 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 msgr2=0x7f33c00a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 0x7f33c00a4f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005cc0 tx=0x7f33cc032e40 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 msgr2=0x7f33d40fc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000edf0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 shutdown_connections 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f33c00a2ac0 0x7f33c00a4f80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33d40fcb10 0x7f33d41a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33d41059f0 0x7f33d40fc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 --2- 172.21.15.116:0/3603507914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33d4101040 0x7f33d40fc090 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 >> 172.21.15.116:0/3603507914 conn(0x7f33d4076c40 msgr2=0x7f33d40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 shutdown_connections 2024-03-19T02:28:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.277+0000 7f33db2ff640 1 -- 172.21.15.116:0/3603507914 wait complete. 2024-03-19T02:28:46.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:28:46.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:28:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 -- 172.21.15.116:0/2985705609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41066a0 msgr2=0x7f7bc4106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/2985705609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41066a0 0x7f7bc4106a80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009930 tx=0x7f7bb402bc30 comp rx=0 tx=0).stop 2024-03-19T02:28:46.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 -- 172.21.15.116:0/2985705609 shutdown_connections 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/2985705609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc4101050 0x7f7bc410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/2985705609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 0x7f7bc4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/2985705609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc41066a0 0x7f7bc4106a80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 -- 172.21.15.116:0/2985705609 >> 172.21.15.116:0/2985705609 conn(0x7f7bc4077820 msgr2=0x7f7bc4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 -- 172.21.15.116:0/2985705609 shutdown_connections 2024-03-19T02:28:46.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.444+0000 7f7bc8e27640 1 -- 172.21.15.116:0/2985705609 wait complete. 2024-03-19T02:28:46.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 Processor -- start 2024-03-19T02:28:46.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 -- start start 2024-03-19T02:28:46.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 0x7f7bc4195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc41066a0 0x7f7bc419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f850 con 0x7f7bc4101050 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f6d0 con 0x7f7bc41066a0 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.445+0000 7f7bc8e27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f9d0 con 0x7f7bc4100690 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 0x7f7bc4195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36742/0 (socket says 172.21.15.116:36742) 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/813307126 learned_addr learned my addr 172.21.15.116:0/813307126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc41066a0 0x7f7bc419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/813307126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 msgr2=0x7f7bc4195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 0x7f7bc4195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/813307126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc41066a0 msgr2=0x7f7bc419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc41066a0 0x7f7bc419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/813307126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc419ade0 con 0x7f7bc4101050 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800e9f0 tx=0x7f7bb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bb37fe640 1 -- 172.21.15.116:0/813307126 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800ccc0 con 0x7f7bc4101050 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bb37fe640 1 -- 172.21.15.116:0/813307126 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bb800ce60 con 0x7f7bc4101050 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bb37fe640 1 -- 172.21.15.116:0/813307126 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8010430 con 0x7f7bc4101050 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc419b0d0 con 0x7f7bc4101050 2024-03-19T02:28:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.446+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc41a2960 con 0x7f7bc4101050 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.448+0000 7f7bb37fe640 1 -- 172.21.15.116:0/813307126 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f7bb8005020 con 0x7f7bc4101050 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bb37fe640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 0x7f7bac0a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 Processor -- start 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bb37fe640 1 -- 172.21.15.116:0/813307126 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7bb80cf8a0 con 0x7f7bc4101050 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bc259c640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 0x7f7bac0a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 -- start start 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb50fd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bcb5055500 con 0x55bcb50faf30 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcb4ffdbe0 con 0x55bcb50f2350 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.449+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcb50515a0 con 0x55bcb50f1a30 2024-03-19T02:28:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc259c640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 0x7f7bac0a7160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005ab0 tx=0x7f7bb4005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb50fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53346/0 (socket says 172.21.15.116:53346) 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/1407123123 learned_addr learned my addr 172.21.15.116:0/1407123123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/1407123123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 msgr2=0x55bcb50f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/1407123123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 msgr2=0x55bcb50fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb50fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 -- 172.21.15.116:0/1407123123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb501f000 con 0x55bcb50f2350 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb50fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7bb809b630 tx=0x7f7bb809b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e10258df3e4f2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bb0ff9640 1 -- 172.21.15.116:0/1407123123 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80a40d0 con 0x55bcb50f2350 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bb0ff9640 1 -- 172.21.15.116:0/1407123123 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bb80046c0 con 0x55bcb50f2350 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.450+0000 7f7bb0ff9640 1 -- 172.21.15.116:0/1407123123 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80a3c40 con 0x55bcb50f2350 2024-03-19T02:28:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1407123123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 msgr2=0x55bcb50fa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7bb809b630 tx=0x7f7bb809b5a0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1407123123 shutdown_connections 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb50fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50fa9f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1407123123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1407123123 >> 172.21.15.116:0/1407123123 conn(0x55bcb50ec160 msgr2=0x55bcb50ec540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1407123123 shutdown_connections 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1407123123 wait complete. 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.451+0000 7f7bcba0f740 1 -- start start 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 Processor -- start 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb50f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bcb50fdd20 con 0x55bcb50f1a30 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcb5011160 con 0x55bcb50f2350 2024-03-19T02:28:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcb50fdea0 con 0x55bcb50faf30 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bcba0f740 10 client.0 _ll_register_callbacks cb 0x55bcb504da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36760/0 (socket says 172.21.15.116:36760) 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 -- 172.21.15.116:0/4175717016 learned_addr learned my addr 172.21.15.116:0/4175717016 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb50f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 -- 172.21.15.116:0/4175717016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 msgr2=0x55bcb50f8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb50f8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 -- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 msgr2=0x55bcb50f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 -- 172.21.15.116:0/4175717016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb4e337d0 con 0x55bcb50f1a30 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7bc259c640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009930 tx=0x7f7bb4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ced730e5e901a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.452+0000 7f7ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4050030 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7bb403a040 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bb80046c0 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7bb405d050 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 10 client.16098 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcb4d693f0 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bb80046c0 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f7bb40037a0 con 0x55bcb50f1a30 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 20 client.16098 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb510b320 0x55bcb510d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 4 client.16098 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcb5062a40 con 0x55bcb510b320 2024-03-19T02:28:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.453+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb510b320 0x55bcb510d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.454+0000 7f7ba9ffb640 10 client.16098 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.454+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb510b320 0x55bcb510d780 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12e51fb0b5731133 server_cookie=259a888b728b84b2 in_seq=0 out_seq=0 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/4175717016 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bcb5062a40 con 0x55bcb510b320 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7ba9ffb640 10 client.16098 handle_command_reply: tid=0 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 client.16098 shutdown 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 msgr2=0x55bcb50f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009930 tx=0x7f7bb4009260 comp rx=0 tx=0).stop 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 shutdown_connections 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb510b320 0x55bcb510d780 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb50f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7ba9ffb640 0 client.16098 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/4175717016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f7f80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 >> 172.21.15.116:0/4175717016 conn(0x55bcb50ec160 msgr2=0x55bcb50edeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 shutdown_connections 2024-03-19T02:28:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.455+0000 7f7bcba0f740 1 -- 172.21.15.116:0/4175717016 wait complete. 2024-03-19T02:28:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.456+0000 7f7bcba0f740 20 client.16098 trim_cache size 0 max 16384 2024-03-19T02:28:46.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 Processor -- start 2024-03-19T02:28:46.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 -- start start 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50fa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb5106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41ac960 con 0x55bcb50faf30 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41ae4d0 con 0x55bcb50f2350 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41ad030 con 0x55bcb50f1a30 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36772/0 (socket says 172.21.15.116:36772) 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 -- 172.21.15.116:0/2106091907 learned_addr learned my addr 172.21.15.116:0/2106091907 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb5106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc259c640 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50fa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 -- 172.21.15.116:0/2106091907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 msgr2=0x55bcb50fa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50fa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 -- 172.21.15.116:0/2106091907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 msgr2=0x55bcb5106ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb5106ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 -- 172.21.15.116:0/2106091907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb4d290f0 con 0x55bcb50faf30 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.473+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0127a0 tx=0x7f7bbc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392a30831d060e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7babfff640 1 -- 172.21.15.116:0/2106091907 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc022070 con 0x55bcb50faf30 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7babfff640 1 -- 172.21.15.116:0/2106091907 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bbc003e20 con 0x55bcb50faf30 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7babfff640 1 -- 172.21.15.116:0/2106091907 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc004050 con 0x55bcb50faf30 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 -- 172.21.15.116:0/2106091907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 msgr2=0x55bcb5113050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0127a0 tx=0x7f7bbc009500 comp rx=0 tx=0).stop 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 -- 172.21.15.116:0/2106091907 shutdown_connections 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50faf30 0x55bcb5113050 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb5106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/2106091907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50f1a30 0x55bcb50fa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 -- 172.21.15.116:0/2106091907 >> 172.21.15.116:0/2106091907 conn(0x55bcb50ec160 msgr2=0x55bcb50eb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 -- 172.21.15.116:0/2106091907 shutdown_connections 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.474+0000 7f7bcba0f740 1 -- 172.21.15.116:0/2106091907 wait complete. 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 -- start start 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 Processor -- start 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb5170680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41addd0 con 0x55bcb50f1a30 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41ad700 con 0x55bcb50f2350 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc41aa940 con 0x55bcb50faf30 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bcba0f740 10 client.0 _ll_register_callbacks cb 0x7f7bb4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36780/0 (socket says 172.21.15.116:36780) 2024-03-19T02:28:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 -- 172.21.15.116:0/1346487553 learned_addr learned my addr 172.21.15.116:0/1346487553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc2d9d640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb5170680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 -- 172.21.15.116:0/1346487553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 msgr2=0x55bcb5170680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb5170680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 -- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 msgr2=0x55bcb50f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 -- 172.21.15.116:0/1346487553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb50f5cc0 con 0x55bcb50f1a30 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7bc259c640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009230 tx=0x7f7bb4004030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a7b3345f2eb506 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.475+0000 7f7ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4050040 con 0x55bcb50f1a30 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7bb403a040 con 0x55bcb50f1a30 2024-03-19T02:28:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bbc003e20 con 0x55bcb50f1a30 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7bb406e050 con 0x55bcb50f1a30 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 10 client.16104 fetch_fsmap learned FSMap version 29 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcb5171ce0 con 0x55bcb50f1a30 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bbc003e20 con 0x55bcb50f1a30 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f7bb4004ae0 con 0x55bcb50f1a30 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 10 client.16104 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 20 client.16104 populate_metadata read hostname 'smithi116' 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb5172480 0x55bcb51748a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 4 client.16104 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bcb505cd90 con 0x55bcb5172480 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.476+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb5172480 0x55bcb51748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:28:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bc1d9b640 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb5172480 0x55bcb51748a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461b3931b2242b51 server_cookie=9c958133c4576131 in_seq=0 out_seq=0 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7ba9ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7ba9ffb640 1 -- 172.21.15.116:0/1346487553 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bcb505cd90 con 0x55bcb5172480 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7ba9ffb640 10 client.16104 handle_command_reply: tid=0 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 client.16104 shutdown 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 msgr2=0x55bcb50f4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009230 tx=0x7f7bb4004030 comp rx=0 tx=0).stop 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 shutdown_connections 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb5172480 0x55bcb51748a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb50faf30 0x55bcb5170680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb50f2350 0x55bcb50f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7ba9ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 --2- 172.21.15.116:0/1346487553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb50f1a30 0x55bcb50f4ea0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.477+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 >> 172.21.15.116:0/1346487553 conn(0x55bcb50ec160 msgr2=0x55bcb50eb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 shutdown_connections 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bcba0f740 1 -- 172.21.15.116:0/1346487553 wait complete. 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bcba0f740 20 client.16104 trim_cache size 0 max 16384 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:28:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "38c709b9-f3b0-4812-bc29-e1cdf836ed5c": { 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "38c709b9-f3b0-4812-bc29-e1cdf836ed5c", 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:28:46.488 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 msgr2=0x7f7bac0a7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 0x7f7bac0a7160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005ab0 tx=0x7f7bb4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 msgr2=0x7f7bc4196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800e9f0 tx=0x7f7bb800edf0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 shutdown_connections 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bac0a4ca0 0x7f7bac0a7160 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc41066a0 0x7f7bc419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bc4101050 0x7f7bc4196330 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 --2- 172.21.15.116:0/813307126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bc4100690 0x7f7bc4195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 >> 172.21.15.116:0/813307126 conn(0x7f7bc4077820 msgr2=0x7f7bc4103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:28:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 shutdown_connections 2024-03-19T02:28:46.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:28:46.478+0000 7f7bc8e27640 1 -- 172.21.15.116:0/813307126 wait complete. 2024-03-19T02:28:46.491 INFO:tasks.cephfs.filesystem:scrub status for tag:38c709b9-f3b0-4812-bc29-e1cdf836ed5c - {'path': '/', 'tag': '38c709b9-f3b0-4812-bc29-e1cdf836ed5c', 'options': 'recursive,force'} 2024-03-19T02:28:46.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 5992 5.25 MB/sec warmup 9 sec latency 27.350 ms 2024-03-19T02:28:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:46 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:28:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:46 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:46 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:28:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:46 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:46 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:28:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:46 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:46 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:28:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:46 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:46 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:28:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:46 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:46 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:28:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:46 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:47.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 6854 5.45 MB/sec warmup 10 sec latency 18.485 ms 2024-03-19T02:28:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:28:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:47 smithi162 ceph-mon[35761]: pgmap v32: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:28:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:47 smithi154 ceph-mon[35693]: pgmap v32: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:28:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:28:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:47 smithi116 ceph-mon[32263]: pgmap v32: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:48.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 7356 5.35 MB/sec warmup 11 sec latency 15.968 ms 2024-03-19T02:28:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 7759 4.94 MB/sec warmup 12 sec latency 30.781 ms 2024-03-19T02:28:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:49 smithi162 ceph-mon[35761]: pgmap v33: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:49 smithi154 ceph-mon[35693]: pgmap v33: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:49 smithi116 ceph-mon[32263]: pgmap v33: 129 pgs: 129 active+clean; 27 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-03-19T02:28:50.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 8360 4.72 MB/sec warmup 13 sec latency 21.947 ms 2024-03-19T02:28:51.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 9082 4.62 MB/sec warmup 14 sec latency 17.081 ms 2024-03-19T02:28:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:51 smithi162 ceph-mon[35761]: pgmap v34: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.2 MiB/s wr, 102 op/s 2024-03-19T02:28:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:51 smithi154 ceph-mon[35693]: pgmap v34: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.2 MiB/s wr, 102 op/s 2024-03-19T02:28:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:51 smithi116 ceph-mon[32263]: pgmap v34: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.2 MiB/s wr, 102 op/s 2024-03-19T02:28:52.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 9691 4.48 MB/sec warmup 15 sec latency 17.234 ms 2024-03-19T02:28:53.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 10423 4.60 MB/sec warmup 16 sec latency 18.740 ms 2024-03-19T02:28:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:53 smithi162 ceph-mon[35761]: pgmap v35: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-03-19T02:28:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:53 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:28:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:53 smithi154 ceph-mon[35693]: pgmap v35: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-03-19T02:28:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:53 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:28:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:53 smithi116 ceph-mon[32263]: pgmap v35: 129 pgs: 129 active+clean; 27 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.3 MiB/s wr, 77 op/s 2024-03-19T02:28:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:53 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:28:54.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 10945 4.59 MB/sec warmup 17 sec latency 21.602 ms 2024-03-19T02:28:55.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: pgmap v36: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 121 op/s 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:28:55.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: pgmap v36: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 121 op/s 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:28:55.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:28:55.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: pgmap v36: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 2.0 MiB/s wr, 121 op/s 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:28:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:28:55.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 11341 4.36 MB/sec warmup 18 sec latency 35.562 ms 2024-03-19T02:28:56.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 12037 4.39 MB/sec warmup 19 sec latency 15.942 ms 2024-03-19T02:28:57.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 12736 4.20 MB/sec warmup 20 sec latency 26.032 ms 2024-03-19T02:28:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:57 smithi162 ceph-mon[35761]: pgmap v37: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:28:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:57 smithi154 ceph-mon[35693]: pgmap v37: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:28:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:57 smithi116 ceph-mon[32263]: pgmap v37: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:28:58.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 13426 4.25 MB/sec warmup 21 sec latency 16.534 ms 2024-03-19T02:28:59.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 14002 4.22 MB/sec warmup 22 sec latency 16.768 ms 2024-03-19T02:29:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:28:59 smithi116 ceph-mon[32263]: pgmap v38: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:28:59 smithi162 ceph-mon[35761]: pgmap v38: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:28:59 smithi154 ceph-mon[35693]: pgmap v38: 129 pgs: 129 active+clean; 28 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:00.777 INFO:tasks.workunit.client.0.smithi116.stdout: 1 14575 4.24 MB/sec warmup 23 sec latency 26.226 ms 2024-03-19T02:29:01.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 15059 4.09 MB/sec warmup 24 sec latency 33.656 ms 2024-03-19T02:29:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:01 smithi162 ceph-mon[35761]: pgmap v39: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-03-19T02:29:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:01 smithi154 ceph-mon[35693]: pgmap v39: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-03-19T02:29:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:01 smithi116 ceph-mon[32263]: pgmap v39: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.9 MiB/s wr, 124 op/s 2024-03-19T02:29:02.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 15689 4.10 MB/sec warmup 25 sec latency 54.191 ms 2024-03-19T02:29:03.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16380 3.97 MB/sec warmup 26 sec latency 16.927 ms 2024-03-19T02:29:04.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:03 smithi162 ceph-mon[35761]: pgmap v40: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:03 smithi154 ceph-mon[35693]: pgmap v40: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:03 smithi116 ceph-mon[32263]: pgmap v40: 129 pgs: 129 active+clean; 30 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:04.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 17130 4.07 MB/sec warmup 27 sec latency 22.235 ms 2024-03-19T02:29:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 17720 4.10 MB/sec warmup 28 sec latency 16.035 ms 2024-03-19T02:29:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:05 smithi162 ceph-mon[35761]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-03-19T02:29:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:05 smithi154 ceph-mon[35693]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-03-19T02:29:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:05 smithi116 ceph-mon[32263]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 2.0 MiB/s wr, 143 op/s 2024-03-19T02:29:06.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 18211 4.02 MB/sec warmup 29 sec latency 18.418 ms 2024-03-19T02:29:07.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 18747 3.92 MB/sec warmup 30 sec latency 33.010 ms 2024-03-19T02:29:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:07 smithi162 ceph-mon[35761]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:07 smithi154 ceph-mon[35693]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:07 smithi116 ceph-mon[32263]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:08.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 19456 3.93 MB/sec warmup 31 sec latency 17.592 ms 2024-03-19T02:29:09.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 20176 3.86 MB/sec warmup 32 sec latency 35.613 ms 2024-03-19T02:29:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:09 smithi162 ceph-mon[35761]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:09 smithi154 ceph-mon[35693]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:09 smithi116 ceph-mon[32263]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-03-19T02:29:10.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 21054 3.97 MB/sec warmup 33 sec latency 20.784 ms 2024-03-19T02:29:11.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 21527 3.98 MB/sec warmup 34 sec latency 16.698 ms 2024-03-19T02:29:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:11 smithi162 ceph-mon[35761]: pgmap v44: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.9 MiB/s wr, 143 op/s 2024-03-19T02:29:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:11 smithi154 ceph-mon[35693]: pgmap v44: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.9 MiB/s wr, 143 op/s 2024-03-19T02:29:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:11 smithi116 ceph-mon[32263]: pgmap v44: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 1.9 MiB/s wr, 143 op/s 2024-03-19T02:29:12.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 21927 3.88 MB/sec warmup 35 sec latency 33.080 ms 2024-03-19T02:29:13.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 22504 3.83 MB/sec warmup 36 sec latency 35.951 ms 2024-03-19T02:29:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:13 smithi162 ceph-mon[35761]: pgmap v45: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-03-19T02:29:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:13 smithi154 ceph-mon[35693]: pgmap v45: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-03-19T02:29:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:13 smithi116 ceph-mon[32263]: pgmap v45: 129 pgs: 129 active+clean; 34 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 1.4 MiB/s wr, 101 op/s 2024-03-19T02:29:14.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 23196 3.81 MB/sec warmup 37 sec latency 18.987 ms 2024-03-19T02:29:15.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 23825 3.78 MB/sec warmup 38 sec latency 17.260 ms 2024-03-19T02:29:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:15 smithi162 ceph-mon[35761]: pgmap v46: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-19T02:29:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:15 smithi154 ceph-mon[35693]: pgmap v46: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-19T02:29:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:15 smithi116 ceph-mon[32263]: pgmap v46: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.0 MiB/s wr, 142 op/s 2024-03-19T02:29:16.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:29:16.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.601+0000 7f391f1f4640 1 -- 172.21.15.116:0/1135106647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3918102180 msgr2=0x7f391810e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.601+0000 7f391f1f4640 1 --2- 172.21.15.116:0/1135106647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3918102180 0x7f391810e690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3910009ca0 tx=0x7f391002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- 172.21.15.116:0/1135106647 shutdown_connections 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- 172.21.15.116:0/1135106647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3918102180 0x7f391810e690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- 172.21.15.116:0/1135106647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- 172.21.15.116:0/1135106647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39181077d0 0x7f3918107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- 172.21.15.116:0/1135106647 >> 172.21.15.116:0/1135106647 conn(0x7f39180fd2b0 msgr2=0x7f39180ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- 172.21.15.116:0/1135106647 shutdown_connections 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- 172.21.15.116:0/1135106647 wait complete. 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 Processor -- start 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- start start 2024-03-19T02:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3918102180 0x7f39181984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39181077d0 0x7f391819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3918111a90 con 0x7f39181017c0 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3918111910 con 0x7f39181077d0 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391f1f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3918111c10 con 0x7f3918102180 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.602+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45418/0 (socket says 172.21.15.116:45418) 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 -- 172.21.15.116:0/4244616387 learned_addr learned my addr 172.21.15.116:0/4244616387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f3917fff640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3918102180 0x7f39181984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391d791640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39181077d0 0x7f391819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 -- 172.21.15.116:0/4244616387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3918102180 msgr2=0x7f39181984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3918102180 0x7f39181984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 -- 172.21.15.116:0/4244616387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39181077d0 msgr2=0x7f391819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39181077d0 0x7f391819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 -- 172.21.15.116:0/4244616387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391819cf80 con 0x7f39181017c0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391cf90640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f390c00e9f0 tx=0x7f390c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f3915ffb640 1 -- 172.21.15.116:0/4244616387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c00cd80 con 0x7f39181017c0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f3915ffb640 1 -- 172.21.15.116:0/4244616387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f390c0108e0 con 0x7f39181017c0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391819d270 con 0x7f39181017c0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f3915ffb640 1 -- 172.21.15.116:0/4244616387 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c018920 con 0x7f39181017c0 2024-03-19T02:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.603+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39181a4b00 con 0x7f39181017c0 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.606+0000 7f3915ffb640 1 -- 172.21.15.116:0/4244616387 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f390c010a90 con 0x7f39181017c0 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.606+0000 7f3921de4740 1 Processor -- start 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.606+0000 7f3921de4740 1 -- start start 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef24e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef30320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ee88500 con 0x55bb3ef25350 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ee30be0 con 0x55bb3ef24a30 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3921de4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ee845a0 con 0x55bb3ef2df30 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3915ffb640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45422/0 (socket says 172.21.15.116:45422) 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391cf90640 1 -- 172.21.15.116:0/2314592364 learned_addr learned my addr 172.21.15.116:0/2314592364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef30320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391d791640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef24e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391d791640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3915ffb640 1 -- 172.21.15.116:0/4244616387 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f390c0cf750 con 0x7f39181017c0 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 -- 172.21.15.116:0/2314592364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 msgr2=0x55bb3ef24e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef24e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 -- 172.21.15.116:0/2314592364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 msgr2=0x55bb3ef2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 -- 172.21.15.116:0/2314592364 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3ee52000 con 0x55bb3ef2df30 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391cf90640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f391d791640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 0x7f39080a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3910004af0 tx=0x7f3910005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.607+0000 7f3917fff640 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef30320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3904009bf0 tx=0x7f3904011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60116ac22eabccdd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3902ffd640 1 -- 172.21.15.116:0/2314592364 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904004050 con 0x55bb3ef2df30 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3902ffd640 1 -- 172.21.15.116:0/2314592364 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39040054d0 con 0x55bb3ef2df30 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3902ffd640 1 -- 172.21.15.116:0/2314592364 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904004c60 con 0x55bb3ef2df30 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 -- 172.21.15.116:0/2314592364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 msgr2=0x55bb3ef30320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef30320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3904009bf0 tx=0x7f3904011810 comp rx=0 tx=0).stop 2024-03-19T02:29:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 -- 172.21.15.116:0/2314592364 shutdown_connections 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef30320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 --2- 172.21.15.116:0/2314592364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef24e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 -- 172.21.15.116:0/2314592364 >> 172.21.15.116:0/2314592364 conn(0x55bb3ef1f160 msgr2=0x55bb3ef1f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 -- 172.21.15.116:0/2314592364 shutdown_connections 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.608+0000 7f3921de4740 1 -- 172.21.15.116:0/2314592364 wait complete. 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 -- start start 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 Processor -- start 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef25350 0x55bb3ef38540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef32410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ef30d20 con 0x55bb3ef24a30 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ee44160 con 0x55bb3ef25350 2024-03-19T02:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb3ef30ea0 con 0x55bb3ef2df30 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3921de4740 10 client.0 _ll_register_callbacks cb 0x55bb3ee80a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f3917fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef25350 0x55bb3ef38540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f391cf90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45430/0 (socket says 172.21.15.116:45430) 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f391cf90640 1 -- 172.21.15.116:0/1381137761 learned_addr learned my addr 172.21.15.116:0/1381137761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.609+0000 7f391d791640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef32410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 -- 172.21.15.116:0/1381137761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 msgr2=0x55bb3ef32410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef32410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 -- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef25350 msgr2=0x55bb3ef38540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef25350 0x55bb3ef38540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 -- 172.21.15.116:0/1381137761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3ec667d0 con 0x55bb3ef24a30 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f391cf90640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f390c098260 tx=0x7f390c09f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcf2419fd5c1a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0860d0 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39040054d0 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f390c0cf390 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f390c086270 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 10 client.16116 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb3eb9c3f0 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39040054d0 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f390c0ccc80 con 0x55bb3ef24a30 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 10 client.16116 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.610+0000 7f3921de4740 20 client.16116 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3ef424c0 0x55bb3ef44930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f3921de4740 4 client.16116 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb3ee95a40 con 0x55bb3ef424c0 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f3917fff640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3ef424c0 0x55bb3ef44930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f3917fff640 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3ef424c0 0x55bb3ef44930 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f406f1d517c5b304 server_cookie=5289242489247b4f in_seq=0 out_seq=0 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.611+0000 7f38f7fff640 10 client.16116 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f38f7fff640 1 -- 172.21.15.116:0/1381137761 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb3ee95a40 con 0x55bb3ef424c0 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f38f7fff640 10 client.16116 handle_command_reply: tid=0 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 client.16116 shutdown 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 msgr2=0x55bb3ef38000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f390c098260 tx=0x7f390c09f9f0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 shutdown_connections 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3ef424c0 0x55bb3ef44930 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef32410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef25350 0x55bb3ef38540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 --2- 172.21.15.116:0/1381137761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38000 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f38f7fff640 0 client.16116 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 >> 172.21.15.116:0/1381137761 conn(0x55bb3ef1f160 msgr2=0x55bb3ef20eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 shutdown_connections 2024-03-19T02:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 1 -- 172.21.15.116:0/1381137761 wait complete. 2024-03-19T02:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.612+0000 7f3921de4740 20 client.16116 trim_cache size 0 max 16384 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 Processor -- start 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 -- start start 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef2c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39181ac970 con 0x55bb3ef25350 2024-03-19T02:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39181ae4e0 con 0x55bb3ef24a30 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3921de4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39181ad040 con 0x55bb3ef2df30 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45434/0 (socket says 172.21.15.116:45434) 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 -- 172.21.15.116:0/2974095534 learned_addr learned my addr 172.21.15.116:0/2974095534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f391cf90640 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f391d791640 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef2c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 -- 172.21.15.116:0/2974095534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 msgr2=0x55bb3ef2c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef2c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 -- 172.21.15.116:0/2974095534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 msgr2=0x55bb3ef38260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 -- 172.21.15.116:0/2974095534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3eb5c0f0 con 0x55bb3ef25350 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.632+0000 7f3917fff640 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55bb3ef202f0 tx=0x7f3904014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8282773ffcbe89ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f39027fc640 1 -- 172.21.15.116:0/2974095534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904034070 con 0x55bb3ef25350 2024-03-19T02:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f39027fc640 1 -- 172.21.15.116:0/2974095534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3904004050 con 0x55bb3ef25350 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f39027fc640 1 -- 172.21.15.116:0/2974095534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390402e760 con 0x55bb3ef25350 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 -- 172.21.15.116:0/2974095534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 msgr2=0x55bb3ef4a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55bb3ef202f0 tx=0x7f3904014000 comp rx=0 tx=0).stop 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 -- 172.21.15.116:0/2974095534 shutdown_connections 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef2df30 0x55bb3ef2c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef25350 0x55bb3ef4a1d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 --2- 172.21.15.116:0/2974095534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef38260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 -- 172.21.15.116:0/2974095534 >> 172.21.15.116:0/2974095534 conn(0x55bb3ef1f160 msgr2=0x55bb3ef37b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 -- 172.21.15.116:0/2974095534 shutdown_connections 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.633+0000 7f3921de4740 1 -- 172.21.15.116:0/2974095534 wait complete. 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 -- start start 2024-03-19T02:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 Processor -- start 2024-03-19T02:29:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 0x55bb3ef3c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39181adde0 con 0x55bb3ef2df30 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39181ad710 con 0x55bb3ef24a30 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39181aa950 con 0x55bb3ef25350 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3921de4740 10 client.0 _ll_register_callbacks cb 0x7f390c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f391d791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f391cf90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f391d791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45446/0 (socket says 172.21.15.116:45446) 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f3917fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 0x55bb3ef3c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f391d791640 1 -- 172.21.15.116:0/2374783980 learned_addr learned my addr 172.21.15.116:0/2374783980 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.634+0000 7f391cf90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41078/0 (socket says 172.21.15.116:41078) 2024-03-19T02:29:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 -- 172.21.15.116:0/2374783980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 msgr2=0x55bb3ef3c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 0x55bb3ef3c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 -- 172.21.15.116:0/2374783980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 msgr2=0x55bb3ef3c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 -- 172.21.15.116:0/2374783980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3efa7a10 con 0x55bb3ef24a30 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391d791640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f3917fff640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 0x55bb3ef3c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f391cf90640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f390c099b40 tx=0x7f390c09b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762cf6142aed60e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f38f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0b3070 con 0x55bb3ef24a30 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f390c0cf390 con 0x55bb3ef24a30 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3904004050 con 0x55bb3ef24a30 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f390c0860d0 con 0x55bb3ef24a30 2024-03-19T02:29:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f3921de4740 10 client.25952 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb3efa8d00 con 0x55bb3ef24a30 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.635+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3904004050 con 0x55bb3ef24a30 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f390c0a7020 con 0x55bb3ef24a30 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 10 client.25952 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 10 client.25952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 20 client.25952 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3efa9450 0x55bb3efab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 4 client.25952 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bb3ee8fd90 con 0x55bb3efa9450 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3917fff640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3efa9450 0x55bb3efab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f38f7fff640 10 client.25952 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.636+0000 7f3917fff640 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3efa9450 0x55bb3efab870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a52efff0cdf88d server_cookie=e06c4d3ecad907e7 in_seq=0 out_seq=0 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f38f7fff640 1 -- 172.21.15.116:0/2374783980 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bb3ee8fd90 con 0x55bb3efa9450 2024-03-19T02:29:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f38f7fff640 10 client.25952 handle_command_reply: tid=0 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 client.25952 shutdown 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 msgr2=0x55bb3ef3bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f390c099b40 tx=0x7f390c09b660 comp rx=0 tx=0).stop 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 shutdown_connections 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3efa9450 0x55bb3efab870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3ef2df30 0x55bb3ef3c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3ef25350 0x55bb3ef3c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f38f7fff640 0 client.25952 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 --2- 172.21.15.116:0/2374783980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3ef24a30 0x55bb3ef3bee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 >> 172.21.15.116:0/2374783980 conn(0x55bb3ef1f160 msgr2=0x55bb3ef1d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.637+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 shutdown_connections 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f3921de4740 1 -- 172.21.15.116:0/2374783980 wait complete. 2024-03-19T02:29:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f3921de4740 20 client.25952 trim_cache size 0 max 16384 2024-03-19T02:29:16.645 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:16.645 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:29:16.645 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:29:16.645 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 msgr2=0x7f39080a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 0x7f39080a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3910004af0 tx=0x7f3910005c50 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 msgr2=0x7f3918197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f390c00e9f0 tx=0x7f390c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 shutdown_connections 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39181077d0 0x7f391819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3918102180 0x7f39181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 --2- 172.21.15.116:0/4244616387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39181017c0 0x7f3918197f90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 >> 172.21.15.116:0/4244616387 conn(0x7f39180fd2b0 msgr2=0x7f39180fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 shutdown_connections 2024-03-19T02:29:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.638+0000 7f391f1f4640 1 -- 172.21.15.116:0/4244616387 wait complete. 2024-03-19T02:29:16.650 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:29:16.651 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:29:16.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- 172.21.15.116:0/3030477794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 msgr2=0x7f7800106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 --2- 172.21.15.116:0/3030477794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f7800106b10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009890 tx=0x7f77fc02bb70 comp rx=0 tx=0).stop 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- 172.21.15.116:0/3030477794 shutdown_connections 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 --2- 172.21.15.116:0/3030477794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 0x7f780010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 --2- 172.21.15.116:0/3030477794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 0x7f7800100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 --2- 172.21.15.116:0/3030477794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f7800106b10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- 172.21.15.116:0/3030477794 >> 172.21.15.116:0/3030477794 conn(0x7f7800077820 msgr2=0x7f7800077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- 172.21.15.116:0/3030477794 shutdown_connections 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- 172.21.15.116:0/3030477794 wait complete. 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 Processor -- start 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.757+0000 7f7807880640 1 -- start start 2024-03-19T02:29:16.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 0x7f78001980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 0x7f78001985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78001109f0 con 0x7f7800106730 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7800110870 con 0x7f78001010e0 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7800110b70 con 0x7f7800100720 2024-03-19T02:29:16.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45458/0 (socket says 172.21.15.116:45458) 2024-03-19T02:29:16.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7804e1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 0x7f78001985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 -- 172.21.15.116:0/3497402585 learned_addr learned my addr 172.21.15.116:0/3497402585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:16.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f780561c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 0x7f78001980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 -- 172.21.15.116:0/3497402585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 msgr2=0x7f78001980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 0x7f78001980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 -- 172.21.15.116:0/3497402585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 msgr2=0x7f78001985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 0x7f78001985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 -- 172.21.15.116:0/3497402585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780019d090 con 0x7f7800106730 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7805e1d640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f77f800b970 tx=0x7f77f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8019070 con 0x7f7800106730 2024-03-19T02:29:16.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780019d380 con 0x7f7800106730 2024-03-19T02:29:16.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77f8014440 con 0x7f7800106730 2024-03-19T02:29:16.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f80075d0 con 0x7f7800106730 2024-03-19T02:29:16.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.758+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78001a4c10 con 0x7f7800106730 2024-03-19T02:29:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.760+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7800068610 con 0x7f7800106730 2024-03-19T02:29:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.760+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f77f801c070 con 0x7f7800106730 2024-03-19T02:29:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.761+0000 7f77f67fc640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 0x7f77ec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.761+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f77f80ce930 con 0x7f7800106730 2024-03-19T02:29:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.761+0000 7f780561c640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 0x7f77ec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:16.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.762+0000 7f780561c640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 0x7f77ec0a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000c00 tx=0x7f77fc005890 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:16.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:16.766+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77f809ab80 con 0x7f7800106730 2024-03-19T02:29:16.778 INFO:tasks.workunit.client.0.smithi116.stdout: 1 24630 3.85 MB/sec warmup 39 sec latency 16.486 ms 2024-03-19T02:29:16.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.000+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7800109cd0 con 0x7f7800106730 2024-03-19T02:29:17.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.000+0000 7f77f67fc640 1 -- 172.21.15.116:0/3497402585 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11951 (secure 0 0 0) 0x7f77f809ec20 con 0x7f7800106730 2024-03-19T02:29:17.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:29:17.000 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:28:39.411853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:29:17.000 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-19T02:29:17.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.002+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 msgr2=0x7f77ec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.002+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 0x7f77ec0a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000c00 tx=0x7f77fc005890 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 msgr2=0x7f780019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f77f800b970 tx=0x7f77f800be30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 shutdown_connections 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f77ec0a2a90 0x7f77ec0a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7800106730 0x7f780019c960 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78001010e0 0x7f78001985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 --2- 172.21.15.116:0/3497402585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7800100720 0x7f78001980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 >> 172.21.15.116:0/3497402585 conn(0x7f7800077820 msgr2=0x7f7800068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 shutdown_connections 2024-03-19T02:29:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.003+0000 7f7807880640 1 -- 172.21.15.116:0/3497402585 wait complete. 2024-03-19T02:29:17.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 -- 172.21.15.116:0/341844290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c105b50 msgr2=0x7fa70c105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 --2- 172.21.15.116:0/341844290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c105b50 0x7fa70c105fe0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa704009ca0 tx=0x7fa70402c180 comp rx=0 tx=0).stop 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 -- 172.21.15.116:0/341844290 shutdown_connections 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 --2- 172.21.15.116:0/341844290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c105b50 0x7fa70c105fe0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 --2- 172.21.15.116:0/341844290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c101150 0x7fa70c105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 --2- 172.21.15.116:0/341844290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 -- 172.21.15.116:0/341844290 >> 172.21.15.116:0/341844290 conn(0x7fa70c0778d0 msgr2=0x7fa70c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 -- 172.21.15.116:0/341844290 shutdown_connections 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.126+0000 7fa712953640 1 -- 172.21.15.116:0/341844290 wait complete. 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 Processor -- start 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 -- start start 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c105b50 0x7fa70c1a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c118020 con 0x7fa70c101150 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c117ea0 con 0x7fa70c100830 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa712953640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1181a0 con 0x7fa70c105b50 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45464/0 (socket says 172.21.15.116:45464) 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 -- 172.21.15.116:0/3890855984 learned_addr learned my addr 172.21.15.116:0/3890855984 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c105b50 0x7fa70c1a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 -- 172.21.15.116:0/3890855984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c105b50 msgr2=0x7fa70c1a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c105b50 0x7fa70c1a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 -- 172.21.15.116:0/3890855984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 msgr2=0x7fa70c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.127+0000 7fa70b7fe640 1 -- 172.21.15.116:0/3890855984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c1a3530 con 0x7fa70c101150 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa70000e9f0 tx=0x7fa70000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa7097fa640 1 -- 172.21.15.116:0/3890855984 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70000cd80 con 0x7fa70c101150 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70c1a3820 con 0x7fa70c101150 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa7097fa640 1 -- 172.21.15.116:0/3890855984 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7000054d0 con 0x7fa70c101150 2024-03-19T02:29:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa7097fa640 1 -- 172.21.15.116:0/3890855984 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700010490 con 0x7fa70c101150 2024-03-19T02:29:17.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.128+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70c1ab0b0 con 0x7fa70c101150 2024-03-19T02:29:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.130+0000 7fa7097fa640 1 -- 172.21.15.116:0/3890855984 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7fa700005020 con 0x7fa70c101150 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.130+0000 7fa71561d740 1 Processor -- start 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.130+0000 7fa71561d740 1 -- start start 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa7097fa640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 0x7fa6f40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3a5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c0b2ff4b0 con 0x558c0b3a57d0 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558c0b2a7d70 con 0x558c0b39c2d0 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa71561d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c0b2fb550 con 0x558c0b39cbf0 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa7097fa640 1 -- 172.21.15.116:0/3890855984 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa7000cf760 con 0x7fa70c101150 2024-03-19T02:29:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 0x7fa6f40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41142/0 (socket says 172.21.15.116:41142) 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 -- 172.21.15.116:0/3637341362 learned_addr learned my addr 172.21.15.116:0/3637341362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 -- 172.21.15.116:0/3637341362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 msgr2=0x558c0b3a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 -- 172.21.15.116:0/3637341362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 msgr2=0x558c0b3a7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3a7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 -- 172.21.15.116:0/3637341362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0b2ca210 con 0x558c0b39c2d0 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0115f0 tx=0x7fa6fc011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7014633554aac92d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 0x7fa6f40a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa704004af0 tx=0x7fa70402e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.131+0000 7fa6f27fc640 1 -- 172.21.15.116:0/3637341362 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc01e070 con 0x558c0b39c2d0 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa6f27fc640 1 -- 172.21.15.116:0/3637341362 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6fc00fa90 con 0x558c0b39c2d0 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa6f27fc640 1 -- 172.21.15.116:0/3637341362 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc019aa0 con 0x558c0b39c2d0 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- 172.21.15.116:0/3637341362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 msgr2=0x558c0b39c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0115f0 tx=0x7fa6fc011ab0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- 172.21.15.116:0/3637341362 shutdown_connections 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 --2- 172.21.15.116:0/3637341362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b39c6b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- 172.21.15.116:0/3637341362 >> 172.21.15.116:0/3637341362 conn(0x558c0b396a00 msgr2=0x558c0b396de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- 172.21.15.116:0/3637341362 shutdown_connections 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- 172.21.15.116:0/3637341362 wait complete. 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.132+0000 7fa71561d740 1 -- start start 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 Processor -- start 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39cbf0 0x558c0b3a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b3a3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c0b3a85c0 con 0x558c0b39c2d0 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558c0b2bb310 con 0x558c0b39cbf0 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c0b3a8740 con 0x558c0b3a57d0 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa71561d740 10 client.0 _ll_register_callbacks cb 0x558c0b2af2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45492/0 (socket says 172.21.15.116:45492) 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 -- 172.21.15.116:0/3174182136 learned_addr learned my addr 172.21.15.116:0/3174182136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39cbf0 0x558c0b3a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b3a3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 -- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 msgr2=0x558c0b3a3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b3a3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 -- 172.21.15.116:0/3174182136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39cbf0 msgr2=0x558c0b3a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39cbf0 0x558c0b3a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 -- 172.21.15.116:0/3174182136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c0b0133f0 con 0x558c0b39c2d0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.133+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00f8f0 tx=0x7fa6fc00f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7845a9f76a1e8ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc01e040 con 0x558c0b39c2d0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa6fc004030 con 0x558c0b39c2d0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6fc00fa90 con 0x558c0b39c2d0 2024-03-19T02:29:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa6fc02f050 con 0x558c0b39c2d0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 10 client.16140 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c0b0dd7d0 con 0x558c0b39c2d0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6fc00fa90 con 0x558c0b39c2d0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fa6fc02a030 con 0x558c0b39c2d0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 10 client.16140 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 10 client.16140 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 20 client.16140 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b3b59e0 0x558c0b3b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 4 client.16140 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c0b35d3e0 con 0x558c0b3b59e0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.134+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b3b59e0 0x558c0b3b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.135+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b3b59e0 0x558c0b3b7dd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97682b2c9236a736 server_cookie=505385b2f59097bc in_seq=0 out_seq=0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.135+0000 7fa6eb7fe640 10 client.16140 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.135+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/3174182136 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c0b35d3e0 con 0x558c0b3b59e0 2024-03-19T02:29:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.135+0000 7fa6eb7fe640 10 client.16140 handle_command_reply: tid=0 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.135+0000 7fa71561d740 1 client.16140 shutdown 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 msgr2=0x558c0b3a2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00f8f0 tx=0x7fa6fc00f920 comp rx=0 tx=0).stop 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 shutdown_connections 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b3b59e0 0x558c0b3b7dd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b3a3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa6eb7fe640 0 client.16140 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39cbf0 0x558c0b3a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 --2- 172.21.15.116:0/3174182136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39c2d0 0x558c0b3a2800 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 >> 172.21.15.116:0/3174182136 conn(0x558c0b396a00 msgr2=0x558c0b398750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 shutdown_connections 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 1 -- 172.21.15.116:0/3174182136 wait complete. 2024-03-19T02:29:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.136+0000 7fa71561d740 20 client.16140 trim_cache size 0 max 16384 2024-03-19T02:29:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 Processor -- start 2024-03-19T02:29:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 -- start start 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3ad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b50b0 con 0x558c0b3a57d0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b6c20 con 0x558c0b39c2d0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa71561d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b5780 con 0x558c0b39cbf0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41158/0 (socket says 172.21.15.116:41158) 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 -- 172.21.15.116:0/3196129173 learned_addr learned my addr 172.21.15.116:0/3196129173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa710ef0640 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 -- 172.21.15.116:0/3196129173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 msgr2=0x558c0b3bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3bd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 -- 172.21.15.116:0/3196129173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 msgr2=0x558c0b3ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3ad980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.149+0000 7fa70bfff640 1 -- 172.21.15.116:0/3196129173 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c0b39bcd0 con 0x558c0b39c2d0 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa70bfff640 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc019710 tx=0x7fa6fc012fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=775d5d51c7bfb263 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa6f1ffb640 1 -- 172.21.15.116:0/3196129173 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc01e070 con 0x558c0b39c2d0 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa6f1ffb640 1 -- 172.21.15.116:0/3196129173 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6fc0194d0 con 0x558c0b39c2d0 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa6f1ffb640 1 -- 172.21.15.116:0/3196129173 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0197b0 con 0x558c0b39c2d0 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 -- 172.21.15.116:0/3196129173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 msgr2=0x558c0b3b1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc019710 tx=0x7fa6fc012fc0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 -- 172.21.15.116:0/3196129173 shutdown_connections 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b3a57d0 0x558c0b3ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b39cbf0 0x558c0b3bd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 --2- 172.21.15.116:0/3196129173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3b1360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 -- 172.21.15.116:0/3196129173 >> 172.21.15.116:0/3196129173 conn(0x558c0b396a00 msgr2=0x558c0b395ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 -- 172.21.15.116:0/3196129173 shutdown_connections 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.150+0000 7fa71561d740 1 -- 172.21.15.116:0/3196129173 wait complete. 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 -- start start 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 Processor -- start 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3aab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b413be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b6520 con 0x558c0b39cbf0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b5e50 con 0x558c0b39c2d0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1b3090 con 0x558c0b3a57d0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa71561d740 10 client.0 _ll_register_callbacks cb 0x7fa6fc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45512/0 (socket says 172.21.15.116:45512) 2024-03-19T02:29:17.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70b7fe640 1 -- 172.21.15.116:0/541966556 learned_addr learned my addr 172.21.15.116:0/541966556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa710ef0640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b413be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 -- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 msgr2=0x558c0b413be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b413be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 -- 172.21.15.116:0/541966556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 msgr2=0x558c0b3ab080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70bfff640 1 -- 172.21.15.116:0/541966556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c0b4141d0 con 0x558c0b39c2d0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.151+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa70bfff640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3aab40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc003870 tx=0x7fa6fc009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca88afcdfd7c9bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc042060 con 0x558c0b39c2d0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6fc0194d0 con 0x558c0b39c2d0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa6fc005040 con 0x558c0b39c2d0 2024-03-19T02:29:17.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa6fc049050 con 0x558c0b39c2d0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 10 client.25973 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c0b41b550 con 0x558c0b39c2d0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6fc0194d0 con 0x558c0b39c2d0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fa6fc02a040 con 0x558c0b39c2d0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 10 client.25973 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 10 client.25973 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 20 client.25973 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b41bb60 0x558c0b41df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 4 client.25973 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.152+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c0b30c9f0 con 0x558c0b41bb60 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b41bb60 0x558c0b41df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa6eb7fe640 10 client.25973 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa70b7fe640 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b41bb60 0x558c0b41df50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=59ac23e50222752 server_cookie=e5b6da8f78b659bf in_seq=0 out_seq=0 2024-03-19T02:29:17.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa6eb7fe640 1 -- 172.21.15.116:0/541966556 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c0b30c9f0 con 0x558c0b41bb60 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa6eb7fe640 10 client.25973 handle_command_reply: tid=0 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 client.25973 shutdown 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 msgr2=0x558c0b3aab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3aab40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc003870 tx=0x7fa6fc009240 comp rx=0 tx=0).stop 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 shutdown_connections 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558c0b41bb60 0x558c0b41df50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c0b3a57d0 0x558c0b413be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c0b39cbf0 0x558c0b3ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa6eb7fe640 0 client.25973 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 --2- 172.21.15.116:0/541966556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c0b39c2d0 0x558c0b3aab40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 >> 172.21.15.116:0/541966556 conn(0x558c0b396a00 msgr2=0x558c0b397d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 shutdown_connections 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 1 -- 172.21.15.116:0/541966556 wait complete. 2024-03-19T02:29:17.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.153+0000 7fa71561d740 20 client.25973 trim_cache size 0 max 16384 2024-03-19T02:29:17.161 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:17.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 msgr2=0x7fa6f40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 0x7fa6f40a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa704004af0 tx=0x7fa70402e040 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 msgr2=0x7fa70c19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa70000e9f0 tx=0x7fa70000edf0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 shutdown_connections 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6f40a2a90 0x7fa6f40a4f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa70c105b50 0x7fa70c1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa70c101150 0x7fa70c19ea80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 --2- 172.21.15.116:0/3890855984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70c100830 0x7fa70c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 >> 172.21.15.116:0/3890855984 conn(0x7fa70c0778d0 msgr2=0x7fa70c1035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 shutdown_connections 2024-03-19T02:29:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.154+0000 7fa712953640 1 -- 172.21.15.116:0/3890855984 wait complete. 2024-03-19T02:29:17.164 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:29:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 -- 172.21.15.116:0/1954491308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 msgr2=0x7f96b810b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 --2- 172.21.15.116:0/1954491308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b810b840 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009900 tx=0x7f96a002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 -- 172.21.15.116:0/1954491308 shutdown_connections 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 --2- 172.21.15.116:0/1954491308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b810bd80 0x7f96b810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 --2- 172.21.15.116:0/1954491308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b810b840 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.275+0000 7f96be619640 1 --2- 172.21.15.116:0/1954491308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b8102a40 0x7f96b8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- 172.21.15.116:0/1954491308 >> 172.21.15.116:0/1954491308 conn(0x7f96b80778d0 msgr2=0x7f96b8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- 172.21.15.116:0/1954491308 shutdown_connections 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- 172.21.15.116:0/1954491308 wait complete. 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 Processor -- start 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- start start 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 0x7f96b8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 0x7f96b819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96b8111a30 con 0x7f96b8103360 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81118b0 con 0x7f96b8102a40 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.276+0000 7f96be619640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96b8111bb0 con 0x7f96b810bd80 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45534/0 (socket says 172.21.15.116:45534) 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 -- 172.21.15.116:0/3970305457 learned_addr learned my addr 172.21.15.116:0/3970305457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96b7fff640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 0x7f96b8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 0x7f96b819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 -- 172.21.15.116:0/3970305457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 msgr2=0x7f96b819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 0x7f96b819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 -- 172.21.15.116:0/3970305457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 msgr2=0x7f96b8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 0x7f96b8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 -- 172.21.15.116:0/3970305457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b819cf40 con 0x7f96b8103360 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96b7fff640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 0x7f96b8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 0x7f96b819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96affff640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f96a0005940 tx=0x7f96a002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96b5ffb640 1 -- 172.21.15.116:0/3970305457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0043020 con 0x7f96b8103360 2024-03-19T02:29:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96b5ffb640 1 -- 172.21.15.116:0/3970305457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96a0004e00 con 0x7f96b8103360 2024-03-19T02:29:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96b5ffb640 1 -- 172.21.15.116:0/3970305457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0003970 con 0x7f96b8103360 2024-03-19T02:29:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b819d1d0 con 0x7f96b8103360 2024-03-19T02:29:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.277+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b81a4ac0 con 0x7f96b8103360 2024-03-19T02:29:17.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.280+0000 7f96b5ffb640 1 -- 172.21.15.116:0/3970305457 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f96a0046020 con 0x7f96b8103360 2024-03-19T02:29:17.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.280+0000 7f96c1208740 1 Processor -- start 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 -- start start 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x56423808c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564237fe24b0 con 0x56423808cbe0 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564237f8ad70 con 0x56423807ef80 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96c1208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564237fde550 con 0x56423807f8a0 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41200/0 (socket says 172.21.15.116:41200) 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 -- 172.21.15.116:0/2954863213 learned_addr learned my addr 172.21.15.116:0/2954863213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96b7fff640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x56423808c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96affff640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96b5ffb640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 0x7f96a40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 -- 172.21.15.116:0/2954863213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 msgr2=0x56423808c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x56423808c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 -- 172.21.15.116:0/2954863213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 msgr2=0x56423808efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 -- 172.21.15.116:0/2954863213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564237fad210 con 0x56423807ef80 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96b7fff640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x56423808c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 0x7f96a40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.281+0000 7f96b5ffb640 1 -- 172.21.15.116:0/3970305457 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f96a00f12c0 con 0x7f96b8103360 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96b00118d0 tx=0x7f96b0011d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5750b3860932d6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96ae7fc640 1 -- 172.21.15.116:0/2954863213 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0017050 con 0x56423807ef80 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 0x7f96a40a4fd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96b0023000 tx=0x7f96b0024000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96ae7fc640 1 -- 172.21.15.116:0/2954863213 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b000e3b0 con 0x56423807ef80 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96ae7fc640 1 -- 172.21.15.116:0/2954863213 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b001a5c0 con 0x56423807ef80 2024-03-19T02:29:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96c1208740 1 -- 172.21.15.116:0/2954863213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 msgr2=0x56423807f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.282+0000 7f96c1208740 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96b00118d0 tx=0x7f96b0011d90 comp rx=0 tx=0).stop 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- 172.21.15.116:0/2954863213 shutdown_connections 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x56423808c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- 172.21.15.116:0/2954863213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807ef80 0x56423807f360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- 172.21.15.116:0/2954863213 >> 172.21.15.116:0/2954863213 conn(0x564238079a00 msgr2=0x564238079de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- 172.21.15.116:0/2954863213 shutdown_connections 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- 172.21.15.116:0/2954863213 wait complete. 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- start start 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 Processor -- start 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238089bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x56423808a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56423808fa20 con 0x56423807f8a0 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564237f9e310 con 0x56423808cbe0 2024-03-19T02:29:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56423808fba0 con 0x56423807ef80 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.283+0000 7f96c1208740 10 client.0 _ll_register_callbacks cb 0x564237f922d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96b7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238089bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x56423808a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45552/0 (socket says 172.21.15.116:45552) 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96affff640 1 -- 172.21.15.116:0/3269847915 learned_addr learned my addr 172.21.15.116:0/3269847915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 -- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 msgr2=0x564238089bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238089bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 -- 172.21.15.116:0/3269847915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 msgr2=0x56423808a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 -- 172.21.15.116:0/3269847915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564237cf63f0 con 0x56423808cbe0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96b7fff640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238089bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96affff640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x56423808a670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96b00118d0 tx=0x7f96b000e790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83c668569ef6162 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96977fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b002b040 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f96b0004c40 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b000e3b0 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96b003a050 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96c1208740 10 client.25985 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.284+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564237dc07d0 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b000e3b0 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f96b001a7f0 con 0x56423808cbe0 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 10 client.25985 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 10 client.25985 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 20 client.25985 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56423809cfc0 0x56423809f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 4 client.25985 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642380403e0 con 0x56423809cfc0 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96b7fff640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56423809cfc0 0x56423809f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96b7fff640 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56423809cfc0 0x56423809f3b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5ac5d743fa59695d server_cookie=6328fe30a3edf97a in_seq=0 out_seq=0 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.285+0000 7f96977fe640 10 client.25985 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.286+0000 7f96977fe640 1 -- 172.21.15.116:0/3269847915 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642380403e0 con 0x56423809cfc0 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.286+0000 7f96977fe640 10 client.25985 handle_command_reply: tid=0 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.286+0000 7f96c1208740 1 client.25985 shutdown 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 msgr2=0x56423808a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x56423808a670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96b00118d0 tx=0x7f96b000e790 comp rx=0 tx=0).stop 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 shutdown_connections 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x56423808a670 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56423809cfc0 0x56423809f3b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807f8a0 0x56423808a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 --2- 172.21.15.116:0/3269847915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238089bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 >> 172.21.15.116:0/3269847915 conn(0x564238079a00 msgr2=0x56423807b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96977fe640 0 client.25985 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 shutdown_connections 2024-03-19T02:29:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 1 -- 172.21.15.116:0/3269847915 wait complete. 2024-03-19T02:29:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.287+0000 7f96c1208740 20 client.25985 trim_cache size 0 max 16384 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.300+0000 7f96c1208740 1 Processor -- start 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.300+0000 7f96c1208740 1 -- start start 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238088d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807f8a0 0x564238098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81ac810 con 0x56423808cbe0 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81ae380 con 0x56423807f8a0 2024-03-19T02:29:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96c1208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81acee0 con 0x56423807ef80 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45556/0 (socket says 172.21.15.116:45556) 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96affff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807f8a0 0x564238098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 -- 172.21.15.116:0/1640372808 learned_addr learned my addr 172.21.15.116:0/1640372808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96b7fff640 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238088d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 -- 172.21.15.116:0/1640372808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 msgr2=0x564238088d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238088d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 -- 172.21.15.116:0/1640372808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807f8a0 msgr2=0x564238098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807f8a0 0x564238098860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 -- 172.21.15.116:0/1640372808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564238079530 con 0x56423808cbe0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009690 tx=0x7f96b0005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981c3da16bb5dcc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96adffb640 1 -- 172.21.15.116:0/1640372808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0043070 con 0x56423808cbe0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96adffb640 1 -- 172.21.15.116:0/1640372808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b000e3b0 con 0x56423808cbe0 2024-03-19T02:29:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.301+0000 7f96adffb640 1 -- 172.21.15.116:0/1640372808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0017070 con 0x56423808cbe0 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- 172.21.15.116:0/1640372808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 msgr2=0x56423808a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009690 tx=0x7f96b0005fb0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- 172.21.15.116:0/1640372808 shutdown_connections 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423808cbe0 0x56423808a490 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423807f8a0 0x564238098860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 --2- 172.21.15.116:0/1640372808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807ef80 0x564238088d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- 172.21.15.116:0/1640372808 >> 172.21.15.116:0/1640372808 conn(0x564238079a00 msgr2=0x56423807ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- 172.21.15.116:0/1640372808 shutdown_connections 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- 172.21.15.116:0/1640372808 wait complete. 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.302+0000 7f96c1208740 1 -- start start 2024-03-19T02:29:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 Processor -- start 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x564238090360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x5642380908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81adc80 con 0x56423807ef80 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81ad5b0 con 0x56423808cbe0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81aa7f0 con 0x56423807f8a0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96c1208740 10 client.0 _ll_register_callbacks cb 0x7f96a00b9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45562/0 (socket says 172.21.15.116:45562) 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 -- 172.21.15.116:0/1760022041 learned_addr learned my addr 172.21.15.116:0/1760022041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96bcbb6640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x5642380908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96affff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x564238090360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 -- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 msgr2=0x564238090360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x564238090360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 -- 172.21.15.116:0/1760022041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 msgr2=0x5642380908a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x5642380908a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 -- 172.21.15.116:0/1760022041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642381013e0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96b7fff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x56423809fb10 tx=0x7f96a8012ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d68930d237be13a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.303+0000 7f96977fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8020c50 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f96a8020df0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b000e3b0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96a8032050 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 10 client.16164 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5642381026e0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b000e3b0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f96a80035f0 con 0x56423807ef80 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 10 client.16164 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 10 client.16164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 20 client.16164 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564238102e40 0x564238105260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 4 client.16164 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564237fef9f0 con 0x564238102e40 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.304+0000 7f96affff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564238102e40 0x564238105260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96977fe640 10 client.16164 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96affff640 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564238102e40 0x564238105260 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf900cb1b625e534 server_cookie=b9ebc65ee4acc05c in_seq=0 out_seq=0 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96977fe640 1 -- 172.21.15.116:0/1760022041 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564237fef9f0 con 0x564238102e40 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96977fe640 10 client.16164 handle_command_reply: tid=0 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 client.16164 shutdown 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 msgr2=0x56423808fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x56423809fb10 tx=0x7f96a8012ed0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 shutdown_connections 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56423808cbe0 0x5642380908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56423807f8a0 0x564238090360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564238102e40 0x564238105260 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 --2- 172.21.15.116:0/1760022041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56423807ef80 0x56423808fe20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 >> 172.21.15.116:0/1760022041 conn(0x564238079a00 msgr2=0x56423807ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96977fe640 0 client.16164 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 shutdown_connections 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 1 -- 172.21.15.116:0/1760022041 wait complete. 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.305+0000 7f96c1208740 20 client.16164 trim_cache size 0 max 16384 2024-03-19T02:29:17.311 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 msgr2=0x7f96a40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 0x7f96a40a4fd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96b0023000 tx=0x7f96b0024000 comp rx=0 tx=0).stop 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 msgr2=0x7f96b8198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f96a0005940 tx=0x7f96a002c690 comp rx=0 tx=0).stop 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 shutdown_connections 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96a40a2b10 0x7f96a40a4fd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96b810bd80 0x7f96b819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96b8103360 0x7f96b8198490 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 --2- 172.21.15.116:0/3970305457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96b8102a40 0x7f96b8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 >> 172.21.15.116:0/3970305457 conn(0x7f96b80778d0 msgr2=0x7f96b80fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 shutdown_connections 2024-03-19T02:29:17.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.306+0000 7f96be619640 1 -- 172.21.15.116:0/3970305457 wait complete. 2024-03-19T02:29:17.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- 172.21.15.116:0/3102126742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 msgr2=0x7f08bc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 --2- 172.21.15.116:0/3102126742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc100b20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009930 tx=0x7f08ac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- 172.21.15.116:0/3102126742 shutdown_connections 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 --2- 172.21.15.116:0/3102126742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc101060 0x7f08bc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 --2- 172.21.15.116:0/3102126742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc100b20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 --2- 172.21.15.116:0/3102126742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc1066b0 0x7f08bc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- 172.21.15.116:0/3102126742 >> 172.21.15.116:0/3102126742 conn(0x7f08bc0778d0 msgr2=0x7f08bc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- 172.21.15.116:0/3102126742 shutdown_connections 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- 172.21.15.116:0/3102126742 wait complete. 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 Processor -- start 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.430+0000 7f08c3297640 1 -- start start 2024-03-19T02:29:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc101060 0x7f08bc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc1066b0 0x7f08bc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc110930 con 0x7f08bc1006a0 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1107b0 con 0x7f08bc1066b0 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc110ab0 con 0x7f08bc101060 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45590/0 (socket says 172.21.15.116:45590) 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 -- 172.21.15.116:0/95904230 learned_addr learned my addr 172.21.15.116:0/95904230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1834640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc1066b0 0x7f08bc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c0832640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc101060 0x7f08bc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 -- 172.21.15.116:0/95904230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc101060 msgr2=0x7f08bc199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc101060 0x7f08bc199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 -- 172.21.15.116:0/95904230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc1066b0 msgr2=0x7f08bc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc1066b0 0x7f08bc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 -- 172.21.15.116:0/95904230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc1a43b0 con 0x7f08bc1006a0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c1033640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f08b800e9f0 tx=0x7f08b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08b27fc640 1 -- 172.21.15.116:0/95904230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b800cd80 con 0x7f08bc1006a0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc1a4580 con 0x7f08bc1006a0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08b27fc640 1 -- 172.21.15.116:0/95904230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b80108e0 con 0x7f08bc1006a0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08b27fc640 1 -- 172.21.15.116:0/95904230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8018920 con 0x7f08bc1006a0 2024-03-19T02:29:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.431+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08bc1a4af0 con 0x7f08bc1006a0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.433+0000 7f08b27fc640 1 -- 172.21.15.116:0/95904230 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f08b8010a90 con 0x7f08bc1006a0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 Processor -- start 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 -- start start 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08b27fc640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 0x7f08a80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498ccabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563498c224b0 con 0x563498cc87d0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563498bcad70 con 0x563498cbf2d0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c5e82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563498c1e550 con 0x563498cbfbf0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08c1834640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 0x7f08a80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.434+0000 7f08b27fc640 1 -- 172.21.15.116:0/95904230 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f08b80cf750 con 0x7f08bc1006a0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41280/0 (socket says 172.21.15.116:41280) 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c0832640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 -- 172.21.15.116:0/615100536 learned_addr learned my addr 172.21.15.116:0/615100536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1834640 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498ccabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 -- 172.21.15.116:0/615100536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 msgr2=0x563498cc8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 -- 172.21.15.116:0/615100536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 msgr2=0x563498ccabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498ccabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 -- 172.21.15.116:0/615100536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563498bed210 con 0x563498cbf2d0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1834640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 0x7f08a80a4f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08b4005fd0 tx=0x7f08b4013b60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08c1033640 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08b800c6a0 tx=0x7f08b809b320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a8e5b7f260fafc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08a77fe640 1 -- 172.21.15.116:0/615100536 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8078b60 con 0x563498cbf2d0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08a77fe640 1 -- 172.21.15.116:0/615100536 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b809b890 con 0x563498cbf2d0 2024-03-19T02:29:17.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.435+0000 7f08a77fe640 1 -- 172.21.15.116:0/615100536 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b807dd80 con 0x563498cbf2d0 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- 172.21.15.116:0/615100536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 msgr2=0x563498cbf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08b800c6a0 tx=0x7f08b809b320 comp rx=0 tx=0).stop 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- 172.21.15.116:0/615100536 shutdown_connections 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498ccabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 --2- 172.21.15.116:0/615100536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cbf6b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- 172.21.15.116:0/615100536 >> 172.21.15.116:0/615100536 conn(0x563498cb9a00 msgr2=0x563498cb9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- 172.21.15.116:0/615100536 shutdown_connections 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- 172.21.15.116:0/615100536 wait complete. 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 -- start start 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.436+0000 7f08c5e82740 1 Processor -- start 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563498ccb5c0 con 0x563498cc87d0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563498bde310 con 0x563498cbf2d0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563498ccb740 con 0x563498cbfbf0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c5e82740 10 client.0 _ll_register_callbacks cb 0x563498bd22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c0832640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45602/0 (socket says 172.21.15.116:45602) 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41288/0 (socket says 172.21.15.116:41288) 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 -- 172.21.15.116:0/2427868599 learned_addr learned my addr 172.21.15.116:0/2427868599 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 -- 172.21.15.116:0/2427868599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 msgr2=0x563498cc5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 -- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 msgr2=0x563498cc5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 -- 172.21.15.116:0/2427868599 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634989363f0 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08c1834640 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f08b400fe40 tx=0x7f08b4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150054f53d9286a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.437+0000 7f08a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4024030 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f08b4009040 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b809b890 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f08b4031050 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 10 client.16176 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563498a007d0 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b809b890 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f08b401f070 con 0x563498cc87d0 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 10 client.16176 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 10 client.16176 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 20 client.16176 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498cd8bd0 0x563498cdb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 4 client.16176 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563498c803e0 con 0x563498cd8bd0 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.438+0000 7f08c1033640 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498cd8bd0 0x563498cdb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.439+0000 7f08a4ff9640 10 client.16176 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.439+0000 7f08c1033640 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498cd8bd0 0x563498cdb020 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f78a1815b8420869 server_cookie=1e150bdf77292d9d in_seq=0 out_seq=0 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08a4ff9640 1 -- 172.21.15.116:0/2427868599 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563498c803e0 con 0x563498cd8bd0 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08a4ff9640 10 client.16176 handle_command_reply: tid=0 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 client.16176 shutdown 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 msgr2=0x563498cc62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f08b400fe40 tx=0x7f08b4004ab0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 shutdown_connections 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498cd8bd0 0x563498cdb020 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cc62b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08a4ff9640 0 client.16176 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:17.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 --2- 172.21.15.116:0/2427868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 >> 172.21.15.116:0/2427868599 conn(0x563498cb9a00 msgr2=0x563498cbb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 shutdown_connections 2024-03-19T02:29:17.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 1 -- 172.21.15.116:0/2427868599 wait complete. 2024-03-19T02:29:17.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.440+0000 7f08c5e82740 20 client.16176 trim_cache size 0 max 16384 2024-03-19T02:29:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.453+0000 7f08c5e82740 1 Processor -- start 2024-03-19T02:29:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 -- start start 2024-03-19T02:29:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cd4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1aeb20 con 0x563498cc87d0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1b0690 con 0x563498cbf2d0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c5e82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1af1f0 con 0x563498cbfbf0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c0832640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41294/0 (socket says 172.21.15.116:41294) 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 -- 172.21.15.116:0/3614086961 learned_addr learned my addr 172.21.15.116:0/3614086961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1834640 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cd4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 -- 172.21.15.116:0/3614086961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 msgr2=0x563498cc4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 -- 172.21.15.116:0/3614086961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 msgr2=0x563498cd4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cd4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.454+0000 7f08c1033640 1 -- 172.21.15.116:0/3614086961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563498c8f880 con 0x563498cbf2d0 2024-03-19T02:29:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.455+0000 7f08c1033640 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f08b809b590 tx=0x7f08b8097b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8542b7512919c4a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.455+0000 7f08a6ffd640 1 -- 172.21.15.116:0/3614086961 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b809f930 con 0x563498cbf2d0 2024-03-19T02:29:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.455+0000 7f08a6ffd640 1 -- 172.21.15.116:0/3614086961 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b809b890 con 0x563498cbf2d0 2024-03-19T02:29:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.455+0000 7f08a6ffd640 1 -- 172.21.15.116:0/3614086961 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8097f00 con 0x563498cbf2d0 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- 172.21.15.116:0/3614086961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 msgr2=0x563498cc7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f08b809b590 tx=0x7f08b8097b30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- 172.21.15.116:0/3614086961 shutdown_connections 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cc87d0 0x563498cd4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498cc4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- 172.21.15.116:0/3614086961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cbf2d0 0x563498cc7f60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- 172.21.15.116:0/3614086961 >> 172.21.15.116:0/3614086961 conn(0x563498cb9a00 msgr2=0x563498cb8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- 172.21.15.116:0/3614086961 shutdown_connections 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- 172.21.15.116:0/3614086961 wait complete. 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- start start 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 Processor -- start 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498ccbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cc87d0 0x563498ccc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1aff90 con 0x563498cbf2d0 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1af8c0 con 0x563498cc87d0 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc1acb00 con 0x563498cbfbf0 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.456+0000 7f08c5e82740 10 client.0 _ll_register_callbacks cb 0x7f08b4012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45628/0 (socket says 172.21.15.116:45628) 2024-03-19T02:29:17.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1834640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cc87d0 0x563498ccc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 -- 172.21.15.116:0/1159661831 learned_addr learned my addr 172.21.15.116:0/1159661831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c0832640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498ccbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 -- 172.21.15.116:0/1159661831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 msgr2=0x563498ccbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498ccbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 -- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cc87d0 msgr2=0x563498ccc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cc87d0 0x563498ccc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 -- 172.21.15.116:0/1159661831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563498cccb30 con 0x563498cbf2d0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c1033640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f08b807a010 tx=0x7f08b800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d365e9fe42364866 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b80b3030 con 0x563498cbf2d0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f08b8099b90 con 0x563498cbf2d0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b809b890 con 0x563498cbf2d0 2024-03-19T02:29:17.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f08b809f930 con 0x563498cbf2d0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c5e82740 10 client.16182 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563498d3ed90 con 0x563498cbf2d0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.457+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b809b890 con 0x563498cbf2d0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f08b80afce0 con 0x563498cbf2d0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 10 client.16182 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 10 client.16182 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 20 client.16182 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498d3f010 0x563498d41400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 4 client.16182 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563498c2f9f0 con 0x563498d3f010 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c0832640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498d3f010 0x563498d41400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08a4ff9640 10 client.16182 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c0832640 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498d3f010 0x563498d41400 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=934520d0f1c8d205 server_cookie=37ac847cdfaaff8a in_seq=0 out_seq=0 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08a4ff9640 1 -- 172.21.15.116:0/1159661831 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563498c2f9f0 con 0x563498d3f010 2024-03-19T02:29:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08a4ff9640 10 client.16182 handle_command_reply: tid=0 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.458+0000 7f08c5e82740 1 client.16182 shutdown 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 msgr2=0x563498cd0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f08b807a010 tx=0x7f08b800cf20 comp rx=0 tx=0).stop 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 shutdown_connections 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x563498d3f010 0x563498d41400 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563498cc87d0 0x563498ccc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08a4ff9640 0 client.16182 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563498cbfbf0 0x563498ccbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 --2- 172.21.15.116:0/1159661831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563498cbf2d0 0x563498cd0aa0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 >> 172.21.15.116:0/1159661831 conn(0x563498cb9a00 msgr2=0x563498cc57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 shutdown_connections 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 1 -- 172.21.15.116:0/1159661831 wait complete. 2024-03-19T02:29:17.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c5e82740 20 client.16182 trim_cache size 0 max 16384 2024-03-19T02:29:17.465 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:17.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 msgr2=0x7f08a80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 0x7f08a80a4f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08b4005fd0 tx=0x7f08b4013b60 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 msgr2=0x7f08bc198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.459+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f08b800e9f0 tx=0x7f08b800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 shutdown_connections 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08a80a2a90 0x7f08a80a4f50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08bc1066b0 0x7f08bc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc101060 0x7f08bc199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 --2- 172.21.15.116:0/95904230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08bc1006a0 0x7f08bc198df0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 >> 172.21.15.116:0/95904230 conn(0x7f08bc0778d0 msgr2=0x7f08bc102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 shutdown_connections 2024-03-19T02:29:17.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.460+0000 7f08c3297640 1 -- 172.21.15.116:0/95904230 wait complete. 2024-03-19T02:29:17.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2883461874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8107890 msgr2=0x7fb9c8107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2883461874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8107890 0x7fb9c8107c70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009860 tx=0x7fb9b802bb70 comp rx=0 tx=0).stop 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2883461874 shutdown_connections 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2883461874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c806a830 0x7fb9c810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2883461874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c8069e90 0x7fb9c806a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2883461874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8107890 0x7fb9c8107c70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2883461874 >> 172.21.15.116:0/2883461874 conn(0x7fb9c8077500 msgr2=0x7fb9c8077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2883461874 shutdown_connections 2024-03-19T02:29:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.586+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2883461874 wait complete. 2024-03-19T02:29:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 Processor -- start 2024-03-19T02:29:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 -- start start 2024-03-19T02:29:17.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 0x7fb9c8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c8107890 0x7fb9c819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8111a20 con 0x7fb9c8069e90 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81118a0 con 0x7fb9c8107890 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9ce37e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8111ba0 con 0x7fb9c806a830 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45646/0 (socket says 172.21.15.116:45646) 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 0x7fb9c8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9c7fff640 1 -- 172.21.15.116:0/2306501241 learned_addr learned my addr 172.21.15.116:0/2306501241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.587+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c8107890 0x7fb9c819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 -- 172.21.15.116:0/2306501241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 msgr2=0x7fb9c8198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 0x7fb9c8198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 -- 172.21.15.116:0/2306501241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c8107890 msgr2=0x7fb9c819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c8107890 0x7fb9c819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 -- 172.21.15.116:0/2306501241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c819cef0 con 0x7fb9c8069e90 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 0x7fb9c8198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8037040 tx=0x7fb9b8035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c57fa640 1 -- 172.21.15.116:0/2306501241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8041020 con 0x7fb9c8069e90 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c57fa640 1 -- 172.21.15.116:0/2306501241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9b802c9c0 con 0x7fb9c8069e90 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c819d180 con 0x7fb9c8069e90 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9c57fa640 1 -- 172.21.15.116:0/2306501241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8039040 con 0x7fb9c8069e90 2024-03-19T02:29:17.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.588+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c81a4a70 con 0x7fb9c8069e90 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.590+0000 7fb9c57fa640 1 -- 172.21.15.116:0/2306501241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7fb9b8044020 con 0x7fb9c8069e90 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 Processor -- start 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 -- start start 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 0x5631624e7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 0x5631624e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5631624414b0 con 0x5631624e77d0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5631623e9d70 con 0x5631624debf0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56316243d550 con 0x5631624de2d0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c57fa640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 0x7fb9b00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 0x5631624e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 0x5631624e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 0x5631624e9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45656/0 (socket says 172.21.15.116:45656) 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9cc91b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c77fe640 1 -- 172.21.15.116:0/1261323660 learned_addr learned my addr 172.21.15.116:0/1261323660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 0x7fb9b00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.591+0000 7fb9c57fa640 1 -- 172.21.15.116:0/2306501241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb9b80f0b70 con 0x7fb9c8069e90 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1261323660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 msgr2=0x5631624de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1261323660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 msgr2=0x5631624e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 0x5631624e9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1261323660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56316240c210 con 0x5631624debf0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 0x5631624e7290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80eded0 tx=0x7fb9b8005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6b3ab78a0f8c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 0x7fb9b00a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005c40 tx=0x7fb9c0009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9ae7fc640 1 -- 172.21.15.116:0/1261323660 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80bd410 con 0x5631624debf0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9ae7fc640 1 -- 172.21.15.116:0/1261323660 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9b80bcd10 con 0x5631624debf0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.592+0000 7fb9ae7fc640 1 -- 172.21.15.116:0/1261323660 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80035f0 con 0x5631624debf0 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1261323660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 msgr2=0x5631624e7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 0x5631624e7290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80eded0 tx=0x7fb9b8005dc0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1261323660 shutdown_connections 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624e77d0 0x5631624e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624debf0 0x5631624e7290 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1261323660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1261323660 >> 172.21.15.116:0/1261323660 conn(0x5631624d8a00 msgr2=0x5631624d8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1261323660 shutdown_connections 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1261323660 wait complete. 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 -- start start 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.593+0000 7fb9d0f69740 1 Processor -- start 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624e4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5631624ea380 con 0x5631624debf0 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5631623fd310 con 0x5631624e77d0 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5631624ea500 con 0x5631624de2d0 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 10 client.0 _ll_register_callbacks cb 0x5631623f12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45672/0 (socket says 172.21.15.116:45672) 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 -- 172.21.15.116:0/2565942322 learned_addr learned my addr 172.21.15.116:0/2565942322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 -- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 msgr2=0x5631624e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624e4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 -- 172.21.15.116:0/2565942322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 msgr2=0x5631624e5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 -- 172.21.15.116:0/2565942322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631621553f0 con 0x5631624debf0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc012d20 tx=0x7fb9bc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576592063d16e9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc010430 con 0x5631624debf0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb9bc0105d0 con 0x5631624debf0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.594+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b80bcd10 con 0x5631624debf0 2024-03-19T02:29:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9bc010810 con 0x5631624debf0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 10 client.16194 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56316221f7d0 con 0x5631624debf0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9b80bcd10 con 0x5631624debf0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb9bc01c790 con 0x5631624debf0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 10 client.16194 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 10 client.16194 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 20 client.16194 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5631624f78f0 0x5631624f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 4 client.16194 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56316249f3e0 con 0x5631624f78f0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.595+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5631624f78f0 0x5631624f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9a77fe640 10 client.16194 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:17.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5631624f78f0 0x5631624f9d20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd671679804f7b17 server_cookie=2773d3b12ccf6fee in_seq=0 out_seq=0 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9a77fe640 1 -- 172.21.15.116:0/2565942322 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56316249f3e0 con 0x5631624f78f0 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9a77fe640 10 client.16194 handle_command_reply: tid=0 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 client.16194 shutdown 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 msgr2=0x5631624e4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc012d20 tx=0x7fb9bc00f790 comp rx=0 tx=0).stop 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 shutdown_connections 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5631624f78f0 0x5631624f9d20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624debf0 0x5631624e4dc0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/2565942322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624de2d0 0x5631624e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.596+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 >> 172.21.15.116:0/2565942322 conn(0x5631624d8a00 msgr2=0x5631624da750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.597+0000 7fb9a77fe640 0 client.16194 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:17.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.597+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 shutdown_connections 2024-03-19T02:29:17.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.597+0000 7fb9d0f69740 1 -- 172.21.15.116:0/2565942322 wait complete. 2024-03-19T02:29:17.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.597+0000 7fb9d0f69740 20 client.16194 trim_cache size 0 max 16384 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 Processor -- start 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 -- start start 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624f3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81b0cc0 con 0x5631624de2d0 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81b2830 con 0x5631624e77d0 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81b1390 con 0x5631624debf0 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.611+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33964/0 (socket says 172.21.15.116:33964) 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1892830377 learned_addr learned my addr 172.21.15.116:0/1892830377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1892830377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 msgr2=0x5631624f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1892830377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 msgr2=0x5631624ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 -- 172.21.15.116:0/1892830377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631624dac40 con 0x5631624de2d0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bca10 tx=0x7fb9b802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1a22811f39c09b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9adffb640 1 -- 172.21.15.116:0/1892830377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80d9070 con 0x5631624de2d0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9adffb640 1 -- 172.21.15.116:0/1892830377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9b80bcd10 con 0x5631624de2d0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.612+0000 7fb9adffb640 1 -- 172.21.15.116:0/1892830377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80035f0 con 0x5631624de2d0 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1892830377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 msgr2=0x5631624e6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80bca10 tx=0x7fb9b802bb70 comp rx=0 tx=0).stop 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1892830377 shutdown_connections 2024-03-19T02:29:17.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/1892830377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624e6fb0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1892830377 >> 172.21.15.116:0/1892830377 conn(0x5631624d8a00 msgr2=0x5631624d8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1892830377 shutdown_connections 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- 172.21.15.116:0/1892830377 wait complete. 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 -- start start 2024-03-19T02:29:17.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.613+0000 7fb9d0f69740 1 Processor -- start 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624e1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81b2130 con 0x5631624de2d0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81b1a60 con 0x5631624e77d0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c81aeca0 con 0x5631624debf0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9d0f69740 10 client.0 _ll_register_callbacks cb 0x7fb9bc009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33978/0 (socket says 172.21.15.116:33978) 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 -- 172.21.15.116:0/778292253 learned_addr learned my addr 172.21.15.116:0/778292253 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624e1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9cc91b640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 -- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 msgr2=0x5631624e1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624e1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 -- 172.21.15.116:0/778292253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 msgr2=0x5631624e1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 -- 172.21.15.116:0/778292253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631624e1cb0 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9c7fff640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80c4f80 tx=0x7fb9b80bcae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeaade616d33f3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.614+0000 7fb9a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b80a2060 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9b80bcd10 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb9b80035f0 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9b80d9030 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 10 client.16200 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56316255da70 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9b80bcd10 con 0x5631624de2d0 2024-03-19T02:29:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb9b803c030 con 0x5631624de2d0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 10 client.16200 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 10 client.16200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 20 client.16200 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56316255dec0 0x5631625602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 4 client.16200 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56316244e9f0 con 0x56316255dec0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.615+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56316255dec0 0x5631625602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9a77fe640 10 client.16200 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9c77fe640 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56316255dec0 0x5631625602e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39d490ee7d9caf0f server_cookie=d97c7ea312cd73b1 in_seq=0 out_seq=0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9a77fe640 1 -- 172.21.15.116:0/778292253 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56316244e9f0 con 0x56316255dec0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9a77fe640 10 client.16200 handle_command_reply: tid=0 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 client.16200 shutdown 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 msgr2=0x5631624ef7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80c4f80 tx=0x7fb9b80bcae0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 shutdown_connections 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56316255dec0 0x5631625602e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631624e77d0 0x5631624e1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9a77fe640 0 client.16200 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631624debf0 0x5631624e1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 --2- 172.21.15.116:0/778292253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631624de2d0 0x5631624ef7f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 >> 172.21.15.116:0/778292253 conn(0x5631624d8a00 msgr2=0x5631624dfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 shutdown_connections 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 1 -- 172.21.15.116:0/778292253 wait complete. 2024-03-19T02:29:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.616+0000 7fb9d0f69740 20 client.16200 trim_cache size 0 max 16384 2024-03-19T02:29:17.623 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 msgr2=0x7fb9b00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 0x7fb9b00a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0005c40 tx=0x7fb9c0009500 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 msgr2=0x7fb9c8197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8037040 tx=0x7fb9b8035990 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 shutdown_connections 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb9b00a2a90 0x7fb9b00a4f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9c8107890 0x7fb9c819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9c806a830 0x7fb9c8198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 --2- 172.21.15.116:0/2306501241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9c8069e90 0x7fb9c8197f00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 >> 172.21.15.116:0/2306501241 conn(0x7fb9c8077500 msgr2=0x7fb9c80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 shutdown_connections 2024-03-19T02:29:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.617+0000 7fb9ce37e640 1 -- 172.21.15.116:0/2306501241 wait complete. 2024-03-19T02:29:17.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 -- 172.21.15.116:0/2192868039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001006b0 msgr2=0x7fec00100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 --2- 172.21.15.116:0/2192868039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001006b0 0x7fec00100b30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7febe8009930 tx=0x7febe802bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 -- 172.21.15.116:0/2192868039 shutdown_connections 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 --2- 172.21.15.116:0/2192868039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec0010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 --2- 172.21.15.116:0/2192868039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001006b0 0x7fec00100b30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 --2- 172.21.15.116:0/2192868039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001066c0 0x7fec00106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 -- 172.21.15.116:0/2192868039 >> 172.21.15.116:0/2192868039 conn(0x7fec00076aa0 msgr2=0x7fec00076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 -- 172.21.15.116:0/2192868039 shutdown_connections 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.742+0000 7fec05137640 1 -- 172.21.15.116:0/2192868039 wait complete. 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 Processor -- start 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 -- start start 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001006b0 0x7fec00197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec001984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 0x7fec0019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec00110980 con 0x7fec001066c0 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec00110800 con 0x7fec00101070 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7fec05137640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fec00110b00 con 0x7fec001006b0 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 0x7fec0019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec001984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 0x7fec0019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33994/0 (socket says 172.21.15.116:33994) 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febfed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001006b0 0x7fec00197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febff59e640 1 -- 172.21.15.116:0/1760504054 learned_addr learned my addr 172.21.15.116:0/1760504054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 -- 172.21.15.116:0/1760504054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001006b0 msgr2=0x7fec00197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001006b0 0x7fec00197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 -- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 msgr2=0x7fec0019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 0x7fec0019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.743+0000 7febf7fff640 1 -- 172.21.15.116:0/1760504054 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0019cf80 con 0x7fec00101070 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7febf7fff640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec001984d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7febe8009230 tx=0x7febe80045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7febfcd99640 1 -- 172.21.15.116:0/1760504054 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe804a070 con 0x7fec00101070 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0019d210 con 0x7fec00101070 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec000fdf00 con 0x7fec00101070 2024-03-19T02:29:17.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7febfcd99640 1 -- 172.21.15.116:0/1760504054 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe802ca70 con 0x7fec00101070 2024-03-19T02:29:17.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.744+0000 7febfcd99640 1 -- 172.21.15.116:0/1760504054 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe8003a10 con 0x7fec00101070 2024-03-19T02:29:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.746+0000 7fec07dfb740 1 Processor -- start 2024-03-19T02:29:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.746+0000 7fec07dfb740 1 -- start start 2024-03-19T02:29:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7820bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dc77784b0 con 0x564dc781e7d0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564dc7720d70 con 0x564dc78152d0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7fec07dfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564dc7774550 con 0x564dc7815bf0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febfcd99640 1 -- 172.21.15.116:0/1760504054 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7febe804d070 con 0x7fec00101070 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febfed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55342/0 (socket says 172.21.15.116:55342) 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 -- 172.21.15.116:0/3777655778 learned_addr learned my addr 172.21.15.116:0/3777655778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febfed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37896/0 (socket says 172.21.15.116:37896) 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febf7fff640 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7820bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 -- 172.21.15.116:0/3777655778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 msgr2=0x564dc781e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 -- 172.21.15.116:0/3777655778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 msgr2=0x564dc7820bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7820bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 -- 172.21.15.116:0/3777655778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dc7743210 con 0x564dc78152d0 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febfed9d640 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febff59e640 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7febf8011570 tx=0x7febf8011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50a6b11a9cfae73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.747+0000 7febfcd99640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 0x7febec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febf57fa640 1 -- 172.21.15.116:0/3777655778 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8004b20 con 0x564dc78152d0 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febfed9d640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 0x7febec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febfcd99640 1 -- 172.21.15.116:0/1760504054 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febe80f43a0 con 0x7fec00101070 2024-03-19T02:29:17.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febf57fa640 1 -- 172.21.15.116:0/3777655778 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febf8004cc0 con 0x564dc78152d0 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febf57fa640 1 -- 172.21.15.116:0/3777655778 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf80053e0 con 0x564dc78152d0 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 -- 172.21.15.116:0/3777655778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 msgr2=0x564dc78156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7febfed9d640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 0x7febec0a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x564dc7814da0 tx=0x7febf0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7febf8011570 tx=0x7febf8011a30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 -- 172.21.15.116:0/3777655778 shutdown_connections 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7820bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc781e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 --2- 172.21.15.116:0/3777655778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78156b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 -- 172.21.15.116:0/3777655778 >> 172.21.15.116:0/3777655778 conn(0x564dc780fa00 msgr2=0x564dc780fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 -- 172.21.15.116:0/3777655778 shutdown_connections 2024-03-19T02:29:17.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.748+0000 7fec07dfb740 1 -- 172.21.15.116:0/3777655778 wait complete. 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 -- start start 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 Processor -- start 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc781b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc781e7d0 0x564dc781c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564dc78215c0 con 0x564dc7815bf0 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564dc7734310 con 0x564dc78152d0 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564dc7821740 con 0x564dc781e7d0 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7fec07dfb740 10 client.0 _ll_register_callbacks cb 0x564dc77282d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7febf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7febf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34016/0 (socket says 172.21.15.116:34016) 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7febfed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc781b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7febf7fff640 1 -- 172.21.15.116:0/2074384107 learned_addr learned my addr 172.21.15.116:0/2074384107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.749+0000 7febff59e640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc781e7d0 0x564dc781c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 -- 172.21.15.116:0/2074384107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc781e7d0 msgr2=0x564dc781c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc781e7d0 0x564dc781c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 -- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 msgr2=0x564dc781b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc781b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 -- 172.21.15.116:0/2074384107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dc748c3f0 con 0x564dc7815bf0 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febfed9d640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc781b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febf7fff640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7febe802c170 tx=0x7febe802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c236c87fac728b0f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80bc9b0 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febe80c4540 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf8004cc0 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febe80c0070 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 10 client.16212 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564dc75567d0 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febf8004cc0 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7febe80c4150 con 0x564dc7815bf0 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 10 client.16212 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 20 client.16212 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc782e990 0x564dc7830e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 4 client.16212 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.750+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564dc77d63e0 con 0x564dc782e990 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.751+0000 7febfed9d640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc782e990 0x564dc7830e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.751+0000 7febe67fc640 10 client.16212 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.751+0000 7febfed9d640 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc782e990 0x564dc7830e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a52216b82d8fe68 server_cookie=24e6727b9732f359 in_seq=0 out_seq=0 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7febe67fc640 1 -- 172.21.15.116:0/2074384107 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564dc77d63e0 con 0x564dc782e990 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7febe67fc640 10 client.16212 handle_command_reply: tid=0 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 client.16212 shutdown 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 msgr2=0x564dc781bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7febe802c170 tx=0x7febe802bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:17.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 shutdown_connections 2024-03-19T02:29:17.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc782e990 0x564dc7830e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc781e7d0 0x564dc781c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc7815bf0 0x564dc781bbf0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7febe67fc640 0 client.16212 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2074384107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc781b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 >> 172.21.15.116:0/2074384107 conn(0x564dc780fa00 msgr2=0x564dc7811750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 shutdown_connections 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 1 -- 172.21.15.116:0/2074384107 wait complete. 2024-03-19T02:29:17.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.752+0000 7fec07dfb740 20 client.16212 trim_cache size 0 max 16384 2024-03-19T02:29:17.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.766+0000 7fec07dfb740 1 Processor -- start 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.766+0000 7fec07dfb740 1 -- start start 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.766+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.766+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc782a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc781e7d0 0x564dc7836720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7fec07dfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febe000fe20 con 0x564dc78152d0 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7fec07dfb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febe0011990 con 0x564dc781e7d0 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7fec07dfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febe00104f0 con 0x564dc7815bf0 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34026/0 (socket says 172.21.15.116:34026) 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 -- 172.21.15.116:0/4230409774 learned_addr learned my addr 172.21.15.116:0/4230409774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febf7fff640 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc782a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febff59e640 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc781e7d0 0x564dc7836720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 -- 172.21.15.116:0/4230409774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 msgr2=0x564dc782a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc782a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 -- 172.21.15.116:0/4230409774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc781e7d0 msgr2=0x564dc7836720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc781e7d0 0x564dc7836720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 -- 172.21.15.116:0/4230409774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dc7831490 con 0x564dc78152d0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febfed9d640 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x564dc78316c0 tx=0x7febf001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53784ebb3111cf45 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febf4ff9640 1 -- 172.21.15.116:0/4230409774 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0003db0 con 0x564dc78152d0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febf4ff9640 1 -- 172.21.15.116:0/4230409774 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febf002bb90 con 0x564dc78152d0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.767+0000 7febf4ff9640 1 -- 172.21.15.116:0/4230409774 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00047a0 con 0x564dc78152d0 2024-03-19T02:29:17.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- 172.21.15.116:0/4230409774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 msgr2=0x564dc781dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x564dc78316c0 tx=0x7febf001b000 comp rx=0 tx=0).stop 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- 172.21.15.116:0/4230409774 shutdown_connections 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc781e7d0 0x564dc7836720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc782a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 --2- 172.21.15.116:0/4230409774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc78152d0 0x564dc781dde0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- 172.21.15.116:0/4230409774 >> 172.21.15.116:0/4230409774 conn(0x564dc780fa00 msgr2=0x564dc7811670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- 172.21.15.116:0/4230409774 shutdown_connections 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- 172.21.15.116:0/4230409774 wait complete. 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 -- start start 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.768+0000 7fec07dfb740 1 Processor -- start 2024-03-19T02:29:17.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc7818f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febe0011290 con 0x564dc781e7d0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febe0010bc0 con 0x564dc78152d0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febe000de00 con 0x564dc7815bf0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7fec07dfb740 10 client.0 _ll_register_callbacks cb 0x7febe8005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febfed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febf7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc7818f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34034/0 (socket says 172.21.15.116:34034) 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 -- 172.21.15.116:0/2171214307 learned_addr learned my addr 172.21.15.116:0/2171214307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 -- 172.21.15.116:0/2171214307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 msgr2=0x564dc7818f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc7818f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 -- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 msgr2=0x564dc78189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78189c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 -- 172.21.15.116:0/2171214307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dc77e5880 con 0x564dc781e7d0 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febfed9d640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78189c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febff59e640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7febf8009cf0 tx=0x7febf8004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8622389948dcfc5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.769+0000 7febe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf802f060 con 0x564dc781e7d0 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febf800fb00 con 0x564dc781e7d0 2024-03-19T02:29:17.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf002bb90 con 0x564dc781e7d0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febf8036050 con 0x564dc781e7d0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 10 client.16218 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564dc780f830 con 0x564dc781e7d0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febf002bb90 con 0x564dc781e7d0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7febf80056f0 con 0x564dc781e7d0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 10 client.16218 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 20 client.16218 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc78958b0 0x564dc7897cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 4 client.16218 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564dc77859f0 con 0x564dc78958b0 2024-03-19T02:29:17.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.770+0000 7febfed9d640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc78958b0 0x564dc7897cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7febfed9d640 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc78958b0 0x564dc7897cd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fbf3f2cca9e154a server_cookie=485a64a7aba41fbb in_seq=0 out_seq=0 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7febe67fc640 10 client.16218 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7febe67fc640 1 -- 172.21.15.116:0/2171214307 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564dc77859f0 con 0x564dc78958b0 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7febe67fc640 10 client.16218 handle_command_reply: tid=0 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 client.16218 shutdown 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 msgr2=0x564dc7893d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7febf8009cf0 tx=0x7febf8004f90 comp rx=0 tx=0).stop 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 shutdown_connections 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564dc78958b0 0x564dc7897cd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564dc781e7d0 0x564dc7893d50 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7febe67fc640 0 client.16218 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564dc7815bf0 0x564dc7818f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 --2- 172.21.15.116:0/2171214307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dc78152d0 0x564dc78189c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 >> 172.21.15.116:0/2171214307 conn(0x564dc780fa00 msgr2=0x564dc780e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 shutdown_connections 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 1 -- 172.21.15.116:0/2171214307 wait complete. 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.771+0000 7fec07dfb740 20 client.16218 trim_cache size 0 max 16384 2024-03-19T02:29:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 msgr2=0x7febec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 0x7febec0a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x564dc7814da0 tx=0x7febf0005f90 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 msgr2=0x7fec001984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec001984d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7febe8009230 tx=0x7febe80045e0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 shutdown_connections 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febec0a2a90 0x7febec0a4f50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec001066c0 0x7fec0019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec00101070 0x7fec001984d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 --2- 172.21.15.116:0/1760504054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec001006b0 0x7fec00197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:17.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 >> 172.21.15.116:0/1760504054 conn(0x7fec00076aa0 msgr2=0x7fec0006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 shutdown_connections 2024-03-19T02:29:17.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:17.772+0000 7fec05137640 1 -- 172.21.15.116:0/1760504054 wait complete. 2024-03-19T02:29:17.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 25106 3.86 MB/sec warmup 40 sec latency 43.886 ms 2024-03-19T02:29:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:17 smithi116 ceph-mon[32263]: pgmap v47: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:18.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:17 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3497402585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:17 smithi162 ceph-mon[35761]: pgmap v47: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:17 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3497402585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:17 smithi154 ceph-mon[35693]: pgmap v47: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:17 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3497402585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:18.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 25531 3.78 MB/sec warmup 41 sec latency 45.611 ms 2024-03-19T02:29:18.786 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:29:18.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:29:18.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 -- 172.21.15.116:0/319788450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0069d90 msgr2=0x7f71c006a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 --2- 172.21.15.116:0/319788450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0069d90 0x7f71c006a210 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009930 tx=0x7f71a802bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 -- 172.21.15.116:0/319788450 shutdown_connections 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 --2- 172.21.15.116:0/319788450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c006a750 0x7f71c010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 --2- 172.21.15.116:0/319788450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0069d90 0x7f71c006a210 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 --2- 172.21.15.116:0/319788450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c01077c0 0x7f71c0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 -- 172.21.15.116:0/319788450 >> 172.21.15.116:0/319788450 conn(0x7f71c0077700 msgr2=0x7f71c0077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 -- 172.21.15.116:0/319788450 shutdown_connections 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.902+0000 7f71c531b640 1 -- 172.21.15.116:0/319788450 wait complete. 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 Processor -- start 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 -- start start 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c0069d90 0x7f71c0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 0x7f71c0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0111a80 con 0x7f71c01077c0 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0111900 con 0x7f71c006a750 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71c531b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0111c00 con 0x7f71c0069d90 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34052/0 (socket says 172.21.15.116:34052) 2024-03-19T02:29:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.903+0000 7f71bf7fe640 1 -- 172.21.15.116:0/2760138467 learned_addr learned my addr 172.21.15.116:0/2760138467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71beffd640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c0069d90 0x7f71c0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71b67fc640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 0x7f71c0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 -- 172.21.15.116:0/2760138467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c0069d90 msgr2=0x7f71c0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c0069d90 0x7f71c0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 -- 172.21.15.116:0/2760138467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 msgr2=0x7f71c0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 0x7f71c0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 -- 172.21.15.116:0/2760138467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c019cff0 con 0x7f71c01077c0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71b67fc640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 0x7f71c0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bf7fe640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f71b800eb10 tx=0x7f71b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bcff9640 1 -- 172.21.15.116:0/2760138467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8004b20 con 0x7f71c01077c0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c019d2e0 con 0x7f71c01077c0 2024-03-19T02:29:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bcff9640 1 -- 172.21.15.116:0/2760138467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b8004cc0 con 0x7f71c01077c0 2024-03-19T02:29:18.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71bcff9640 1 -- 172.21.15.116:0/2760138467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8010690 con 0x7f71c01077c0 2024-03-19T02:29:18.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.904+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01a4b70 con 0x7f71c01077c0 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.906+0000 7f71bcff9640 1 -- 172.21.15.116:0/2760138467 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f71b8005020 con 0x7f71c01077c0 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 Processor -- start 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 -- start start 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abc7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c52ab2b160 con 0x55c52abc7380 2024-03-19T02:29:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c52aad3e10 con 0x55c52abd0880 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c52ab27200 con 0x55c52abc7ca0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71bcff9640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 0x7f71ac0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abc7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abc7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34066/0 (socket says 172.21.15.116:34066) 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71b67fc640 1 -- 172.21.15.116:0/2922698449 learned_addr learned my addr 172.21.15.116:0/2922698449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71b67fc640 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.907+0000 7f71bf7fe640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 0x7f71ac0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 -- 172.21.15.116:0/2922698449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 msgr2=0x55c52abd2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71bcff9640 1 -- 172.21.15.116:0/2760138467 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71b80cfb60 con 0x7f71c01077c0 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 -- 172.21.15.116:0/2922698449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 msgr2=0x55c52abc7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abc7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 -- 172.21.15.116:0/2922698449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52a9077e0 con 0x55c52abc7ca0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71bf7fe640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 0x7f71ac0a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71b80cce10 tx=0x7f71b8097850 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71beffd640 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd0340 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009c20 tx=0x7f71b0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c8abee6dfcd1b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71b57fa640 1 -- 172.21.15.116:0/2922698449 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0004050 con 0x55c52abc7ca0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71b57fa640 1 -- 172.21.15.116:0/2922698449 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b00054d0 con 0x55c52abc7ca0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.908+0000 7f71b57fa640 1 -- 172.21.15.116:0/2922698449 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0004bc0 con 0x55c52abc7ca0 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- 172.21.15.116:0/2922698449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 msgr2=0x55c52abd0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd0340 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009c20 tx=0x7f71b0011810 comp rx=0 tx=0).stop 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- 172.21.15.116:0/2922698449 shutdown_connections 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd0340 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/2922698449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abc7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- 172.21.15.116:0/2922698449 >> 172.21.15.116:0/2922698449 conn(0x55c52abc1ab0 msgr2=0x55c52abc1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- 172.21.15.116:0/2922698449 shutdown_connections 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- 172.21.15.116:0/2922698449 wait complete. 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 -- start start 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.909+0000 7f71c7fe4740 1 Processor -- start 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 0x55c52abcdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c52abd3670 con 0x55c52abc7ca0 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c52aae71a0 con 0x55c52abd0880 2024-03-19T02:29:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c52abd37f0 con 0x55c52abc7380 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71c7fe4740 10 client.0 _ll_register_callbacks cb 0x55c52a83ec60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 0x55c52abcdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37952/0 (socket says 172.21.15.116:37952) 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 -- 172.21.15.116:0/3703031143 learned_addr learned my addr 172.21.15.116:0/3703031143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 -- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 msgr2=0x55c52abce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 -- 172.21.15.116:0/3703031143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 msgr2=0x55c52abcdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 0x55c52abcdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 -- 172.21.15.116:0/3703031143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52a967f70 con 0x55c52abc7380 2024-03-19T02:29:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71beffd640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 0x55c52abcdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71b67fc640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71a8005ab0 tx=0x7f71a8005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fad48ef546a05e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a802cc90 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.910+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71a8003800 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b00054d0 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71a8003a00 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 10 client.25993 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c52a7ff170 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b00054d0 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f71a8050c30 con 0x55c52abc7380 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 10 client.25993 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:18.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 10 client.25993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 20 client.25993 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52abe2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 4 client.25993 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c52ab386a0 con 0x55c52abe0b80 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.911+0000 7f71beffd640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52abe2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.912+0000 7f71a67fc640 10 client.25993 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.912+0000 7f71beffd640 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52abe2f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e3527334e6e0635 server_cookie=8c1754186ec933d3 in_seq=0 out_seq=0 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71a67fc640 1 -- 172.21.15.116:0/3703031143 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c52ab386a0 con 0x55c52abe0b80 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71a67fc640 10 client.25993 handle_command_reply: tid=0 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 client.25993 shutdown 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 msgr2=0x55c52abcd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71a8005ab0 tx=0x7f71a8005d90 comp rx=0 tx=0).stop 2024-03-19T02:29:18.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 shutdown_connections 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52abe2f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71a67fc640 0 client.25993 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7ca0 0x55c52abcdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/3703031143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7380 0x55c52abcd830 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 >> 172.21.15.116:0/3703031143 conn(0x55c52abc1ab0 msgr2=0x55c52abc3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 shutdown_connections 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 1 -- 172.21.15.116:0/3703031143 wait complete. 2024-03-19T02:29:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.913+0000 7f71c7fe4740 20 client.25993 trim_cache size 0 max 16384 2024-03-19T02:29:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 Processor -- start 2024-03-19T02:29:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 -- start start 2024-03-19T02:29:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 0x55c52abcff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 0x55c52abe8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01aec60 con 0x55c52abd0880 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b07d0 con 0x55c52abc7380 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.934+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01af330 con 0x55c52abc7ca0 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 0x55c52abe8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 0x55c52abe8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34092/0 (socket says 172.21.15.116:34092) 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37958/0 (socket says 172.21.15.116:37958) 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b67fc640 1 -- 172.21.15.116:0/1798503163 learned_addr learned my addr 172.21.15.116:0/1798503163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b67fc640 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 0x55c52abcff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 -- 172.21.15.116:0/1798503163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 msgr2=0x55c52abcff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 0x55c52abcff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 -- 172.21.15.116:0/1798503163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 msgr2=0x55c52abe8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 0x55c52abe8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 -- 172.21.15.116:0/1798503163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52abe32e0 con 0x55c52abc7ca0 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b67fc640 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 0x55c52abcff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71beffd640 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71b001b040 tx=0x7f71b0009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=734b0186b5784488 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b4ff9640 1 -- 172.21.15.116:0/1798503163 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00097a0 con 0x55c52abc7ca0 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b4ff9640 1 -- 172.21.15.116:0/1798503163 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b0004360 con 0x55c52abc7ca0 2024-03-19T02:29:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.935+0000 7f71b4ff9640 1 -- 172.21.15.116:0/1798503163 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0003850 con 0x55c52abc7ca0 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 -- 172.21.15.116:0/1798503163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 msgr2=0x55c52abdc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71b001b040 tx=0x7f71b0009680 comp rx=0 tx=0).stop 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 -- 172.21.15.116:0/1798503163 shutdown_connections 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abd0880 0x55c52abe8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abdc410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/1798503163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abc7380 0x55c52abcff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 -- 172.21.15.116:0/1798503163 >> 172.21.15.116:0/1798503163 conn(0x55c52abc1ab0 msgr2=0x55c52abc3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 -- 172.21.15.116:0/1798503163 shutdown_connections 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.936+0000 7f71c7fe4740 1 -- 172.21.15.116:0/1798503163 wait complete. 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 -- start start 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 Processor -- start 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abd8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b00d0 con 0x55c52abc7380 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01afa00 con 0x55c52abd0880 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01acc40 con 0x55c52abc7ca0 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71c7fe4740 10 client.0 _ll_register_callbacks cb 0x7f71a8009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71b67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abd8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37970/0 (socket says 172.21.15.116:37970) 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 -- 172.21.15.116:0/840413744 learned_addr learned my addr 172.21.15.116:0/840413744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71b67fc640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 -- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 msgr2=0x55c52abd5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.937+0000 7f71beffd640 1 -- 172.21.15.116:0/840413744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 msgr2=0x55c52abd8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71beffd640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abd8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71beffd640 1 -- 172.21.15.116:0/840413744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52abd8fb0 con 0x55c52abc7ca0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71b67fc640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71beffd640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f71b00057c0 tx=0x7f71b0009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6bd8e1fb288f26c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b003b020 con 0x55c52abc7ca0 2024-03-19T02:29:18.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f71b002e630 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b0004360 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71b002eab0 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71c7fe4740 10 client.25999 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c52ac46e40 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.938+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b0004360 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f71b0004800 con 0x55c52abc7ca0 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 10 client.25999 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 10 client.25999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 20 client.25999 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52ac494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:18.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 4 client.25999 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c52ab329f0 con 0x55c52abe0b80 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71b67fc640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52ac494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71a67fc640 10 client.25999 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.939+0000 7f71b67fc640 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52ac494a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2f7bbf280cc305d server_cookie=7627a1556f0524a8 in_seq=0 out_seq=0 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.940+0000 7f71a67fc640 1 -- 172.21.15.116:0/840413744 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c52ab329f0 con 0x55c52abe0b80 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.940+0000 7f71a67fc640 10 client.25999 handle_command_reply: tid=0 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.940+0000 7f71c7fe4740 1 client.25999 shutdown 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 msgr2=0x55c52abd5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f71b00057c0 tx=0x7f71b0009830 comp rx=0 tx=0).stop 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 shutdown_connections 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c52abe0b80 0x55c52ac494a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c52abd0880 0x55c52abd5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c52abc7ca0 0x55c52abd5a10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 --2- 172.21.15.116:0/840413744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c52abc7380 0x55c52abd8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 >> 172.21.15.116:0/840413744 conn(0x55c52abc1ab0 msgr2=0x55c52abc2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71a67fc640 0 client.25999 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 shutdown_connections 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 1 -- 172.21.15.116:0/840413744 wait complete. 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c7fe4740 20 client.25999 trim_cache size 0 max 16384 2024-03-19T02:29:18.947 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:18.948 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:29:18.948 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "67de1b39-feb1-4487-87ce-27fe29e9c19f", 2024-03-19T02:29:18.948 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:29:18.948 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 msgr2=0x7f71ac0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 0x7f71ac0a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f71b80cce10 tx=0x7f71b8097850 comp rx=0 tx=0).stop 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 msgr2=0x7f71c019c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.941+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f71b800eb10 tx=0x7f71b800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 shutdown_connections 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71ac0a2a90 0x7f71ac0a4f50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01077c0 0x7f71c019c8c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c006a750 0x7f71c0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 --2- 172.21.15.116:0/2760138467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c0069d90 0x7f71c0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 >> 172.21.15.116:0/2760138467 conn(0x7f71c0077700 msgr2=0x7f71c00ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 shutdown_connections 2024-03-19T02:29:18.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:18.942+0000 7f71c531b640 1 -- 172.21.15.116:0/2760138467 wait complete. 2024-03-19T02:29:18.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:29:18.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:29:19.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.067+0000 7f474dd09640 1 -- 172.21.15.116:0/3213632174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4748069bc0 msgr2=0x7f474806a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.067+0000 7f474dd09640 1 --2- 172.21.15.116:0/3213632174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4748069bc0 0x7f474806a020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f473c009900 tx=0x7f473c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 -- 172.21.15.116:0/3213632174 shutdown_connections 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 --2- 172.21.15.116:0/3213632174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f474806a560 0x7f474810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 --2- 172.21.15.116:0/3213632174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4748069bc0 0x7f474806a020 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 --2- 172.21.15.116:0/3213632174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f4748107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 -- 172.21.15.116:0/3213632174 >> 172.21.15.116:0/3213632174 conn(0x7f47480777b0 msgr2=0x7f4748077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 -- 172.21.15.116:0/3213632174 shutdown_connections 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 -- 172.21.15.116:0/3213632174 wait complete. 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 Processor -- start 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.068+0000 7f474dd09640 1 -- start start 2024-03-19T02:29:19.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 0x7f4748198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4748111a80 con 0x7f4748107810 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4748111900 con 0x7f4748069bc0 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4748111c00 con 0x7f474806a560 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f47477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4746ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 0x7f4748198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f47477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55426/0 (socket says 172.21.15.116:55426) 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34110/0 (socket says 172.21.15.116:34110) 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4746ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 0x7f4748198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37988/0 (socket says 172.21.15.116:37988) 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f47477fe640 1 -- 172.21.15.116:0/2114797057 learned_addr learned my addr 172.21.15.116:0/2114797057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 -- 172.21.15.116:0/2114797057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 msgr2=0x7f4748198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 0x7f4748198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 -- 172.21.15.116:0/2114797057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 msgr2=0x7f4748198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 -- 172.21.15.116:0/2114797057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474819d010 con 0x7f4748107810 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f47477fe640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4747fff640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4740009fd0 tx=0x7f474000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f4744ff9640 1 -- 172.21.15.116:0/2114797057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474000ce20 con 0x7f4748107810 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.069+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474819d300 con 0x7f4748107810 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.070+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47481a4b90 con 0x7f4748107810 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.070+0000 7f4744ff9640 1 -- 172.21.15.116:0/2114797057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4740010430 con 0x7f4748107810 2024-03-19T02:29:19.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.070+0000 7f4744ff9640 1 -- 172.21.15.116:0/2114797057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474000f620 con 0x7f4748107810 2024-03-19T02:29:19.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f4744ff9640 1 -- 172.21.15.116:0/2114797057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 137524+0+0 (secure 0 0 0) 0x7f47400105e0 con 0x7f4748107810 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 Processor -- start 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 -- start start 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f4744ff9640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 0x7f47300a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1039350 0x5575d10419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d1044320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5575d0f9c500 con 0x5575d1039350 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5575d0f44be0 con 0x5575d1038a30 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47509d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5575d0f985a0 con 0x5575d1041f30 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f4744ff9640 1 -- 172.21.15.116:0/2114797057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f47400cf760 con 0x7f4748107810 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f4746ffd640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 0x7f47300a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f47477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55442/0 (socket says 172.21.15.116:55442) 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.072+0000 7f4746ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1039350 0x5575d10419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 -- 172.21.15.116:0/64948012 learned_addr learned my addr 172.21.15.116:0/64948012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f4747fff640 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d1044320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 -- 172.21.15.116:0/64948012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 msgr2=0x5575d1044320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d1044320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 -- 172.21.15.116:0/64948012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1039350 msgr2=0x5575d10419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1039350 0x5575d10419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 -- 172.21.15.116:0/64948012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575d0f66000 con 0x5575d1038a30 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f4746ffd640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 0x7f47300a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473c005ab0 tx=0x7f473c02bcd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f47477fe640 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f47380115c0 tx=0x7f4738011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8a0ca880620dd79 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f472dffb640 1 -- 172.21.15.116:0/64948012 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473800f660 con 0x5575d1038a30 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f472dffb640 1 -- 172.21.15.116:0/64948012 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f473800fcb0 con 0x5575d1038a30 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.073+0000 7f472dffb640 1 -- 172.21.15.116:0/64948012 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738019c80 con 0x5575d1038a30 2024-03-19T02:29:19.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- 172.21.15.116:0/64948012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 msgr2=0x5575d1038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f47380115c0 tx=0x7f4738011a80 comp rx=0 tx=0).stop 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- 172.21.15.116:0/64948012 shutdown_connections 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d1044320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1039350 0x5575d10419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 --2- 172.21.15.116:0/64948012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1038a30 0x5575d1038e10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- 172.21.15.116:0/64948012 >> 172.21.15.116:0/64948012 conn(0x5575d1033160 msgr2=0x5575d1033540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- 172.21.15.116:0/64948012 shutdown_connections 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- 172.21.15.116:0/64948012 wait complete. 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.074+0000 7f47509d2740 1 -- start start 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 Processor -- start 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5575d1049130 con 0x5575d1038a30 2024-03-19T02:29:19.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5575d0f58160 con 0x5575d1039350 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5575d10492b0 con 0x5575d1041f30 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47509d2740 10 client.0 _ll_register_callbacks cb 0x5575d0f94a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34116/0 (socket says 172.21.15.116:34116) 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 -- 172.21.15.116:0/1525395643 learned_addr learned my addr 172.21.15.116:0/1525395643 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f4746ffd640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f4747fff640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 -- 172.21.15.116:0/1525395643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 msgr2=0x5575d103f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 -- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 msgr2=0x5575d103f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 -- 172.21.15.116:0/1525395643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575d0d7a7d0 con 0x5575d1038a30 2024-03-19T02:29:19.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f47477fe640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4738009350 tx=0x7f4738019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4da96d00282e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.075+0000 7f4726ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738019e30 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4738016050 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473800fcb0 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f473802f050 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 10 client.16248 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5575d0cb03f0 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f473800fcb0 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f47380050d0 con 0x5575d1038a30 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 10 client.16248 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 20 client.16248 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d1056550 0x5575d1058940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 4 client.16248 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575d0fa9a40 con 0x5575d1056550 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.076+0000 7f4746ffd640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d1056550 0x5575d1058940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.077+0000 7f4746ffd640 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d1056550 0x5575d1058940 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f20c17b027aa29b server_cookie=b72e28cb923ab669 in_seq=0 out_seq=0 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.077+0000 7f4726ffd640 10 client.16248 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.077+0000 7f4726ffd640 1 -- 172.21.15.116:0/1525395643 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575d0fa9a40 con 0x5575d1056550 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.077+0000 7f4726ffd640 10 client.16248 handle_command_reply: tid=0 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.077+0000 7f47509d2740 1 client.16248 shutdown 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 msgr2=0x5575d103ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4738009350 tx=0x7f4738019640 comp rx=0 tx=0).stop 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 shutdown_connections 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d1056550 0x5575d1058940 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f4726ffd640 0 client.16248 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 --2- 172.21.15.116:0/1525395643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103ee50 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 >> 172.21.15.116:0/1525395643 conn(0x5575d1033160 msgr2=0x5575d1034eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 shutdown_connections 2024-03-19T02:29:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 1 -- 172.21.15.116:0/1525395643 wait complete. 2024-03-19T02:29:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.078+0000 7f47509d2740 20 client.16248 trim_cache size 0 max 16384 2024-03-19T02:29:19.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 Processor -- start 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 -- start start 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d1051ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ac950 con 0x5575d1038a30 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ae4c0 con 0x5575d1039350 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47509d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ad020 con 0x5575d1041f30 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34126/0 (socket says 172.21.15.116:34126) 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f47477fe640 1 -- 172.21.15.116:0/31519974 learned_addr learned my addr 172.21.15.116:0/31519974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f4746ffd640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d1051ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.096+0000 7f4747fff640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 -- 172.21.15.116:0/31519974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 msgr2=0x5575d103f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 -- 172.21.15.116:0/31519974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 msgr2=0x5575d1051ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d1051ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 -- 172.21.15.116:0/31519974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575d0c700f0 con 0x5575d1038a30 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f4746ffd640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d1051ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f47477fe640 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4738000c00 tx=0x7f47380056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3842a42add463270 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f472d7fa640 1 -- 172.21.15.116:0/31519974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380196b0 con 0x5575d1038a30 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f472d7fa640 1 -- 172.21.15.116:0/31519974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f473800f870 con 0x5575d1038a30 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.097+0000 7f472d7fa640 1 -- 172.21.15.116:0/31519974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380042d0 con 0x5575d1038a30 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- 172.21.15.116:0/31519974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 msgr2=0x5575d1041580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4738000c00 tx=0x7f47380056e0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- 172.21.15.116:0/31519974 shutdown_connections 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d103f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d1051ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 --2- 172.21.15.116:0/31519974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d1041580 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- 172.21.15.116:0/31519974 >> 172.21.15.116:0/31519974 conn(0x5575d1033160 msgr2=0x5575d103dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- 172.21.15.116:0/31519974 shutdown_connections 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- 172.21.15.116:0/31519974 wait complete. 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 -- start start 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 Processor -- start 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.098+0000 7f47509d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d10bb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47481addc0 con 0x5575d1038a30 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ad6f0 con 0x5575d1039350 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47481aa930 con 0x5575d1041f30 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 10 client.0 _ll_register_callbacks cb 0x7f47400093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34130/0 (socket says 172.21.15.116:34130) 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 -- 172.21.15.116:0/616284604 learned_addr learned my addr 172.21.15.116:0/616284604 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4747fff640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d10bb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4746ffd640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 -- 172.21.15.116:0/616284604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 msgr2=0x5575d10bb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d10bb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 -- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 msgr2=0x5575d103c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 -- 172.21.15.116:0/616284604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575d10bbcc0 con 0x5575d1038a30 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47477fe640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f47380199b0 tx=0x7f4738011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e937518bb8bbb62e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4726ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473800fa20 con 0x5575d1038a30 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f473800fbc0 con 0x5575d1038a30 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473800f870 con 0x5575d1038a30 2024-03-19T02:29:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4738044050 con 0x5575d1038a30 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 10 client.16254 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.099+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5575d10bcf70 con 0x5575d1038a30 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f473800f870 con 0x5575d1038a30 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f473803e4c0 con 0x5575d1038a30 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 10 client.16254 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 10 client.16254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 20 client.16254 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d10bd520 0x5575d10bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 4 client.16254 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5575d0fa3d90 con 0x5575d10bd520 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f4746ffd640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d10bd520 0x5575d10bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f4726ffd640 10 client.16254 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.100+0000 7f4746ffd640 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d10bd520 0x5575d10bf940 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ae58eb91e990cf server_cookie=5049fbd89143ef09 in_seq=0 out_seq=0 2024-03-19T02:29:19.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f4726ffd640 1 -- 172.21.15.116:0/616284604 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5575d0fa3d90 con 0x5575d10bd520 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f4726ffd640 10 client.16254 handle_command_reply: tid=0 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 client.16254 shutdown 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 msgr2=0x5575d103c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f47380199b0 tx=0x7f4738011440 comp rx=0 tx=0).stop 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 shutdown_connections 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5575d10bd520 0x5575d10bf940 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5575d1041f30 0x5575d10bb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575d1039350 0x5575d103c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f4726ffd640 0 client.16254 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 --2- 172.21.15.116:0/616284604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575d1038a30 0x5575d103c0c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 >> 172.21.15.116:0/616284604 conn(0x5575d1033160 msgr2=0x5575d10321e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 shutdown_connections 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 1 -- 172.21.15.116:0/616284604 wait complete. 2024-03-19T02:29:19.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.101+0000 7f47509d2740 20 client.16254 trim_cache size 0 max 16384 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "67de1b39-feb1-4487-87ce-27fe29e9c19f": { 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "67de1b39-feb1-4487-87ce-27fe29e9c19f", 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:29:19.108 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 msgr2=0x7f47300a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 0x7f47300a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473c005ab0 tx=0x7f473c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 msgr2=0x7f474819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4740009fd0 tx=0x7f474000ef10 comp rx=0 tx=0).stop 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 shutdown_connections 2024-03-19T02:29:19.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47300a2a90 0x7f47300a4f50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748107810 0x7f474819c8e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f474806a560 0x7f4748198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 --2- 172.21.15.116:0/2114797057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4748069bc0 0x7f4748198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 >> 172.21.15.116:0/2114797057 conn(0x7f47480777b0 msgr2=0x7f47480ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 shutdown_connections 2024-03-19T02:29:19.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:19.102+0000 7f474dd09640 1 -- 172.21.15.116:0/2114797057 wait complete. 2024-03-19T02:29:19.112 INFO:tasks.cephfs.filesystem:scrub status for tag:67de1b39-feb1-4487-87ce-27fe29e9c19f - {'path': '/', 'tag': '67de1b39-feb1-4487-87ce-27fe29e9c19f', 'options': 'recursive,force'} 2024-03-19T02:29:19.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 26149 3.74 MB/sec warmup 42 sec latency 25.089 ms 2024-03-19T02:29:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: pgmap v48: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:29:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:29:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:29:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: pgmap v48: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: pgmap v48: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:29:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:29:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:29:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:29:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:20.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 26856 3.73 MB/sec warmup 43 sec latency 17.399 ms 2024-03-19T02:29:21.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:20 smithi162 ceph-mon[35761]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-19T02:29:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:20 smithi154 ceph-mon[35693]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-19T02:29:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:20 smithi116 ceph-mon[32263]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-19T02:29:21.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 27489 3.70 MB/sec warmup 44 sec latency 17.202 ms 2024-03-19T02:29:22.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:21 smithi154 ceph-mon[35693]: pgmap v49: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:21 smithi162 ceph-mon[35761]: pgmap v49: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:21 smithi116 ceph-mon[32263]: pgmap v49: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:22.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 28195 3.76 MB/sec warmup 45 sec latency 18.079 ms 2024-03-19T02:29:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 28755 3.78 MB/sec warmup 46 sec latency 16.697 ms 2024-03-19T02:29:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:23 smithi162 ceph-mon[35761]: pgmap v50: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:23 smithi154 ceph-mon[35693]: pgmap v50: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:23 smithi116 ceph-mon[32263]: pgmap v50: 129 pgs: 129 active+clean; 41 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:24.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 29222 3.71 MB/sec warmup 47 sec latency 42.205 ms 2024-03-19T02:29:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:24 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:29:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:24 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:29:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:24 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:29:25.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 29865 3.73 MB/sec warmup 48 sec latency 16.994 ms 2024-03-19T02:29:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:29:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:25 smithi162 ceph-mon[35761]: pgmap v51: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:29:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:29:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:25 smithi154 ceph-mon[35693]: pgmap v51: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:29:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:29:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:25 smithi116 ceph-mon[32263]: pgmap v51: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:29:26.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 30561 3.67 MB/sec warmup 49 sec latency 23.298 ms 2024-03-19T02:29:27.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 31290 3.70 MB/sec warmup 50 sec latency 19.460 ms 2024-03-19T02:29:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:27 smithi162 ceph-mon[35761]: pgmap v52: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:27 smithi154 ceph-mon[35693]: pgmap v52: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:27 smithi116 ceph-mon[32263]: pgmap v52: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:28.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 31805 3.70 MB/sec warmup 51 sec latency 18.765 ms 2024-03-19T02:29:29.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 32393 3.71 MB/sec warmup 52 sec latency 16.038 ms 2024-03-19T02:29:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:29 smithi162 ceph-mon[35761]: pgmap v53: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:29 smithi154 ceph-mon[35693]: pgmap v53: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:29 smithi116 ceph-mon[32263]: pgmap v53: 129 pgs: 129 active+clean; 42 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:29:30.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 32905 3.66 MB/sec warmup 53 sec latency 33.332 ms 2024-03-19T02:29:31.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 33580 3.67 MB/sec warmup 54 sec latency 15.069 ms 2024-03-19T02:29:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:31 smithi162 ceph-mon[35761]: pgmap v54: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:29:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:31 smithi154 ceph-mon[35693]: pgmap v54: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:29:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:31 smithi116 ceph-mon[32263]: pgmap v54: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:29:32.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 34300 3.64 MB/sec warmup 55 sec latency 22.618 ms 2024-03-19T02:29:33.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 35159 3.69 MB/sec warmup 56 sec latency 15.197 ms 2024-03-19T02:29:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:33 smithi162 ceph-mon[35761]: pgmap v55: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:33 smithi154 ceph-mon[35693]: pgmap v55: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:33 smithi116 ceph-mon[32263]: pgmap v55: 129 pgs: 129 active+clean; 43 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:34.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 35630 3.70 MB/sec warmup 57 sec latency 17.767 ms 2024-03-19T02:29:35.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 36081 3.66 MB/sec warmup 58 sec latency 14.952 ms 2024-03-19T02:29:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:35 smithi162 ceph-mon[35761]: pgmap v56: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:35 smithi154 ceph-mon[35693]: pgmap v56: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:35 smithi116 ceph-mon[32263]: pgmap v56: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:36.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 36631 3.63 MB/sec warmup 59 sec latency 38.419 ms 2024-03-19T02:29:37.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 37320 3.63 MB/sec warmup 60 sec latency 50.866 ms 2024-03-19T02:29:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:37 smithi162 ceph-mon[35761]: pgmap v57: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:37 smithi154 ceph-mon[35693]: pgmap v57: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:37 smithi116 ceph-mon[32263]: pgmap v57: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:38.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 37937 3.60 MB/sec warmup 61 sec latency 24.528 ms 2024-03-19T02:29:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 38794 3.65 MB/sec warmup 62 sec latency 41.526 ms 2024-03-19T02:29:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:39 smithi162 ceph-mon[35761]: pgmap v58: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:39 smithi154 ceph-mon[35693]: pgmap v58: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:39 smithi116 ceph-mon[32263]: pgmap v58: 129 pgs: 129 active+clean; 45 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:40.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 39266 3.67 MB/sec warmup 63 sec latency 17.369 ms 2024-03-19T02:29:41.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 39668 3.61 MB/sec warmup 64 sec latency 47.079 ms 2024-03-19T02:29:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:41 smithi162 ceph-mon[35761]: pgmap v59: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:41 smithi154 ceph-mon[35693]: pgmap v59: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:41 smithi116 ceph-mon[32263]: pgmap v59: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:29:42.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 40310 3.59 MB/sec warmup 65 sec latency 14.718 ms 2024-03-19T02:29:43.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 40983 3.59 MB/sec warmup 66 sec latency 23.031 ms 2024-03-19T02:29:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:43 smithi162 ceph-mon[35761]: pgmap v60: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:43 smithi154 ceph-mon[35693]: pgmap v60: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:43 smithi116 ceph-mon[32263]: pgmap v60: 129 pgs: 129 active+clean; 46 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:44.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 41637 3.57 MB/sec warmup 67 sec latency 23.045 ms 2024-03-19T02:29:45.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 42372 3.61 MB/sec warmup 68 sec latency 21.183 ms 2024-03-19T02:29:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:45 smithi162 ceph-mon[35761]: pgmap v61: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:45 smithi154 ceph-mon[35693]: pgmap v61: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:45 smithi116 ceph-mon[32263]: pgmap v61: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:46.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 42910 3.63 MB/sec warmup 69 sec latency 17.080 ms 2024-03-19T02:29:47.779 INFO:tasks.workunit.client.0.smithi116.stdout: 1 43342 3.58 MB/sec warmup 70 sec latency 44.905 ms 2024-03-19T02:29:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:47 smithi162 ceph-mon[35761]: pgmap v62: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:47 smithi154 ceph-mon[35693]: pgmap v62: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:47 smithi116 ceph-mon[32263]: pgmap v62: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:48.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 44000 3.60 MB/sec warmup 71 sec latency 16.782 ms 2024-03-19T02:29:49.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:29:49.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 -- 172.21.15.116:0/2582574003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc1066a0 msgr2=0x7f49bc106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 --2- 172.21.15.116:0/2582574003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc1066a0 0x7f49bc106a80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009930 tx=0x7f49ac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 -- 172.21.15.116:0/2582574003 shutdown_connections 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 --2- 172.21.15.116:0/2582574003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc101050 0x7f49bc10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 --2- 172.21.15.116:0/2582574003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 0x7f49bc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 --2- 172.21.15.116:0/2582574003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc1066a0 0x7f49bc106a80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 -- 172.21.15.116:0/2582574003 >> 172.21.15.116:0/2582574003 conn(0x7f49bc077820 msgr2=0x7f49bc077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 -- 172.21.15.116:0/2582574003 shutdown_connections 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.228+0000 7f49c2990640 1 -- 172.21.15.116:0/2582574003 wait complete. 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 Processor -- start 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 -- start start 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 0x7f49bc198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc1066a0 0x7f49bc1a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc110920 con 0x7f49bc101050 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1107a0 con 0x7f49bc1066a0 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c2990640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc110aa0 con 0x7f49bc100690 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 0x7f49bc198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54816/0 (socket says 172.21.15.116:54816) 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49bb7fe640 1 -- 172.21.15.116:0/1424049778 learned_addr learned my addr 172.21.15.116:0/1424049778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.229+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc1066a0 0x7f49bc1a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 -- 172.21.15.116:0/1424049778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 msgr2=0x7f49bc198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 0x7f49bc198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 -- 172.21.15.116:0/1424049778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc1066a0 msgr2=0x7f49bc1a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc1066a0 0x7f49bc1a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 -- 172.21.15.116:0/1424049778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49bc1a43c0 con 0x7f49bc101050 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f49b000e9f0 tx=0x7f49b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49b97fa640 1 -- 172.21.15.116:0/1424049778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b000cd80 con 0x7f49bc101050 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49b97fa640 1 -- 172.21.15.116:0/1424049778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49b00108e0 con 0x7f49bc101050 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49b97fa640 1 -- 172.21.15.116:0/1424049778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b0018920 con 0x7f49bc101050 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49bc1a4590 con 0x7f49bc101050 2024-03-19T02:29:49.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.230+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49bc1a4ae0 con 0x7f49bc101050 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.232+0000 7f49b97fa640 1 -- 172.21.15.116:0/1424049778 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f49b0010a90 con 0x7f49bc101050 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 Processor -- start 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49b97fa640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 0x7f49a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 -- start start 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 0x7f49a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49b97fa640 1 -- 172.21.15.116:0/1424049778 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f49b00cf440 con 0x7f49bc101050 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90cece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90cf8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c90c50500 con 0x563c90ced350 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563c90bf8be0 con 0x563c90cf5f30 2024-03-19T02:29:49.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c557f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c90c4c5a0 con 0x563c90ceca30 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54818/0 (socket says 172.21.15.116:54818) 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90cece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 -- 172.21.15.116:0/575340964 learned_addr learned my addr 172.21.15.116:0/575340964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90cf8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 -- 172.21.15.116:0/575340964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 msgr2=0x563c90cece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90cece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 -- 172.21.15.116:0/575340964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 msgr2=0x563c90cf8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90cf8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49bb7fe640 1 -- 172.21.15.116:0/575340964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c90c1a000 con 0x563c90ced350 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.233+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 0x7f49a40a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49b4005fd0 tx=0x7f49b4011810 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f49b000e9c0 tx=0x7f49b009b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67c350f6f11a111 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49a27fc640 1 -- 172.21.15.116:0/575340964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b00cf150 con 0x563c90ced350 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49a27fc640 1 -- 172.21.15.116:0/575340964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49b00ccb30 con 0x563c90ced350 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49a27fc640 1 -- 172.21.15.116:0/575340964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b00afba0 con 0x563c90ced350 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49c557f740 1 -- 172.21.15.116:0/575340964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 msgr2=0x563c90cf59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.234+0000 7f49c557f740 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f49b000e9c0 tx=0x7f49b009b870 comp rx=0 tx=0).stop 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- 172.21.15.116:0/575340964 shutdown_connections 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90cf8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90cf59f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- 172.21.15.116:0/575340964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90cece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- 172.21.15.116:0/575340964 >> 172.21.15.116:0/575340964 conn(0x563c90ce7160 msgr2=0x563c90ce7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- 172.21.15.116:0/575340964 shutdown_connections 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- 172.21.15.116:0/575340964 wait complete. 2024-03-19T02:29:49.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- start start 2024-03-19T02:29:49.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 Processor -- start 2024-03-19T02:29:49.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90cf34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cf3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c90cfd0d0 con 0x563c90ceca30 2024-03-19T02:29:49.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563c90c0c160 con 0x563c90ced350 2024-03-19T02:29:49.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c90cfd250 con 0x563c90cf5f30 2024-03-19T02:29:49.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.235+0000 7f49c557f740 10 client.0 _ll_register_callbacks cb 0x563c90c48a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54826/0 (socket says 172.21.15.116:54826) 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 -- 172.21.15.116:0/1545362939 learned_addr learned my addr 172.21.15.116:0/1545362939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90cf34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cf3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 -- 172.21.15.116:0/1545362939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 msgr2=0x563c90cf3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cf3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 -- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 msgr2=0x563c90cf34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90cf34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 -- 172.21.15.116:0/1545362939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c90a2e7d0 con 0x563c90ceca30 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49bbfff640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f49ac036040 tx=0x7f49ac02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63d2358b6c62ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f499b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac02c5d0 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f49ac005030 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49b00ccb30 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f49ac052050 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49c557f740 10 client.16266 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.236+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c909643f0 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49b00ccb30 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f49ac0035f0 con 0x563c90ceca30 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 10 client.16266 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 20 client.16266 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d0a7e0 0x563c90d0cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 4 client.16266 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c90c5da40 con 0x563c90d0a7e0 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d0a7e0 0x563c90d0cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d0a7e0 0x563c90d0cc80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2db9af5f3f4b7f server_cookie=79c97d48cf5c5745 in_seq=0 out_seq=0 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.237+0000 7f499b7fe640 10 client.16266 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f499b7fe640 1 -- 172.21.15.116:0/1545362939 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c90c5da40 con 0x563c90d0a7e0 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f499b7fe640 10 client.16266 handle_command_reply: tid=0 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 client.16266 shutdown 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 msgr2=0x563c90cf2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f49ac036040 tx=0x7f49ac02c9f0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 shutdown_connections 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d0a7e0 0x563c90d0cc80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cf3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90cf34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f499b7fe640 0 client.16266 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 --2- 172.21.15.116:0/1545362939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90cf2f90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 >> 172.21.15.116:0/1545362939 conn(0x563c90ce7160 msgr2=0x563c90ce8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 shutdown_connections 2024-03-19T02:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.238+0000 7f49c557f740 1 -- 172.21.15.116:0/1545362939 wait complete. 2024-03-19T02:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.239+0000 7f49c557f740 20 client.16266 trim_cache size 0 max 16384 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 Processor -- start 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 -- start start 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90ce61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90d12550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1ac930 con 0x563c90ced350 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1ae4a0 con 0x563c90cf5f30 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c557f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1ad000 con 0x563c90ceca30 2024-03-19T02:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90ce61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49c0f2d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90d12550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54840/0 (socket says 172.21.15.116:54840) 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.259+0000 7f49bb7fe640 1 -- 172.21.15.116:0/4187036855 learned_addr learned my addr 172.21.15.116:0/4187036855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 -- 172.21.15.116:0/4187036855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 msgr2=0x563c90d12550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90d12550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 -- 172.21.15.116:0/4187036855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 msgr2=0x563c90d05ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 -- 172.21.15.116:0/4187036855 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c909240f0 con 0x563c90ceca30 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49bbfff640 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90ce61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003b80 tx=0x7f49ac02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fafe8631cc42340c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49a1ffb640 1 -- 172.21.15.116:0/4187036855 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac000e60 con 0x563c90ceca30 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49a1ffb640 1 -- 172.21.15.116:0/4187036855 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49ac009520 con 0x563c90ceca30 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.260+0000 7f49a1ffb640 1 -- 172.21.15.116:0/4187036855 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0054e0 con 0x563c90ceca30 2024-03-19T02:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- 172.21.15.116:0/4187036855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 msgr2=0x563c90ce61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90ce61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003b80 tx=0x7f49ac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- 172.21.15.116:0/4187036855 shutdown_connections 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90cf5f30 0x563c90d12550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ced350 0x563c90d05ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 --2- 172.21.15.116:0/4187036855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90ceca30 0x563c90ce61e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- 172.21.15.116:0/4187036855 >> 172.21.15.116:0/4187036855 conn(0x563c90ce7160 msgr2=0x563c90cf2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- 172.21.15.116:0/4187036855 shutdown_connections 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- 172.21.15.116:0/4187036855 wait complete. 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.261+0000 7f49c557f740 1 -- start start 2024-03-19T02:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 Processor -- start 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90d02620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cfe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1adda0 con 0x563c90ceca30 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1ad6d0 con 0x563c90ced350 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1aa910 con 0x563c90cf5f30 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c557f740 10 client.0 _ll_register_callbacks cb 0x7f49ac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54852/0 (socket says 172.21.15.116:54852) 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 -- 172.21.15.116:0/3117967363 learned_addr learned my addr 172.21.15.116:0/3117967363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90d02620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49c0f2d640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cfe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 -- 172.21.15.116:0/3117967363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 msgr2=0x563c90cfe5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cfe5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 -- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 msgr2=0x563c90d02620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90d02620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 -- 172.21.15.116:0/3117967363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c90cfed40 con 0x563c90ceca30 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f49bbfff640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003a10 tx=0x7f49ac02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4abfacd9b45f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.262+0000 7f499b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac05a030 con 0x563c90ceca30 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f49ac005040 con 0x563c90ceca30 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49ac009520 con 0x563c90ceca30 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f49ac069050 con 0x563c90ceca30 2024-03-19T02:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 10 client.16272 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c90d71600 con 0x563c90ceca30 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49ac009520 con 0x563c90ceca30 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f49ac004880 con 0x563c90ceca30 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 10 client.16272 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 20 client.16272 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d71cc0 0x563c90d740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 4 client.16272 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.263+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c90c57d90 con 0x563c90d71cc0 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d71cc0 0x563c90d740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f499b7fe640 10 client.16272 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f49bb7fe640 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d71cc0 0x563c90d740e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b44ea3e45ca1f9a server_cookie=58f84f7c48751159 in_seq=0 out_seq=0 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f499b7fe640 1 -- 172.21.15.116:0/3117967363 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563c90c57d90 con 0x563c90d71cc0 2024-03-19T02:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f499b7fe640 10 client.16272 handle_command_reply: tid=0 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f49c557f740 1 client.16272 shutdown 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.264+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 msgr2=0x563c90d020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f49ac003a10 tx=0x7f49ac02c9f0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 shutdown_connections 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c90d71cc0 0x563c90d740e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c90cf5f30 0x563c90cfe5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f499b7fe640 0 client.16272 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c90ced350 0x563c90d02620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 --2- 172.21.15.116:0/3117967363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c90ceca30 0x563c90d020e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 >> 172.21.15.116:0/3117967363 conn(0x563c90ce7160 msgr2=0x563c90ce8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 shutdown_connections 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 1 -- 172.21.15.116:0/3117967363 wait complete. 2024-03-19T02:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c557f740 20 client.16272 trim_cache size 0 max 16384 2024-03-19T02:29:49.272 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:29:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:29:49.272 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 msgr2=0x7f49a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 0x7f49a40a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49b4005fd0 tx=0x7f49b4011810 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 msgr2=0x7f49bc199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f49b000e9f0 tx=0x7f49b000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 shutdown_connections 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49a40a2970 0x7f49a40a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49bc1066a0 0x7f49bc1a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49bc101050 0x7f49bc199340 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 --2- 172.21.15.116:0/1424049778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49bc100690 0x7f49bc198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 >> 172.21.15.116:0/1424049778 conn(0x7f49bc077820 msgr2=0x7f49bc102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.265+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 shutdown_connections 2024-03-19T02:29:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.266+0000 7f49c2990640 1 -- 172.21.15.116:0/1424049778 wait complete. 2024-03-19T02:29:49.278 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:29:49.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:29:49.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- 172.21.15.116:0/3960266898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd8100720 msgr2=0x7fddd8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 --2- 172.21.15.116:0/3960266898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd8100720 0x7fddd8100ba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009890 tx=0x7fddc802bb70 comp rx=0 tx=0).stop 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- 172.21.15.116:0/3960266898 shutdown_connections 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 --2- 172.21.15.116:0/3960266898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd81010e0 0x7fddd810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 --2- 172.21.15.116:0/3960266898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd8100720 0x7fddd8100ba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 --2- 172.21.15.116:0/3960266898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd8106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- 172.21.15.116:0/3960266898 >> 172.21.15.116:0/3960266898 conn(0x7fddd8077820 msgr2=0x7fddd8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- 172.21.15.116:0/3960266898 shutdown_connections 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- 172.21.15.116:0/3960266898 wait complete. 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 Processor -- start 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.388+0000 7fdddfe34640 1 -- start start 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 0x7fddd8198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 0x7fddd81993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fddd81109f0 con 0x7fddd8106730 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8110870 con 0x7fddd8100720 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8110b70 con 0x7fddd81010e0 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54862/0 (socket says 172.21.15.116:54862) 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 -- 172.21.15.116:0/2951396137 learned_addr learned my addr 172.21.15.116:0/2951396137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddd3cf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 0x7fddd81993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddddbd0640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 0x7fddd8198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 -- 172.21.15.116:0/2951396137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 msgr2=0x7fddd81993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 0x7fddd81993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 -- 172.21.15.116:0/2951396137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 msgr2=0x7fddd8198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 0x7fddd8198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 -- 172.21.15.116:0/2951396137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd81a4450 con 0x7fddd8106730 2024-03-19T02:29:49.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddddbd0640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 0x7fddd8198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddd3cf640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 0x7fddd81993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddde3d1640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fddd000eb10 tx=0x7fddd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0004b20 con 0x7fddd8106730 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd81a4620 con 0x7fddd8106730 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddd0004cc0 con 0x7fddd8106730 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.389+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0010690 con 0x7fddd8106730 2024-03-19T02:29:49.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.390+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd81a4b40 con 0x7fddd8106730 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.391+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd8068610 con 0x7fddd8106730 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.392+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fddd0005020 con 0x7fddd8106730 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.392+0000 7fddceffd640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 0x7fddc40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.392+0000 7fddddbd0640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 0x7fddc40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.392+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fddd00cf8b0 con 0x7fddd8106730 2024-03-19T02:29:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.393+0000 7fddddbd0640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 0x7fddc40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddd40096c0 tx=0x7fddd4009260 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.398+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fddd009c290 con 0x7fddd8106730 2024-03-19T02:29:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.651+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd8108cb0 con 0x7fddd8106730 2024-03-19T02:29:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.651+0000 7fddceffd640 1 -- 172.21.15.116:0/2951396137 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11951 (secure 0 0 0) 0x7fddd009bb90 con 0x7fddd8106730 2024-03-19T02:29:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:29:49.652 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:28:39.411853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:29:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-19T02:29:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 msgr2=0x7fddc40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 0x7fddc40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddd40096c0 tx=0x7fddd4009260 comp rx=0 tx=0).stop 2024-03-19T02:29:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 msgr2=0x7fddd81a3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fddd000eb10 tx=0x7fddd000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 shutdown_connections 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fddc40a2970 0x7fddc40a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8106730 0x7fddd81a3f10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd81010e0 0x7fddd81993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 --2- 172.21.15.116:0/2951396137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddd8100720 0x7fddd8198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.653+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 >> 172.21.15.116:0/2951396137 conn(0x7fddd8077820 msgr2=0x7fddd8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.654+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 shutdown_connections 2024-03-19T02:29:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.654+0000 7fdddfe34640 1 -- 172.21.15.116:0/2951396137 wait complete. 2024-03-19T02:29:49.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:29:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 44633 3.56 MB/sec warmup 72 sec latency 18.336 ms 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 -- 172.21.15.116:0/1894913781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc01066b0 msgr2=0x7fcfc0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/1894913781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc01066b0 0x7fcfc0106a90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009930 tx=0x7fcfb402bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 -- 172.21.15.116:0/1894913781 shutdown_connections 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/1894913781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc0101060 0x7fcfc010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/1894913781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 0x7fcfc0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/1894913781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc01066b0 0x7fcfc0106a90 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 -- 172.21.15.116:0/1894913781 >> 172.21.15.116:0/1894913781 conn(0x7fcfc00778d0 msgr2=0x7fcfc0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 -- 172.21.15.116:0/1894913781 shutdown_connections 2024-03-19T02:29:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.785+0000 7fcfc6d10640 1 -- 172.21.15.116:0/1894913781 wait complete. 2024-03-19T02:29:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 Processor -- start 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 -- start start 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 0x7fcfc0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc01066b0 0x7fcfc01a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0110930 con 0x7fcfc0101060 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01107b0 con 0x7fcfc01006a0 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc6d10640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0110ab0 con 0x7fcfc01066b0 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54878/0 (socket says 172.21.15.116:54878) 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 -- 172.21.15.116:0/957432237 learned_addr learned my addr 172.21.15.116:0/957432237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 0x7fcfc0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfc52ad640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc01066b0 0x7fcfc01a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 -- 172.21.15.116:0/957432237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc01066b0 msgr2=0x7fcfc01a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc01066b0 0x7fcfc01a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 -- 172.21.15.116:0/957432237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 msgr2=0x7fcfc0198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 0x7fcfc0198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.786+0000 7fcfbffff640 1 -- 172.21.15.116:0/957432237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc01a43b0 con 0x7fcfc0101060 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfbffff640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00e9f0 tx=0x7fcfac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfbdffb640 1 -- 172.21.15.116:0/957432237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac00cd80 con 0x7fcfc0101060 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc01a4580 con 0x7fcfc0101060 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfbdffb640 1 -- 172.21.15.116:0/957432237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfac0108e0 con 0x7fcfc0101060 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfbdffb640 1 -- 172.21.15.116:0/957432237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac018920 con 0x7fcfc0101060 2024-03-19T02:29:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.787+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc01a4af0 con 0x7fcfc0101060 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfbdffb640 1 -- 172.21.15.116:0/957432237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcfac010a90 con 0x7fcfc0101060 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 Processor -- start 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 -- start start 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb46fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617bb3c74b0 con 0x5617bb46d7d0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617bb36fd70 con 0x5617bb4642d0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.790+0000 7fcfc99de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617bb3c3550 con 0x5617bb464bf0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfbdffb640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 0x7fcfb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc52ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37936/0 (socket says 172.21.15.116:37936) 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc52ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4646b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40948/0 (socket says 172.21.15.116:40948) 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 -- 172.21.15.116:0/1997442256 learned_addr learned my addr 172.21.15.116:0/1997442256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfbffff640 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb46fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc52ad640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 0x7fcfb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 -- 172.21.15.116:0/1997442256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 msgr2=0x5617bb4646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 -- 172.21.15.116:0/1997442256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 msgr2=0x5617bb46fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfbdffb640 1 -- 172.21.15.116:0/957432237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fcfac0cf440 con 0x7fcfc0101060 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb46fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 -- 172.21.15.116:0/1997442256 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617bb392210 con 0x5617bb464bf0 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc52ad640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 0x7fcfb00a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009fd0 tx=0x7fcfb8004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.791+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4036040 tx=0x7fcfb4009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7db697cac6768ae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfaaffd640 1 -- 172.21.15.116:0/1997442256 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4045020 con 0x5617bb464bf0 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfaaffd640 1 -- 172.21.15.116:0/1997442256 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb402c7e0 con 0x5617bb464bf0 2024-03-19T02:29:49.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfaaffd640 1 -- 172.21.15.116:0/1997442256 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40054f0 con 0x5617bb464bf0 2024-03-19T02:29:49.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 -- 172.21.15.116:0/1997442256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 msgr2=0x5617bb46d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4036040 tx=0x7fcfb4009560 comp rx=0 tx=0).stop 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 -- 172.21.15.116:0/1997442256 shutdown_connections 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb46fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb464bf0 0x5617bb46d290 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 --2- 172.21.15.116:0/1997442256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.792+0000 7fcfc99de740 1 -- 172.21.15.116:0/1997442256 >> 172.21.15.116:0/1997442256 conn(0x5617bb45ea00 msgr2=0x5617bb45ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.793+0000 7fcfc99de740 1 -- 172.21.15.116:0/1997442256 shutdown_connections 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.793+0000 7fcfc99de740 1 -- 172.21.15.116:0/1997442256 wait complete. 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.793+0000 7fcfc99de740 1 -- start start 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.793+0000 7fcfc99de740 1 Processor -- start 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb46a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb46b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617bb470560 con 0x5617bb464bf0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617bb383310 con 0x5617bb4642d0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617bb4706e0 con 0x5617bb46d7d0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc99de740 10 client.0 _ll_register_callbacks cb 0x5617bb3772d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54904/0 (socket says 172.21.15.116:54904) 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 -- 172.21.15.116:0/309319411 learned_addr learned my addr 172.21.15.116:0/309319411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc52ad640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb46b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb46a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 -- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 msgr2=0x5617bb46b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb46b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 -- 172.21.15.116:0/309319411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 msgr2=0x5617bb46a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb46a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.794+0000 7fcfbffff640 1 -- 172.21.15.116:0/309319411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617bb0db3f0 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfbffff640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcfac09b570 tx=0x7fcfac09b1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a2886836cc545c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac09f910 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fcfac0860b0 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb402c7e0 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfac0a3f60 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 10 client.16290 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617bb1a57d0 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfb402c7e0 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fcfac097650 con 0x5617bb464bf0 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 10 client.16290 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 10 client.16290 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:29:49.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 20 client.16290 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb47dc00 0x5617bb47fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 4 client.16290 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.795+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617bb4253e0 con 0x5617bb47dc00 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.796+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb47dc00 0x5617bb47fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.796+0000 7fcf9ffff640 10 client.16290 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.796+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb47dc00 0x5617bb47fff0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1843442eef561d20 server_cookie=60f129daee229457 in_seq=0 out_seq=0 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcf9ffff640 1 -- 172.21.15.116:0/309319411 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617bb4253e0 con 0x5617bb47dc00 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcf9ffff640 10 client.16290 handle_command_reply: tid=0 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 client.16290 shutdown 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 msgr2=0x5617bb46adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcfac09b570 tx=0x7fcfac09b1d0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 shutdown_connections 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb47dc00 0x5617bb47fff0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb46b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb46adb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 --2- 172.21.15.116:0/309319411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb46a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcf9ffff640 0 client.16290 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 >> 172.21.15.116:0/309319411 conn(0x5617bb45ea00 msgr2=0x5617bb460750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 shutdown_connections 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 1 -- 172.21.15.116:0/309319411 wait complete. 2024-03-19T02:29:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.797+0000 7fcfc99de740 20 client.16290 trim_cache size 0 max 16384 2024-03-19T02:29:49.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.811+0000 7fcfc99de740 1 Processor -- start 2024-03-19T02:29:49.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 -- start start 2024-03-19T02:29:49.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 0x5617bb479300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01aeaa0 con 0x5617bb46d7d0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01b0610 con 0x5617bb464bf0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc99de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01af170 con 0x5617bb4642d0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc52ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 0x5617bb479300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc4aac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc4aac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37950/0 (socket says 172.21.15.116:37950) 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc52ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54914/0 (socket says 172.21.15.116:54914) 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc4aac640 1 -- 172.21.15.116:0/975446320 learned_addr learned my addr 172.21.15.116:0/975446320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 -- 172.21.15.116:0/975446320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 msgr2=0x5617bb46cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 -- 172.21.15.116:0/975446320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 msgr2=0x5617bb4699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 -- 172.21.15.116:0/975446320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617bb468de0 con 0x5617bb464bf0 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfc52ad640 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.812+0000 7fcfbffff640 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 0x5617bb479300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcfac097cd0 tx=0x7fcfac0cf380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b7e0db41390fde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfaa7fc640 1 -- 172.21.15.116:0/975446320 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac09f910 con 0x5617bb464bf0 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfaa7fc640 1 -- 172.21.15.116:0/975446320 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfac097ee0 con 0x5617bb464bf0 2024-03-19T02:29:49.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfaa7fc640 1 -- 172.21.15.116:0/975446320 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac0afe00 con 0x5617bb464bf0 2024-03-19T02:29:49.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 -- 172.21.15.116:0/975446320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 msgr2=0x5617bb479300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 0x5617bb479300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcfac097cd0 tx=0x7fcfac0cf380 comp rx=0 tx=0).stop 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 -- 172.21.15.116:0/975446320 shutdown_connections 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb46d7d0 0x5617bb4699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb464bf0 0x5617bb479300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 --2- 172.21.15.116:0/975446320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb4642d0 0x5617bb46cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 -- 172.21.15.116:0/975446320 >> 172.21.15.116:0/975446320 conn(0x5617bb45ea00 msgr2=0x5617bb45fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 -- 172.21.15.116:0/975446320 shutdown_connections 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.813+0000 7fcfc99de740 1 -- 172.21.15.116:0/975446320 wait complete. 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 -- start start 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 Processor -- start 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb467f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01aff10 con 0x5617bb464bf0 2024-03-19T02:29:49.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01af840 con 0x5617bb4642d0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01aca80 con 0x5617bb46d7d0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc99de740 10 client.0 _ll_register_callbacks cb 0x7fcfac00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc52ad640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb467f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfbffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54924/0 (socket says 172.21.15.116:54924) 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfbffff640 1 -- 172.21.15.116:0/2369980555 learned_addr learned my addr 172.21.15.116:0/2369980555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 -- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 msgr2=0x5617bb467f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb467f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 -- 172.21.15.116:0/2369980555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 msgr2=0x5617bb467a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.814+0000 7fcfc4aac640 1 -- 172.21.15.116:0/2369980555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617bb4e2f50 con 0x5617bb4642d0 2024-03-19T02:29:49.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfbffff640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc4aac640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4674f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfb402c780 tx=0x7fcfb402c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d54fb8c3f49b1ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4051070 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfac097ee0 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fcfb40055d0 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfb405c050 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 10 client.26120 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617bb4e4210 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfac097ee0 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fcfb4003a10 con 0x5617bb4642d0 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 10 client.26120 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 10 client.26120 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 20 client.26120 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb4e4500 0x5617bb4e68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 4 client.26120 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.815+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617bb3d49f0 con 0x5617bb4e4500 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfbffff640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb4e4500 0x5617bb4e68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcf9ffff640 10 client.26120 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfbffff640 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb4e4500 0x5617bb4e68f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db77174ce15c94dd server_cookie=ac7f36cef5d7138 in_seq=0 out_seq=0 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcf9ffff640 1 -- 172.21.15.116:0/2369980555 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617bb3d49f0 con 0x5617bb4e4500 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcf9ffff640 10 client.26120 handle_command_reply: tid=0 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 client.26120 shutdown 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 msgr2=0x5617bb4674f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4674f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfb402c780 tx=0x7fcfb402c710 comp rx=0 tx=0).stop 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 shutdown_connections 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5617bb4e4500 0x5617bb4e68f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617bb46d7d0 0x5617bb467f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcf9ffff640 0 client.26120 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:29:49.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617bb464bf0 0x5617bb467a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 --2- 172.21.15.116:0/2369980555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617bb4642d0 0x5617bb4674f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.816+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 >> 172.21.15.116:0/2369980555 conn(0x5617bb45ea00 msgr2=0x5617bb4606e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 shutdown_connections 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc99de740 1 -- 172.21.15.116:0/2369980555 wait complete. 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc99de740 20 client.26120 trim_cache size 0 max 16384 2024-03-19T02:29:49.824 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 msgr2=0x7fcfb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 0x7fcfb00a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009fd0 tx=0x7fcfb8004ab0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 msgr2=0x7fcfc0199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00e9f0 tx=0x7fcfac00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 shutdown_connections 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcfb00a2970 0x7fcfb00a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcfc01066b0 0x7fcfc01a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc0101060 0x7fcfc0199330 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 --2- 172.21.15.116:0/957432237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfc01006a0 0x7fcfc0198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 >> 172.21.15.116:0/957432237 conn(0x7fcfc00778d0 msgr2=0x7fcfc0103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 shutdown_connections 2024-03-19T02:29:49.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.817+0000 7fcfc6d10640 1 -- 172.21.15.116:0/957432237 wait complete. 2024-03-19T02:29:49.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 -- 172.21.15.116:0/1208759559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c101060 msgr2=0x7f909c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/1208759559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c101060 0x7f909c10d570 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9094009cd0 tx=0x7f909402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 -- 172.21.15.116:0/1208759559 shutdown_connections 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/1208759559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c101060 0x7f909c10d570 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/1208759559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c1006a0 0x7f909c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/1208759559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1066b0 0x7f909c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 -- 172.21.15.116:0/1208759559 >> 172.21.15.116:0/1208759559 conn(0x7f909c0778d0 msgr2=0x7f909c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 -- 172.21.15.116:0/1208759559 shutdown_connections 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.943+0000 7f90a1b0c640 1 -- 172.21.15.116:0/1208759559 wait complete. 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 Processor -- start 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 -- start start 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 0x7f909c0fd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 0x7f909c0fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 0x7f909c0fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f909c110970 con 0x7f909c1066b0 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1107f0 con 0x7f909c1006a0 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f90a1b0c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f909c110af0 con 0x7f909c101060 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 0x7f909c0fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 0x7f909c0fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 0x7f909c0fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 0x7f909c0fe0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54950/0 (socket says 172.21.15.116:54950) 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 0x7f909c0fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41002/0 (socket says 172.21.15.116:41002) 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.944+0000 7f909bfff640 1 -- 172.21.15.116:0/3271284673 learned_addr learned my addr 172.21.15.116:0/3271284673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 -- 172.21.15.116:0/3271284673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 msgr2=0x7f909c0fd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 0x7f909c0fd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 -- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 msgr2=0x7f909c0fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 0x7f909c0fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 -- 172.21.15.116:0/3271284673 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c1a6410 con 0x7f909c101060 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f909affd640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 0x7f909c0fdbb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f909000e9f0 tx=0x7f909000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f9098ff9640 1 -- 172.21.15.116:0/3271284673 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909000ccc0 con 0x7f909c101060 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909c1a6700 con 0x7f909c101060 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f9098ff9640 1 -- 172.21.15.116:0/3271284673 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90900108e0 con 0x7f909c101060 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f9098ff9640 1 -- 172.21.15.116:0/3271284673 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090018880 con 0x7f909c101060 2024-03-19T02:29:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.945+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909c1a6c50 con 0x7f909c101060 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.947+0000 7f9098ff9640 1 -- 172.21.15.116:0/3271284673 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9090010a90 con 0x7f909c101060 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 Processor -- start 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 -- start start 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f9098ff9640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 0x7f90840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a927056bf0 0x55a92705f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 0x55a927061bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a926fb94b0 con 0x55a9270562d0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a926f61d70 con 0x55a92705f7d0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f90a47cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a926fb5550 con 0x55a927056bf0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909bfff640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 0x7f90840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54954/0 (socket says 172.21.15.116:54954) 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 -- 172.21.15.116:0/719976924 learned_addr learned my addr 172.21.15.116:0/719976924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f9098ff9640 1 -- 172.21.15.116:0/3271284673 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f90900cf490 con 0x7f909c101060 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909affd640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a927056bf0 0x55a92705f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909bfff640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 0x55a927061bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 -- 172.21.15.116:0/719976924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a927056bf0 msgr2=0x55a92705f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a927056bf0 0x55a92705f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 -- 172.21.15.116:0/719976924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 msgr2=0x55a927061bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 0x55a927061bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909b7fe640 1 -- 172.21.15.116:0/719976924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a926f84210 con 0x55a9270562d0 2024-03-19T02:29:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.948+0000 7f909bfff640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 0x55a927061bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.949+0000 7f909b7fe640 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f908c009c70 tx=0x7f908c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f46831e7f861c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.949+0000 7f909bfff640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 0x7f90840a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55a927051b90 tx=0x7f9094032340 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.949+0000 7f9081ffb640 1 -- 172.21.15.116:0/719976924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c004230 con 0x55a9270562d0 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.949+0000 7f9081ffb640 1 -- 172.21.15.116:0/719976924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f908c0043d0 con 0x55a9270562d0 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.949+0000 7f9081ffb640 1 -- 172.21.15.116:0/719976924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c0196d0 con 0x55a9270562d0 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- 172.21.15.116:0/719976924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 msgr2=0x55a9270566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f908c009c70 tx=0x7f908c011810 comp rx=0 tx=0).stop 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- 172.21.15.116:0/719976924 shutdown_connections 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a92705f7d0 0x55a927061bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a927056bf0 0x55a92705f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- 172.21.15.116:0/719976924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a9270566b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- 172.21.15.116:0/719976924 >> 172.21.15.116:0/719976924 conn(0x55a927050a00 msgr2=0x55a927050de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- 172.21.15.116:0/719976924 shutdown_connections 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- 172.21.15.116:0/719976924 wait complete. 2024-03-19T02:29:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- start start 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 Processor -- start 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9270562d0 0x55a92705c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a92705d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9270625c0 con 0x55a927056bf0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a926f75310 con 0x55a9270562d0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.950+0000 7f90a47cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a927062740 con 0x55a92705f7d0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f90a47cb740 10 client.0 _ll_register_callbacks cb 0x55a926f692d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9270562d0 0x55a92705c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54958/0 (socket says 172.21.15.116:54958) 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 -- 172.21.15.116:0/2310597340 learned_addr learned my addr 172.21.15.116:0/2310597340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909bfff640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a92705d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 -- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 msgr2=0x55a92705d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a92705d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 -- 172.21.15.116:0/2310597340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9270562d0 msgr2=0x55a92705c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9270562d0 0x55a92705c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 -- 172.21.15.116:0/2310597340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a926ccd3f0 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f909affd640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f909009b5c0 tx=0x7f9090097b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9b6141e5b2bb67 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f907affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90900a3fb0 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f909007f360 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f908c0043d0 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90900cf170 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f90a47cb740 10 client.16308 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.951+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a926d977d0 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f908c0043d0 con 0x55a927056bf0 2024-03-19T02:29:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f9090082600 con 0x55a927056bf0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 10 client.16308 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 10 client.16308 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 20 client.16308 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a92706f9a0 0x55a927071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 4 client.16308 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9270173e0 con 0x55a92706f9a0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f909b7fe640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a92706f9a0 0x55a927071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f907affd640 10 client.16308 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.952+0000 7f909b7fe640 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a92706f9a0 0x55a927071e10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ffd615d20110dee9 server_cookie=7f664cd82ce6cb02 in_seq=0 out_seq=0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f907affd640 1 -- 172.21.15.116:0/2310597340 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9270173e0 con 0x55a92706f9a0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f907affd640 10 client.16308 handle_command_reply: tid=0 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 client.16308 shutdown 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 msgr2=0x55a92705cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f909009b5c0 tx=0x7f9090097b60 comp rx=0 tx=0).stop 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 shutdown_connections 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a92705d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a92706f9a0 0x55a927071e10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a927056bf0 0x55a92705cd10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 --2- 172.21.15.116:0/2310597340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9270562d0 0x55a92705c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f907affd640 0 client.16308 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.953+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 >> 172.21.15.116:0/2310597340 conn(0x55a927050a00 msgr2=0x55a927052750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.954+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 shutdown_connections 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.954+0000 7f90a47cb740 1 -- 172.21.15.116:0/2310597340 wait complete. 2024-03-19T02:29:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.954+0000 7f90a47cb740 20 client.16308 trim_cache size 0 max 16384 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.972+0000 7f90a47cb740 1 Processor -- start 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.972+0000 7f90a47cb740 1 -- start start 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.972+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.972+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a92706b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a9270776b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f90a47cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1b0d60 con 0x55a9270562d0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f90a47cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1b28d0 con 0x55a927056bf0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f90a47cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1b1430 con 0x55a92705f7d0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54964/0 (socket says 172.21.15.116:54964) 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 -- 172.21.15.116:0/1877622314 learned_addr learned my addr 172.21.15.116:0/1877622314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909affd640 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a92706b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909bfff640 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a9270776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 -- 172.21.15.116:0/1877622314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 msgr2=0x55a9270776b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a9270776b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 -- 172.21.15.116:0/1877622314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 msgr2=0x55a92706b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a92706b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 -- 172.21.15.116:0/1877622314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a927056a20 con 0x55a9270562d0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f909b7fe640 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f908c009c70 tx=0x7f908c003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a513e3f56568fa56 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.973+0000 7f90817fa640 1 -- 172.21.15.116:0/1877622314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c009e20 con 0x55a9270562d0 2024-03-19T02:29:49.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90817fa640 1 -- 172.21.15.116:0/1877622314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f908c0054f0 con 0x55a9270562d0 2024-03-19T02:29:49.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90817fa640 1 -- 172.21.15.116:0/1877622314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c01a440 con 0x55a9270562d0 2024-03-19T02:29:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 -- 172.21.15.116:0/1877622314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 msgr2=0x55a92705ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f908c009c70 tx=0x7f908c003ea0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 -- 172.21.15.116:0/1877622314 shutdown_connections 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a92705f7d0 0x55a9270776b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a92706b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 --2- 172.21.15.116:0/1877622314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9270562d0 0x55a92705ef00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 -- 172.21.15.116:0/1877622314 >> 172.21.15.116:0/1877622314 conn(0x55a927050a00 msgr2=0x55a92704f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 -- 172.21.15.116:0/1877622314 shutdown_connections 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.974+0000 7f90a47cb740 1 -- 172.21.15.116:0/1877622314 wait complete. 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 -- start start 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 Processor -- start 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9270562d0 0x55a927059490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a9270599d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1b21d0 con 0x55a92705f7d0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1b1b00 con 0x55a927056bf0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f909c1aed40 con 0x55a9270562d0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f90a47cb740 10 client.0 _ll_register_callbacks cb 0x7f909009b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f909bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f909bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54976/0 (socket says 172.21.15.116:54976) 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f909bfff640 1 -- 172.21.15.116:0/566372788 learned_addr learned my addr 172.21.15.116:0/566372788 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f909affd640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a9270599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.975+0000 7f909b7fe640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9270562d0 0x55a927059490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 -- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9270562d0 msgr2=0x55a927059490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9270562d0 0x55a927059490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 -- 172.21.15.116:0/566372788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 msgr2=0x55a9270599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a9270599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 -- 172.21.15.116:0/566372788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9270d4de0 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f909bfff640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9094037040 tx=0x7f90940095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56363082a67e2c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094055020 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9094004450 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f908c0054f0 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9094039040 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f90a47cb740 10 client.16314 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a9270d60e0 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f908c0054f0 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.976+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f909403c070 con 0x55a92705f7d0 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 10 client.16314 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 10 client.16314 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 20 client.16314 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a9270d68e0 0x55a9270d8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 4 client.16314 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a926fc69f0 con 0x55a9270d68e0 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f909b7fe640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a9270d68e0 0x55a9270d8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f907affd640 10 client.16314 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f909b7fe640 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a9270d68e0 0x55a9270d8d00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dfa75576471dcda8 server_cookie=aa2ad38e1aa58cc9 in_seq=0 out_seq=0 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f907affd640 1 -- 172.21.15.116:0/566372788 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a926fc69f0 con 0x55a9270d68e0 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f907affd640 10 client.16314 handle_command_reply: tid=0 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.977+0000 7f90a47cb740 1 client.16314 shutdown 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 msgr2=0x55a927059f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9094037040 tx=0x7f90940095c0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 shutdown_connections 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55a9270d68e0 0x55a9270d8d00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f907affd640 0 client.16314 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a92705f7d0 0x55a927059f10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a927056bf0 0x55a9270599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 --2- 172.21.15.116:0/566372788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9270562d0 0x55a927059490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 >> 172.21.15.116:0/566372788 conn(0x55a927050a00 msgr2=0x55a92705d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 shutdown_connections 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 1 -- 172.21.15.116:0/566372788 wait complete. 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a47cb740 20 client.16314 trim_cache size 0 max 16384 2024-03-19T02:29:49.985 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 msgr2=0x7f90840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 0x7f90840a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55a927051b90 tx=0x7f9094032340 comp rx=0 tx=0).stop 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 msgr2=0x7f909c0fdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 0x7f909c0fdbb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f909000e9f0 tx=0x7f909000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 shutdown_connections 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90840a2970 0x7f90840a4e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f909c1066b0 0x7f909c0fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f909c101060 0x7f909c0fdbb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 --2- 172.21.15.116:0/3271284673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909c1006a0 0x7f909c0fd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.978+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 >> 172.21.15.116:0/3271284673 conn(0x7f909c0778d0 msgr2=0x7f909c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.979+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 shutdown_connections 2024-03-19T02:29:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:49.979+0000 7f90a1b0c640 1 -- 172.21.15.116:0/3271284673 wait complete. 2024-03-19T02:29:49.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:49 smithi116 ceph-mon[32263]: pgmap v63: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:49.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:49 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2951396137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:49.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- 172.21.15.116:0/1019048401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1017a0 msgr2=0x7f2c6c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/1019048401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c101c20 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009930 tx=0x7f2c5802bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- 172.21.15.116:0/1019048401 shutdown_connections 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/1019048401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/1019048401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c101c20 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/1019048401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- 172.21.15.116:0/1019048401 >> 172.21.15.116:0/1019048401 conn(0x7f2c6c0fd2d0 msgr2=0x7f2c6c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- 172.21.15.116:0/1019048401 shutdown_connections 2024-03-19T02:29:50.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- 172.21.15.116:0/1019048401 wait complete. 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 Processor -- start 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- start start 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c111a30 con 0x7f2c6c1077b0 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1118b0 con 0x7f2c6c102160 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.105+0000 7f2c72dc6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c111bb0 con 0x7f2c6c1017a0 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c70b62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54998/0 (socket says 172.21.15.116:54998) 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c70b62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38044/0 (socket says 172.21.15.116:38044) 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 -- 172.21.15.116:0/2562925102 learned_addr learned my addr 172.21.15.116:0/2562925102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41060/0 (socket says 172.21.15.116:41060) 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 -- 172.21.15.116:0/2562925102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 msgr2=0x7f2c6c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 -- 172.21.15.116:0/2562925102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 msgr2=0x7f2c6c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 -- 172.21.15.116:0/2562925102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c19cf80 con 0x7f2c6c1077b0 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c71363640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400eb10 tx=0x7f2c6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c69ffb640 1 -- 172.21.15.116:0/2562925102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64004b20 con 0x7f2c6c1077b0 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c69ffb640 1 -- 172.21.15.116:0/2562925102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c64004cc0 con 0x7f2c6c1077b0 2024-03-19T02:29:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c19d270 con 0x7f2c6c1077b0 2024-03-19T02:29:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.106+0000 7f2c69ffb640 1 -- 172.21.15.116:0/2562925102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64010690 con 0x7f2c6c1077b0 2024-03-19T02:29:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.107+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c1a4b00 con 0x7f2c6c1077b0 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.109+0000 7f2c69ffb640 1 -- 172.21.15.116:0/2562925102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2c64005020 con 0x7f2c6c1077b0 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.109+0000 7f2c759b1740 1 Processor -- start 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.109+0000 7f2c69ffb640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 0x7f2c5c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.109+0000 7f2c759b1740 1 -- start start 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.109+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 0x7f2c5c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c69ffb640 1 -- 172.21.15.116:0/2562925102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2c640cf6e0 con 0x7f2c6c1077b0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 0x55a34fe8c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe97bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a34fdef4b0 con 0x55a34fe957d0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a34fd97d70 con 0x55a34fe8c2d0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c759b1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a34fdeb550 con 0x55a34fe8cbf0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe97bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38050/0 (socket says 172.21.15.116:38050) 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c6bfff640 1 -- 172.21.15.116:0/1920046288 learned_addr learned my addr 172.21.15.116:0/1920046288 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c70b62640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 0x55a34fe8c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 0x7f2c5c0a4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c580096c0 tx=0x7f2c580042d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 -- 172.21.15.116:0/1920046288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 msgr2=0x55a34fe95290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 -- 172.21.15.116:0/1920046288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 msgr2=0x55a34fe8c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 0x55a34fe8c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 -- 172.21.15.116:0/1920046288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a34fdba210 con 0x55a34fe957d0 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c70b62640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 0x55a34fe8c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c71363640 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe97bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ccab0 tx=0x7f2c6409b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9267fcd40871f300 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c56ffd640 1 -- 172.21.15.116:0/1920046288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6409b9c0 con 0x55a34fe957d0 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c56ffd640 1 -- 172.21.15.116:0/1920046288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c6409fb10 con 0x55a34fe957d0 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.110+0000 7f2c56ffd640 1 -- 172.21.15.116:0/1920046288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640981c0 con 0x55a34fe957d0 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 -- 172.21.15.116:0/1920046288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 msgr2=0x55a34fe97bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe97bc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ccab0 tx=0x7f2c6409b5c0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 -- 172.21.15.116:0/1920046288 shutdown_connections 2024-03-19T02:29:50.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe97bc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8cbf0 0x55a34fe95290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 --2- 172.21.15.116:0/1920046288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8c2d0 0x55a34fe8c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 -- 172.21.15.116:0/1920046288 >> 172.21.15.116:0/1920046288 conn(0x55a34fe86a00 msgr2=0x55a34fe86de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 -- 172.21.15.116:0/1920046288 shutdown_connections 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.111+0000 7f2c759b1740 1 -- 172.21.15.116:0/1920046288 wait complete. 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 -- start start 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 Processor -- start 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34fe92d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a34fe98560 con 0x55a34fe957d0 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c70b62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a34fdab310 con 0x55a34fe8cbf0 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34fe92d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55024/0 (socket says 172.21.15.116:55024) 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c71363640 1 -- 172.21.15.116:0/3880076886 learned_addr learned my addr 172.21.15.116:0/3880076886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.112+0000 7f2c759b1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a34fe986e0 con 0x55a34fe8c2d0 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c759b1740 10 client.0 _ll_register_callbacks cb 0x55a34fd9f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 -- 172.21.15.116:0/3880076886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 msgr2=0x55a34fe927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 -- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 msgr2=0x55a34fe92d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34fe92d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 -- 172.21.15.116:0/3880076886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a34fb033f0 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34fe92d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c70b62640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c71363640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ccb60 tx=0x7f2c6409b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da8a1c4bd88ad26 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640cf530 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2c64099cc0 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.113+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6409fb10 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2c640981a0 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 10 client.16326 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a34fbcd7d0 con 0x55a34fe957d0 2024-03-19T02:29:50.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c6409fb10 con 0x55a34fe957d0 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f2c64005310 con 0x55a34fe957d0 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 10 client.16326 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 10 client.16326 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 20 client.16326 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34fea5910 0x55a34fea7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 4 client.16326 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a34fe4d3e0 con 0x55a34fea5910 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.114+0000 7f2c70b62640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34fea5910 0x55a34fea7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.115+0000 7f2c4bfff640 10 client.16326 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.115+0000 7f2c70b62640 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34fea5910 0x55a34fea7d00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4c78e405f7ded5d server_cookie=fa643df8acb427f4 in_seq=0 out_seq=0 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.115+0000 7f2c4bfff640 1 -- 172.21.15.116:0/3880076886 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a34fe4d3e0 con 0x55a34fea5910 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.115+0000 7f2c4bfff640 10 client.16326 handle_command_reply: tid=0 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.115+0000 7f2c759b1740 1 client.16326 shutdown 2024-03-19T02:29:50.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 msgr2=0x55a34fe93260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2c640ccb60 tx=0x7f2c6409b630 comp rx=0 tx=0).stop 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 shutdown_connections 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34fea5910 0x55a34fea7d00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe93260 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34fe92d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 --2- 172.21.15.116:0/3880076886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 >> 172.21.15.116:0/3880076886 conn(0x55a34fe86a00 msgr2=0x55a34fe88750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c4bfff640 0 client.16326 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 shutdown_connections 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 1 -- 172.21.15.116:0/3880076886 wait complete. 2024-03-19T02:29:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.116+0000 7f2c759b1740 20 client.16326 trim_cache size 0 max 16384 2024-03-19T02:29:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 Processor -- start 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 -- start start 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe94610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1aec00 con 0x55a34fe957d0 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1b0770 con 0x55a34fe8cbf0 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c759b1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1af2d0 con 0x55a34fe8c2d0 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41088/0 (socket says 172.21.15.116:41088) 2024-03-19T02:29:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c71363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55034/0 (socket says 172.21.15.116:55034) 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 -- 172.21.15.116:0/592432047 learned_addr learned my addr 172.21.15.116:0/592432047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c70b62640 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe94610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 -- 172.21.15.116:0/592432047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 msgr2=0x55a34fe94610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe94610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 -- 172.21.15.116:0/592432047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 msgr2=0x55a34fe8fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c6bfff640 1 -- 172.21.15.116:0/592432047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a34fe86830 con 0x55a34fe8cbf0 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.131+0000 7f2c71363640 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.132+0000 7f2c6bfff640 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2c58006010 tx=0x7f2c58046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40f3d4558c61312 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.132+0000 7f2c567fc640 1 -- 172.21.15.116:0/592432047 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5802cc70 con 0x55a34fe8cbf0 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.132+0000 7f2c567fc640 1 -- 172.21.15.116:0/592432047 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c58052070 con 0x55a34fe8cbf0 2024-03-19T02:29:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.132+0000 7f2c567fc640 1 -- 172.21.15.116:0/592432047 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5804e8e0 con 0x55a34fe8cbf0 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- 172.21.15.116:0/592432047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 msgr2=0x55a34feaa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2c58006010 tx=0x7f2c58046000 comp rx=0 tx=0).stop 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- 172.21.15.116:0/592432047 shutdown_connections 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe957d0 0x55a34fe8fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaa940 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- 172.21.15.116:0/592432047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe8c2d0 0x55a34fe94610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- 172.21.15.116:0/592432047 >> 172.21.15.116:0/592432047 conn(0x55a34fe86a00 msgr2=0x55a34fe93f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- 172.21.15.116:0/592432047 shutdown_connections 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- 172.21.15.116:0/592432047 wait complete. 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- start start 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 Processor -- start 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 0x55a34feaa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 0x55a34fea7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1b0070 con 0x55a34fe8c2d0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1af9a0 con 0x55a34fe8cbf0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.133+0000 7f2c759b1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c1acbe0 con 0x55a34fe957d0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c759b1740 10 client.0 _ll_register_callbacks cb 0x7f2c600097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c70b62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 0x55a34feaa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 0x55a34fea7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c70b62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 0x55a34feaa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55040/0 (socket says 172.21.15.116:55040) 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c6bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41094/0 (socket says 172.21.15.116:41094) 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c70b62640 1 -- 172.21.15.116:0/2521546920 learned_addr learned my addr 172.21.15.116:0/2521546920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 -- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 msgr2=0x55a34feaab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 -- 172.21.15.116:0/2521546920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 msgr2=0x55a34feaa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 0x55a34feaa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 -- 172.21.15.116:0/2521546920 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a34fea8200 con 0x55a34fe957d0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c71363640 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 0x55a34fea7a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2c64099c70 tx=0x7f2c64097a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29e87e7b15ee2ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6409b790 con 0x55a34fe957d0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c58052070 con 0x55a34fe957d0 2024-03-19T02:29:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2c64099cc0 con 0x55a34fe957d0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2c640cf530 con 0x55a34fe957d0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c759b1740 10 client.26095 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.134+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a34fea8200 con 0x55a34fe957d0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c58052070 con 0x55a34fe957d0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f2c64086220 con 0x55a34fe957d0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 10 client.26095 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 10 client.26095 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 20 client.26095 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34ff0b710 0x55a34ff0db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 4 client.26095 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a34fdfc9f0 con 0x55a34ff0b710 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c70b62640 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34ff0b710 0x55a34ff0db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c4bfff640 10 client.26095 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.135+0000 7f2c70b62640 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34ff0b710 0x55a34ff0db00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a1237207b7b45f1 server_cookie=16bf86aa15464420 in_seq=0 out_seq=0 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2521546920 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a34fdfc9f0 con 0x55a34ff0b710 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c4bfff640 10 client.26095 handle_command_reply: tid=0 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 client.26095 shutdown 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 msgr2=0x55a34fea7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 0x55a34fea7a00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2c64099c70 tx=0x7f2c64097a80 comp rx=0 tx=0).stop 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 shutdown_connections 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a34ff0b710 0x55a34ff0db00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a34fe957d0 0x55a34fea7a00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a34fe8cbf0 0x55a34feaab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 --2- 172.21.15.116:0/2521546920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34fe8c2d0 0x55a34feaa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 >> 172.21.15.116:0/2521546920 conn(0x55a34fe86a00 msgr2=0x55a34fe93030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c4bfff640 0 client.26095 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 shutdown_connections 2024-03-19T02:29:50.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 1 -- 172.21.15.116:0/2521546920 wait complete. 2024-03-19T02:29:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.136+0000 7f2c759b1740 20 client.26095 trim_cache size 0 max 16384 2024-03-19T02:29:50.144 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 msgr2=0x7f2c5c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 0x7f2c5c0a4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c580096c0 tx=0x7f2c580042d0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 msgr2=0x7f2c6c19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400eb10 tx=0x7f2c6400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 shutdown_connections 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2c5c0a29f0 0x7f2c5c0a4eb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c6c1077b0 0x7f2c6c19c850 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c6c102160 0x7f2c6c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 --2- 172.21.15.116:0/2562925102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c6c1017a0 0x7f2c6c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 >> 172.21.15.116:0/2562925102 conn(0x7f2c6c0fd2d0 msgr2=0x7f2c6c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 shutdown_connections 2024-03-19T02:29:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.137+0000 7f2c72dc6640 1 -- 172.21.15.116:0/2562925102 wait complete. 2024-03-19T02:29:50.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:29:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:49 smithi162 ceph-mon[35761]: pgmap v63: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:49 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2951396137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:49 smithi154 ceph-mon[35693]: pgmap v63: 129 pgs: 129 active+clean; 46 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:29:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:49 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2951396137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/3106563984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 msgr2=0x7f8c08100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/3106563984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08100b20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009930 tx=0x7f8bf402bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/3106563984 shutdown_connections 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/3106563984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c08101060 0x7f8c0810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/3106563984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08100b20 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/3106563984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c081066b0 0x7f8c08106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/3106563984 >> 172.21.15.116:0/3106563984 conn(0x7f8c080778d0 msgr2=0x7f8c08077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/3106563984 shutdown_connections 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.264+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/3106563984 wait complete. 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 Processor -- start 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 -- start start 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 0x7f8c08197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c081066b0 0x7f8c0819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c08110970 con 0x7f8c081006a0 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081107f0 con 0x7f8c08101060 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0f1b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c08110af0 con 0x7f8c081066b0 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55056/0 (socket says 172.21.15.116:55056) 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/58808323 learned_addr learned my addr 172.21.15.116:0/58808323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c07fff640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 0x7f8c08197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0d750640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c081066b0 0x7f8c0819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/58808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c081066b0 msgr2=0x7f8c0819b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c081066b0 0x7f8c0819b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/58808323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 msgr2=0x7f8c08197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 0x7f8c08197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/58808323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0819bee0 con 0x7f8c081006a0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.265+0000 7f8c07fff640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 0x7f8c08197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e9f0 tx=0x7f8bfc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c05ffb640 1 -- 172.21.15.116:0/58808323 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc00cd80 con 0x7f8c081006a0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0819c1d0 con 0x7f8c081006a0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c05ffb640 1 -- 172.21.15.116:0/58808323 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bfc0108e0 con 0x7f8c081006a0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c05ffb640 1 -- 172.21.15.116:0/58808323 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc018920 con 0x7f8c081006a0 2024-03-19T02:29:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.266+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c081a3a60 con 0x7f8c081006a0 2024-03-19T02:29:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.268+0000 7f8c05ffb640 1 -- 172.21.15.116:0/58808323 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8bfc010a90 con 0x7f8c081006a0 2024-03-19T02:29:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.269+0000 7f8c05ffb640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 0x7f8bf80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.269+0000 7f8c11d9a740 1 Processor -- start 2024-03-19T02:29:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.269+0000 7f8c11d9a740 1 -- start start 2024-03-19T02:29:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.269+0000 7f8c05ffb640 1 -- 172.21.15.116:0/58808323 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8bfc0cf490 con 0x7f8c081006a0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.269+0000 7f8c07fff640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 0x7f8bf80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 0x55c5714004a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c571402dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c5713584b0 con 0x55c5713f5660 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c571300d70 con 0x55c5714009e0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c571354550 con 0x55c5713f4cb0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38090/0 (socket says 172.21.15.116:38090) 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/1337895668 learned_addr learned my addr 172.21.15.116:0/1337895668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c07fff640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 0x55c5714004a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c571402dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/1337895668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 msgr2=0x55c571402dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c571402dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/1337895668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 msgr2=0x55c5714004a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 0x55c5714004a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 -- 172.21.15.116:0/1337895668 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c571323210 con 0x55c5713f4cb0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c07fff640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 0x55c5714004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c07fff640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 0x7f8bf80a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4005ab0 tx=0x7f8bf40042b0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0981f0 tx=0x7f8bfc09b2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a135913682cd8fce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8bf2ffd640 1 -- 172.21.15.116:0/1337895668 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc0a3f10 con 0x55c5713f4cb0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8bf2ffd640 1 -- 172.21.15.116:0/1337895668 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bfc0ccae0 con 0x55c5713f4cb0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.270+0000 7f8bf2ffd640 1 -- 172.21.15.116:0/1337895668 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc09ac70 con 0x55c5713f4cb0 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1337895668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 msgr2=0x55c5713f5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0981f0 tx=0x7f8bfc09b2b0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1337895668 shutdown_connections 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c571402dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f5660 0x55c5714004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1337895668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c5713f5090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1337895668 >> 172.21.15.116:0/1337895668 conn(0x55c5713efa00 msgr2=0x55c5713efde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.271+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1337895668 shutdown_connections 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1337895668 wait complete. 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 -- start start 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 Processor -- start 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f4cb0 0x55c5713fc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c5713fc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c5713f6570 con 0x55c5713f4cb0 2024-03-19T02:29:50.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c571314310 con 0x55c5714009e0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c5713f66f0 con 0x55c5713f5660 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.272+0000 7f8c11d9a740 10 client.0 _ll_register_callbacks cb 0x55c5713082d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41132/0 (socket says 172.21.15.116:41132) 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 -- 172.21.15.116:0/1083303708 learned_addr learned my addr 172.21.15.116:0/1083303708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f4cb0 0x55c5713fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c07fff640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c5713fc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 -- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 msgr2=0x55c5713fc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c5713fc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 -- 172.21.15.116:0/1083303708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f4cb0 msgr2=0x55c5713fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f4cb0 0x55c5713fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 -- 172.21.15.116:0/1083303708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57106c3f0 con 0x55c5714009e0 2024-03-19T02:29:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c07fff640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c5713fc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c00011d40 tx=0x7f8c0000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34de88322805adfb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000fe60 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bfc0ccae0 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.273+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8c00004b20 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c00024050 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 10 client.26168 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5711367d0 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bfc0ccae0 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f8c00019030 con 0x55c5714009e0 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 10 client.26168 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 10 client.26168 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:29:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 20 client.26168 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571412ef0 0x55c571415330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 4 client.26168 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5713b63e0 con 0x55c571412ef0 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.274+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571412ef0 0x55c571415330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.275+0000 7f8be7fff640 10 client.26168 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.275+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571412ef0 0x55c571415330 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4030a084f0337c3d server_cookie=d6038dd6ae9f712a in_seq=0 out_seq=0 2024-03-19T02:29:50.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8be7fff640 1 -- 172.21.15.116:0/1083303708 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c5713b63e0 con 0x55c571412ef0 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8be7fff640 10 client.26168 handle_command_reply: tid=0 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 client.26168 shutdown 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 msgr2=0x55c5714087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c00011d40 tx=0x7f8c0000f790 comp rx=0 tx=0).stop 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 shutdown_connections 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571412ef0 0x55c571415330 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5714009e0 0x55c5714087c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c5713fc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1083303708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5713f4cb0 0x55c5713fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8be7fff640 0 client.26168 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 >> 172.21.15.116:0/1083303708 conn(0x55c5713efa00 msgr2=0x55c5713f1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 shutdown_connections 2024-03-19T02:29:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1083303708 wait complete. 2024-03-19T02:29:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.276+0000 7f8c11d9a740 20 client.26168 trim_cache size 0 max 16384 2024-03-19T02:29:50.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 Processor -- start 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 -- start start 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f4cb0 0x55c5714087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c57140f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081aca20 con 0x55c5714009e0 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081ae590 con 0x55c5713f4cb0 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081ad0f0 con 0x55c5713f5660 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c07fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c57140f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55094/0 (socket says 172.21.15.116:55094) 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 -- 172.21.15.116:0/1637157736 learned_addr learned my addr 172.21.15.116:0/1637157736 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f4cb0 0x55c5714087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 -- 172.21.15.116:0/1637157736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 msgr2=0x55c57140f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c57140f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 -- 172.21.15.116:0/1637157736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f4cb0 msgr2=0x55c5714087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.291+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f4cb0 0x55c5714087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.292+0000 7f8c0d750640 1 -- 172.21.15.116:0/1637157736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c571411af0 con 0x55c5714009e0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.292+0000 7f8c0d750640 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000f80 tx=0x7f8c00011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9b586a57287c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.292+0000 7f8bf27fc640 1 -- 172.21.15.116:0/1637157736 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0002c020 con 0x55c5714009e0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.292+0000 7f8bf27fc640 1 -- 172.21.15.116:0/1637157736 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c0001a5b0 con 0x55c5714009e0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.292+0000 7f8bf27fc640 1 -- 172.21.15.116:0/1637157736 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00028630 con 0x55c5714009e0 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1637157736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 msgr2=0x55c57141ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000f80 tx=0x7f8c00011980 comp rx=0 tx=0).stop 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1637157736 shutdown_connections 2024-03-19T02:29:50.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57141ac00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f5660 0x55c57140f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/1637157736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f4cb0 0x55c5714087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1637157736 >> 172.21.15.116:0/1637157736 conn(0x55c5713efa00 msgr2=0x55c5713f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1637157736 shutdown_connections 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- 172.21.15.116:0/1637157736 wait complete. 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- start start 2024-03-19T02:29:50.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 Processor -- start 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c57140b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f5660 0x55c57140b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081ade90 con 0x55c5714009e0 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081ad7c0 con 0x55c5713f5660 2024-03-19T02:29:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081aaa00 con 0x55c5713f4cb0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.293+0000 7f8c11d9a740 10 client.0 _ll_register_callbacks cb 0x7f8c00009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0cf4f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c57140b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55100/0 (socket says 172.21.15.116:55100) 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 -- 172.21.15.116:0/3579914850 learned_addr learned my addr 172.21.15.116:0/3579914850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c07fff640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f5660 0x55c57140b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 -- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 msgr2=0x55c57140b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c57140b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 -- 172.21.15.116:0/3579914850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f5660 msgr2=0x55c57140b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f5660 0x55c57140b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 -- 172.21.15.116:0/3579914850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c571478230 con 0x55c5714009e0 2024-03-19T02:29:50.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c57140b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c0d750640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000fd10 tx=0x7f8c0001a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff93ba56792a971 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00035030 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8c00017040 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c0001a5b0 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c00042050 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c11d9a740 10 client.16350 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c571479410 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.294+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c0001a5b0 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f8c00019070 con 0x55c5714009e0 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 10 client.16350 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 10 client.16350 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:29:50.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 20 client.16350 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571479a20 0x55c57147be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 4 client.16350 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5713659f0 con 0x55c571479a20 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571479a20 0x55c57147be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c0cf4f640 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571479a20 0x55c57147be40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7478d48effb45e0d server_cookie=4157514211d38cb3 in_seq=0 out_seq=0 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8be7fff640 10 client.16350 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8be7fff640 1 -- 172.21.15.116:0/3579914850 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5713659f0 con 0x55c571479a20 2024-03-19T02:29:50.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8be7fff640 10 client.16350 handle_command_reply: tid=0 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 1 client.16350 shutdown 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 msgr2=0x55c57140bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.295+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000fd10 tx=0x7f8c0001a9f0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 shutdown_connections 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c571479a20 0x55c57147be40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5714009e0 0x55c57140bca0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5713f5660 0x55c57140b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 --2- 172.21.15.116:0/3579914850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5713f4cb0 0x55c57140b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 >> 172.21.15.116:0/3579914850 conn(0x55c5713efa00 msgr2=0x55c5713f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8be7fff640 0 client.16350 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:29:50.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 shutdown_connections 2024-03-19T02:29:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 1 -- 172.21.15.116:0/3579914850 wait complete. 2024-03-19T02:29:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c11d9a740 20 client.16350 trim_cache size 0 max 16384 2024-03-19T02:29:50.305 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 msgr2=0x7f8bf80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 0x7f8bf80a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4005ab0 tx=0x7f8bf40042b0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 msgr2=0x7f8c08196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e9f0 tx=0x7f8bfc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 shutdown_connections 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8bf80a2970 0x7f8bf80a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c081066b0 0x7f8c0819b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08101060 0x7f8c08197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 --2- 172.21.15.116:0/58808323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c081006a0 0x7f8c08196ef0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 >> 172.21.15.116:0/58808323 conn(0x7f8c080778d0 msgr2=0x7f8c08102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.296+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 shutdown_connections 2024-03-19T02:29:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.297+0000 7f8c0f1b3640 1 -- 172.21.15.116:0/58808323 wait complete. 2024-03-19T02:29:50.308 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 -- 172.21.15.116:0/3437614282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4103360 msgr2=0x7f22d410b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 --2- 172.21.15.116:0/3437614282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4103360 0x7f22d410b840 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009900 tx=0x7f22c002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 -- 172.21.15.116:0/3437614282 shutdown_connections 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 --2- 172.21.15.116:0/3437614282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d410bd80 0x7f22d410e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 --2- 172.21.15.116:0/3437614282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4103360 0x7f22d410b840 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 --2- 172.21.15.116:0/3437614282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d4102a40 0x7f22d4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 -- 172.21.15.116:0/3437614282 >> 172.21.15.116:0/3437614282 conn(0x7f22d40778d0 msgr2=0x7f22d4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 -- 172.21.15.116:0/3437614282 shutdown_connections 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.422+0000 7f22db0f8640 1 -- 172.21.15.116:0/3437614282 wait complete. 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 Processor -- start 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 -- start start 2024-03-19T02:29:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d4103360 0x7f22d419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d410bd80 0x7f22d419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4111a30 con 0x7f22d4102a40 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41118b0 con 0x7f22d410bd80 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22db0f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4111bb0 con 0x7f22d4103360 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55108/0 (socket says 172.21.15.116:55108) 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22d8e94640 1 -- 172.21.15.116:0/1827543618 learned_addr learned my addr 172.21.15.116:0/1827543618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22d9695640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d410bd80 0x7f22d419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.423+0000 7f22d3fff640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d4103360 0x7f22d419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 -- 172.21.15.116:0/1827543618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d4103360 msgr2=0x7f22d419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d4103360 0x7f22d419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 -- 172.21.15.116:0/1827543618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d410bd80 msgr2=0x7f22d419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d410bd80 0x7f22d419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 -- 172.21.15.116:0/1827543618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d419f130 con 0x7f22d4102a40 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d8e94640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f22c800e9f0 tx=0x7f22c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d1ffb640 1 -- 172.21.15.116:0/1827543618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c800cd80 con 0x7f22d4102a40 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d1ffb640 1 -- 172.21.15.116:0/1827543618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22c80108e0 con 0x7f22d4102a40 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d419f420 con 0x7f22d4102a40 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22d1ffb640 1 -- 172.21.15.116:0/1827543618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8018920 con 0x7f22d4102a40 2024-03-19T02:29:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.424+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d41a6cb0 con 0x7f22d4102a40 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.427+0000 7f22d1ffb640 1 -- 172.21.15.116:0/1827543618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f22c8010a90 con 0x7f22d4102a40 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.427+0000 7f22dddc0740 1 Processor -- start 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.427+0000 7f22dddc0740 1 -- start start 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06feee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06fef9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a06fe514b0 con 0x55a06feeebf0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a06fdf9d70 con 0x55a06fef77d0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22dddc0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a06fe4d550 con 0x55a06feee2d0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06feee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55112/0 (socket says 172.21.15.116:55112) 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d8e94640 1 -- 172.21.15.116:0/1223233558 learned_addr learned my addr 172.21.15.116:0/1223233558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d3fff640 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06fef9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d1ffb640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 0x7f22c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 -- 172.21.15.116:0/1223233558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 msgr2=0x55a06fef9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06fef9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 -- 172.21.15.116:0/1223233558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 msgr2=0x55a06fef7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 -- 172.21.15.116:0/1223233558 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06fe1c210 con 0x55a06feee2d0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d8e94640 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 0x7f22c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d1ffb640 1 -- 172.21.15.116:0/1827543618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22c80cf440 con 0x7f22d4102a40 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22d9695640 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06feee6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009fa0 tx=0x7f22cc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64ba46996ffaa470 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.428+0000 7f22beffd640 1 -- 172.21.15.116:0/1223233558 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc017050 con 0x55a06feee2d0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22beffd640 1 -- 172.21.15.116:0/1223233558 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22cc00fb90 con 0x55a06feee2d0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22beffd640 1 -- 172.21.15.116:0/1223233558 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc01b790 con 0x55a06feee2d0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 -- 172.21.15.116:0/1223233558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 msgr2=0x55a06feee6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22d9695640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 0x7f22c40a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22cc022000 tx=0x7f22cc023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06feee6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009fa0 tx=0x7f22cc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 -- 172.21.15.116:0/1223233558 shutdown_connections 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06fef9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 --2- 172.21.15.116:0/1223233558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06feee6b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 -- 172.21.15.116:0/1223233558 >> 172.21.15.116:0/1223233558 conn(0x55a06fee8a00 msgr2=0x55a06fee8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 -- 172.21.15.116:0/1223233558 shutdown_connections 2024-03-19T02:29:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.429+0000 7f22dddc0740 1 -- 172.21.15.116:0/1223233558 wait complete. 2024-03-19T02:29:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 -- start start 2024-03-19T02:29:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 Processor -- start 2024-03-19T02:29:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 0x55a06fef4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06fef4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a06fefa5c0 con 0x55a06fef77d0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a06fe0d310 con 0x55a06feee2d0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a06fefa740 con 0x55a06feeebf0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22dddc0740 10 client.0 _ll_register_callbacks cb 0x55a06fe012d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d9695640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 0x55a06fef4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d9695640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55128/0 (socket says 172.21.15.116:55128) 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 0x55a06fef4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41202/0 (socket says 172.21.15.116:41202) 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d9695640 1 -- 172.21.15.116:0/167173220 learned_addr learned my addr 172.21.15.116:0/167173220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.430+0000 7f22d3fff640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06fef4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 -- 172.21.15.116:0/167173220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 msgr2=0x55a06fef4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06fef4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 -- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 msgr2=0x55a06fef4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 0x55a06fef4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 -- 172.21.15.116:0/167173220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06fb653f0 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22d9695640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009700 tx=0x7f22cc00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933aef53b7838cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc034060 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22cc00fb90 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22cc01bcb0 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f22cc03b050 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22dddc0740 10 client.16362 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a06fc2f7d0 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.431+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22cc00fb90 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f22cc01a070 con 0x55a06fef77d0 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 10 client.16362 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 20 client.16362 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff07a30 0x55a06ff09e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 4 client.16362 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a06feaf3e0 con 0x55a06ff07a30 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22d8e94640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff07a30 0x55a06ff09e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22d8e94640 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff07a30 0x55a06ff09e20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f055adefd0c9e009 server_cookie=c68bc6f96b3c8cc5 in_seq=0 out_seq=0 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.432+0000 7f22b3fff640 10 client.16362 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22b3fff640 1 -- 172.21.15.116:0/167173220 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a06feaf3e0 con 0x55a06ff07a30 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22b3fff640 10 client.16362 handle_command_reply: tid=0 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 client.16362 shutdown 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 msgr2=0x55a06fef51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009700 tx=0x7f22cc00ff70 comp rx=0 tx=0).stop 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 shutdown_connections 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff07a30 0x55a06ff09e20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06fef77d0 0x55a06fef51d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22b3fff640 0 client.16362 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06fef4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 --2- 172.21.15.116:0/167173220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06feee2d0 0x55a06fef4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 >> 172.21.15.116:0/167173220 conn(0x55a06fee8a00 msgr2=0x55a06feea750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 shutdown_connections 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 1 -- 172.21.15.116:0/167173220 wait complete. 2024-03-19T02:29:50.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.434+0000 7f22dddc0740 20 client.16362 trim_cache size 0 max 16384 2024-03-19T02:29:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 Processor -- start 2024-03-19T02:29:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 -- start start 2024-03-19T02:29:50.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06ff03360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff0f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41b0d20 con 0x55a06feee2d0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41b2890 con 0x55a06fef77d0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22dddc0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41b13f0 con 0x55a06feeebf0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06ff03360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55142/0 (socket says 172.21.15.116:55142) 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 -- 172.21.15.116:0/907000066 learned_addr learned my addr 172.21.15.116:0/907000066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d9695640 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff0f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 -- 172.21.15.116:0/907000066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 msgr2=0x55a06ff03360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06ff03360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 -- 172.21.15.116:0/907000066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 msgr2=0x55a06ff0f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff0f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d8e94640 1 -- 172.21.15.116:0/907000066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a06ff0a3e0 con 0x55a06feee2d0 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.450+0000 7f22d3fff640 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06ff03360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.451+0000 7f22d8e94640 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f22c800c6a0 tx=0x7f22c80d19c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7393a493f7811a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.451+0000 7f22be7fc640 1 -- 172.21.15.116:0/907000066 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c8010de0 con 0x55a06feee2d0 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.451+0000 7f22be7fc640 1 -- 172.21.15.116:0/907000066 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22c809b870 con 0x55a06feee2d0 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.451+0000 7f22be7fc640 1 -- 172.21.15.116:0/907000066 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c80ccb30 con 0x55a06feee2d0 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.451+0000 7f22dddc0740 1 -- 172.21.15.116:0/907000066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 msgr2=0x55a06fef6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f22c800c6a0 tx=0x7f22c80d19c0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 -- 172.21.15.116:0/907000066 shutdown_connections 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff0f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feeebf0 0x55a06ff03360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- 172.21.15.116:0/907000066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feee2d0 0x55a06fef6e80 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 -- 172.21.15.116:0/907000066 >> 172.21.15.116:0/907000066 conn(0x55a06fee8a00 msgr2=0x55a06fee9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 -- 172.21.15.116:0/907000066 shutdown_connections 2024-03-19T02:29:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 -- 172.21.15.116:0/907000066 wait complete. 2024-03-19T02:29:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 -- start start 2024-03-19T02:29:50.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 Processor -- start 2024-03-19T02:29:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06fef1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.452+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff6cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41b2190 con 0x55a06feeebf0 2024-03-19T02:29:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41b1ac0 con 0x55a06fef77d0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22d41aed00 con 0x55a06feee2d0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55144/0 (socket says 172.21.15.116:55144) 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 -- 172.21.15.116:0/3933414740 learned_addr learned my addr 172.21.15.116:0/3933414740 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d9695640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff6cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d8e94640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06fef1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 10 client.0 _ll_register_callbacks cb 0x7f22c80d2ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 -- 172.21.15.116:0/3933414740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 msgr2=0x55a06fef1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06fef1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 -- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 msgr2=0x55a06ff6cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff6cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 -- 172.21.15.116:0/3933414740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a06febe880 con 0x55a06feeebf0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22d3fff640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009900 tx=0x7f22c0005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fcad5013be381 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c002c880 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.453+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c809b870 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22c002ca20 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f22c0053050 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 10 client.16368 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a06ff6e1c0 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22c809b870 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f22c00035f0 con 0x55a06feeebf0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 10 client.16368 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 20 client.16368 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff6e950 0x55a06ff70d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 4 client.16368 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a06fe5e9f0 con 0x55a06ff6e950 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.454+0000 7f22d8e94640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff6e950 0x55a06ff70d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.455+0000 7f22b3fff640 10 client.16368 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.455+0000 7f22d8e94640 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff6e950 0x55a06ff70d70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf7be79fc183ada server_cookie=fc21e8e5bbfed93f in_seq=0 out_seq=0 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22b3fff640 1 -- 172.21.15.116:0/3933414740 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a06fe5e9f0 con 0x55a06ff6e950 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22b3fff640 10 client.16368 handle_command_reply: tid=0 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 client.16368 shutdown 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 msgr2=0x55a06fef1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009900 tx=0x7f22c0005250 comp rx=0 tx=0).stop 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 shutdown_connections 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a06ff6e950 0x55a06ff70d70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22b3fff640 0 client.16368 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a06fef77d0 0x55a06ff6cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a06feeebf0 0x55a06fef1ea0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 --2- 172.21.15.116:0/3933414740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a06feee2d0 0x55a06fef1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.456+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 >> 172.21.15.116:0/3933414740 conn(0x55a06fee8a00 msgr2=0x55a06fee7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 shutdown_connections 2024-03-19T02:29:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22dddc0740 1 -- 172.21.15.116:0/3933414740 wait complete. 2024-03-19T02:29:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22dddc0740 20 client.16368 trim_cache size 0 max 16384 2024-03-19T02:29:50.462 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 msgr2=0x7f22c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 0x7f22c40a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22cc022000 tx=0x7f22cc023000 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 msgr2=0x7f22d419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f22c800e9f0 tx=0x7f22c800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 shutdown_connections 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22c40a2970 0x7f22c40a4e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.457+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22d410bd80 0x7f22d419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.458+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22d4103360 0x7f22d419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.458+0000 7f22db0f8640 1 --2- 172.21.15.116:0/1827543618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22d4102a40 0x7f22d419a190 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.458+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 >> 172.21.15.116:0/1827543618 conn(0x7f22d40778d0 msgr2=0x7f22d4107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.458+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 shutdown_connections 2024-03-19T02:29:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:50.458+0000 7f22db0f8640 1 -- 172.21.15.116:0/1827543618 wait complete. 2024-03-19T02:29:50.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 45206 3.54 MB/sec warmup 73 sec latency 20.541 ms 2024-03-19T02:29:51.474 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:29:51.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:29:51.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:29:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- 172.21.15.116:0/1421074889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c100690 msgr2=0x7f009c100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- 172.21.15.116:0/1421074889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c100690 0x7f009c100b10 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0084009930 tx=0x7f008402bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- 172.21.15.116:0/1421074889 shutdown_connections 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- 172.21.15.116:0/1421074889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- 172.21.15.116:0/1421074889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c100690 0x7f009c100b10 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- 172.21.15.116:0/1421074889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c1066a0 0x7f009c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- 172.21.15.116:0/1421074889 >> 172.21.15.116:0/1421074889 conn(0x7f009c077820 msgr2=0x7f009c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- 172.21.15.116:0/1421074889 shutdown_connections 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- 172.21.15.116:0/1421074889 wait complete. 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 Processor -- start 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- start start 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 0x7f009c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 0x7f009c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f009c110960 con 0x7f009c1066a0 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1107e0 con 0x7f009c100690 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.587+0000 7f00a14bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f009c110ae0 con 0x7f009c101050 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 0x7f009c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 0x7f009c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f0093fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 0x7f009c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41244/0 (socket says 172.21.15.116:41244) 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009affd640 1 -- 172.21.15.116:0/3314097470 learned_addr learned my addr 172.21.15.116:0/3314097470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 -- 172.21.15.116:0/3314097470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 msgr2=0x7f009c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 -- 172.21.15.116:0/3314097470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 msgr2=0x7f009c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 0x7f009c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 -- 172.21.15.116:0/3314097470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c19cfe0 con 0x7f009c1066a0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f0093fff640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 0x7f009c19c8b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0094009fd0 tx=0x7f009400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f0098ff9640 1 -- 172.21.15.116:0/3314097470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009400ce20 con 0x7f009c1066a0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f0098ff9640 1 -- 172.21.15.116:0/3314097470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0094005020 con 0x7f009c1066a0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c19d2d0 con 0x7f009c1066a0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f0098ff9640 1 -- 172.21.15.116:0/3314097470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094010690 con 0x7f009c1066a0 2024-03-19T02:29:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.588+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c1a4b60 con 0x7f009c1066a0 2024-03-19T02:29:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.590+0000 7f0098ff9640 1 -- 172.21.15.116:0/3314097470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f00940051d0 con 0x7f009c1066a0 2024-03-19T02:29:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 Processor -- start 2024-03-19T02:29:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 -- start start 2024-03-19T02:29:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f18340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f1ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0098ff9640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 0x7f00880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d79e73160 con 0x562d79f0fca0 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d79e1be10 con 0x562d79f0f380 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f00a4182740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562d79e6f200 con 0x562d79f18880 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f009affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f18340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0093fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0093fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f0f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41254/0 (socket says 172.21.15.116:41254) 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f1ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0093fff640 1 -- 172.21.15.116:0/1796512169 learned_addr learned my addr 172.21.15.116:0/1796512169 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0093fff640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 0x7f00880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.591+0000 7f0098ff9640 1 -- 172.21.15.116:0/3314097470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f00940cf830 con 0x7f009c1066a0 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 -- 172.21.15.116:0/1796512169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 msgr2=0x562d79f0f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f0f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 -- 172.21.15.116:0/1796512169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 msgr2=0x562d79f18340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f18340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 -- 172.21.15.116:0/1796512169 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d79c4f7e0 con 0x562d79f18880 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f009b7fe640 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f1ac70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f00940ccc00 tx=0x7f009409b530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=957b83dd6a30db67 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f0093fff640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 0x7f00880a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0084009900 tx=0x7f0084005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f00917fa640 1 -- 172.21.15.116:0/1796512169 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009409bb10 con 0x562d79f18880 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f00917fa640 1 -- 172.21.15.116:0/1796512169 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f009409fc60 con 0x562d79f18880 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.592+0000 7f00917fa640 1 -- 172.21.15.116:0/1796512169 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940b35c0 con 0x562d79f18880 2024-03-19T02:29:51.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- 172.21.15.116:0/1796512169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 msgr2=0x562d79f1ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f1ac70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f00940ccc00 tx=0x7f009409b530 comp rx=0 tx=0).stop 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- 172.21.15.116:0/1796512169 shutdown_connections 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f1ac70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f18340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- 172.21.15.116:0/1796512169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- 172.21.15.116:0/1796512169 >> 172.21.15.116:0/1796512169 conn(0x562d79f09ab0 msgr2=0x562d79f09e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- 172.21.15.116:0/1796512169 shutdown_connections 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- 172.21.15.116:0/1796512169 wait complete. 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- start start 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 Processor -- start 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0f380 0x562d79f15860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0fca0 0x562d79f15da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.593+0000 7f00a4182740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d79f1b670 con 0x562d79f18880 2024-03-19T02:29:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d79e2f1a0 con 0x562d79f0fca0 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562d79f1b7f0 con 0x562d79f0f380 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 10 client.0 _ll_register_callbacks cb 0x562d79b86c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55188/0 (socket says 172.21.15.116:55188) 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 -- 172.21.15.116:0/3663754155 learned_addr learned my addr 172.21.15.116:0/3663754155 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0f380 0x562d79f15860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f0093fff640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0fca0 0x562d79f15da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 -- 172.21.15.116:0/3663754155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0f380 msgr2=0x562d79f15860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0f380 0x562d79f15860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 -- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0fca0 msgr2=0x562d79f15da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0fca0 0x562d79f15da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 -- 172.21.15.116:0/3663754155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d79caff70 con 0x562d79f18880 2024-03-19T02:29:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f009409baa0 tx=0x7f009400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acda7eadc6017aa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00827fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940b8020 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f00940ba2c0 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f009409fc60 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f00940cf670 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 10 client.16380 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.594+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d79b47170 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f009409fc60 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f00940a3e20 con 0x562d79f18880 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 10 client.16380 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 20 client.16380 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f28a40 0x562d79f2ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 4 client.16380 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d79e806a0 con 0x562d79f28a40 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f009affd640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f28a40 0x562d79f2ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f009affd640 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f28a40 0x562d79f2ae30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c88f2abbbc43aae9 server_cookie=5018654322dfe71f in_seq=0 out_seq=0 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.595+0000 7f00827fc640 10 client.16380 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00827fc640 1 -- 172.21.15.116:0/3663754155 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d79e806a0 con 0x562d79f28a40 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00827fc640 10 client.16380 handle_command_reply: tid=0 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 client.16380 shutdown 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 msgr2=0x562d79f162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f009409baa0 tx=0x7f009400c880 comp rx=0 tx=0).stop 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 shutdown_connections 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f28a40 0x562d79f2ae30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f162e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00827fc640 0 client.16380 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0fca0 0x562d79f15da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 --2- 172.21.15.116:0/3663754155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0f380 0x562d79f15860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 >> 172.21.15.116:0/3663754155 conn(0x562d79f09ab0 msgr2=0x562d79f0b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 shutdown_connections 2024-03-19T02:29:51.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 1 -- 172.21.15.116:0/3663754155 wait complete. 2024-03-19T02:29:51.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.597+0000 7f00a4182740 20 client.16380 trim_cache size 0 max 16384 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.617+0000 7f00a4182740 1 Processor -- start 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.617+0000 7f00a4182740 1 -- start start 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f17f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0fca0 0x562d79f24150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1aeb90 con 0x562d79f18880 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1b0700 con 0x562d79f0f380 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f00a4182740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1af260 con 0x562d79f0fca0 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55194/0 (socket says 172.21.15.116:55194) 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 -- 172.21.15.116:0/3623500976 learned_addr learned my addr 172.21.15.116:0/3623500976 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f0093fff640 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0fca0 0x562d79f24150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009affd640 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f17f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 -- 172.21.15.116:0/3623500976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0fca0 msgr2=0x562d79f24150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0fca0 0x562d79f24150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 -- 172.21.15.116:0/3623500976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 msgr2=0x562d79f17f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f17f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 -- 172.21.15.116:0/3623500976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d79f0e6a0 con 0x562d79f18880 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f009b7fe640 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f009409b810 tx=0x7f0094005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf239c9e7996d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.618+0000 7f0090ff9640 1 -- 172.21.15.116:0/3623500976 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940b8030 con 0x562d79f18880 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f0090ff9640 1 -- 172.21.15.116:0/3623500976 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f009409fc60 con 0x562d79f18880 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f0090ff9640 1 -- 172.21.15.116:0/3623500976 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940ba2c0 con 0x562d79f18880 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 -- 172.21.15.116:0/3623500976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 msgr2=0x562d79f2de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f009409b810 tx=0x7f0094005f90 comp rx=0 tx=0).stop 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 -- 172.21.15.116:0/3623500976 shutdown_connections 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f18880 0x562d79f2de50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f0fca0 0x562d79f24150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 --2- 172.21.15.116:0/3623500976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f17f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 -- 172.21.15.116:0/3623500976 >> 172.21.15.116:0/3623500976 conn(0x562d79f09ab0 msgr2=0x562d79f23a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 -- 172.21.15.116:0/3623500976 shutdown_connections 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.619+0000 7f00a4182740 1 -- 172.21.15.116:0/3623500976 wait complete. 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 -- start start 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 Processor -- start 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f2b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f2c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1b0000 con 0x562d79f0fca0 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1af930 con 0x562d79f0f380 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1acb70 con 0x562d79f18880 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f00a4182740 10 client.0 _ll_register_callbacks cb 0x7f008c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f009affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f2b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55204/0 (socket says 172.21.15.116:55204) 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 -- 172.21.15.116:0/2004217972 learned_addr learned my addr 172.21.15.116:0/2004217972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f009b7fe640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f2c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 -- 172.21.15.116:0/2004217972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 msgr2=0x562d79f2c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f2c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 -- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 msgr2=0x562d79f2b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f2b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f0093fff640 1 -- 172.21.15.116:0/2004217972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d79f8dc30 con 0x562d79f0fca0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.620+0000 7f009affd640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f2b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f0093fff640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0084009930 tx=0x7f008402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45fd979d001638a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084003d10 con 0x562d79f0fca0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f008405c480 con 0x562d79f0fca0 2024-03-19T02:29:51.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f009409fc60 con 0x562d79f0fca0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0084065050 con 0x562d79f0fca0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 10 client.16386 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562d79f8ef10 con 0x562d79f0fca0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f009409fc60 con 0x562d79f0fca0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f008404e040 con 0x562d79f0fca0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 10 client.16386 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 10 client.16386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 20 client.16386 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f8f720 0x562d79f91b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 4 client.16386 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.621+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d79e7a9f0 con 0x562d79f8f720 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.622+0000 7f009affd640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f8f720 0x562d79f91b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.622+0000 7f00827fc640 10 client.16386 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.622+0000 7f009affd640 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f8f720 0x562d79f91b40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e858e456a68e04f8 server_cookie=c9906e742c48a8c6 in_seq=0 out_seq=0 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.623+0000 7f00827fc640 1 -- 172.21.15.116:0/2004217972 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d79e7a9f0 con 0x562d79f8f720 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.623+0000 7f00827fc640 10 client.16386 handle_command_reply: tid=0 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.623+0000 7f00a4182740 1 client.16386 shutdown 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 msgr2=0x562d79f2bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0084009930 tx=0x7f008402bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 shutdown_connections 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562d79f8f720 0x562d79f91b40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562d79f18880 0x562d79f2c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d79f0fca0 0x562d79f2bf10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00827fc640 0 client.16386 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 --2- 172.21.15.116:0/2004217972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d79f0f380 0x562d79f2b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 >> 172.21.15.116:0/2004217972 conn(0x562d79f09ab0 msgr2=0x562d79f242d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 shutdown_connections 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 1 -- 172.21.15.116:0/2004217972 wait complete. 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a4182740 20 client.16386 trim_cache size 0 max 16384 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "c9e6fee9-3ccd-45c7-859c-4c877c2dd672", 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:29:51.630 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:51.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 msgr2=0x7f00880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 0x7f00880a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0084009900 tx=0x7f0084005fd0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 msgr2=0x7f009c19c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 0x7f009c19c8b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0094009fd0 tx=0x7f009400ef10 comp rx=0 tx=0).stop 2024-03-19T02:29:51.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 shutdown_connections 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00880a29f0 0x7f00880a4eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f009c1066a0 0x7f009c19c8b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f009c101050 0x7f009c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 --2- 172.21.15.116:0/3314097470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f009c100690 0x7f009c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 >> 172.21.15.116:0/3314097470 conn(0x7f009c077820 msgr2=0x7f009c068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 shutdown_connections 2024-03-19T02:29:51.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.624+0000 7f00a14bf640 1 -- 172.21.15.116:0/3314097470 wait complete. 2024-03-19T02:29:51.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:29:51.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:29:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 -- 172.21.15.116:0/2958878478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8107870 msgr2=0x7ffbc8107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/2958878478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8107870 0x7ffbc8107c50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009930 tx=0x7ffbc402bc30 comp rx=0 tx=0).stop 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 -- 172.21.15.116:0/2958878478 shutdown_connections 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/2958878478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc80693e0 0x7ffbc810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/2958878478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc8068a20 0x7ffbc8068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/2958878478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8107870 0x7ffbc8107c50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 -- 172.21.15.116:0/2958878478 >> 172.21.15.116:0/2958878478 conn(0x7ffbc8077820 msgr2=0x7ffbc8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 -- 172.21.15.116:0/2958878478 shutdown_connections 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.746+0000 7ffbd00c6640 1 -- 172.21.15.116:0/2958878478 wait complete. 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 Processor -- start 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 -- start start 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 0x7ffbc81006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc8107870 0x7ffbc8101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8111a70 con 0x7ffbc8068a20 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81118f0 con 0x7ffbc80693e0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbd00c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8111bf0 con 0x7ffbc8107870 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbcde62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 0x7ffbc81006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41308/0 (socket says 172.21.15.116:41308) 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbcde62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 0x7ffbc81006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55234/0 (socket says 172.21.15.116:55234) 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbcd661640 1 -- 172.21.15.116:0/3088776348 learned_addr learned my addr 172.21.15.116:0/3088776348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.747+0000 7ffbce663640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc8107870 0x7ffbc8101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 -- 172.21.15.116:0/3088776348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc8107870 msgr2=0x7ffbc8101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc8107870 0x7ffbc8101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 -- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 msgr2=0x7ffbc81006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 0x7ffbc81006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 -- 172.21.15.116:0/3088776348 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc81a64d0 con 0x7ffbc80693e0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800e9f0 tx=0x7ffbb800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbbeffd640 1 -- 172.21.15.116:0/3088776348 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb800cd80 con 0x7ffbc80693e0 2024-03-19T02:29:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbbeffd640 1 -- 172.21.15.116:0/3088776348 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbb80108e0 con 0x7ffbc80693e0 2024-03-19T02:29:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc81a67c0 con 0x7ffbc80693e0 2024-03-19T02:29:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbbeffd640 1 -- 172.21.15.116:0/3088776348 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8018920 con 0x7ffbc80693e0 2024-03-19T02:29:51.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.748+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc81a6d10 con 0x7ffbc80693e0 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 Processor -- start 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 -- start start 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbbeffd640 1 -- 172.21.15.116:0/3088776348 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ffbb8010a90 con 0x7ffbc80693e0 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 0x55dfebf229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf22f30 0x55dfebf25320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dfebe7d500 con 0x55dfebf1a350 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dfebe25be0 con 0x55dfebf22f30 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfebe795a0 con 0x55dfebf19a30 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbcde62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 0x55dfebf229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38272/0 (socket says 172.21.15.116:38272) 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 -- 172.21.15.116:0/909237904 learned_addr learned my addr 172.21.15.116:0/909237904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbcd661640 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf22f30 0x55dfebf25320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 -- 172.21.15.116:0/909237904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf22f30 msgr2=0x55dfebf25320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf22f30 0x55dfebf25320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 -- 172.21.15.116:0/909237904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 msgr2=0x55dfebf229f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 0x55dfebf229f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbce663640 1 -- 172.21.15.116:0/909237904 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfebe47000 con 0x55dfebf19a30 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.751+0000 7ffbcde62640 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 0x55dfebf229f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbbeffd640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 0x7ffbb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbce663640 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009f50 tx=0x7ffbc00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbd78df859f8a553 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 0x7ffbb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbbeffd640 1 -- 172.21.15.116:0/3088776348 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ffbb80cf440 con 0x7ffbc80693e0 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbb3fff640 1 -- 172.21.15.116:0/909237904 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0004d20 con 0x55dfebf19a30 2024-03-19T02:29:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbb3fff640 1 -- 172.21.15.116:0/909237904 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0004050 con 0x55dfebf19a30 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.752+0000 7ffbb3fff640 1 -- 172.21.15.116:0/909237904 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0005280 con 0x55dfebf19a30 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/909237904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 msgr2=0x55dfebf19e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 0x7ffbb40a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4005ab0 tx=0x7ffbc4005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009f50 tx=0x7ffbc00118d0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/909237904 shutdown_connections 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf22f30 0x55dfebf25320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf1a350 0x55dfebf229f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/909237904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf19e10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/909237904 >> 172.21.15.116:0/909237904 conn(0x55dfebf14160 msgr2=0x55dfebf14540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/909237904 shutdown_connections 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/909237904 wait complete. 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 -- start start 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.753+0000 7ffbd2d8e740 1 Processor -- start 2024-03-19T02:29:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf20470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf22f30 0x55dfebf209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dfebf25d20 con 0x55dfebf19a30 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dfebe39160 con 0x55dfebf1a350 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfebf25ea0 con 0x55dfebf22f30 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 10 client.0 _ll_register_callbacks cb 0x55dfebe75a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcde62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf20470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcde62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55250/0 (socket says 172.21.15.116:55250) 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbce663640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf22f30 0x55dfebf209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcde62640 1 -- 172.21.15.116:0/3395144822 learned_addr learned my addr 172.21.15.116:0/3395144822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 -- 172.21.15.116:0/3395144822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf22f30 msgr2=0x55dfebf209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf22f30 0x55dfebf209b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 -- 172.21.15.116:0/3395144822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 msgr2=0x55dfebf1ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 -- 172.21.15.116:0/3395144822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfebc5b7d0 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf20470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cc8b0 tx=0x7ffbb8097510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd51731267708a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb809b3b0 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ffbb8097ee0 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.754+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc0004050 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbb809b1d0 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 10 client.26222 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dfebb913f0 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0004050 con 0x55dfebf1a350 2024-03-19T02:29:51.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ffbb809acc0 con 0x55dfebf1a350 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 10 client.26222 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 20 client.26222 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf33220 0x55dfebf35610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 4 client.26222 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfebe8aa40 con 0x55dfebf33220 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.755+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf33220 0x55dfebf35610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.756+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf33220 0x55dfebf35610 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af1e417b1a5ab3c server_cookie=50dc1d8102bae3d4 in_seq=0 out_seq=0 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.756+0000 7ffbb17fa640 10 client.26222 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbb17fa640 1 -- 172.21.15.116:0/3395144822 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfebe8aa40 con 0x55dfebf33220 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbb17fa640 10 client.26222 handle_command_reply: tid=0 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 client.26222 shutdown 2024-03-19T02:29:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 msgr2=0x55dfebf20470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf20470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cc8b0 tx=0x7ffbb8097510 comp rx=0 tx=0).stop 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 shutdown_connections 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf33220 0x55dfebf35610 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf22f30 0x55dfebf209b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbb17fa640 0 client.26222 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf20470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3395144822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf19a30 0x55dfebf1ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 >> 172.21.15.116:0/3395144822 conn(0x55dfebf14160 msgr2=0x55dfebf15eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 shutdown_connections 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3395144822 wait complete. 2024-03-19T02:29:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.757+0000 7ffbd2d8e740 20 client.26222 trim_cache size 0 max 16384 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 Processor -- start 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 -- start start 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf225f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf1ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81b0e10 con 0x55dfebf22f30 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81b2980 con 0x55dfebf1a350 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81b14e0 con 0x55dfebf19a30 2024-03-19T02:29:51.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55256/0 (socket says 172.21.15.116:55256) 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf1ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 -- 172.21.15.116:0/3848352779 learned_addr learned my addr 172.21.15.116:0/3848352779 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbcde62640 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf225f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 -- 172.21.15.116:0/3848352779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 msgr2=0x55dfebf225f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf225f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 -- 172.21.15.116:0/3848352779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 msgr2=0x55dfebf1ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.776+0000 7ffbce663640 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf1ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbce663640 1 -- 172.21.15.116:0/3848352779 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfebf1a180 con 0x55dfebf22f30 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbcd661640 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf1ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbce663640 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0006010 tx=0x7ffbc0009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e325fe681b91a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbb37fe640 1 -- 172.21.15.116:0/3848352779 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0003db0 con 0x55dfebf22f30 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbb37fe640 1 -- 172.21.15.116:0/3848352779 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0005560 con 0x55dfebf22f30 2024-03-19T02:29:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.777+0000 7ffbb37fe640 1 -- 172.21.15.116:0/3848352779 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000fb60 con 0x55dfebf22f30 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3848352779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 msgr2=0x55dfebf2eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0006010 tx=0x7ffbc0009cc0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3848352779 shutdown_connections 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf2eac0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf1ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/3848352779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf225f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3848352779 >> 172.21.15.116:0/3848352779 conn(0x55dfebf14160 msgr2=0x55dfebf131e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3848352779 shutdown_connections 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/3848352779 wait complete. 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- start start 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 Processor -- start 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf26c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81b2280 con 0x55dfebf22f30 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81b1bb0 con 0x55dfebf1a350 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81aedf0 con 0x55dfebf19a30 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.778+0000 7ffbd2d8e740 10 client.0 _ll_register_callbacks cb 0x7ffbb80d2d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcd661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41354/0 (socket says 172.21.15.116:41354) 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcd661640 1 -- 172.21.15.116:0/873513343 learned_addr learned my addr 172.21.15.116:0/873513343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcde62640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf26c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 -- 172.21.15.116:0/873513343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 msgr2=0x55dfebf26c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf26c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 -- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 msgr2=0x55dfebf271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 -- 172.21.15.116:0/873513343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfebb510f0 con 0x55dfebf22f30 2024-03-19T02:29:51.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcd661640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbcde62640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:29:51.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbce663640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf276e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0005000 tx=0x7ffbc0005940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc3304f882555ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:29:51.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbb17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:29:51.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 45919 3.58 MB/sec warmup 74 sec latency 24.111 ms 2024-03-19T02:29:51.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000f860 con 0x55dfebf22f30 2024-03-19T02:29:51.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ffbc000fa00 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc0005560 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc003e050 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbd2d8e740 10 client.16404 fetch_fsmap learned FSMap version 29 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.779+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dfebf99830 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbc0005560 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ffbc00238e0 con 0x55dfebf22f30 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 10 client.16404 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 20 client.16404 populate_metadata read hostname 'smithi116' 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf99d50 0x55dfebf9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 4 client.16404 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfebe84d90 con 0x55dfebf99d50 2024-03-19T02:29:51.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbcde62640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf99d50 0x55dfebf9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbcde62640 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf99d50 0x55dfebf9c140 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef832cbc5bd32bce server_cookie=a427260b7bcb9765 in_seq=0 out_seq=0 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.780+0000 7ffbb17fa640 10 client.16404 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbb17fa640 1 -- 172.21.15.116:0/873513343 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dfebe84d90 con 0x55dfebf99d50 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbb17fa640 10 client.16404 handle_command_reply: tid=0 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 client.16404 shutdown 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 msgr2=0x55dfebf276e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf276e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0005000 tx=0x7ffbc0005940 comp rx=0 tx=0).stop 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 shutdown_connections 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dfebf99d50 0x55dfebf9c140 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dfebf22f30 0x55dfebf276e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfebf1a350 0x55dfebf271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbb17fa640 0 client.16404 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 --2- 172.21.15.116:0/873513343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfebf19a30 0x55dfebf26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 >> 172.21.15.116:0/873513343 conn(0x55dfebf14160 msgr2=0x55dfebf12950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 shutdown_connections 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 1 -- 172.21.15.116:0/873513343 wait complete. 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.781+0000 7ffbd2d8e740 20 client.16404 trim_cache size 0 max 16384 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "c9e6fee9-3ccd-45c7-859c-4c877c2dd672": { 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "c9e6fee9-3ccd-45c7-859c-4c877c2dd672", 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:29:51.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:29:51.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:29:51.790 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:29:51.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 msgr2=0x7ffbb40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 0x7ffbb40a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4005ab0 tx=0x7ffbc4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 msgr2=0x7ffbc8100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800e9f0 tx=0x7ffbb800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 shutdown_connections 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ffbb40a2970 0x7ffbb40a4e30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffbc8107870 0x7ffbc8101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbc80693e0 0x7ffbc8100c20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 --2- 172.21.15.116:0/3088776348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbc8068a20 0x7ffbc81006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 >> 172.21.15.116:0/3088776348 conn(0x7ffbc8077820 msgr2=0x7ffbc80fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 shutdown_connections 2024-03-19T02:29:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:29:51.782+0000 7ffbd00c6640 1 -- 172.21.15.116:0/3088776348 wait complete. 2024-03-19T02:29:51.793 INFO:tasks.cephfs.filesystem:scrub status for tag:c9e6fee9-3ccd-45c7-859c-4c877c2dd672 - {'path': '/', 'tag': 'c9e6fee9-3ccd-45c7-859c-4c877c2dd672', 'options': 'recursive,force'} 2024-03-19T02:29:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:51 smithi162 ceph-mon[35761]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:51 smithi154 ceph-mon[35693]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:51 smithi116 ceph-mon[32263]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-19T02:29:52.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 46466 3.59 MB/sec warmup 75 sec latency 16.899 ms 2024-03-19T02:29:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:29:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:29:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:29:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:52 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:52 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:52 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:29:53.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 46903 3.55 MB/sec warmup 76 sec latency 40.533 ms 2024-03-19T02:29:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:53 smithi162 ceph-mon[35761]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-19T02:29:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:53 smithi154 ceph-mon[35693]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-19T02:29:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:53 smithi116 ceph-mon[32263]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-03-19T02:29:54.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 47557 3.57 MB/sec warmup 77 sec latency 16.574 ms 2024-03-19T02:29:55.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 48203 3.53 MB/sec warmup 78 sec latency 20.923 ms 2024-03-19T02:29:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:55 smithi162 ceph-mon[35761]: pgmap v66: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:55 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:29:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:55 smithi154 ceph-mon[35693]: pgmap v66: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:55 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:29:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:55 smithi116 ceph-mon[32263]: pgmap v66: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:29:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:55 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:29:56.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 48863 3.55 MB/sec warmup 79 sec latency 19.973 ms 2024-03-19T02:29:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:56 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:29:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:56 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:29:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:56 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:29:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:56 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:29:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:56 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:29:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:56 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:29:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:56 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:29:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:56 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:29:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:56 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:29:57.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 49468 3.55 MB/sec warmup 80 sec latency 23.631 ms 2024-03-19T02:29:58.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:57 smithi116 ceph-mon[32263]: pgmap v67: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:57 smithi162 ceph-mon[35761]: pgmap v67: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:57 smithi154 ceph-mon[35693]: pgmap v67: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:29:58.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 50027 3.56 MB/sec warmup 81 sec latency 18.076 ms 2024-03-19T02:29:59.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 50479 3.53 MB/sec warmup 82 sec latency 32.991 ms 2024-03-19T02:30:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:59 smithi162 ceph-mon[35761]: pgmap v68: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:30:00.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:29:59 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:30:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:59 smithi154 ceph-mon[35693]: pgmap v68: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:30:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:29:59 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:30:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:59 smithi116 ceph-mon[32263]: pgmap v68: 129 pgs: 129 active+clean; 52 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:30:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:29:59 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:30:00.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 51114 3.54 MB/sec warmup 83 sec latency 28.868 ms 2024-03-19T02:30:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:30:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:00 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:30:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:30:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:00 smithi154 ceph-mon[35693]: overall HEALTH_OK 2024-03-19T02:30:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:30:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:00 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:30:01.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 51759 3.50 MB/sec warmup 84 sec latency 16.782 ms 2024-03-19T02:30:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:01 smithi162 ceph-mon[35761]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-19T02:30:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:01 smithi154 ceph-mon[35693]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-19T02:30:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:01 smithi116 ceph-mon[32263]: pgmap v69: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 2.0 MiB/s wr, 123 op/s 2024-03-19T02:30:02.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 52461 3.53 MB/sec warmup 85 sec latency 18.044 ms 2024-03-19T02:30:03.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 53028 3.52 MB/sec warmup 86 sec latency 17.456 ms 2024-03-19T02:30:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:03 smithi116 ceph-mon[32263]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:03 smithi162 ceph-mon[35761]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:03 smithi154 ceph-mon[35693]: pgmap v70: 129 pgs: 129 active+clean; 53 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:04.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 53620 3.54 MB/sec warmup 87 sec latency 15.593 ms 2024-03-19T02:30:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:04 smithi154 ceph-mon[35693]: pgmap v71: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:30:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:04 smithi116 ceph-mon[32263]: pgmap v71: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:30:05.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:04 smithi162 ceph-mon[35761]: pgmap v71: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:30:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 54149 3.51 MB/sec warmup 88 sec latency 31.717 ms 2024-03-19T02:30:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 54830 3.52 MB/sec warmup 89 sec latency 17.375 ms 2024-03-19T02:30:07.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 55535 3.50 MB/sec warmup 90 sec latency 23.834 ms 2024-03-19T02:30:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:07 smithi162 ceph-mon[35761]: pgmap v72: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:07 smithi154 ceph-mon[35693]: pgmap v72: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:07 smithi116 ceph-mon[32263]: pgmap v72: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:08.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 56242 3.52 MB/sec warmup 91 sec latency 17.012 ms 2024-03-19T02:30:09.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 56666 3.51 MB/sec warmup 92 sec latency 16.976 ms 2024-03-19T02:30:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:09 smithi162 ceph-mon[35761]: pgmap v73: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:09 smithi154 ceph-mon[35693]: pgmap v73: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:09 smithi116 ceph-mon[32263]: pgmap v73: 129 pgs: 129 active+clean; 56 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.4 MiB/s wr, 98 op/s 2024-03-19T02:30:10.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 57245 3.52 MB/sec warmup 93 sec latency 15.023 ms 2024-03-19T02:30:11.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 57732 3.49 MB/sec warmup 94 sec latency 44.684 ms 2024-03-19T02:30:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:11 smithi162 ceph-mon[35761]: pgmap v74: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-03-19T02:30:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:11 smithi154 ceph-mon[35693]: pgmap v74: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-03-19T02:30:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:11 smithi116 ceph-mon[32263]: pgmap v74: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 2.1 MiB/s wr, 139 op/s 2024-03-19T02:30:12.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 58399 3.50 MB/sec warmup 95 sec latency 19.015 ms 2024-03-19T02:30:13.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 59107 3.48 MB/sec warmup 96 sec latency 24.089 ms 2024-03-19T02:30:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:13 smithi162 ceph-mon[35761]: pgmap v75: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:30:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:13 smithi154 ceph-mon[35693]: pgmap v75: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:30:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:13 smithi116 ceph-mon[32263]: pgmap v75: 129 pgs: 129 active+clean; 57 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:30:14.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 59807 3.50 MB/sec warmup 97 sec latency 22.114 ms 2024-03-19T02:30:15.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 60332 3.51 MB/sec warmup 98 sec latency 18.626 ms 2024-03-19T02:30:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:15 smithi162 ceph-mon[35761]: pgmap v76: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:30:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:15 smithi154 ceph-mon[35693]: pgmap v76: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:30:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:15 smithi116 ceph-mon[32263]: pgmap v76: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:30:16.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 60815 3.50 MB/sec warmup 99 sec latency 16.027 ms 2024-03-19T02:30:17.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 61396 3.48 MB/sec warmup 100 sec latency 30.421 ms 2024-03-19T02:30:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:17 smithi162 ceph-mon[35761]: pgmap v77: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:17 smithi154 ceph-mon[35693]: pgmap v77: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:17 smithi116 ceph-mon[32263]: pgmap v77: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:18.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 62066 3.48 MB/sec warmup 101 sec latency 16.758 ms 2024-03-19T02:30:19.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 62722 3.46 MB/sec warmup 102 sec latency 22.331 ms 2024-03-19T02:30:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:19 smithi162 ceph-mon[35761]: pgmap v78: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:19 smithi154 ceph-mon[35693]: pgmap v78: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:19 smithi116 ceph-mon[32263]: pgmap v78: 129 pgs: 129 active+clean; 59 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 85 op/s 2024-03-19T02:30:20.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 63596 3.50 MB/sec warmup 103 sec latency 17.577 ms 2024-03-19T02:30:21.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 64030 3.50 MB/sec warmup 104 sec latency 19.225 ms 2024-03-19T02:30:21.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:30:21.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:30:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2bb7fe640 1 -- 172.21.15.116:0/2814877513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40328b0 con 0x7ff2bc102170 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/2814877513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc102170 msgr2=0x7ff2bc10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/2814877513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc102170 0x7ff2bc10e680 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009cd0 tx=0x7ff2b402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/2814877513 shutdown_connections 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/2814877513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc102170 0x7ff2bc10e680 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/2814877513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc1017b0 0x7ff2bc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/2814877513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/2814877513 >> 172.21.15.116:0/2814877513 conn(0x7ff2bc078c20 msgr2=0x7ff2bc079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/2814877513 shutdown_connections 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.915+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/2814877513 wait complete. 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 Processor -- start 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 -- start start 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1017b0 0x7ff2bc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc102170 0x7ff2bc198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111a80 con 0x7ff2bc1077c0 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111900 con 0x7ff2bc1017b0 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c2e6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111c00 con 0x7ff2bc102170 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41706/0 (socket says 172.21.15.116:41706) 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c0c08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1017b0 0x7ff2bc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 -- 172.21.15.116:0/1800150274 learned_addr learned my addr 172.21.15.116:0/1800150274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc102170 0x7ff2bc198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 -- 172.21.15.116:0/1800150274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc102170 msgr2=0x7ff2bc198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc102170 0x7ff2bc198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 -- 172.21.15.116:0/1800150274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1017b0 msgr2=0x7ff2bc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1017b0 0x7ff2bc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.916+0000 7ff2c1409640 1 -- 172.21.15.116:0/1800150274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc19cfb0 con 0x7ff2bc1077c0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2c1409640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4004af0 tx=0x7ff2b4035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2b9ffb640 1 -- 172.21.15.116:0/1800150274 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4039040 con 0x7ff2bc1077c0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2b9ffb640 1 -- 172.21.15.116:0/1800150274 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2b4044070 con 0x7ff2bc1077c0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc19d240 con 0x7ff2bc1077c0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2b9ffb640 1 -- 172.21.15.116:0/1800150274 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40042a0 con 0x7ff2bc1077c0 2024-03-19T02:30:21.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.917+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a4b30 con 0x7ff2bc1077c0 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.919+0000 7ff2b9ffb640 1 -- 172.21.15.116:0/1800150274 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff2b4004440 con 0x7ff2bc1077c0 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2b9ffb640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 0x7ff2ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 Processor -- start 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 -- start start 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 0x7ff2ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2b9ffb640 1 -- 172.21.15.116:0/1800150274 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff2b40f1610 con 0x7ff2bc1077c0 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e292e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c04e1f6500 con 0x55c04e29bf30 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c04e19ebe0 con 0x55c04e292a30 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c04e1f25a0 con 0x55c04e293350 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c1409640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56144/0 (socket says 172.21.15.116:56144) 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c1409640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41712/0 (socket says 172.21.15.116:41712) 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 -- 172.21.15.116:0/777411304 learned_addr learned my addr 172.21.15.116:0/777411304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e292e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 0x7ff2ac0a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005d00 tx=0x7ff2b00103d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 -- 172.21.15.116:0/777411304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 msgr2=0x55c04e292e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e292e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 -- 172.21.15.116:0/777411304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 msgr2=0x55c04e29e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2bbfff640 1 -- 172.21.15.116:0/777411304 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c04e1c0000 con 0x55c04e293350 2024-03-19T02:30:21.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.920+0000 7ff2c1409640 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.921+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009ca0 tx=0x7ff2a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=881dbc8e9017c3db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.921+0000 7ff2a6ffd640 1 -- 172.21.15.116:0/777411304 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a801e070 con 0x55c04e293350 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.921+0000 7ff2a6ffd640 1 -- 172.21.15.116:0/777411304 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2a8009e50 con 0x55c04e293350 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.921+0000 7ff2a6ffd640 1 -- 172.21.15.116:0/777411304 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a800e840 con 0x55c04e293350 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- 172.21.15.116:0/777411304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 msgr2=0x55c04e29b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009ca0 tx=0x7ff2a8011810 comp rx=0 tx=0).stop 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- 172.21.15.116:0/777411304 shutdown_connections 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e29bf30 0x55c04e29e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e293350 0x55c04e29b9f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/777411304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e292e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- 172.21.15.116:0/777411304 >> 172.21.15.116:0/777411304 conn(0x55c04e28d160 msgr2=0x55c04e28d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- 172.21.15.116:0/777411304 shutdown_connections 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- 172.21.15.116:0/777411304 wait complete. 2024-03-19T02:30:21.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 -- start start 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 Processor -- start 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.922+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e298e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e299890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c04e29ed20 con 0x55c04e293350 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c04e1b2160 con 0x55c04e292a30 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c04e29eea0 con 0x55c04e29bf30 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 10 client.0 _ll_register_callbacks cb 0x55c04e1eea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41714/0 (socket says 172.21.15.116:41714) 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2bbfff640 1 -- 172.21.15.116:0/2322236422 learned_addr learned my addr 172.21.15.116:0/2322236422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e299890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e298e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 -- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 msgr2=0x55c04e298e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e298e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 -- 172.21.15.116:0/2322236422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 msgr2=0x55c04e299350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 -- 172.21.15.116:0/2322236422 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c04dfd47d0 con 0x55c04e29bf30 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c1409640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e299890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40eeaf0 tx=0x7ff2b40bd630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74bce02d6551d5a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff29bfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40be100 con 0x55c04e29bf30 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff2b40ba070 con 0x55c04e29bf30 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.923+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2a8009e50 con 0x55c04e29bf30 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2b40bda00 con 0x55c04e29bf30 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 10 client.26179 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:21.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c04df0a3f0 con 0x55c04e29bf30 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2a8009e50 con 0x55c04e29bf30 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff2b403c030 con 0x55c04e29bf30 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 10 client.26179 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 10 client.26179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 20 client.26179 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e2ac130 0x55c04e2ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 4 client.26179 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c04e203a40 con 0x55c04e2ac130 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.924+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e2ac130 0x55c04e2ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.925+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e2ac130 0x55c04e2ae540 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc353ec8e6887378 server_cookie=33df1752276c6c35 in_seq=0 out_seq=0 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.925+0000 7ff29bfff640 10 client.26179 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff29bfff640 1 -- 172.21.15.116:0/2322236422 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c04e203a40 con 0x55c04e2ac130 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff29bfff640 10 client.26179 handle_command_reply: tid=0 2024-03-19T02:30:21.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 client.26179 shutdown 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 msgr2=0x55c04e299890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e299890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40eeaf0 tx=0x7ff2b40bd630 comp rx=0 tx=0).stop 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 shutdown_connections 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e2ac130 0x55c04e2ae540 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e299890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e299350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff29bfff640 0 client.26179 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/2322236422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e298e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 >> 172.21.15.116:0/2322236422 conn(0x55c04e28d160 msgr2=0x55c04e28eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 shutdown_connections 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 1 -- 172.21.15.116:0/2322236422 wait complete. 2024-03-19T02:30:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.926+0000 7ff2c5a58740 20 client.26179 trim_cache size 0 max 16384 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 Processor -- start 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 -- start start 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e293350 0x55c04e28d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e2a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1aeb30 con 0x55c04e292a30 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1b06a0 con 0x55c04e293350 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1af200 con 0x55c04e29bf30 2024-03-19T02:30:21.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c0c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c0c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41730/0 (socket says 172.21.15.116:41730) 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c0c08640 1 -- 172.21.15.116:0/745478560 learned_addr learned my addr 172.21.15.116:0/745478560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e2a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e293350 0x55c04e28d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 -- 172.21.15.116:0/745478560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e293350 msgr2=0x55c04e28d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e293350 0x55c04e28d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 -- 172.21.15.116:0/745478560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 msgr2=0x55c04e29b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 -- 172.21.15.116:0/745478560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c04deca0f0 con 0x55c04e29bf30 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.946+0000 7ff2c1409640 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e2a7af0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff2b403d040 tx=0x7ff2b4004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3177e945a908b4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2a67fc640 1 -- 172.21.15.116:0/745478560 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40be100 con 0x55c04e29bf30 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2a67fc640 1 -- 172.21.15.116:0/745478560 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2b40c1b50 con 0x55c04e29bf30 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2a67fc640 1 -- 172.21.15.116:0/745478560 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40c5d10 con 0x55c04e29bf30 2024-03-19T02:30:21.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 -- 172.21.15.116:0/745478560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 msgr2=0x55c04e2a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e2a7af0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff2b403d040 tx=0x7ff2b4004940 comp rx=0 tx=0).stop 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 -- 172.21.15.116:0/745478560 shutdown_connections 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e2a7af0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e293350 0x55c04e28d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/745478560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e292a30 0x55c04e29b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 -- 172.21.15.116:0/745478560 >> 172.21.15.116:0/745478560 conn(0x55c04e28d160 msgr2=0x55c04e28e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 -- 172.21.15.116:0/745478560 shutdown_connections 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.947+0000 7ff2c5a58740 1 -- 172.21.15.116:0/745478560 wait complete. 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 -- start start 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 Processor -- start 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e295410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e295950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1affa0 con 0x55c04e293350 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1af8d0 con 0x55c04e292a30 2024-03-19T02:30:21.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1acb10 con 0x55c04e29bf30 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c5a58740 10 client.0 _ll_register_callbacks cb 0x7ff2b40bd840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c0c08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e295410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c0c08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47706/0 (socket says 172.21.15.116:47706) 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c0c08640 1 -- 172.21.15.116:0/1641963024 learned_addr learned my addr 172.21.15.116:0/1641963024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.948+0000 7ff2c1409640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e295950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 -- 172.21.15.116:0/1641963024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 msgr2=0x55c04e295950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e295950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 -- 172.21.15.116:0/1641963024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 msgr2=0x55c04e295410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e295410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 -- 172.21.15.116:0/1641963024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c04e295e90 con 0x55c04e292a30 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c1409640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e295950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e295410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c0c08640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0011780 tx=0x7ff2b0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f97667efb37f5f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff29bfff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b002f060 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b40c1b50 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff2b002ad90 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2b0036050 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c5a58740 10 client.26246 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c04e312230 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.949+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2b40c1b50 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7ff2b40dc160 con 0x55c04e292a30 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 10 client.26246 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 10 client.26246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 20 client.26246 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e3129c0 0x55c04e314de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 4 client.26246 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:30:21.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c04e1fdd90 con 0x55c04e3129c0 2024-03-19T02:30:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e3129c0 0x55c04e314de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff29bfff640 10 client.26246 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:21.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.950+0000 7ff2bbfff640 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e3129c0 0x55c04e314de0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969caca3b5a43971 server_cookie=4f4d49d5f92131cd in_seq=0 out_seq=0 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff29bfff640 1 -- 172.21.15.116:0/1641963024 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c04e1fdd90 con 0x55c04e3129c0 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff29bfff640 10 client.26246 handle_command_reply: tid=0 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 client.26246 shutdown 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 msgr2=0x55c04e2a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0011780 tx=0x7ff2b0004730 comp rx=0 tx=0).stop 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 shutdown_connections 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c04e3129c0 0x55c04e314de0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c04e29bf30 0x55c04e295950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff29bfff640 0 client.26246 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c04e293350 0x55c04e295410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 --2- 172.21.15.116:0/1641963024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c04e292a30 0x55c04e2a4120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 >> 172.21.15.116:0/1641963024 conn(0x55c04e28d160 msgr2=0x55c04e299660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 shutdown_connections 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 1 -- 172.21.15.116:0/1641963024 wait complete. 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.951+0000 7ff2c5a58740 20 client.26246 trim_cache size 0 max 16384 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:30:21.957 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:30:21.958 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 msgr2=0x7ff2ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 0x7ff2ac0a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0005d00 tx=0x7ff2b00103d0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 msgr2=0x7ff2bc19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4004af0 tx=0x7ff2b4035ce0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 shutdown_connections 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2ac0a2970 0x7ff2ac0a4e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2bc1077c0 0x7ff2bc19c880 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2bc102170 0x7ff2bc198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 --2- 172.21.15.116:0/1800150274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2bc1017b0 0x7ff2bc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 >> 172.21.15.116:0/1800150274 conn(0x7ff2bc078c20 msgr2=0x7ff2bc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 shutdown_connections 2024-03-19T02:30:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:21.952+0000 7ff2c2e6c640 1 -- 172.21.15.116:0/1800150274 wait complete. 2024-03-19T02:30:21.962 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:30:21.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:30:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 -- 172.21.15.116:0/1571919573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd680100720 msgr2=0x7fd680100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 --2- 172.21.15.116:0/1571919573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd680100720 0x7fd680100ba0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd670009930 tx=0x7fd67002bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 -- 172.21.15.116:0/1571919573 shutdown_connections 2024-03-19T02:30:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 --2- 172.21.15.116:0/1571919573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6801010e0 0x7fd68010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 --2- 172.21.15.116:0/1571919573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd680100720 0x7fd680100ba0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 --2- 172.21.15.116:0/1571919573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680106730 0x7fd680106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 -- 172.21.15.116:0/1571919573 >> 172.21.15.116:0/1571919573 conn(0x7fd680077820 msgr2=0x7fd680077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 -- 172.21.15.116:0/1571919573 shutdown_connections 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.075+0000 7fd688310640 1 -- 172.21.15.116:0/1571919573 wait complete. 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 Processor -- start 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 -- start start 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680100720 0x7fd6800fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd680106730 0x7fd6800fe140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd680110a70 con 0x7fd6801010e0 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801108f0 con 0x7fd680106730 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd688310640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd680110bf0 con 0x7fd680100720 2024-03-19T02:30:22.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd6858ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd6858ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41754/0 (socket says 172.21.15.116:41754) 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd6860ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680100720 0x7fd6800fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd6858ab640 1 -- 172.21.15.116:0/2479330189 learned_addr learned my addr 172.21.15.116:0/2479330189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.076+0000 7fd6868ad640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd680106730 0x7fd6800fe140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 -- 172.21.15.116:0/2479330189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680100720 msgr2=0x7fd6800fd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680100720 0x7fd6800fd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 -- 172.21.15.116:0/2479330189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd680106730 msgr2=0x7fd6800fe140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd680106730 0x7fd6800fe140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 -- 172.21.15.116:0/2479330189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6801a6620 con 0x7fd6801010e0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6858ab640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd670005940 tx=0x7fd67002ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670043020 con 0x7fd6801010e0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6801a68b0 con 0x7fd6801010e0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd67002cc80 con 0x7fd6801010e0 2024-03-19T02:30:22.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670003c00 con 0x7fd6801010e0 2024-03-19T02:30:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.077+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801a6e50 con 0x7fd6801010e0 2024-03-19T02:30:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.078+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd680068610 con 0x7fd6801010e0 2024-03-19T02:30:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.079+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd670046020 con 0x7fd6801010e0 2024-03-19T02:30:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.080+0000 7fd6777fe640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 0x7fd66c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.080+0000 7fd6860ac640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 0x7fd66c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.080+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd6700f1990 con 0x7fd6801010e0 2024-03-19T02:30:22.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.080+0000 7fd6860ac640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 0x7fd66c0a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009990 tx=0x7fd67c009110 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.084+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd67004d020 con 0x7fd6801010e0 2024-03-19T02:30:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:21 smithi162 ceph-mon[35761]: pgmap v79: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:21 smithi154 ceph-mon[35693]: pgmap v79: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:21 smithi116 ceph-mon[32263]: pgmap v79: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:22.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.322+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6801a7080 con 0x7fd6801010e0 2024-03-19T02:30:22.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.323+0000 7fd6777fe640 1 -- 172.21.15.116:0/2479330189 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11951 (secure 0 0 0) 0x7fd6700be480 con 0x7fd6801010e0 2024-03-19T02:30:22.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:30:22.323 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:28:39.411853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:30:22.323 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 msgr2=0x7fd66c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 0x7fd66c0a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009990 tx=0x7fd67c009110 comp rx=0 tx=0).stop 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 msgr2=0x7fd6800fdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd670005940 tx=0x7fd67002ca90 comp rx=0 tx=0).stop 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 shutdown_connections 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd66c0a2970 0x7fd66c0a4e30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd680106730 0x7fd6800fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6801010e0 0x7fd6800fdc00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 --2- 172.21.15.116:0/2479330189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd680100720 0x7fd6800fd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 >> 172.21.15.116:0/2479330189 conn(0x7fd680077820 msgr2=0x7fd6800ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 shutdown_connections 2024-03-19T02:30:22.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.325+0000 7fd688310640 1 -- 172.21.15.116:0/2479330189 wait complete. 2024-03-19T02:30:22.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:30:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.444+0000 7f368426e640 1 -- 172.21.15.116:0/691935046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 msgr2=0x7f367c107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.444+0000 7f368426e640 1 --2- 172.21.15.116:0/691935046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c107be0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3678009930 tx=0x7f367802bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- 172.21.15.116:0/691935046 shutdown_connections 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- 172.21.15.116:0/691935046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 0x7f367c10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- 172.21.15.116:0/691935046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- 172.21.15.116:0/691935046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c107be0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- 172.21.15.116:0/691935046 >> 172.21.15.116:0/691935046 conn(0x7f367c0fd2c0 msgr2=0x7f367c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- 172.21.15.116:0/691935046 shutdown_connections 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- 172.21.15.116:0/691935046 wait complete. 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 Processor -- start 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- start start 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 0x7f367c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f367c111ac0 con 0x7f367c107800 2024-03-19T02:30:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f367c111940 con 0x7f367c1021b0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368426e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f367c111c40 con 0x7f367c1017f0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368280b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368200a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368280b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41770/0 (socket says 172.21.15.116:41770) 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f368280b640 1 -- 172.21.15.116:0/3372445339 learned_addr learned my addr 172.21.15.116:0/3372445339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.445+0000 7f3681809640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 0x7f367c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 -- 172.21.15.116:0/3372445339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 msgr2=0x7f367c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 0x7f367c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 -- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 msgr2=0x7f367c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 -- 172.21.15.116:0/3372445339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367c19d010 con 0x7f367c1017f0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368280b640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368200a640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c198020 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3678009230 tx=0x7f36780045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f36737fe640 1 -- 172.21.15.116:0/3372445339 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678043020 con 0x7f367c1017f0 2024-03-19T02:30:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367c19d2a0 con 0x7f367c1017f0 2024-03-19T02:30:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f367c1a4b90 con 0x7f367c1017f0 2024-03-19T02:30:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f36737fe640 1 -- 172.21.15.116:0/3372445339 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f367802ca70 con 0x7f367c1017f0 2024-03-19T02:30:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.446+0000 7f36737fe640 1 -- 172.21.15.116:0/3372445339 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678003a10 con 0x7f367c1017f0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.448+0000 7f36737fe640 1 -- 172.21.15.116:0/3372445339 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3678046020 con 0x7f367c1017f0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 Processor -- start 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 -- start start 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d2f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d38290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556e24c924b0 con 0x556e24d387d0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556e24c3ad70 con 0x556e24d2f2d0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f3686e57740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556e24c8e550 con 0x556e24d2fbf0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f36737fe640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.449+0000 7f368200a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d38290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3681809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368200a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d38290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56228/0 (socket says 172.21.15.116:56228) 2024-03-19T02:30:22.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d2f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3681809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41778/0 (socket says 172.21.15.116:41778) 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3681809640 1 -- 172.21.15.116:0/2588004337 learned_addr learned my addr 172.21.15.116:0/2588004337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f36737fe640 1 -- 172.21.15.116:0/3372445339 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f36780f09a0 con 0x7f367c1017f0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 -- 172.21.15.116:0/2588004337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 msgr2=0x556e24d38290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d38290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 -- 172.21.15.116:0/2588004337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 msgr2=0x556e24d3abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 -- 172.21.15.116:0/2588004337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e24c5d210 con 0x556e24d2f2d0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3681809640 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3674011f70 tx=0x7f3674018000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f368280b640 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d2f6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f36740049d0 tx=0x7f3674011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9eda8283bc41df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3670bfe640 1 -- 172.21.15.116:0/2588004337 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674025070 con 0x556e24d2f2d0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3670bfe640 1 -- 172.21.15.116:0/2588004337 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3674013cd0 con 0x556e24d2f2d0 2024-03-19T02:30:22.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.450+0000 7f3670bfe640 1 -- 172.21.15.116:0/2588004337 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674005650 con 0x556e24d2f2d0 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 -- 172.21.15.116:0/2588004337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 msgr2=0x556e24d2f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d2f6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f36740049d0 tx=0x7f3674011a70 comp rx=0 tx=0).stop 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 -- 172.21.15.116:0/2588004337 shutdown_connections 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d3abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d38290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 --2- 172.21.15.116:0/2588004337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d2f6b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 -- 172.21.15.116:0/2588004337 >> 172.21.15.116:0/2588004337 conn(0x556e24d29a00 msgr2=0x556e24d29de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 -- 172.21.15.116:0/2588004337 shutdown_connections 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.451+0000 7f3686e57740 1 -- 172.21.15.116:0/2588004337 wait complete. 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 -- start start 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 Processor -- start 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24d35d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d362d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556e24d3b560 con 0x556e24d387d0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556e24c4e310 con 0x556e24d2fbf0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556e24d3b6e0 con 0x556e24d2f2d0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3686e57740 10 client.0 _ll_register_callbacks cb 0x556e24c422d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368200a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368200a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56238/0 (socket says 172.21.15.116:56238) 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368200a640 1 -- 172.21.15.116:0/1313811389 learned_addr learned my addr 172.21.15.116:0/1313811389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f3681809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24d35d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 -- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 msgr2=0x556e24d35850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 -- 172.21.15.116:0/1313811389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 msgr2=0x556e24d35d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24d35d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368280b640 1 -- 172.21.15.116:0/1313811389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e249a63f0 con 0x556e24d387d0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.452+0000 7f368200a640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f368280b640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d362d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3674009270 tx=0x7f367400e3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a873cc8ef3290851 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367401c040 con 0x556e24d387d0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3674005000 con 0x556e24d387d0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3674013cd0 con 0x556e24d387d0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3674025040 con 0x556e24d387d0 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 10 client.16440 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e24a707d0 con 0x556e24d387d0 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3674013cd0 con 0x556e24d387d0 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f36740329e0 con 0x556e24d387d0 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 10 client.16440 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 10 client.16440 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 20 client.16440 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24d48b40 0x556e24d4af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 4 client.16440 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.453+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e24cf03e0 con 0x556e24d48b40 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.454+0000 7f368200a640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24d48b40 0x556e24d4af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.454+0000 7f368200a640 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24d48b40 0x556e24d4af30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a68e73938a549b2c server_cookie=1676d8cfcf4d74d2 in_seq=0 out_seq=0 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.454+0000 7f3665ffb640 10 client.16440 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3665ffb640 1 -- 172.21.15.116:0/1313811389 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e24cf03e0 con 0x556e24d48b40 2024-03-19T02:30:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3665ffb640 10 client.16440 handle_command_reply: tid=0 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 client.16440 shutdown 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 msgr2=0x556e24d362d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d362d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3674009270 tx=0x7f367400e3d0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 shutdown_connections 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24d48b40 0x556e24d4af30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d362d0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3665ffb640 0 client.16440 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24d35d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 --2- 172.21.15.116:0/1313811389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2f2d0 0x556e24d35850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 >> 172.21.15.116:0/1313811389 conn(0x556e24d29a00 msgr2=0x556e24d2b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 shutdown_connections 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 1 -- 172.21.15.116:0/1313811389 wait complete. 2024-03-19T02:30:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.455+0000 7f3686e57740 20 client.16440 trim_cache size 0 max 16384 2024-03-19T02:30:22.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 Processor -- start 2024-03-19T02:30:22.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 -- start start 2024-03-19T02:30:22.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d37f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d2ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1aeb30 con 0x556e24d387d0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1b06a0 con 0x556e24d2f2d0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3686e57740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1af200 con 0x556e24d2fbf0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f368280b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f368280b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41804/0 (socket says 172.21.15.116:41804) 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f368200a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d37f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f368280b640 1 -- 172.21.15.116:0/666524427 learned_addr learned my addr 172.21.15.116:0/666524427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.469+0000 7f3681809640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d2ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 -- 172.21.15.116:0/666524427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 msgr2=0x556e24d2ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d2ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 -- 172.21.15.116:0/666524427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 msgr2=0x556e24d37f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d37f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 -- 172.21.15.116:0/666524427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e24d3af30 con 0x556e24d387d0 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f368280b640 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3674032d50 tx=0x7f3674011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb015aecbfee5c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f3667fff640 1 -- 172.21.15.116:0/666524427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674013cd0 con 0x556e24d387d0 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f3667fff640 1 -- 172.21.15.116:0/666524427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3674005610 con 0x556e24d387d0 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.470+0000 7f3667fff640 1 -- 172.21.15.116:0/666524427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36740112a0 con 0x556e24d387d0 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- 172.21.15.116:0/666524427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 msgr2=0x556e24d44300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3674032d50 tx=0x7f3674011a70 comp rx=0 tx=0).stop 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- 172.21.15.116:0/666524427 shutdown_connections 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d387d0 0x556e24d44300 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d2fbf0 0x556e24d2ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- 172.21.15.116:0/666524427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2f2d0 0x556e24d37f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- 172.21.15.116:0/666524427 >> 172.21.15.116:0/666524427 conn(0x556e24d29a00 msgr2=0x556e24d2ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- 172.21.15.116:0/666524427 shutdown_connections 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- 172.21.15.116:0/666524427 wait complete. 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- start start 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 Processor -- start 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24daa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d387d0 0x556e24daa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1affa0 con 0x556e24d2f2d0 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1af8d0 con 0x556e24d2fbf0 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f367c1acb10 con 0x556e24d387d0 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3686e57740 10 client.0 _ll_register_callbacks cb 0x7f366c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f368200a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f368200a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41808/0 (socket says 172.21.15.116:41808) 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f368200a640 1 -- 172.21.15.116:0/3561794011 learned_addr learned my addr 172.21.15.116:0/3561794011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.471+0000 7f3681809640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24daa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f368280b640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d387d0 0x556e24daa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 -- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d387d0 msgr2=0x556e24daa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d387d0 0x556e24daa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 -- 172.21.15.116:0/3561794011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 msgr2=0x556e24d40930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 -- 172.21.15.116:0/3561794011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e24d32390 con 0x556e24d2fbf0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f368200a640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3681809640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24daa320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x556e24d3b6a0 tx=0x7f366c012bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f6bcfc6807e6dd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3665ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c023060 con 0x556e24d2fbf0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3674005610 con 0x556e24d2fbf0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f366c004230 con 0x556e24d2fbf0 2024-03-19T02:30:22.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f366c004430 con 0x556e24d2fbf0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3686e57740 10 client.26267 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.472+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e24db3550 con 0x556e24d2fbf0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3674005610 con 0x556e24d2fbf0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f366c003b20 con 0x556e24d2fbf0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 10 client.26267 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 10 client.26267 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 20 client.26267 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24db3c80 0x556e24db60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 4 client.26267 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e24c9f9f0 con 0x556e24db3c80 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f368200a640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24db3c80 0x556e24db60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3665ffb640 10 client.26267 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f368200a640 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24db3c80 0x556e24db60a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e79b85380823e9e5 server_cookie=a26ebd939f07faf1 in_seq=0 out_seq=0 2024-03-19T02:30:22.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3665ffb640 1 -- 172.21.15.116:0/3561794011 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e24c9f9f0 con 0x556e24db3c80 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3665ffb640 10 client.26267 handle_command_reply: tid=0 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.473+0000 7f3686e57740 1 client.26267 shutdown 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 msgr2=0x556e24daa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24daa320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x556e24d3b6a0 tx=0x7f366c012bd0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 shutdown_connections 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x556e24db3c80 0x556e24db60a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e24d387d0 0x556e24daa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3665ffb640 0 client.26267 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e24d2fbf0 0x556e24daa320 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 --2- 172.21.15.116:0/3561794011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e24d2f2d0 0x556e24d40930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 >> 172.21.15.116:0/3561794011 conn(0x556e24d29a00 msgr2=0x556e24d28ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 shutdown_connections 2024-03-19T02:30:22.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 1 -- 172.21.15.116:0/3561794011 wait complete. 2024-03-19T02:30:22.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f3686e57740 20 client.26267 trim_cache size 0 max 16384 2024-03-19T02:30:22.481 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:22.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 msgr2=0x7f36680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3674011f70 tx=0x7f3674018000 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 msgr2=0x7f367c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c198020 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3678009230 tx=0x7f36780045e0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 shutdown_connections 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367c107800 0x7f367c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f367c1021b0 0x7f367c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 --2- 172.21.15.116:0/3372445339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f367c1017f0 0x7f367c198020 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 >> 172.21.15.116:0/3372445339 conn(0x7f367c0fd2c0 msgr2=0x7f367c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 shutdown_connections 2024-03-19T02:30:22.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.474+0000 7f368426e640 1 -- 172.21.15.116:0/3372445339 wait complete. 2024-03-19T02:30:22.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:30:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.594+0000 7fbee8f90640 1 -- 172.21.15.116:0/658895029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41006a0 msgr2=0x7fbee4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.594+0000 7fbee8f90640 1 --2- 172.21.15.116:0/658895029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41006a0 0x7fbee4100b20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009890 tx=0x7fbed802bb70 comp rx=0 tx=0).stop 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 -- 172.21.15.116:0/658895029 shutdown_connections 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 --2- 172.21.15.116:0/658895029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee4101060 0x7fbee410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 --2- 172.21.15.116:0/658895029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41006a0 0x7fbee4100b20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 --2- 172.21.15.116:0/658895029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee41066b0 0x7fbee4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 -- 172.21.15.116:0/658895029 >> 172.21.15.116:0/658895029 conn(0x7fbee40778d0 msgr2=0x7fbee4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 -- 172.21.15.116:0/658895029 shutdown_connections 2024-03-19T02:30:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 -- 172.21.15.116:0/658895029 wait complete. 2024-03-19T02:30:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 Processor -- start 2024-03-19T02:30:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.595+0000 7fbee8f90640 1 -- start start 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee4101060 0x7fbee4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 0x7fbee41a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbee4110970 con 0x7fbee41006a0 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41107f0 con 0x7fbee4101060 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbee4110af0 con 0x7fbee41066b0 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41832/0 (socket says 172.21.15.116:41832) 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 -- 172.21.15.116:0/2779899432 learned_addr learned my addr 172.21.15.116:0/2779899432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 0x7fbee41a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee4101060 0x7fbee4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 -- 172.21.15.116:0/2779899432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 msgr2=0x7fbee41a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 0x7fbee41a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 -- 172.21.15.116:0/2779899432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee4101060 msgr2=0x7fbee4199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee4101060 0x7fbee4199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 -- 172.21.15.116:0/2779899432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee41a43d0 con 0x7fbee41006a0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 0x7fbee41a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee259c640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbed37fe640 1 -- 172.21.15.116:0/2779899432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed400cd80 con 0x7fbee41006a0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee4196c00 con 0x7fbee41006a0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbed37fe640 1 -- 172.21.15.116:0/2779899432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed40108e0 con 0x7fbee41006a0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.596+0000 7fbed37fe640 1 -- 172.21.15.116:0/2779899432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4018920 con 0x7fbee41006a0 2024-03-19T02:30:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.597+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee4197170 con 0x7fbee41006a0 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.599+0000 7fbed37fe640 1 -- 172.21.15.116:0/2779899432 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fbed4010a90 con 0x7fbee41006a0 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.599+0000 7fbeebb78740 1 Processor -- start 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 -- start start 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc754290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc756bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568fc6ae4b0 con 0x5568fc74b2d0 2024-03-19T02:30:22.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5568fc656d70 con 0x5568fc7547d0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbeebb78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5568fc6aa550 con 0x5568fc74bbf0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc754290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41836/0 (socket says 172.21.15.116:41836) 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee2d9d640 1 -- 172.21.15.116:0/2767958331 learned_addr learned my addr 172.21.15.116:0/2767958331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc756bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbed37fe640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 0x7fbecc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 -- 172.21.15.116:0/2767958331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 msgr2=0x5568fc756bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc756bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 -- 172.21.15.116:0/2767958331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 msgr2=0x5568fc74b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee259c640 1 -- 172.21.15.116:0/2767958331 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568fc679210 con 0x5568fc74bbf0 2024-03-19T02:30:22.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.600+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 0x7fbecc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbed37fe640 1 -- 172.21.15.116:0/2779899432 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fbed40cf470 con 0x7fbee41006a0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbee259c640 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc754290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbed40cc840 tx=0x7fbed405bad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717fbebd2d89d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 0x7fbecc0a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbedc005c40 tx=0x7fbedc010040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbed0ff9640 1 -- 172.21.15.116:0/2767958331 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed409b160 con 0x5568fc74bbf0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbed0ff9640 1 -- 172.21.15.116:0/2767958331 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed4010de0 con 0x5568fc74bbf0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.601+0000 7fbed0ff9640 1 -- 172.21.15.116:0/2767958331 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4020770 con 0x5568fc74bbf0 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- 172.21.15.116:0/2767958331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 msgr2=0x5568fc754290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc754290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbed40cc840 tx=0x7fbed405bad0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- 172.21.15.116:0/2767958331 shutdown_connections 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc756bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc754290 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- 172.21.15.116:0/2767958331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc74b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- 172.21.15.116:0/2767958331 >> 172.21.15.116:0/2767958331 conn(0x5568fc745a00 msgr2=0x5568fc745de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- 172.21.15.116:0/2767958331 shutdown_connections 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- 172.21.15.116:0/2767958331 wait complete. 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- start start 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 Processor -- start 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc7522e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568fc757560 con 0x5568fc74b2d0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5568fc66a310 con 0x5568fc7547d0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5568fc7576e0 con 0x5568fc74bbf0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.602+0000 7fbeebb78740 10 client.0 _ll_register_callbacks cb 0x5568fc65e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee1d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41840/0 (socket says 172.21.15.116:41840) 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 -- 172.21.15.116:0/1761210085 learned_addr learned my addr 172.21.15.116:0/1761210085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee1d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56282/0 (socket says 172.21.15.116:56282) 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc7522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 -- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 msgr2=0x5568fc751da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 -- 172.21.15.116:0/1761210085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 msgr2=0x5568fc7522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc7522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 -- 172.21.15.116:0/1761210085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568fc3c23f0 con 0x5568fc74b2d0 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbee259c640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbed40a39c0 tx=0x7fbed409b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff7d92211c4e1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbec9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed409b160 con 0x5568fc74b2d0 2024-03-19T02:30:22.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fbed409ba60 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568fc70c3e0 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbed409b800 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbeebb78740 10 client.16458 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.603+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568fc48c7d0 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5568fc70c3e0 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fbed405bb40 con 0x5568fc74b2d0 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 10 client.16458 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 10 client.16458 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:30:22.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 20 client.16458 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7649e0 0x5568fc766dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 4 client.16458 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568fc6bb9f0 con 0x5568fc7649e0 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7649e0 0x5568fc766dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbec9ffb640 10 client.16458 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.604+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7649e0 0x5568fc766dd0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e22bf5f9df9e2ba server_cookie=a897a6e0563d96df in_seq=0 out_seq=0 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1761210085 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568fc6bb9f0 con 0x5568fc7649e0 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbec9ffb640 10 client.16458 handle_command_reply: tid=0 2024-03-19T02:30:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 client.16458 shutdown 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 msgr2=0x5568fc751860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbed40a39c0 tx=0x7fbed409b020 comp rx=0 tx=0).stop 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 shutdown_connections 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc7522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc751da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7649e0 0x5568fc766dd0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1761210085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc751860 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbec9ffb640 0 client.16458 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 >> 172.21.15.116:0/1761210085 conn(0x5568fc745a00 msgr2=0x5568fc747750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 shutdown_connections 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 1 -- 172.21.15.116:0/1761210085 wait complete. 2024-03-19T02:30:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.605+0000 7fbeebb78740 20 client.16458 trim_cache size 0 max 16384 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 Processor -- start 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 -- start start 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc7520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc769a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41ac900 con 0x5568fc74b2d0 2024-03-19T02:30:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41ae470 con 0x5568fc7547d0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbeebb78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41acfd0 con 0x5568fc74bbf0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbee259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41852/0 (socket says 172.21.15.116:41852) 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbee259c640 1 -- 172.21.15.116:0/372617694 learned_addr learned my addr 172.21.15.116:0/372617694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc769a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.620+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc7520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 -- 172.21.15.116:0/372617694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 msgr2=0x5568fc7520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc7520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 -- 172.21.15.116:0/372617694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 msgr2=0x5568fc769a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc769a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 -- 172.21.15.116:0/372617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568fc7465d0 con 0x5568fc74b2d0 2024-03-19T02:30:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbee259c640 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbed40cce40 tx=0x7fbed40cc810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2847fc447b1f524 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbecbfff640 1 -- 172.21.15.116:0/372617694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed409f8a0 con 0x5568fc74b2d0 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbecbfff640 1 -- 172.21.15.116:0/372617694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed4020e10 con 0x5568fc74b2d0 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbecbfff640 1 -- 172.21.15.116:0/372617694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed409ba60 con 0x5568fc74b2d0 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbeebb78740 1 -- 172.21.15.116:0/372617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 msgr2=0x5568fc753f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbeebb78740 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbed40cce40 tx=0x7fbed40cc810 comp rx=0 tx=0).stop 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.621+0000 7fbeebb78740 1 -- 172.21.15.116:0/372617694 shutdown_connections 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc7547d0 0x5568fc769a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74bbf0 0x5568fc7520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- 172.21.15.116:0/372617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc74b2d0 0x5568fc753f90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- 172.21.15.116:0/372617694 >> 172.21.15.116:0/372617694 conn(0x5568fc745a00 msgr2=0x5568fc744ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- 172.21.15.116:0/372617694 shutdown_connections 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- 172.21.15.116:0/372617694 wait complete. 2024-03-19T02:30:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- start start 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 Processor -- start 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 0x5568fc7c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 0x5568fc7c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41add70 con 0x5568fc7547d0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41ad6a0 con 0x5568fc74bbf0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbee41aa8e0 con 0x5568fc74b2d0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbeebb78740 10 client.0 _ll_register_callbacks cb 0x7fbed40d3f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbee2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbee2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41856/0 (socket says 172.21.15.116:41856) 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbee2d9d640 1 -- 172.21.15.116:0/1955296286 learned_addr learned my addr 172.21.15.116:0/1955296286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbee259c640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 0x5568fc7c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.622+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 0x5568fc7c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 -- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 msgr2=0x5568fc7c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 0x5568fc7c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 -- 172.21.15.116:0/1955296286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 msgr2=0x5568fc7c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 0x5568fc7c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 -- 172.21.15.116:0/1955296286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568fc7674f0 con 0x5568fc7547d0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee259c640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 0x5568fc7c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee1d9b640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 0x5568fc7c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbee2d9d640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00efd0 tx=0x7fbedc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5de5d347bfc2a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc02c070 con 0x5568fc7547d0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fbedc028680 con 0x5568fc7547d0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbed4020e10 con 0x5568fc7547d0 2024-03-19T02:30:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbedc038050 con 0x5568fc7547d0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbeebb78740 10 client.16464 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568fc7ca570 con 0x5568fc7547d0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed4020e10 con 0x5568fc7547d0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.623+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fbedc026040 con 0x5568fc7547d0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 10 client.16464 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 10 client.16464 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 20 client.16464 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7cac90 0x5568fc7cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 4 client.16464 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568fc6b5d40 con 0x5568fc7cac90 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbee259c640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7cac90 0x5568fc7cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbee259c640 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7cac90 0x5568fc7cd0b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6a789bba7ec1ef5 server_cookie=d90d50e38e413b64 in_seq=0 out_seq=0 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbec9ffb640 10 client.16464 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbec9ffb640 1 -- 172.21.15.116:0/1955296286 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568fc6b5d40 con 0x5568fc7cac90 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbec9ffb640 10 client.16464 handle_command_reply: tid=0 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 client.16464 shutdown 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 msgr2=0x5568fc766e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00efd0 tx=0x7fbedc0095c0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 shutdown_connections 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5568fc7cac90 0x5568fc7cd0b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568fc7547d0 0x5568fc766e00 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568fc74bbf0 0x5568fc7c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbec9ffb640 0 client.16464 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 --2- 172.21.15.116:0/1955296286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568fc74b2d0 0x5568fc7c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 >> 172.21.15.116:0/1955296286 conn(0x5568fc745a00 msgr2=0x5568fc75f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.624+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 shutdown_connections 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbeebb78740 1 -- 172.21.15.116:0/1955296286 wait complete. 2024-03-19T02:30:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbeebb78740 20 client.16464 trim_cache size 0 max 16384 2024-03-19T02:30:22.632 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 msgr2=0x7fbecc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 0x7fbecc0a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbedc005c40 tx=0x7fbedc010040 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 msgr2=0x7fbee4198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbed400e9f0 tx=0x7fbed400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 shutdown_connections 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbecc0a2970 0x7fbecc0a4e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbee41066b0 0x7fbee41a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee4101060 0x7fbee4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 --2- 172.21.15.116:0/2779899432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbee41006a0 0x7fbee4198e10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 >> 172.21.15.116:0/2779899432 conn(0x7fbee40778d0 msgr2=0x7fbee4102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 shutdown_connections 2024-03-19T02:30:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.625+0000 7fbee8f90640 1 -- 172.21.15.116:0/2779899432 wait complete. 2024-03-19T02:30:22.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 -- 172.21.15.116:0/3154922332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d01006a0 msgr2=0x7f69d0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 --2- 172.21.15.116:0/3154922332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d01006a0 0x7f69d0100b20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009930 tx=0x7f69c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 -- 172.21.15.116:0/3154922332 shutdown_connections 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 --2- 172.21.15.116:0/3154922332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d0101060 0x7f69d010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 --2- 172.21.15.116:0/3154922332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d01006a0 0x7f69d0100b20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 --2- 172.21.15.116:0/3154922332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01066b0 0x7f69d0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 -- 172.21.15.116:0/3154922332 >> 172.21.15.116:0/3154922332 conn(0x7f69d00778d0 msgr2=0x7f69d0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 -- 172.21.15.116:0/3154922332 shutdown_connections 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.746+0000 7f69d7353640 1 -- 172.21.15.116:0/3154922332 wait complete. 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 Processor -- start 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 -- start start 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 0x7f69d0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d01066b0 0x7f69d019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69d0110970 con 0x7f69d0101060 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01107f0 con 0x7f69d01066b0 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d7353640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69d0110af0 con 0x7f69d01006a0 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41874/0 (socket says 172.21.15.116:41874) 2024-03-19T02:30:22.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 0x7f69d0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 -- 172.21.15.116:0/2418760261 learned_addr learned my addr 172.21.15.116:0/2418760261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d01066b0 0x7f69d019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 -- 172.21.15.116:0/2418760261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 msgr2=0x7f69d0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 0x7f69d0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 -- 172.21.15.116:0/2418760261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d01066b0 msgr2=0x7f69d019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d01066b0 0x7f69d019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d48ee640 1 -- 172.21.15.116:0/2418760261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d019cfd0 con 0x7f69d0101060 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.747+0000 7f69d50ef640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 0x7f69d0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69d48ee640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009230 tx=0x7f69c002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69c67fc640 1 -- 172.21.15.116:0/2418760261 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0043020 con 0x7f69d0101060 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69c67fc640 1 -- 172.21.15.116:0/2418760261 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69c002ca70 con 0x7f69d0101060 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d019d260 con 0x7f69d0101060 2024-03-19T02:30:22.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69c67fc640 1 -- 172.21.15.116:0/2418760261 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0003a50 con 0x7f69d0101060 2024-03-19T02:30:22.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.748+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d01a4b50 con 0x7f69d0101060 2024-03-19T02:30:22.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69c67fc640 1 -- 172.21.15.116:0/2418760261 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f69c0046020 con 0x7f69d0101060 2024-03-19T02:30:22.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 Processor -- start 2024-03-19T02:30:22.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 -- start start 2024-03-19T02:30:22.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a837dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a83866a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8388fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5a82dc4b0 con 0x55a5a8386be0 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8284d70 con 0x55a5a837e000 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69da01a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5a82d8550 con 0x55a5a837d6e0 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8388fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69d58f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a837dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a83866a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69d58f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a837dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56318/0 (socket says 172.21.15.116:56318) 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.751+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8388fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41884/0 (socket says 172.21.15.116:41884) 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d58f0640 1 -- 172.21.15.116:0/952527028 learned_addr learned my addr 172.21.15.116:0/952527028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69c67fc640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 0x7f69bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 -- 172.21.15.116:0/952527028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 msgr2=0x55a5a837dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a837dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 -- 172.21.15.116:0/952527028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 msgr2=0x55a5a8388fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8388fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 0x7f69bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 -- 172.21.15.116:0/952527028 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a82a7210 con 0x55a5a837e000 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69c67fc640 1 -- 172.21.15.116:0/2418760261 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f69c00f10b0 con 0x7f69d0101060 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d50ef640 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a83866a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009fb0 tx=0x7f69cc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d2165cba2e848b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 0x7f69bc0a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009fd0 tx=0x7f69c8004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69bb7fe640 1 -- 172.21.15.116:0/952527028 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc004310 con 0x55a5a837e000 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69bb7fe640 1 -- 172.21.15.116:0/952527028 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69cc0044b0 con 0x55a5a837e000 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.752+0000 7f69bb7fe640 1 -- 172.21.15.116:0/952527028 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0035f0 con 0x55a5a837e000 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- 172.21.15.116:0/952527028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 msgr2=0x55a5a83866a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a83866a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009fb0 tx=0x7f69cc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- 172.21.15.116:0/952527028 shutdown_connections 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8388fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a83866a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 --2- 172.21.15.116:0/952527028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a837dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- 172.21.15.116:0/952527028 >> 172.21.15.116:0/952527028 conn(0x55a5a8377e10 msgr2=0x55a5a83781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- 172.21.15.116:0/952527028 shutdown_connections 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- 172.21.15.116:0/952527028 wait complete. 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.753+0000 7f69da01a740 1 -- start start 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 Processor -- start 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a8383b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a8386be0 0x55a5a8384610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5a83899d0 con 0x55a5a837e000 2024-03-19T02:30:22.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8298310 con 0x55a5a8386be0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8389b50 con 0x55a5a837d6e0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69da01a740 10 client.0 _ll_register_callbacks cb 0x55a5a828c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41892/0 (socket says 172.21.15.116:41892) 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 -- 172.21.15.116:0/672039830 learned_addr learned my addr 172.21.15.116:0/672039830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d50ef640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a8383b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d58f0640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a8386be0 0x55a5a8384610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 -- 172.21.15.116:0/672039830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 msgr2=0x55a5a8383b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a8383b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 -- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a8386be0 msgr2=0x55a5a8384610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a8386be0 0x55a5a8384610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 -- 172.21.15.116:0/672039830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a7ff03f0 con 0x55a5a837e000 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.754+0000 7f69d48ee640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f69c0000c00 tx=0x7f69c0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827dee5c29628db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c00bce00 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f69c00b91f0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69cc0044b0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69c00bdba0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 10 client.16476 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5a80ba7d0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69cc0044b0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f69c00053d0 con 0x55a5a837e000 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 10 client.16476 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 10 client.16476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 20 client.16476 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83992b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 4 client.16476 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.755+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a833a3e0 con 0x55a5a8396e60 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.756+0000 7f69d50ef640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.756+0000 7f69b8ff9640 10 client.16476 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.756+0000 7f69d50ef640 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83992b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef6449b484b8735e server_cookie=3be57e0f3fd91fb8 in_seq=0 out_seq=0 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69b8ff9640 1 -- 172.21.15.116:0/672039830 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5a833a3e0 con 0x55a5a8396e60 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69b8ff9640 10 client.16476 handle_command_reply: tid=0 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 client.16476 shutdown 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 msgr2=0x55a5a83840d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f69c0000c00 tx=0x7f69c0005bd0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 shutdown_connections 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83992b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a8386be0 0x55a5a8384610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69b8ff9640 0 client.16476 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837e000 0x55a5a83840d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 --2- 172.21.15.116:0/672039830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837d6e0 0x55a5a8383b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 >> 172.21.15.116:0/672039830 conn(0x55a5a8377e10 msgr2=0x55a5a8372ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 shutdown_connections 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 1 -- 172.21.15.116:0/672039830 wait complete. 2024-03-19T02:30:22.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.757+0000 7f69da01a740 20 client.16476 trim_cache size 0 max 16384 2024-03-19T02:30:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 Processor -- start 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 -- start start 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 0x55a5a83927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 0x55a5a839ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01ac8b0 con 0x55a5a8386be0 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01ae420 con 0x55a5a837d6e0 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69da01a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01acf80 con 0x55a5a837e000 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 0x55a5a83927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d48ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 0x55a5a839ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41902/0 (socket says 172.21.15.116:41902) 2024-03-19T02:30:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 0x55a5a83927a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47846/0 (socket says 172.21.15.116:47846) 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 -- 172.21.15.116:0/2020708077 learned_addr learned my addr 172.21.15.116:0/2020708077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d48ee640 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 0x55a5a839ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56336/0 (socket says 172.21.15.116:56336) 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 -- 172.21.15.116:0/2020708077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 msgr2=0x55a5a839ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 0x55a5a839ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 -- 172.21.15.116:0/2020708077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 msgr2=0x55a5a83927a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 0x55a5a83927a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.773+0000 7f69d58f0640 1 -- 172.21.15.116:0/2020708077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a8377c40 con 0x55a5a8386be0 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69d58f0640 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f69c8013400 tx=0x7f69c8009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38485f26cd9d334e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69baffd640 1 -- 172.21.15.116:0/2020708077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8022740 con 0x55a5a8386be0 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69baffd640 1 -- 172.21.15.116:0/2020708077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69c8022d90 con 0x55a5a8386be0 2024-03-19T02:30:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69baffd640 1 -- 172.21.15.116:0/2020708077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8010e40 con 0x55a5a8386be0 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69da01a740 1 -- 172.21.15.116:0/2020708077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 msgr2=0x55a5a8380e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69da01a740 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f69c8013400 tx=0x7f69c8009be0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.774+0000 7f69da01a740 1 -- 172.21.15.116:0/2020708077 shutdown_connections 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a8386be0 0x55a5a8380e10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a837e000 0x55a5a839ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- 172.21.15.116:0/2020708077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837d6e0 0x55a5a83927a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- 172.21.15.116:0/2020708077 >> 172.21.15.116:0/2020708077 conn(0x55a5a8377e10 msgr2=0x55a5a837eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- 172.21.15.116:0/2020708077 shutdown_connections 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- 172.21.15.116:0/2020708077 wait complete. 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- start start 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 Processor -- start 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a8380d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a8386be0 0x55a5a83812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01add20 con 0x55a5a837d6e0 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01ad650 con 0x55a5a837e000 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01aa890 con 0x55a5a8386be0 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69da01a740 10 client.0 _ll_register_callbacks cb 0x7f69c00057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69d50ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41912/0 (socket says 172.21.15.116:41912) 2024-03-19T02:30:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69d50ef640 1 -- 172.21.15.116:0/155650413 learned_addr learned my addr 172.21.15.116:0/155650413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.775+0000 7f69d58f0640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a8386be0 0x55a5a83812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d48ee640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a8380d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 -- 172.21.15.116:0/155650413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a8386be0 msgr2=0x55a5a83812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a8386be0 0x55a5a83812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 -- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 msgr2=0x55a5a8380d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a8380d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 -- 172.21.15.116:0/155650413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a836f0a0 con 0x55a5a837d6e0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69d50ef640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009fb0 tx=0x7f69cc004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b868450aa99cb6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc02e340 con 0x55a5a837d6e0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69c8022d90 con 0x55a5a837d6e0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f69cc02e4e0 con 0x55a5a837d6e0 2024-03-19T02:30:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69cc03c050 con 0x55a5a837d6e0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69da01a740 10 client.16482 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5a8349880 con 0x55a5a837d6e0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.776+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69c8022d90 con 0x55a5a837d6e0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f69cc0047d0 con 0x55a5a837d6e0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 10 client.16482 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 10 client.16482 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 20 client.16482 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83ff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 4 client.16482 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5a82e99f0 con 0x55a5a8396e60 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69d48ee640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69d48ee640 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83ff870 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d237d8a1d802ecf3 server_cookie=71488381da434606 in_seq=0 out_seq=0 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69b8ff9640 10 client.16482 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69b8ff9640 1 -- 172.21.15.116:0/155650413 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5a82e99f0 con 0x55a5a8396e60 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69b8ff9640 10 client.16482 handle_command_reply: tid=0 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 client.16482 shutdown 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 msgr2=0x55a5a8380830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009fb0 tx=0x7f69cc004230 comp rx=0 tx=0).stop 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 shutdown_connections 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a5a8396e60 0x55a5a83ff870 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5a8386be0 0x55a5a83812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5a837e000 0x55a5a8380d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69b8ff9640 0 client.16482 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 --2- 172.21.15.116:0/155650413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5a837d6e0 0x55a5a8380830 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.777+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 >> 172.21.15.116:0/155650413 conn(0x55a5a8377e10 msgr2=0x55a5a8372310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 shutdown_connections 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69da01a740 1 -- 172.21.15.116:0/155650413 wait complete. 2024-03-19T02:30:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69da01a740 20 client.16482 trim_cache size 0 max 16384 2024-03-19T02:30:22.785 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 msgr2=0x7f69bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 0x7f69bc0a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009fd0 tx=0x7f69c8004ab0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 msgr2=0x7f69d0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f69c0009230 tx=0x7f69c002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 shutdown_connections 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69bc0a2970 0x7f69bc0a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d01066b0 0x7f69d019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69d0101060 0x7f69d0198520 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 --2- 172.21.15.116:0/2418760261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69d01006a0 0x7f69d0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 >> 172.21.15.116:0/2418760261 conn(0x7f69d00778d0 msgr2=0x7f69d0068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 shutdown_connections 2024-03-19T02:30:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.778+0000 7f69d7353640 1 -- 172.21.15.116:0/2418760261 wait complete. 2024-03-19T02:30:22.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 64451 3.48 MB/sec warmup 105 sec latency 15.809 ms 2024-03-19T02:30:22.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 -- 172.21.15.116:0/2117279901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 msgr2=0x7faeb0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 --2- 172.21.15.116:0/2117279901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb0106a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7faea0009930 tx=0x7faea002bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 -- 172.21.15.116:0/2117279901 shutdown_connections 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 --2- 172.21.15.116:0/2117279901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 --2- 172.21.15.116:0/2117279901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 --2- 172.21.15.116:0/2117279901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb0106a90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 -- 172.21.15.116:0/2117279901 >> 172.21.15.116:0/2117279901 conn(0x7faeb00778d0 msgr2=0x7faeb0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 -- 172.21.15.116:0/2117279901 shutdown_connections 2024-03-19T02:30:22.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.899+0000 7faeb5226640 1 -- 172.21.15.116:0/2117279901 wait complete. 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 Processor -- start 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 -- start start 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0110970 con 0x7faeb01066b0 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01107f0 con 0x7faeb0101060 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeb5226640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0110af0 con 0x7faeb01006a0 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faea7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56366/0 (socket says 172.21.15.116:56366) 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaed9d640 1 -- 172.21.15.116:0/988947385 learned_addr learned my addr 172.21.15.116:0/988947385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41916/0 (socket says 172.21.15.116:41916) 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faea7fff640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47868/0 (socket says 172.21.15.116:47868) 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 -- 172.21.15.116:0/988947385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 msgr2=0x7faeb019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 -- 172.21.15.116:0/988947385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 msgr2=0x7faeb019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaf59e640 1 -- 172.21.15.116:0/988947385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb019f170 con 0x7faeb01066b0 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.900+0000 7faeaed9d640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeaf59e640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7faea8009fd0 tx=0x7faea800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeacd99640 1 -- 172.21.15.116:0/988947385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea800ce20 con 0x7faeb01066b0 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeacd99640 1 -- 172.21.15.116:0/988947385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faea8005020 con 0x7faeb01066b0 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb019f460 con 0x7faeb01066b0 2024-03-19T02:30:22.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeacd99640 1 -- 172.21.15.116:0/988947385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8010690 con 0x7faeb01066b0 2024-03-19T02:30:22.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.901+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb01a6cf0 con 0x7faeb01066b0 2024-03-19T02:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.903+0000 7faeacd99640 1 -- 172.21.15.116:0/988947385 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faea80051d0 con 0x7faeb01066b0 2024-03-19T02:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.903+0000 7faeb7e0b740 1 Processor -- start 2024-03-19T02:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.903+0000 7faeb7e0b740 1 -- start start 2024-03-19T02:30:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a935f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630a92c24b0 con 0x5630a93687d0 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5630a926ad70 con 0x5630a935fbf0 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630a92be550 con 0x5630a935f2d0 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeacd99640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41922/0 (socket says 172.21.15.116:41922) 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47882/0 (socket says 172.21.15.116:47882) 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 -- 172.21.15.116:0/3596996239 learned_addr learned my addr 172.21.15.116:0/3596996239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a935f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeaf59e640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 -- 172.21.15.116:0/3596996239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 msgr2=0x5630a935f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a935f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 -- 172.21.15.116:0/3596996239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 msgr2=0x5630a9368290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faea7fff640 1 -- 172.21.15.116:0/3596996239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630a928d210 con 0x5630a93687d0 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeacd99640 1 -- 172.21.15.116:0/988947385 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7faea80cf880 con 0x7faeb01066b0 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.904+0000 7faeaed9d640 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faea7fff640 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fae98011810 tx=0x7fae980097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ceedde298d15ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faeaf59e640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 0x7fae9c0a4e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faea809b800 tx=0x7faea8086040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faea57fa640 1 -- 172.21.15.116:0/3596996239 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98004040 con 0x5630a93687d0 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faea57fa640 1 -- 172.21.15.116:0/3596996239 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae98011e50 con 0x5630a93687d0 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faea57fa640 1 -- 172.21.15.116:0/3596996239 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98003850 con 0x5630a93687d0 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faeb7e0b740 1 -- 172.21.15.116:0/3596996239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 msgr2=0x5630a936abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.905+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fae98011810 tx=0x7fae980097d0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- 172.21.15.116:0/3596996239 shutdown_connections 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a936abc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9368290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/3596996239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a935f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- 172.21.15.116:0/3596996239 >> 172.21.15.116:0/3596996239 conn(0x5630a9359a00 msgr2=0x5630a9359de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- 172.21.15.116:0/3596996239 shutdown_connections 2024-03-19T02:30:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- 172.21.15.116:0/3596996239 wait complete. 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- start start 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 Processor -- start 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a9365840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9365d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630a936b5c0 con 0x5630a93687d0 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5630a927e310 con 0x5630a935fbf0 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630a936b740 con 0x5630a935f2d0 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faeb7e0b740 10 client.0 _ll_register_callbacks cb 0x5630a92722d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.906+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41926/0 (socket says 172.21.15.116:41926) 2024-03-19T02:30:22.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9365d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 -- 172.21.15.116:0/30564262 learned_addr learned my addr 172.21.15.116:0/30564262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a9365840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 -- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 msgr2=0x5630a9365840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a9365840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 -- 172.21.15.116:0/30564262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 msgr2=0x5630a9365d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9365d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 -- 172.21.15.116:0/30564262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630a8fd63f0 con 0x5630a93687d0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faeaed9d640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9365d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faea7fff640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fae98011620 tx=0x7fae98004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2b91130fd76649 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7fae967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9800fc20 con 0x5630a93687d0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fae9800fdc0 con 0x5630a93687d0 2024-03-19T02:30:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae98011e50 con 0x5630a93687d0 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fae98035050 con 0x5630a93687d0 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faeb7e0b740 10 client.16494 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.907+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5630a90a07d0 con 0x5630a93687d0 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae98011e50 con 0x5630a93687d0 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fae98004a40 con 0x5630a93687d0 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 10 client.16494 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 10 client.16494 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 20 client.16494 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a9378d70 0x5630a937b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 4 client.16494 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630a93203e0 con 0x5630a9378d70 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeaed9d640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a9378d70 0x5630a937b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7fae967fc640 10 client.16494 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.908+0000 7faeaed9d640 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a9378d70 0x5630a937b1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4aae7a5adf4b517 server_cookie=1698d2a8bbf9b8af in_seq=0 out_seq=0 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7fae967fc640 1 -- 172.21.15.116:0/30564262 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630a93203e0 con 0x5630a9378d70 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7fae967fc640 10 client.16494 handle_command_reply: tid=0 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 client.16494 shutdown 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 msgr2=0x5630a93662c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fae98011620 tx=0x7fae98004420 comp rx=0 tx=0).stop 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 shutdown_connections 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a9378d70 0x5630a937b1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a93687d0 0x5630a93662c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9365d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/30564262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a9365840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7fae967fc640 0 client.16494 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 >> 172.21.15.116:0/30564262 conn(0x5630a9359a00 msgr2=0x5630a935b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 shutdown_connections 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 1 -- 172.21.15.116:0/30564262 wait complete. 2024-03-19T02:30:22.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.909+0000 7faeb7e0b740 20 client.16494 trim_cache size 0 max 16384 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.926+0000 7faeb7e0b740 1 Processor -- start 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.926+0000 7faeb7e0b740 1 -- start start 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9374360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a93687d0 0x5630a9380a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b0d60 con 0x5630a935f2d0 2024-03-19T02:30:22.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b28d0 con 0x5630a935fbf0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b1430 con 0x5630a93687d0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9374360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41932/0 (socket says 172.21.15.116:41932) 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 -- 172.21.15.116:0/458045865 learned_addr learned my addr 172.21.15.116:0/458045865 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a93687d0 0x5630a9380a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 -- 172.21.15.116:0/458045865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a93687d0 msgr2=0x5630a9380a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a93687d0 0x5630a9380a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 -- 172.21.15.116:0/458045865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 msgr2=0x5630a9374360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9374360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 -- 172.21.15.116:0/458045865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630a932f880 con 0x5630a935f2d0 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faeaed9d640 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9374360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:22.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea7fff640 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fae98005e00 tx=0x7fae980036c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d391c59cc1f5b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea4ff9640 1 -- 172.21.15.116:0/458045865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980037b0 con 0x5630a935f2d0 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea4ff9640 1 -- 172.21.15.116:0/458045865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae98011e50 con 0x5630a935f2d0 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.927+0000 7faea4ff9640 1 -- 172.21.15.116:0/458045865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae98011290 con 0x5630a935f2d0 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 -- 172.21.15.116:0/458045865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 msgr2=0x5630a9367f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fae98005e00 tx=0x7fae980036c0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 -- 172.21.15.116:0/458045865 shutdown_connections 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a93687d0 0x5630a9380a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a935fbf0 0x5630a9374360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/458045865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935f2d0 0x5630a9367f70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 -- 172.21.15.116:0/458045865 >> 172.21.15.116:0/458045865 conn(0x5630a9359a00 msgr2=0x5630a9360b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 -- 172.21.15.116:0/458045865 shutdown_connections 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.928+0000 7faeb7e0b740 1 -- 172.21.15.116:0/458045865 wait complete. 2024-03-19T02:30:22.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 -- start start 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 Processor -- start 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a936b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a93687d0 0x5630a93dd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b21d0 con 0x5630a935fbf0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b1b00 con 0x5630a93687d0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01aed40 con 0x5630a935f2d0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeb7e0b740 10 client.0 _ll_register_callbacks cb 0x7fae98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41948/0 (socket says 172.21.15.116:41948) 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faea7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a936b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 -- 172.21.15.116:0/523294102 learned_addr learned my addr 172.21.15.116:0/523294102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:22.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faea7fff640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a93687d0 0x5630a93dd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 -- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 msgr2=0x5630a936b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a936b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 -- 172.21.15.116:0/523294102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a93687d0 msgr2=0x5630a93dd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a93687d0 0x5630a93dd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 -- 172.21.15.116:0/523294102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630a936c780 con 0x5630a935fbf0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7faeaed9d640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7faea0005ec0 tx=0x7faea002c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144148d4c3246300 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.929+0000 7fae967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0051070 con 0x5630a935fbf0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7faea004ab10 con 0x5630a935fbf0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae98011e50 con 0x5630a935fbf0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faea005c050 con 0x5630a935fbf0 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 10 client.16500 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5630a93de740 con 0x5630a935fbf0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae98011e50 con 0x5630a935fbf0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7faea00038f0 con 0x5630a935fbf0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 10 client.16500 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 10 client.16500 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 20 client.16500 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a93deea0 0x5630a93e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 4 client.16500 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630a92cf9f0 con 0x5630a93deea0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.930+0000 7faea7fff640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a93deea0 0x5630a93e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faea7fff640 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a93deea0 0x5630a93e12c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68ef469146484ccb server_cookie=8d51fea8cf06745b in_seq=0 out_seq=0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7fae967fc640 10 client.16500 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7fae967fc640 1 -- 172.21.15.116:0/523294102 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630a92cf9f0 con 0x5630a93deea0 2024-03-19T02:30:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7fae967fc640 10 client.16500 handle_command_reply: tid=0 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 client.16500 shutdown 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 msgr2=0x5630a936be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7faea0005ec0 tx=0x7faea002c940 comp rx=0 tx=0).stop 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 shutdown_connections 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5630a93deea0 0x5630a93e12c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630a93687d0 0x5630a93dd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630a935fbf0 0x5630a936be90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7fae967fc640 0 client.16500 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 --2- 172.21.15.116:0/523294102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630a935f2d0 0x5630a936b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 >> 172.21.15.116:0/523294102 conn(0x5630a9359a00 msgr2=0x5630a9365800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 shutdown_connections 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 1 -- 172.21.15.116:0/523294102 wait complete. 2024-03-19T02:30:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.931+0000 7faeb7e0b740 20 client.16500 trim_cache size 0 max 16384 2024-03-19T02:30:22.939 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 msgr2=0x7fae9c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 0x7fae9c0a4e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faea809b800 tx=0x7faea8086040 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 msgr2=0x7faeb019ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7faea8009fd0 tx=0x7faea800ef10 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 shutdown_connections 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb01066b0 0x7faeb019ea40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb0101060 0x7faeb019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 --2- 172.21.15.116:0/988947385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeb01006a0 0x7faeb019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 >> 172.21.15.116:0/988947385 conn(0x7faeb00778d0 msgr2=0x7faeb010c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 shutdown_connections 2024-03-19T02:30:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:22.932+0000 7faeb5226640 1 -- 172.21.15.116:0/988947385 wait complete. 2024-03-19T02:30:22.941 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:22 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2479330189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:22.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:30:23.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.054+0000 7f4116587640 1 -- 172.21.15.116:0/3867423575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4110101060 msgr2=0x7f411010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.054+0000 7f4116587640 1 --2- 172.21.15.116:0/3867423575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4110101060 0x7f411010d570 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4108009cd0 tx=0x7f410802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 -- 172.21.15.116:0/3867423575 shutdown_connections 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 --2- 172.21.15.116:0/3867423575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4110101060 0x7f411010d570 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 --2- 172.21.15.116:0/3867423575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41101006a0 0x7f4110100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 --2- 172.21.15.116:0/3867423575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101066b0 0x7f4110106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 -- 172.21.15.116:0/3867423575 >> 172.21.15.116:0/3867423575 conn(0x7f41100778d0 msgr2=0x7f4110077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 -- 172.21.15.116:0/3867423575 shutdown_connections 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 -- 172.21.15.116:0/3867423575 wait complete. 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 Processor -- start 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.055+0000 7f4116587640 1 -- start start 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 0x7f4110110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4110101060 0x7f41101106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4110069240 con 0x7f41101066b0 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41100690c0 con 0x7f4110101060 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41100693c0 con 0x7f41101006a0 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 0x7f4110110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4110101060 0x7f41101106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41966/0 (socket says 172.21.15.116:41966) 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 0x7f4110110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56418/0 (socket says 172.21.15.116:56418) 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 -- 172.21.15.116:0/302545470 learned_addr learned my addr 172.21.15.116:0/302545470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 -- 172.21.15.116:0/302545470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 msgr2=0x7f4110110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 0x7f4110110180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 -- 172.21.15.116:0/302545470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4110101060 msgr2=0x7f41101106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4110101060 0x7f41101106c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 -- 172.21.15.116:0/302545470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41101a6460 con 0x7f41101066b0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4114b24640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4108004af0 tx=0x7f4108035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410d7fa640 1 -- 172.21.15.116:0/302545470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108039040 con 0x7f41101066b0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41101a66f0 con 0x7f41101066b0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410d7fa640 1 -- 172.21.15.116:0/302545470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4108044070 con 0x7f41101066b0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f410d7fa640 1 -- 172.21.15.116:0/302545470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41080042a0 con 0x7f41101066b0 2024-03-19T02:30:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.056+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41101a6c30 con 0x7f41101066b0 2024-03-19T02:30:23.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.059+0000 7f410d7fa640 1 -- 172.21.15.116:0/302545470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4108003970 con 0x7f41101066b0 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.059+0000 7f4119171740 1 Processor -- start 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.059+0000 7f4119171740 1 -- start start 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bcb94fb4b0 con 0x55bcb95982d0 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcb94a3d70 con 0x55bcb95a17d0 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4119171740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcb94f7550 con 0x55bcb9598bf0 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41972/0 (socket says 172.21.15.116:41972) 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 -- 172.21.15.116:0/4264548734 learned_addr learned my addr 172.21.15.116:0/4264548734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410ffff640 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410d7fa640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 0x7f40f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 -- 172.21.15.116:0/4264548734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 msgr2=0x55bcb95a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 -- 172.21.15.116:0/4264548734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 msgr2=0x55bcb95a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f4114b24640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 0x7f40f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 -- 172.21.15.116:0/4264548734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb94c6210 con 0x55bcb95982d0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410d7fa640 1 -- 172.21.15.116:0/302545470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f41080f1470 con 0x7f41101066b0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.060+0000 7f410f7fe640 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4104013840 tx=0x7f4104005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98ce3ea9b5d9c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4114b24640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 0x7f40f80a4e60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41080eea00 tx=0x7f41080326f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f40f67fc640 1 -- 172.21.15.116:0/4264548734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104013170 con 0x55bcb95982d0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f40f67fc640 1 -- 172.21.15.116:0/4264548734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41040054d0 con 0x55bcb95982d0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f40f67fc640 1 -- 172.21.15.116:0/4264548734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104004c60 con 0x55bcb95982d0 2024-03-19T02:30:23.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 -- 172.21.15.116:0/4264548734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 msgr2=0x55bcb95986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4104013840 tx=0x7f4104005f90 comp rx=0 tx=0).stop 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 -- 172.21.15.116:0/4264548734 shutdown_connections 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 --2- 172.21.15.116:0/4264548734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95986b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.061+0000 7f4119171740 1 -- 172.21.15.116:0/4264548734 >> 172.21.15.116:0/4264548734 conn(0x55bcb9592a00 msgr2=0x55bcb9592de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- 172.21.15.116:0/4264548734 shutdown_connections 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- 172.21.15.116:0/4264548734 wait complete. 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- start start 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 Processor -- start 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb959f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bcb95a4560 con 0x55bcb9598bf0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcb94b7310 con 0x55bcb95a17d0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.062+0000 7f4119171740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bcb95a46e0 con 0x55bcb95982d0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f4119171740 10 client.0 _ll_register_callbacks cb 0x55bcb94ab2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41978/0 (socket says 172.21.15.116:41978) 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56426/0 (socket says 172.21.15.116:56426) 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 -- 172.21.15.116:0/317977405 learned_addr learned my addr 172.21.15.116:0/317977405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410f7fe640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb959f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 -- 172.21.15.116:0/317977405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 msgr2=0x55bcb959e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 -- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 msgr2=0x55bcb959f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb959f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 -- 172.21.15.116:0/317977405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb920f3f0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410f7fe640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f410ffff640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f41000126e0 tx=0x7f4100012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a7c09b38100a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f40ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41000120d0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f410000f920 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41040054d0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4100012270 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f4119171740 10 client.16512 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.063+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcb92d97d0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41040054d0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f4100010cd0 con 0x55bcb9598bf0 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 10 client.16512 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 20 client.16512 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb95b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 4 client.16512 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcb95593e0 con 0x55bcb95b1c30 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f410f7fe640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb95b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f410f7fe640 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb95b4020 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab48ba1eba462533 server_cookie=eb747b092aa4ef41 in_seq=0 out_seq=0 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.064+0000 7f40ef7fe640 10 client.16512 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f40ef7fe640 1 -- 172.21.15.116:0/317977405 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bcb95593e0 con 0x55bcb95b1c30 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f40ef7fe640 10 client.16512 handle_command_reply: tid=0 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 client.16512 shutdown 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 msgr2=0x55bcb959ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f41000126e0 tx=0x7f4100012ba0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 shutdown_connections 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb95b4020 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb959f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f40ef7fe640 0 client.16512 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb9598bf0 0x55bcb959ed10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 --2- 172.21.15.116:0/317977405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb95982d0 0x55bcb959e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 >> 172.21.15.116:0/317977405 conn(0x55bcb9592a00 msgr2=0x55bcb9594750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 shutdown_connections 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.065+0000 7f4119171740 1 -- 172.21.15.116:0/317977405 wait complete. 2024-03-19T02:30:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.066+0000 7f4119171740 20 client.16512 trim_cache size 0 max 16384 2024-03-19T02:30:23.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 Processor -- start 2024-03-19T02:30:23.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 -- start start 2024-03-19T02:30:23.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95982d0 0x55bcb95a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb959d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41101b0c20 con 0x55bcb95a17d0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41101b2790 con 0x55bcb95982d0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f4119171740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41101b12f0 con 0x55bcb9598bf0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb959d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41986/0 (socket says 172.21.15.116:41986) 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb959d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56438/0 (socket says 172.21.15.116:56438) 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 -- 172.21.15.116:0/2098716570 learned_addr learned my addr 172.21.15.116:0/2098716570 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95982d0 0x55bcb95a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 -- 172.21.15.116:0/2098716570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 msgr2=0x55bcb959d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb959d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 -- 172.21.15.116:0/2098716570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95982d0 msgr2=0x55bcb95a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95982d0 0x55bcb95a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.081+0000 7f410f7fe640 1 -- 172.21.15.116:0/2098716570 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb9598a20 con 0x55bcb95a17d0 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f410f7fe640 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41040098e0 tx=0x7f4104005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684ed4e3fdbbe925 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f40f5ffb640 1 -- 172.21.15.116:0/2098716570 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104028070 con 0x55bcb95a17d0 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f40f5ffb640 1 -- 172.21.15.116:0/2098716570 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4104031800 con 0x55bcb95a17d0 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f40f5ffb640 1 -- 172.21.15.116:0/2098716570 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104004040 con 0x55bcb95a17d0 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 -- 172.21.15.116:0/2098716570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 msgr2=0x55bcb95ad330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41040098e0 tx=0x7f4104005f90 comp rx=0 tx=0).stop 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 -- 172.21.15.116:0/2098716570 shutdown_connections 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95a17d0 0x55bcb95ad330 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb959d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 --2- 172.21.15.116:0/2098716570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95982d0 0x55bcb95a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 -- 172.21.15.116:0/2098716570 >> 172.21.15.116:0/2098716570 conn(0x55bcb9592a00 msgr2=0x55bcb9591310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 -- 172.21.15.116:0/2098716570 shutdown_connections 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.082+0000 7f4119171740 1 -- 172.21.15.116:0/2098716570 wait complete. 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 -- start start 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 Processor -- start 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41101b2090 con 0x55bcb95982d0 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41101b19c0 con 0x55bcb95a17d0 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41101aec00 con 0x55bcb9598bf0 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f4119171740 10 client.0 _ll_register_callbacks cb 0x7f4100009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f410f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41998/0 (socket says 172.21.15.116:41998) 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f410f7fe640 1 -- 172.21.15.116:0/1726768071 learned_addr learned my addr 172.21.15.116:0/1726768071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f410ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.083+0000 7f410f7fe640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 -- 172.21.15.116:0/1726768071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 msgr2=0x55bcb95a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 -- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 msgr2=0x55bcb95a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 -- 172.21.15.116:0/1726768071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb95a5780 con 0x55bcb95982d0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410ffff640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f410f7fe640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4104005e00 tx=0x7f4104031c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b295c308436175 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f40ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104028050 con 0x55bcb95982d0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4104031e00 con 0x55bcb95982d0 2024-03-19T02:30:23.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bcb95089f0 con 0x55bcb95982d0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f410403f050 con 0x55bcb95982d0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f4119171740 10 client.16518 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcb9617c50 con 0x55bcb95982d0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.084+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bcb95089f0 con 0x55bcb95982d0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f4104004150 con 0x55bcb95982d0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 10 client.16518 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 20 client.16518 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb961a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 4 client.16518 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcb9502d40 con 0x55bcb95b1c30 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f410ffff640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb961a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:23.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f410ffff640 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb961a1c0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99fee364720c920e server_cookie=f1b0c80ad37b3e3b in_seq=0 out_seq=0 2024-03-19T02:30:23.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.085+0000 7f40ef7fe640 10 client.16518 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f40ef7fe640 1 -- 172.21.15.116:0/1726768071 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcb9502d40 con 0x55bcb95b1c30 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f40ef7fe640 10 client.16518 handle_command_reply: tid=0 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 client.16518 shutdown 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 msgr2=0x55bcb95a98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4104005e00 tx=0x7f4104031c60 comp rx=0 tx=0).stop 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 shutdown_connections 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bcb95b1c30 0x55bcb961a1c0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcb95a17d0 0x55bcb95a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f40ef7fe640 0 client.16518 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.086+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bcb9598bf0 0x55bcb95a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4119171740 1 --2- 172.21.15.116:0/1726768071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcb95982d0 0x55bcb95a98d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 >> 172.21.15.116:0/1726768071 conn(0x55bcb9592a00 msgr2=0x55bcb9591ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 shutdown_connections 2024-03-19T02:30:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4119171740 1 -- 172.21.15.116:0/1726768071 wait complete. 2024-03-19T02:30:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4119171740 20 client.16518 trim_cache size 0 max 16384 2024-03-19T02:30:23.093 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 msgr2=0x7f40f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 0x7f40f80a4e60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41080eea00 tx=0x7f41080326f0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 msgr2=0x7f4110110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4108004af0 tx=0x7f4108035ce0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 shutdown_connections 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f40f80a29a0 0x7f40f80a4e60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41101066b0 0x7f4110110c00 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4110101060 0x7f41101106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 --2- 172.21.15.116:0/302545470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41101006a0 0x7f4110110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 >> 172.21.15.116:0/302545470 conn(0x7f41100778d0 msgr2=0x7f4110102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 shutdown_connections 2024-03-19T02:30:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:23.087+0000 7f4116587640 1 -- 172.21.15.116:0/302545470 wait complete. 2024-03-19T02:30:23.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:22 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2479330189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:22 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2479330189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 65011 3.46 MB/sec warmup 106 sec latency 44.007 ms 2024-03-19T02:30:24.099 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:30:24.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:30:24.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:30:24.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:23 smithi116 ceph-mon[32263]: pgmap v80: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:30:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:23 smithi162 ceph-mon[35761]: pgmap v80: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 -- 172.21.15.116:0/2176002015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301006e0 msgr2=0x7f5f30100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2176002015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301006e0 0x7f5f30100b40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009900 tx=0x7f5f2002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 -- 172.21.15.116:0/2176002015 shutdown_connections 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2176002015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f30101080 0x7f5f3010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2176002015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301006e0 0x7f5f30100b40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2176002015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f301055c0 0x7f5f301059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 -- 172.21.15.116:0/2176002015 >> 172.21.15.116:0/2176002015 conn(0x7f5f300777b0 msgr2=0x7f5f30077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 -- 172.21.15.116:0/2176002015 shutdown_connections 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.217+0000 7f5f381bc640 1 -- 172.21.15.116:0/2176002015 wait complete. 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 Processor -- start 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 -- start start 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f301006e0 0x7f5f300fe180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 0x7f5f300fe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30110960 con 0x7f5f301055c0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301107e0 con 0x7f5f301006e0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f381bc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30110ae0 con 0x7f5f30101080 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42020/0 (socket says 172.21.15.116:42020) 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f35f58640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f301006e0 0x7f5f300fe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f35757640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 0x7f5f300fe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 -- 172.21.15.116:0/2378305066 learned_addr learned my addr 172.21.15.116:0/2378305066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 -- 172.21.15.116:0/2378305066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 msgr2=0x7f5f300fe6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 0x7f5f300fe6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 -- 172.21.15.116:0/2378305066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f301006e0 msgr2=0x7f5f300fe180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f301006e0 0x7f5f300fe180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f36759640 1 -- 172.21.15.116:0/2378305066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f301a6510 con 0x7f5f301055c0 2024-03-19T02:30:24.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.218+0000 7f5f35757640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 0x7f5f300fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f36759640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800eb10 tx=0x7f5f2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f26ffd640 1 -- 172.21.15.116:0/2378305066 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28004b20 con 0x7f5f301055c0 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f301a6800 con 0x7f5f301055c0 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f26ffd640 1 -- 172.21.15.116:0/2378305066 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f28004cc0 con 0x7f5f301055c0 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f26ffd640 1 -- 172.21.15.116:0/2378305066 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28010690 con 0x7f5f301055c0 2024-03-19T02:30:24.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.219+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f301a6d20 con 0x7f5f301055c0 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 Processor -- start 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f26ffd640 1 -- 172.21.15.116:0/2378305066 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5f28005020 con 0x7f5f301055c0 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 -- start start 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b0633760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b063c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b063ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5567b0597160 con 0x5567b0633380 2024-03-19T02:30:24.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5567b053fe10 con 0x5567b063c880 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5567b0593200 con 0x5567b0633ca0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b0633760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b0633760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42024/0 (socket says 172.21.15.116:42024) 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f35757640 1 -- 172.21.15.116:0/2913999838 learned_addr learned my addr 172.21.15.116:0/2913999838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f35f58640 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b063c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f35757640 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b063ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.222+0000 7f5f26ffd640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 0x7f5f1c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f36759640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 0x7f5f1c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 -- 172.21.15.116:0/2913999838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 msgr2=0x5567b063ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b063ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 -- 172.21.15.116:0/2913999838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 msgr2=0x5567b0633760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b0633760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f26ffd640 1 -- 172.21.15.116:0/2378305066 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f5f280cf780 con 0x7f5f301055c0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 -- 172.21.15.116:0/2913999838 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b03737e0 con 0x5567b0633ca0 2024-03-19T02:30:24.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f35f58640 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b063c340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009c00 tx=0x7f5f2c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fca47625b0036e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f36759640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 0x7f5f1c0a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f280ccc00 tx=0x7f5f28005b40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f1bfff640 1 -- 172.21.15.116:0/2913999838 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c009db0 con 0x5567b0633ca0 2024-03-19T02:30:24.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f1bfff640 1 -- 172.21.15.116:0/2913999838 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f2c005020 con 0x5567b0633ca0 2024-03-19T02:30:24.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.223+0000 7f5f1bfff640 1 -- 172.21.15.116:0/2913999838 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c004a00 con 0x5567b0633ca0 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/2913999838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 msgr2=0x5567b063c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b063c340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009c00 tx=0x7f5f2c011810 comp rx=0 tx=0).stop 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/2913999838 shutdown_connections 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b063ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b063c340 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/2913999838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b0633760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/2913999838 >> 172.21.15.116:0/2913999838 conn(0x5567b062dab0 msgr2=0x5567b062de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/2913999838 shutdown_connections 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.224+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/2913999838 wait complete. 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 -- start start 2024-03-19T02:30:24.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 Processor -- start 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633380 0x5567b0639860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0639da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5567b063f670 con 0x5567b063c880 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5567b05531a0 con 0x5567b0633380 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5567b063f7f0 con 0x5567b0633ca0 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f3ad9b740 10 client.0 _ll_register_callbacks cb 0x5567b02aac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42036/0 (socket says 172.21.15.116:42036) 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f35757640 1 -- 172.21.15.116:0/674664694 learned_addr learned my addr 172.21.15.116:0/674664694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f35f58640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0639da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.225+0000 7f5f35757640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633380 0x5567b0639860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 -- 172.21.15.116:0/674664694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 msgr2=0x5567b0639da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0639da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 -- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633380 msgr2=0x5567b0639860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633380 0x5567b0639860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 -- 172.21.15.116:0/674664694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b03d3f70 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f35757640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5f2002c210 tx=0x7f5f20005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49e1ec39f75b578 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002c940 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f5f2002cae0 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2c005020 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f20055050 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f3ad9b740 10 client.16530 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567b026b170 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.226+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f2c005020 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f5f20003820 con 0x5567b063c880 2024-03-19T02:30:24.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 10 client.16530 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 20 client.16530 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b064cbb0 0x5567b064f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 4 client.16530 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567b05a46a0 con 0x5567b064cbb0 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f35f58640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b064cbb0 0x5567b064f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f35f58640 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b064cbb0 0x5567b064f010 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30c0d80246486d88 server_cookie=48b5ae54fcbb34c0 in_seq=0 out_seq=0 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.227+0000 7f5f197fa640 10 client.16530 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.228+0000 7f5f197fa640 1 -- 172.21.15.116:0/674664694 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567b05a46a0 con 0x5567b064cbb0 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.228+0000 7f5f197fa640 10 client.16530 handle_command_reply: tid=0 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.228+0000 7f5f3ad9b740 1 client.16530 shutdown 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.228+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 msgr2=0x5567b063a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5f2002c210 tx=0x7f5f20005e40 comp rx=0 tx=0).stop 2024-03-19T02:30:24.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 shutdown_connections 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b064cbb0 0x5567b064f010 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b063a2e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0639da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/674664694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633380 0x5567b0639860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f197fa640 0 client.16530 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 >> 172.21.15.116:0/674664694 conn(0x5567b062dab0 msgr2=0x5567b062f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 shutdown_connections 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/674664694 wait complete. 2024-03-19T02:30:24.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.229+0000 7f5f3ad9b740 20 client.16530 trim_cache size 0 max 16384 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 Processor -- start 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 -- start start 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0648410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b06548e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301b0d60 con 0x5567b0633380 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301b28d0 con 0x5567b063c880 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301b1430 con 0x5567b0633ca0 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42050/0 (socket says 172.21.15.116:42050) 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35f58640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0648410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 -- 172.21.15.116:0/4090975668 learned_addr learned my addr 172.21.15.116:0/4090975668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b06548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 -- 172.21.15.116:0/4090975668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 msgr2=0x5567b0648410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.248+0000 7f5f35757640 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0648410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f35757640 1 -- 172.21.15.116:0/4090975668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 msgr2=0x5567b06548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f35757640 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b06548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f35757640 1 -- 172.21.15.116:0/4090975668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567b0633ad0 con 0x5567b0633380 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f35f58640 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0648410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f35757640 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005d00 tx=0x7f5f20005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f731b783e5fe25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f1b7fe640 1 -- 172.21.15.116:0/4090975668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002cb40 con 0x5567b0633380 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f1b7fe640 1 -- 172.21.15.116:0/4090975668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f200035f0 con 0x5567b0633380 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.249+0000 7f5f1b7fe640 1 -- 172.21.15.116:0/4090975668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002e530 con 0x5567b0633380 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/4090975668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 msgr2=0x5567b063bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005d00 tx=0x7f5f20005c90 comp rx=0 tx=0).stop 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/4090975668 shutdown_connections 2024-03-19T02:30:24.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b063c880 0x5567b06548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633ca0 0x5567b0648410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/4090975668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b0633380 0x5567b063bf90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/4090975668 >> 172.21.15.116:0/4090975668 conn(0x5567b062dab0 msgr2=0x5567b062ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/4090975668 shutdown_connections 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/4090975668 wait complete. 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 -- start start 2024-03-19T02:30:24.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.250+0000 7f5f3ad9b740 1 Processor -- start 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633380 0x5567b0644a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633ca0 0x5567b0641a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301b21d0 con 0x5567b063c880 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301b1b00 con 0x5567b0633ca0 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f301aed40 con 0x5567b0633380 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f3ad9b740 10 client.0 _ll_register_callbacks cb 0x7f5f200044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42056/0 (socket says 172.21.15.116:42056) 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 -- 172.21.15.116:0/809819210 learned_addr learned my addr 172.21.15.116:0/809819210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35f58640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633ca0 0x5567b0641a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633380 0x5567b0644a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 -- 172.21.15.116:0/809819210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633380 msgr2=0x5567b0644a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633380 0x5567b0644a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 -- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633ca0 msgr2=0x5567b0641a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633ca0 0x5567b0641a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 -- 172.21.15.116:0/809819210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567b0645030 con 0x5567b063c880 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f35757640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5f2002e9b0 tx=0x7f5f20005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93df0439ddcfa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.251+0000 7f5f197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2002e4b0 con 0x5567b063c880 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f5f20003e60 con 0x5567b063c880 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f200035f0 con 0x5567b063c880 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f200039b0 con 0x5567b063c880 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 10 client.16536 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:24.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567b06b3920 con 0x5567b063c880 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f200035f0 con 0x5567b063c880 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f5f20041b30 con 0x5567b063c880 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 10 client.16536 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 20 client.16536 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b06b3fb0 0x5567b06b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 4 client.16536 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5567b059e9f0 con 0x5567b06b3fb0 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.252+0000 7f5f35f58640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b06b3fb0 0x5567b06b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.253+0000 7f5f197fa640 10 client.16536 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.253+0000 7f5f35f58640 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b06b3fb0 0x5567b06b63a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8141c40504538893 server_cookie=6a44d46bf09a6078 in_seq=0 out_seq=0 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f197fa640 1 -- 172.21.15.116:0/809819210 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5567b059e9f0 con 0x5567b06b3fb0 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f197fa640 10 client.16536 handle_command_reply: tid=0 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 client.16536 shutdown 2024-03-19T02:30:24.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 msgr2=0x5567b0641f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5f2002e9b0 tx=0x7f5f20005bc0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 shutdown_connections 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5567b06b3fb0 0x5567b06b63a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.254+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567b063c880 0x5567b0641f50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f197fa640 0 client.16536 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567b0633ca0 0x5567b0641a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 1 --2- 172.21.15.116:0/809819210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567b0633380 0x5567b0644a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 >> 172.21.15.116:0/809819210 conn(0x5567b062dab0 msgr2=0x5567b062f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 shutdown_connections 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 1 -- 172.21.15.116:0/809819210 wait complete. 2024-03-19T02:30:24.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f3ad9b740 20 client.16536 trim_cache size 0 max 16384 2024-03-19T02:30:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:24.261 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:30:24.261 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "68028166-92f6-4387-b709-ced552bced39", 2024-03-19T02:30:24.261 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:30:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 msgr2=0x7f5f1c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 0x7f5f1c0a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f280ccc00 tx=0x7f5f28005b40 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 msgr2=0x7f5f300fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800eb10 tx=0x7f5f2800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 shutdown_connections 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f1c0a2970 0x7f5f1c0a4e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f301055c0 0x7f5f300fec00 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f30101080 0x7f5f300fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 --2- 172.21.15.116:0/2378305066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f301006e0 0x7f5f300fe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 >> 172.21.15.116:0/2378305066 conn(0x7f5f300777b0 msgr2=0x7f5f30069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 shutdown_connections 2024-03-19T02:30:24.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.255+0000 7f5f381bc640 1 -- 172.21.15.116:0/2378305066 wait complete. 2024-03-19T02:30:24.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:23 smithi154 ceph-mon[35693]: pgmap v80: 129 pgs: 129 active+clean; 59 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:30:24.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:30:24.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:30:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.420+0000 7fb361812640 1 -- 172.21.15.116:0/1777593232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0693e0 msgr2=0x7fb35c10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.420+0000 7fb361812640 1 --2- 172.21.15.116:0/1777593232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0693e0 0x7fb35c10e680 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009890 tx=0x7fb34c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:30:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- 172.21.15.116:0/1777593232 shutdown_connections 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- 172.21.15.116:0/1777593232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0693e0 0x7fb35c10e680 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- 172.21.15.116:0/1777593232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 0x7fb35c068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- 172.21.15.116:0/1777593232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c107870 0x7fb35c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- 172.21.15.116:0/1777593232 >> 172.21.15.116:0/1777593232 conn(0x7fb35c077820 msgr2=0x7fb35c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- 172.21.15.116:0/1777593232 shutdown_connections 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- 172.21.15.116:0/1777593232 wait complete. 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 Processor -- start 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- start start 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 0x7fb35c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c0693e0 0x7fb35c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c111a70 con 0x7fb35c107870 2024-03-19T02:30:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1118f0 con 0x7fb35c068a20 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.421+0000 7fb361812640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c111bf0 con 0x7fb35c0693e0 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42070/0 (socket says 172.21.15.116:42070) 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 -- 172.21.15.116:0/3286196490 learned_addr learned my addr 172.21.15.116:0/3286196490 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35affd640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 0x7fb35c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c0693e0 0x7fb35c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 -- 172.21.15.116:0/3286196490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c0693e0 msgr2=0x7fb35c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c0693e0 0x7fb35c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 -- 172.21.15.116:0/3286196490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 msgr2=0x7fb35c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 0x7fb35c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 -- 172.21.15.116:0/3286196490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c19ceb0 con 0x7fb35c107870 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb34c02c0b0 tx=0x7fb34c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb360810640 1 -- 172.21.15.116:0/3286196490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c041020 con 0x7fb35c107870 2024-03-19T02:30:24.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c19d140 con 0x7fb35c107870 2024-03-19T02:30:24.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb360810640 1 -- 172.21.15.116:0/3286196490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb34c02c9c0 con 0x7fb35c107870 2024-03-19T02:30:24.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb360810640 1 -- 172.21.15.116:0/3286196490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c039040 con 0x7fb35c107870 2024-03-19T02:30:24.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.422+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c1a4a30 con 0x7fb35c107870 2024-03-19T02:30:24.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.424+0000 7fb360810640 1 -- 172.21.15.116:0/3286196490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb34c044020 con 0x7fb35c107870 2024-03-19T02:30:24.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 Processor -- start 2024-03-19T02:30:24.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 -- start start 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb360810640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 0x7fb3480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f9577619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957761f30 0x55f957764320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9576bc500 con 0x55f957758a30 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f957664be0 con 0x55f957759350 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb364403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f9576b85a0 con 0x55f957761f30 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42084/0 (socket says 172.21.15.116:42084) 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35affd640 1 -- 172.21.15.116:0/4288557976 learned_addr learned my addr 172.21.15.116:0/4288557976 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 0x7fb3480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb360810640 1 -- 172.21.15.116:0/3286196490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb34c0f0890 con 0x7fb35c107870 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f9577619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.425+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957761f30 0x55f957764320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 -- 172.21.15.116:0/4288557976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957761f30 msgr2=0x55f957764320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957761f30 0x55f957764320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 -- 172.21.15.116:0/4288557976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 msgr2=0x55f9577619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f9577619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 -- 172.21.15.116:0/4288557976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f957686000 con 0x55f957758a30 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 0x7fb3480a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb344005e10 tx=0x7fb34400ab10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb35affd640 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e5d0 tx=0x7fb3540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a69014c7e2cb25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb351ffb640 1 -- 172.21.15.116:0/4288557976 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35400dad0 con 0x55f957758a30 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb351ffb640 1 -- 172.21.15.116:0/4288557976 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb35401c070 con 0x55f957758a30 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.426+0000 7fb351ffb640 1 -- 172.21.15.116:0/4288557976 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354016380 con 0x55f957758a30 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- 172.21.15.116:0/4288557976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 msgr2=0x55f957758e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e5d0 tx=0x7fb3540095c0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- 172.21.15.116:0/4288557976 shutdown_connections 2024-03-19T02:30:24.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957761f30 0x55f957764320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f9577619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 --2- 172.21.15.116:0/4288557976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957758e10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- 172.21.15.116:0/4288557976 >> 172.21.15.116:0/4288557976 conn(0x55f957753160 msgr2=0x55f957753540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- 172.21.15.116:0/4288557976 shutdown_connections 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- 172.21.15.116:0/4288557976 wait complete. 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 -- start start 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.427+0000 7fb364403740 1 Processor -- start 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95775ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95775f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 0x55f95775f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f957764d20 con 0x55f957761f30 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f957678160 con 0x55f957759350 2024-03-19T02:30:24.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f957764ea0 con 0x55f957758a30 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 10 client.0 _ll_register_callbacks cb 0x55f9576b4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 0x55f95775f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95775f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95775ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95775f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48062/0 (socket says 172.21.15.116:48062) 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35a7fc640 1 -- 172.21.15.116:0/2448766392 learned_addr learned my addr 172.21.15.116:0/2448766392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95775ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56556/0 (socket says 172.21.15.116:56556) 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 -- 172.21.15.116:0/2448766392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 msgr2=0x55f95775ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95775ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 -- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 msgr2=0x55f95775f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95775f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 -- 172.21.15.116:0/2448766392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f95749a7d0 con 0x55f957761f30 2024-03-19T02:30:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 0x55f95775f8a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0edd70 tx=0x7fb34c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc766250ae8afefc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb33affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0bd380 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb35401c070 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.428+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb34c0baf80 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb34c0bcc80 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 10 client.16548 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f9573d03f0 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb35401c070 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb34c03c030 con 0x55f957761f30 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 10 client.16548 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 20 client.16548 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957772340 0x55f9577747a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 4 client.16548 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9576c9a40 con 0x55f957772340 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb35affd640 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957772340 0x55f9577747a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb35affd640 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957772340 0x55f9577747a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db83cf59e7386fac server_cookie=ee5becdef0cc5c18 in_seq=0 out_seq=0 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.429+0000 7fb33affd640 10 client.16548 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.430+0000 7fb33affd640 1 -- 172.21.15.116:0/2448766392 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9576c9a40 con 0x55f957772340 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.430+0000 7fb33affd640 10 client.16548 handle_command_reply: tid=0 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.430+0000 7fb364403740 1 client.16548 shutdown 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 msgr2=0x55f95775f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 0x55f95775f8a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0edd70 tx=0x7fb34c048000 comp rx=0 tx=0).stop 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 shutdown_connections 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957772340 0x55f9577747a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957761f30 0x55f95775f8a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95775f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb33affd640 0 client.16548 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 --2- 172.21.15.116:0/2448766392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95775ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 >> 172.21.15.116:0/2448766392 conn(0x55f957753160 msgr2=0x55f957754eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 shutdown_connections 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 1 -- 172.21.15.116:0/2448766392 wait complete. 2024-03-19T02:30:24.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.431+0000 7fb364403740 20 client.16548 trim_cache size 0 max 16384 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.449+0000 7fb364403740 1 Processor -- start 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.449+0000 7fb364403740 1 -- start start 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95776daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95777a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1aea90 con 0x55f957753160 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1b0600 con 0x55f957759350 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb364403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1af160 con 0x55f957758a30 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42106/0 (socket says 172.21.15.116:42106) 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 -- 172.21.15.116:0/683526395 learned_addr learned my addr 172.21.15.116:0/683526395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95776daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95777a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 -- 172.21.15.116:0/683526395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 msgr2=0x55f95776daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95776daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 -- 172.21.15.116:0/683526395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 msgr2=0x55f95777a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95777a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 -- 172.21.15.116:0/683526395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9573900f0 con 0x55f957753160 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb35affd640 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb354010010 tx=0x7fb354005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddb7ef3136ac42c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb3517fa640 1 -- 172.21.15.116:0/683526395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35402cb10 con 0x55f957753160 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb3517fa640 1 -- 172.21.15.116:0/683526395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb35401c070 con 0x55f957753160 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.450+0000 7fb3517fa640 1 -- 172.21.15.116:0/683526395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354003bd0 con 0x55f957753160 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- 172.21.15.116:0/683526395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 msgr2=0x55f957761550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb354010010 tx=0x7fb354005ad0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- 172.21.15.116:0/683526395 shutdown_connections 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f95777a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957758a30 0x55f95776daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 --2- 172.21.15.116:0/683526395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957753160 0x55f957761550 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- 172.21.15.116:0/683526395 >> 172.21.15.116:0/683526395 conn(0x55f95775da60 msgr2=0x55f95775f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- 172.21.15.116:0/683526395 shutdown_connections 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- 172.21.15.116:0/683526395 wait complete. 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.451+0000 7fb364403740 1 -- start start 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 Processor -- start 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 0x55f95776a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f957766ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1aff00 con 0x55f957758a30 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1af830 con 0x55f957759350 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1aca70 con 0x55f957753160 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb364403740 10 client.0 _ll_register_callbacks cb 0x7fb34c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 0x55f95776a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42118/0 (socket says 172.21.15.116:42118) 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 -- 172.21.15.116:0/359014661 learned_addr learned my addr 172.21.15.116:0/359014661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35b7fe640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f957766ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 -- 172.21.15.116:0/359014661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 msgr2=0x55f95776a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 0x55f95776a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 -- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 msgr2=0x55f957766ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f957766ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35a7fc640 1 -- 172.21.15.116:0/359014661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9577670c0 con 0x55f957758a30 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.452+0000 7fb35affd640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 0x55f95776a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb35a7fc640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55f95775e680 tx=0x7fb34401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465887ea23255665 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34402f060 con 0x55f957758a30 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb344010b90 con 0x55f957758a30 2024-03-19T02:30:24.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb35401c070 con 0x55f957758a30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb344036050 con 0x55f957758a30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 10 client.16554 fetch_fsmap learned FSMap version 29 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f9577d9530 con 0x55f957758a30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb35401c070 con 0x55f957758a30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb344029070 con 0x55f957758a30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 10 client.16554 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 10 client.16554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 20 client.16554 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957761f30 0x55f9577dbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 4 client.16554 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.453+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f9576c3d90 con 0x55f957761f30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb35affd640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957761f30 0x55f9577dbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb35affd640 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957761f30 0x55f9577dbd80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e967249a3f8f2410 server_cookie=ca87cea30a02845a in_seq=0 out_seq=0 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb33affd640 10 client.16554 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb33affd640 1 -- 172.21.15.116:0/359014661 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f9576c3d90 con 0x55f957761f30 2024-03-19T02:30:24.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb33affd640 10 client.16554 handle_command_reply: tid=0 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 client.16554 shutdown 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 msgr2=0x55f957766590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55f95775e680 tx=0x7fb34401b000 comp rx=0 tx=0).stop 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 shutdown_connections 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f957761f30 0x55f9577dbd80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f957759350 0x55f957766ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f957758a30 0x55f957766590 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 --2- 172.21.15.116:0/359014661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f957753160 0x55f95776a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 >> 172.21.15.116:0/359014661 conn(0x55f95775da60 msgr2=0x55f957751950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb33affd640 0 client.16554 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 shutdown_connections 2024-03-19T02:30:24.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.454+0000 7fb364403740 1 -- 172.21.15.116:0/359014661 wait complete. 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb364403740 20 client.16554 trim_cache size 0 max 16384 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stdout: "68028166-92f6-4387-b709-ced552bced39": { 2024-03-19T02:30:24.462 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:30:24.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "68028166-92f6-4387-b709-ced552bced39", 2024-03-19T02:30:24.463 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:30:24.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:30:24.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:30:24.463 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 msgr2=0x7fb3480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 0x7fb3480a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb344005e10 tx=0x7fb34400ab10 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 msgr2=0x7fb35c19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb34c02c0b0 tx=0x7fb34c004040 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 shutdown_connections 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb3480a2970 0x7fb3480a4e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c107870 0x7fb35c19c780 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb35c0693e0 0x7fb35c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 --2- 172.21.15.116:0/3286196490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb35c068a20 0x7fb35c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 >> 172.21.15.116:0/3286196490 conn(0x7fb35c077820 msgr2=0x7fb35c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:24.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 shutdown_connections 2024-03-19T02:30:24.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:24.455+0000 7fb361812640 1 -- 172.21.15.116:0/3286196490 wait complete. 2024-03-19T02:30:24.467 INFO:tasks.cephfs.filesystem:scrub status for tag:68028166-92f6-4387-b709-ced552bced39 - {'path': '/', 'tag': '68028166-92f6-4387-b709-ced552bced39', 'options': 'recursive,force'} 2024-03-19T02:30:24.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 65682 3.46 MB/sec warmup 107 sec latency 16.788 ms 2024-03-19T02:30:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:24 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:30:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:24 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:30:25.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:24 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:30:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:24 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:30:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:24 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:30:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:24 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:30:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:25.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 66303 3.45 MB/sec warmup 108 sec latency 21.826 ms 2024-03-19T02:30:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:25 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:30:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:25 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:25 smithi162 ceph-mon[35761]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:30:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:25 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:30:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:25 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:25 smithi154 ceph-mon[35693]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:25 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:25 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:25 smithi116 ceph-mon[32263]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:30:26.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 67160 3.48 MB/sec warmup 109 sec latency 19.575 ms 2024-03-19T02:30:27.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 67657 3.49 MB/sec warmup 110 sec latency 17.091 ms 2024-03-19T02:30:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:27 smithi162 ceph-mon[35761]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:27 smithi154 ceph-mon[35693]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:27 smithi116 ceph-mon[32263]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:28.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 68087 3.46 MB/sec warmup 111 sec latency 26.584 ms 2024-03-19T02:30:29.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 68804 3.48 MB/sec warmup 112 sec latency 15.428 ms 2024-03-19T02:30:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:29 smithi162 ceph-mon[35761]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:29 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:30:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:29 smithi154 ceph-mon[35693]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:29 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:30:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:29 smithi116 ceph-mon[32263]: pgmap v83: 129 pgs: 129 active+clean; 64 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:30:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:29 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:30:30.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 69446 3.45 MB/sec warmup 113 sec latency 16.108 ms 2024-03-19T02:30:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:30:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:30:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:30:31.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 70023 3.44 MB/sec warmup 114 sec latency 16.820 ms 2024-03-19T02:30:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:31 smithi162 ceph-mon[35761]: pgmap v84: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:30:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:31 smithi154 ceph-mon[35693]: pgmap v84: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:30:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:31 smithi116 ceph-mon[32263]: pgmap v84: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:30:32.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 70741 3.47 MB/sec warmup 115 sec latency 16.128 ms 2024-03-19T02:30:33.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 71274 3.48 MB/sec warmup 116 sec latency 20.325 ms 2024-03-19T02:30:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:33 smithi162 ceph-mon[35761]: pgmap v85: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:30:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:33 smithi154 ceph-mon[35693]: pgmap v85: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:30:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:33 smithi116 ceph-mon[32263]: pgmap v85: 129 pgs: 129 active+clean; 66 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:30:34.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 71689 3.45 MB/sec warmup 117 sec latency 25.427 ms 2024-03-19T02:30:35.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 72383 3.46 MB/sec warmup 118 sec latency 16.751 ms 2024-03-19T02:30:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:35 smithi162 ceph-mon[35761]: pgmap v86: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-03-19T02:30:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:35 smithi154 ceph-mon[35693]: pgmap v86: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-03-19T02:30:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:35 smithi116 ceph-mon[32263]: pgmap v86: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-03-19T02:30:36.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 73068 3.44 MB/sec warmup 119 sec latency 18.075 ms 2024-03-19T02:30:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:37 smithi162 ceph-mon[35761]: pgmap v87: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:37 smithi154 ceph-mon[35693]: pgmap v87: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:37 smithi116 ceph-mon[32263]: pgmap v87: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:38.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 74330 3.42 MB/sec execute 1 sec latency 23.547 ms 2024-03-19T02:30:39.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 74910 4.05 MB/sec execute 2 sec latency 16.685 ms 2024-03-19T02:30:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:39 smithi162 ceph-mon[35761]: pgmap v88: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:39 smithi154 ceph-mon[35693]: pgmap v88: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:39 smithi116 ceph-mon[32263]: pgmap v88: 129 pgs: 129 active+clean; 67 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:30:40.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 75367 2.88 MB/sec execute 3 sec latency 48.485 ms 2024-03-19T02:30:41.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 76036 3.30 MB/sec execute 4 sec latency 15.922 ms 2024-03-19T02:30:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:41 smithi162 ceph-mon[35761]: pgmap v89: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:30:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:41 smithi154 ceph-mon[35693]: pgmap v89: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:30:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:41 smithi116 ceph-mon[32263]: pgmap v89: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:30:42.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 76738 2.78 MB/sec execute 5 sec latency 28.157 ms 2024-03-19T02:30:43.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 77515 3.45 MB/sec execute 6 sec latency 18.910 ms 2024-03-19T02:30:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:43 smithi162 ceph-mon[35761]: pgmap v90: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:30:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:43 smithi154 ceph-mon[35693]: pgmap v90: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:30:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:43 smithi116 ceph-mon[32263]: pgmap v90: 129 pgs: 129 active+clean; 69 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:30:44.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 77928 3.24 MB/sec execute 7 sec latency 18.129 ms 2024-03-19T02:30:45.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 78507 3.40 MB/sec execute 8 sec latency 16.970 ms 2024-03-19T02:30:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:45 smithi162 ceph-mon[35761]: pgmap v91: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-03-19T02:30:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:45 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:30:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:45 smithi154 ceph-mon[35693]: pgmap v91: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-03-19T02:30:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:45 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:30:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:45 smithi116 ceph-mon[32263]: pgmap v91: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.1 MiB/s wr, 123 op/s 2024-03-19T02:30:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:45 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:30:46.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 79024 3.12 MB/sec execute 9 sec latency 44.139 ms 2024-03-19T02:30:47.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 79695 3.24 MB/sec execute 10 sec latency 15.876 ms 2024-03-19T02:30:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:47 smithi162 ceph-mon[35761]: pgmap v92: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:47 smithi154 ceph-mon[35693]: pgmap v92: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:47 smithi116 ceph-mon[32263]: pgmap v92: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:48.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 80391 3.10 MB/sec execute 11 sec latency 53.615 ms 2024-03-19T02:30:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 81085 3.32 MB/sec execute 12 sec latency 21.592 ms 2024-03-19T02:30:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:49 smithi162 ceph-mon[35761]: pgmap v93: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:49 smithi154 ceph-mon[35693]: pgmap v93: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:49 smithi116 ceph-mon[32263]: pgmap v93: 129 pgs: 129 active+clean; 71 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:50.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 81652 3.44 MB/sec execute 13 sec latency 17.527 ms 2024-03-19T02:30:51.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 82121 3.31 MB/sec execute 14 sec latency 16.213 ms 2024-03-19T02:30:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:51 smithi162 ceph-mon[35761]: pgmap v94: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-03-19T02:30:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:51 smithi154 ceph-mon[35693]: pgmap v94: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-03-19T02:30:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:51 smithi116 ceph-mon[32263]: pgmap v94: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-03-19T02:30:52.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 82639 3.15 MB/sec execute 15 sec latency 36.203 ms 2024-03-19T02:30:53.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 83285 3.23 MB/sec execute 16 sec latency 18.278 ms 2024-03-19T02:30:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:53 smithi162 ceph-mon[35761]: pgmap v95: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:53 smithi154 ceph-mon[35693]: pgmap v95: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:53 smithi116 ceph-mon[32263]: pgmap v95: 129 pgs: 129 active+clean; 72 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:30:54.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:30:54.468 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:30:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 -- 172.21.15.116:0/2639695829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af8102170 msgr2=0x7f4af810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 --2- 172.21.15.116:0/2639695829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af8102170 0x7f4af810e680 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009cd0 tx=0x7f4af002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 -- 172.21.15.116:0/2639695829 shutdown_connections 2024-03-19T02:30:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 --2- 172.21.15.116:0/2639695829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af8102170 0x7f4af810e680 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 --2- 172.21.15.116:0/2639695829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af81017b0 0x7f4af8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 --2- 172.21.15.116:0/2639695829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81077c0 0x7f4af8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 -- 172.21.15.116:0/2639695829 >> 172.21.15.116:0/2639695829 conn(0x7f4af8078c20 msgr2=0x7f4af8079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 -- 172.21.15.116:0/2639695829 shutdown_connections 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.580+0000 7f4b00387640 1 -- 172.21.15.116:0/2639695829 wait complete. 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 Processor -- start 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 -- start start 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 0x7f4af819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8111a80 con 0x7f4af81077c0 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8111900 con 0x7f4af8102170 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4b00387640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8111c00 con 0x7f4af81017b0 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afe123640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55720/0 (socket says 172.21.15.116:55720) 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afe123640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42876/0 (socket says 172.21.15.116:42876) 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afe924640 1 -- 172.21.15.116:0/2746084367 learned_addr learned my addr 172.21.15.116:0/2746084367 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.581+0000 7f4afd922640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 0x7f4af819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 -- 172.21.15.116:0/2746084367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 msgr2=0x7f4af819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 -- 172.21.15.116:0/2746084367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 msgr2=0x7f4af819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 0x7f4af819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 -- 172.21.15.116:0/2746084367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af819f210 con 0x7f4af81077c0 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe123640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afd922640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 0x7f4af819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4afe924640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4af0004af0 tx=0x7f4af0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4aef7fe640 1 -- 172.21.15.116:0/2746084367 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0039040 con 0x7f4af81077c0 2024-03-19T02:30:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4aef7fe640 1 -- 172.21.15.116:0/2746084367 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af0044070 con 0x7f4af81077c0 2024-03-19T02:30:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af819f4a0 con 0x7f4af81077c0 2024-03-19T02:30:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4aef7fe640 1 -- 172.21.15.116:0/2746084367 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00042a0 con 0x7f4af81077c0 2024-03-19T02:30:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.582+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af81a6d90 con 0x7f4af81077c0 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.584+0000 7f4aef7fe640 1 -- 172.21.15.116:0/2746084367 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4af0003970 con 0x7f4af81077c0 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 Processor -- start 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 -- start start 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4aef7fe640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 0x7f4ae40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308fe49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560308f3f500 con 0x560308fe4f30 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560308ee7be0 con 0x560308fdc350 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4b02f75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560308f3b5a0 con 0x560308fdba30 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4afd922640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 0x7f4ae40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.585+0000 7f4afe123640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4aef7fe640 1 -- 172.21.15.116:0/2746084367 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4af00f1470 con 0x7f4af81077c0 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42884/0 (socket says 172.21.15.116:42884) 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 -- 172.21.15.116:0/695323296 learned_addr learned my addr 172.21.15.116:0/695323296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe924640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afd922640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308fe49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 -- 172.21.15.116:0/695323296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 msgr2=0x560308fe49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308fe49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 -- 172.21.15.116:0/695323296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 msgr2=0x560308fe7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 -- 172.21.15.116:0/695323296 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560308f09000 con 0x560308fdba30 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe924640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afd922640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 0x7f4ae40a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009fd0 tx=0x7f4ae8009ee0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4afe123640 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af40114c0 tx=0x7f4af40118c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a79d8c081248c8c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4aecbfe640 1 -- 172.21.15.116:0/695323296 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4004020 con 0x560308fdba30 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4aecbfe640 1 -- 172.21.15.116:0/695323296 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af40041c0 con 0x560308fdba30 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.586+0000 7f4aecbfe640 1 -- 172.21.15.116:0/695323296 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af400f780 con 0x560308fdba30 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- 172.21.15.116:0/695323296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 msgr2=0x560308fdbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af40114c0 tx=0x7f4af40118c0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- 172.21.15.116:0/695323296 shutdown_connections 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308fe49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 --2- 172.21.15.116:0/695323296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fdbe10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- 172.21.15.116:0/695323296 >> 172.21.15.116:0/695323296 conn(0x560308fd6160 msgr2=0x560308fd6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- 172.21.15.116:0/695323296 shutdown_connections 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- 172.21.15.116:0/695323296 wait complete. 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.587+0000 7f4b02f75740 1 -- start start 2024-03-19T02:30:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 Processor -- start 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fe2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fe4f30 0x560308fe28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560308fe7d20 con 0x560308fdba30 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560308efb160 con 0x560308fe4f30 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560308fe7ea0 con 0x560308fdc350 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4b02f75740 10 client.0 _ll_register_callbacks cb 0x560308f37a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afd922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fe2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55730/0 (socket says 172.21.15.116:55730) 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 -- 172.21.15.116:0/603278141 learned_addr learned my addr 172.21.15.116:0/603278141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe924640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fe4f30 0x560308fe28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 -- 172.21.15.116:0/603278141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 msgr2=0x560308fe2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fe2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 -- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fe4f30 msgr2=0x560308fe28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fe4f30 0x560308fe28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 -- 172.21.15.116:0/603278141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560308d1d7d0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4afe123640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005f10 tx=0x7f4af4009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02578b4f79bf322 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.588+0000 7f4ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4027020 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4af40035f0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4af40041c0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4af4032050 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 10 client.16566 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560308c533f0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af40041c0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4af4004ca0 con 0x560308fdba30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 10 client.16566 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 20 client.16566 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560308ff51d0 0x560308ff75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 4 client.16566 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560308f4ca40 con 0x560308ff51d0 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.589+0000 7f4afd922640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560308ff51d0 0x560308ff75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.590+0000 7f4ae1ffb640 10 client.16566 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.590+0000 7f4afd922640 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560308ff51d0 0x560308ff75c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2b707f311fd0ddf server_cookie=3f17870cc6e974c6 in_seq=0 out_seq=0 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/603278141 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560308f4ca40 con 0x560308ff51d0 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4ae1ffb640 10 client.16566 handle_command_reply: tid=0 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 client.16566 shutdown 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 msgr2=0x560308fe1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005f10 tx=0x7f4af4009d00 comp rx=0 tx=0).stop 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 shutdown_connections 2024-03-19T02:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560308ff51d0 0x560308ff75c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fe4f30 0x560308fe28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fe2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4ae1ffb640 0 client.16566 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 --2- 172.21.15.116:0/603278141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fdba30 0x560308fe1e50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 >> 172.21.15.116:0/603278141 conn(0x560308fd6160 msgr2=0x560308fd7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 shutdown_connections 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 1 -- 172.21.15.116:0/603278141 wait complete. 2024-03-19T02:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.591+0000 7f4b02f75740 20 client.16566 trim_cache size 0 max 16384 2024-03-19T02:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 Processor -- start 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 -- start start 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fe4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308ff0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81b0ce0 con 0x560308fe4f30 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81b2820 con 0x560308fdc350 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4b02f75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81b13b0 con 0x560308fdba30 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55744/0 (socket says 172.21.15.116:55744) 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 -- 172.21.15.116:0/602705241 learned_addr learned my addr 172.21.15.116:0/602705241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afd922640 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308ff0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe123640 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fe4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 -- 172.21.15.116:0/602705241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 msgr2=0x560308fe4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fe4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 -- 172.21.15.116:0/602705241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 msgr2=0x560308ff0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308ff0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.612+0000 7f4afe924640 1 -- 172.21.15.116:0/602705241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560308c130f0 con 0x560308fe4f30 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4afe924640 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4af0006010 tx=0x7f4af00323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7e7d6320bd4af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4ae3fff640 1 -- 172.21.15.116:0/602705241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00bd3a0 con 0x560308fe4f30 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4ae3fff640 1 -- 172.21.15.116:0/602705241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af0034700 con 0x560308fe4f30 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4ae3fff640 1 -- 172.21.15.116:0/602705241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00a1540 con 0x560308fe4f30 2024-03-19T02:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4b02f75740 1 -- 172.21.15.116:0/602705241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 msgr2=0x560308ffce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4b02f75740 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4af0006010 tx=0x7f4af00323c0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.613+0000 7f4b02f75740 1 -- 172.21.15.116:0/602705241 shutdown_connections 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308ffce40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdc350 0x560308ff0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- 172.21.15.116:0/602705241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdba30 0x560308fe4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- 172.21.15.116:0/602705241 >> 172.21.15.116:0/602705241 conn(0x560308fd6160 msgr2=0x560308fe26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- 172.21.15.116:0/602705241 shutdown_connections 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- 172.21.15.116:0/602705241 wait complete. 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- start start 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 Processor -- start 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81b2150 con 0x560308fe4f30 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81b1a80 con 0x560308fdba30 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81aecc0 con 0x560308fdc350 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4b02f75740 10 client.0 _ll_register_callbacks cb 0x7f4ae8011880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afe123640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afd922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afe924640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55746/0 (socket says 172.21.15.116:55746) 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afd922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42924/0 (socket says 172.21.15.116:42924) 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afe924640 1 -- 172.21.15.116:0/78197493 learned_addr learned my addr 172.21.15.116:0/78197493 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.614+0000 7f4afe123640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33842/0 (socket says 172.21.15.116:33842) 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 -- 172.21.15.116:0/78197493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 msgr2=0x560308fed380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 -- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 msgr2=0x560308fece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 -- 172.21.15.116:0/78197493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560308fe8a20 con 0x560308fe4f30 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afd922640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe123640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4afe924640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4af00c5ad0 tx=0x7f4af00bd3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff71c5610f198ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00a0020 con 0x560308fe4f30 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4af00b9ed0 con 0x560308fe4f30 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560308f46d90 con 0x560308fe4f30 2024-03-19T02:30:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4af00bd1c0 con 0x560308fe4f30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 10 client.16572 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56030905af90 con 0x560308fe4f30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560308f46d90 con 0x560308fe4f30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4af003c030 con 0x560308fe4f30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 10 client.16572 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 10 client.16572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 20 client.16572 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56030905b540 0x56030905d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 4 client.16572 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.615+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560308ec4540 con 0x56030905b540 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4afe123640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56030905b540 0x56030905d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4afe123640 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56030905b540 0x56030905d930 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89e36726b986dd9c server_cookie=45d10fe6299298c in_seq=0 out_seq=0 2024-03-19T02:30:54.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4ae1ffb640 10 client.16572 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4ae1ffb640 1 -- 172.21.15.116:0/78197493 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560308ec4540 con 0x56030905b540 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4ae1ffb640 10 client.16572 handle_command_reply: tid=0 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.616+0000 7f4b02f75740 1 client.16572 shutdown 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 msgr2=0x560308fe8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4af00c5ad0 tx=0x7f4af00bd3a0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 shutdown_connections 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56030905b540 0x56030905d930 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560308fe4f30 0x560308fe8220 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560308fdc350 0x560308fed380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4ae1ffb640 0 client.16572 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 --2- 172.21.15.116:0/78197493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560308fdba30 0x560308fece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 >> 172.21.15.116:0/78197493 conn(0x560308fd6160 msgr2=0x560308fe1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 shutdown_connections 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 1 -- 172.21.15.116:0/78197493 wait complete. 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b02f75740 20 client.16572 trim_cache size 0 max 16384 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:30:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:30:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:54.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 msgr2=0x7f4ae40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 0x7f4ae40a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009fd0 tx=0x7f4ae8009ee0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 msgr2=0x7f4af819eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4af0004af0 tx=0x7f4af0035ce0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 shutdown_connections 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4ae40a29f0 0x7f4ae40a4eb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4af81077c0 0x7f4af819eae0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4af8102170 0x7f4af819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 --2- 172.21.15.116:0/2746084367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4af81017b0 0x7f4af819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 >> 172.21.15.116:0/2746084367 conn(0x7f4af8078c20 msgr2=0x7f4af810cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 shutdown_connections 2024-03-19T02:30:54.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.617+0000 7f4b00387640 1 -- 172.21.15.116:0/2746084367 wait complete. 2024-03-19T02:30:54.631 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:30:54.631 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:30:54.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- 172.21.15.116:0/4227436853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c106730 msgr2=0x7f482c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 --2- 172.21.15.116:0/4227436853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c106730 0x7f482c106b10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4828009930 tx=0x7f482802bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- 172.21.15.116:0/4227436853 shutdown_connections 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 --2- 172.21.15.116:0/4227436853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c1010e0 0x7f482c10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 --2- 172.21.15.116:0/4227436853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 --2- 172.21.15.116:0/4227436853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c106730 0x7f482c106b10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- 172.21.15.116:0/4227436853 >> 172.21.15.116:0/4227436853 conn(0x7f482c077820 msgr2=0x7f482c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- 172.21.15.116:0/4227436853 shutdown_connections 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- 172.21.15.116:0/4227436853 wait complete. 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 Processor -- start 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.740+0000 7f48344ea640 1 -- start start 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 0x7f482c1a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f482c1109f0 con 0x7f482c1010e0 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f482c110870 con 0x7f482c106730 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48344ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f482c110b70 con 0x7f482c100720 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55758/0 (socket says 172.21.15.116:55758) 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 -- 172.21.15.116:0/602448983 learned_addr learned my addr 172.21.15.116:0/602448983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4832286640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4832a87640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 0x7f482c1a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 -- 172.21.15.116:0/602448983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 msgr2=0x7f482c198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 -- 172.21.15.116:0/602448983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 msgr2=0x7f482c1a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 0x7f482c1a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 -- 172.21.15.116:0/602448983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482c1a4470 con 0x7f482c1010e0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4832286640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4832a87640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 0x7f482c1a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f4831a85640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f481c00e9f0 tx=0x7f481c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.741+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c00cd80 con 0x7f482c1010e0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.742+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f481c0054d0 con 0x7f482c1010e0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.742+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482c1a4640 con 0x7f482c1010e0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.742+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c010490 con 0x7f482c1010e0 2024-03-19T02:30:54.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.742+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f482c1a4b10 con 0x7f482c1010e0 2024-03-19T02:30:54.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.743+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f482c101e60 con 0x7f482c1010e0 2024-03-19T02:30:54.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.744+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f481c005020 con 0x7f482c1010e0 2024-03-19T02:30:54.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.745+0000 7f48237fe640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 0x7f48180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:54.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.745+0000 7f4832286640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 0x7f48180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:54.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.745+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f481c0cf550 con 0x7f482c1010e0 2024-03-19T02:30:54.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.745+0000 7f4832286640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 0x7f48180a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4828005ab0 tx=0x7f4828005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:54.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:54.750+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f481c09bf30 con 0x7f482c1010e0 2024-03-19T02:30:54.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 83983 3.14 MB/sec execute 17 sec latency 20.814 ms 2024-03-19T02:30:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.008+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f482c068f70 con 0x7f482c1010e0 2024-03-19T02:30:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.009+0000 7f48237fe640 1 -- 172.21.15.116:0/602448983 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f481c09b830 con 0x7f482c1010e0 2024-03-19T02:30:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:30:55.010 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:30:55.010 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:30:55.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 msgr2=0x7f48180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 0x7f48180a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4828005ab0 tx=0x7f4828005fd0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 msgr2=0x7f482c1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f481c00e9f0 tx=0x7f481c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 shutdown_connections 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48180a2970 0x7f48180a4e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482c106730 0x7f482c1a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f482c1010e0 0x7f482c1993f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 --2- 172.21.15.116:0/602448983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f482c100720 0x7f482c198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 >> 172.21.15.116:0/602448983 conn(0x7f482c077820 msgr2=0x7f482c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 shutdown_connections 2024-03-19T02:30:55.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.011+0000 7f48344ea640 1 -- 172.21.15.116:0/602448983 wait complete. 2024-03-19T02:30:55.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:30:55.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- 172.21.15.116:0/588243429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2798100e00 msgr2=0x7f2798101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- 172.21.15.116:0/588243429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2798100e00 0x7f2798101280 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2784009930 tx=0x7f278402bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- 172.21.15.116:0/588243429 shutdown_connections 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- 172.21.15.116:0/588243429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 0x7f279810d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- 172.21.15.116:0/588243429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2798100e00 0x7f2798101280 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- 172.21.15.116:0/588243429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f27981068f0 0x7f2798106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- 172.21.15.116:0/588243429 >> 172.21.15.116:0/588243429 conn(0x7f2798076c30 msgr2=0x7f27980fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- 172.21.15.116:0/588243429 shutdown_connections 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- 172.21.15.116:0/588243429 wait complete. 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 Processor -- start 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- start start 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 0x7f2798198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 0x7f27981987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2798110b70 con 0x7f27981068f0 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27981109f0 con 0x7f27981017c0 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.136+0000 7f279f236640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2798110cf0 con 0x7f2798100e00 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 0x7f2798198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 0x7f2798198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42964/0 (socket says 172.21.15.116:42964) 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55774/0 (socket says 172.21.15.116:55774) 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279cfd2640 1 -- 172.21.15.116:0/306237475 learned_addr learned my addr 172.21.15.116:0/306237475 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f2797fff640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 0x7f27981987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 -- 172.21.15.116:0/306237475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 msgr2=0x7f2798198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 0x7f2798198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 -- 172.21.15.116:0/306237475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 msgr2=0x7f27981987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 0x7f27981987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 -- 172.21.15.116:0/306237475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279819d250 con 0x7f27981068f0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279d7d3640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2790009fd0 tx=0x7f279000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f2795ffb640 1 -- 172.21.15.116:0/306237475 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279000ce20 con 0x7f27981068f0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f2795ffb640 1 -- 172.21.15.116:0/306237475 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2790005020 con 0x7f27981068f0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279819d540 con 0x7f27981068f0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f2795ffb640 1 -- 172.21.15.116:0/306237475 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790010690 con 0x7f27981068f0 2024-03-19T02:30:55.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.137+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981a4dd0 con 0x7f27981068f0 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.139+0000 7f2795ffb640 1 -- 172.21.15.116:0/306237475 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f27900051d0 con 0x7f27981068f0 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f2795ffb640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 0x7f27880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f2795ffb640 1 -- 172.21.15.116:0/306237475 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f27900cf8e0 con 0x7f27981068f0 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f279cfd2640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 0x7f27880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 Processor -- start 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 -- start start 2024-03-19T02:30:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f279cfd2640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 0x7f27880a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f278c0095f0 tx=0x7f278c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d120e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d1210bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7d11684b0 con 0x55f7d120e7d0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.140+0000 7f27a1e29740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f7d1110d70 con 0x55f7d1205bf0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f27a1e29740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f7d1164550 con 0x55f7d12052d0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279d7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d1210bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42978/0 (socket says 172.21.15.116:42978) 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f2797fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d120e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 -- 172.21.15.116:0/3331990376 learned_addr learned my addr 172.21.15.116:0/3331990376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 -- 172.21.15.116:0/3331990376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 msgr2=0x55f7d120e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d120e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 -- 172.21.15.116:0/3331990376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 msgr2=0x55f7d1210bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d1210bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 -- 172.21.15.116:0/3331990376 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d1133210 con 0x55f7d12052d0 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f279cfd2640 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f278c009f70 tx=0x7f278c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdcaaf3b944631cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f2782ffd640 1 -- 172.21.15.116:0/3331990376 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c02b070 con 0x55f7d12052d0 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f2782ffd640 1 -- 172.21.15.116:0/3331990376 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f278c026ad0 con 0x55f7d12052d0 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.141+0000 7f2782ffd640 1 -- 172.21.15.116:0/3331990376 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c01a9b0 con 0x55f7d12052d0 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- 172.21.15.116:0/3331990376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 msgr2=0x55f7d12056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f278c009f70 tx=0x7f278c014000 comp rx=0 tx=0).stop 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- 172.21.15.116:0/3331990376 shutdown_connections 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d1210bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d120e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 --2- 172.21.15.116:0/3331990376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d12056b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- 172.21.15.116:0/3331990376 >> 172.21.15.116:0/3331990376 conn(0x55f7d11ffa00 msgr2=0x55f7d11ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- 172.21.15.116:0/3331990376 shutdown_connections 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- 172.21.15.116:0/3331990376 wait complete. 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 -- start start 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.142+0000 7f27a1e29740 1 Processor -- start 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d1218ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d120e7d0 0x55f7d1212ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7d12115c0 con 0x55f7d12052d0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f7d1124310 con 0x55f7d1205bf0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f7d1211740 con 0x55f7d120e7d0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f27a1e29740 10 client.0 _ll_register_callbacks cb 0x55f7d11182d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55800/0 (socket says 172.21.15.116:55800) 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f279cfd2640 1 -- 172.21.15.116:0/685031507 learned_addr learned my addr 172.21.15.116:0/685031507 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d1218ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f279d7d3640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d120e7d0 0x55f7d1212ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 -- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d120e7d0 msgr2=0x55f7d1212ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d120e7d0 0x55f7d1212ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 -- 172.21.15.116:0/685031507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 msgr2=0x55f7d1218ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f2797fff640 1 -- 172.21.15.116:0/685031507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d0e7c3f0 con 0x55f7d1205bf0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.143+0000 7f279cfd2640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2797fff640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d1218ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f278402ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f1a34a206b7a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278402c5d0 con 0x55f7d1205bf0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2784005030 con 0x55f7d1205bf0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f278c026ad0 con 0x55f7d1205bf0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2784005230 con 0x55f7d1205bf0 2024-03-19T02:30:55.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 10 client.26408 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f7d0f467d0 con 0x55f7d1205bf0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f278c026ad0 con 0x55f7d1205bf0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f27840035f0 con 0x55f7d1205bf0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 10 client.26408 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 10 client.26408 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 20 client.26408 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d1223020 0x55f7d1225410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 4 client.26408 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.144+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7d11c63e0 con 0x55f7d1223020 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.145+0000 7f279cfd2640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d1223020 0x55f7d1225410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.145+0000 7f279cfd2640 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d1223020 0x55f7d1225410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6224fea8cd03831f server_cookie=29492347dc32b876 in_seq=0 out_seq=0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.145+0000 7f2777fff640 10 client.26408 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f2777fff640 1 -- 172.21.15.116:0/685031507 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7d11c63e0 con 0x55f7d1223020 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f2777fff640 10 client.26408 handle_command_reply: tid=0 2024-03-19T02:30:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 client.26408 shutdown 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 msgr2=0x55f7d1218ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d1218ff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f278402ca10 comp rx=0 tx=0).stop 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 shutdown_connections 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d1223020 0x55f7d1225410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d120e7d0 0x55f7d1212ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f2777fff640 0 client.26408 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d1205bf0 0x55f7d1218ff0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 --2- 172.21.15.116:0/685031507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d12052d0 0x55f7d1218ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 >> 172.21.15.116:0/685031507 conn(0x55f7d11ffa00 msgr2=0x55f7d1201750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 shutdown_connections 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 1 -- 172.21.15.116:0/685031507 wait complete. 2024-03-19T02:30:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.146+0000 7f27a1e29740 20 client.26408 trim_cache size 0 max 16384 2024-03-19T02:30:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 Processor -- start 2024-03-19T02:30:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 -- start start 2024-03-19T02:30:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 0x55f7d1212e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27981aca10 con 0x55f7d120e7d0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27981ae550 con 0x55f7d12052d0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f27a1e29740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27981ad0e0 con 0x55f7d1205bf0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 0x55f7d1212e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f2797fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55804/0 (socket says 172.21.15.116:55804) 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279cfd2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 0x55f7d1212e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33906/0 (socket says 172.21.15.116:33906) 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f2797fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42984/0 (socket says 172.21.15.116:42984) 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 -- 172.21.15.116:0/105814074 learned_addr learned my addr 172.21.15.116:0/105814074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 -- 172.21.15.116:0/105814074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 msgr2=0x55f7d122acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 -- 172.21.15.116:0/105814074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 msgr2=0x55f7d1212e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 0x55f7d1212e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f279d7d3640 1 -- 172.21.15.116:0/105814074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d120b890 con 0x55f7d120e7d0 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.161+0000 7f2797fff640 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f279d7d3640 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f279009bb50 tx=0x7f279009fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fced33b47e24d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27827fc640 1 -- 172.21.15.116:0/105814074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790098230 con 0x55f7d120e7d0 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27827fc640 1 -- 172.21.15.116:0/105814074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27900ba300 con 0x55f7d120e7d0 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27827fc640 1 -- 172.21.15.116:0/105814074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27900a4130 con 0x55f7d120e7d0 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 -- 172.21.15.116:0/105814074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 msgr2=0x55f7d120cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f279009bb50 tx=0x7f279009fdd0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 -- 172.21.15.116:0/105814074 shutdown_connections 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d120e7d0 0x55f7d120cf20 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d1205bf0 0x55f7d122acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 --2- 172.21.15.116:0/105814074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d12052d0 0x55f7d1212e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.162+0000 7f27a1e29740 1 -- 172.21.15.116:0/105814074 >> 172.21.15.116:0/105814074 conn(0x55f7d11ffa00 msgr2=0x55f7d1212750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- 172.21.15.116:0/105814074 shutdown_connections 2024-03-19T02:30:55.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- 172.21.15.116:0/105814074 wait complete. 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- start start 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 Processor -- start 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d121ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d120e7d0 0x55f7d121c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27981ade80 con 0x55f7d1205bf0 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27981ad7b0 con 0x55f7d120e7d0 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27981aa9f0 con 0x55f7d12052d0 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f27a1e29740 10 client.0 _ll_register_callbacks cb 0x7f2784009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f2797fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f2797fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55814/0 (socket says 172.21.15.116:55814) 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f2797fff640 1 -- 172.21.15.116:0/4174508831 learned_addr learned my addr 172.21.15.116:0/4174508831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f279cfd2640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d121ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.163+0000 7f279d7d3640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d120e7d0 0x55f7d121c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 -- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 msgr2=0x55f7d121ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d121ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 -- 172.21.15.116:0/4174508831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d120e7d0 msgr2=0x55f7d121c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d120e7d0 0x55f7d121c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 -- 172.21.15.116:0/4174508831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d121c920 con 0x55f7d1205bf0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f279cfd2640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d121ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2797fff640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x55f7d1214440 tx=0x7f2784003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711f9c6ae02fb436 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2777fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784009520 con 0x55f7d1205bf0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f27840055e0 con 0x55f7d1205bf0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27900ba300 con 0x55f7d1205bf0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2784063050 con 0x55f7d1205bf0 2024-03-19T02:30:55.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f27a1e29740 10 client.16596 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f7d1288db0 con 0x55f7d1205bf0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.164+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27900ba300 con 0x55f7d1205bf0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2784004cc0 con 0x55f7d1205bf0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 10 client.16596 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 10 client.16596 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 20 client.16596 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d12895b0 0x55f7d128b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 4 client.16596 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7d11759f0 con 0x55f7d12895b0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f279cfd2640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d12895b0 0x55f7d128b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f2777fff640 10 client.16596 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f279cfd2640 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d12895b0 0x55f7d128b9d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eb6c5d4d81156486 server_cookie=1b150da79a1347ff in_seq=0 out_seq=0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f2777fff640 1 -- 172.21.15.116:0/4174508831 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7d11759f0 con 0x55f7d12895b0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f2777fff640 10 client.16596 handle_command_reply: tid=0 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.165+0000 7f27a1e29740 1 client.16596 shutdown 2024-03-19T02:30:55.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 msgr2=0x55f7d121f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x55f7d1214440 tx=0x7f2784003820 comp rx=0 tx=0).stop 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 shutdown_connections 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f7d12895b0 0x55f7d128b9d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7d120e7d0 0x55f7d121c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7d1205bf0 0x55f7d121f250 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 --2- 172.21.15.116:0/4174508831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d12052d0 0x55f7d121ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 >> 172.21.15.116:0/4174508831 conn(0x55f7d11ffa00 msgr2=0x55f7d1218600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f2777fff640 0 client.16596 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 shutdown_connections 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 1 -- 172.21.15.116:0/4174508831 wait complete. 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.166+0000 7f27a1e29740 20 client.16596 trim_cache size 0 max 16384 2024-03-19T02:30:55.173 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:55.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 msgr2=0x7f27880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 0x7f27880a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f278c0095f0 tx=0x7f278c005f90 comp rx=0 tx=0).stop 2024-03-19T02:30:55.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 msgr2=0x7f279819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2790009fd0 tx=0x7f279000ef10 comp rx=0 tx=0).stop 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 shutdown_connections 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f27880a2970 0x7f27880a4e30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27981068f0 0x7f279819cb20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27981017c0 0x7f27981987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 --2- 172.21.15.116:0/306237475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2798100e00 0x7f2798198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 >> 172.21.15.116:0/306237475 conn(0x7f2798076c30 msgr2=0x7f27980fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 shutdown_connections 2024-03-19T02:30:55.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.167+0000 7f279f236640 1 -- 172.21.15.116:0/306237475 wait complete. 2024-03-19T02:30:55.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 -- 172.21.15.116:0/3046060827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c1017a0 msgr2=0x7fd58c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 --2- 172.21.15.116:0/3046060827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c1017a0 0x7fd58c101c20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009930 tx=0x7fd57c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 -- 172.21.15.116:0/3046060827 shutdown_connections 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 --2- 172.21.15.116:0/3046060827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c102160 0x7fd58c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 --2- 172.21.15.116:0/3046060827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c1017a0 0x7fd58c101c20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 --2- 172.21.15.116:0/3046060827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1077b0 0x7fd58c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 -- 172.21.15.116:0/3046060827 >> 172.21.15.116:0/3046060827 conn(0x7fd58c0fd2d0 msgr2=0x7fd58c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 -- 172.21.15.116:0/3046060827 shutdown_connections 2024-03-19T02:30:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 -- 172.21.15.116:0/3046060827 wait complete. 2024-03-19T02:30:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.289+0000 7fd593c4b640 1 Processor -- start 2024-03-19T02:30:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 -- start start 2024-03-19T02:30:55.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c1077b0 0x7fd58c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c111a70 con 0x7fd58c102160 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1118f0 con 0x7fd58c1017a0 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd593c4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c111bf0 con 0x7fd58c1077b0 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33930/0 (socket says 172.21.15.116:33930) 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55830/0 (socket says 172.21.15.116:55830) 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5919e7640 1 -- 172.21.15.116:0/594690915 learned_addr learned my addr 172.21.15.116:0/594690915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5921e8640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c1077b0 0x7fd58c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 -- 172.21.15.116:0/594690915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c1077b0 msgr2=0x7fd58c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c1077b0 0x7fd58c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 -- 172.21.15.116:0/594690915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 msgr2=0x7fd58c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 -- 172.21.15.116:0/594690915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c19cfa0 con 0x7fd58c102160 2024-03-19T02:30:55.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5919e7640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.290+0000 7fd5911e6640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009230 tx=0x7fd57c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.291+0000 7fd582ffd640 1 -- 172.21.15.116:0/594690915 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c043020 con 0x7fd58c102160 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.291+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c19d230 con 0x7fd58c102160 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.291+0000 7fd582ffd640 1 -- 172.21.15.116:0/594690915 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd57c02ca70 con 0x7fd58c102160 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.291+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c1a4b20 con 0x7fd58c102160 2024-03-19T02:30:55.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.291+0000 7fd582ffd640 1 -- 172.21.15.116:0/594690915 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0039b0 con 0x7fd58c102160 2024-03-19T02:30:55.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.293+0000 7fd582ffd640 1 -- 172.21.15.116:0/594690915 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd57c046020 con 0x7fd58c102160 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.293+0000 7fd596836740 1 Processor -- start 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 -- start start 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd582ffd640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 0x7fd5780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b45290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5921e8640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 0x7fd5780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 0x557147b47bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd582ffd640 1 -- 172.21.15.116:0/594690915 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd57c0f1000 con 0x7fd58c102160 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b45290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557147a9f4b0 con 0x557147b3cbf0 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557147a47d70 con 0x557147b457d0 2024-03-19T02:30:55.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd596836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557147a9b550 con 0x557147b3c2d0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43032/0 (socket says 172.21.15.116:43032) 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5919e7640 1 -- 172.21.15.116:0/2227991441 learned_addr learned my addr 172.21.15.116:0/2227991441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b45290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55832/0 (socket says 172.21.15.116:55832) 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.294+0000 7fd5921e8640 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 0x557147b47bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 -- 172.21.15.116:0/2227991441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 msgr2=0x557147b47bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 0x557147b47bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 -- 172.21.15.116:0/2227991441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 msgr2=0x557147b45290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b45290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 -- 172.21.15.116:0/2227991441 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557147a6a210 con 0x557147b3c2d0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5921e8640 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 0x557147b47bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5921e8640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 0x7fd5780a4e30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd584005fd0 tx=0x7fd584013b60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd5880138d0 tx=0x7fd588013d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76e7edec0dcc4c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd577fff640 1 -- 172.21.15.116:0/2227991441 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588021070 con 0x557147b3c2d0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd577fff640 1 -- 172.21.15.116:0/2227991441 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd58800ede0 con 0x557147b3c2d0 2024-03-19T02:30:55.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.295+0000 7fd577fff640 1 -- 172.21.15.116:0/2227991441 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58801ce40 con 0x557147b3c2d0 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- 172.21.15.116:0/2227991441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 msgr2=0x557147b3c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd5880138d0 tx=0x7fd588013d90 comp rx=0 tx=0).stop 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- 172.21.15.116:0/2227991441 shutdown_connections 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b457d0 0x557147b47bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b45290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 --2- 172.21.15.116:0/2227991441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3c2d0 0x557147b3c6b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- 172.21.15.116:0/2227991441 >> 172.21.15.116:0/2227991441 conn(0x557147b36a00 msgr2=0x557147b36de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- 172.21.15.116:0/2227991441 shutdown_connections 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- 172.21.15.116:0/2227991441 wait complete. 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 -- start start 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.296+0000 7fd596836740 1 Processor -- start 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557147b485c0 con 0x557147b457d0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557147a5b310 con 0x557147b3c2d0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557147b48740 con 0x557147b3cbf0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd596836740 10 client.0 _ll_register_callbacks cb 0x557147a4f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55842/0 (socket says 172.21.15.116:55842) 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43046/0 (socket says 172.21.15.116:43046) 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 -- 172.21.15.116:0/2819480297 learned_addr learned my addr 172.21.15.116:0/2819480297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33950/0 (socket says 172.21.15.116:33950) 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 -- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 msgr2=0x557147b42d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 -- 172.21.15.116:0/2819480297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 msgr2=0x557147b42820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 -- 172.21.15.116:0/2819480297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571477b33f0 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5911e6640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5921e8640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd58400fd70 tx=0x7fd584004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f68a27aff7d19f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.297+0000 7fd5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584003970 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd584003b10 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd58800ede0 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd584031050 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 10 client.16608 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55714787d7d0 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd58800ede0 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd58401f070 con 0x557147b457d0 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 10 client.16608 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 10 client.16608 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 20 client.16608 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147b55bc0 0x557147b58020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 4 client.16608 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557147afd3e0 con 0x557147b55bc0 2024-03-19T02:30:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.298+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147b55bc0 0x557147b58020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.299+0000 7fd5757fa640 10 client.16608 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.299+0000 7fd5919e7640 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147b55bc0 0x557147b58020 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=696f1579521ad45e server_cookie=98dabde29e6874f4 in_seq=0 out_seq=0 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd5757fa640 1 -- 172.21.15.116:0/2819480297 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557147afd3e0 con 0x557147b55bc0 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd5757fa640 10 client.16608 handle_command_reply: tid=0 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 client.16608 shutdown 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 msgr2=0x557147b432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd58400fd70 tx=0x7fd584004ab0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 shutdown_connections 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147b55bc0 0x557147b58020 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b432a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b42d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd5757fa640 0 client.16608 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 --2- 172.21.15.116:0/2819480297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b42820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 >> 172.21.15.116:0/2819480297 conn(0x557147b36a00 msgr2=0x557147b38750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 shutdown_connections 2024-03-19T02:30:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 1 -- 172.21.15.116:0/2819480297 wait complete. 2024-03-19T02:30:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.300+0000 7fd596836740 20 client.16608 trim_cache size 0 max 16384 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 Processor -- start 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 -- start start 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b51360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b44650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ac9c0 con 0x557147b457d0 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ae500 con 0x557147b3c2d0 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd596836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ad090 con 0x557147b3cbf0 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd5921e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b51360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b44650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd5921e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55846/0 (socket says 172.21.15.116:55846) 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.315+0000 7fd5921e8640 1 -- 172.21.15.116:0/1414673037 learned_addr learned my addr 172.21.15.116:0/1414673037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 -- 172.21.15.116:0/1414673037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 msgr2=0x557147b51360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b51360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 -- 172.21.15.116:0/1414673037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 msgr2=0x557147b5d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 -- 172.21.15.116:0/1414673037 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557147b3ca20 con 0x557147b3cbf0 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5919e7640 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b51360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5921e8640 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5911e6640 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b44650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ee4e0 tx=0x7fd57c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af3aaf22ae980867 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5777fe640 1 -- 172.21.15.116:0/1414673037 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c008cf0 con 0x557147b3cbf0 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5777fe640 1 -- 172.21.15.116:0/1414673037 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd57c0bdaf0 con 0x557147b3cbf0 2024-03-19T02:30:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.316+0000 7fd5777fe640 1 -- 172.21.15.116:0/1414673037 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c008b10 con 0x557147b3cbf0 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- 172.21.15.116:0/1414673037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 msgr2=0x557147b44650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b44650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ee4e0 tx=0x7fd57c000bc0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- 172.21.15.116:0/1414673037 shutdown_connections 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b457d0 0x557147b5d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b3cbf0 0x557147b44650 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 --2- 172.21.15.116:0/1414673037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b51360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- 172.21.15.116:0/1414673037 >> 172.21.15.116:0/1414673037 conn(0x557147b36a00 msgr2=0x557147b43f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- 172.21.15.116:0/1414673037 shutdown_connections 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- 172.21.15.116:0/1414673037 wait complete. 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 -- start start 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.317+0000 7fd596836740 1 Processor -- start 2024-03-19T02:30:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b4d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b457d0 0x557147b3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ade30 con 0x557147b3cbf0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ad760 con 0x557147b3c2d0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1aa9a0 con 0x557147b457d0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 10 client.0 _ll_register_callbacks cb 0x7fd584014dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5919e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b4d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55860/0 (socket says 172.21.15.116:55860) 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 -- 172.21.15.116:0/3508502763 learned_addr learned my addr 172.21.15.116:0/3508502763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5921e8640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b457d0 0x557147b3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 -- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b457d0 msgr2=0x557147b3f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b457d0 0x557147b3f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 -- 172.21.15.116:0/3508502763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 msgr2=0x557147b4d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b4d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 -- 172.21.15.116:0/3508502763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557147b4e050 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5911e6640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ee590 tx=0x7fd57c037350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1644d7a5aa40a07b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0bd3f0 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd57c0c5910 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.318+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd57c0bdaf0 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd57c0c1540 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 10 client.16614 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557147bbb9a0 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd57c0bdaf0 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd57c0bc840 con 0x557147b3cbf0 2024-03-19T02:30:55.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 10 client.16614 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 10 client.16614 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 20 client.16614 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147bbc040 0x557147bbe460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 4 client.16614 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557147aac9f0 con 0x557147bbc040 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.319+0000 7fd5919e7640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147bbc040 0x557147bbe460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd5757fa640 10 client.16614 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd5919e7640 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147bbc040 0x557147bbe460 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=25ddbe116a5c8fb3 server_cookie=ec46bbed7ea09fa6 in_seq=0 out_seq=0 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd5757fa640 1 -- 172.21.15.116:0/3508502763 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557147aac9f0 con 0x557147bbc040 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd5757fa640 10 client.16614 handle_command_reply: tid=0 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 client.16614 shutdown 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 msgr2=0x557147b3eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ee590 tx=0x7fd57c037350 comp rx=0 tx=0).stop 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 shutdown_connections 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557147b457d0 0x557147b3f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557147bbc040 0x557147bbe460 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557147b3cbf0 0x557147b3eb50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 --2- 172.21.15.116:0/3508502763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557147b3c2d0 0x557147b4d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd5757fa640 0 client.16614 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 >> 172.21.15.116:0/3508502763 conn(0x557147b36a00 msgr2=0x557147b35ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 shutdown_connections 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 1 -- 172.21.15.116:0/3508502763 wait complete. 2024-03-19T02:30:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.320+0000 7fd596836740 20 client.16614 trim_cache size 0 max 16384 2024-03-19T02:30:55.328 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 msgr2=0x7fd5780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 0x7fd5780a4e30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd584005fd0 tx=0x7fd584013b60 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 msgr2=0x7fd58c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009230 tx=0x7fd57c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 shutdown_connections 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd5780a2970 0x7fd5780a4e30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd58c1077b0 0x7fd58c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd58c102160 0x7fd58c1984f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 --2- 172.21.15.116:0/594690915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd58c1017a0 0x7fd58c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 >> 172.21.15.116:0/594690915 conn(0x7fd58c0fd2d0 msgr2=0x7fd58c0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 shutdown_connections 2024-03-19T02:30:55.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.321+0000 7fd593c4b640 1 -- 172.21.15.116:0/594690915 wait complete. 2024-03-19T02:30:55.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:30:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 -- 172.21.15.116:0/2082526894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 msgr2=0x7ff8341009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 --2- 172.21.15.116:0/2082526894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff8341009a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff824009900 tx=0x7ff82402bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 -- 172.21.15.116:0/2082526894 shutdown_connections 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 --2- 172.21.15.116:0/2082526894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff83410e170 0x7ff834110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 --2- 172.21.15.116:0/2082526894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff834100ee0 0x7ff83410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 --2- 172.21.15.116:0/2082526894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff8341009a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 -- 172.21.15.116:0/2082526894 >> 172.21.15.116:0/2082526894 conn(0x7ff8340778d0 msgr2=0x7ff834077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 -- 172.21.15.116:0/2082526894 shutdown_connections 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.446+0000 7ff8394c0640 1 -- 172.21.15.116:0/2082526894 wait complete. 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 Processor -- start 2024-03-19T02:30:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 -- start start 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834100ee0 0x7ff83419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83410e170 0x7ff83419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff834113c50 con 0x7ff8341005c0 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff834113ad0 con 0x7ff83410e170 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8394c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff834113dd0 con 0x7ff834100ee0 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff832ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff832ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55880/0 (socket says 172.21.15.116:55880) 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff832ffd640 1 -- 172.21.15.116:0/305409332 learned_addr learned my addr 172.21.15.116:0/305409332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8327fc640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834100ee0 0x7ff83419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.447+0000 7ff8337fe640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83410e170 0x7ff83419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 -- 172.21.15.116:0/305409332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834100ee0 msgr2=0x7ff83419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834100ee0 0x7ff83419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 -- 172.21.15.116:0/305409332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83410e170 msgr2=0x7ff83419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83410e170 0x7ff83419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 -- 172.21.15.116:0/305409332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83419f150 con 0x7ff8341005c0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff832ffd640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff824005940 tx=0x7ff82402c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff81bfff640 1 -- 172.21.15.116:0/305409332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824043020 con 0x7ff8341005c0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83419f3e0 con 0x7ff8341005c0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff81bfff640 1 -- 172.21.15.116:0/305409332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8240042d0 con 0x7ff8341005c0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff81bfff640 1 -- 172.21.15.116:0/305409332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824003ab0 con 0x7ff8341005c0 2024-03-19T02:30:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.448+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8341a6cd0 con 0x7ff8341005c0 2024-03-19T02:30:55.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.450+0000 7ff81bfff640 1 -- 172.21.15.116:0/305409332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff824046020 con 0x7ff8341005c0 2024-03-19T02:30:55.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 Processor -- start 2024-03-19T02:30:55.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 -- start start 2024-03-19T02:30:55.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff81bfff640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 0x7ff81c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70c2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559aa701c4b0 con 0x559aa70b9bf0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559aa6fc4d70 con 0x559aa70c27d0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff83c185740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559aa7018550 con 0x559aa70b92d0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff832ffd640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 0x7ff81c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43094/0 (socket says 172.21.15.116:43094) 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8337fe640 1 -- 172.21.15.116:0/2792789453 learned_addr learned my addr 172.21.15.116:0/2792789453 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8337fe640 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff81bfff640 1 -- 172.21.15.116:0/305409332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff8240f10f0 con 0x7ff8341005c0 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 -- 172.21.15.116:0/2792789453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 msgr2=0x559aa70b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 -- 172.21.15.116:0/2792789453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 msgr2=0x559aa70c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.451+0000 7ff8327fc640 1 -- 172.21.15.116:0/2792789453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aa6fe7210 con 0x559aa70b9bf0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff832ffd640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 0x7ff81c0a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8240bd470 tx=0x7ff82408f110 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff8327fc640 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70c2290 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff828009bf0 tx=0x7ff828011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22b6bc018b31909 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff8197fa640 1 -- 172.21.15.116:0/2792789453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828004050 con 0x559aa70b9bf0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff8197fa640 1 -- 172.21.15.116:0/2792789453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8280054d0 con 0x559aa70b9bf0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff8197fa640 1 -- 172.21.15.116:0/2792789453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828004c80 con 0x559aa70b9bf0 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 -- 172.21.15.116:0/2792789453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 msgr2=0x559aa70c2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70c2290 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff828009bf0 tx=0x7ff828011810 comp rx=0 tx=0).stop 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 -- 172.21.15.116:0/2792789453 shutdown_connections 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70c2290 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 --2- 172.21.15.116:0/2792789453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 -- 172.21.15.116:0/2792789453 >> 172.21.15.116:0/2792789453 conn(0x559aa70b3a00 msgr2=0x559aa70b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 -- 172.21.15.116:0/2792789453 shutdown_connections 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.452+0000 7ff83c185740 1 -- 172.21.15.116:0/2792789453 wait complete. 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 -- start start 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 Processor -- start 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559aa70c55c0 con 0x559aa70b9bf0 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559aa6fd8310 con 0x559aa70c27d0 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559aa70c5740 con 0x559aa70b92d0 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff83c185740 10 client.0 _ll_register_callbacks cb 0x559aa6fcc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55894/0 (socket says 172.21.15.116:55894) 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 -- 172.21.15.116:0/885006003 learned_addr learned my addr 172.21.15.116:0/885006003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8337fe640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8337fe640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 -- 172.21.15.116:0/885006003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 msgr2=0x559aa70bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70bf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 -- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 msgr2=0x559aa70c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.453+0000 7ff8327fc640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff8327fc640 1 -- 172.21.15.116:0/885006003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aa6d303f0 con 0x559aa70b9bf0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff8327fc640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff828004500 tx=0x7ff828009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cb47e532905292 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828029070 con 0x559aa70b9bf0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff828003a00 con 0x559aa70b9bf0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8280054d0 con 0x559aa70b9bf0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff828035050 con 0x559aa70b9bf0 2024-03-19T02:30:55.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff83c185740 10 client.16626 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559aa6dfa7d0 con 0x559aa70b9bf0 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8280054d0 con 0x559aa70b9bf0 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff828004fe0 con 0x559aa70b9bf0 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff83c185740 10 client.16626 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.454+0000 7ff83c185740 10 client.16626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff83c185740 20 client.16626 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa70d2a10 0x559aa70d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff83c185740 4 client.16626 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559aa707a3e0 con 0x559aa70d2a10 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff8337fe640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa70d2a10 0x559aa70d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff816ff5640 10 client.16626 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:55.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.455+0000 7ff8337fe640 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa70d2a10 0x559aa70d4e00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50e9b7b57c92548f server_cookie=8824f295c2d2f6e8 in_seq=0 out_seq=0 2024-03-19T02:30:55.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff816ff5640 1 -- 172.21.15.116:0/885006003 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559aa707a3e0 con 0x559aa70d2a10 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff816ff5640 10 client.16626 handle_command_reply: tid=0 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 client.16626 shutdown 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 msgr2=0x559aa70bfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff828004500 tx=0x7ff828009fd0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 shutdown_connections 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa70d2a10 0x559aa70d4e00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa70c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70bfcf0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 --2- 172.21.15.116:0/885006003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b92d0 0x559aa70bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 >> 172.21.15.116:0/885006003 conn(0x559aa70b3a00 msgr2=0x559aa70b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff816ff5640 0 client.16626 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 shutdown_connections 2024-03-19T02:30:55.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 1 -- 172.21.15.116:0/885006003 wait complete. 2024-03-19T02:30:55.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.456+0000 7ff83c185740 20 client.16626 trim_cache size 0 max 16384 2024-03-19T02:30:55.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 Processor -- start 2024-03-19T02:30:55.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 -- start start 2024-03-19T02:30:55.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70b92d0 0x559aa70bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 0x559aa70da5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b0d00 con 0x559aa70b9bf0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b2840 con 0x559aa70b92d0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff83c185740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b13d0 con 0x559aa70c27d0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55900/0 (socket says 172.21.15.116:55900) 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 -- 172.21.15.116:0/3141508809 learned_addr learned my addr 172.21.15.116:0/3141508809 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8337fe640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70b92d0 0x559aa70bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8337fe640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 0x559aa70da5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 -- 172.21.15.116:0/3141508809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 msgr2=0x559aa70da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 0x559aa70da5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 -- 172.21.15.116:0/3141508809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70b92d0 msgr2=0x559aa70bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.470+0000 7ff8327fc640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70b92d0 0x559aa70bf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff8327fc640 1 -- 172.21.15.116:0/3141508809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aa70d5170 con 0x559aa70b9bf0 2024-03-19T02:30:55.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff8337fe640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 0x559aa70da5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff8327fc640 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff828000c00 tx=0x7ff828009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d88223512f2664 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff818ff9640 1 -- 172.21.15.116:0/3141508809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828029070 con 0x559aa70b9bf0 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff818ff9640 1 -- 172.21.15.116:0/3141508809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8280235a0 con 0x559aa70b9bf0 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff818ff9640 1 -- 172.21.15.116:0/3141508809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828003b60 con 0x559aa70b9bf0 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff83c185740 1 -- 172.21.15.116:0/3141508809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 msgr2=0x559aa70ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.471+0000 7ff83c185740 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff828000c00 tx=0x7ff828009a70 comp rx=0 tx=0).stop 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- 172.21.15.116:0/3141508809 shutdown_connections 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70c27d0 0x559aa70da5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b9bf0 0x559aa70ce390 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- 172.21.15.116:0/3141508809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70b92d0 0x559aa70bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- 172.21.15.116:0/3141508809 >> 172.21.15.116:0/3141508809 conn(0x559aa70b3a00 msgr2=0x559aa70b56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- 172.21.15.116:0/3141508809 shutdown_connections 2024-03-19T02:30:55.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- 172.21.15.116:0/3141508809 wait complete. 2024-03-19T02:30:55.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- start start 2024-03-19T02:30:55.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 Processor -- start 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b9bf0 0x559aa70c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa7136be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b2170 con 0x559aa70b92d0 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341b1aa0 con 0x559aa70c27d0 2024-03-19T02:30:55.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8341aece0 con 0x559aa70b9bf0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff83c185740 10 client.0 _ll_register_callbacks cb 0x7ff82c011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff8337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff8337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55912/0 (socket says 172.21.15.116:55912) 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff8337fe640 1 -- 172.21.15.116:0/530271437 learned_addr learned my addr 172.21.15.116:0/530271437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff8327fc640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b9bf0 0x559aa70c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.472+0000 7ff8337fe640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa7136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 -- 172.21.15.116:0/530271437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b9bf0 msgr2=0x559aa70c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b9bf0 0x559aa70c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 -- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 msgr2=0x559aa7136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa7136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 -- 172.21.15.116:0/530271437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aa7137120 con 0x559aa70b92d0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff8337fe640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x559aa70d5530 tx=0x7ff82c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1826c9e8d488f2c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff816ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c027700 con 0x559aa70b92d0 2024-03-19T02:30:55.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff82c004b20 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8280235a0 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff82c004d20 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff83c185740 10 client.16632 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559aa7138480 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.473+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8280235a0 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff82c032020 con 0x559aa70b92d0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 10 client.16632 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 10 client.16632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 20 client.16632 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa7138bf0 0x559aa713b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 4 client.16632 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559aa70299f0 con 0x559aa7138bf0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff8327fc640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa7138bf0 0x559aa713b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff8327fc640 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa7138bf0 0x559aa713b010 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e16e83e8b775d414 server_cookie=280012e699b21675 in_seq=0 out_seq=0 2024-03-19T02:30:55.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff816ff5640 10 client.16632 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff816ff5640 1 -- 172.21.15.116:0/530271437 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559aa70299f0 con 0x559aa7138bf0 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff816ff5640 10 client.16632 handle_command_reply: tid=0 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 client.16632 shutdown 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 msgr2=0x559aa70c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x559aa70d5530 tx=0x7ff82c013040 comp rx=0 tx=0).stop 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 shutdown_connections 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559aa7138bf0 0x559aa713b010 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559aa70c27d0 0x559aa7136be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559aa70b9bf0 0x559aa70c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff816ff5640 0 client.16632 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 --2- 172.21.15.116:0/530271437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559aa70b92d0 0x559aa70c5b70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.474+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 >> 172.21.15.116:0/530271437 conn(0x559aa70b3a00 msgr2=0x559aa70bf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 shutdown_connections 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff83c185740 1 -- 172.21.15.116:0/530271437 wait complete. 2024-03-19T02:30:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff83c185740 20 client.16632 trim_cache size 0 max 16384 2024-03-19T02:30:55.482 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 msgr2=0x7ff81c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 0x7ff81c0a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8240bd470 tx=0x7ff82408f110 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 msgr2=0x7ff83419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff824005940 tx=0x7ff82402c5f0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 shutdown_connections 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff81c0a29f0 0x7ff81c0a4eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83410e170 0x7ff83419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff834100ee0 0x7ff83419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 --2- 172.21.15.116:0/305409332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8341005c0 0x7ff83419a1b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 >> 172.21.15.116:0/305409332 conn(0x7ff8340778d0 msgr2=0x7ff83410f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 shutdown_connections 2024-03-19T02:30:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.475+0000 7ff8394c0640 1 -- 172.21.15.116:0/305409332 wait complete. 2024-03-19T02:30:55.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:30:55.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 -- 172.21.15.116:0/4065359881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1010e170 msgr2=0x7fce10110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 --2- 172.21.15.116:0/4065359881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1010e170 0x7fce10110560 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fce08009ca0 tx=0x7fce0802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 -- 172.21.15.116:0/4065359881 shutdown_connections 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 --2- 172.21.15.116:0/4065359881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1010e170 0x7fce10110560 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 --2- 172.21.15.116:0/4065359881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce10100ee0 0x7fce1010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 --2- 172.21.15.116:0/4065359881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce101005c0 0x7fce101009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 -- 172.21.15.116:0/4065359881 >> 172.21.15.116:0/4065359881 conn(0x7fce100778d0 msgr2=0x7fce10077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 -- 172.21.15.116:0/4065359881 shutdown_connections 2024-03-19T02:30:55.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.602+0000 7fce14d10640 1 -- 172.21.15.116:0/4065359881 wait complete. 2024-03-19T02:30:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 Processor -- start 2024-03-19T02:30:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 -- start start 2024-03-19T02:30:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113c50 con 0x7fce101005c0 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113ad0 con 0x7fce10100ee0 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce14d10640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113dd0 con 0x7fce1010e170 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43152/0 (socket says 172.21.15.116:43152) 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34052/0 (socket says 172.21.15.116:34052) 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0ed9d640 1 -- 172.21.15.116:0/3553018308 learned_addr learned my addr 172.21.15.116:0/3553018308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55926/0 (socket says 172.21.15.116:55926) 2024-03-19T02:30:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 -- 172.21.15.116:0/3553018308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 msgr2=0x7fce1019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 -- 172.21.15.116:0/3553018308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 msgr2=0x7fce1019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0e59c640 1 -- 172.21.15.116:0/3553018308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1019f220 con 0x7fce101005c0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.603+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fce0e59c640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fce0000e9f0 tx=0x7fce0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fcdff7fe640 1 -- 172.21.15.116:0/3553018308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0000ccc0 con 0x7fce101005c0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1019f510 con 0x7fce101005c0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fcdff7fe640 1 -- 172.21.15.116:0/3553018308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce000108e0 con 0x7fce101005c0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fcdff7fe640 1 -- 172.21.15.116:0/3553018308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00018880 con 0x7fce101005c0 2024-03-19T02:30:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.604+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce101a6da0 con 0x7fce101005c0 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.606+0000 7fcdff7fe640 1 -- 172.21.15.116:0/3553018308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fce00010a90 con 0x7fce101005c0 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 Processor -- start 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 -- start start 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fcdff7fe640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 0x7fcdf80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0febc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564b8d0564b0 con 0x564b8d0fc7d0 2024-03-19T02:30:55.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564b8cffed70 con 0x564b8d0f32d0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce179d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b8d052550 con 0x564b8d0f3bf0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43168/0 (socket says 172.21.15.116:43168) 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 -- 172.21.15.116:0/554961957 learned_addr learned my addr 172.21.15.116:0/554961957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34064/0 (socket says 172.21.15.116:34064) 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0e59c640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 0x7fcdf80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fcdff7fe640 1 -- 172.21.15.116:0/3553018308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce000cf550 con 0x7fce101005c0 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 -- 172.21.15.116:0/554961957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 msgr2=0x564b8d0f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 -- 172.21.15.116:0/554961957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 msgr2=0x564b8d0febc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0febc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.607+0000 7fce0dd9b640 1 -- 172.21.15.116:0/554961957 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8d021210 con 0x564b8d0f3bf0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.608+0000 7fce0e59c640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 0x7fcdf80a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce000ccf20 tx=0x7fce00086040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.608+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce04011470 tx=0x7fce04011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=540a47930d9a464b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.608+0000 7fcdfcff9640 1 -- 172.21.15.116:0/554961957 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0401e070 con 0x564b8d0f3bf0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.608+0000 7fcdfcff9640 1 -- 172.21.15.116:0/554961957 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04004310 con 0x564b8d0f3bf0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.608+0000 7fcdfcff9640 1 -- 172.21.15.116:0/554961957 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0400f880 con 0x564b8d0f3bf0 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- 172.21.15.116:0/554961957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 msgr2=0x564b8d0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce04011470 tx=0x7fce04011930 comp rx=0 tx=0).stop 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- 172.21.15.116:0/554961957 shutdown_connections 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0fc290 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- 172.21.15.116:0/554961957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- 172.21.15.116:0/554961957 >> 172.21.15.116:0/554961957 conn(0x564b8d0eda00 msgr2=0x564b8d0edde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- 172.21.15.116:0/554961957 shutdown_connections 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- 172.21.15.116:0/554961957 wait complete. 2024-03-19T02:30:55.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 -- start start 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 Processor -- start 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.609+0000 7fce179d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564b8d0ff5c0 con 0x564b8d0fc7d0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564b8d012310 con 0x564b8d0f32d0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b8d0ff740 con 0x564b8d0f3bf0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 10 client.0 _ll_register_callbacks cb 0x564b8d0062d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55942/0 (socket says 172.21.15.116:55942) 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43170/0 (socket says 172.21.15.116:43170) 2024-03-19T02:30:55.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 -- 172.21.15.116:0/1078255808 learned_addr learned my addr 172.21.15.116:0/1078255808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 -- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 msgr2=0x564b8d0f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 -- 172.21.15.116:0/1078255808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 msgr2=0x564b8d0f9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 -- 172.21.15.116:0/1078255808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8cd6a3f0 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fce08037040 tx=0x7fce0802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178b0151efdbb181 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fcdf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08039040 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce08003d80 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce04004310 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fce08056050 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.610+0000 7fce179d8740 10 client.16644 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564b8ce347d0 con 0x564b8d0fc7d0 2024-03-19T02:30:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04004310 con 0x564b8d0fc7d0 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fce0803f070 con 0x564b8d0fc7d0 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 10 client.16644 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 10 client.16644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 20 client.16644 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d10cb30 0x564b8d10efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 4 client.16644 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b8d0b43e0 con 0x564b8d10cb30 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d10cb30 0x564b8d10efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d10cb30 0x564b8d10efb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0d8be17e7f23c0 server_cookie=89b48f4633849507 in_seq=0 out_seq=0 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.611+0000 7fcdf5ffb640 10 client.16644 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/1078255808 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b8d0b43e0 con 0x564b8d10cb30 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fcdf5ffb640 10 client.16644 handle_command_reply: tid=0 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 client.16644 shutdown 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 msgr2=0x564b8d0fa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fce08037040 tx=0x7fce0802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 shutdown_connections 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d10cb30 0x564b8d10efb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0fc7d0 0x564b8d0fa2f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 --2- 172.21.15.116:0/1078255808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0f32d0 0x564b8d0f9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 >> 172.21.15.116:0/1078255808 conn(0x564b8d0eda00 msgr2=0x564b8d0ef750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fcdf5ffb640 0 client.16644 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 shutdown_connections 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.612+0000 7fce179d8740 1 -- 172.21.15.116:0/1078255808 wait complete. 2024-03-19T02:30:55.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.613+0000 7fce179d8740 20 client.16644 trim_cache size 0 max 16384 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 Processor -- start 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 -- start start 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d0fbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b0d80 con 0x564b8d0f3bf0 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b28c0 con 0x564b8d0fc7d0 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce179d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b1450 con 0x564b8d0f32d0 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55948/0 (socket says 172.21.15.116:55948) 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 -- 172.21.15.116:0/3653503722 learned_addr learned my addr 172.21.15.116:0/3653503722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d0fbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 -- 172.21.15.116:0/3653503722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 msgr2=0x564b8d0fbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d0fbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 -- 172.21.15.116:0/3653503722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 msgr2=0x564b8d108390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d108390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 -- 172.21.15.116:0/3653503722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8d0f7dc0 con 0x564b8d0f3bf0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d0fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fce04000c00 tx=0x7fce0400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61373cab5e373b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.627+0000 7fcdf7fff640 1 -- 172.21.15.116:0/3653503722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0401e040 con 0x564b8d0f3bf0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fcdf7fff640 1 -- 172.21.15.116:0/3653503722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04030070 con 0x564b8d0f3bf0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fcdf7fff640 1 -- 172.21.15.116:0/3653503722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce040039e0 con 0x564b8d0f3bf0 2024-03-19T02:30:55.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 -- 172.21.15.116:0/3653503722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 msgr2=0x564b8d0f89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fce04000c00 tx=0x7fce0400f580 comp rx=0 tx=0).stop 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 -- 172.21.15.116:0/3653503722 shutdown_connections 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d108390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d0f89d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 --2- 172.21.15.116:0/3653503722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d0fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 -- 172.21.15.116:0/3653503722 >> 172.21.15.116:0/3653503722 conn(0x564b8d0eda00 msgr2=0x564b8d0ec310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 -- 172.21.15.116:0/3653503722 shutdown_connections 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.628+0000 7fce179d8740 1 -- 172.21.15.116:0/3653503722 wait complete. 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 -- start start 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 Processor -- start 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d104540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d0ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b21f0 con 0x564b8d0f3bf0 2024-03-19T02:30:55.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b1b20 con 0x564b8d0fc7d0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce101aed60 con 0x564b8d0f32d0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce179d8740 10 client.0 _ll_register_callbacks cb 0x7fce08009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55950/0 (socket says 172.21.15.116:55950) 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 -- 172.21.15.116:0/2008539828 learned_addr learned my addr 172.21.15.116:0/2008539828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d0ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 -- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 msgr2=0x564b8d104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 -- 172.21.15.116:0/2008539828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 msgr2=0x564b8d0ff9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d0ff9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 -- 172.21.15.116:0/2008539828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8d1000d0 con 0x564b8d0f3bf0 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:30:55.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fce0dd9b640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fce0401c040 tx=0x7fce04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40d09e297d77f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.629+0000 7fcdf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04038020 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce04003960 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce04030070 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fce04043050 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 10 client.16650 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564b8d172890 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04030070 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fce0400f710 con 0x564b8d0f3bf0 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 10 client.16650 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 10 client.16650 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 20 client.16650 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d172e10 0x564b8d175230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 4 client.16650 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:55.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b8d0639f0 con 0x564b8d172e10 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.630+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d172e10 0x564b8d175230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fcdf5ffb640 10 client.16650 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce0ed9d640 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d172e10 0x564b8d175230 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=33f02371658e0be2 server_cookie=3c8d367bc3b3f541 in_seq=0 out_seq=0 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fcdf5ffb640 1 -- 172.21.15.116:0/2008539828 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b8d0639f0 con 0x564b8d172e10 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fcdf5ffb640 10 client.16650 handle_command_reply: tid=0 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 client.16650 shutdown 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 msgr2=0x564b8d104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fce0401c040 tx=0x7fce04009bd0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 shutdown_connections 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x564b8d172e10 0x564b8d175230 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b8d0fc7d0 0x564b8d0ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b8d0f3bf0 0x564b8d104a80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 --2- 172.21.15.116:0/2008539828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b8d0f32d0 0x564b8d104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 >> 172.21.15.116:0/2008539828 conn(0x564b8d0eda00 msgr2=0x564b8d0eed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fcdf5ffb640 0 client.16650 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:30:55.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 shutdown_connections 2024-03-19T02:30:55.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 1 -- 172.21.15.116:0/2008539828 wait complete. 2024-03-19T02:30:55.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.631+0000 7fce179d8740 20 client.16650 trim_cache size 0 max 16384 2024-03-19T02:30:55.639 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 msgr2=0x7fcdf80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 0x7fcdf80a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce000ccf20 tx=0x7fce00086040 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 msgr2=0x7fce1019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fce0000e9f0 tx=0x7fce0000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 shutdown_connections 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2970 0x7fcdf80a4e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce1010e170 0x7fce1019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce10100ee0 0x7fce1019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 --2- 172.21.15.116:0/3553018308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce101005c0 0x7fce1019a230 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 >> 172.21.15.116:0/3553018308 conn(0x7fce100778d0 msgr2=0x7fce1010f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 shutdown_connections 2024-03-19T02:30:55.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.632+0000 7fce14d10640 1 -- 172.21.15.116:0/3553018308 wait complete. 2024-03-19T02:30:55.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/2374721463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d78101060 msgr2=0x7f9d7810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/2374721463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d78101060 0x7f9d7810d570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009cd0 tx=0x7f9d7002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/2374721463 shutdown_connections 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/2374721463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d78101060 0x7f9d7810d570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/2374721463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/2374721463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d781066b0 0x7f9d78106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/2374721463 >> 172.21.15.116:0/2374721463 conn(0x7f9d780778d0 msgr2=0x7f9d78077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/2374721463 shutdown_connections 2024-03-19T02:30:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.755+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/2374721463 wait complete. 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 Processor -- start 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 -- start start 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 0x7f9d781a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d78110970 con 0x7f9d78101060 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781107f0 con 0x7f9d781006a0 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7c9e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d78110af0 con 0x7f9d781066b0 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55966/0 (socket says 172.21.15.116:55966) 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d6ffff640 1 -- 172.21.15.116:0/1277970025 learned_addr learned my addr 172.21.15.116:0/1277970025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.756+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 0x7f9d781a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 -- 172.21.15.116:0/1277970025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 msgr2=0x7f9d78198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 -- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 msgr2=0x7f9d78199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 -- 172.21.15.116:0/1277970025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d781a43d0 con 0x7f9d781066b0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d7659c640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 0x7f9d781a3e90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d7002c630 tx=0x7f9d70035d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d6f7fe640 1 -- 172.21.15.116:0/1277970025 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70039040 con 0x7f9d781066b0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d6f7fe640 1 -- 172.21.15.116:0/1277970025 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d70044070 con 0x7f9d781066b0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d781a45a0 con 0x7f9d781066b0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d6f7fe640 1 -- 172.21.15.116:0/1277970025 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d700042a0 con 0x7f9d781066b0 2024-03-19T02:30:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.757+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d781a4a00 con 0x7f9d781066b0 2024-03-19T02:30:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.759+0000 7f9d6f7fe640 1 -- 172.21.15.116:0/1277970025 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9d70004440 con 0x7f9d781066b0 2024-03-19T02:30:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 Processor -- start 2024-03-19T02:30:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 -- start start 2024-03-19T02:30:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15bf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d0a15224b0 con 0x55d0a15bfbf0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d0a14cad70 con 0x55d0a15c87d0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0a151e550 con 0x55d0a15bf2d0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d6f7fe640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 0x7f9d640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15bf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.760+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55978/0 (socket says 172.21.15.116:55978) 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d7659c640 1 -- 172.21.15.116:0/206329341 learned_addr learned my addr 172.21.15.116:0/206329341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6f7fe640 1 -- 172.21.15.116:0/1277970025 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9d700f14b0 con 0x7f9d781066b0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 0x7f9d640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 -- 172.21.15.116:0/206329341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 msgr2=0x55d0a15cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 -- 172.21.15.116:0/206329341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 msgr2=0x55d0a15c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 -- 172.21.15.116:0/206329341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0a14ed210 con 0x55d0a15bf2d0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d7659c640 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15bf6b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d780694d0 tx=0x7f9d60009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d6a7ef07fd2652 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d76d9d640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 0x7f9d640a4e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9d700ba270 tx=0x7f9d7002c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6cff9640 1 -- 172.21.15.116:0/206329341 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000fc50 con 0x55d0a15bf2d0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6cff9640 1 -- 172.21.15.116:0/206329341 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d6000fdf0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.761+0000 7f9d6cff9640 1 -- 172.21.15.116:0/206329341 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60004e40 con 0x55d0a15bf2d0 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/206329341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 msgr2=0x55d0a15bf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15bf6b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d780694d0 tx=0x7f9d60009590 comp rx=0 tx=0).stop 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/206329341 shutdown_connections 2024-03-19T02:30:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/206329341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15bf6b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/206329341 >> 172.21.15.116:0/206329341 conn(0x55d0a15b9a00 msgr2=0x55d0a15b9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/206329341 shutdown_connections 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/206329341 wait complete. 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 -- start start 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.762+0000 7f9d7f5d3740 1 Processor -- start 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d0a15cb5c0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d0a14de310 con 0x55d0a15c87d0 2024-03-19T02:30:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55980/0 (socket says 172.21.15.116:55980) 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0a15cb740 con 0x55d0a15bfbf0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 -- 172.21.15.116:0/824947222 learned_addr learned my addr 172.21.15.116:0/824947222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7f5d3740 10 client.0 _ll_register_callbacks cb 0x55d0a14d22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d7659c640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 -- 172.21.15.116:0/824947222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bfbf0 msgr2=0x55d0a15c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 -- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 msgr2=0x55d0a15c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 -- 172.21.15.116:0/824947222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0a12363f0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009c50 tx=0x7f9d600097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e04a6922ed0cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.763+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60028030 con 0x55d0a15bf2d0 2024-03-19T02:30:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9d60019040 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d6000fdf0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9d60035050 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 10 client.16662 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d0a13007d0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d6000fdf0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9d6001d030 con 0x55d0a15bf2d0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 10 client.16662 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 20 client.16662 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a15d8af0 0x55d0a15daee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 4 client.16662 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0a15803e0 con 0x55d0a15d8af0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.764+0000 7f9d7659c640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a15d8af0 0x55d0a15daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.765+0000 7f9d5dffb640 10 client.16662 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.765+0000 7f9d7659c640 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a15d8af0 0x55d0a15daee0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc629cd20ac42cd7 server_cookie=5387f56937e34947 in_seq=0 out_seq=0 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d5dffb640 1 -- 172.21.15.116:0/824947222 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0a15803e0 con 0x55d0a15d8af0 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d5dffb640 10 client.16662 handle_command_reply: tid=0 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 client.16662 shutdown 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 msgr2=0x55d0a15c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009c50 tx=0x7f9d600097a0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 shutdown_connections 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a15d8af0 0x55d0a15daee0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15c87d0 0x55d0a15c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d5dffb640 0 client.16662 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/824947222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c5930 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 >> 172.21.15.116:0/824947222 conn(0x55d0a15b9a00 msgr2=0x55d0a15bb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 shutdown_connections 2024-03-19T02:30:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.766+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/824947222 wait complete. 2024-03-19T02:30:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.767+0000 7f9d7f5d3740 20 client.16662 trim_cache size 0 max 16384 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 Processor -- start 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 -- start start 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 0x55d0a15bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781aeaa0 con 0x55d0a15bfbf0 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781b05e0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781af170 con 0x55d0a15c87d0 2024-03-19T02:30:55.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55982/0 (socket says 172.21.15.116:55982) 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 -- 172.21.15.116:0/1396591125 learned_addr learned my addr 172.21.15.116:0/1396591125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 0x55d0a15bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 -- 172.21.15.116:0/1396591125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 msgr2=0x55d0a15bb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 0x55d0a15bb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 -- 172.21.15.116:0/1396591125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bf2d0 msgr2=0x55d0a15c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d7659c640 1 -- 172.21.15.116:0/1396591125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0a15be560 con 0x55d0a15bfbf0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.780+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 0x55d0a15bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7659c640 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9d68005890 tx=0x7f9d68005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4464223b8277a473 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d5ffff640 1 -- 172.21.15.116:0/1396591125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68004310 con 0x55d0a15bfbf0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d5ffff640 1 -- 172.21.15.116:0/1396591125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d68027070 con 0x55d0a15bfbf0 2024-03-19T02:30:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d5ffff640 1 -- 172.21.15.116:0/1396591125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68003a80 con 0x55d0a15bfbf0 2024-03-19T02:30:55.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 84803 3.31 MB/sec execute 18 sec latency 15.223 ms 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/1396591125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 msgr2=0x55d0a15c61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9d68005890 tx=0x7f9d68005f90 comp rx=0 tx=0).stop 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/1396591125 shutdown_connections 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15c87d0 0x55d0a15bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15bfbf0 0x55d0a15c61d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/1396591125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bf2d0 0x55d0a15c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.781+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/1396591125 >> 172.21.15.116:0/1396591125 conn(0x55d0a15b9a00 msgr2=0x55d0a15bae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/1396591125 shutdown_connections 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/1396591125 wait complete. 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- start start 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 Processor -- start 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 0x55d0a15cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15c87d0 0x55d0a15cce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781aff10 con 0x55d0a15c87d0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781af840 con 0x55d0a15bfbf0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d781aca80 con 0x55d0a15bf2d0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7f5d3740 10 client.0 _ll_register_callbacks cb 0x7f9d68009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15c87d0 0x55d0a15cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d6ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43260/0 (socket says 172.21.15.116:43260) 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d6ffff640 1 -- 172.21.15.116:0/3535335968 learned_addr learned my addr 172.21.15.116:0/3535335968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:55.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.782+0000 7f9d7659c640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 0x55d0a15cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 -- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 msgr2=0x55d0a15cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 0x55d0a15cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 -- 172.21.15.116:0/3535335968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15c87d0 msgr2=0x55d0a15cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15c87d0 0x55d0a15cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 -- 172.21.15.116:0/3535335968 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0a163dde0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d7659c640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 0x55d0a15cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d6ffff640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d6001ff60 tx=0x7f9d60004bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f416ef8e4be53e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60028040 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d68027070 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9d6002f0d0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9d6001e800 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d7f5d3740 10 client.26428 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d0a163f0e0 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.783+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d68027070 con 0x55d0a15bf2d0 2024-03-19T02:30:55.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9d6001d040 con 0x55d0a15bf2d0 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 10 client.26428 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 10 client.26428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 20 client.26428 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a163f670 0x55d0a1641a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 4 client.26428 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0a152f9f0 con 0x55d0a163f670 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7659c640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a163f670 0x55d0a1641a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d5dffb640 10 client.26428 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7659c640 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a163f670 0x55d0a1641a90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49cdb01e169594f9 server_cookie=a61997589bdac0a6 in_seq=0 out_seq=0 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d5dffb640 1 -- 172.21.15.116:0/3535335968 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0a152f9f0 con 0x55d0a163f670 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d5dffb640 10 client.26428 handle_command_reply: tid=0 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 1 client.26428 shutdown 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 msgr2=0x55d0a15cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.784+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d6001ff60 tx=0x7f9d60004bb0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 shutdown_connections 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0a163f670 0x55d0a1641a90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0a15c87d0 0x55d0a15cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0a15bfbf0 0x55d0a15cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 --2- 172.21.15.116:0/3535335968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0a15bf2d0 0x55d0a15cc390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d5dffb640 0 client.26428 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 >> 172.21.15.116:0/3535335968 conn(0x55d0a15b9a00 msgr2=0x55d0a15bb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 shutdown_connections 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 1 -- 172.21.15.116:0/3535335968 wait complete. 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7f5d3740 20 client.26428 trim_cache size 0 max 16384 2024-03-19T02:30:55.792 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:30:55.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 msgr2=0x7f9d640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 0x7f9d640a4e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9d700ba270 tx=0x7f9d7002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 msgr2=0x7f9d781a3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:55.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 0x7f9d781a3e90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9d7002c630 tx=0x7f9d70035d60 comp rx=0 tx=0).stop 2024-03-19T02:30:55.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 shutdown_connections 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9d640a2970 0x7f9d640a4e30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d781066b0 0x7f9d781a3e90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d78101060 0x7f9d78199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 --2- 172.21.15.116:0/1277970025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d781006a0 0x7f9d78198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 >> 172.21.15.116:0/1277970025 conn(0x7f9d780778d0 msgr2=0x7f9d78102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 shutdown_connections 2024-03-19T02:30:55.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:55.785+0000 7f9d7c9e6640 1 -- 172.21.15.116:0/1277970025 wait complete. 2024-03-19T02:30:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:55 smithi116 ceph-mon[32263]: pgmap v96: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:55 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/602448983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:55 smithi162 ceph-mon[35761]: pgmap v96: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:55 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/602448983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:55 smithi154 ceph-mon[35693]: pgmap v96: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:30:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:55 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/602448983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:30:56.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 85212 3.36 MB/sec execute 19 sec latency 18.013 ms 2024-03-19T02:30:56.798 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:30:56.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:30:56.798 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.925+0000 7f05b0c4b640 1 -- 172.21.15.116:0/2391495841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac1066a0 msgr2=0x7f05ac106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.925+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/2391495841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac1066a0 0x7f05ac106a80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f059c009930 tx=0x7f059c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.925+0000 7f05b0c4b640 1 -- 172.21.15.116:0/2391495841 shutdown_connections 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/2391495841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac101050 0x7f05ac10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/2391495841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac100690 0x7f05ac100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/2391495841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac1066a0 0x7f05ac106a80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 -- 172.21.15.116:0/2391495841 >> 172.21.15.116:0/2391495841 conn(0x7f05ac077820 msgr2=0x7f05ac077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 -- 172.21.15.116:0/2391495841 shutdown_connections 2024-03-19T02:30:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 -- 172.21.15.116:0/2391495841 wait complete. 2024-03-19T02:30:56.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 Processor -- start 2024-03-19T02:30:56.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.926+0000 7f05b0c4b640 1 -- start start 2024-03-19T02:30:56.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac101050 0x7f05ac103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac1066a0 0x7f05ac1037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac110960 con 0x7f05ac101050 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1107e0 con 0x7f05ac1066a0 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05b0c4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac110ae0 con 0x7f05ac100690 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43282/0 (socket says 172.21.15.116:43282) 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 -- 172.21.15.116:0/285201134 learned_addr learned my addr 172.21.15.116:0/285201134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac1066a0 0x7f05ac1037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac101050 0x7f05ac103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 -- 172.21.15.116:0/285201134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac1066a0 msgr2=0x7f05ac1037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac1066a0 0x7f05ac1037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 -- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac101050 msgr2=0x7f05ac103260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac101050 0x7f05ac103260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 -- 172.21.15.116:0/285201134 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac1a64d0 con 0x7f05ac100690 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.927+0000 7f05aa59c640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f059c009230 tx=0x7f059c02c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.928+0000 7f05a37fe640 1 -- 172.21.15.116:0/285201134 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c043020 con 0x7f05ac100690 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.928+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ac1a6760 con 0x7f05ac100690 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.928+0000 7f05a37fe640 1 -- 172.21.15.116:0/285201134 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f059c02ca70 con 0x7f05ac100690 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.928+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ac1a6d00 con 0x7f05ac100690 2024-03-19T02:30:56.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.928+0000 7f05a37fe640 1 -- 172.21.15.116:0/285201134 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c003a50 con 0x7f05ac100690 2024-03-19T02:30:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.930+0000 7f05a37fe640 1 -- 172.21.15.116:0/285201134 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f059c046020 con 0x7f05ac100690 2024-03-19T02:30:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.930+0000 7f05b3838740 1 Processor -- start 2024-03-19T02:30:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 -- start start 2024-03-19T02:30:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 0x55cbe95ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 0x55cbe95f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cbe954a160 con 0x55cbe95ef880 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cbe94f2e10 con 0x55cbe95e6380 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05b3838740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbe9546200 con 0x55cbe95e6ca0 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 0x55cbe95f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34164/0 (socket says 172.21.15.116:34164) 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 -- 172.21.15.116:0/193744618 learned_addr learned my addr 172.21.15.116:0/193744618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aa59c640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 0x55cbe95ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05a37fe640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 0x7f05980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 -- 172.21.15.116:0/193744618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 msgr2=0x55cbe95ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 0x55cbe95ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 -- 172.21.15.116:0/193744618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 msgr2=0x55cbe95f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 0x55cbe95f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 -- 172.21.15.116:0/193744618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe93267e0 con 0x55cbe95e6380 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05a3fff640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 0x55cbe95f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aad9d640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 0x7f05980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.931+0000 7f05aa59c640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 0x55cbe95ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05a37fe640 1 -- 172.21.15.116:0/285201134 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f059c0f0fe0 con 0x7f05ac100690 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05aad9d640 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05a4011470 tx=0x7f05a4011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f8e17eb7573bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05aad9d640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 0x7f05980a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f05a4023000 tx=0x7f05a4024000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05a0ff9640 1 -- 172.21.15.116:0/193744618 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4017050 con 0x55cbe95e6380 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05a0ff9640 1 -- 172.21.15.116:0/193744618 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a401bc30 con 0x55cbe95e6380 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.932+0000 7f05a0ff9640 1 -- 172.21.15.116:0/193744618 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4026af0 con 0x55cbe95e6380 2024-03-19T02:30:56.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- 172.21.15.116:0/193744618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 msgr2=0x55cbe95e6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05a4011470 tx=0x7f05a4011930 comp rx=0 tx=0).stop 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- 172.21.15.116:0/193744618 shutdown_connections 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95ef880 0x55cbe95f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6ca0 0x55cbe95ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 --2- 172.21.15.116:0/193744618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95e6760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- 172.21.15.116:0/193744618 >> 172.21.15.116:0/193744618 conn(0x55cbe95e0ab0 msgr2=0x55cbe95e0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- 172.21.15.116:0/193744618 shutdown_connections 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- 172.21.15.116:0/193744618 wait complete. 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 -- start start 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.933+0000 7f05b3838740 1 Processor -- start 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe95ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cbe95f2670 con 0x55cbe95e6380 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cbe95061a0 con 0x55cbe95e6ca0 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbe95f27f0 con 0x55cbe95ef880 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05b3838740 10 client.0 _ll_register_callbacks cb 0x55cbe925dc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe95ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56008/0 (socket says 172.21.15.116:56008) 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 -- 172.21.15.116:0/516743992 learned_addr learned my addr 172.21.15.116:0/516743992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05a3fff640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34170/0 (socket says 172.21.15.116:34170) 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 -- 172.21.15.116:0/516743992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 msgr2=0x55cbe95ed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe95ed1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 -- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 msgr2=0x55cbe95eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 -- 172.21.15.116:0/516743992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe9386f70 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05a3fff640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f05aa59c640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f059c009900 tx=0x7f059c0bcf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788218d9e85480df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.934+0000 7f0591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0ee7f0 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f059c0f3080 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a401bc30 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f059c0f3280 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f05b3838740 10 client.16680 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.935+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cbe921e170 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a401bc30 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f059c0bd6d0 con 0x55cbe95e6380 2024-03-19T02:30:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 10 client.16680 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 20 client.16680 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe95ffa60 0x55cbe9601ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 4 client.16680 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbe95576a0 con 0x55cbe95ffa60 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05a3fff640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe95ffa60 0x55cbe9601ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f05a3fff640 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe95ffa60 0x55cbe9601ee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78a0f70a3cf38c84 server_cookie=5e047dce79ae703a in_seq=0 out_seq=0 2024-03-19T02:30:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.936+0000 7f0591ffb640 10 client.16680 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.938+0000 7f0591ffb640 1 -- 172.21.15.116:0/516743992 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbe95576a0 con 0x55cbe95ffa60 2024-03-19T02:30:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.938+0000 7f0591ffb640 10 client.16680 handle_command_reply: tid=0 2024-03-19T02:30:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.938+0000 7f05b3838740 1 client.16680 shutdown 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 msgr2=0x55cbe95ec770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f059c009900 tx=0x7f059c0bcf90 comp rx=0 tx=0).stop 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 shutdown_connections 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe95ffa60 0x55cbe9601ee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f0591ffb640 0 client.16680 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe95ed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6ca0 0x55cbe95eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 --2- 172.21.15.116:0/516743992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6380 0x55cbe95ec770 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 >> 172.21.15.116:0/516743992 conn(0x55cbe95e0ab0 msgr2=0x55cbe95e2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 shutdown_connections 2024-03-19T02:30:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 1 -- 172.21.15.116:0/516743992 wait complete. 2024-03-19T02:30:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.939+0000 7f05b3838740 20 client.16680 trim_cache size 0 max 16384 2024-03-19T02:30:56.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 Processor -- start 2024-03-19T02:30:56.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 -- start start 2024-03-19T02:30:56.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe96077f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0100d0 con 0x55cbe95e6ca0 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f058c011c10 con 0x55cbe95e6380 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.957+0000 7f05b3838740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0107a0 con 0x55cbe95ef880 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56020/0 (socket says 172.21.15.116:56020) 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 -- 172.21.15.116:0/2696915476 learned_addr learned my addr 172.21.15.116:0/2696915476 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05aad9d640 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe96077f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 -- 172.21.15.116:0/2696915476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 msgr2=0x55cbe96077f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe96077f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 -- 172.21.15.116:0/2696915476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 msgr2=0x55cbe95eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 -- 172.21.15.116:0/2696915476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe9602560 con 0x55cbe95e6ca0 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05aa59c640 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f05a3fff640 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05ac1027a0 tx=0x7f0594005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d9bfea6c27e33a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f0593fff640 1 -- 172.21.15.116:0/2696915476 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594022170 con 0x55cbe95e6ca0 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f0593fff640 1 -- 172.21.15.116:0/2696915476 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05940227c0 con 0x55cbe95e6ca0 2024-03-19T02:30:56.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.958+0000 7f0593fff640 1 -- 172.21.15.116:0/2696915476 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594010690 con 0x55cbe95e6ca0 2024-03-19T02:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 -- 172.21.15.116:0/2696915476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 msgr2=0x55cbe95fb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f05ac1027a0 tx=0x7f0594005f90 comp rx=0 tx=0).stop 2024-03-19T02:30:56.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 -- 172.21.15.116:0/2696915476 shutdown_connections 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95ef880 0x55cbe96077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95fb4a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 --2- 172.21.15.116:0/2696915476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95e6380 0x55cbe95eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 -- 172.21.15.116:0/2696915476 >> 172.21.15.116:0/2696915476 conn(0x55cbe95e0ab0 msgr2=0x55cbe95dfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.959+0000 7f05b3838740 1 -- 172.21.15.116:0/2696915476 shutdown_connections 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 -- 172.21.15.116:0/2696915476 wait complete. 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 -- start start 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 Processor -- start 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 0x55cbe95f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95ef880 0x55cbe95f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f058c011540 con 0x55cbe95e6ca0 2024-03-19T02:30:56.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f058c010e70 con 0x55cbe95ef880 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f058c00e0b0 con 0x55cbe95e6380 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05b3838740 10 client.0 _ll_register_callbacks cb 0x7f059c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 0x55cbe95f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56034/0 (socket says 172.21.15.116:56034) 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05a3fff640 1 -- 172.21.15.116:0/1330791719 learned_addr learned my addr 172.21.15.116:0/1330791719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.960+0000 7f05aad9d640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95ef880 0x55cbe95f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 -- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95ef880 msgr2=0x55cbe95f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95ef880 0x55cbe95f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 -- 172.21.15.116:0/1330791719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 msgr2=0x55cbe95f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 -- 172.21.15.116:0/1330791719 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe95e02f0 con 0x55cbe95e6380 2024-03-19T02:30:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05a3fff640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05aa59c640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 0x55cbe95f7d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f059c0b9d60 tx=0x7f059c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac6d9631eb92ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f0591ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0b9630 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05940227c0 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f059c0bd800 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f059c0d6240 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05b3838740 10 client.26443 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.961+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cbe95e02f0 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05940227c0 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f059c0f4170 con 0x55cbe95e6380 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 10 client.26443 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 10 client.26443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 20 client.26443 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe9667010 0x55cbe9669430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 4 client.26443 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cbe95519f0 con 0x55cbe9667010 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.962+0000 7f05a3fff640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe9667010 0x55cbe9669430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.963+0000 7f05a3fff640 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe9667010 0x55cbe9669430 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a4d277c7583724d server_cookie=1d6c837a08a5cf19 in_seq=0 out_seq=0 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.963+0000 7f0591ffb640 10 client.26443 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f0591ffb640 1 -- 172.21.15.116:0/1330791719 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cbe95519f0 con 0x55cbe9667010 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f0591ffb640 10 client.26443 handle_command_reply: tid=0 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 client.26443 shutdown 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 msgr2=0x55cbe95f7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 0x55cbe95f7d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f059c0b9d60 tx=0x7f059c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 shutdown_connections 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cbe9667010 0x55cbe9669430 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbe95ef880 0x55cbe95f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cbe95e6ca0 0x55cbe95f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 --2- 172.21.15.116:0/1330791719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbe95e6380 0x55cbe95f7d30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 >> 172.21.15.116:0/1330791719 conn(0x55cbe95e0ab0 msgr2=0x55cbe95e1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f0591ffb640 0 client.26443 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 shutdown_connections 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 1 -- 172.21.15.116:0/1330791719 wait complete. 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.965+0000 7f05b3838740 20 client.26443 trim_cache size 0 max 16384 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:56.971 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:30:56.972 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "c737b807-1882-4f1c-959b-5298c02c515f", 2024-03-19T02:30:56.972 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:30:56.972 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 msgr2=0x7f05980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 0x7f05980a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f05a4023000 tx=0x7f05a4024000 comp rx=0 tx=0).stop 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 msgr2=0x7f05ac102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f059c009230 tx=0x7f059c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 shutdown_connections 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a2970 0x7f05980a4e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05ac1066a0 0x7f05ac1037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05ac101050 0x7f05ac103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 --2- 172.21.15.116:0/285201134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05ac100690 0x7f05ac102d20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 >> 172.21.15.116:0/285201134 conn(0x7f05ac077820 msgr2=0x7f05ac068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 shutdown_connections 2024-03-19T02:30:56.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:56.966+0000 7f05b0c4b640 1 -- 172.21.15.116:0/285201134 wait complete. 2024-03-19T02:30:56.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:30:56.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:30:57.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/4153937408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 msgr2=0x7fdd04106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/4153937408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd04106a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009860 tx=0x7fdd0002bb70 comp rx=0 tx=0).stop 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/4153937408 shutdown_connections 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/4153937408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 0x7fdd0410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/4153937408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/4153937408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd04106a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/4153937408 >> 172.21.15.116:0/4153937408 conn(0x7fdd04077820 msgr2=0x7fdd04077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/4153937408 shutdown_connections 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.099+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/4153937408 wait complete. 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 Processor -- start 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 -- start start 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 0x7fdd04199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd041a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd04110960 con 0x7fdd04100690 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041107e0 con 0x7fdd041066a0 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd0b5c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd04110ae0 con 0x7fdd04101050 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd09360640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd09360640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56046/0 (socket says 172.21.15.116:56046) 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd09360640 1 -- 172.21.15.116:0/2755275698 learned_addr learned my addr 172.21.15.116:0/2755275698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 0x7fdd04199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.100+0000 7fdd09b61640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd041a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 -- 172.21.15.116:0/2755275698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 msgr2=0x7fdd04199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 0x7fdd04199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 -- 172.21.15.116:0/2755275698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 msgr2=0x7fdd041a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd041a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 -- 172.21.15.116:0/2755275698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd041a43a0 con 0x7fdd04100690 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd09360640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdd00037040 tx=0x7fdd00004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdcfa7fc640 1 -- 172.21.15.116:0/2755275698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00041020 con 0x7fdd04100690 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd041a4570 con 0x7fdd04100690 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdcfa7fc640 1 -- 172.21.15.116:0/2755275698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd0002c9c0 con 0x7fdd04100690 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdcfa7fc640 1 -- 172.21.15.116:0/2755275698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00039040 con 0x7fdd04100690 2024-03-19T02:30:57.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.101+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd041a4a30 con 0x7fdd04100690 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 Processor -- start 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 -- start start 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x55750131b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x55750131e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557501276500 con 0x557501313350 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55750121ebe0 con 0x557501312a30 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5575012725a0 con 0x55750131bf30 2024-03-19T02:30:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdcfa7fc640 1 -- 172.21.15.116:0/2755275698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdd00044020 con 0x7fdd04100690 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34218/0 (socket says 172.21.15.116:34218) 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 -- 172.21.15.116:0/2033989718 learned_addr learned my addr 172.21.15.116:0/2033989718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x55750131e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09360640 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x55750131b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 -- 172.21.15.116:0/2033989718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 msgr2=0x55750131e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x55750131e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 -- 172.21.15.116:0/2033989718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 msgr2=0x55750131b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x55750131b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.104+0000 7fdd09b61640 1 -- 172.21.15.116:0/2033989718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557501240000 con 0x557501312a30 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdd09b61640 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009f50 tx=0x7fdcfc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb415f37d4cc2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdcfa7fc640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 0x7fdcf00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdcef7fe640 1 -- 172.21.15.116:0/2033989718 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc004d20 con 0x557501312a30 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdd09360640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 0x7fdcf00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdcef7fe640 1 -- 172.21.15.116:0/2033989718 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcfc004050 con 0x557501312a30 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdcef7fe640 1 -- 172.21.15.116:0/2033989718 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc005280 con 0x557501312a30 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdcfa7fc640 1 -- 172.21.15.116:0/2755275698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd000f0900 con 0x7fdd04100690 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/2033989718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 msgr2=0x557501312e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdd09360640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 0x7fdcf00a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd000ede90 tx=0x7fdd000f6550 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.105+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009f50 tx=0x7fdcfc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/2033989718 shutdown_connections 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x55750131e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x55750131b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/2033989718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501312a30 0x557501312e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/2033989718 >> 172.21.15.116:0/2033989718 conn(0x55750130d160 msgr2=0x55750130d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/2033989718 shutdown_connections 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/2033989718 wait complete. 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- start start 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 Processor -- start 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x557501319390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x5575013198d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55750131ed20 con 0x557501312a30 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.106+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557501232160 con 0x557501313350 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55750131eea0 con 0x55750131bf30 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd0e1ab740 10 client.0 _ll_register_callbacks cb 0x55750126ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56062/0 (socket says 172.21.15.116:56062) 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd09b61640 1 -- 172.21.15.116:0/1406003835 learned_addr learned my addr 172.21.15.116:0/1406003835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x557501319390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd09b61640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x5575013198d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 -- 172.21.15.116:0/1406003835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 msgr2=0x5575013198d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x5575013198d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 -- 172.21.15.116:0/1406003835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 msgr2=0x557501318e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 -- 172.21.15.116:0/1406003835 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575010547d0 con 0x557501313350 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd09b61640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x557501319390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd040694d0 tx=0x7fdcf4011b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3caebf56f83920 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf401e060 con 0x557501313350 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcfc004050 con 0x557501313350 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdcf400fbc0 con 0x557501313350 2024-03-19T02:30:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdcf4025050 con 0x557501313350 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd0e1ab740 10 client.26516 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.107+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557500f8a3f0 con 0x557501313350 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcfc004050 con 0x557501313350 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdcf4019070 con 0x557501313350 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 10 client.26516 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 20 client.26516 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55750132c1a0 0x55750132e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 4 client.26516 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557501283a40 con 0x55750132c1a0 2024-03-19T02:30:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd09b61640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55750132c1a0 0x55750132e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdcecff9640 10 client.26516 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.108+0000 7fdd09b61640 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55750132c1a0 0x55750132e5d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f618f187e99225 server_cookie=9c1edc4d6ba9c296 in_seq=0 out_seq=0 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.109+0000 7fdcecff9640 1 -- 172.21.15.116:0/1406003835 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557501283a40 con 0x55750132c1a0 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.109+0000 7fdcecff9640 10 client.26516 handle_command_reply: tid=0 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.109+0000 7fdd0e1ab740 1 client.26516 shutdown 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 msgr2=0x557501319390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x557501319390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd040694d0 tx=0x7fdcf4011b30 comp rx=0 tx=0).stop 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 shutdown_connections 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55750132c1a0 0x55750132e5d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55750131bf30 0x5575013198d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x557501319390 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdcecff9640 0 client.26516 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/1406003835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501312a30 0x557501318e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 >> 172.21.15.116:0/1406003835 conn(0x55750130d160 msgr2=0x55750130eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 shutdown_connections 2024-03-19T02:30:57.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/1406003835 wait complete. 2024-03-19T02:30:57.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.110+0000 7fdd0e1ab740 20 client.26516 trim_cache size 0 max 16384 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 Processor -- start 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 -- start start 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x5575013196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55750131bf30 0x557501333ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041aeb10 con 0x557501313350 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041b0650 con 0x55750131bf30 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041af1e0 con 0x557501312a30 2024-03-19T02:30:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x5575013196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56066/0 (socket says 172.21.15.116:56066) 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x5575013196a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43374/0 (socket says 172.21.15.116:43374) 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 -- 172.21.15.116:0/3142606269 learned_addr learned my addr 172.21.15.116:0/3142606269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd09b61640 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55750131bf30 0x557501333ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 -- 172.21.15.116:0/3142606269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 msgr2=0x5575013196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x5575013196a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 -- 172.21.15.116:0/3142606269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55750131bf30 msgr2=0x557501333ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55750131bf30 0x557501333ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.130+0000 7fdd08b5f640 1 -- 172.21.15.116:0/3142606269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557500f4a0f0 con 0x557501313350 2024-03-19T02:30:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.131+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000c00 tx=0x7fdcf400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2478cda25c25ff89 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.131+0000 7fdceeffd640 1 -- 172.21.15.116:0/3142606269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf402d070 con 0x557501313350 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.131+0000 7fdceeffd640 1 -- 172.21.15.116:0/3142606269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4005070 con 0x557501313350 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.131+0000 7fdceeffd640 1 -- 172.21.15.116:0/3142606269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40043a0 con 0x557501313350 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/3142606269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 msgr2=0x557501327ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000c00 tx=0x7fdcf400f790 comp rx=0 tx=0).stop 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/3142606269 shutdown_connections 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55750131bf30 0x557501333ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557501313350 0x557501327ac0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/3142606269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x5575013196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/3142606269 >> 172.21.15.116:0/3142606269 conn(0x55750130d160 msgr2=0x55750130e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/3142606269 shutdown_connections 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/3142606269 wait complete. 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 -- start start 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 Processor -- start 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.132+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x55750131fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x5575013200b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041aff80 con 0x55750131bf30 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041af8b0 con 0x557501313350 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd041acaf0 con 0x557501312a30 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 10 client.0 _ll_register_callbacks cb 0x7fdcf4009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x55750131fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56074/0 (socket says 172.21.15.116:56074) 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd08b5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x5575013200b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 -- 172.21.15.116:0/328694492 learned_addr learned my addr 172.21.15.116:0/328694492 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 -- 172.21.15.116:0/328694492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 msgr2=0x55750131fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x55750131fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 -- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 msgr2=0x5575013200b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x5575013200b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 -- 172.21.15.116:0/328694492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557501391680 con 0x55750131bf30 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd09b61640 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009ef0 tx=0x7fdcfc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9ea06271ef302b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdcecff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:30:57.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0042d0 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdcfc01f040 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.133+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf4005070 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdcfc040050 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 10 client.16704 fetch_fsmap learned FSMap version 30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557501392950 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4005070 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdcfc00f480 con 0x55750131bf30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 10 client.16704 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 10 client.16704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 20 client.16704 populate_metadata read hostname 'smithi116' 2024-03-19T02:30:57.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557501392e80 0x557501395270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 4 client.16704 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55750127dd90 con 0x557501392e80 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.134+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557501392e80 0x557501395270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdcecff9640 10 client.16704 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd08b5f640 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557501392e80 0x557501395270 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4bd64f9298fbe8 server_cookie=260d58d6d4521bd5 in_seq=0 out_seq=0 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdcecff9640 1 -- 172.21.15.116:0/328694492 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55750127dd90 con 0x557501392e80 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdcecff9640 10 client.16704 handle_command_reply: tid=0 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 client.16704 shutdown 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 msgr2=0x5575013205f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009ef0 tx=0x7fdcfc0047b0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 shutdown_connections 2024-03-19T02:30:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557501392e80 0x557501395270 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdcecff9640 0 client.16704 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55750131bf30 0x5575013205f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557501313350 0x5575013200b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 --2- 172.21.15.116:0/328694492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557501312a30 0x55750131fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 >> 172.21.15.116:0/328694492 conn(0x55750130d160 msgr2=0x55750130e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 shutdown_connections 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 1 -- 172.21.15.116:0/328694492 wait complete. 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.135+0000 7fdd0e1ab740 20 client.16704 trim_cache size 0 max 16384 2024-03-19T02:30:57.144 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "c737b807-1882-4f1c-959b-5298c02c515f": { 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "c737b807-1882-4f1c-959b-5298c02c515f", 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:30:57.145 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 msgr2=0x7fdcf00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 0x7fdcf00a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd000ede90 tx=0x7fdd000f6550 comp rx=0 tx=0).stop 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 msgr2=0x7fdd04198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdd00037040 tx=0x7fdd00004040 comp rx=0 tx=0).stop 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 shutdown_connections 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcf00a2970 0x7fdcf00a4e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd041066a0 0x7fdd041a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd04101050 0x7fdd04199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 --2- 172.21.15.116:0/2755275698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd04100690 0x7fdd04198de0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:30:57.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 >> 172.21.15.116:0/2755275698 conn(0x7fdd04077820 msgr2=0x7fdd04102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:30:57.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 shutdown_connections 2024-03-19T02:30:57.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:30:57.136+0000 7fdd0b5c4640 1 -- 172.21.15.116:0/2755275698 wait complete. 2024-03-19T02:30:57.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:56 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:30:57.148 INFO:tasks.cephfs.filesystem:scrub status for tag:c737b807-1882-4f1c-959b-5298c02c515f - {'path': '/', 'tag': 'c737b807-1882-4f1c-959b-5298c02c515f', 'options': 'recursive,force'} 2024-03-19T02:30:57.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:56 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:30:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:56 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:30:57.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 85656 3.27 MB/sec execute 20 sec latency 17.611 ms 2024-03-19T02:30:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: pgmap v97: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:30:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:30:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:30:58.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:57 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: pgmap v97: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:30:58.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:30:58.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:57 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:30:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: pgmap v97: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:30:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:30:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:30:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:57 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:30:58.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 86233 3.21 MB/sec execute 21 sec latency 28.771 ms 2024-03-19T02:30:59.781 INFO:tasks.workunit.client.0.smithi116.stdout: 1 86897 3.22 MB/sec execute 22 sec latency 15.397 ms 2024-03-19T02:31:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:30:59 smithi162 ceph-mon[35761]: pgmap v98: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:31:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:30:59 smithi154 ceph-mon[35693]: pgmap v98: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:31:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:30:59 smithi116 ceph-mon[32263]: pgmap v98: 129 pgs: 129 active+clean; 72 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-03-19T02:31:00.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 87566 3.15 MB/sec execute 23 sec latency 17.203 ms 2024-03-19T02:31:01.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 88458 3.33 MB/sec execute 24 sec latency 16.958 ms 2024-03-19T02:31:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:01 smithi162 ceph-mon[35761]: pgmap v99: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:31:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:01 smithi154 ceph-mon[35693]: pgmap v99: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:31:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:01 smithi116 ceph-mon[32263]: pgmap v99: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-19T02:31:02.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 88863 3.33 MB/sec execute 25 sec latency 17.109 ms 2024-03-19T02:31:03.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 89328 3.25 MB/sec execute 26 sec latency 16.126 ms 2024-03-19T02:31:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:03 smithi162 ceph-mon[35761]: pgmap v100: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:03 smithi154 ceph-mon[35693]: pgmap v100: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:03 smithi116 ceph-mon[32263]: pgmap v100: 129 pgs: 129 active+clean; 77 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:04.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 89902 3.21 MB/sec execute 27 sec latency 31.986 ms 2024-03-19T02:31:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:04 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:31:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:04 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:31:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:04 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:31:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 90600 3.21 MB/sec execute 28 sec latency 20.813 ms 2024-03-19T02:31:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:31:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:05 smithi162 ceph-mon[35761]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-03-19T02:31:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:31:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:05 smithi154 ceph-mon[35693]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-03-19T02:31:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:31:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:05 smithi116 ceph-mon[32263]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 2.1 MiB/s wr, 141 op/s 2024-03-19T02:31:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 91227 3.19 MB/sec execute 29 sec latency 16.349 ms 2024-03-19T02:31:07.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:06 smithi162 ceph-mon[35761]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:06 smithi154 ceph-mon[35693]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:06 smithi116 ceph-mon[32263]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:07.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 92032 3.30 MB/sec execute 30 sec latency 18.816 ms 2024-03-19T02:31:08.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 92502 3.34 MB/sec execute 31 sec latency 18.486 ms 2024-03-19T02:31:09.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 92891 3.24 MB/sec execute 32 sec latency 41.508 ms 2024-03-19T02:31:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:09 smithi162 ceph-mon[35761]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:09 smithi154 ceph-mon[35693]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:09 smithi116 ceph-mon[32263]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.5 MiB/s wr, 98 op/s 2024-03-19T02:31:10.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 93480 3.21 MB/sec execute 33 sec latency 17.134 ms 2024-03-19T02:31:11.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 94173 3.21 MB/sec execute 34 sec latency 25.569 ms 2024-03-19T02:31:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:11 smithi162 ceph-mon[35761]: pgmap v104: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:31:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:11 smithi154 ceph-mon[35693]: pgmap v104: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:31:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:11 smithi116 ceph-mon[32263]: pgmap v104: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.2 MiB/s wr, 138 op/s 2024-03-19T02:31:12.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 94772 3.19 MB/sec execute 35 sec latency 19.454 ms 2024-03-19T02:31:13.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 95580 3.28 MB/sec execute 36 sec latency 21.221 ms 2024-03-19T02:31:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:13 smithi162 ceph-mon[35761]: pgmap v105: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.5 MiB/s wr, 97 op/s 2024-03-19T02:31:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:13 smithi154 ceph-mon[35693]: pgmap v105: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.5 MiB/s wr, 97 op/s 2024-03-19T02:31:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:13 smithi116 ceph-mon[32263]: pgmap v105: 129 pgs: 129 active+clean; 82 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.5 MiB/s wr, 97 op/s 2024-03-19T02:31:14.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 96093 3.31 MB/sec execute 37 sec latency 25.259 ms 2024-03-19T02:31:15.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 96509 3.24 MB/sec execute 38 sec latency 31.769 ms 2024-03-19T02:31:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:15 smithi162 ceph-mon[35761]: pgmap v106: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 2.3 MiB/s wr, 141 op/s 2024-03-19T02:31:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:15 smithi154 ceph-mon[35693]: pgmap v106: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 2.3 MiB/s wr, 141 op/s 2024-03-19T02:31:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:15 smithi116 ceph-mon[32263]: pgmap v106: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 2.3 MiB/s wr, 141 op/s 2024-03-19T02:31:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 97096 3.21 MB/sec execute 39 sec latency 38.126 ms 2024-03-19T02:31:17.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 97742 3.21 MB/sec execute 40 sec latency 17.768 ms 2024-03-19T02:31:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:17 smithi162 ceph-mon[35761]: pgmap v107: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:17 smithi154 ceph-mon[35693]: pgmap v107: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:17 smithi116 ceph-mon[32263]: pgmap v107: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:18.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 98397 3.19 MB/sec execute 41 sec latency 16.106 ms 2024-03-19T02:31:19.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 99145 3.27 MB/sec execute 42 sec latency 16.170 ms 2024-03-19T02:31:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:19 smithi162 ceph-mon[35761]: pgmap v108: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:19 smithi154 ceph-mon[35693]: pgmap v108: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:19 smithi116 ceph-mon[32263]: pgmap v108: 129 pgs: 129 active+clean; 84 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.4 MiB/s wr, 83 op/s 2024-03-19T02:31:20.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 99719 3.30 MB/sec execute 43 sec latency 16.988 ms 2024-03-19T02:31:21.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 100188 3.24 MB/sec execute 44 sec latency 36.680 ms 2024-03-19T02:31:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:21 smithi162 ceph-mon[35761]: pgmap v109: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:21 smithi154 ceph-mon[35693]: pgmap v109: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:21 smithi116 ceph-mon[32263]: pgmap v109: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:22.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 100834 3.27 MB/sec execute 45 sec latency 20.193 ms 2024-03-19T02:31:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 101452 3.21 MB/sec execute 46 sec latency 16.882 ms 2024-03-19T02:31:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:23 smithi162 ceph-mon[35761]: pgmap v110: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:23 smithi154 ceph-mon[35693]: pgmap v110: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:23 smithi116 ceph-mon[32263]: pgmap v110: 129 pgs: 129 active+clean; 86 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:24.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 102117 3.26 MB/sec execute 47 sec latency 16.778 ms 2024-03-19T02:31:25.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 102684 3.26 MB/sec execute 48 sec latency 48.242 ms 2024-03-19T02:31:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:25 smithi162 ceph-mon[35761]: pgmap v111: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-19T02:31:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:25 smithi154 ceph-mon[35693]: pgmap v111: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-19T02:31:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:25 smithi116 ceph-mon[32263]: pgmap v111: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-19T02:31:26.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 103251 3.29 MB/sec execute 49 sec latency 16.084 ms 2024-03-19T02:31:27.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:31:27.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:31:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 -- 172.21.15.116:0/3243815018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0106660 msgr2=0x7febf0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 --2- 172.21.15.116:0/3243815018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0106660 0x7febf0106a40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7febe0009900 tx=0x7febe002bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 -- 172.21.15.116:0/3243815018 shutdown_connections 2024-03-19T02:31:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 --2- 172.21.15.116:0/3243815018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf0100f60 0x7febf010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 --2- 172.21.15.116:0/3243815018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf01005a0 0x7febf0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 --2- 172.21.15.116:0/3243815018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0106660 0x7febf0106a40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 -- 172.21.15.116:0/3243815018 >> 172.21.15.116:0/3243815018 conn(0x7febf00fc070 msgr2=0x7febf00fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 -- 172.21.15.116:0/3243815018 shutdown_connections 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 -- 172.21.15.116:0/3243815018 wait complete. 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.260+0000 7febf579d640 1 Processor -- start 2024-03-19T02:31:27.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 -- start start 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 0x7febf0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 0x7febf0199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf0106660 0x7febf01a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febf0110850 con 0x7febf0100f60 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febf01106d0 con 0x7febf0106660 2024-03-19T02:31:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf579d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febf01109d0 con 0x7febf01005a0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 0x7febf0199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 0x7febf0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 0x7febf0199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50614/0 (socket says 172.21.15.116:50614) 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febef7fe640 1 -- 172.21.15.116:0/3184498512 learned_addr learned my addr 172.21.15.116:0/3184498512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febf4f9c640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf0106660 0x7febf01a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 -- 172.21.15.116:0/3184498512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf0106660 msgr2=0x7febf01a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf0106660 0x7febf01a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 -- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 msgr2=0x7febf0199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 0x7febf0199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.261+0000 7febeffff640 1 -- 172.21.15.116:0/3184498512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf01a41d0 con 0x7febf01005a0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febeffff640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 0x7febf0198c10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7febe0036040 tx=0x7febe002cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febed7fa640 1 -- 172.21.15.116:0/3184498512 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0043020 con 0x7febf01005a0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf01a43a0 con 0x7febf01005a0 2024-03-19T02:31:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febed7fa640 1 -- 172.21.15.116:0/3184498512 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe0004410 con 0x7febf01005a0 2024-03-19T02:31:27.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf01a48e0 con 0x7febf01005a0 2024-03-19T02:31:27.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.262+0000 7febed7fa640 1 -- 172.21.15.116:0/3184498512 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0003b20 con 0x7febf01005a0 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.264+0000 7febed7fa640 1 -- 172.21.15.116:0/3184498512 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7febe0046020 con 0x7febf01005a0 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.264+0000 7febf95ee740 1 Processor -- start 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.264+0000 7febed7fa640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 0x7febd80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 -- start start 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf4f9c640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 0x7febd80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febed7fa640 1 -- 172.21.15.116:0/3184498512 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febe00f0d10 con 0x7febf01005a0 2024-03-19T02:31:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x56146920dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x5614692104e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561469168500 con 0x56146920e0f0 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561469110be0 con 0x561469204bf0 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf95ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5614691645a0 con 0x561469205510 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47560/0 (socket says 172.21.15.116:47560) 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x56146920dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 -- 172.21.15.116:0/2757441652 learned_addr learned my addr 172.21.15.116:0/2757441652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf4f9c640 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x5614692104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 -- 172.21.15.116:0/2757441652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 msgr2=0x56146920dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x56146920dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 -- 172.21.15.116:0/2757441652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 msgr2=0x5614692104e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x5614692104e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 -- 172.21.15.116:0/2757441652 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561469132000 con 0x561469204bf0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febf4f9c640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 0x7febd80a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febe8005fd0 tx=0x7febe8011b40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.265+0000 7febeffff640 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7febe00b9560 tx=0x7febe00035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19db66f4bf1b7dff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febd67fc640 1 -- 172.21.15.116:0/2757441652 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe00bd800 con 0x561469204bf0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febd67fc640 1 -- 172.21.15.116:0/2757441652 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe00bd100 con 0x561469204bf0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febd67fc640 1 -- 172.21.15.116:0/2757441652 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe00ee4f0 con 0x561469204bf0 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 -- 172.21.15.116:0/2757441652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 msgr2=0x561469204fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7febe00b9560 tx=0x7febe00035f0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 -- 172.21.15.116:0/2757441652 shutdown_connections 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x5614692104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x56146920dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 --2- 172.21.15.116:0/2757441652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469204fd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 -- 172.21.15.116:0/2757441652 >> 172.21.15.116:0/2757441652 conn(0x5614691ff320 msgr2=0x5614691ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 -- 172.21.15.116:0/2757441652 shutdown_connections 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.266+0000 7febf95ee740 1 -- 172.21.15.116:0/2757441652 wait complete. 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 -- start start 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 Processor -- start 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469205510 0x56146920b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56146920e0f0 0x56146920bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561469110be0 con 0x561469204bf0 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561469124160 con 0x561469205510 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561469210ca0 con 0x56146920e0f0 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf95ee740 10 client.0 _ll_register_callbacks cb 0x561469160a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50638/0 (socket says 172.21.15.116:50638) 2024-03-19T02:31:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febeffff640 1 -- 172.21.15.116:0/2920218611 learned_addr learned my addr 172.21.15.116:0/2920218611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febef7fe640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469205510 0x56146920b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.267+0000 7febf4f9c640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56146920e0f0 0x56146920bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 -- 172.21.15.116:0/2920218611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56146920e0f0 msgr2=0x56146920bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56146920e0f0 0x56146920bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 -- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469205510 msgr2=0x56146920b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469205510 0x56146920b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 -- 172.21.15.116:0/2920218611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561468f467d0 con 0x561469204bf0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febeffff640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7febe00ee2a0 tx=0x7febe00ee3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608ae9877da9414b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe00c58a0 con 0x561469204bf0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febe00c5410 con 0x561469204bf0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe00bd100 con 0x561469204bf0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7febe00bd800 con 0x561469204bf0 2024-03-19T02:31:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febf95ee740 10 client.16716 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561468e7c3f0 con 0x561469204bf0 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe00bd100 con 0x561469204bf0 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.268+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7febe0050060 con 0x561469204bf0 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 10 client.16716 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 10 client.16716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 20 client.16716 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56146921df00 0x5614692202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 4 client.16716 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561469175a40 con 0x56146921df00 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febef7fe640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56146921df00 0x5614692202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febef7fe640 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56146921df00 0x5614692202f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d342959a23d075b4 server_cookie=a7ce35bf871f412c in_seq=0 out_seq=0 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.269+0000 7febcf7fe640 10 client.16716 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febcf7fe640 1 -- 172.21.15.116:0/2920218611 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561469175a40 con 0x56146921df00 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febcf7fe640 10 client.16716 handle_command_reply: tid=0 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 client.16716 shutdown 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 msgr2=0x56146920b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7febe00ee2a0 tx=0x7febe00ee3f0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 shutdown_connections 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56146921df00 0x5614692202f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56146920e0f0 0x56146920bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469205510 0x56146920b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 --2- 172.21.15.116:0/2920218611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920b170 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 >> 172.21.15.116:0/2920218611 conn(0x5614691ff320 msgr2=0x561469201070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.270+0000 7febcf7fe640 0 client.16716 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.271+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 shutdown_connections 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.271+0000 7febf95ee740 1 -- 172.21.15.116:0/2920218611 wait complete. 2024-03-19T02:31:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.271+0000 7febf95ee740 20 client.16716 trim_cache size 0 max 16384 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 Processor -- start 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 -- start start 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x561469219810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56146920e0f0 0x561469225aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febf01ae940 con 0x561469204bf0 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febf01b0480 con 0x56146920e0f0 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf95ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febf01af010 con 0x561469205510 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50650/0 (socket says 172.21.15.116:50650) 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febeffff640 1 -- 172.21.15.116:0/333611909 learned_addr learned my addr 172.21.15.116:0/333611909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x561469219810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febf4f9c640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56146920e0f0 0x561469225aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 -- 172.21.15.116:0/333611909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56146920e0f0 msgr2=0x561469225aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56146920e0f0 0x561469225aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 -- 172.21.15.116:0/333611909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 msgr2=0x56146920d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febef7fe640 1 -- 172.21.15.116:0/333611909 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561468e3c0f0 con 0x561469205510 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.289+0000 7febeffff640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.290+0000 7febef7fe640 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x561469219810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x561469220a40 tx=0x7febe4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36a2b22f0c6ca77 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.290+0000 7febd5ffb640 1 -- 172.21.15.116:0/333611909 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4027070 con 0x561469205510 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.290+0000 7febd5ffb640 1 -- 172.21.15.116:0/333611909 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe40221b0 con 0x561469205510 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.290+0000 7febd5ffb640 1 -- 172.21.15.116:0/333611909 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4003850 con 0x561469205510 2024-03-19T02:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- 172.21.15.116:0/333611909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 msgr2=0x561469219810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x561469219810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x561469220a40 tx=0x7febe4005f90 comp rx=0 tx=0).stop 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- 172.21.15.116:0/333611909 shutdown_connections 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56146920e0f0 0x561469225aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x561469219810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- 172.21.15.116:0/333611909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561469204bf0 0x56146920d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- 172.21.15.116:0/333611909 >> 172.21.15.116:0/333611909 conn(0x5614691ff320 msgr2=0x5614692006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- 172.21.15.116:0/333611909 shutdown_connections 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- 172.21.15.116:0/333611909 wait complete. 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- start start 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 Processor -- start 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x5614692080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x561469208610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febf01afdb0 con 0x56146920e0f0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.291+0000 7febf95ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febf01af6e0 con 0x561469204bf0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf95ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febf01ac920 con 0x561469205510 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf95ee740 10 client.0 _ll_register_callbacks cb 0x7febe00d4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47586/0 (socket says 172.21.15.116:47586) 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 -- 172.21.15.116:0/1713815700 learned_addr learned my addr 172.21.15.116:0/1713815700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febef7fe640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x5614692080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf4f9c640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x561469208610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 -- 172.21.15.116:0/1713815700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 msgr2=0x5614692080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x5614692080d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 -- 172.21.15.116:0/1713815700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 msgr2=0x561469208610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x561469208610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 -- 172.21.15.116:0/1713815700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614692830d0 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febeffff640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7febe002c170 tx=0x7febe0005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557b85ffeb481a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe00bced0 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7febe00b9170 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe40221b0 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7febe00bca60 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf95ee740 10 client.26540 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.292+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5614692830d0 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febe40221b0 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7febe00c5410 con 0x561469204bf0 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 10 client.26540 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 10 client.26540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 20 client.26540 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5614692845c0 0x5614692869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 4 client.26540 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56146916fd90 con 0x5614692845c0 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febef7fe640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5614692845c0 0x5614692869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febef7fe640 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5614692845c0 0x5614692869b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ec53031acc37a2 server_cookie=c776e06efd390a2d in_seq=0 out_seq=0 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.293+0000 7febcf7fe640 10 client.26540 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febcf7fe640 1 -- 172.21.15.116:0/1713815700 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56146916fd90 con 0x5614692845c0 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febcf7fe640 10 client.26540 handle_command_reply: tid=0 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 client.26540 shutdown 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 msgr2=0x561469207b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7febe002c170 tx=0x7febe0005c90 comp rx=0 tx=0).stop 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 shutdown_connections 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5614692845c0 0x5614692869b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56146920e0f0 0x561469208610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561469205510 0x5614692080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febcf7fe640 0 client.26540 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 --2- 172.21.15.116:0/1713815700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561469204bf0 0x561469207b90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 >> 172.21.15.116:0/1713815700 conn(0x5614691ff320 msgr2=0x5614691fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 shutdown_connections 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 1 -- 172.21.15.116:0/1713815700 wait complete. 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.294+0000 7febf95ee740 20 client.26540 trim_cache size 0 max 16384 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:31:27.301 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:31:27.302 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:31:27.302 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:31:27.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 msgr2=0x7febd80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 0x7febd80a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febe8005fd0 tx=0x7febe8011b40 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 msgr2=0x7febf0198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 0x7febf0198c10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7febe0036040 tx=0x7febe002cd20 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 shutdown_connections 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7febd80a2970 0x7febd80a4e30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febf0106660 0x7febf01a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febf0100f60 0x7febf0199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 --2- 172.21.15.116:0/3184498512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febf01005a0 0x7febf0198c10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 >> 172.21.15.116:0/3184498512 conn(0x7febf00fc070 msgr2=0x7febf00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 shutdown_connections 2024-03-19T02:31:27.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.295+0000 7febf579d640 1 -- 172.21.15.116:0/3184498512 wait complete. 2024-03-19T02:31:27.306 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:31:27.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:31:27.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 -- 172.21.15.116:0/40991558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d4102a90 msgr2=0x7f89d4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 --2- 172.21.15.116:0/40991558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d4102a90 0x7f89d4102e70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009900 tx=0x7f89c402bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 -- 172.21.15.116:0/40991558 shutdown_connections 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 --2- 172.21.15.116:0/40991558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d410be60 0x7f89d410e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 --2- 172.21.15.116:0/40991558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d41033b0 0x7f89d410b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 --2- 172.21.15.116:0/40991558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d4102a90 0x7f89d4102e70 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 -- 172.21.15.116:0/40991558 >> 172.21.15.116:0/40991558 conn(0x7f89d4077820 msgr2=0x7f89d4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 -- 172.21.15.116:0/40991558 shutdown_connections 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.413+0000 7f89d8b90640 1 -- 172.21.15.116:0/40991558 wait complete. 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 Processor -- start 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 -- start start 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d4102a90 0x7f89d4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d410be60 0x7f89d419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4111b10 con 0x7f89d41033b0 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4111990 con 0x7f89d410be60 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89d8b90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4111c90 con 0x7f89d4102a90 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50670/0 (socket says 172.21.15.116:50670) 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.414+0000 7f89cbfff640 1 -- 172.21.15.116:0/3938083806 learned_addr learned my addr 172.21.15.116:0/3938083806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89d259c640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d4102a90 0x7f89d4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89d2d9d640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d410be60 0x7f89d419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 -- 172.21.15.116:0/3938083806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d4102a90 msgr2=0x7f89d4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d4102a90 0x7f89d4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 -- 172.21.15.116:0/3938083806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d410be60 msgr2=0x7f89d419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d410be60 0x7f89d419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 -- 172.21.15.116:0/3938083806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d419d000 con 0x7f89d41033b0 2024-03-19T02:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cbfff640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e9f0 tx=0x7f89bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc00cd80 con 0x7f89d41033b0 2024-03-19T02:31:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d419d2f0 con 0x7f89d41033b0 2024-03-19T02:31:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89bc0108e0 con 0x7f89d41033b0 2024-03-19T02:31:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc018920 con 0x7f89d41033b0 2024-03-19T02:31:27.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.415+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d41a4b80 con 0x7f89d41033b0 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.416+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4068610 con 0x7f89d41033b0 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.417+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f89bc010a90 con 0x7f89d41033b0 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.418+0000 7f89cb7fe640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 0x7f89c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.418+0000 7f89d259c640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 0x7f89c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.418+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f89bc0cf460 con 0x7f89d41033b0 2024-03-19T02:31:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.418+0000 7f89d259c640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 0x7f89c00a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006010 tx=0x7f89c4005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.422+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89bc09b890 con 0x7f89d41033b0 2024-03-19T02:31:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.661+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89d4106f50 con 0x7f89d41033b0 2024-03-19T02:31:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.662+0000 7f89cb7fe640 1 -- 172.21.15.116:0/3938083806 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f89bc09f930 con 0x7f89d41033b0 2024-03-19T02:31:27.663 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:31:27.663 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:31:27.663 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:31:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 msgr2=0x7f89c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 0x7f89c00a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006010 tx=0x7f89c4005e40 comp rx=0 tx=0).stop 2024-03-19T02:31:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 msgr2=0x7f89d4198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e9f0 tx=0x7f89bc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 shutdown_connections 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f89c00a2970 0x7f89c00a4e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89d410be60 0x7f89d419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d41033b0 0x7f89d4198550 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 --2- 172.21.15.116:0/3938083806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89d4102a90 0x7f89d4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 >> 172.21.15.116:0/3938083806 conn(0x7f89d4077820 msgr2=0x7f89d40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 shutdown_connections 2024-03-19T02:31:27.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.664+0000 7f89d8b90640 1 -- 172.21.15.116:0/3938083806 wait complete. 2024-03-19T02:31:27.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:31:27.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 103651 3.23 MB/sec execute 50 sec latency 60.836 ms 2024-03-19T02:31:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 -- 172.21.15.116:0/2080709547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01017a0 msgr2=0x7f8ed0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2080709547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01017a0 0x7f8ed0101c20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009930 tx=0x7f8ec402bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 -- 172.21.15.116:0/2080709547 shutdown_connections 2024-03-19T02:31:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2080709547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed0102160 0x7f8ed010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2080709547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01017a0 0x7f8ed0101c20 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2080709547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 -- 172.21.15.116:0/2080709547 >> 172.21.15.116:0/2080709547 conn(0x7f8ed00fd2d0 msgr2=0x7f8ed00ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 -- 172.21.15.116:0/2080709547 shutdown_connections 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.790+0000 7f8ed5412640 1 -- 172.21.15.116:0/2080709547 wait complete. 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 Processor -- start 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 -- start start 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 0x7f8ed0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed0111a70 con 0x7f8ed0102160 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01118f0 con 0x7f8ed01077b0 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ed5412640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed0111bf0 con 0x7f8ed01017a0 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8eceffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 0x7f8ed0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39272/0 (socket says 172.21.15.116:39272) 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ece7fc640 1 -- 172.21.15.116:0/2408584805 learned_addr learned my addr 172.21.15.116:0/2408584805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.791+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 -- 172.21.15.116:0/2408584805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 msgr2=0x7f8ed019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 -- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 msgr2=0x7f8ed01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 -- 172.21.15.116:0/2408584805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed019cfa0 con 0x7f8ed01017a0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 0x7f8ed0197fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e9f0 tx=0x7f8ec000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eb7fff640 1 -- 172.21.15.116:0/2408584805 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000cd80 con 0x7f8ed01017a0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed019d290 con 0x7f8ed01017a0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eb7fff640 1 -- 172.21.15.116:0/2408584805 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ec00035f0 con 0x7f8ed01017a0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed01a4b20 con 0x7f8ed01017a0 2024-03-19T02:31:27.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.792+0000 7f8eb7fff640 1 -- 172.21.15.116:0/2408584805 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0010690 con 0x7f8ed01017a0 2024-03-19T02:31:27.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.794+0000 7f8ed8002740 1 Processor -- start 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.794+0000 7f8eb7fff640 1 -- 172.21.15.116:0/2408584805 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8ec00037a0 con 0x7f8ed01017a0 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.794+0000 7f8ed8002740 1 -- start start 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.794+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ecbf0 0x55a1384f5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384f57d0 0x55a1384f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ed8002740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a13844f4b0 con 0x55a1384ecbf0 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ed8002740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a1383f7d70 con 0x55a1384f57d0 2024-03-19T02:31:27.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ed8002740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a13844b550 con 0x55a1384ec2d0 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8eceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ecbf0 0x55a1384f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38726/0 (socket says 172.21.15.116:38726) 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/2231958094 learned_addr learned my addr 172.21.15.116:0/2231958094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384f57d0 0x55a1384f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/2231958094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384f57d0 msgr2=0x55a1384f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384f57d0 0x55a1384f7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8eb7fff640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 0x7f8eb80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/2231958094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ecbf0 msgr2=0x55a1384f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ecbf0 0x55a1384f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/2231958094 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a13841a210 con 0x55a1384ec2d0 2024-03-19T02:31:27.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8eb7fff640 1 -- 172.21.15.116:0/2408584805 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8ec00cf3e0 con 0x7f8ed01017a0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 0x7f8eb80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80117e0 tx=0x7f8ec8011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96d12e80a378e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.795+0000 7f8eb57fa640 1 -- 172.21.15.116:0/2231958094 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800fb00 con 0x55a1384ec2d0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8eb57fa640 1 -- 172.21.15.116:0/2231958094 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ec8010e00 con 0x55a1384ec2d0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8eb57fa640 1 -- 172.21.15.116:0/2231958094 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec801ca80 con 0x55a1384ec2d0 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 -- 172.21.15.116:0/2231958094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 msgr2=0x55a1384ec6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80117e0 tx=0x7f8ec8011ca0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 0x7f8eb80a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8011490 tx=0x7f8ec8021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 -- 172.21.15.116:0/2231958094 shutdown_connections 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384f57d0 0x55a1384f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ecbf0 0x55a1384f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2231958094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a1384ec6b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 -- 172.21.15.116:0/2231958094 >> 172.21.15.116:0/2231958094 conn(0x55a1384e6a00 msgr2=0x55a1384e6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 -- 172.21.15.116:0/2231958094 shutdown_connections 2024-03-19T02:31:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.796+0000 7f8ed8002740 1 -- 172.21.15.116:0/2231958094 wait complete. 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 -- start start 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 Processor -- start 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 0x55a1384f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 0x55a1384f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a1384f85c0 con 0x55a1384f57d0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a13840b310 con 0x55a1384ec2d0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a1384f8740 con 0x55a1384ecbf0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ed8002740 10 client.0 _ll_register_callbacks cb 0x55a1383ff2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8eceffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 0x55a1384f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 0x55a1384f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39288/0 (socket says 172.21.15.116:39288) 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/3456876669 learned_addr learned my addr 172.21.15.116:0/3456876669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 0x55a1384f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51016/0 (socket says 172.21.15.116:51016) 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 -- 172.21.15.116:0/3456876669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 msgr2=0x55a1384f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 0x55a1384f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 -- 172.21.15.116:0/3456876669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 msgr2=0x55a1384f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ece7fc640 1 -- 172.21.15.116:0/3456876669 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1381633f0 con 0x55a1384ecbf0 2024-03-19T02:31:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.797+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 0x55a1384f2d30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009900 tx=0x7f8ec402ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b0bb515ef0e0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec402c5d0 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8ec40035f0 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ec8010e00 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ec40037f0 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 10 client.26497 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a13822d7d0 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ec8010e00 con 0x55a1384ecbf0 2024-03-19T02:31:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8ec4034b00 con 0x55a1384ecbf0 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 10 client.26497 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 10 client.26497 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 20 client.26497 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a138505b30 0x55a138507f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 4 client.26497 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.798+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1384ad3e0 con 0x55a138505b30 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.799+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a138505b30 0x55a138507f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.799+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a138505b30 0x55a138507f60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3a1573704446aee6 server_cookie=7b229731cb601101 in_seq=0 out_seq=0 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.799+0000 7f8eb2ff5640 10 client.26497 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3456876669 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a1384ad3e0 con 0x55a138505b30 2024-03-19T02:31:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8eb2ff5640 10 client.26497 handle_command_reply: tid=0 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 client.26497 shutdown 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 msgr2=0x55a1384f2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 0x55a1384f2d30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009900 tx=0x7f8ec402ca10 comp rx=0 tx=0).stop 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 shutdown_connections 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a138505b30 0x55a138507f60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ecbf0 0x55a1384f2d30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3456876669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ec2d0 0x55a1384f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 >> 172.21.15.116:0/3456876669 conn(0x55a1384e6a00 msgr2=0x55a1384e8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8eb2ff5640 0 client.26497 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 shutdown_connections 2024-03-19T02:31:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 1 -- 172.21.15.116:0/3456876669 wait complete. 2024-03-19T02:31:27.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.800+0000 7f8ed8002740 20 client.26497 trim_cache size 0 max 16384 2024-03-19T02:31:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.815+0000 7f8ed8002740 1 Processor -- start 2024-03-19T02:31:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.815+0000 7f8ed8002740 1 -- start start 2024-03-19T02:31:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.815+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a138501360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384f57d0 0x55a1384f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ed8002740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01acaa0 con 0x55a1384ec2d0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ed8002740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01ae5e0 con 0x55a1384ecbf0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ed8002740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01ad170 con 0x55a1384f57d0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39298/0 (socket says 172.21.15.116:39298) 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 -- 172.21.15.116:0/2485222924 learned_addr learned my addr 172.21.15.116:0/2485222924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384f57d0 0x55a1384f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a138501360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 -- 172.21.15.116:0/2485222924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384f57d0 msgr2=0x55a1384f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384f57d0 0x55a1384f17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 -- 172.21.15.116:0/2485222924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 msgr2=0x55a138501360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a138501360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 -- 172.21.15.116:0/2485222924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1384e6830 con 0x55a1384ec2d0 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eceffd640 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8ec009b3c0 tx=0x7f8ec0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d071a52ee86432e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eb4ff9640 1 -- 172.21.15.116:0/2485222924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec009b020 con 0x55a1384ec2d0 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eb4ff9640 1 -- 172.21.15.116:0/2485222924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ec00860b0 con 0x55a1384ec2d0 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.816+0000 7f8eb4ff9640 1 -- 172.21.15.116:0/2485222924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00a3e60 con 0x55a1384ec2d0 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- 172.21.15.116:0/2485222924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 msgr2=0x55a1384f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8ec009b3c0 tx=0x7f8ec0086040 comp rx=0 tx=0).stop 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- 172.21.15.116:0/2485222924 shutdown_connections 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384f57d0 0x55a1384f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a138501360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 --2- 172.21.15.116:0/2485222924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384ec2d0 0x55a1384f4f20 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- 172.21.15.116:0/2485222924 >> 172.21.15.116:0/2485222924 conn(0x55a1384e6a00 msgr2=0x55a1384f10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- 172.21.15.116:0/2485222924 shutdown_connections 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- 172.21.15.116:0/2485222924 wait complete. 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 -- start start 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 Processor -- start 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.817+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a138564160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a1385646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01adf10 con 0x55a1384f57d0 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01ad840 con 0x55a1384ecbf0 2024-03-19T02:31:27.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01aaa80 con 0x55a1384ec2d0 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 10 client.0 _ll_register_callbacks cb 0x7f8ec4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39310/0 (socket says 172.21.15.116:39310) 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/3611789716 learned_addr learned my addr 172.21.15.116:0/3611789716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a138564160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ece7fc640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a1385646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 msgr2=0x55a138564160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a138564160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/3611789716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 msgr2=0x55a1385646a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a1385646a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 -- 172.21.15.116:0/3611789716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1384f9140 con 0x55a1384f57d0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a138564160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ecf7fe640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800a010 tx=0x7f8ec800fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1293087c9f4f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8004bf0 con 0x55a1384f57d0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8ec8022040 con 0x55a1384f57d0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ec00860b0 con 0x55a1384f57d0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8ec8045050 con 0x55a1384f57d0 2024-03-19T02:31:27.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 10 client.16746 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.818+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a13856b900 con 0x55a1384f57d0 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ec00860b0 con 0x55a1384f57d0 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8ec80041e0 con 0x55a1384f57d0 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 10 client.16746 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 10 client.16746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 20 client.16746 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a13856bed0 0x55a13856e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 4 client.16746 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a13845c9f0 con 0x55a13856bed0 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a13856bed0 0x55a13856e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8eceffd640 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a13856bed0 0x55a13856e2f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9c96142a0fb5fff3 server_cookie=664e04407ebbed54 in_seq=0 out_seq=0 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.819+0000 7f8eb2ff5640 10 client.16746 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8eb2ff5640 1 -- 172.21.15.116:0/3611789716 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a13845c9f0 con 0x55a13856bed0 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8eb2ff5640 10 client.16746 handle_command_reply: tid=0 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 client.16746 shutdown 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 msgr2=0x55a1384f8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800a010 tx=0x7f8ec800fbd0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 shutdown_connections 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a13856bed0 0x55a13856e2f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a1384f57d0 0x55a1384f8a50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8eb2ff5640 0 client.16746 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1384ecbf0 0x55a1385646a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 --2- 172.21.15.116:0/3611789716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a1384ec2d0 0x55a138564160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 >> 172.21.15.116:0/3611789716 conn(0x55a1384e6a00 msgr2=0x55a1384f2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 shutdown_connections 2024-03-19T02:31:27.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 1 -- 172.21.15.116:0/3611789716 wait complete. 2024-03-19T02:31:27.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed8002740 20 client.16746 trim_cache size 0 max 16384 2024-03-19T02:31:27.830 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:31:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 msgr2=0x7f8eb80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 0x7f8eb80a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8011490 tx=0x7f8ec8021000 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 msgr2=0x7f8ed0197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 0x7f8ed0197fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e9f0 tx=0x7f8ec000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 shutdown_connections 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8eb80a2970 0x7f8eb80a4e30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed01077b0 0x7f8ed019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed0102160 0x7f8ed01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 --2- 172.21.15.116:0/2408584805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ed01017a0 0x7f8ed0197fb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 >> 172.21.15.116:0/2408584805 conn(0x7f8ed00fd2d0 msgr2=0x7f8ed00fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.820+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 shutdown_connections 2024-03-19T02:31:27.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.821+0000 7f8ed5412640 1 -- 172.21.15.116:0/2408584805 wait complete. 2024-03-19T02:31:27.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 -- 172.21.15.116:0/3547407723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c1006a0 msgr2=0x7f431c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 --2- 172.21.15.116:0/3547407723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c1006a0 0x7f431c100b20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f430c009930 tx=0x7f430c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 -- 172.21.15.116:0/3547407723 shutdown_connections 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 --2- 172.21.15.116:0/3547407723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c101060 0x7f431c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 --2- 172.21.15.116:0/3547407723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c1006a0 0x7f431c100b20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 --2- 172.21.15.116:0/3547407723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 0x7f431c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 -- 172.21.15.116:0/3547407723 >> 172.21.15.116:0/3547407723 conn(0x7f431c0778d0 msgr2=0x7f431c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 -- 172.21.15.116:0/3547407723 shutdown_connections 2024-03-19T02:31:27.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.953+0000 7f43241b9640 1 -- 172.21.15.116:0/3547407723 wait complete. 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 Processor -- start 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 -- start start 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 0x7f431c196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 0x7f431c19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f431c110970 con 0x7f431c101060 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1107f0 con 0x7f431c1066b0 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f43241b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f431c110af0 con 0x7f431c1006a0 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321f55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 0x7f431c196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39330/0 (socket says 172.21.15.116:39330) 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 -- 172.21.15.116:0/2540270529 learned_addr learned my addr 172.21.15.116:0/2540270529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4322756640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 0x7f431c19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 -- 172.21.15.116:0/2540270529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 msgr2=0x7f431c196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 0x7f431c196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 -- 172.21.15.116:0/2540270529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 msgr2=0x7f431c19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 0x7f431c19b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321754640 1 -- 172.21.15.116:0/2540270529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c19bec0 con 0x7f431c101060 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.954+0000 7f4321f55640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 0x7f431c196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f4321754640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f430c009230 tx=0x7f430c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f4312ffd640 1 -- 172.21.15.116:0/2540270529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c02ca70 con 0x7f431c101060 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431c19c150 con 0x7f431c101060 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f4312ffd640 1 -- 172.21.15.116:0/2540270529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f430c02cc10 con 0x7f431c101060 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f4312ffd640 1 -- 172.21.15.116:0/2540270529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c003c60 con 0x7f431c101060 2024-03-19T02:31:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.955+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431c068980 con 0x7f431c101060 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.957+0000 7f4312ffd640 1 -- 172.21.15.116:0/2540270529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f430c004c50 con 0x7f431c101060 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.957+0000 7f4312ffd640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 0x7f43080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 Processor -- start 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 0x7f43080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4312ffd640 1 -- 172.21.15.116:0/2540270529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f430c0f2be0 con 0x7f431c101060 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 -- start start 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589dac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589dac7d0 0x55b589daebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b589d064b0 con 0x55b589dac7d0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b589caed70 con 0x55b589da32d0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4326da3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b589d02550 con 0x55b589da3bf0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 0x7f43080a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4318009f40 tx=0x7f4318005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51046/0 (socket says 172.21.15.116:51046) 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 -- 172.21.15.116:0/2852737061 learned_addr learned my addr 172.21.15.116:0/2852737061 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4322756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589dac7d0 0x55b589daebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321754640 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589dac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 -- 172.21.15.116:0/2852737061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 msgr2=0x55b589dac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589dac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 -- 172.21.15.116:0/2852737061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589dac7d0 msgr2=0x55b589daebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589dac7d0 0x55b589daebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.958+0000 7f4321f55640 1 -- 172.21.15.116:0/2852737061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b589cd1210 con 0x55b589da32d0 2024-03-19T02:31:27.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.959+0000 7f4321f55640 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4318004230 tx=0x7f4318014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aefbc43d6a34f2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.959+0000 7f4307fff640 1 -- 172.21.15.116:0/2852737061 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800f480 con 0x55b589da32d0 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.959+0000 7f4307fff640 1 -- 172.21.15.116:0/2852737061 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f431800fad0 con 0x55b589da32d0 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.959+0000 7f4307fff640 1 -- 172.21.15.116:0/2852737061 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431801ea70 con 0x55b589da32d0 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- 172.21.15.116:0/2852737061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 msgr2=0x55b589da36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4318004230 tx=0x7f4318014000 comp rx=0 tx=0).stop 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- 172.21.15.116:0/2852737061 shutdown_connections 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589dac7d0 0x55b589daebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589dac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 --2- 172.21.15.116:0/2852737061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589da32d0 0x55b589da36b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- 172.21.15.116:0/2852737061 >> 172.21.15.116:0/2852737061 conn(0x55b589d9da00 msgr2=0x55b589d9dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- 172.21.15.116:0/2852737061 shutdown_connections 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- 172.21.15.116:0/2852737061 wait complete. 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 -- start start 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.960+0000 7f4326da3740 1 Processor -- start 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589da9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589daa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b589daf5c0 con 0x55b589da32d0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b589cc2310 con 0x55b589dac7d0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b589daf740 con 0x55b589da3bf0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39350/0 (socket says 172.21.15.116:39350) 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 -- 172.21.15.116:0/4266505394 learned_addr learned my addr 172.21.15.116:0/4266505394 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4322756640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589daa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321754640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589da9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4326da3740 10 client.0 _ll_register_callbacks cb 0x55b589cb62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 -- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 msgr2=0x55b589da9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589da9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 -- 172.21.15.116:0/4266505394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 msgr2=0x55b589daa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589daa350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 -- 172.21.15.116:0/4266505394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b589a1a3f0 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f4321f55640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4318009700 tx=0x7f4318003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3aa8bf7d4ef753c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f43057fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800f690 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.961+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f4318009050 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f431800fad0 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4318038050 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 10 client.16758 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b589ae47d0 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f431800fad0 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f431801eb00 con 0x55b589da32d0 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 10 client.16758 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 10 client.16758 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 20 client.16758 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589dbcbe0 0x55b589dbefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 4 client.16758 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:27.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b589d643e0 con 0x55b589dbcbe0 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.962+0000 7f4321754640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589dbcbe0 0x55b589dbefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.963+0000 7f43057fa640 10 client.16758 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.963+0000 7f4321754640 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589dbcbe0 0x55b589dbefd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e4372b77333431a0 server_cookie=bb79ae1d75483a85 in_seq=0 out_seq=0 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f43057fa640 1 -- 172.21.15.116:0/4266505394 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b589d643e0 con 0x55b589dbcbe0 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f43057fa640 10 client.16758 handle_command_reply: tid=0 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 client.16758 shutdown 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 msgr2=0x55b589da98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4318009700 tx=0x7f4318003f20 comp rx=0 tx=0).stop 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 shutdown_connections 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589daa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589da9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589dbcbe0 0x55b589dbefd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 --2- 172.21.15.116:0/4266505394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589da98d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f43057fa640 0 client.16758 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:31:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 >> 172.21.15.116:0/4266505394 conn(0x55b589d9da00 msgr2=0x55b589d9f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 shutdown_connections 2024-03-19T02:31:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 1 -- 172.21.15.116:0/4266505394 wait complete. 2024-03-19T02:31:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.964+0000 7f4326da3740 20 client.16758 trim_cache size 0 max 16384 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 Processor -- start 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 -- start start 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589daa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589dc4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1ac9b0 con 0x55b589da32d0 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1ae4f0 con 0x55b589dac7d0 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.977+0000 7f4326da3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1ad080 con 0x55b589da3bf0 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39356/0 (socket says 172.21.15.116:39356) 2024-03-19T02:31:27.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 -- 172.21.15.116:0/3989586312 learned_addr learned my addr 172.21.15.116:0/3989586312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321754640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589daa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4322756640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589dc4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 -- 172.21.15.116:0/3989586312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 msgr2=0x55b589daa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589daa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 -- 172.21.15.116:0/3989586312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 msgr2=0x55b589dc4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589dc4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 -- 172.21.15.116:0/3989586312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b589da8880 con 0x55b589da32d0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4322756640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589dc4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f4321f55640 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4318011ca0 tx=0x7f4318009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d5f9753083bf73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f43077fe640 1 -- 172.21.15.116:0/3989586312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800f8a0 con 0x55b589da32d0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f43077fe640 1 -- 172.21.15.116:0/3989586312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f431800fa40 con 0x55b589da32d0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.978+0000 7f43077fe640 1 -- 172.21.15.116:0/3989586312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318004be0 con 0x55b589da32d0 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- 172.21.15.116:0/3989586312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 msgr2=0x55b589db8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4318011ca0 tx=0x7f4318009f70 comp rx=0 tx=0).stop 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- 172.21.15.116:0/3989586312 shutdown_connections 2024-03-19T02:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589dc4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589daa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- 172.21.15.116:0/3989586312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db8360 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- 172.21.15.116:0/3989586312 >> 172.21.15.116:0/3989586312 conn(0x55b589d9da00 msgr2=0x55b589d9cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- 172.21.15.116:0/3989586312 shutdown_connections 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- 172.21.15.116:0/3989586312 wait complete. 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 -- start start 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 Processor -- start 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.979+0000 7f4326da3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589db0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589db0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1ade20 con 0x55b589da32d0 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1ad750 con 0x55b589dac7d0 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f431c1aa990 con 0x55b589da3bf0 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 10 client.0 _ll_register_callbacks cb 0x7f4314009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39368/0 (socket says 172.21.15.116:39368) 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 -- 172.21.15.116:0/3562271183 learned_addr learned my addr 172.21.15.116:0/3562271183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321754640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589db0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4322756640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589db0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 -- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 msgr2=0x55b589db0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589db0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 -- 172.21.15.116:0/3562271183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 msgr2=0x55b589db0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589db0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 -- 172.21.15.116:0/3562271183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b589e21c80 con 0x55b589da32d0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4321f55640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f431801eac0 tx=0x7f431801eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299a52ccfefb843b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f43057fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800f8a0 con 0x55b589da32d0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f431802c040 con 0x55b589da32d0 2024-03-19T02:31:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f431800fa40 con 0x55b589da32d0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4318003a80 con 0x55b589da32d0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 10 client.16764 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.980+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b589e22f40 con 0x55b589da32d0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f431800fa40 con 0x55b589da32d0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f431803b070 con 0x55b589da32d0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 10 client.16764 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 10 client.16764 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 20 client.16764 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589e23550 0x55b589e25970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 4 client.16764 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b589d139f0 con 0x55b589e23550 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4321754640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589e23550 0x55b589e25970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f43057fa640 10 client.16764 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.981+0000 7f4321754640 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589e23550 0x55b589e25970 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b4c261afb848b1c server_cookie=42bde41d76a05416 in_seq=0 out_seq=0 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43057fa640 1 -- 172.21.15.116:0/3562271183 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b589d139f0 con 0x55b589e23550 2024-03-19T02:31:27.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43057fa640 10 client.16764 handle_command_reply: tid=0 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 client.16764 shutdown 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 msgr2=0x55b589db02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f431801eac0 tx=0x7f431801eaf0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 shutdown_connections 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b589dac7d0 0x55b589db0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b589da3bf0 0x55b589db0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b589e23550 0x55b589e25970 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 --2- 172.21.15.116:0/3562271183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b589da32d0 0x55b589db02d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 >> 172.21.15.116:0/3562271183 conn(0x55b589d9da00 msgr2=0x55b589d9ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43057fa640 0 client.16764 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 shutdown_connections 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 1 -- 172.21.15.116:0/3562271183 wait complete. 2024-03-19T02:31:27.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f4326da3740 20 client.16764 trim_cache size 0 max 16384 2024-03-19T02:31:27.989 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:31:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 msgr2=0x7f43080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 0x7f43080a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4318009f40 tx=0x7f4318005f90 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 msgr2=0x7f431c197410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f430c009230 tx=0x7f430c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 shutdown_connections 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f43080a2970 0x7f43080a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f431c1066b0 0x7f431c19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431c101060 0x7f431c197410 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 --2- 172.21.15.116:0/2540270529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f431c1006a0 0x7f431c196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.982+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 >> 172.21.15.116:0/2540270529 conn(0x7f431c0778d0 msgr2=0x7f431c102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.983+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 shutdown_connections 2024-03-19T02:31:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:27.983+0000 7f43241b9640 1 -- 172.21.15.116:0/2540270529 wait complete. 2024-03-19T02:31:27.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:27 smithi116 ceph-mon[32263]: pgmap v112: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:27.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:27 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3938083806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:31:27.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:31:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- 172.21.15.116:0/1295393629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad00103360 msgr2=0x7fad0010b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 --2- 172.21.15.116:0/1295393629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad00103360 0x7fad0010b840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7facf0009900 tx=0x7facf002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- 172.21.15.116:0/1295393629 shutdown_connections 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 --2- 172.21.15.116:0/1295393629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0010bd80 0x7fad0010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 --2- 172.21.15.116:0/1295393629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad00103360 0x7fad0010b840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 --2- 172.21.15.116:0/1295393629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 0x7fad00102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- 172.21.15.116:0/1295393629 >> 172.21.15.116:0/1295393629 conn(0x7fad000778d0 msgr2=0x7fad00077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- 172.21.15.116:0/1295393629 shutdown_connections 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- 172.21.15.116:0/1295393629 wait complete. 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 Processor -- start 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.110+0000 7fad0800c640 1 -- start start 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 0x7fad00197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad0010bd80 0x7fad0019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad00111a30 con 0x7fad00103360 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad001118b0 con 0x7fad00102a40 2024-03-19T02:31:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad00111bb0 con 0x7fad0010bd80 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad05da8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 0x7fad00197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39386/0 (socket says 172.21.15.116:39386) 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 -- 172.21.15.116:0/3933038634 learned_addr learned my addr 172.21.15.116:0/3933038634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad065a9640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad0010bd80 0x7fad0019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 -- 172.21.15.116:0/3933038634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad0010bd80 msgr2=0x7fad0019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad0010bd80 0x7fad0019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 -- 172.21.15.116:0/3933038634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 msgr2=0x7fad00197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 0x7fad00197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 -- 172.21.15.116:0/3933038634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0019cfc0 con 0x7fad00103360 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad055a7640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7facf0005940 tx=0x7facf002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7facf6ffd640 1 -- 172.21.15.116:0/3933038634 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0043020 con 0x7fad00103360 2024-03-19T02:31:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0019d250 con 0x7fad00103360 2024-03-19T02:31:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7facf6ffd640 1 -- 172.21.15.116:0/3933038634 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facf0004e00 con 0x7fad00103360 2024-03-19T02:31:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7facf6ffd640 1 -- 172.21.15.116:0/3933038634 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0003850 con 0x7fad00103360 2024-03-19T02:31:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.111+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad001a4b40 con 0x7fad00103360 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.114+0000 7facf6ffd640 1 -- 172.21.15.116:0/3933038634 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7facf0046020 con 0x7fad00103360 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.114+0000 7fad0abf0740 1 Processor -- start 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.114+0000 7fad0abf0740 1 -- start start 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.114+0000 7facf6ffd640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 0x7facec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bb4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9bb6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5635c9b0e4b0 con 0x5635c9bb47d0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5635c9ab6d70 con 0x5635c9babbf0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad0abf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5635c9b0a550 con 0x5635c9bab2d0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad065a9640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 0x7facec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7facf6ffd640 1 -- 172.21.15.116:0/3933038634 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7facf00f10c0 con 0x7fad00103360 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad065a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9bb6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38824/0 (socket says 172.21.15.116:38824) 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 -- 172.21.15.116:0/2736606671 learned_addr learned my addr 172.21.15.116:0/2736606671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad055a7640 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bb4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 -- 172.21.15.116:0/2736606671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 msgr2=0x5635c9bb4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bb4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 -- 172.21.15.116:0/2736606671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 msgr2=0x5635c9bb6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9bb6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 -- 172.21.15.116:0/2736606671 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635c9ad9210 con 0x5635c9bab2d0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad065a9640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 0x7facec0a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7facf8005fd0 tx=0x7facf8009df0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7fad05da8640 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7facfc009d20 tx=0x7facfc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a248b1d19c044eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7facebfff640 1 -- 172.21.15.116:0/2736606671 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc01e070 con 0x5635c9bab2d0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7facebfff640 1 -- 172.21.15.116:0/2736606671 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facfc019ae0 con 0x5635c9bab2d0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.115+0000 7facebfff640 1 -- 172.21.15.116:0/2736606671 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc00e930 con 0x5635c9bab2d0 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 -- 172.21.15.116:0/2736606671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 msgr2=0x5635c9bab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7facfc009d20 tx=0x7facfc011810 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 -- 172.21.15.116:0/2736606671 shutdown_connections 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9bb6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bb4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2736606671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bab2d0 0x5635c9bab6b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 -- 172.21.15.116:0/2736606671 >> 172.21.15.116:0/2736606671 conn(0x5635c9ba5a00 msgr2=0x5635c9ba5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 -- 172.21.15.116:0/2736606671 shutdown_connections 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.116+0000 7fad0abf0740 1 -- 172.21.15.116:0/2736606671 wait complete. 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 -- start start 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 Processor -- start 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5635c9bb75c0 con 0x5635c9bab2d0 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5635c9aca310 con 0x5635c9bb47d0 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5635c9bb7740 con 0x5635c9babbf0 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad05da8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad0abf0740 10 client.0 _ll_register_callbacks cb 0x5635c9abe2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad05da8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39410/0 (socket says 172.21.15.116:39410) 2024-03-19T02:31:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad05da8640 1 -- 172.21.15.116:0/2712298499 learned_addr learned my addr 172.21.15.116:0/2712298499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad055a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.117+0000 7fad065a9640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 -- 172.21.15.116:0/2712298499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 msgr2=0x5635c9bb1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 -- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 msgr2=0x5635c9bb2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 -- 172.21.15.116:0/2712298499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635c98223f0 con 0x5635c9bab2d0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad065a9640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad055a7640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad05da8640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7facfc01c040 tx=0x7facfc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87ad89d706176c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7face97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc027020 con 0x5635c9bab2d0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7facfc005250 con 0x5635c9bab2d0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facfc019ae0 con 0x5635c9bab2d0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7facfc032050 con 0x5635c9bab2d0 2024-03-19T02:31:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad0abf0740 10 client.16776 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635c98ec7d0 con 0x5635c9bab2d0 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.118+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facfc019ae0 con 0x5635c9bab2d0 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7facfc00e6b0 con 0x5635c9bab2d0 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 10 client.16776 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 10 client.16776 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 20 client.16776 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9bc4a60 0x5635c9bc6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 4 client.16776 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635c9b6c3e0 con 0x5635c9bc4a60 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad055a7640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9bc4a60 0x5635c9bc6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7fad055a7640 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9bc4a60 0x5635c9bc6e50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a5fe86bd3a886117 server_cookie=97f399d69425a2e9 in_seq=0 out_seq=0 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.119+0000 7face97fa640 10 client.16776 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7face97fa640 1 -- 172.21.15.116:0/2712298499 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635c9b6c3e0 con 0x5635c9bc4a60 2024-03-19T02:31:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7face97fa640 10 client.16776 handle_command_reply: tid=0 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 client.16776 shutdown 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 msgr2=0x5635c9bb1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7facfc01c040 tx=0x7facfc019660 comp rx=0 tx=0).stop 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 shutdown_connections 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9bc4a60 0x5635c9bc6e50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7face97fa640 0 client.16776 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 --2- 172.21.15.116:0/2712298499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9bb1790 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 >> 172.21.15.116:0/2712298499 conn(0x5635c9ba5a00 msgr2=0x5635c9ba7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 shutdown_connections 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 1 -- 172.21.15.116:0/2712298499 wait complete. 2024-03-19T02:31:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.120+0000 7fad0abf0740 20 client.16776 trim_cache size 0 max 16384 2024-03-19T02:31:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 Processor -- start 2024-03-19T02:31:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 -- start start 2024-03-19T02:31:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bc0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad001b0cd0 con 0x5635c9bab2d0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad001b2810 con 0x5635c9babbf0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad0abf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad001b13a0 con 0x5635c9bb47d0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39412/0 (socket says 172.21.15.116:39412) 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 -- 172.21.15.116:0/1771947654 learned_addr learned my addr 172.21.15.116:0/1771947654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad055a7640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bc0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad065a9640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 -- 172.21.15.116:0/1771947654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bb47d0 msgr2=0x5635c9bb2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 -- 172.21.15.116:0/1771947654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 msgr2=0x5635c9bc0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bc0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 -- 172.21.15.116:0/1771947654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635c9bc7480 con 0x5635c9bab2d0 2024-03-19T02:31:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad055a7640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bc0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.134+0000 7fad05da8640 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7facfc009ff0 tx=0x7facfc009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecbec4b80debaa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7faceb7fe640 1 -- 172.21.15.116:0/1771947654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc027070 con 0x5635c9bab2d0 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7faceb7fe640 1 -- 172.21.15.116:0/1771947654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facfc01f040 con 0x5635c9bab2d0 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7faceb7fe640 1 -- 172.21.15.116:0/1771947654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc01e070 con 0x5635c9bab2d0 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 -- 172.21.15.116:0/1771947654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 msgr2=0x5635c9ba4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7facfc009ff0 tx=0x7facfc009620 comp rx=0 tx=0).stop 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 -- 172.21.15.116:0/1771947654 shutdown_connections 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9bb47d0 0x5635c9bb2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9babbf0 0x5635c9bc0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1771947654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bab2d0 0x5635c9ba4de0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 -- 172.21.15.116:0/1771947654 >> 172.21.15.116:0/1771947654 conn(0x5635c9ba5a00 msgr2=0x5635c9ba74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 -- 172.21.15.116:0/1771947654 shutdown_connections 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.135+0000 7fad0abf0740 1 -- 172.21.15.116:0/1771947654 wait complete. 2024-03-19T02:31:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 -- start start 2024-03-19T02:31:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 Processor -- start 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 0x5635c9bb7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad001b2140 con 0x5635c9bb47d0 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad001b1a70 con 0x5635c9bab2d0 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad001aecb0 con 0x5635c9babbf0 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad0abf0740 10 client.0 _ll_register_callbacks cb 0x7facfc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad055a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad065a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad05da8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 0x5635c9bb7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad055a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38862/0 (socket says 172.21.15.116:38862) 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad065a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39428/0 (socket says 172.21.15.116:39428) 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad05da8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 0x5635c9bb7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51112/0 (socket says 172.21.15.116:51112) 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.136+0000 7fad055a7640 1 -- 172.21.15.116:0/1555134657 learned_addr learned my addr 172.21.15.116:0/1555134657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 -- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 msgr2=0x5635c9bb7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 0x5635c9bb7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 -- 172.21.15.116:0/1555134657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 msgr2=0x5635c9c22ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 -- 172.21.15.116:0/1555134657 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635c9bb6bc0 con 0x5635c9babbf0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad065a9640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad055a7640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7facf00b9910 tx=0x7facf00dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d74e101b2ef2b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf00d6060 con 0x5635c9babbf0 2024-03-19T02:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facfc01f040 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7facf00d6200 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7facf00a6070 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 10 client.26539 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635c9ba5830 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facfc01f040 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7facf00bd710 con 0x5635c9babbf0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 10 client.26539 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 10 client.26539 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.137+0000 7fad0abf0740 20 client.26539 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9c2aa00 0x5635c9c2ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 4 client.26539 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635c9b1b9f0 con 0x5635c9c2aa00 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad05da8640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9c2aa00 0x5635c9c2ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7face97fa640 10 client.26539 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad05da8640 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9c2aa00 0x5635c9c2ce20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4226ee490b5b11a server_cookie=6682a0a1f9f70639 in_seq=0 out_seq=0 2024-03-19T02:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7face97fa640 1 -- 172.21.15.116:0/1555134657 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635c9b1b9f0 con 0x5635c9c2aa00 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7face97fa640 10 client.26539 handle_command_reply: tid=0 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 1 client.26539 shutdown 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 msgr2=0x5635c9bb80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.138+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7facf00b9910 tx=0x7facf00dd040 comp rx=0 tx=0).stop 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 shutdown_connections 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5635c9c2aa00 0x5635c9c2ce20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635c9bb47d0 0x5635c9c22ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5635c9babbf0 0x5635c9bb80b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 --2- 172.21.15.116:0/1555134657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5635c9bab2d0 0x5635c9bb7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7face97fa640 0 client.26539 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 >> 172.21.15.116:0/1555134657 conn(0x5635c9ba5a00 msgr2=0x5635c9bb2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 shutdown_connections 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 1 -- 172.21.15.116:0/1555134657 wait complete. 2024-03-19T02:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0abf0740 20 client.26539 trim_cache size 0 max 16384 2024-03-19T02:31:28.146 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:31:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 msgr2=0x7facec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 0x7facec0a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7facf8005fd0 tx=0x7facf8009df0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 msgr2=0x7fad00198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7facf0005940 tx=0x7facf002c690 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 shutdown_connections 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7facec0a2970 0x7facec0a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad0010bd80 0x7fad0019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00103360 0x7fad00198510 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 --2- 172.21.15.116:0/3933038634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad00102a40 0x7fad00197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 >> 172.21.15.116:0/3933038634 conn(0x7fad000778d0 msgr2=0x7fad000fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 shutdown_connections 2024-03-19T02:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.139+0000 7fad0800c640 1 -- 172.21.15.116:0/3933038634 wait complete. 2024-03-19T02:31:28.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:31:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:27 smithi162 ceph-mon[35761]: pgmap v112: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:27 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3938083806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:31:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:27 smithi154 ceph-mon[35693]: pgmap v112: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:27 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3938083806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 -- 172.21.15.116:0/3044810325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701077c0 msgr2=0x7fce70107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 --2- 172.21.15.116:0/3044810325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701077c0 0x7fce70107ba0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fce60009930 tx=0x7fce6002bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 -- 172.21.15.116:0/3044810325 shutdown_connections 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 --2- 172.21.15.116:0/3044810325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce7010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 --2- 172.21.15.116:0/3044810325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701017b0 0x7fce70101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 --2- 172.21.15.116:0/3044810325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701077c0 0x7fce70107ba0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 -- 172.21.15.116:0/3044810325 >> 172.21.15.116:0/3044810325 conn(0x7fce700fd2e0 msgr2=0x7fce700ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 -- 172.21.15.116:0/3044810325 shutdown_connections 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.267+0000 7fce754d1640 1 -- 172.21.15.116:0/3044810325 wait complete. 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 Processor -- start 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 -- start start 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce70198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701077c0 0x7fce7019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce70111a80 con 0x7fce701017b0 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce70111900 con 0x7fce70102170 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce754d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce70111c00 con 0x7fce701077c0 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce67fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce70198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39434/0 (socket says 172.21.15.116:39434) 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 -- 172.21.15.116:0/2385332286 learned_addr learned my addr 172.21.15.116:0/2385332286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701077c0 0x7fce7019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 -- 172.21.15.116:0/2385332286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701077c0 msgr2=0x7fce7019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701077c0 0x7fce7019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 -- 172.21.15.116:0/2385332286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 msgr2=0x7fce70198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce70198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce6effd640 1 -- 172.21.15.116:0/2385332286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7019d010 con 0x7fce701017b0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.268+0000 7fce67fff640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce70198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce6effd640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fce60009230 tx=0x7fce6002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce6cff9640 1 -- 172.21.15.116:0/2385332286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60043020 con 0x7fce701017b0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce6cff9640 1 -- 172.21.15.116:0/2385332286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce6002ca70 con 0x7fce701017b0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce6cff9640 1 -- 172.21.15.116:0/2385332286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60003a50 con 0x7fce701017b0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7019d2a0 con 0x7fce701017b0 2024-03-19T02:31:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.269+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce701a4b90 con 0x7fce701017b0 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.271+0000 7fce6cff9640 1 -- 172.21.15.116:0/2385332286 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fce60046020 con 0x7fce701017b0 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.271+0000 7fce78198740 1 Processor -- start 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 -- start start 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6cff9640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 0x7fce5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbd9be0 0x556d8cbdbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d8cb2f4b0 con 0x556d8cbd9be0 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556d8cad7d70 con 0x556d8cbcc8e0 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce78198740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d8cb2b550 con 0x556d8cbcbfc0 2024-03-19T02:31:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce67fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38886/0 (socket says 172.21.15.116:38886) 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6f7fe640 1 -- 172.21.15.116:0/2809988532 learned_addr learned my addr 172.21.15.116:0/2809988532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6cff9640 1 -- 172.21.15.116:0/2385332286 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce600f10b0 con 0x7fce701017b0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6effd640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 0x7fce5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.272+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbd9be0 0x556d8cbdbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 -- 172.21.15.116:0/2809988532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 msgr2=0x556d8cbcc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 -- 172.21.15.116:0/2809988532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbd9be0 msgr2=0x556d8cbdbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbd9be0 0x556d8cbdbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 -- 172.21.15.116:0/2809988532 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8cafa210 con 0x556d8cbcc8e0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce6effd640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 0x7fce5c0a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce600ee640 tx=0x7fce600dd040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce67fff640 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd96a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fa0 tx=0x7fce580118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e32d109dbaa442c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce657fa640 1 -- 172.21.15.116:0/2809988532 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58004310 con 0x556d8cbcc8e0 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce657fa640 1 -- 172.21.15.116:0/2809988532 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce580044b0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.273+0000 7fce657fa640 1 -- 172.21.15.116:0/2809988532 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58004e50 con 0x556d8cbcc8e0 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- 172.21.15.116:0/2809988532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 msgr2=0x556d8cbd96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd96a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fa0 tx=0x7fce580118d0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- 172.21.15.116:0/2809988532 shutdown_connections 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbd9be0 0x556d8cbdbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd96a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 --2- 172.21.15.116:0/2809988532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcbfc0 0x556d8cbcc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- 172.21.15.116:0/2809988532 >> 172.21.15.116:0/2809988532 conn(0x556d8cbc6a00 msgr2=0x556d8cbc6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- 172.21.15.116:0/2809988532 shutdown_connections 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- 172.21.15.116:0/2809988532 wait complete. 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 -- start start 2024-03-19T02:31:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.274+0000 7fce78198740 1 Processor -- start 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbd9be0 0x556d8cbd7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d8cbdc9d0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556d8caeb310 con 0x556d8cbcc8e0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556d8cbdcb50 con 0x556d8cbd9be0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 10 client.0 _ll_register_callbacks cb 0x556d8cadf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39452/0 (socket says 172.21.15.116:39452) 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 -- 172.21.15.116:0/1663333229 learned_addr learned my addr 172.21.15.116:0/1663333229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce67fff640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbd9be0 0x556d8cbd7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 -- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbd9be0 msgr2=0x556d8cbd7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbd9be0 0x556d8cbd7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 -- 172.21.15.116:0/1663333229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 msgr2=0x556d8cbd7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 -- 172.21.15.116:0/1663333229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8c8433f0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fce68003d70 tx=0x7fce68009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75f6c9b962c64db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce567fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce68004740 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce6801e3c0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce580044b0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fce6801e5c0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 10 client.16794 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.275+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556d8c90d7d0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce580044b0 con 0x556d8cbcbfc0 2024-03-19T02:31:28.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fce68021660 con 0x556d8cbcbfc0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 10 client.16794 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 10 client.16794 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 20 client.16794 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cbe9ff0 0x556d8cbec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 4 client.16794 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d8cb8d3e0 con 0x556d8cbe9ff0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce67fff640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cbe9ff0 0x556d8cbec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce567fc640 10 client.16794 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.276+0000 7fce67fff640 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cbe9ff0 0x556d8cbec400 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a23fb37b7f60bb8 server_cookie=b56af875ae12fb64 in_seq=0 out_seq=0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce567fc640 1 -- 172.21.15.116:0/1663333229 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d8cb8d3e0 con 0x556d8cbe9ff0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce567fc640 10 client.16794 handle_command_reply: tid=0 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 client.16794 shutdown 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 msgr2=0x556d8cbd6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fce68003d70 tx=0x7fce68009500 comp rx=0 tx=0).stop 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 shutdown_connections 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cbe9ff0 0x556d8cbec400 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbd9be0 0x556d8cbd7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce567fc640 0 client.16794 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 --2- 172.21.15.116:0/1663333229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd6be0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 >> 172.21.15.116:0/1663333229 conn(0x556d8cbc6a00 msgr2=0x556d8cbc8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.277+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 shutdown_connections 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.278+0000 7fce78198740 1 -- 172.21.15.116:0/1663333229 wait complete. 2024-03-19T02:31:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.278+0000 7fce78198740 20 client.16794 trim_cache size 0 max 16384 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 Processor -- start 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 -- start start 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbe57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce701ac990 con 0x556d8cbcbfc0 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce701ae4d0 con 0x556d8cbd9be0 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce78198740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce701ad060 con 0x556d8cbcc8e0 2024-03-19T02:31:28.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39464/0 (socket says 172.21.15.116:39464) 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce6f7fe640 1 -- 172.21.15.116:0/320907126 learned_addr learned my addr 172.21.15.116:0/320907126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38908/0 (socket says 172.21.15.116:38908) 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.292+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbe57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 -- 172.21.15.116:0/320907126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 msgr2=0x556d8cbe57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbe57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 -- 172.21.15.116:0/320907126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 msgr2=0x556d8cbd9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 -- 172.21.15.116:0/320907126 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d8cb9c880 con 0x556d8cbcc8e0 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce67fff640 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce58005e00 tx=0x7fce580118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4298017508dd349 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce64ff9640 1 -- 172.21.15.116:0/320907126 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5801d040 con 0x556d8cbcc8e0 2024-03-19T02:31:28.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce64ff9640 1 -- 172.21.15.116:0/320907126 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce580044b0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.293+0000 7fce64ff9640 1 -- 172.21.15.116:0/320907126 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce580035f0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- 172.21.15.116:0/320907126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 msgr2=0x556d8cbd64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce58005e00 tx=0x7fce580118d0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- 172.21.15.116:0/320907126 shutdown_connections 2024-03-19T02:31:28.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbe57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbd64c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 --2- 172.21.15.116:0/320907126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbd9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- 172.21.15.116:0/320907126 >> 172.21.15.116:0/320907126 conn(0x556d8cbc6a00 msgr2=0x556d8cbc5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- 172.21.15.116:0/320907126 shutdown_connections 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- 172.21.15.116:0/320907126 wait complete. 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 -- start start 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.294+0000 7fce78198740 1 Processor -- start 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbdceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce701ade00 con 0x556d8cbcbfc0 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce701ad730 con 0x556d8cbd9be0 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce701aa970 con 0x556d8cbcc8e0 2024-03-19T02:31:28.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce78198740 10 client.0 _ll_register_callbacks cb 0x7fce68003bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38914/0 (socket says 172.21.15.116:38914) 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39476/0 (socket says 172.21.15.116:39476) 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 -- 172.21.15.116:0/1626584892 learned_addr learned my addr 172.21.15.116:0/1626584892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbdceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 -- 172.21.15.116:0/1626584892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 msgr2=0x556d8cbdceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbdceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 -- 172.21.15.116:0/1626584892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 msgr2=0x556d8cbe1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 -- 172.21.15.116:0/1626584892 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8cbc20a0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce67fff640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce5801b040 tx=0x7fce58004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819b3a6065c0503c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.295+0000 7fce567fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5801d070 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce580044b0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fce5802b350 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fce58042050 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 10 client.26557 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556d8cc4fbc0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce580044b0 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fce5803d030 con 0x556d8cbcc8e0 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 10 client.26557 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 10 client.26557 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 20 client.26557 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cc500f0 0x556d8cc52510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 4 client.26557 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d8cb3c9f0 con 0x556d8cc500f0 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.296+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cc500f0 0x556d8cc52510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce6f7fe640 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cc500f0 0x556d8cc52510 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5429a0c5a1868c4 server_cookie=eff2aef88bb38598 in_seq=0 out_seq=0 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce567fc640 10 client.26557 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce567fc640 1 -- 172.21.15.116:0/1626584892 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d8cb3c9f0 con 0x556d8cc500f0 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce567fc640 10 client.26557 handle_command_reply: tid=0 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 client.26557 shutdown 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 msgr2=0x556d8cbe1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce5801b040 tx=0x7fce58004250 comp rx=0 tx=0).stop 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 shutdown_connections 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556d8cc500f0 0x556d8cc52510 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d8cbd9be0 0x556d8cbdceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce567fc640 0 client.26557 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:31:28.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556d8cbcc8e0 0x556d8cbe1ff0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 --2- 172.21.15.116:0/1626584892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d8cbcbfc0 0x556d8cbe1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 >> 172.21.15.116:0/1626584892 conn(0x556d8cbc6a00 msgr2=0x556d8cbc7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 shutdown_connections 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 1 -- 172.21.15.116:0/1626584892 wait complete. 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.297+0000 7fce78198740 20 client.26557 trim_cache size 0 max 16384 2024-03-19T02:31:28.305 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 msgr2=0x7fce5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 0x7fce5c0a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce600ee640 tx=0x7fce600dd040 comp rx=0 tx=0).stop 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 msgr2=0x7fce70198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fce60009230 tx=0x7fce6002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 shutdown_connections 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fce5c0a2970 0x7fce5c0a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce701077c0 0x7fce7019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce70102170 0x7fce70198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 --2- 172.21.15.116:0/2385332286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701017b0 0x7fce70198020 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 >> 172.21.15.116:0/2385332286 conn(0x7fce700fd2e0 msgr2=0x7fce700fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 shutdown_connections 2024-03-19T02:31:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.298+0000 7fce754d1640 1 -- 172.21.15.116:0/2385332286 wait complete. 2024-03-19T02:31:28.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 -- 172.21.15.116:0/692007043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 msgr2=0x7fa0c4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/692007043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c4107b90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009930 tx=0x7fa0b402bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 -- 172.21.15.116:0/692007043 shutdown_connections 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/692007043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c4102160 0x7fa0c410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/692007043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c41017a0 0x7fa0c4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/692007043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c4107b90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 -- 172.21.15.116:0/692007043 >> 172.21.15.116:0/692007043 conn(0x7fa0c40fd2d0 msgr2=0x7fa0c40ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 -- 172.21.15.116:0/692007043 shutdown_connections 2024-03-19T02:31:28.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.425+0000 7fa0c8e27640 1 -- 172.21.15.116:0/692007043 wait complete. 2024-03-19T02:31:28.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 Processor -- start 2024-03-19T02:31:28.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 -- start start 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 0x7fa0c4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c4111a30 con 0x7fa0c41077b0 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41118b0 con 0x7fa0c41017a0 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c8e27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c4111bb0 con 0x7fa0c4102160 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c1d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 0x7fa0c4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c1d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 0x7fa0c4198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38934/0 (socket says 172.21.15.116:38934) 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51192/0 (socket says 172.21.15.116:51192) 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c259c640 1 -- 172.21.15.116:0/329886307 learned_addr learned my addr 172.21.15.116:0/329886307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.426+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39488/0 (socket says 172.21.15.116:39488) 2024-03-19T02:31:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 -- 172.21.15.116:0/329886307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 msgr2=0x7fa0c4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 0x7fa0c4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 -- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 msgr2=0x7fa0c419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 -- 172.21.15.116:0/329886307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c419cfe0 con 0x7fa0c41017a0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c259c640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009230 tx=0x7fa0b402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0b37fe640 1 -- 172.21.15.116:0/329886307 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4043020 con 0x7fa0c41017a0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c419d270 con 0x7fa0c41017a0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0b37fe640 1 -- 172.21.15.116:0/329886307 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0b402ca70 con 0x7fa0c41017a0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0b37fe640 1 -- 172.21.15.116:0/329886307 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4003a50 con 0x7fa0c41017a0 2024-03-19T02:31:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.427+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c41a4b60 con 0x7fa0c41017a0 2024-03-19T02:31:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.429+0000 7fa0b37fe640 1 -- 172.21.15.116:0/329886307 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa0b4046020 con 0x7fa0c41017a0 2024-03-19T02:31:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 Processor -- start 2024-03-19T02:31:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 -- start start 2024-03-19T02:31:28.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5019290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e501bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1e4f734b0 con 0x55c1e50197d0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1e4f1bd70 con 0x55c1e5010bf0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0cba14740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1e4f6f550 con 0x55c1e50102d0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0b37fe640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 0x7fa0ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5019290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38944/0 (socket says 172.21.15.116:38944) 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c2d9d640 1 -- 172.21.15.116:0/725187643 learned_addr learned my addr 172.21.15.116:0/725187643 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 0x7fa0ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.430+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e501bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 -- 172.21.15.116:0/725187643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 msgr2=0x55c1e5019290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0b37fe640 1 -- 172.21.15.116:0/329886307 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa0b40f10b0 con 0x7fa0c41017a0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5019290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 -- 172.21.15.116:0/725187643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 msgr2=0x55c1e501bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e501bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 -- 172.21.15.116:0/725187643 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e4f3e210 con 0x55c1e50102d0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009fd0 tx=0x7fa0bc00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c8359a4b4efb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 0x7fa0ac0a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0099e0 tx=0x7fa0bc018000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0b0ff9640 1 -- 172.21.15.116:0/725187643 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc022070 con 0x55c1e50102d0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0b0ff9640 1 -- 172.21.15.116:0/725187643 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0bc0115f0 con 0x55c1e50102d0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.431+0000 7fa0b0ff9640 1 -- 172.21.15.116:0/725187643 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00edf0 con 0x55c1e50102d0 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- 172.21.15.116:0/725187643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 msgr2=0x55c1e50106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009fd0 tx=0x7fa0bc00fa00 comp rx=0 tx=0).stop 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- 172.21.15.116:0/725187643 shutdown_connections 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e501bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5019290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 --2- 172.21.15.116:0/725187643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e50106b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- 172.21.15.116:0/725187643 >> 172.21.15.116:0/725187643 conn(0x55c1e500aa00 msgr2=0x55c1e500ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- 172.21.15.116:0/725187643 shutdown_connections 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- 172.21.15.116:0/725187643 wait complete. 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 -- start start 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.432+0000 7fa0cba14740 1 Processor -- start 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5016db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1e501c500 con 0x55c1e50197d0 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1e4f2f310 con 0x55c1e5010bf0 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1e501c680 con 0x55c1e50102d0 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0cba14740 10 client.0 _ll_register_callbacks cb 0x55c1e4f232d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38954/0 (socket says 172.21.15.116:38954) 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39514/0 (socket says 172.21.15.116:39514) 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 -- 172.21.15.116:0/182146370 learned_addr learned my addr 172.21.15.116:0/182146370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5016db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 -- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 msgr2=0x55c1e5016db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5016db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 -- 172.21.15.116:0/182146370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 msgr2=0x55c1e50172f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 -- 172.21.15.116:0/182146370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e4c873f0 con 0x55c1e50102d0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0c259c640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40b9e70 tx=0x7fa0b402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ef12725f8bf7d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b40d6060 con 0x55c1e50102d0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.433+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa0b40b9b10 con 0x55c1e50102d0 2024-03-19T02:31:28.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1e4fd13e0 con 0x55c1e50102d0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa0b40bd4a0 con 0x55c1e50102d0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 10 client.26569 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c1e4d517d0 con 0x55c1e50102d0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c1e4fd13e0 con 0x55c1e50102d0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa0b40c1170 con 0x55c1e50102d0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 10 client.26569 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 10 client.26569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 20 client.26569 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5029a40 0x55c1e502be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 4 client.26569 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1e4f809f0 con 0x55c1e5029a40 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.434+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5029a40 0x55c1e502be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0a9ffb640 10 client.26569 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5029a40 0x55c1e502be90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8104c41cbbf3f32 server_cookie=fc3593f3523633ba in_seq=0 out_seq=0 2024-03-19T02:31:28.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/182146370 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1e4f809f0 con 0x55c1e5029a40 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0a9ffb640 10 client.26569 handle_command_reply: tid=0 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0cba14740 1 client.26569 shutdown 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 msgr2=0x55c1e5016870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.435+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40b9e70 tx=0x7fa0b402bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 shutdown_connections 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5029a40 0x55c1e502be90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50172f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0a9ffb640 0 client.26569 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5016db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 --2- 172.21.15.116:0/182146370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5016870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 >> 172.21.15.116:0/182146370 conn(0x55c1e500aa00 msgr2=0x55c1e500c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 shutdown_connections 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 1 -- 172.21.15.116:0/182146370 wait complete. 2024-03-19T02:31:28.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.436+0000 7fa0cba14740 20 client.26569 trim_cache size 0 max 16384 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 Processor -- start 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 -- start start 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e50102d0 0x55c1e50252a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50197d0 0x55c1e5031760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41aebd0 con 0x55c1e5010bf0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b0710 con 0x55c1e50102d0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0cba14740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41af2a0 con 0x55c1e50197d0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39516/0 (socket says 172.21.15.116:39516) 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 -- 172.21.15.116:0/122135567 learned_addr learned my addr 172.21.15.116:0/122135567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50197d0 0x55c1e5031760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c259c640 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e50102d0 0x55c1e50252a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 -- 172.21.15.116:0/122135567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50197d0 msgr2=0x55c1e5031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50197d0 0x55c1e5031760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 -- 172.21.15.116:0/122135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e50102d0 msgr2=0x55c1e50252a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e50102d0 0x55c1e50252a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.450+0000 7fa0c1d9b640 1 -- 172.21.15.116:0/122135567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e4fe0880 con 0x55c1e5010bf0 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.451+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009ca0 tx=0x7fa0b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa930376895059a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.451+0000 7fa0abfff640 1 -- 172.21.15.116:0/122135567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8003d60 con 0x55c1e5010bf0 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.451+0000 7fa0abfff640 1 -- 172.21.15.116:0/122135567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0b801f500 con 0x55c1e5010bf0 2024-03-19T02:31:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.451+0000 7fa0abfff640 1 -- 172.21.15.116:0/122135567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8010a50 con 0x55c1e5010bf0 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- 172.21.15.116:0/122135567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 msgr2=0x55c1e5017110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009ca0 tx=0x7fa0b8005f90 comp rx=0 tx=0).stop 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- 172.21.15.116:0/122135567 shutdown_connections 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50197d0 0x55c1e5031760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e5010bf0 0x55c1e5017110 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- 172.21.15.116:0/122135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e50102d0 0x55c1e50252a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- 172.21.15.116:0/122135567 >> 172.21.15.116:0/122135567 conn(0x55c1e500aa00 msgr2=0x55c1e500c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- 172.21.15.116:0/122135567 shutdown_connections 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- 172.21.15.116:0/122135567 wait complete. 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- start start 2024-03-19T02:31:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 Processor -- start 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5013b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b0040 con 0x55c1e50197d0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41af970 con 0x55c1e5010bf0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41acbb0 con 0x55c1e50102d0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.452+0000 7fa0cba14740 10 client.0 _ll_register_callbacks cb 0x7fa0b40057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5013b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38968/0 (socket says 172.21.15.116:38968) 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 -- 172.21.15.116:0/3552067165 learned_addr learned my addr 172.21.15.116:0/3552067165 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39520/0 (socket says 172.21.15.116:39520) 2024-03-19T02:31:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 -- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 msgr2=0x55c1e5013b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5013b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 -- 172.21.15.116:0/3552067165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 msgr2=0x55c1e50140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 -- 172.21.15.116:0/3552067165 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e508efe0 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c2d9d640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0c259c640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009900 tx=0x7fa0b40bd320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2fdca8a7c052682 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b40ed030 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0b801f500 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa0b40c57b0 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa0b40bce00 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0cba14740 10 client.26575 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.453+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c1e508efe0 con 0x55c1e50102d0 2024-03-19T02:31:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0b801f500 con 0x55c1e50102d0 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa0b40eebd0 con 0x55c1e50102d0 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 10 client.26575 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 10 client.26575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 20 client.26575 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5090680 0x55c1e5092a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 4 client.26575 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1e4f7ad40 con 0x55c1e5090680 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5090680 0x55c1e5092a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0a9ffb640 10 client.26575 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.454+0000 7fa0c1d9b640 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5090680 0x55c1e5092a70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffe81481bb984c36 server_cookie=9d039021b5cb7c6e in_seq=0 out_seq=0 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.455+0000 7fa0a9ffb640 1 -- 172.21.15.116:0/3552067165 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1e4f7ad40 con 0x55c1e5090680 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.455+0000 7fa0a9ffb640 10 client.26575 handle_command_reply: tid=0 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.455+0000 7fa0cba14740 1 client.26575 shutdown 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 msgr2=0x55c1e5013640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009900 tx=0x7fa0b40bd320 comp rx=0 tx=0).stop 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 shutdown_connections 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c1e5090680 0x55c1e5092a70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e50197d0 0x55c1e50140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e5010bf0 0x55c1e5013b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0a9ffb640 0 client.26575 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 --2- 172.21.15.116:0/3552067165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e50102d0 0x55c1e5013640 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 >> 172.21.15.116:0/3552067165 conn(0x55c1e500aa00 msgr2=0x55c1e500be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 shutdown_connections 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 1 -- 172.21.15.116:0/3552067165 wait complete. 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0cba14740 20 client.26575 trim_cache size 0 max 16384 2024-03-19T02:31:28.461 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 msgr2=0x7fa0ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 0x7fa0ac0a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0099e0 tx=0x7fa0bc018000 comp rx=0 tx=0).stop 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 msgr2=0x7fa0c4197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009230 tx=0x7fa0b402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 shutdown_connections 2024-03-19T02:31:28.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa0ac0a2970 0x7fa0ac0a4e30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c41077b0 0x7fa0c419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c4102160 0x7fa0c4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 --2- 172.21.15.116:0/329886307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0c41017a0 0x7fa0c4197ff0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 >> 172.21.15.116:0/329886307 conn(0x7fa0c40fd2d0 msgr2=0x7fa0c40fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 shutdown_connections 2024-03-19T02:31:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:28.456+0000 7fa0c8e27640 1 -- 172.21.15.116:0/329886307 wait complete. 2024-03-19T02:31:28.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 104320 3.26 MB/sec execute 51 sec latency 14.763 ms 2024-03-19T02:31:29.469 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:31:29.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:31:29.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.586+0000 7fdb92309640 1 -- 172.21.15.116:0/2854212041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 msgr2=0x7fdb8c1007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.586+0000 7fdb92309640 1 --2- 172.21.15.116:0/2854212041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1007a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009900 tx=0x7fdb7c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.586+0000 7fdb92309640 1 -- 172.21.15.116:0/2854212041 shutdown_connections 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.586+0000 7fdb92309640 1 --2- 172.21.15.116:0/2854212041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 0x7fdb8c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.586+0000 7fdb92309640 1 --2- 172.21.15.116:0/2854212041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c100d70 0x7fdb8c106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 --2- 172.21.15.116:0/2854212041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1007a0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 -- 172.21.15.116:0/2854212041 >> 172.21.15.116:0/2854212041 conn(0x7fdb8c075ad0 msgr2=0x7fdb8c075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 -- 172.21.15.116:0/2854212041 shutdown_connections 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 -- 172.21.15.116:0/2854212041 wait complete. 2024-03-19T02:31:29.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 Processor -- start 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.587+0000 7fdb92309640 1 -- start start 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 0x7fdb8c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 0x7fdb8c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c111b30 con 0x7fdb8c1003c0 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1119b0 con 0x7fdb8c19c9b0 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c111cb0 con 0x7fdb8c106c40 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39542/0 (socket says 172.21.15.116:39542) 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2602061823 learned_addr learned my addr 172.21.15.116:0/2602061823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 0x7fdb8c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 0x7fdb8c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2602061823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 msgr2=0x7fdb8c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 0x7fdb8c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2602061823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 msgr2=0x7fdb8c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 0x7fdb8c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2602061823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8c19d4a0 con 0x7fdb8c1003c0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 0x7fdb8c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005940 tx=0x7fdb7c02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb897fa640 1 -- 172.21.15.116:0/2602061823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c043020 con 0x7fdb8c1003c0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8c19d730 con 0x7fdb8c1003c0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb897fa640 1 -- 172.21.15.116:0/2602061823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb7c02ca70 con 0x7fdb8c1003c0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.588+0000 7fdb897fa640 1 -- 172.21.15.116:0/2602061823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c003a80 con 0x7fdb8c1003c0 2024-03-19T02:31:29.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.589+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8c1a4f50 con 0x7fdb8c1003c0 2024-03-19T02:31:29.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb897fa640 1 -- 172.21.15.116:0/2602061823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdb7c046020 con 0x7fdb8c1003c0 2024-03-19T02:31:29.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 Processor -- start 2024-03-19T02:31:29.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb897fa640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 0x7fdb740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 -- start start 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 0x7fdb740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb897fa640 1 -- 172.21.15.116:0/2602061823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdb7c0f1100 con 0x7fdb8c1003c0 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c49340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 0x562b00c4bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b00ba4160 con 0x562b00c40ca0 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b00b4ce10 con 0x562b00c40380 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.591+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b00ba0200 con 0x562b00c49880 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c49340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51248/0 (socket says 172.21.15.116:51248) 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2628840336 learned_addr learned my addr 172.21.15.116:0/2628840336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c49340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39552/0 (socket says 172.21.15.116:39552) 2024-03-19T02:31:29.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 0x562b00c4bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 0x7fdb740a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdb84005fd0 tx=0x7fdb84009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2628840336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 msgr2=0x562b00c4bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 0x562b00c4bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2628840336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 msgr2=0x562b00c49340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c49340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 -- 172.21.15.116:0/2628840336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b009807e0 con 0x562b00c40380 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb908a6640 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 0x562b00c4bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0b9950 tx=0x7fdb7c0bce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=704093a57e3a19ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb727fc640 1 -- 172.21.15.116:0/2628840336 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0bd4f0 con 0x562b00c40380 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb727fc640 1 -- 172.21.15.116:0/2628840336 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb7c0c1640 con 0x562b00c40380 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.592+0000 7fdb727fc640 1 -- 172.21.15.116:0/2628840336 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0b9cf0 con 0x562b00c40380 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- 172.21.15.116:0/2628840336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 msgr2=0x562b00c40760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0b9950 tx=0x7fdb7c0bce50 comp rx=0 tx=0).stop 2024-03-19T02:31:29.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- 172.21.15.116:0/2628840336 shutdown_connections 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c49880 0x562b00c4bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c49340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/2628840336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c40760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- 172.21.15.116:0/2628840336 >> 172.21.15.116:0/2628840336 conn(0x562b00c3aab0 msgr2=0x562b00c3ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- 172.21.15.116:0/2628840336 shutdown_connections 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- 172.21.15.116:0/2628840336 wait complete. 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.593+0000 7fdb94fd2740 1 -- start start 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 Processor -- start 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c467e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c46d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b00c4c430 con 0x562b00c49880 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b00b601a0 con 0x562b00c40380 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b00c4c5b0 con 0x562b00c40ca0 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb94fd2740 10 client.0 _ll_register_callbacks cb 0x562b008b7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c46d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39554/0 (socket says 172.21.15.116:39554) 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 -- 172.21.15.116:0/3899322306 learned_addr learned my addr 172.21.15.116:0/3899322306 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 -- 172.21.15.116:0/3899322306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 msgr2=0x562b00c46d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c46d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 -- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 msgr2=0x562b00c467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb908a6640 1 -- 172.21.15.116:0/3899322306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b009e0f70 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.594+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb908a6640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400fa40 tx=0x7fdb84004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e678093f0b71a7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84004420 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdb840045c0 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb7c0c1640 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdb84004800 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 10 client.16830 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562b00878170 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb7c0c1640 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdb8401c420 con 0x562b00c49880 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 10 client.16830 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 10 client.16830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 20 client.16830 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:29.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00c59930 0x562b00c5bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 4 client.16830 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.595+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b00bb16a0 con 0x562b00c59930 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.596+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00c59930 0x562b00c5bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.596+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00c59930 0x562b00c5bd70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eb48e7045151f2b server_cookie=e50dd050479c2c8 in_seq=0 out_seq=0 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.596+0000 7fdb6b7fe640 10 client.16830 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3899322306 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b00bb16a0 con 0x562b00c59930 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb6b7fe640 10 client.16830 handle_command_reply: tid=0 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 client.16830 shutdown 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 msgr2=0x562b00c47260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400fa40 tx=0x7fdb84004ab0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 shutdown_connections 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00c59930 0x562b00c5bd70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c49880 0x562b00c47260 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb6b7fe640 0 client.16830 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c46d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3899322306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c40380 0x562b00c467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 >> 172.21.15.116:0/3899322306 conn(0x562b00c3aab0 msgr2=0x562b00c3c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 shutdown_connections 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3899322306 wait complete. 2024-03-19T02:31:29.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.597+0000 7fdb94fd2740 20 client.16830 trim_cache size 0 max 16384 2024-03-19T02:31:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 Processor -- start 2024-03-19T02:31:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 -- start start 2024-03-19T02:31:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 0x562b00c48f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c55060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00c3c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1ac920 con 0x562b00c40380 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1ae460 con 0x562b00c49880 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1acff0 con 0x562b00c40ca0 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.620+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 0x562b00c48f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c55060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 0x562b00c48f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39562/0 (socket says 172.21.15.116:39562) 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8bfff640 1 -- 172.21.15.116:0/760343033 learned_addr learned my addr 172.21.15.116:0/760343033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb908a6640 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00c3c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/760343033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 msgr2=0x562b00c3c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00c3c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/760343033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 msgr2=0x562b00c48f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 0x562b00c48f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/760343033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b00c3a8e0 con 0x562b00c40ca0 2024-03-19T02:31:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c55060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x562b00c3fe50 tx=0x7fdb80005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c70fca689f29a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb71ffb640 1 -- 172.21.15.116:0/760343033 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80026070 con 0x562b00c40ca0 2024-03-19T02:31:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb71ffb640 1 -- 172.21.15.116:0/760343033 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb800165c0 con 0x562b00c40ca0 2024-03-19T02:31:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.621+0000 7fdb71ffb640 1 -- 172.21.15.116:0/760343033 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8000fd50 con 0x562b00c40ca0 2024-03-19T02:31:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 -- 172.21.15.116:0/760343033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 msgr2=0x562b00c55060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c55060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x562b00c3fe50 tx=0x7fdb80005f90 comp rx=0 tx=0).stop 2024-03-19T02:31:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 -- 172.21.15.116:0/760343033 shutdown_connections 2024-03-19T02:31:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00c3c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40ca0 0x562b00c55060 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/760343033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40380 0x562b00c48f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 -- 172.21.15.116:0/760343033 >> 172.21.15.116:0/760343033 conn(0x562b00c3aab0 msgr2=0x562b00c3be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 -- 172.21.15.116:0/760343033 shutdown_connections 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.622+0000 7fdb94fd2740 1 -- 172.21.15.116:0/760343033 wait complete. 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 -- start start 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 Processor -- start 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 0x562b00c43b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00cbec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1add90 con 0x562b00c40ca0 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1ad6c0 con 0x562b00c49880 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1aa900 con 0x562b00c40380 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb94fd2740 10 client.0 _ll_register_callbacks cb 0x7fdb84003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb8bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 0x562b00c43b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39572/0 (socket says 172.21.15.116:39572) 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/3163617328 learned_addr learned my addr 172.21.15.116:0/3163617328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.623+0000 7fdb908a6640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00cbec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/3163617328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 msgr2=0x562b00c43b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 0x562b00c43b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 msgr2=0x562b00cbec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00cbec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 -- 172.21.15.116:0/3163617328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b00cbf1e0 con 0x562b00c40ca0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 0x562b00c43b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb908a6640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00cbec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb8b7fe640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x562b00c46040 tx=0x7fdb80009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fae5acde0de0bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb80026040 con 0x562b00c40ca0 2024-03-19T02:31:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb800165c0 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdb80004050 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdb80038050 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb94fd2740 10 client.16836 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562b00cc0500 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.624+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb800165c0 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdb800035f0 con 0x562b00c40ca0 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 10 client.16836 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 10 client.16836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 20 client.16836 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00cc0c30 0x562b00cc3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 4 client.16836 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:31:29.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562b00bab9f0 con 0x562b00cc0c30 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00cc0c30 0x562b00cc3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb6b7fe640 10 client.16836 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.625+0000 7fdb8bfff640 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00cc0c30 0x562b00cc3050 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c069f6e9db62c2 server_cookie=d0cabc5818fee018 in_seq=0 out_seq=0 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb6b7fe640 1 -- 172.21.15.116:0/3163617328 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562b00bab9f0 con 0x562b00cc0c30 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb6b7fe640 10 client.16836 handle_command_reply: tid=0 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 client.16836 shutdown 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 msgr2=0x562b00c440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x562b00c46040 tx=0x7fdb80009fb0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 shutdown_connections 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b00cc0c30 0x562b00cc3050 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b00c49880 0x562b00cbec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b00c40ca0 0x562b00c440a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.626+0000 7fdb94fd2740 1 --2- 172.21.15.116:0/3163617328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b00c40380 0x562b00c43b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb6b7fe640 0 client.16836 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 >> 172.21.15.116:0/3163617328 conn(0x562b00c3aab0 msgr2=0x562b00c47030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 shutdown_connections 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb94fd2740 1 -- 172.21.15.116:0/3163617328 wait complete. 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb94fd2740 20 client.16836 trim_cache size 0 max 16384 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "15561b5e-0d3a-404b-a5b1-8f354c35ccf0", 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:31:29.633 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:31:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 msgr2=0x7fdb740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 0x7fdb740a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdb84005fd0 tx=0x7fdb84009500 comp rx=0 tx=0).stop 2024-03-19T02:31:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 msgr2=0x7fdb8c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005940 tx=0x7fdb7c02c5f0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 shutdown_connections 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb740a2970 0x7fdb740a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb8c19c9b0 0x7fdb8c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb8c106c40 0x7fdb8c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 --2- 172.21.15.116:0/2602061823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c1003c0 0x7fdb8c1980f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 >> 172.21.15.116:0/2602061823 conn(0x7fdb8c075ad0 msgr2=0x7fdb8c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 shutdown_connections 2024-03-19T02:31:29.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.627+0000 7fdb92309640 1 -- 172.21.15.116:0/2602061823 wait complete. 2024-03-19T02:31:29.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:31:29.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:31:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.754+0000 7f52170c0640 1 -- 172.21.15.116:0/1154744421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101066a0 msgr2=0x7f5210106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.754+0000 7f52170c0640 1 --2- 172.21.15.116:0/1154744421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101066a0 0x7f5210106a80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5204009930 tx=0x7f520402bc30 comp rx=0 tx=0).stop 2024-03-19T02:31:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- 172.21.15.116:0/1154744421 shutdown_connections 2024-03-19T02:31:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- 172.21.15.116:0/1154744421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210101050 0x7f521010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- 172.21.15.116:0/1154744421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5210100690 0x7f5210100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- 172.21.15.116:0/1154744421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101066a0 0x7f5210106a80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- 172.21.15.116:0/1154744421 >> 172.21.15.116:0/1154744421 conn(0x7f5210077820 msgr2=0x7f5210077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- 172.21.15.116:0/1154744421 shutdown_connections 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- 172.21.15.116:0/1154744421 wait complete. 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 Processor -- start 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- start start 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210100690 0x7f5210198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52101066a0 0x7f52101a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5210110920 con 0x7f5210101050 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52101107a0 con 0x7f5210100690 2024-03-19T02:31:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.755+0000 7f52170c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5210110aa0 con 0x7f52101066a0 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39600/0 (socket says 172.21.15.116:39600) 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 -- 172.21.15.116:0/500031430 learned_addr learned my addr 172.21.15.116:0/500031430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f5214e5c640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210100690 0x7f5210198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f521565d640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52101066a0 0x7f52101a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 -- 172.21.15.116:0/500031430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52101066a0 msgr2=0x7f52101a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52101066a0 0x7f52101a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 -- 172.21.15.116:0/500031430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210100690 msgr2=0x7f5210198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210100690 0x7f5210198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 -- 172.21.15.116:0/500031430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52101a43c0 con 0x7f5210101050 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520ffff640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00e9f0 tx=0x7f51fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520dffb640 1 -- 172.21.15.116:0/500031430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00cd80 con 0x7f5210101050 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52101a4590 con 0x7f5210101050 2024-03-19T02:31:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520dffb640 1 -- 172.21.15.116:0/500031430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51fc0108e0 con 0x7f5210101050 2024-03-19T02:31:29.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f520dffb640 1 -- 172.21.15.116:0/500031430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc018920 con 0x7f5210101050 2024-03-19T02:31:29.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.756+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52101a4ae0 con 0x7f5210101050 2024-03-19T02:31:29.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f520dffb640 1 -- 172.21.15.116:0/500031430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f51fc010a90 con 0x7f5210101050 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 Processor -- start 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 -- start start 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5adf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560ae5a37500 con 0x560ae5ad3a30 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560ae59dfbe0 con 0x560ae5ad4350 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5219d86740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560ae5a335a0 con 0x560ae5adcf30 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f520dffb640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 0x7f52000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f521565d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f520ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5adf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51286/0 (socket says 172.21.15.116:51286) 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f521565d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39610/0 (socket says 172.21.15.116:39610) 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.759+0000 7f5214e5c640 1 -- 172.21.15.116:0/311612874 learned_addr learned my addr 172.21.15.116:0/311612874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f521565d640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 0x7f52000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f520dffb640 1 -- 172.21.15.116:0/500031430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f51fc0cf440 con 0x7f5210101050 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 -- 172.21.15.116:0/311612874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 msgr2=0x560ae5adf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5adf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 -- 172.21.15.116:0/311612874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 msgr2=0x560ae5ad3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 -- 172.21.15.116:0/311612874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ae5a01000 con 0x560ae5ad4350 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f521565d640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 0x7f52000a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f5208009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f5214e5c640 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5204036040 tx=0x7f5204009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af7893fd6cd923 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f51faffd640 1 -- 172.21.15.116:0/311612874 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204045020 con 0x560ae5ad4350 2024-03-19T02:31:29.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f51faffd640 1 -- 172.21.15.116:0/311612874 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f520402c7e0 con 0x560ae5ad4350 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.760+0000 7f51faffd640 1 -- 172.21.15.116:0/311612874 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204005450 con 0x560ae5ad4350 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- 172.21.15.116:0/311612874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 msgr2=0x560ae5adc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5204036040 tx=0x7f5204009560 comp rx=0 tx=0).stop 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- 172.21.15.116:0/311612874 shutdown_connections 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5adf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad4350 0x560ae5adc9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 --2- 172.21.15.116:0/311612874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- 172.21.15.116:0/311612874 >> 172.21.15.116:0/311612874 conn(0x560ae5ace160 msgr2=0x560ae5ace540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- 172.21.15.116:0/311612874 shutdown_connections 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- 172.21.15.116:0/311612874 wait complete. 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.761+0000 7f5219d86740 1 -- start start 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 Processor -- start 2024-03-19T02:31:29.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad4350 0x560ae5ada4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5adaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560ae5adfcc0 con 0x560ae5ad3a30 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560ae59f3160 con 0x560ae5adcf30 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560ae5adfe40 con 0x560ae5ad4350 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5219d86740 10 client.0 _ll_register_callbacks cb 0x560ae5a2fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39616/0 (socket says 172.21.15.116:39616) 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 -- 172.21.15.116:0/3726980868 learned_addr learned my addr 172.21.15.116:0/3726980868 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f521565d640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5adaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f520ffff640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad4350 0x560ae5ada4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 -- 172.21.15.116:0/3726980868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad4350 msgr2=0x560ae5ada4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad4350 0x560ae5ada4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 -- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 msgr2=0x560ae5adaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5adaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.762+0000 7f5214e5c640 1 -- 172.21.15.116:0/3726980868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ae58157d0 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5214e5c640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5204005470 tx=0x7f52040054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcefbc5d4ad7430 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204050070 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f5204004450 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f520402c7e0 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5204059050 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 10 client.16848 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ae574b3f0 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f520402c7e0 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5204053020 con 0x560ae5ad3a30 2024-03-19T02:31:29.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 10 client.16848 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 20 client.16848 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5af1510 0x560ae5af3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 4 client.16848 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.763+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ae5a44a40 con 0x560ae5af1510 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.764+0000 7f520ffff640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5af1510 0x560ae5af3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.764+0000 7f520ffff640 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5af1510 0x560ae5af3900 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cde3bdfb0e15e6d server_cookie=96a6cfe2e89ba6b1 in_seq=0 out_seq=0 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.764+0000 7f51effff640 10 client.16848 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f51effff640 1 -- 172.21.15.116:0/3726980868 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ae5a44a40 con 0x560ae5af1510 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f51effff640 10 client.16848 handle_command_reply: tid=0 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 client.16848 shutdown 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 msgr2=0x560ae5ad9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5204005470 tx=0x7f52040054a0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 shutdown_connections 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5af1510 0x560ae5af3900 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5adaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad4350 0x560ae5ada4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 --2- 172.21.15.116:0/3726980868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad3a30 0x560ae5ad9f80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 >> 172.21.15.116:0/3726980868 conn(0x560ae5ace160 msgr2=0x560ae5acfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f51effff640 0 client.16848 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 shutdown_connections 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 1 -- 172.21.15.116:0/3726980868 wait complete. 2024-03-19T02:31:29.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.765+0000 7f5219d86740 20 client.16848 trim_cache size 0 max 16384 2024-03-19T02:31:29.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 104972 3.21 MB/sec execute 52 sec latency 22.593 ms 2024-03-19T02:31:29.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.783+0000 7f5219d86740 1 Processor -- start 2024-03-19T02:31:29.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.783+0000 7f5219d86740 1 -- start start 2024-03-19T02:31:29.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 0x560ae5adc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5ae8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52101ac8d0 con 0x560ae5ad4350 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52101ae410 con 0x560ae5adcf30 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5219d86740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52101acfa0 con 0x560ae5ad3a30 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39626/0 (socket says 172.21.15.116:39626) 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 0x560ae5adc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 -- 172.21.15.116:0/147433262 learned_addr learned my addr 172.21.15.116:0/147433262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f521565d640 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5ae8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 -- 172.21.15.116:0/147433262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 msgr2=0x560ae5adc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 0x560ae5adc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 -- 172.21.15.116:0/147433262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 msgr2=0x560ae5ae8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5ae8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 -- 172.21.15.116:0/147433262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ae570b0f0 con 0x560ae5ad4350 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f5214e5c640 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 0x560ae5adc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f520ffff640 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097cd0 tx=0x7f51fc06e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb1f895f3a7f19a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f51fa7fc640 1 -- 172.21.15.116:0/147433262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc026040 con 0x560ae5ad4350 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f51fa7fc640 1 -- 172.21.15.116:0/147433262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51fc09b870 con 0x560ae5ad4350 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.784+0000 7f51fa7fc640 1 -- 172.21.15.116:0/147433262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc097ee0 con 0x560ae5ad4350 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 -- 172.21.15.116:0/147433262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 msgr2=0x560ae5ad90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097cd0 tx=0x7f51fc06e610 comp rx=0 tx=0).stop 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 -- 172.21.15.116:0/147433262 shutdown_connections 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5adcf30 0x560ae5ae8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad90e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 --2- 172.21.15.116:0/147433262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5ad3a30 0x560ae5adc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 -- 172.21.15.116:0/147433262 >> 172.21.15.116:0/147433262 conn(0x560ae5ace160 msgr2=0x560ae5acc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 -- 172.21.15.116:0/147433262 shutdown_connections 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.785+0000 7f5219d86740 1 -- 172.21.15.116:0/147433262 wait complete. 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 -- start start 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 Processor -- start 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 0x560ae5ad6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5ad7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52101add40 con 0x560ae5ad4350 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52101ad670 con 0x560ae5ad3a30 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52101aa8b0 con 0x560ae5adcf30 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5219d86740 10 client.0 _ll_register_callbacks cb 0x7f5208009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5214e5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 0x560ae5ad6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39628/0 (socket says 172.21.15.116:39628) 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 -- 172.21.15.116:0/4036671252 learned_addr learned my addr 172.21.15.116:0/4036671252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f521565d640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5ad7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 -- 172.21.15.116:0/4036671252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 msgr2=0x560ae5ad7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5ad7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 -- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 msgr2=0x560ae5ad6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 0x560ae5ad6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f520ffff640 1 -- 172.21.15.116:0/4036671252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ae5b566a0 con 0x560ae5ad4350 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f521565d640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5ad7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.786+0000 7f5214e5c640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 0x560ae5ad6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f520ffff640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097c80 tx=0x7f51fc0a40e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dea2d8b5b0d742 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0af060 con 0x560ae5ad4350 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f51fc0ccb30 con 0x560ae5ad4350 2024-03-19T02:31:29.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51fc09b870 con 0x560ae5ad4350 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51fc010de0 con 0x560ae5ad4350 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 10 client.16854 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ae5b57990 con 0x560ae5ad4350 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51fc09b870 con 0x560ae5ad4350 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f51fc0aaba0 con 0x560ae5ad4350 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 10 client.16854 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 10 client.16854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 20 client.16854 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5b57fb0 0x560ae5b5a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 4 client.16854 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.787+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ae5a3ed90 con 0x560ae5b57fb0 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.788+0000 7f5214e5c640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5b57fb0 0x560ae5b5a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:29.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.788+0000 7f5214e5c640 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5b57fb0 0x560ae5b5a3d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d40e45f49a4733 server_cookie=86afe2386f9ca388 in_seq=0 out_seq=0 2024-03-19T02:31:29.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.788+0000 7f51effff640 10 client.16854 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f51effff640 1 -- 172.21.15.116:0/4036671252 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560ae5a3ed90 con 0x560ae5b57fb0 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f51effff640 10 client.16854 handle_command_reply: tid=0 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 client.16854 shutdown 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 msgr2=0x560ae5ad7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097c80 tx=0x7f51fc0a40e0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 shutdown_connections 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ae5b57fb0 0x560ae5b5a3d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ae5adcf30 0x560ae5ad7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f51effff640 0 client.16854 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:29.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ae5ad4350 0x560ae5ad7220 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 --2- 172.21.15.116:0/4036671252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ae5ad3a30 0x560ae5ad6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 >> 172.21.15.116:0/4036671252 conn(0x560ae5ace160 msgr2=0x560ae5acf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 shutdown_connections 2024-03-19T02:31:29.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.789+0000 7f5219d86740 1 -- 172.21.15.116:0/4036671252 wait complete. 2024-03-19T02:31:29.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f5219d86740 20 client.16854 trim_cache size 0 max 16384 2024-03-19T02:31:29.799 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:31:29.799 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:31:29.799 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:31:29.799 INFO:teuthology.orchestra.run.smithi116.stdout: "15561b5e-0d3a-404b-a5b1-8f354c35ccf0": { 2024-03-19T02:31:29.799 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:31:29.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "15561b5e-0d3a-404b-a5b1-8f354c35ccf0", 2024-03-19T02:31:29.800 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:31:29.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:31:29.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:31:29.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:31:29.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 msgr2=0x7f52000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 0x7f52000a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f5208009500 comp rx=0 tx=0).stop 2024-03-19T02:31:29.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 msgr2=0x7f5210199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:29.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00e9f0 tx=0x7f51fc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 shutdown_connections 2024-03-19T02:31:29.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f52000a2970 0x7f52000a4e30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52101066a0 0x7f52101a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5210101050 0x7f5210199340 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 --2- 172.21.15.116:0/500031430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5210100690 0x7f5210198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 >> 172.21.15.116:0/500031430 conn(0x7f5210077820 msgr2=0x7f5210102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 shutdown_connections 2024-03-19T02:31:29.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:29.790+0000 7f52170c0640 1 -- 172.21.15.116:0/500031430 wait complete. 2024-03-19T02:31:29.805 INFO:tasks.cephfs.filesystem:scrub status for tag:15561b5e-0d3a-404b-a5b1-8f354c35ccf0 - {'path': '/', 'tag': '15561b5e-0d3a-404b-a5b1-8f354c35ccf0', 'options': 'recursive,force'} 2024-03-19T02:31:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:29 smithi162 ceph-mon[35761]: pgmap v113: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:29 smithi154 ceph-mon[35693]: pgmap v113: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:29 smithi116 ceph-mon[32263]: pgmap v113: 129 pgs: 129 active+clean; 87 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.3 MiB/s wr, 83 op/s 2024-03-19T02:31:30.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 105528 3.19 MB/sec execute 53 sec latency 94.909 ms 2024-03-19T02:31:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:31:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:31:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:31:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:30 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:30 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:30 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:31:31.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 106242 3.25 MB/sec execute 54 sec latency 22.699 ms 2024-03-19T02:31:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:31 smithi162 ceph-mon[35761]: pgmap v114: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:31:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:31 smithi154 ceph-mon[35693]: pgmap v114: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:31:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:31 smithi116 ceph-mon[32263]: pgmap v114: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-03-19T02:31:32.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 106831 3.28 MB/sec execute 55 sec latency 17.391 ms 2024-03-19T02:31:33.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 107275 3.23 MB/sec execute 56 sec latency 30.813 ms 2024-03-19T02:31:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:33 smithi162 ceph-mon[35761]: pgmap v115: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:31:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:33 smithi154 ceph-mon[35693]: pgmap v115: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:31:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:33 smithi116 ceph-mon[32263]: pgmap v115: 129 pgs: 129 active+clean; 89 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:31:34.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 107922 3.25 MB/sec execute 57 sec latency 16.395 ms 2024-03-19T02:31:35.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 108590 3.21 MB/sec execute 58 sec latency 22.271 ms 2024-03-19T02:31:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:35 smithi162 ceph-mon[35761]: pgmap v116: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:35 smithi154 ceph-mon[35693]: pgmap v116: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:35 smithi116 ceph-mon[32263]: pgmap v116: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:36.782 INFO:tasks.workunit.client.0.smithi116.stdout: 1 109341 3.25 MB/sec execute 59 sec latency 16.207 ms 2024-03-19T02:31:37.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 109829 3.25 MB/sec execute 60 sec latency 19.240 ms 2024-03-19T02:31:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:37 smithi162 ceph-mon[35761]: pgmap v117: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:37 smithi154 ceph-mon[35693]: pgmap v117: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:37 smithi116 ceph-mon[32263]: pgmap v117: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:38.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 110431 3.27 MB/sec execute 61 sec latency 17.067 ms 2024-03-19T02:31:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 110921 3.23 MB/sec execute 62 sec latency 44.779 ms 2024-03-19T02:31:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:39 smithi162 ceph-mon[35761]: pgmap v118: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:39 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:31:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:39 smithi154 ceph-mon[35693]: pgmap v118: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:39 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:31:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:39 smithi116 ceph-mon[32263]: pgmap v118: 129 pgs: 129 active+clean; 94 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:31:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:39 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:31:40.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 111558 3.25 MB/sec execute 63 sec latency 21.966 ms 2024-03-19T02:31:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:31:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:31:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:31:41.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 112255 3.21 MB/sec execute 64 sec latency 22.169 ms 2024-03-19T02:31:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:41 smithi162 ceph-mon[35761]: pgmap v119: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:31:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:41 smithi154 ceph-mon[35693]: pgmap v119: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:31:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:41 smithi116 ceph-mon[32263]: pgmap v119: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:31:42.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 112989 3.26 MB/sec execute 65 sec latency 16.429 ms 2024-03-19T02:31:43.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 113525 3.28 MB/sec execute 66 sec latency 18.781 ms 2024-03-19T02:31:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:43 smithi162 ceph-mon[35761]: pgmap v120: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:43 smithi154 ceph-mon[35693]: pgmap v120: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:43 smithi116 ceph-mon[32263]: pgmap v120: 129 pgs: 129 active+clean; 96 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:44.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 114016 3.26 MB/sec execute 67 sec latency 21.787 ms 2024-03-19T02:31:45.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 114580 3.24 MB/sec execute 68 sec latency 43.639 ms 2024-03-19T02:31:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:45 smithi162 ceph-mon[35761]: pgmap v121: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:31:46.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:45 smithi154 ceph-mon[35693]: pgmap v121: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:31:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:45 smithi116 ceph-mon[32263]: pgmap v121: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:31:46.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 115229 3.24 MB/sec execute 69 sec latency 15.522 ms 2024-03-19T02:31:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 115909 3.22 MB/sec execute 70 sec latency 15.837 ms 2024-03-19T02:31:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:47 smithi162 ceph-mon[35761]: pgmap v122: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:47 smithi154 ceph-mon[35693]: pgmap v122: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:47 smithi116 ceph-mon[32263]: pgmap v122: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:48.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 116808 3.28 MB/sec execute 71 sec latency 14.903 ms 2024-03-19T02:31:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 117219 3.28 MB/sec execute 72 sec latency 16.132 ms 2024-03-19T02:31:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:49 smithi162 ceph-mon[35761]: pgmap v123: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:49 smithi154 ceph-mon[35693]: pgmap v123: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:49 smithi116 ceph-mon[32263]: pgmap v123: 129 pgs: 129 active+clean; 98 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:31:50.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 117654 3.25 MB/sec execute 73 sec latency 15.773 ms 2024-03-19T02:31:51.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 118224 3.24 MB/sec execute 74 sec latency 22.340 ms 2024-03-19T02:31:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:51 smithi162 ceph-mon[35761]: pgmap v124: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:51 smithi154 ceph-mon[35693]: pgmap v124: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:51 smithi116 ceph-mon[32263]: pgmap v124: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 2.1 MiB/s wr, 124 op/s 2024-03-19T02:31:52.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 118922 3.24 MB/sec execute 75 sec latency 16.488 ms 2024-03-19T02:31:53.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 119539 3.22 MB/sec execute 76 sec latency 17.956 ms 2024-03-19T02:31:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:53 smithi162 ceph-mon[35761]: pgmap v125: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:31:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:53 smithi154 ceph-mon[35693]: pgmap v125: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:31:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:53 smithi116 ceph-mon[32263]: pgmap v125: 129 pgs: 129 active+clean; 100 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:31:54.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 120380 3.27 MB/sec execute 77 sec latency 19.352 ms 2024-03-19T02:31:55.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 120867 3.29 MB/sec execute 78 sec latency 18.069 ms 2024-03-19T02:31:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:55 smithi162 ceph-mon[35761]: pgmap v126: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:55 smithi154 ceph-mon[35693]: pgmap v126: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:55 smithi116 ceph-mon[32263]: pgmap v126: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:31:56.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 121243 3.25 MB/sec execute 79 sec latency 44.079 ms 2024-03-19T02:31:57.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 121812 3.23 MB/sec execute 80 sec latency 16.531 ms 2024-03-19T02:31:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:57 smithi162 ceph-mon[35761]: pgmap v127: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:57 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:31:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:57 smithi154 ceph-mon[35693]: pgmap v127: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:57 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:31:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:57 smithi116 ceph-mon[32263]: pgmap v127: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:31:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:57 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:31:58.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 122480 3.24 MB/sec execute 81 sec latency 32.453 ms 2024-03-19T02:31:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:58 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:31:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:58 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:31:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:58 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:31:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:58 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:31:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:58 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:31:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:58 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:31:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:58 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:31:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:58 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:31:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:58 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:31:59.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 123111 3.23 MB/sec execute 82 sec latency 18.167 ms 2024-03-19T02:31:59.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:31:59.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:31:59.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 -- 172.21.15.116:0/2980103931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 msgr2=0x7f2d8c10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2980103931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c10e110 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009ca0 tx=0x7f2d8402c180 comp rx=0 tx=0).stop 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 -- 172.21.15.116:0/2980103931 shutdown_connections 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2980103931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c10e110 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2980103931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c103230 0x7f2d8c10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2980103931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c102910 0x7f2d8c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 -- 172.21.15.116:0/2980103931 >> 172.21.15.116:0/2980103931 conn(0x7f2d8c0fc070 msgr2=0x7f2d8c0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 -- 172.21.15.116:0/2980103931 shutdown_connections 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.919+0000 7f2d9159d640 1 -- 172.21.15.116:0/2980103931 wait complete. 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 Processor -- start 2024-03-19T02:31:59.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 -- start start 2024-03-19T02:31:59.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 0x7f2d8c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c111920 con 0x7f2d8c10bce0 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1117a0 con 0x7f2d8c103230 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d9159d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c111aa0 con 0x7f2d8c102910 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52974/0 (socket says 172.21.15.116:52974) 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d8bfff640 1 -- 172.21.15.116:0/2962379742 learned_addr learned my addr 172.21.15.116:0/2962379742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.920+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 0x7f2d8c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 -- 172.21.15.116:0/2962379742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 msgr2=0x7f2d8c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 -- 172.21.15.116:0/2962379742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 msgr2=0x7f2d8c19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 0x7f2d8c19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 -- 172.21.15.116:0/2962379742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c19f050 con 0x7f2d8c10bce0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 0x7f2d8c19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c19e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2d84037040 tx=0x7f2d84035d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d897fa640 1 -- 172.21.15.116:0/2962379742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84039040 con 0x7f2d8c10bce0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8c19f2e0 con 0x7f2d8c10bce0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d897fa640 1 -- 172.21.15.116:0/2962379742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d84044070 con 0x7f2d8c10bce0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d897fa640 1 -- 172.21.15.116:0/2962379742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84004860 con 0x7f2d8c10bce0 2024-03-19T02:31:59.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.921+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8c1a6bd0 con 0x7f2d8c10bce0 2024-03-19T02:31:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d897fa640 1 -- 172.21.15.116:0/2962379742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2d84003d00 con 0x7f2d8c10bce0 2024-03-19T02:31:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 Processor -- start 2024-03-19T02:31:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 -- start start 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d897fa640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 0x7f2d740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4c510 0x55cf33b54bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b574e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cf33aaf500 con 0x55cf33b4bbf0 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cf33a57be0 con 0x55cf33b4c510 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d953ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cf33aab5a0 con 0x55cf33b550f0 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 0x7f2d740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d897fa640 1 -- 172.21.15.116:0/2962379742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2d840f13a0 con 0x7f2d8c10bce0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36090/0 (socket says 172.21.15.116:36090) 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1654166773 learned_addr learned my addr 172.21.15.116:0/1654166773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.924+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4c510 0x55cf33b54bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1654166773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 msgr2=0x55cf33b574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1654166773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4c510 msgr2=0x55cf33b54bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4c510 0x55cf33b54bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1654166773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf33a79000 con 0x55cf33b4bbf0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c011650 tx=0x7f2d7c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfdfbbb6f3e953 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 0x7f2d740a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c102580 tx=0x7f2d80009eb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d727fc640 1 -- 172.21.15.116:0/1654166773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c01e070 con 0x55cf33b4bbf0 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d727fc640 1 -- 172.21.15.116:0/1654166773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d7c00fb70 con 0x55cf33b4bbf0 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.925+0000 7f2d727fc640 1 -- 172.21.15.116:0/1654166773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c019bc0 con 0x55cf33b4bbf0 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- 172.21.15.116:0/1654166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 msgr2=0x55cf33b4bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c011650 tx=0x7f2d7c009f90 comp rx=0 tx=0).stop 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- 172.21.15.116:0/1654166773 shutdown_connections 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4c510 0x55cf33b54bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1654166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4bbf0 0x55cf33b4bfd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- 172.21.15.116:0/1654166773 >> 172.21.15.116:0/1654166773 conn(0x55cf33b46320 msgr2=0x55cf33b46700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- 172.21.15.116:0/1654166773 shutdown_connections 2024-03-19T02:31:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- 172.21.15.116:0/1654166773 wait complete. 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 -- start start 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.926+0000 7f2d953ee740 1 Processor -- start 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b52570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cf33b57ee0 con 0x55cf33b550f0 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cf33a6b160 con 0x55cf33b4bbf0 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cf33b58060 con 0x55cf33b4c510 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 10 client.0 _ll_register_callbacks cb 0x55cf33aa7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36092/0 (socket says 172.21.15.116:36092) 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 -- 172.21.15.116:0/3594048833 learned_addr learned my addr 172.21.15.116:0/3594048833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47714/0 (socket says 172.21.15.116:47714) 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d8b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b52570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 -- 172.21.15.116:0/3594048833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 msgr2=0x55cf33b52570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b52570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 -- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 msgr2=0x55cf33b52030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 -- 172.21.15.116:0/3594048833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf3388d7d0 con 0x55cf33b550f0 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b52570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2d840b9bf0 tx=0x7f2d840bd2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f244b4a0b04e2289 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8403eed0 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2d840b9e00 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.927+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d7c00fb70 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2d840bde90 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 10 client.16866 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf337c33f0 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d7c00fb70 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2d8403c030 con 0x55cf33b550f0 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 10 client.16866 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 20 client.16866 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33b65410 0x55cf33b67840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 4 client.16866 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf33abca40 con 0x55cf33b65410 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.928+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33b65410 0x55cf33b67840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d6b7fe640 10 client.16866 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33b65410 0x55cf33b67840 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d2d3aba9439ea9 server_cookie=75c043dc6c4f3444 in_seq=0 out_seq=0 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/3594048833 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf33abca40 con 0x55cf33b65410 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d6b7fe640 10 client.16866 handle_command_reply: tid=0 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 client.16866 shutdown 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 msgr2=0x55cf33b52ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2d840b9bf0 tx=0x7f2d840bd2d0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 shutdown_connections 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33b65410 0x55cf33b67840 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.929+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b52ab0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d6b7fe640 0 client.16866 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b52570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 1 --2- 172.21.15.116:0/3594048833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b52030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 >> 172.21.15.116:0/3594048833 conn(0x55cf33b46320 msgr2=0x55cf33b48070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 shutdown_connections 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 1 -- 172.21.15.116:0/3594048833 wait complete. 2024-03-19T02:31:59.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.930+0000 7f2d953ee740 20 client.16866 trim_cache size 0 max 16384 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.950+0000 7f2d953ee740 1 Processor -- start 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.950+0000 7f2d953ee740 1 -- start start 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b54760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b60c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1b0b00 con 0x55cf33b550f0 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1b2640 con 0x55cf33b4bbf0 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d953ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1b11d0 con 0x55cf33b4c510 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b54760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36094/0 (socket says 172.21.15.116:36094) 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 -- 172.21.15.116:0/4149877989 learned_addr learned my addr 172.21.15.116:0/4149877989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b60c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 -- 172.21.15.116:0/4149877989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 msgr2=0x55cf33b60c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b60c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 -- 172.21.15.116:0/4149877989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 msgr2=0x55cf33b54760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b54760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 -- 172.21.15.116:0/4149877989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf337830f0 con 0x55cf33b550f0 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2d840ba160 tx=0x7f2d84031420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60727e81f7d6811b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d71ffb640 1 -- 172.21.15.116:0/4149877989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8403eed0 con 0x55cf33b550f0 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d71ffb640 1 -- 172.21.15.116:0/4149877989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d840bd790 con 0x55cf33b550f0 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.951+0000 7f2d71ffb640 1 -- 172.21.15.116:0/4149877989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d840bd0f0 con 0x55cf33b550f0 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- 172.21.15.116:0/4149877989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 msgr2=0x55cf33b47e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2d840ba160 tx=0x7f2d84031420 comp rx=0 tx=0).stop 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- 172.21.15.116:0/4149877989 shutdown_connections 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b550f0 0x55cf33b47e60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b4c510 0x55cf33b60c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 --2- 172.21.15.116:0/4149877989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- 172.21.15.116:0/4149877989 >> 172.21.15.116:0/4149877989 conn(0x55cf33b46320 msgr2=0x55cf33b47750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- 172.21.15.116:0/4149877989 shutdown_connections 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- 172.21.15.116:0/4149877989 wait complete. 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.952+0000 7f2d953ee740 1 -- start start 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 Processor -- start 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b5d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b58b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1b1f70 con 0x55cf33b4c510 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1b18a0 con 0x55cf33b4bbf0 2024-03-19T02:31:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c1aeae0 con 0x55cf33b550f0 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d953ee740 10 client.0 _ll_register_callbacks cb 0x7f2d84009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36102/0 (socket says 172.21.15.116:36102) 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b5d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8b7fe640 1 -- 172.21.15.116:0/1415587517 learned_addr learned my addr 172.21.15.116:0/1415587517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d90d9c640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b58b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1415587517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 msgr2=0x55cf33b58b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b58b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1415587517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 msgr2=0x55cf33b58640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:31:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1415587517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf33b590f0 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.953+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b5d2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c01c040 tx=0x7f2d7c019510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d86a3cee6be716eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c01e040 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d840bd790 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2d7c0050b0 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2d7c038050 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 10 client.26690 fetch_fsmap learned FSMap version 30 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf33bcb480 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d840bd790 con 0x55cf33b4bbf0 2024-03-19T02:31:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2d7c02a680 con 0x55cf33b4bbf0 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 10 client.26690 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 10 client.26690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 20 client.26690 populate_metadata read hostname 'smithi116' 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33bcba90 0x55cf33bcdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.954+0000 7f2d953ee740 4 client.26690 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cf33ab6d90 con 0x55cf33bcba90 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33bcba90 0x55cf33bcdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d6b7fe640 10 client.26690 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d8b7fe640 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33bcba90 0x55cf33bcdeb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31176a94ac4f8214 server_cookie=baf586cc2ef66994 in_seq=0 out_seq=0 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d6b7fe640 1 -- 172.21.15.116:0/1415587517 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cf33ab6d90 con 0x55cf33bcba90 2024-03-19T02:31:59.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.955+0000 7f2d6b7fe640 10 client.26690 handle_command_reply: tid=0 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 client.26690 shutdown 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 msgr2=0x55cf33b5d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b5d2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c01c040 tx=0x7f2d7c019510 comp rx=0 tx=0).stop 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 shutdown_connections 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cf33bcba90 0x55cf33bcdeb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cf33b550f0 0x55cf33b58b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf33b4c510 0x55cf33b58640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d6b7fe640 0 client.26690 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 --2- 172.21.15.116:0/1415587517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cf33b4bbf0 0x55cf33b5d2d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 >> 172.21.15.116:0/1415587517 conn(0x55cf33b46320 msgr2=0x55cf33b454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 shutdown_connections 2024-03-19T02:31:59.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 1 -- 172.21.15.116:0/1415587517 wait complete. 2024-03-19T02:31:59.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d953ee740 20 client.26690 trim_cache size 0 max 16384 2024-03-19T02:31:59.965 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:31:59.965 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:31:59.965 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:31:59.965 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:31:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 msgr2=0x7f2d740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 0x7f2d740a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c102580 tx=0x7f2d80009eb0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 msgr2=0x7f2d8c19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:31:59.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.956+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c19e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2d84037040 tx=0x7f2d84035d50 comp rx=0 tx=0).stop 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 shutdown_connections 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2d740a2970 0x7f2d740a4e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c10bce0 0x7f2d8c19e920 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d8c103230 0x7f2d8c19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 --2- 172.21.15.116:0/2962379742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d8c102910 0x7f2d8c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 >> 172.21.15.116:0/2962379742 conn(0x7f2d8c0fc070 msgr2=0x7f2d8c0fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 shutdown_connections 2024-03-19T02:31:59.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:31:59.957+0000 7f2d9159d640 1 -- 172.21.15.116:0/2962379742 wait complete. 2024-03-19T02:31:59.968 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:31:59.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 -- 172.21.15.116:0/1133474033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 msgr2=0x7f5e24100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 --2- 172.21.15.116:0/1133474033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e24100ba0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009930 tx=0x7f5e0c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 -- 172.21.15.116:0/1133474033 shutdown_connections 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 --2- 172.21.15.116:0/1133474033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 0x7f5e2410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 --2- 172.21.15.116:0/1133474033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e24100ba0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 --2- 172.21.15.116:0/1133474033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 0x7f5e24106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 -- 172.21.15.116:0/1133474033 >> 172.21.15.116:0/1133474033 conn(0x7f5e24077820 msgr2=0x7f5e24077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 -- 172.21.15.116:0/1133474033 shutdown_connections 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.080+0000 7f5e28e34640 1 -- 172.21.15.116:0/1133474033 wait complete. 2024-03-19T02:32:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 Processor -- start 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 -- start start 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 0x7f5e24198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 0x7f5e2419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24110a40 con 0x7f5e24100720 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241108c0 con 0x7f5e241010e0 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e28e34640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24110bc0 con 0x7f5e24106730 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36126/0 (socket says 172.21.15.116:36126) 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 -- 172.21.15.116:0/700496742 learned_addr learned my addr 172.21.15.116:0/700496742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e22d9d640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 0x7f5e2419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e1bfff640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 0x7f5e24198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 -- 172.21.15.116:0/700496742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 msgr2=0x7f5e2419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 0x7f5e2419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 -- 172.21.15.116:0/700496742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 msgr2=0x7f5e24198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 0x7f5e24198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 -- 172.21.15.116:0/700496742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2419d0c0 con 0x7f5e24100720 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.081+0000 7f5e2259c640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400e9f0 tx=0x7f5e1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.082+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1400cd80 con 0x7f5e24100720 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.082+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2419d3b0 con 0x7f5e24100720 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.082+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e140054d0 con 0x7f5e24100720 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.082+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14010490 con 0x7f5e24100720 2024-03-19T02:32:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.082+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e241a4c40 con 0x7f5e24100720 2024-03-19T02:32:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.083+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e24068610 con 0x7f5e24100720 2024-03-19T02:32:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.084+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5e14005020 con 0x7f5e24100720 2024-03-19T02:32:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.084+0000 7f5e1b7fe640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 0x7f5e100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.085+0000 7f5e1bfff640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 0x7f5e100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.085+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f5e140cf4d0 con 0x7f5e24100720 2024-03-19T02:32:00.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.085+0000 7f5e1bfff640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 0x7f5e100a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009900 tx=0x7f5e0c005a20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.090+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e1409beb0 con 0x7f5e24100720 2024-03-19T02:32:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:31:59 smithi162 ceph-mon[35761]: pgmap v128: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:31:59 smithi154 ceph-mon[35693]: pgmap v128: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:31:59 smithi116 ceph-mon[32263]: pgmap v128: 129 pgs: 129 active+clean; 102 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.344+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e24109ca0 con 0x7f5e24100720 2024-03-19T02:32:00.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.345+0000 7f5e1b7fe640 1 -- 172.21.15.116:0/700496742 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f5e1409b7b0 con 0x7f5e24100720 2024-03-19T02:32:00.346 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:32:00.346 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:32:00.346 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:32:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.347+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 msgr2=0x7f5e100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.347+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 0x7f5e100a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009900 tx=0x7f5e0c005a20 comp rx=0 tx=0).stop 2024-03-19T02:32:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.347+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 msgr2=0x7f5e241980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.347+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400e9f0 tx=0x7f5e1400edf0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 shutdown_connections 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e100a2970 0x7f5e100a4e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24106730 0x7f5e2419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e241010e0 0x7f5e24198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 --2- 172.21.15.116:0/700496742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e24100720 0x7f5e241980d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 >> 172.21.15.116:0/700496742 conn(0x7f5e24077820 msgr2=0x7f5e24068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 shutdown_connections 2024-03-19T02:32:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.348+0000 7f5e28e34640 1 -- 172.21.15.116:0/700496742 wait complete. 2024-03-19T02:32:00.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 -- 172.21.15.116:0/1422761516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c102a40 msgr2=0x7ff14c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 --2- 172.21.15.116:0/1422761516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c102a40 0x7ff14c102e20 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009900 tx=0x7ff13c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 -- 172.21.15.116:0/1422761516 shutdown_connections 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 --2- 172.21.15.116:0/1422761516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c10bd80 0x7ff14c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 --2- 172.21.15.116:0/1422761516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c103360 0x7ff14c10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 --2- 172.21.15.116:0/1422761516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c102a40 0x7ff14c102e20 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 -- 172.21.15.116:0/1422761516 >> 172.21.15.116:0/1422761516 conn(0x7ff14c0778d0 msgr2=0x7ff14c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 -- 172.21.15.116:0/1422761516 shutdown_connections 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.469+0000 7ff1518f8640 1 -- 172.21.15.116:0/1422761516 wait complete. 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 Processor -- start 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 -- start start 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c102a40 0x7ff14c100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c103360 0x7ff14c100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c111a30 con 0x7ff14c10bd80 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1118b0 con 0x7ff14c103360 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff1518f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c111bb0 con 0x7ff14c102a40 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36140/0 (socket says 172.21.15.116:36140) 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 -- 172.21.15.116:0/413741243 learned_addr learned my addr 172.21.15.116:0/413741243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff143fff640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c103360 0x7ff14c100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14affd640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c102a40 0x7ff14c100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 -- 172.21.15.116:0/413741243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c102a40 msgr2=0x7ff14c100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c102a40 0x7ff14c100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 -- 172.21.15.116:0/413741243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c103360 msgr2=0x7ff14c100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c103360 0x7ff14c100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.470+0000 7ff14b7fe640 1 -- 172.21.15.116:0/413741243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c1a6430 con 0x7ff14c10bd80 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff14b7fe640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff14400eb10 tx=0x7ff14400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff1508f6640 1 -- 172.21.15.116:0/413741243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144004b20 con 0x7ff14c10bd80 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff1508f6640 1 -- 172.21.15.116:0/413741243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff144004cc0 con 0x7ff14c10bd80 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c1a6720 con 0x7ff14c10bd80 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff1508f6640 1 -- 172.21.15.116:0/413741243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144010690 con 0x7ff14c10bd80 2024-03-19T02:32:00.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.471+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c1a6c20 con 0x7ff14c10bd80 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.473+0000 7ff1508f6640 1 -- 172.21.15.116:0/413741243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff144005020 con 0x7ff14c10bd80 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.473+0000 7ff1545c0740 1 Processor -- start 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 -- start start 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75fefcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d75fe544b0 con 0x55d75fefa7d0 2024-03-19T02:32:00.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d75fdfcd70 con 0x55d75fef1bf0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1545c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d75fe50550 con 0x55d75fef12d0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1508f6640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 0x7ff1380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff143fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff143fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53082/0 (socket says 172.21.15.116:53082) 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff143fff640 1 -- 172.21.15.116:0/1694623069 learned_addr learned my addr 172.21.15.116:0/1694623069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff143fff640 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75fefcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14b7fe640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 0x7ff1380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff1508f6640 1 -- 172.21.15.116:0/413741243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff1440cf750 con 0x7ff14c10bd80 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 -- 172.21.15.116:0/1694623069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 msgr2=0x55d75fef16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 -- 172.21.15.116:0/1694623069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 msgr2=0x55d75fefcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75fefcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff14affd640 1 -- 172.21.15.116:0/1694623069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75fe1f210 con 0x55d75fef1bf0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.474+0000 7ff143fff640 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff14b7fe640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 0x7ff1380a4e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff14409b960 tx=0x7ff144066740 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff14affd640 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefa290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff13c000c00 tx=0x7ff13c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcaa5d4ef701d8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff141ffb640 1 -- 172.21.15.116:0/1694623069 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c004040 con 0x55d75fef1bf0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff141ffb640 1 -- 172.21.15.116:0/1694623069 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff13c0054f0 con 0x55d75fef1bf0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff141ffb640 1 -- 172.21.15.116:0/1694623069 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c02cd80 con 0x55d75fef1bf0 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff1545c0740 1 -- 172.21.15.116:0/1694623069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 msgr2=0x55d75fefa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.475+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefa290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff13c000c00 tx=0x7ff13c0095f0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- 172.21.15.116:0/1694623069 shutdown_connections 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75fefcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefa290 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1694623069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fef16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- 172.21.15.116:0/1694623069 >> 172.21.15.116:0/1694623069 conn(0x55d75feeba00 msgr2=0x55d75feebde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- 172.21.15.116:0/1694623069 shutdown_connections 2024-03-19T02:32:00.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- 172.21.15.116:0/1694623069 wait complete. 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- start start 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 Processor -- start 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef1bf0 0x55d75fef7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75fef8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d75fefd560 con 0x55d75fef12d0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d75fe10310 con 0x55d75fefa7d0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d75fefd6e0 con 0x55d75fef1bf0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.476+0000 7ff1545c0740 10 client.0 _ll_register_callbacks cb 0x55d75fe042d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36154/0 (socket says 172.21.15.116:36154) 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 -- 172.21.15.116:0/3995452057 learned_addr learned my addr 172.21.15.116:0/3995452057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff14affd640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef1bf0 0x55d75fef7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75fef8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 -- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef1bf0 msgr2=0x55d75fef7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef1bf0 0x55d75fef7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 -- 172.21.15.116:0/3995452057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 msgr2=0x55d75fef8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75fef8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 -- 172.21.15.116:0/3995452057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75fb683f0 con 0x55d75fef12d0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff143fff640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff1340037a0 tx=0x7ff134005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b568bfc50a6c0537 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff12affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134011bb0 con 0x55d75fef12d0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff134011d50 con 0x55d75fef12d0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff13c0054f0 con 0x55d75fef12d0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff13402b050 con 0x55d75fef12d0 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff1545c0740 10 client.16890 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.477+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d75fc327d0 con 0x55d75fef12d0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff13c0054f0 con 0x55d75fef12d0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff134004b10 con 0x55d75fef12d0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 10 client.16890 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 10 client.16890 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 20 client.16890 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff0cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 4 client.16890 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d75feb23e0 con 0x55d75ff0ab40 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff14affd640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff0cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff12affd640 10 client.16890 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.478+0000 7ff14affd640 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff0cfa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3841de10f7ada0eb server_cookie=1c30f1ad31d20339 in_seq=0 out_seq=0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff12affd640 1 -- 172.21.15.116:0/3995452057 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d75feb23e0 con 0x55d75ff0ab40 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff12affd640 10 client.16890 handle_command_reply: tid=0 2024-03-19T02:32:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 client.16890 shutdown 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 msgr2=0x55d75fef7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff1340037a0 tx=0x7ff134005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 shutdown_connections 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff0cfa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75fef8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff12affd640 0 client.16890 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef1bf0 0x55d75fef7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 --2- 172.21.15.116:0/3995452057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef12d0 0x55d75fef7800 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 >> 172.21.15.116:0/3995452057 conn(0x55d75feeba00 msgr2=0x55d75feed750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 shutdown_connections 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 1 -- 172.21.15.116:0/3995452057 wait complete. 2024-03-19T02:32:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.479+0000 7ff1545c0740 20 client.16890 trim_cache size 0 max 16384 2024-03-19T02:32:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 Processor -- start 2024-03-19T02:32:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 -- start start 2024-03-19T02:32:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75ff06300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75ff12870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1b0c90 con 0x55d75fef1bf0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1b27d0 con 0x55d75fefa7d0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff1545c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1b1360 con 0x55d75fef12d0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36168/0 (socket says 172.21.15.116:36168) 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 -- 172.21.15.116:0/1240123063 learned_addr learned my addr 172.21.15.116:0/1240123063 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff143fff640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75ff06300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff143fff640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75ff12870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 -- 172.21.15.116:0/1240123063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 msgr2=0x55d75ff06300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75ff06300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 -- 172.21.15.116:0/1240123063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 msgr2=0x55d75ff12870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75ff12870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff14affd640 1 -- 172.21.15.116:0/1240123063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75fef1a20 con 0x55d75fef1bf0 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.493+0000 7ff143fff640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75ff06300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff14affd640 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff13c000c00 tx=0x7ff13c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434e3ddffe48ce63 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1417fa640 1 -- 172.21.15.116:0/1240123063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c050aa0 con 0x55d75fef1bf0 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1417fa640 1 -- 172.21.15.116:0/1240123063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff13c004030 con 0x55d75fef1bf0 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1417fa640 1 -- 172.21.15.116:0/1240123063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c005280 con 0x55d75fef1bf0 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 -- 172.21.15.116:0/1240123063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 msgr2=0x55d75fef6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff13c000c00 tx=0x7ff13c005e40 comp rx=0 tx=0).stop 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 -- 172.21.15.116:0/1240123063 shutdown_connections 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fefa7d0 0x55d75ff12870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fef1bf0 0x55d75fef6fb0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1240123063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75ff06300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.494+0000 7ff1545c0740 1 -- 172.21.15.116:0/1240123063 >> 172.21.15.116:0/1240123063 conn(0x55d75feeba00 msgr2=0x55d75feeaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- 172.21.15.116:0/1240123063 shutdown_connections 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- 172.21.15.116:0/1240123063 wait complete. 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- start start 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 Processor -- start 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fefdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefe040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1b2100 con 0x55d75fefa7d0 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1b1a30 con 0x55d75fef1bf0 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1aec70 con 0x55d75fef12d0 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff1545c0740 10 client.0 _ll_register_callbacks cb 0x7ff1340035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36180/0 (socket says 172.21.15.116:36180) 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff143fff640 1 -- 172.21.15.116:0/1625080985 learned_addr learned my addr 172.21.15.116:0/1625080985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff143fff640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fefdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.495+0000 7ff14affd640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 -- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 msgr2=0x55d75fefdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fefdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 -- 172.21.15.116:0/1625080985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 msgr2=0x55d75fefe040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefe040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 -- 172.21.15.116:0/1625080985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75ff694a0 con 0x55d75fefa7d0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff143fff640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff1340225c0 tx=0x7ff134005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a405aa2b4fb9e6dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff134011b90 con 0x55d75fefa7d0 2024-03-19T02:32:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff134011d30 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff13c004030 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff1340047a0 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 10 client.16896 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d75ff70810 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff13c004030 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff134004fd0 con 0x55d75fefa7d0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 10 client.16896 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 10 client.16896 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 20 client.16896 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff72f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 4 client.16896 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.496+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d75fe619f0 con 0x55d75ff0ab40 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff14affd640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff72f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff14affd640 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff72f40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=693121018c190314 server_cookie=208cedca866c00b9 in_seq=0 out_seq=0 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff12affd640 10 client.16896 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff12affd640 1 -- 172.21.15.116:0/1625080985 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d75fe619f0 con 0x55d75ff0ab40 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff12affd640 10 client.16896 handle_command_reply: tid=0 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 client.16896 shutdown 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 msgr2=0x55d75ff68e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff1340225c0 tx=0x7ff134005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 shutdown_connections 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d75ff0ab40 0x55d75ff72f40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d75fefa7d0 0x55d75ff68e60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff12affd640 0 client.16896 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d75fef1bf0 0x55d75fefe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 --2- 172.21.15.116:0/1625080985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d75fef12d0 0x55d75fefdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 >> 172.21.15.116:0/1625080985 conn(0x55d75feeba00 msgr2=0x55d75feea310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 shutdown_connections 2024-03-19T02:32:00.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 1 -- 172.21.15.116:0/1625080985 wait complete. 2024-03-19T02:32:00.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.497+0000 7ff1545c0740 20 client.16896 trim_cache size 0 max 16384 2024-03-19T02:32:00.506 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 msgr2=0x7ff1380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 0x7ff1380a4e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff14409b960 tx=0x7ff144066740 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 msgr2=0x7ff14c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff14400eb10 tx=0x7ff14400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 shutdown_connections 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff1380a29a0 0x7ff1380a4e60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff14c10bd80 0x7ff14c101110 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c103360 0x7ff14c100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 --2- 172.21.15.116:0/413741243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff14c102a40 0x7ff14c100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 >> 172.21.15.116:0/413741243 conn(0x7ff14c0778d0 msgr2=0x7ff14c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 shutdown_connections 2024-03-19T02:32:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.498+0000 7ff1518f8640 1 -- 172.21.15.116:0/413741243 wait complete. 2024-03-19T02:32:00.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:32:00.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 -- 172.21.15.116:0/718724815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1066b0 msgr2=0x7eff6c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 --2- 172.21.15.116:0/718724815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1066b0 0x7eff6c106a90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7eff68009930 tx=0x7eff6802bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 -- 172.21.15.116:0/718724815 shutdown_connections 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 --2- 172.21.15.116:0/718724815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c101060 0x7eff6c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 --2- 172.21.15.116:0/718724815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c1006a0 0x7eff6c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 --2- 172.21.15.116:0/718724815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1066b0 0x7eff6c106a90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.631+0000 7eff73dcd640 1 -- 172.21.15.116:0/718724815 >> 172.21.15.116:0/718724815 conn(0x7eff6c0778d0 msgr2=0x7eff6c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- 172.21.15.116:0/718724815 shutdown_connections 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- 172.21.15.116:0/718724815 wait complete. 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 Processor -- start 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- start start 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c101060 0x7eff6c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c1066b0 0x7eff6c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c110970 con 0x7eff6c1006a0 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1107f0 con 0x7eff6c1066b0 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.632+0000 7eff73dcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c110af0 con 0x7eff6c101060 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36188/0 (socket says 172.21.15.116:36188) 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 -- 172.21.15.116:0/1303936512 learned_addr learned my addr 172.21.15.116:0/1303936512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71368640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c101060 0x7eff6c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff7236a640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c1066b0 0x7eff6c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 -- 172.21.15.116:0/1303936512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c101060 msgr2=0x7eff6c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c101060 0x7eff6c198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 -- 172.21.15.116:0/1303936512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c1066b0 msgr2=0x7eff6c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c1066b0 0x7eff6c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 -- 172.21.15.116:0/1303936512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c19cfd0 con 0x7eff6c1006a0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff71b69640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7eff68009230 tx=0x7eff6802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff62ffd640 1 -- 172.21.15.116:0/1303936512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68043020 con 0x7eff6c1006a0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff62ffd640 1 -- 172.21.15.116:0/1303936512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff6802ca70 con 0x7eff6c1006a0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff62ffd640 1 -- 172.21.15.116:0/1303936512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68003a50 con 0x7eff6c1006a0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6c19d260 con 0x7eff6c1006a0 2024-03-19T02:32:00.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.633+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6c1a4b50 con 0x7eff6c1006a0 2024-03-19T02:32:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.636+0000 7eff62ffd640 1 -- 172.21.15.116:0/1303936512 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7eff68046020 con 0x7eff6c1006a0 2024-03-19T02:32:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.636+0000 7eff769ba740 1 Processor -- start 2024-03-19T02:32:00.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 -- start start 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff62ffd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 0x7eff580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea7012c4b0 con 0x55ea701d27d0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ea700d4d70 con 0x55ea701c92d0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff769ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea70128550 con 0x55ea701c9bf0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff7236a640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 0x7eff580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71b69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53140/0 (socket says 172.21.15.116:53140) 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 -- 172.21.15.116:0/2073141839 learned_addr learned my addr 172.21.15.116:0/2073141839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff7236a640 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff62ffd640 1 -- 172.21.15.116:0/1303936512 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7eff680f10b0 con 0x7eff6c1006a0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 -- 172.21.15.116:0/2073141839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 msgr2=0x55ea701c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 -- 172.21.15.116:0/2073141839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 msgr2=0x55ea701d4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.637+0000 7eff71368640 1 -- 172.21.15.116:0/2073141839 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea700f7210 con 0x55ea701c9bf0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff7236a640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 0x7eff580a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff64005fd0 tx=0x7eff64009e70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff71368640 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009ca0 tx=0x7eff5c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c63ecdd0717c8b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff57fff640 1 -- 172.21.15.116:0/2073141839 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c01e070 con 0x55ea701c9bf0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff57fff640 1 -- 172.21.15.116:0/2073141839 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5c019ae0 con 0x55ea701c9bf0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff57fff640 1 -- 172.21.15.116:0/2073141839 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00e8e0 con 0x55ea701c9bf0 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff769ba740 1 -- 172.21.15.116:0/2073141839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 msgr2=0x55ea701d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff769ba740 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009ca0 tx=0x7eff5c011810 comp rx=0 tx=0).stop 2024-03-19T02:32:00.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff769ba740 1 -- 172.21.15.116:0/2073141839 shutdown_connections 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.638+0000 7eff769ba740 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701d2290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 --2- 172.21.15.116:0/2073141839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- 172.21.15.116:0/2073141839 >> 172.21.15.116:0/2073141839 conn(0x55ea701c3a00 msgr2=0x55ea701c3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- 172.21.15.116:0/2073141839 shutdown_connections 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- 172.21.15.116:0/2073141839 wait complete. 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- start start 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 Processor -- start 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea701d55c0 con 0x55ea701d27d0 2024-03-19T02:32:00.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ea700e8310 con 0x55ea701c92d0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.639+0000 7eff769ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea701d5740 con 0x55ea701c9bf0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff769ba740 10 client.0 _ll_register_callbacks cb 0x55ea700dc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71b69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff7236a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff7236a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36204/0 (socket says 172.21.15.116:36204) 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53148/0 (socket says 172.21.15.116:53148) 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff7236a640 1 -- 172.21.15.116:0/1412068167 learned_addr learned my addr 172.21.15.116:0/1412068167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71b69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701cf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47842/0 (socket says 172.21.15.116:47842) 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 -- 172.21.15.116:0/1412068167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 msgr2=0x55ea701cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 -- 172.21.15.116:0/1412068167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 msgr2=0x55ea701d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 -- 172.21.15.116:0/1412068167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea6fe403f0 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff7236a640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff71368640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009c70 tx=0x7eff5c005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c1f89a024a6a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff557fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c027020 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5c019ae0 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7eff5c005460 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff5c032050 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff769ba740 10 client.26665 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.640+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ea6ff0a7d0 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5c019ae0 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7eff5c00e7d0 con 0x55ea701c9bf0 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 10 client.26665 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 10 client.26665 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 20 client.26665 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea701e2a50 0x55ea701e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 4 client.26665 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea7018a3e0 con 0x55ea701e2a50 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff71b69640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea701e2a50 0x55ea701e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff71b69640 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea701e2a50 0x55ea701e4e40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e6de0419d849fad1 server_cookie=7d2e396456bccd33 in_seq=0 out_seq=0 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.641+0000 7eff557fa640 10 client.26665 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff557fa640 1 -- 172.21.15.116:0/1412068167 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea7018a3e0 con 0x55ea701e2a50 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff557fa640 10 client.26665 handle_command_reply: tid=0 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 client.26665 shutdown 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 msgr2=0x55ea701cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009c70 tx=0x7eff5c005d30 comp rx=0 tx=0).stop 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 shutdown_connections 2024-03-19T02:32:00.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea701e2a50 0x55ea701e4e40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cfd30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff557fa640 0 client.26665 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 --2- 172.21.15.116:0/1412068167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 >> 172.21.15.116:0/1412068167 conn(0x55ea701c3a00 msgr2=0x55ea701c5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 shutdown_connections 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.642+0000 7eff769ba740 1 -- 172.21.15.116:0/1412068167 wait complete. 2024-03-19T02:32:00.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.643+0000 7eff769ba740 20 client.26665 trim_cache size 0 max 16384 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 Processor -- start 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 -- start start 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701d27d0 0x55ea701de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1ac950 con 0x55ea701c9bf0 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1ae490 con 0x55ea701c92d0 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff769ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1ad020 con 0x55ea701d27d0 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff71368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff71368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36210/0 (socket says 172.21.15.116:36210) 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.657+0000 7eff71368640 1 -- 172.21.15.116:0/44171633 learned_addr learned my addr 172.21.15.116:0/44171633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff7236a640 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701d27d0 0x55ea701de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71b69640 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 -- 172.21.15.116:0/44171633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701d27d0 msgr2=0x55ea701de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701d27d0 0x55ea701de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 -- 172.21.15.116:0/44171633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 msgr2=0x55ea701d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 -- 172.21.15.116:0/44171633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea701e53d0 con 0x55ea701c9bf0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff71368640 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00eba0 tx=0x7eff5c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707b2e4b8c4dbff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff577fe640 1 -- 172.21.15.116:0/44171633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c027040 con 0x55ea701c9bf0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff577fe640 1 -- 172.21.15.116:0/44171633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5c01f040 con 0x55ea701c9bf0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.658+0000 7eff577fe640 1 -- 172.21.15.116:0/44171633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c01e070 con 0x55ea701c9bf0 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- 172.21.15.116:0/44171633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 msgr2=0x55ea701c2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00eba0 tx=0x7eff5c005970 comp rx=0 tx=0).stop 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- 172.21.15.116:0/44171633 shutdown_connections 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701d27d0 0x55ea701de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701c9bf0 0x55ea701c2de0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 --2- 172.21.15.116:0/44171633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- 172.21.15.116:0/44171633 >> 172.21.15.116:0/44171633 conn(0x55ea701c3a00 msgr2=0x55ea701d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- 172.21.15.116:0/44171633 shutdown_connections 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- 172.21.15.116:0/44171633 wait complete. 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 -- start start 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.659+0000 7eff769ba740 1 Processor -- start 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1addc0 con 0x55ea701d27d0 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1ad6f0 con 0x55ea701c92d0 2024-03-19T02:32:00.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c1aa930 con 0x55ea701c9bf0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 10 client.0 _ll_register_callbacks cb 0x7eff5c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36226/0 (socket says 172.21.15.116:36226) 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 -- 172.21.15.116:0/492290098 learned_addr learned my addr 172.21.15.116:0/492290098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff71b69640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff71368640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 -- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 msgr2=0x55ea701cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 -- 172.21.15.116:0/492290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 msgr2=0x55ea701dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701dabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 -- 172.21.15.116:0/492290098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea701d4bc0 con 0x55ea701d27d0 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff71b69640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701dabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff7236a640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff6400fc50 tx=0x7eff64009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75714e74ede84d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff557fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6400fc80 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7eff6401a040 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5c01f040 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff64036050 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 10 client.16914 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.660+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ea70248910 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5c01f040 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7eff6401e070 con 0x55ea701d27d0 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 10 client.16914 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 10 client.16914 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 20 client.16914 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea70249110 0x55ea7024b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 4 client.16914 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea701399f0 con 0x55ea70249110 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff71b69640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea70249110 0x55ea7024b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff557fa640 10 client.16914 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.661+0000 7eff71b69640 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea70249110 0x55ea7024b530 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f1d8e9a1e85b053 server_cookie=71f72879eb29de69 in_seq=0 out_seq=0 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff557fa640 1 -- 172.21.15.116:0/492290098 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea701399f0 con 0x55ea70249110 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff557fa640 10 client.16914 handle_command_reply: tid=0 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 client.16914 shutdown 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 msgr2=0x55ea701cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7eff6400fc50 tx=0x7eff64009500 comp rx=0 tx=0).stop 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 shutdown_connections 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55ea70249110 0x55ea7024b530 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea701d27d0 0x55ea701cc560 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea701c9bf0 0x55ea701cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 --2- 172.21.15.116:0/492290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea701c92d0 0x55ea701dabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff557fa640 0 client.16914 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 >> 172.21.15.116:0/492290098 conn(0x55ea701c3a00 msgr2=0x55ea701cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 shutdown_connections 2024-03-19T02:32:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 1 -- 172.21.15.116:0/492290098 wait complete. 2024-03-19T02:32:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff769ba740 20 client.16914 trim_cache size 0 max 16384 2024-03-19T02:32:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 msgr2=0x7eff580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 0x7eff580a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7eff64005fd0 tx=0x7eff64009e70 comp rx=0 tx=0).stop 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 msgr2=0x7eff6c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7eff68009230 tx=0x7eff6802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 shutdown_connections 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.662+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7eff580a2970 0x7eff580a4e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c1066b0 0x7eff6c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff6c101060 0x7eff6c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 --2- 172.21.15.116:0/1303936512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c1006a0 0x7eff6c197fe0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 >> 172.21.15.116:0/1303936512 conn(0x7eff6c0778d0 msgr2=0x7eff6c068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 shutdown_connections 2024-03-19T02:32:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.663+0000 7eff73dcd640 1 -- 172.21.15.116:0/1303936512 wait complete. 2024-03-19T02:32:00.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:32:00.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 123924 3.27 MB/sec execute 83 sec latency 19.382 ms 2024-03-19T02:32:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 -- 172.21.15.116:0/4188492375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01066b0 msgr2=0x7fdda0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 --2- 172.21.15.116:0/4188492375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01066b0 0x7fdda0106a90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009900 tx=0x7fdd9002bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 -- 172.21.15.116:0/4188492375 shutdown_connections 2024-03-19T02:32:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 --2- 172.21.15.116:0/4188492375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda0101060 0x7fdda010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 --2- 172.21.15.116:0/4188492375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01006a0 0x7fdda0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 --2- 172.21.15.116:0/4188492375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda01066b0 0x7fdda0106a90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 -- 172.21.15.116:0/4188492375 >> 172.21.15.116:0/4188492375 conn(0x7fdda00778d0 msgr2=0x7fdda0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 -- 172.21.15.116:0/4188492375 shutdown_connections 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.785+0000 7fdda5297640 1 -- 172.21.15.116:0/4188492375 wait complete. 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 Processor -- start 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 -- start start 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda010f8a0 con 0x7fdda01066b0 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda010f720 con 0x7fdda0101060 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdda5297640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda010fa20 con 0x7fdda01006a0 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36244/0 (socket says 172.21.15.116:36244) 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47890/0 (socket says 172.21.15.116:47890) 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd9f7fe640 1 -- 172.21.15.116:0/2602012295 learned_addr learned my addr 172.21.15.116:0/2602012295 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.786+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53184/0 (socket says 172.21.15.116:53184) 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 -- 172.21.15.116:0/2602012295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 msgr2=0x7fdda019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 -- 172.21.15.116:0/2602012295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 msgr2=0x7fdda019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 -- 172.21.15.116:0/2602012295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda019f160 con 0x7fdda01066b0 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9effd640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9f7fe640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800efd0 tx=0x7fdd9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9cff9640 1 -- 172.21.15.116:0/2602012295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98003b70 con 0x7fdda01066b0 2024-03-19T02:32:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda019f450 con 0x7fdda01066b0 2024-03-19T02:32:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9cff9640 1 -- 172.21.15.116:0/2602012295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd98003d10 con 0x7fdda01066b0 2024-03-19T02:32:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdd9cff9640 1 -- 172.21.15.116:0/2602012295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98010980 con 0x7fdda01066b0 2024-03-19T02:32:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.787+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda01a6ce0 con 0x7fdda01066b0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd9cff9640 1 -- 172.21.15.116:0/2602012295 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdd98004020 con 0x7fdda01066b0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 Processor -- start 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 -- start start 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 0x5573ec503bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573ec45b4b0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573ec403d70 con 0x5573ec4f82d0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdda7e81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5573ec457550 con 0x5573ec5017d0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36250/0 (socket says 172.21.15.116:36250) 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd9effd640 1 -- 172.21.15.116:0/2874032517 learned_addr learned my addr 172.21.15.116:0/2874032517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4f86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47900/0 (socket says 172.21.15.116:47900) 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.790+0000 7fdd9cff9640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 0x7fdd8c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 0x5573ec503bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9f7fe640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 0x7fdd8c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 -- 172.21.15.116:0/2874032517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 msgr2=0x5573ec503bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 0x5573ec503bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 -- 172.21.15.116:0/2874032517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 msgr2=0x5573ec4f86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4f86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 -- 172.21.15.116:0/2874032517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573ec426210 con 0x5573ec4f8bf0 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 0x5573ec503bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9cff9640 1 -- 172.21.15.116:0/2602012295 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd980cf600 con 0x7fdda01066b0 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9effd640 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009900 tx=0x7fdd900095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea18ea986b5a2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd957fa640 1 -- 172.21.15.116:0/2874032517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90045070 con 0x5573ec4f8bf0 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd9f7fe640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 0x7fdd8c0a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd980cca80 tx=0x7fdd98090d70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd957fa640 1 -- 172.21.15.116:0/2874032517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd9002c7e0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.791+0000 7fdd957fa640 1 -- 172.21.15.116:0/2874032517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90003ab0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- 172.21.15.116:0/2874032517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 msgr2=0x5573ec501290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009900 tx=0x7fdd900095f0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- 172.21.15.116:0/2874032517 shutdown_connections 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec5017d0 0x5573ec503bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec501290 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2874032517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4f86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- 172.21.15.116:0/2874032517 >> 172.21.15.116:0/2874032517 conn(0x5573ec4f2a00 msgr2=0x5573ec4f2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- 172.21.15.116:0/2874032517 shutdown_connections 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- 172.21.15.116:0/2874032517 wait complete. 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 -- start start 2024-03-19T02:32:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.792+0000 7fdda7e81740 1 Processor -- start 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec4fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec5017d0 0x5573ec4ff250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573ec504560 con 0x5573ec4f8bf0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573ec417310 con 0x5573ec5017d0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5573ec5046e0 con 0x5573ec4f82d0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 10 client.0 _ll_register_callbacks cb 0x5573ec40b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36264/0 (socket says 172.21.15.116:36264) 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 -- 172.21.15.116:0/3211726441 learned_addr learned my addr 172.21.15.116:0/3211726441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd97fff640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec4fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd97fff640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec5017d0 0x5573ec4ff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 -- 172.21.15.116:0/3211726441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 msgr2=0x5573ec4fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec4fe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 -- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec5017d0 msgr2=0x5573ec4ff250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec5017d0 0x5573ec4ff250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 -- 172.21.15.116:0/3211726441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573ec16f3f0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd9effd640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdd90036040 tx=0x7fdd90005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a697aeedc66954d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd867fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9004e070 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd9003fc30 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.793+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd9002c7e0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdd9005a050 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 10 client.16926 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5573ec2397d0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd9002c7e0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdd90034440 con 0x5573ec4f8bf0 2024-03-19T02:32:00.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 10 client.16926 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 10 client.16926 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 20 client.16926 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec5119a0 0x5573ec513d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 4 client.16926 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573ec4b93e0 con 0x5573ec5119a0 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.794+0000 7fdd97fff640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec5119a0 0x5573ec513d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.795+0000 7fdd867fc640 10 client.16926 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.795+0000 7fdd97fff640 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec5119a0 0x5573ec513d90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fd4a9cc565ce378a server_cookie=f65bdccc388e4774 in_seq=0 out_seq=0 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdd867fc640 1 -- 172.21.15.116:0/3211726441 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5573ec4b93e0 con 0x5573ec5119a0 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdd867fc640 10 client.16926 handle_command_reply: tid=0 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 client.16926 shutdown 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 msgr2=0x5573ec4fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fdd90036040 tx=0x7fdd90005b50 comp rx=0 tx=0).stop 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 shutdown_connections 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec5119a0 0x5573ec513d90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec5017d0 0x5573ec4ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec4f8bf0 0x5573ec4fed10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 --2- 172.21.15.116:0/3211726441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec4fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 >> 172.21.15.116:0/3211726441 conn(0x5573ec4f2a00 msgr2=0x5573ec4f4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdd867fc640 0 client.16926 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 shutdown_connections 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 1 -- 172.21.15.116:0/3211726441 wait complete. 2024-03-19T02:32:00.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.796+0000 7fdda7e81740 20 client.16926 trim_cache size 0 max 16384 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.811+0000 7fdda7e81740 1 Processor -- start 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.811+0000 7fdda7e81740 1 -- start start 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b0c40 con 0x5573ec5017d0 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b2780 con 0x5573ec4f82d0 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdda7e81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b1310 con 0x5573ec4f8bf0 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36270/0 (socket says 172.21.15.116:36270) 2024-03-19T02:32:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53224/0 (socket says 172.21.15.116:53224) 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 -- 172.21.15.116:0/2072053543 learned_addr learned my addr 172.21.15.116:0/2072053543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 -- 172.21.15.116:0/2072053543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 msgr2=0x5573ec50d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 -- 172.21.15.116:0/2072053543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 msgr2=0x5573ec4fd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4fd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 -- 172.21.15.116:0/2072053543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573ec4fd780 con 0x5573ec5017d0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd9effd640 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd97fff640 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdda0104d30 tx=0x7fdd88005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0cfbc22c97b749 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.812+0000 7fdd94ff9640 1 -- 172.21.15.116:0/2072053543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88004050 con 0x5573ec5017d0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdd94ff9640 1 -- 172.21.15.116:0/2072053543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd880054d0 con 0x5573ec5017d0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdd94ff9640 1 -- 172.21.15.116:0/2072053543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88004b00 con 0x5573ec5017d0 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 -- 172.21.15.116:0/2072053543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 msgr2=0x5573ec4ff070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fdda0104d30 tx=0x7fdd88005850 comp rx=0 tx=0).stop 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 -- 172.21.15.116:0/2072053543 shutdown_connections 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec4ff070 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f8bf0 0x5573ec50d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 --2- 172.21.15.116:0/2072053543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f82d0 0x5573ec4fd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 -- 172.21.15.116:0/2072053543 >> 172.21.15.116:0/2072053543 conn(0x5573ec4f2a00 msgr2=0x5573ec4f4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 -- 172.21.15.116:0/2072053543 shutdown_connections 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.813+0000 7fdda7e81740 1 -- 172.21.15.116:0/2072053543 wait complete. 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 -- start start 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 Processor -- start 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec5052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f8bf0 0x5573ec505810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b20b0 con 0x5573ec5017d0 2024-03-19T02:32:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01b19e0 con 0x5573ec4f8bf0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01aec20 con 0x5573ec4f82d0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdda7e81740 10 client.0 _ll_register_callbacks cb 0x7fdd88009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36286/0 (socket says 172.21.15.116:36286) 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 -- 172.21.15.116:0/1496923179 learned_addr learned my addr 172.21.15.116:0/1496923179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd9effd640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f8bf0 0x5573ec505810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec5052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 -- 172.21.15.116:0/1496923179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 msgr2=0x5573ec5052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec5052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 -- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f8bf0 msgr2=0x5573ec505810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f8bf0 0x5573ec505810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.814+0000 7fdd97fff640 1 -- 172.21.15.116:0/1496923179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573ec576a10 con 0x5573ec5017d0 2024-03-19T02:32:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd97fff640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdd88005020 tx=0x7fdd880045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd36a57c687dfee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88005140 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdd88011050 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd880054d0 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdd880052e0 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 10 client.16932 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5573ec577cf0 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd880054d0 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdd88004840 con 0x5573ec5017d0 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 10 client.16932 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 10 client.16932 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 20 client.16932 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec578290 0x5573ec57a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 4 client.16932 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.815+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573ec4689f0 con 0x5573ec578290 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd9effd640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec578290 0x5573ec57a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd867fc640 10 client.16932 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd9effd640 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec578290 0x5573ec57a6b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d60dee2280e5e987 server_cookie=607fafa0bbc9911f in_seq=0 out_seq=0 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd867fc640 1 -- 172.21.15.116:0/1496923179 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573ec4689f0 con 0x5573ec578290 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd867fc640 10 client.16932 handle_command_reply: tid=0 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 client.16932 shutdown 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 msgr2=0x5573ec505d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdd88005020 tx=0x7fdd880045d0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 shutdown_connections 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5573ec578290 0x5573ec57a6b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573ec5017d0 0x5573ec505d50 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ec4f8bf0 0x5573ec505810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdd867fc640 0 client.16932 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 --2- 172.21.15.116:0/1496923179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5573ec4f82d0 0x5573ec5052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 >> 172.21.15.116:0/1496923179 conn(0x5573ec4f2a00 msgr2=0x5573ec4f3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 shutdown_connections 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 1 -- 172.21.15.116:0/1496923179 wait complete. 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.816+0000 7fdda7e81740 20 client.16932 trim_cache size 0 max 16384 2024-03-19T02:32:00.825 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 msgr2=0x7fdd8c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 0x7fdd8c0a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd980cca80 tx=0x7fdd98090d70 comp rx=0 tx=0).stop 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 msgr2=0x7fdda019ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800efd0 tx=0x7fdd9800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 shutdown_connections 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdd8c0a2970 0x7fdd8c0a4e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda01066b0 0x7fdda019ea30 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdda0101060 0x7fdda019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 --2- 172.21.15.116:0/2602012295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdda01006a0 0x7fdda019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 >> 172.21.15.116:0/2602012295 conn(0x7fdda00778d0 msgr2=0x7fdda0102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 shutdown_connections 2024-03-19T02:32:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.817+0000 7fdda5297640 1 -- 172.21.15.116:0/2602012295 wait complete. 2024-03-19T02:32:00.829 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 -- 172.21.15.116:0/3055542098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 msgr2=0x7fd628107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/3055542098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd628107be0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd624009930 tx=0x7fd62402bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 -- 172.21.15.116:0/3055542098 shutdown_connections 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/3055542098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 0x7fd62810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/3055542098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 0x7fd628101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/3055542098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd628107be0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 -- 172.21.15.116:0/3055542098 >> 172.21.15.116:0/3055542098 conn(0x7fd6280fd2c0 msgr2=0x7fd6280ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 -- 172.21.15.116:0/3055542098 shutdown_connections 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 -- 172.21.15.116:0/3055542098 wait complete. 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.944+0000 7fd62f7d4640 1 Processor -- start 2024-03-19T02:32:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 -- start start 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 0x7fd628073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 0x7fd628074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd628111a80 con 0x7fd628107800 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd628111900 con 0x7fd6281017f0 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62f7d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd628111c00 con 0x7fd6281021b0 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36304/0 (socket says 172.21.15.116:36304) 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 -- 172.21.15.116:0/4045267722 learned_addr learned my addr 172.21.15.116:0/4045267722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62d570640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 0x7fd628073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 0x7fd628074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 -- 172.21.15.116:0/4045267722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 msgr2=0x7fd628074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 0x7fd628074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 -- 172.21.15.116:0/4045267722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 msgr2=0x7fd628073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 0x7fd628073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 -- 172.21.15.116:0/4045267722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6281a64e0 con 0x7fd628107800 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd62dd71640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd62000eb10 tx=0x7fd62000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.945+0000 7fd61e7fc640 1 -- 172.21.15.116:0/4045267722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620004b20 con 0x7fd628107800 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.946+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6281a67d0 con 0x7fd628107800 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.946+0000 7fd61e7fc640 1 -- 172.21.15.116:0/4045267722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd620004cc0 con 0x7fd628107800 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.946+0000 7fd61e7fc640 1 -- 172.21.15.116:0/4045267722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620010690 con 0x7fd628107800 2024-03-19T02:32:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.946+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6281a6cd0 con 0x7fd628107800 2024-03-19T02:32:00.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.948+0000 7fd61e7fc640 1 -- 172.21.15.116:0/4045267722 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd620005020 con 0x7fd628107800 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.948+0000 7fd6323c1740 1 Processor -- start 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.948+0000 7fd6323c1740 1 -- start start 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3c104b0 con 0x55a5c3cadbf0 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3bb8d70 con 0x55a5c3cb67d0 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6323c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3c0c550 con 0x55a5c3cad2d0 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62d570640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62d570640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36312/0 (socket says 172.21.15.116:36312) 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62d570640 1 -- 172.21.15.116:0/3743383305 learned_addr learned my addr 172.21.15.116:0/3743383305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd61e7fc640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 0x7fd6140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 -- 172.21.15.116:0/3743383305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 msgr2=0x55a5c3cb8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 -- 172.21.15.116:0/3743383305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 msgr2=0x55a5c3cb6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 -- 172.21.15.116:0/3743383305 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c3bdb210 con 0x55a5c3cad2d0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62dd71640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 0x7fd6140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cad6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd618009730 tx=0x7fd6180096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857f231f25bb5870 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd61e7fc640 1 -- 172.21.15.116:0/4045267722 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd6200cf730 con 0x7fd628107800 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.949+0000 7fd6137fe640 1 -- 172.21.15.116:0/3743383305 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618004050 con 0x55a5c3cad2d0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6137fe640 1 -- 172.21.15.116:0/3743383305 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6180054d0 con 0x55a5c3cad2d0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6137fe640 1 -- 172.21.15.116:0/3743383305 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618004cd0 con 0x55a5c3cad2d0 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 -- 172.21.15.116:0/3743383305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 msgr2=0x55a5c3cad6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cad6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd618009730 tx=0x7fd6180096c0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd62dd71640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 0x7fd6140a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6200ccbb0 tx=0x7fd620009a70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 -- 172.21.15.116:0/3743383305 shutdown_connections 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 --2- 172.21.15.116:0/3743383305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cad6b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 -- 172.21.15.116:0/3743383305 >> 172.21.15.116:0/3743383305 conn(0x55a5c3ca7a00 msgr2=0x55a5c3ca7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 -- 172.21.15.116:0/3743383305 shutdown_connections 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.950+0000 7fd6323c1740 1 -- 172.21.15.116:0/3743383305 wait complete. 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 -- start start 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 Processor -- start 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3cb95c0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3bcc310 con 0x55a5c3cadbf0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5c3cb9740 con 0x55a5c3cad2d0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd6323c1740 10 client.0 _ll_register_callbacks cb 0x55a5c3bc02d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62d570640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36316/0 (socket says 172.21.15.116:36316) 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4166018433 learned_addr learned my addr 172.21.15.116:0/4166018433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 msgr2=0x55a5c3cb38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.951+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4166018433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 msgr2=0x55a5c3cb3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4166018433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c39243f0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd62d570640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd6180049b0 tx=0x7fd618005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132813ca0bb66bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618023020 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd61800f790 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6180054d0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd618036050 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 10 client.16944 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a5c39ee7d0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6180054d0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd61801f7b0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 10 client.16944 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 10 client.16944 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:32:00.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 20 client.16944 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.952+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3cc6e80 0x55a5c3cc9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.953+0000 7fd6323c1740 4 client.16944 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.953+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5c3c6e3e0 con 0x55a5c3cc6e80 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.953+0000 7fd62d570640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3cc6e80 0x55a5c3cc9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.953+0000 7fd610ff9640 10 client.16944 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.953+0000 7fd62d570640 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3cc6e80 0x55a5c3cc9290 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d45cfa5f7702a86d server_cookie=ff29518166aaabcd in_seq=0 out_seq=0 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd610ff9640 1 -- 172.21.15.116:0/4166018433 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5c3c6e3e0 con 0x55a5c3cc6e80 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd610ff9640 10 client.16944 handle_command_reply: tid=0 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 client.16944 shutdown 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 msgr2=0x55a5c3cb4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd6180049b0 tx=0x7fd618005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 shutdown_connections 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3cc6e80 0x55a5c3cc9290 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb4340 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cb3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd610ff9640 0 client.16944 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4166018433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 >> 172.21.15.116:0/4166018433 conn(0x55a5c3ca7a00 msgr2=0x55a5c3ca9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 shutdown_connections 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 1 -- 172.21.15.116:0/4166018433 wait complete. 2024-03-19T02:32:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.954+0000 7fd6323c1740 20 client.16944 trim_cache size 0 max 16384 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 Processor -- start 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 -- start start 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cc2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281b0e20 con 0x55a5c3cad2d0 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281b2960 con 0x55a5c3cb67d0 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.967+0000 7fd6323c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281b14f0 con 0x55a5c3cadbf0 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36330/0 (socket says 172.21.15.116:36330) 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4180298012 learned_addr learned my addr 172.21.15.116:0/4180298012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62d570640 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cc2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4180298012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cadbf0 msgr2=0x55a5c3cc2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cc2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4180298012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 msgr2=0x55a5c3cceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 -- 172.21.15.116:0/4180298012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5c3cada20 con 0x55a5c3cad2d0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6180059f0 tx=0x7fd618004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329b75e82e468445 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd612ffd640 1 -- 172.21.15.116:0/4180298012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618023070 con 0x55a5c3cad2d0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd612ffd640 1 -- 172.21.15.116:0/4180298012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd618004dd0 con 0x55a5c3cad2d0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.968+0000 7fd612ffd640 1 -- 172.21.15.116:0/4180298012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618026070 con 0x55a5c3cad2d0 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- 172.21.15.116:0/4180298012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 msgr2=0x55a5c3cb5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6180059f0 tx=0x7fd618004f90 comp rx=0 tx=0).stop 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- 172.21.15.116:0/4180298012 shutdown_connections 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cc2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 --2- 172.21.15.116:0/4180298012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cb5ff0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- 172.21.15.116:0/4180298012 >> 172.21.15.116:0/4180298012 conn(0x55a5c3ca7a00 msgr2=0x55a5c3cb2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- 172.21.15.116:0/4180298012 shutdown_connections 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- 172.21.15.116:0/4180298012 wait complete. 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 -- start start 2024-03-19T02:32:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.969+0000 7fd6323c1740 1 Processor -- start 2024-03-19T02:32:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cbeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cafb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281b2290 con 0x55a5c3cb67d0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281b1bc0 con 0x55a5c3cadbf0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6281aee00 con 0x55a5c3cad2d0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd6323c1740 10 client.0 _ll_register_callbacks cb 0x7fd6200093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36344/0 (socket says 172.21.15.116:36344) 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 -- 172.21.15.116:0/2590070562 learned_addr learned my addr 172.21.15.116:0/2590070562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62d570640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cafb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cbeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 -- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 msgr2=0x55a5c3cbeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cbeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 -- 172.21.15.116:0/2590070562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 msgr2=0x55a5c3cafb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cafb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 -- 172.21.15.116:0/2590070562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5c3cb06b0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd62cd6f640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd618009250 tx=0x7fd618009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de13c0264d2071e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.970+0000 7fd610ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618023070 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd618005370 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5c3c1d9f0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd618026040 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 10 client.16950 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a5c3d2c170 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a5c3c1d9f0 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd61800f790 con 0x55a5c3cb67d0 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 10 client.16950 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 10 client.16950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 20 client.16950 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3d2c710 0x55a5c3d2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 4 client.16950 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5c3c17d40 con 0x55a5c3d2c710 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.971+0000 7fd62d570640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3d2c710 0x55a5c3d2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd610ff9640 10 client.16950 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd62d570640 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3d2c710 0x55a5c3d2eb00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26f56af5d8ad36b9 server_cookie=351af23723bce2ab in_seq=0 out_seq=0 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd610ff9640 1 -- 172.21.15.116:0/2590070562 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5c3c17d40 con 0x55a5c3d2c710 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd610ff9640 10 client.16950 handle_command_reply: tid=0 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 client.16950 shutdown 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 msgr2=0x55a5c3cb0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd618009250 tx=0x7fd618009fb0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 shutdown_connections 2024-03-19T02:32:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a5c3d2c710 0x55a5c3d2eb00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5c3cb67d0 0x55a5c3cb0070 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a5c3cadbf0 0x55a5c3cafb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 --2- 172.21.15.116:0/2590070562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5c3cad2d0 0x55a5c3cbeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd610ff9640 0 client.16950 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 >> 172.21.15.116:0/2590070562 conn(0x55a5c3ca7a00 msgr2=0x55a5c3cb3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 shutdown_connections 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 1 -- 172.21.15.116:0/2590070562 wait complete. 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.972+0000 7fd6323c1740 20 client.16950 trim_cache size 0 max 16384 2024-03-19T02:32:00.981 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 msgr2=0x7fd6140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 0x7fd6140a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6200ccbb0 tx=0x7fd620009a70 comp rx=0 tx=0).stop 2024-03-19T02:32:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 msgr2=0x7fd6280745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd62000eb10 tx=0x7fd62000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 shutdown_connections 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd6140a2970 0x7fd6140a4e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd628107800 0x7fd6280745a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6281021b0 0x7fd628074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 --2- 172.21.15.116:0/4045267722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6281017f0 0x7fd628073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 >> 172.21.15.116:0/4045267722 conn(0x7fd6280fd2c0 msgr2=0x7fd6280fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 shutdown_connections 2024-03-19T02:32:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:00.973+0000 7fd62f7d4640 1 -- 172.21.15.116:0/4045267722 wait complete. 2024-03-19T02:32:00.983 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/700496742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:00.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 -- 172.21.15.116:0/3392671836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 msgr2=0x7fad5c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 --2- 172.21.15.116:0/3392671836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c10e220 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fad54009ca0 tx=0x7fad5402c180 comp rx=0 tx=0).stop 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 -- 172.21.15.116:0/3392671836 shutdown_connections 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 --2- 172.21.15.116:0/3392671836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c10e220 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 --2- 172.21.15.116:0/3392671836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c103360 0x7fad5c10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 --2- 172.21.15.116:0/3392671836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c102a40 0x7fad5c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 -- 172.21.15.116:0/3392671836 >> 172.21.15.116:0/3392671836 conn(0x7fad5c0778d0 msgr2=0x7fad5c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 -- 172.21.15.116:0/3392671836 shutdown_connections 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.101+0000 7fad60e8a640 1 -- 172.21.15.116:0/3392671836 wait complete. 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 Processor -- start 2024-03-19T02:32:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 -- start start 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c102a40 0x7fad5c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c103360 0x7fad5c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c111a30 con 0x7fad5c10bd80 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1118b0 con 0x7fad5c103360 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad60e8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c111bb0 con 0x7fad5c102a40 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36350/0 (socket says 172.21.15.116:36350) 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad59d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c103360 0x7fad5c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 -- 172.21.15.116:0/1117300268 learned_addr learned my addr 172.21.15.116:0/1117300268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c102a40 0x7fad5c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 -- 172.21.15.116:0/1117300268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c102a40 msgr2=0x7fad5c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c102a40 0x7fad5c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 -- 172.21.15.116:0/1117300268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c103360 msgr2=0x7fad5c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c103360 0x7fad5c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 -- 172.21.15.116:0/1117300268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5c19cf20 con 0x7fad5c10bd80 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.102+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fad5402c790 tx=0x7fad54035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.103+0000 7fad4b7fe640 1 -- 172.21.15.116:0/1117300268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54039040 con 0x7fad5c10bd80 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.103+0000 7fad4b7fe640 1 -- 172.21.15.116:0/1117300268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad54044070 con 0x7fad5c10bd80 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.103+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5c19d1b0 con 0x7fad5c10bd80 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.103+0000 7fad4b7fe640 1 -- 172.21.15.116:0/1117300268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54032850 con 0x7fad5c10bd80 2024-03-19T02:32:01.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.103+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c1a4aa0 con 0x7fad5c10bd80 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.105+0000 7fad4b7fe640 1 -- 172.21.15.116:0/1117300268 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fad540329f0 con 0x7fad5c10bd80 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.105+0000 7fad63a71740 1 Processor -- start 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 -- start start 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad4b7fe640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 0x7fad440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb255b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 0x561bb255dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561bb24b54b0 con 0x561bb2552bf0 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561bb245dd70 con 0x561bb255b7d0 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad63a71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bb24b1550 con 0x561bb25522d0 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad59d9b640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 0x7fad440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad4b7fe640 1 -- 172.21.15.116:0/1117300268 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad540f1450 con 0x7fad5c10bd80 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad59d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb255b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 0x561bb255dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53314/0 (socket says 172.21.15.116:53314) 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5a59c640 1 -- 172.21.15.116:0/2133488125 learned_addr learned my addr 172.21.15.116:0/2133488125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2133488125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 msgr2=0x561bb25526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2133488125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 msgr2=0x561bb255b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb255b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2133488125 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bb2480210 con 0x561bb255b7d0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.106+0000 7fad5a59c640 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.107+0000 7fad59d9b640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 0x7fad440a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fad50005f60 tx=0x7fad50009ae0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.107+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 0x561bb255dbc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad540ee930 tx=0x7fad54009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eaea8cc1edf8cde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.107+0000 7fad48ff9640 1 -- 172.21.15.116:0/2133488125 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54032ce0 con 0x561bb255b7d0 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.107+0000 7fad48ff9640 1 -- 172.21.15.116:0/2133488125 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad540bdf40 con 0x561bb255b7d0 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.107+0000 7fad48ff9640 1 -- 172.21.15.116:0/2133488125 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540c5fe0 con 0x561bb255b7d0 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- 172.21.15.116:0/2133488125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 msgr2=0x561bb255dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 0x561bb255dbc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fad540ee930 tx=0x7fad54009a20 comp rx=0 tx=0).stop 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- 172.21.15.116:0/2133488125 shutdown_connections 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb255b7d0 0x561bb255dbc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb255b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- 172.21.15.116:0/2133488125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb25522d0 0x561bb25526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- 172.21.15.116:0/2133488125 >> 172.21.15.116:0/2133488125 conn(0x561bb254ca00 msgr2=0x561bb254cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- 172.21.15.116:0/2133488125 shutdown_connections 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- 172.21.15.116:0/2133488125 wait complete. 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- start start 2024-03-19T02:32:01.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 Processor -- start 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb255b7d0 0x561bb2559220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561bb255e380 con 0x561bb2552bf0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561bb2471310 con 0x561bb25522d0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bb255e500 con 0x561bb255b7d0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad63a71740 10 client.0 _ll_register_callbacks cb 0x561bb24652d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad59d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.108+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47994/0 (socket says 172.21.15.116:47994) 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad5a59c640 1 -- 172.21.15.116:0/1098282868 learned_addr learned my addr 172.21.15.116:0/1098282868 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36356/0 (socket says 172.21.15.116:36356) 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb255b7d0 0x561bb2559220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 -- 172.21.15.116:0/1098282868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb255b7d0 msgr2=0x561bb2559220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb255b7d0 0x561bb2559220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 -- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 msgr2=0x561bb25587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 -- 172.21.15.116:0/1098282868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bb21c93f0 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad59d9b640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fad5c1000d0 tx=0x7fad5001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8acae5cbf26c881 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50024070 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad540bdf40 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad50003d80 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad5000fe20 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad63a71740 10 client.16962 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.109+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561bb22937d0 con 0x561bb2552bf0 2024-03-19T02:32:01.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad540bdf40 con 0x561bb2552bf0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fad500048a0 con 0x561bb2552bf0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 10 client.16962 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 20 client.16962 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb256dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 4 client.16962 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bb25133e0 con 0x561bb256b7a0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb256dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad41ffb640 10 client.16962 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.110+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb256dbd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a40e6180c829095 server_cookie=47910f046a9ef4cf in_seq=0 out_seq=0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad41ffb640 1 -- 172.21.15.116:0/1098282868 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561bb25133e0 con 0x561bb256b7a0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad41ffb640 10 client.16962 handle_command_reply: tid=0 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 client.16962 shutdown 2024-03-19T02:32:01.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 msgr2=0x561bb2558ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fad5c1000d0 tx=0x7fad5001d000 comp rx=0 tx=0).stop 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 shutdown_connections 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb256dbd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb255b7d0 0x561bb2559220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad41ffb640 0 client.16962 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb2552bf0 0x561bb2558ce0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 --2- 172.21.15.116:0/1098282868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb25587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 >> 172.21.15.116:0/1098282868 conn(0x561bb254ca00 msgr2=0x561bb254e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 shutdown_connections 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 1 -- 172.21.15.116:0/1098282868 wait complete. 2024-03-19T02:32:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.112+0000 7fad63a71740 20 client.16962 trim_cache size 0 max 16384 2024-03-19T02:32:01.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 Processor -- start 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 -- start start 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb2567070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1aeaf0 con 0x561bb255b7d0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b0630 con 0x561bb25522d0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad63a71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1af1c0 con 0x561bb2552bf0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36362/0 (socket says 172.21.15.116:36362) 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48004/0 (socket says 172.21.15.116:48004) 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad59d9b640 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb2567070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 -- 172.21.15.116:0/893223574 learned_addr learned my addr 172.21.15.116:0/893223574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 -- 172.21.15.116:0/893223574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 msgr2=0x561bb2567070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb2567070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 -- 172.21.15.116:0/893223574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 msgr2=0x561bb255aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5ad9d640 1 -- 172.21.15.116:0/893223574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bb254c830 con 0x561bb255b7d0 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.127+0000 7fad5a59c640 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fad540bd7d0 tx=0x7fad5402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b97347c1991a1ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad43fff640 1 -- 172.21.15.116:0/893223574 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54032ce0 con 0x561bb255b7d0 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad43fff640 1 -- 172.21.15.116:0/893223574 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad540bdf40 con 0x561bb255b7d0 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad43fff640 1 -- 172.21.15.116:0/893223574 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54003b70 con 0x561bb255b7d0 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 -- 172.21.15.116:0/893223574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 msgr2=0x561bb25734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fad540bd7d0 tx=0x7fad5402c180 comp rx=0 tx=0).stop 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 -- 172.21.15.116:0/893223574 shutdown_connections 2024-03-19T02:32:01.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25734e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb2567070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 --2- 172.21.15.116:0/893223574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 -- 172.21.15.116:0/893223574 >> 172.21.15.116:0/893223574 conn(0x561bb254ca00 msgr2=0x561bb254cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 -- 172.21.15.116:0/893223574 shutdown_connections 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.128+0000 7fad63a71740 1 -- 172.21.15.116:0/893223574 wait complete. 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 -- start start 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 Processor -- start 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb255f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1aff60 con 0x561bb255b7d0 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1af890 con 0x561bb25522d0 2024-03-19T02:32:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1acad0 con 0x561bb2552bf0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad63a71740 10 client.0 _ll_register_callbacks cb 0x7fad50004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36370/0 (socket says 172.21.15.116:36370) 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 -- 172.21.15.116:0/4012392568 learned_addr learned my addr 172.21.15.116:0/4012392568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad59d9b640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb255f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5a59c640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 -- 172.21.15.116:0/4012392568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 msgr2=0x561bb255f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb255f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 -- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 msgr2=0x561bb255f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5ad9d640 1 -- 172.21.15.116:0/4012392568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bb255dbc0 con 0x561bb255b7d0 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.129+0000 7fad5a59c640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fad54004af0 tx=0x7fad54035f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d12573f6a191ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540d5070 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fad540b9eb0 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad540bdf40 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad54032ce0 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 10 client.16968 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561bb2551cd0 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad540bdf40 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fad5403c030 con 0x561bb255b7d0 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 10 client.16968 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 20 client.16968 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb25d4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 4 client.16968 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561bb24c29f0 con 0x561bb256b7a0 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.130+0000 7fad5a59c640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb25d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad41ffb640 10 client.16968 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad5a59c640 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb25d4190 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6e755e31ff64d0 server_cookie=9c6a2e9f8c92152 in_seq=0 out_seq=0 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad41ffb640 1 -- 172.21.15.116:0/4012392568 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561bb24c29f0 con 0x561bb256b7a0 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad41ffb640 10 client.16968 handle_command_reply: tid=0 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 client.16968 shutdown 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 msgr2=0x561bb25d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fad54004af0 tx=0x7fad54035f90 comp rx=0 tx=0).stop 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 shutdown_connections 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561bb256b7a0 0x561bb25d4190 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bb255b7d0 0x561bb25d0b10 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bb2552bf0 0x561bb255f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad41ffb640 0 client.16968 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 --2- 172.21.15.116:0/4012392568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bb25522d0 0x561bb255f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.131+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 >> 172.21.15.116:0/4012392568 conn(0x561bb254ca00 msgr2=0x561bb254ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 shutdown_connections 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad63a71740 1 -- 172.21.15.116:0/4012392568 wait complete. 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad63a71740 20 client.16968 trim_cache size 0 max 16384 2024-03-19T02:32:01.139 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 msgr2=0x7fad440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 0x7fad440a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fad50005f60 tx=0x7fad50009ae0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 msgr2=0x7fad5c19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fad5402c790 tx=0x7fad54035db0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 shutdown_connections 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad440a2970 0x7fad440a4e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c10bd80 0x7fad5c19c7f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad5c103360 0x7fad5c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 --2- 172.21.15.116:0/1117300268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad5c102a40 0x7fad5c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 >> 172.21.15.116:0/1117300268 conn(0x7fad5c0778d0 msgr2=0x7fad5c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 shutdown_connections 2024-03-19T02:32:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:01.132+0000 7fad60e8a640 1 -- 172.21.15.116:0/1117300268 wait complete. 2024-03-19T02:32:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/700496742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/700496742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:01.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 124405 3.28 MB/sec execute 84 sec latency 19.606 ms 2024-03-19T02:32:02.145 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:32:02.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:32:02.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:32:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:01 smithi162 ceph-mon[35761]: pgmap v129: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:32:02.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:01 smithi116 ceph-mon[32263]: pgmap v129: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:32:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:01 smithi154 ceph-mon[35693]: pgmap v129: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.0 MiB/s wr, 126 op/s 2024-03-19T02:32:02.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- 172.21.15.116:0/364107984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46981066a0 msgr2=0x7f4698106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 --2- 172.21.15.116:0/364107984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46981066a0 0x7f4698106a80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f468c009930 tx=0x7f468c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- 172.21.15.116:0/364107984 shutdown_connections 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 --2- 172.21.15.116:0/364107984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698101050 0x7f469810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 --2- 172.21.15.116:0/364107984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4698100690 0x7f4698100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 --2- 172.21.15.116:0/364107984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46981066a0 0x7f4698106a80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- 172.21.15.116:0/364107984 >> 172.21.15.116:0/364107984 conn(0x7f4698077820 msgr2=0x7f4698077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- 172.21.15.116:0/364107984 shutdown_connections 2024-03-19T02:32:02.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- 172.21.15.116:0/364107984 wait complete. 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 Processor -- start 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.259+0000 7f469ed32640 1 -- start start 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 0x7f4698198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46981066a0 0x7f469819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4698110960 con 0x7f4698101050 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46981107e0 con 0x7f46981066a0 2024-03-19T02:32:02.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469ed32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4698110ae0 con 0x7f4698100690 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36396/0 (socket says 172.21.15.116:36396) 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469cace640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 0x7f4698198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 -- 172.21.15.116:0/2116623224 learned_addr learned my addr 172.21.15.116:0/2116623224 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469d2cf640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46981066a0 0x7f469819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 -- 172.21.15.116:0/2116623224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 msgr2=0x7f4698198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 0x7f4698198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 -- 172.21.15.116:0/2116623224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46981066a0 msgr2=0x7f469819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46981066a0 0x7f469819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 -- 172.21.15.116:0/2116623224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469819d000 con 0x7f4698101050 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f469cace640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 0x7f4698198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4697fff640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f468400e9f0 tx=0x7f468400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.260+0000 7f4695ffb640 1 -- 172.21.15.116:0/2116623224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468400cd80 con 0x7f4698101050 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.261+0000 7f4695ffb640 1 -- 172.21.15.116:0/2116623224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46840108e0 con 0x7f4698101050 2024-03-19T02:32:02.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.261+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469819d2f0 con 0x7f4698101050 2024-03-19T02:32:02.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.261+0000 7f4695ffb640 1 -- 172.21.15.116:0/2116623224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684018920 con 0x7f4698101050 2024-03-19T02:32:02.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.261+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46981a4b80 con 0x7f4698101050 2024-03-19T02:32:02.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.263+0000 7f4695ffb640 1 -- 172.21.15.116:0/2116623224 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4684010a90 con 0x7f4698101050 2024-03-19T02:32:02.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.263+0000 7f46a19f5740 1 Processor -- start 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.263+0000 7f46a19f5740 1 -- start start 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.263+0000 7f4695ffb640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 0x7f46880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.263+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308030 0x5636bc308410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc315c90 0x5636bc318080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f46a19f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5636bc26c160 con 0x5636bc315c90 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f46a19f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5636bc214e10 con 0x5636bc308030 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f46a19f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5636bc268200 con 0x5636bc308950 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f469d2cf640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 0x7f46880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4695ffb640 1 -- 172.21.15.116:0/2116623224 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f46840cf440 con 0x7f4698101050 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f469cace640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308030 0x5636bc308410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f469d2cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc315c90 0x5636bc318080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53396/0 (socket says 172.21.15.116:53396) 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 -- 172.21.15.116:0/2215360220 learned_addr learned my addr 172.21.15.116:0/2215360220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 -- 172.21.15.116:0/2215360220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308030 msgr2=0x5636bc308410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308030 0x5636bc308410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 -- 172.21.15.116:0/2215360220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc315c90 msgr2=0x5636bc318080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc315c90 0x5636bc318080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 -- 172.21.15.116:0/2215360220 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636bc0487e0 con 0x5636bc308950 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f469d2cf640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 0x7f46880a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4690005fd0 tx=0x7f4690009e90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4697fff640 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f468400c6a0 tx=0x7f468407daa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad4f6f553ef16c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4682ffd640 1 -- 172.21.15.116:0/2215360220 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468409b870 con 0x5636bc308950 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4682ffd640 1 -- 172.21.15.116:0/2215360220 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f468409f910 con 0x5636bc308950 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.264+0000 7f4682ffd640 1 -- 172.21.15.116:0/2215360220 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684086110 con 0x5636bc308950 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- 172.21.15.116:0/2215360220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 msgr2=0x5636bc315750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f468400c6a0 tx=0x7f468407daa0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- 172.21.15.116:0/2215360220 shutdown_connections 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc315c90 0x5636bc318080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc315750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2215360220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308030 0x5636bc308410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- 172.21.15.116:0/2215360220 >> 172.21.15.116:0/2215360220 conn(0x5636bc302ab0 msgr2=0x5636bc302e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- 172.21.15.116:0/2215360220 shutdown_connections 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- 172.21.15.116:0/2215360220 wait complete. 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.265+0000 7f46a19f5740 1 -- start start 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 Processor -- start 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308950 0x5636bc3131d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 0x5636bc313710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5636bc318a70 con 0x5636bc308030 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5636bc2281a0 con 0x5636bc308950 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5636bc318bf0 con 0x5636bc315c90 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f46a19f5740 10 client.0 _ll_register_callbacks cb 0x5636bbf7fc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:02.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36410/0 (socket says 172.21.15.116:36410) 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469d2cf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 0x5636bc313710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 -- 172.21.15.116:0/1187225123 learned_addr learned my addr 172.21.15.116:0/1187225123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f4697fff640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308950 0x5636bc3131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 -- 172.21.15.116:0/1187225123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 msgr2=0x5636bc313710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 0x5636bc313710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 -- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308950 msgr2=0x5636bc3131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308950 0x5636bc3131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469cace640 1 -- 172.21.15.116:0/1187225123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636bc0a8f70 con 0x5636bc308030 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.266+0000 7f469d2cf640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 0x5636bc313710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f469cace640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f468c005e00 tx=0x7f468c02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b76fec594cca41 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c02c5d0 con 0x5636bc308030 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f468c0035f0 con 0x5636bc308030 2024-03-19T02:32:02.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f468409f910 con 0x5636bc308030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f468c0037f0 con 0x5636bc308030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 10 client.16980 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5636bbf40170 con 0x5636bc308030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f468409f910 con 0x5636bc308030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f468c04bb70 con 0x5636bc308030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 10 client.16980 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 10 client.16980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 20 client.16980 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc326030 0x5636bc328460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 4 client.16980 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.267+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636bc2796a0 con 0x5636bc326030 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.268+0000 7f4697fff640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc326030 0x5636bc328460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.268+0000 7f4677fff640 10 client.16980 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.268+0000 7f4697fff640 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc326030 0x5636bc328460 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88e96fd287b56254 server_cookie=6bda2823cf9b7ba2 in_seq=0 out_seq=0 2024-03-19T02:32:02.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f4677fff640 1 -- 172.21.15.116:0/1187225123 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636bc2796a0 con 0x5636bc326030 2024-03-19T02:32:02.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f4677fff640 10 client.16980 handle_command_reply: tid=0 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 client.16980 shutdown 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 msgr2=0x5636bc312c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f468c005e00 tx=0x7f468c02c9f0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 shutdown_connections 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc326030 0x5636bc328460 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc315c90 0x5636bc313710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc308950 0x5636bc3131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f4677fff640 0 client.16980 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 --2- 172.21.15.116:0/1187225123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc312c90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 >> 172.21.15.116:0/1187225123 conn(0x5636bc302ab0 msgr2=0x5636bc304800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 shutdown_connections 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 1 -- 172.21.15.116:0/1187225123 wait complete. 2024-03-19T02:32:02.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.269+0000 7f46a19f5740 20 client.16980 trim_cache size 0 max 16384 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 Processor -- start 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 -- start start 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc311bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc321880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46981ac9f0 con 0x5636bc308030 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46981ae530 con 0x5636bc315c90 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f46a19f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46981ad0c0 con 0x5636bc308950 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f469cace640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f469cace640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36418/0 (socket says 172.21.15.116:36418) 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f469cace640 1 -- 172.21.15.116:0/3778677370 learned_addr learned my addr 172.21.15.116:0/3778677370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc311bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f469d2cf640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc321880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 -- 172.21.15.116:0/3778677370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 msgr2=0x5636bc321880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc321880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 -- 172.21.15.116:0/3778677370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 msgr2=0x5636bc3153c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f4697fff640 1 -- 172.21.15.116:0/3778677370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636bc308780 con 0x5636bc308950 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.289+0000 7f469cace640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f4697fff640 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc311bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4684098240 tx=0x7f468407f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b30671038bb43a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f46827fc640 1 -- 172.21.15.116:0/3778677370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468409b870 con 0x5636bc308950 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f46827fc640 1 -- 172.21.15.116:0/3778677370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46840a3f60 con 0x5636bc308950 2024-03-19T02:32:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f46827fc640 1 -- 172.21.15.116:0/3778677370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840ccb30 con 0x5636bc308950 2024-03-19T02:32:02.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f46a19f5740 1 -- 172.21.15.116:0/3778677370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 msgr2=0x5636bc311bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.290+0000 7f46a19f5740 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc311bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4684098240 tx=0x7f468407f040 comp rx=0 tx=0).stop 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- 172.21.15.116:0/3778677370 shutdown_connections 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc321880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308950 0x5636bc311bd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- 172.21.15.116:0/3778677370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308030 0x5636bc3153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- 172.21.15.116:0/3778677370 >> 172.21.15.116:0/3778677370 conn(0x5636bc302ab0 msgr2=0x5636bc304790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- 172.21.15.116:0/3778677370 shutdown_connections 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- 172.21.15.116:0/3778677370 wait complete. 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- start start 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 Processor -- start 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc31d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46981ade60 con 0x5636bc308950 2024-03-19T02:32:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46981ad790 con 0x5636bc315c90 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46981aa9d0 con 0x5636bc308030 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.291+0000 7f46a19f5740 10 client.0 _ll_register_callbacks cb 0x7f468c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4697fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53426/0 (socket says 172.21.15.116:53426) 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4697fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36420/0 (socket says 172.21.15.116:36420) 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 -- 172.21.15.116:0/2306134501 learned_addr learned my addr 172.21.15.116:0/2306134501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469d2cf640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc31d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 -- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 msgr2=0x5636bc31d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc31d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 -- 172.21.15.116:0/2306134501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 msgr2=0x5636bc31a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 -- 172.21.15.116:0/2306134501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636bc31de00 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4697fff640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f469cace640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f468c003fa0 tx=0x7f468c003fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c980821c81bd90eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4677fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c02c5d0 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46840a3f60 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f468c034910 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f468c063050 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f46a19f5740 10 client.26743 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.292+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5636bc38ce60 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46840a3f60 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f468c04b6b0 con 0x5636bc308030 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 10 client.26743 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 10 client.26743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:02.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 20 client.26743 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc38d690 0x5636bc38fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 4 client.26743 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5636bc2739f0 con 0x5636bc38d690 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f4697fff640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc38d690 0x5636bc38fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f4677fff640 10 client.26743 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.293+0000 7f4697fff640 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc38d690 0x5636bc38fab0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38a8afe032c375d0 server_cookie=b83756be11651712 in_seq=0 out_seq=0 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f4677fff640 1 -- 172.21.15.116:0/2306134501 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f468407c080 con 0x5636bc38d690 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f4677fff640 10 client.26743 handle_command_reply: tid=0 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 client.26743 shutdown 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 msgr2=0x5636bc31a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f468c003fa0 tx=0x7f468c003fd0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 shutdown_connections 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5636bc38d690 0x5636bc38fab0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636bc315c90 0x5636bc31d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636bc308950 0x5636bc31a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 --2- 172.21.15.116:0/2306134501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5636bc308030 0x5636bc31a280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 >> 172.21.15.116:0/2306134501 conn(0x5636bc302ab0 msgr2=0x5636bc312c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f4677fff640 0 client.26743 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 shutdown_connections 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 1 -- 172.21.15.116:0/2306134501 wait complete. 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.295+0000 7f46a19f5740 20 client.26743 trim_cache size 0 max 16384 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:32:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:32:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "8f4ca2ec-dcd7-4683-ad34-b6f4d2fc47ed", 2024-03-19T02:32:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:32:02.302 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:32:02.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 msgr2=0x7f46880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 0x7f46880a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4690005fd0 tx=0x7f4690009e90 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 msgr2=0x7f4698198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f468400e9f0 tx=0x7f468400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 shutdown_connections 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46880a2970 0x7f46880a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46981066a0 0x7f469819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4698101050 0x7f4698198550 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 --2- 172.21.15.116:0/2116623224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4698100690 0x7f4698198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 >> 172.21.15.116:0/2116623224 conn(0x7f4698077820 msgr2=0x7f4698068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 shutdown_connections 2024-03-19T02:32:02.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.296+0000 7f469ed32640 1 -- 172.21.15.116:0/2116623224 wait complete. 2024-03-19T02:32:02.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:32:02.308 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 -- 172.21.15.116:0/1718786813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 msgr2=0x7f47b8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/1718786813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b8107b90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009930 tx=0x7f47a802bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 -- 172.21.15.116:0/1718786813 shutdown_connections 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/1718786813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b8102160 0x7f47b810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/1718786813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b81017a0 0x7f47b8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/1718786813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b8107b90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 -- 172.21.15.116:0/1718786813 >> 172.21.15.116:0/1718786813 conn(0x7f47b80fd2b0 msgr2=0x7f47b80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 -- 172.21.15.116:0/1718786813 shutdown_connections 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.462+0000 7f47bcdd3640 1 -- 172.21.15.116:0/1718786813 wait complete. 2024-03-19T02:32:02.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 Processor -- start 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 -- start start 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b81017a0 0x7f47b8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b8102160 0x7f47b8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47b8111a70 con 0x7f47b81077b0 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81118f0 con 0x7f47b8102160 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47bcdd3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47b8111bf0 con 0x7f47b81017a0 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36430/0 (socket says 172.21.15.116:36430) 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 -- 172.21.15.116:0/2860882173 learned_addr learned my addr 172.21.15.116:0/2860882173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b659c640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b81017a0 0x7f47b8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47affff640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b8102160 0x7f47b8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 -- 172.21.15.116:0/2860882173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b81017a0 msgr2=0x7f47b8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b81017a0 0x7f47b8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 -- 172.21.15.116:0/2860882173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b8102160 msgr2=0x7f47b8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b8102160 0x7f47b8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 -- 172.21.15.116:0/2860882173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b819d030 con 0x7f47b81077b0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.463+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f47b000eb10 tx=0x7f47b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.464+0000 7f47af7fe640 1 -- 172.21.15.116:0/2860882173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0004b20 con 0x7f47b81077b0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.464+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b819d320 con 0x7f47b81077b0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.464+0000 7f47af7fe640 1 -- 172.21.15.116:0/2860882173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b0004cc0 con 0x7f47b81077b0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.464+0000 7f47af7fe640 1 -- 172.21.15.116:0/2860882173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0010690 con 0x7f47b81077b0 2024-03-19T02:32:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.464+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b81a4bb0 con 0x7f47b81077b0 2024-03-19T02:32:02.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.466+0000 7f47af7fe640 1 -- 172.21.15.116:0/2860882173 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f47b0005020 con 0x7f47b81077b0 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.466+0000 7f47bf9b7740 1 Processor -- start 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.466+0000 7f47bf9b7740 1 -- start start 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d9f30 0x557c911dc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47af7fe640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 0x7f47a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c91134500 con 0x557c911d1350 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557c910dcbe0 con 0x557c911d9f30 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557c911305a0 con 0x557c911d0a30 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53456/0 (socket says 172.21.15.116:53456) 2024-03-19T02:32:02.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 -- 172.21.15.116:0/898556380 learned_addr learned my addr 172.21.15.116:0/898556380 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d9f30 0x557c911dc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 0x7f47a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47af7fe640 1 -- 172.21.15.116:0/2860882173 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f47b00cf810 con 0x7f47b81077b0 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 -- 172.21.15.116:0/898556380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 msgr2=0x557c911d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 -- 172.21.15.116:0/898556380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d9f30 msgr2=0x557c911dc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d9f30 0x557c911dc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 -- 172.21.15.116:0/898556380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c910fe000 con 0x557c911d1350 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47b659c640 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d99f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009230 tx=0x7f47a8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4108b42840f1535 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.467+0000 7f47affff640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 0x7f47a40a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009fd0 tx=0x7f47a0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.468+0000 7f47acff9640 1 -- 172.21.15.116:0/898556380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8045020 con 0x557c911d1350 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.468+0000 7f47acff9640 1 -- 172.21.15.116:0/898556380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47a802c7e0 con 0x557c911d1350 2024-03-19T02:32:02.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.468+0000 7f47acff9640 1 -- 172.21.15.116:0/898556380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8005400 con 0x557c911d1350 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- 172.21.15.116:0/898556380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 msgr2=0x557c911d99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d99f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009230 tx=0x7f47a8009560 comp rx=0 tx=0).stop 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- 172.21.15.116:0/898556380 shutdown_connections 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d9f30 0x557c911dc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d99f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/898556380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d0a30 0x557c911d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- 172.21.15.116:0/898556380 >> 172.21.15.116:0/898556380 conn(0x557c911cb160 msgr2=0x557c911cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- 172.21.15.116:0/898556380 shutdown_connections 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- 172.21.15.116:0/898556380 wait complete. 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- start start 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 Processor -- start 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d9f30 0x557c911d78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c911dccc0 con 0x557c911d1350 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557c910f0160 con 0x557c911d0a30 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557c911dce40 con 0x557c911d9f30 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.469+0000 7f47bf9b7740 10 client.0 _ll_register_callbacks cb 0x557c9112ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47b659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36442/0 (socket says 172.21.15.116:36442) 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d9f30 0x557c911d78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47b659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48084/0 (socket says 172.21.15.116:48084) 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 -- 172.21.15.116:0/2667767616 learned_addr learned my addr 172.21.15.116:0/2667767616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 -- 172.21.15.116:0/2667767616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d9f30 msgr2=0x557c911d78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d9f30 0x557c911d78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 -- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 msgr2=0x557c911d6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 -- 172.21.15.116:0/2667767616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c90f127d0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47affff640 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f47a0004590 tx=0x7f47a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52c383abf91dd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f479dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0005220 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f47a00053c0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a802c7e0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f47a00055c0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47bf9b7740 10 client.16998 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.470+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c90e483f0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47a802c7e0 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f47a0028030 con 0x557c911d1350 2024-03-19T02:32:02.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 10 client.16998 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 20 client.16998 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c911ea190 0x557c911ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 4 client.16998 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c91141a40 con 0x557c911ea190 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47b659c640 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c911ea190 0x557c911ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f47b659c640 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c911ea190 0x557c911ec5d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ca8d0a52b9265fc server_cookie=e065afde6667287b in_seq=0 out_seq=0 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.471+0000 7f479dffb640 10 client.16998 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.472+0000 7f479dffb640 1 -- 172.21.15.116:0/2667767616 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c91141a40 con 0x557c911ea190 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.472+0000 7f479dffb640 10 client.16998 handle_command_reply: tid=0 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.472+0000 7f47bf9b7740 1 client.16998 shutdown 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 msgr2=0x557c911d7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f47a0004590 tx=0x7f47a0004730 comp rx=0 tx=0).stop 2024-03-19T02:32:02.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 shutdown_connections 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c911ea190 0x557c911ec5d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d9f30 0x557c911d78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f479dffb640 0 client.16998 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d1350 0x557c911d7380 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/2667767616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 >> 172.21.15.116:0/2667767616 conn(0x557c911cb160 msgr2=0x557c911cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 shutdown_connections 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 1 -- 172.21.15.116:0/2667767616 wait complete. 2024-03-19T02:32:02.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.473+0000 7f47bf9b7740 20 client.16998 trim_cache size 0 max 16384 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 Processor -- start 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 -- start start 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c911e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81aeb30 con 0x557c911d9f30 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81b0670 con 0x557c911d0a30 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81af200 con 0x557c911d1350 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c911e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.493+0000 7f47b659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36450/0 (socket says 172.21.15.116:36450) 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 -- 172.21.15.116:0/1588962729 learned_addr learned my addr 172.21.15.116:0/1588962729 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 -- 172.21.15.116:0/1588962729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 msgr2=0x557c911e5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c911e5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 -- 172.21.15.116:0/1588962729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 msgr2=0x557c911d9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 -- 172.21.15.116:0/1588962729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c90e080f0 con 0x557c911d9f30 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47b00ccc40 tx=0x7f47b000efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359b1bac5dffffa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f479ffff640 1 -- 172.21.15.116:0/1588962729 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b009b450 con 0x557c911d9f30 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f479ffff640 1 -- 172.21.15.116:0/1588962729 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b009c1f0 con 0x557c911d9f30 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.494+0000 7f479ffff640 1 -- 172.21.15.116:0/1588962729 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b000d340 con 0x557c911d9f30 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- 172.21.15.116:0/1588962729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 msgr2=0x557c911d5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47b00ccc40 tx=0x7f47b000efd0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- 172.21.15.116:0/1588962729 shutdown_connections 2024-03-19T02:32:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d5cf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c911e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/1588962729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- 172.21.15.116:0/1588962729 >> 172.21.15.116:0/1588962729 conn(0x557c911cb160 msgr2=0x557c911d55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- 172.21.15.116:0/1588962729 shutdown_connections 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- 172.21.15.116:0/1588962729 wait complete. 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 -- start start 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 Processor -- start 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.495+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c9124ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81affa0 con 0x557c911d9f30 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81af8d0 con 0x557c911d0a30 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47b81acb10 con 0x557c911d1350 2024-03-19T02:32:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 10 client.0 _ll_register_callbacks cb 0x7f47a0004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36462/0 (socket says 172.21.15.116:36462) 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48102/0 (socket says 172.21.15.116:48102) 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 -- 172.21.15.116:0/4250031383 learned_addr learned my addr 172.21.15.116:0/4250031383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47affff640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c9124ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 -- 172.21.15.116:0/4250031383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 msgr2=0x557c9124ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c9124ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 -- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 msgr2=0x557c911e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 -- 172.21.15.116:0/4250031383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c9124c000 con 0x557c911d9f30 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47affff640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c9124ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b659c640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47b6d9d640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f47b0097f50 tx=0x7f47b0098390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628d2b7c3a10b32d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f479dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b009b450 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f47b009bd50 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47b009c1f0 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f47b0099bf0 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 10 client.17004 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.496+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c91254c50 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b009c1f0 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f47b00ba1e0 con 0x557c911d9f30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 10 client.17004 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 10 client.17004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 20 client.17004 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:02.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c91255260 0x557c91257650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 4 client.17004 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:32:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557c9113bd90 con 0x557c91255260 2024-03-19T02:32:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47b659c640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c91255260 0x557c91257650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f47b659c640 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c91255260 0x557c91257650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=116d53397b30cea6 server_cookie=fd8db6224f2e23b1 in_seq=0 out_seq=0 2024-03-19T02:32:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.497+0000 7f479dffb640 10 client.17004 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.498+0000 7f479dffb640 1 -- 172.21.15.116:0/4250031383 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557c9113bd90 con 0x557c91255260 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.498+0000 7f479dffb640 10 client.17004 handle_command_reply: tid=0 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.498+0000 7f47bf9b7740 1 client.17004 shutdown 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 msgr2=0x557c911d3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f47b0097f50 tx=0x7f47b0098390 comp rx=0 tx=0).stop 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 shutdown_connections 2024-03-19T02:32:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x557c91255260 0x557c91257650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c911d9f30 0x557c911d3ac0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557c911d1350 0x557c9124ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f479dffb640 0 client.17004 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 --2- 172.21.15.116:0/4250031383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c911d0a30 0x557c911e2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 >> 172.21.15.116:0/4250031383 conn(0x557c911cb160 msgr2=0x557c911d5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 shutdown_connections 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 1 -- 172.21.15.116:0/4250031383 wait complete. 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bf9b7740 20 client.17004 trim_cache size 0 max 16384 2024-03-19T02:32:02.506 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "8f4ca2ec-dcd7-4683-ad34-b6f4d2fc47ed": { 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "8f4ca2ec-dcd7-4683-ad34-b6f4d2fc47ed", 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:32:02.507 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 msgr2=0x7f47a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 0x7f47a40a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009fd0 tx=0x7f47a0005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 msgr2=0x7f47b819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f47b000eb10 tx=0x7f47b000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 shutdown_connections 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f47a40a2970 0x7f47a40a4e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b81077b0 0x7f47b819c900 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47b8102160 0x7f47b8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 --2- 172.21.15.116:0/2860882173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47b81017a0 0x7f47b8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 >> 172.21.15.116:0/2860882173 conn(0x7f47b80fd2b0 msgr2=0x7f47b80fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 shutdown_connections 2024-03-19T02:32:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:02.499+0000 7f47bcdd3640 1 -- 172.21.15.116:0/2860882173 wait complete. 2024-03-19T02:32:02.511 INFO:tasks.cephfs.filesystem:scrub status for tag:8f4ca2ec-dcd7-4683-ad34-b6f4d2fc47ed - {'path': '/', 'tag': '8f4ca2ec-dcd7-4683-ad34-b6f4d2fc47ed', 'options': 'recursive,force'} 2024-03-19T02:32:02.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 124796 3.25 MB/sec execute 85 sec latency 43.597 ms 2024-03-19T02:32:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:02 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:32:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:32:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:02 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:32:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:32:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:02 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:32:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:32:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:03.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 125415 3.23 MB/sec execute 86 sec latency 16.345 ms 2024-03-19T02:32:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:03 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:32:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:03 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:04.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:03 smithi162 ceph-mon[35761]: pgmap v130: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:32:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:03 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:32:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:03 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:03 smithi154 ceph-mon[35693]: pgmap v130: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:32:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:03 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:32:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:03 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:03 smithi116 ceph-mon[32263]: pgmap v130: 129 pgs: 129 active+clean; 102 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-03-19T02:32:04.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 126096 3.23 MB/sec execute 87 sec latency 18.144 ms 2024-03-19T02:32:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 126766 3.23 MB/sec execute 88 sec latency 16.381 ms 2024-03-19T02:32:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:05 smithi162 ceph-mon[35761]: pgmap v131: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:32:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:05 smithi154 ceph-mon[35693]: pgmap v131: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:32:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:05 smithi116 ceph-mon[32263]: pgmap v131: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:32:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 127505 3.26 MB/sec execute 89 sec latency 16.163 ms 2024-03-19T02:32:07.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 128029 3.27 MB/sec execute 90 sec latency 52.340 ms 2024-03-19T02:32:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:07 smithi162 ceph-mon[35761]: pgmap v132: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:07 smithi154 ceph-mon[35693]: pgmap v132: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:07 smithi116 ceph-mon[32263]: pgmap v132: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:08.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 128448 3.24 MB/sec execute 91 sec latency 35.346 ms 2024-03-19T02:32:09.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 129129 3.26 MB/sec execute 92 sec latency 15.948 ms 2024-03-19T02:32:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:09 smithi162 ceph-mon[35761]: pgmap v133: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:09 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:32:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:09 smithi154 ceph-mon[35693]: pgmap v133: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:09 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:32:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:09 smithi116 ceph-mon[32263]: pgmap v133: 129 pgs: 129 active+clean; 108 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.4 MiB/s wr, 100 op/s 2024-03-19T02:32:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:09 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:32:10.783 INFO:tasks.workunit.client.0.smithi116.stdout: 1 129783 3.23 MB/sec execute 93 sec latency 17.840 ms 2024-03-19T02:32:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:32:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:32:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:32:11.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 130477 3.25 MB/sec execute 94 sec latency 17.778 ms 2024-03-19T02:32:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:11 smithi162 ceph-mon[35761]: pgmap v134: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 2.2 MiB/s wr, 140 op/s 2024-03-19T02:32:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:11 smithi154 ceph-mon[35693]: pgmap v134: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 2.2 MiB/s wr, 140 op/s 2024-03-19T02:32:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:11 smithi116 ceph-mon[32263]: pgmap v134: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 2.2 MiB/s wr, 140 op/s 2024-03-19T02:32:12.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 131050 3.26 MB/sec execute 95 sec latency 76.916 ms 2024-03-19T02:32:13.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:12 smithi162 ceph-mon[35761]: pgmap v135: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.6 MiB/s wr, 98 op/s 2024-03-19T02:32:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:12 smithi154 ceph-mon[35693]: pgmap v135: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.6 MiB/s wr, 98 op/s 2024-03-19T02:32:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:12 smithi116 ceph-mon[32263]: pgmap v135: 129 pgs: 129 active+clean; 113 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 1.6 MiB/s wr, 98 op/s 2024-03-19T02:32:13.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 131624 3.27 MB/sec execute 96 sec latency 24.490 ms 2024-03-19T02:32:14.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 132105 3.24 MB/sec execute 97 sec latency 31.310 ms 2024-03-19T02:32:15.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 132731 3.26 MB/sec execute 98 sec latency 21.109 ms 2024-03-19T02:32:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:15 smithi162 ceph-mon[35761]: pgmap v136: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:32:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:15 smithi154 ceph-mon[35693]: pgmap v136: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:32:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:15 smithi116 ceph-mon[32263]: pgmap v136: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:32:16.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 133384 3.23 MB/sec execute 99 sec latency 25.815 ms 2024-03-19T02:32:17.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 134043 3.25 MB/sec execute 100 sec latency 19.053 ms 2024-03-19T02:32:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:17 smithi162 ceph-mon[35761]: pgmap v137: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:17 smithi154 ceph-mon[35693]: pgmap v137: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:17 smithi116 ceph-mon[32263]: pgmap v137: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:18.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 134625 3.25 MB/sec execute 101 sec latency 16.405 ms 2024-03-19T02:32:19.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 135204 3.27 MB/sec execute 102 sec latency 18.866 ms 2024-03-19T02:32:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:19 smithi162 ceph-mon[35761]: pgmap v138: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:19 smithi154 ceph-mon[35693]: pgmap v138: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:19 smithi116 ceph-mon[32263]: pgmap v138: 129 pgs: 129 active+clean; 116 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:32:20.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 135685 3.24 MB/sec execute 103 sec latency 41.225 ms 2024-03-19T02:32:21.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 136327 3.25 MB/sec execute 104 sec latency 19.947 ms 2024-03-19T02:32:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:21 smithi162 ceph-mon[35761]: pgmap v139: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-19T02:32:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:21 smithi154 ceph-mon[35693]: pgmap v139: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-19T02:32:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:21 smithi116 ceph-mon[32263]: pgmap v139: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-03-19T02:32:22.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 136999 3.23 MB/sec execute 105 sec latency 23.928 ms 2024-03-19T02:32:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 137755 3.26 MB/sec execute 106 sec latency 15.677 ms 2024-03-19T02:32:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:23 smithi162 ceph-mon[35761]: pgmap v140: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:23 smithi154 ceph-mon[35693]: pgmap v140: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:23 smithi116 ceph-mon[32263]: pgmap v140: 129 pgs: 129 active+clean; 118 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:24.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 138205 3.25 MB/sec execute 107 sec latency 17.981 ms 2024-03-19T02:32:25.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 138764 3.26 MB/sec execute 108 sec latency 37.568 ms 2024-03-19T02:32:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:25 smithi162 ceph-mon[35761]: pgmap v141: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:25 smithi154 ceph-mon[35693]: pgmap v141: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:25 smithi116 ceph-mon[32263]: pgmap v141: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:26.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 139275 3.24 MB/sec execute 109 sec latency 31.413 ms 2024-03-19T02:32:27.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 139949 3.25 MB/sec execute 110 sec latency 16.925 ms 2024-03-19T02:32:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:27 smithi162 ceph-mon[35761]: pgmap v142: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:27 smithi154 ceph-mon[35693]: pgmap v142: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:27 smithi116 ceph-mon[32263]: pgmap v142: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:28.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 140667 3.24 MB/sec execute 111 sec latency 16.733 ms 2024-03-19T02:32:29.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 141359 3.26 MB/sec execute 112 sec latency 16.221 ms 2024-03-19T02:32:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:29 smithi162 ceph-mon[35761]: pgmap v143: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:29 smithi154 ceph-mon[35693]: pgmap v143: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:30.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:29 smithi116 ceph-mon[32263]: pgmap v143: 129 pgs: 129 active+clean; 120 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:30.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 141922 3.27 MB/sec execute 113 sec latency 16.243 ms 2024-03-19T02:32:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 142412 3.26 MB/sec execute 114 sec latency 16.477 ms 2024-03-19T02:32:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:31 smithi162 ceph-mon[35761]: pgmap v144: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:32:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:31 smithi154 ceph-mon[35693]: pgmap v144: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:32:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:31 smithi116 ceph-mon[32263]: pgmap v144: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-03-19T02:32:32.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:32:32.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 -- 172.21.15.116:0/4114845852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 msgr2=0x7f31c8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 --2- 172.21.15.116:0/4114845852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8100b40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009900 tx=0x7f31b002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 -- 172.21.15.116:0/4114845852 shutdown_connections 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 --2- 172.21.15.116:0/4114845852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c8101080 0x7f31c810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 --2- 172.21.15.116:0/4114845852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8100b40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 --2- 172.21.15.116:0/4114845852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c81055c0 0x7f31c81059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 -- 172.21.15.116:0/4114845852 >> 172.21.15.116:0/4114845852 conn(0x7f31c80777b0 msgr2=0x7f31c8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 -- 172.21.15.116:0/4114845852 shutdown_connections 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.628+0000 7f31cd945640 1 -- 172.21.15.116:0/4114845852 wait complete. 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 Processor -- start 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 -- start start 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c8101080 0x7f31c8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c81055c0 0x7f31c819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31c8110960 con 0x7f31c81006e0 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81107e0 con 0x7f31c81055c0 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31cd945640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31c8110ae0 con 0x7f31c8101080 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41778/0 (socket says 172.21.15.116:41778) 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2333292786 learned_addr learned my addr 172.21.15.116:0/2333292786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31bffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c8101080 0x7f31c8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c77fe640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c81055c0 0x7f31c819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2333292786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c8101080 msgr2=0x7f31c8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c8101080 0x7f31c8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2333292786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c81055c0 msgr2=0x7f31c819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c81055c0 0x7f31c819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.629+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2333292786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c819cf90 con 0x7f31c81006e0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9f0 tx=0x7f31b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31cc943640 1 -- 172.21.15.116:0/2333292786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800cd80 con 0x7f31c81006e0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31cc943640 1 -- 172.21.15.116:0/2333292786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31b80054d0 con 0x7f31c81006e0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31cc943640 1 -- 172.21.15.116:0/2333292786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8010490 con 0x7f31c81006e0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c819d280 con 0x7f31c81006e0 2024-03-19T02:32:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.630+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c81a4b10 con 0x7f31c81006e0 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.632+0000 7f31cc943640 1 -- 172.21.15.116:0/2333292786 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f31b8005020 con 0x7f31c81006e0 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.632+0000 7f31cc943640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 0x7f31b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31cc943640 1 -- 172.21.15.116:0/2333292786 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f31b80cf4a0 con 0x7f31c81006e0 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 0x7f31b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 Processor -- start 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 -- start start 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x564920428e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x564920434320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56492038c500 con 0x564920429350 2024-03-19T02:32:32.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564920334be0 con 0x564920428a30 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31d060d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649203885a0 con 0x564920431f30 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41792/0 (socket says 172.21.15.116:41792) 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 -- 172.21.15.116:0/3397144725 learned_addr learned my addr 172.21.15.116:0/3397144725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x564920428e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31c77fe640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x564920434320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 0x7f31b40a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006010 tx=0x7f31b0005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 -- 172.21.15.116:0/3397144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 msgr2=0x564920434320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x564920434320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 -- 172.21.15.116:0/3397144725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 msgr2=0x564920428e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x564920428e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31bffff640 1 -- 172.21.15.116:0/3397144725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564920356000 con 0x564920429350 2024-03-19T02:32:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.633+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x564920428e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31bffff640 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f31b0035000 tx=0x7f31b003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a4adb71c8028e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31bdffb640 1 -- 172.21.15.116:0/3397144725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0047050 con 0x564920429350 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31bdffb640 1 -- 172.21.15.116:0/3397144725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31b0051070 con 0x564920429350 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31bdffb640 1 -- 172.21.15.116:0/3397144725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0003850 con 0x564920429350 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 -- 172.21.15.116:0/3397144725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 msgr2=0x5649204319f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f31b0035000 tx=0x7f31b003e000 comp rx=0 tx=0).stop 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 -- 172.21.15.116:0/3397144725 shutdown_connections 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x564920434320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x5649204319f0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 --2- 172.21.15.116:0/3397144725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x564920428e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.634+0000 7f31d060d740 1 -- 172.21.15.116:0/3397144725 >> 172.21.15.116:0/3397144725 conn(0x564920423160 msgr2=0x564920423540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- 172.21.15.116:0/3397144725 shutdown_connections 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- 172.21.15.116:0/3397144725 wait complete. 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- start start 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 Processor -- start 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492042f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x56492042f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564920434cc0 con 0x564920428a30 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564920348160 con 0x564920431f30 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564920434e40 con 0x564920429350 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31d060d740 10 client.0 _ll_register_callbacks cb 0x564920384a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.635+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41800/0 (socket says 172.21.15.116:41800) 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2550234113 learned_addr learned my addr 172.21.15.116:0/2550234113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c77fe640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x56492042f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31bffff640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492042f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2550234113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 msgr2=0x56492042f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492042f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 msgr2=0x56492042f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x56492042f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2550234113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492016a7d0 con 0x564920428a30 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c77fe640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x56492042f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9c0 tx=0x7f31b80053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf677f4fea3785e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b809b780 con 0x564920428a30 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f31b8097df0 con 0x564920428a30 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b0051070 con 0x564920428a30 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f31b809f8d0 con 0x564920428a30 2024-03-19T02:32:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31d060d740 10 client.17016 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5649200a03f0 con 0x564920428a30 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.636+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31b0051070 con 0x564920428a30 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f31b8097040 con 0x564920428a30 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 10 client.17016 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 20 client.17016 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204421e0 0x564920444630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 4 client.17016 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564920399a40 con 0x5649204421e0 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31bffff640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204421e0 0x564920444630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31a6ffd640 10 client.17016 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.637+0000 7f31bffff640 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204421e0 0x564920444630 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=176c903d533183f5 server_cookie=13eadcd770e672e0 in_seq=0 out_seq=0 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31a6ffd640 1 -- 172.21.15.116:0/2550234113 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564920399a40 con 0x5649204421e0 2024-03-19T02:32:32.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31a6ffd640 10 client.17016 handle_command_reply: tid=0 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 client.17016 shutdown 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 msgr2=0x56492042eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9c0 tx=0x7f31b80053d0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 shutdown_connections 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204421e0 0x564920444630 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x56492042f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492042f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 --2- 172.21.15.116:0/2550234113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042eed0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31a6ffd640 0 client.17016 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 >> 172.21.15.116:0/2550234113 conn(0x564920423160 msgr2=0x564920424eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 shutdown_connections 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 1 -- 172.21.15.116:0/2550234113 wait complete. 2024-03-19T02:32:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.638+0000 7f31d060d740 20 client.17016 trim_cache size 0 max 16384 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.656+0000 7f31d060d740 1 Processor -- start 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.656+0000 7f31d060d740 1 -- start start 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492043da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x564920449ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81aeb40 con 0x564920428a30 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81b0680 con 0x564920431f30 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31d060d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81af210 con 0x564920429350 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41816/0 (socket says 172.21.15.116:41816) 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2774656133 learned_addr learned my addr 172.21.15.116:0/2774656133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31bffff640 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492043da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c77fe640 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x564920449ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2774656133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 msgr2=0x56492043da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492043da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2774656133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 msgr2=0x564920449ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x564920449ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 -- 172.21.15.116:0/2774656133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564920434690 con 0x564920428a30 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f31b809b710 tx=0x7f31b8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e8a93225d93957 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31bd7fa640 1 -- 172.21.15.116:0/2774656133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b80a5080 con 0x564920428a30 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31bd7fa640 1 -- 172.21.15.116:0/2774656133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31b80a3f20 con 0x564920428a30 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.657+0000 7f31bd7fa640 1 -- 172.21.15.116:0/2774656133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8097a20 con 0x564920428a30 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- 172.21.15.116:0/2774656133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 msgr2=0x56492042f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f31b809b710 tx=0x7f31b8009310 comp rx=0 tx=0).stop 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- 172.21.15.116:0/2774656133 shutdown_connections 2024-03-19T02:32:32.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920431f30 0x564920449ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920429350 0x56492043da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 --2- 172.21.15.116:0/2774656133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920428a30 0x56492042f720 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- 172.21.15.116:0/2774656133 >> 172.21.15.116:0/2774656133 conn(0x564920423160 msgr2=0x564920421950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- 172.21.15.116:0/2774656133 shutdown_connections 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- 172.21.15.116:0/2774656133 wait complete. 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.658+0000 7f31d060d740 1 -- start start 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 Processor -- start 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x56492043a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x5649204375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81affb0 con 0x564920429350 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81af8e0 con 0x564920428a30 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31c81acb20 con 0x564920431f30 2024-03-19T02:32:32.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31d060d740 10 client.0 _ll_register_callbacks cb 0x7f31b809b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41818/0 (socket says 172.21.15.116:41818) 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x56492043a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31bffff640 1 -- 172.21.15.116:0/94986256 learned_addr learned my addr 172.21.15.116:0/94986256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c77fe640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x5649204375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 -- 172.21.15.116:0/94986256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 msgr2=0x5649204375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x5649204375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 -- 172.21.15.116:0/94986256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 msgr2=0x564920437090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 -- 172.21.15.116:0/94986256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649200600f0 con 0x564920428a30 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31bffff640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c77fe640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x5649204375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:32.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x56492043a320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9c0 tx=0x7f31b80057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0d9821a4c759bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.659+0000 7f31a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b80a5070 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f31b80cf160 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b80a3f20 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f31b8097a20 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 10 client.26840 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5649204a92d0 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31b80a3f20 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f31b8097040 con 0x564920428a30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 10 client.26840 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 10 client.26840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 20 client.26840 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204a9610 0x5649204aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 4 client.26840 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:32:32.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564920393d90 con 0x5649204a9610 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.660+0000 7f31bffff640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204a9610 0x5649204aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31a6ffd640 10 client.26840 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31bffff640 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204a9610 0x5649204aba00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6a269a1471abea server_cookie=5beb0d6459cdb16f in_seq=0 out_seq=0 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31a6ffd640 1 -- 172.21.15.116:0/94986256 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564920393d90 con 0x5649204a9610 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31a6ffd640 10 client.26840 handle_command_reply: tid=0 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 client.26840 shutdown 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 msgr2=0x56492043a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x56492043a320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9c0 tx=0x7f31b80057b0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 shutdown_connections 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5649204a9610 0x5649204aba00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564920431f30 0x5649204375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564920429350 0x564920437090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 --2- 172.21.15.116:0/94986256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564920428a30 0x56492043a320 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31a6ffd640 0 client.26840 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 >> 172.21.15.116:0/94986256 conn(0x564920423160 msgr2=0x56492042f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 shutdown_connections 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 1 -- 172.21.15.116:0/94986256 wait complete. 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.661+0000 7f31d060d740 20 client.26840 trim_cache size 0 max 16384 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:32:32.669 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:32:32.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 msgr2=0x7f31b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 0x7f31b40a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006010 tx=0x7f31b0005e40 comp rx=0 tx=0).stop 2024-03-19T02:32:32.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 msgr2=0x7f31c8197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f31b800e9f0 tx=0x7f31b800edf0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 shutdown_connections 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31b40a2970 0x7f31b40a4e30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31c81055c0 0x7f31c819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31c8101080 0x7f31c8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 --2- 172.21.15.116:0/2333292786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c81006e0 0x7f31c8197ff0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 >> 172.21.15.116:0/2333292786 conn(0x7f31c80777b0 msgr2=0x7f31c8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 shutdown_connections 2024-03-19T02:32:32.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.662+0000 7f31cd945640 1 -- 172.21.15.116:0/2333292786 wait complete. 2024-03-19T02:32:32.674 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:32:32.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:32:32.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 142968 3.25 MB/sec execute 115 sec latency 38.780 ms 2024-03-19T02:32:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 -- 172.21.15.116:0/4221327669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e8100720 msgr2=0x7f65e8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 --2- 172.21.15.116:0/4221327669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e8100720 0x7f65e8100ba0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009930 tx=0x7f65dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 -- 172.21.15.116:0/4221327669 shutdown_connections 2024-03-19T02:32:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 --2- 172.21.15.116:0/4221327669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e81010e0 0x7f65e810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 --2- 172.21.15.116:0/4221327669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e8100720 0x7f65e8100ba0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 --2- 172.21.15.116:0/4221327669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8106730 0x7f65e8106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 -- 172.21.15.116:0/4221327669 >> 172.21.15.116:0/4221327669 conn(0x7f65e8077820 msgr2=0x7f65e8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 -- 172.21.15.116:0/4221327669 shutdown_connections 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.784+0000 7f65ee214640 1 -- 172.21.15.116:0/4221327669 wait complete. 2024-03-19T02:32:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 Processor -- start 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 -- start start 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 0x7f65e8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 0x7f65e819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81109b0 con 0x7f65e81010e0 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8110830 con 0x7f65e8106730 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65ee214640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65e8110b30 con 0x7f65e8100720 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41838/0 (socket says 172.21.15.116:41838) 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65e6ffd640 1 -- 172.21.15.116:0/854756645 learned_addr learned my addr 172.21.15.116:0/854756645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65e77fe640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 0x7f65e8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.785+0000 7f65e7fff640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 0x7f65e819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 -- 172.21.15.116:0/854756645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 msgr2=0x7f65e8198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 0x7f65e8198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 -- 172.21.15.116:0/854756645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 msgr2=0x7f65e819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 0x7f65e819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 -- 172.21.15.116:0/854756645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e819d000 con 0x7f65e81010e0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e77fe640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 0x7f65e8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e7fff640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 0x7f65e819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e6ffd640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009230 tx=0x7f65dc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc043020 con 0x7f65e81010e0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e819d290 con 0x7f65e81010e0 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65dc02ca70 con 0x7f65e81010e0 2024-03-19T02:32:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc003a50 con 0x7f65e81010e0 2024-03-19T02:32:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.786+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e81a4b80 con 0x7f65e81010e0 2024-03-19T02:32:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.787+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e8068610 con 0x7f65e81010e0 2024-03-19T02:32:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.789+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f65dc046020 con 0x7f65e81010e0 2024-03-19T02:32:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.789+0000 7f65e4ff9640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 0x7f65d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.789+0000 7f65e77fe640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 0x7f65d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.789+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f65dc0f10c0 con 0x7f65e81010e0 2024-03-19T02:32:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.790+0000 7f65e77fe640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 0x7f65d00a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65d80095f0 tx=0x7f65d8005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:32.794+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65dc04d020 con 0x7f65e81010e0 2024-03-19T02:32:33.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.041+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65e8109cc0 con 0x7f65e81010e0 2024-03-19T02:32:33.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.042+0000 7f65e4ff9640 1 -- 172.21.15.116:0/854756645 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f65dc0bdcd0 con 0x7f65e81010e0 2024-03-19T02:32:33.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:32:33.043 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:32:33.044 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:32:33.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.046+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 msgr2=0x7f65d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.046+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 0x7f65d00a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65d80095f0 tx=0x7f65d8005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:33.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.046+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 msgr2=0x7f65e81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.046+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009230 tx=0x7f65dc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 shutdown_connections 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65d00a2970 0x7f65d00a4e30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8106730 0x7f65e819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e81010e0 0x7f65e81985a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 --2- 172.21.15.116:0/854756645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65e8100720 0x7f65e8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 >> 172.21.15.116:0/854756645 conn(0x7f65e8077820 msgr2=0x7f65e8068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 shutdown_connections 2024-03-19T02:32:33.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.047+0000 7f65ee214640 1 -- 172.21.15.116:0/854756645 wait complete. 2024-03-19T02:32:33.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:32:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- 172.21.15.116:0/2224997577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c101060 msgr2=0x7fde3c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 --2- 172.21.15.116:0/2224997577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c101060 0x7fde3c10d570 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fde34009cd0 tx=0x7fde3402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- 172.21.15.116:0/2224997577 shutdown_connections 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 --2- 172.21.15.116:0/2224997577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c101060 0x7fde3c10d570 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 --2- 172.21.15.116:0/2224997577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c1006a0 0x7fde3c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 --2- 172.21.15.116:0/2224997577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1066b0 0x7fde3c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- 172.21.15.116:0/2224997577 >> 172.21.15.116:0/2224997577 conn(0x7fde3c0778d0 msgr2=0x7fde3c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- 172.21.15.116:0/2224997577 shutdown_connections 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- 172.21.15.116:0/2224997577 wait complete. 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 Processor -- start 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.176+0000 7fde43fe6640 1 -- start start 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 0x7fde3c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c1066b0 0x7fde3c1a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c110970 con 0x7fde3c1066b0 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1107f0 con 0x7fde3c1006a0 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde43fe6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c110af0 con 0x7fde3c101060 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41581640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 0x7fde3c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde42583640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c1066b0 0x7fde3c1a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38584/0 (socket says 172.21.15.116:38584) 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 -- 172.21.15.116:0/3689749335 learned_addr learned my addr 172.21.15.116:0/3689749335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 -- 172.21.15.116:0/3689749335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 msgr2=0x7fde3c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 0x7fde3c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 -- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c1066b0 msgr2=0x7fde3c1a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c1066b0 0x7fde3c1a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 -- 172.21.15.116:0/3689749335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1a4310 con 0x7fde3c1006a0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41581640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 0x7fde3c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde41d82640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e9f0 tx=0x7fde3800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.177+0000 7fde32ffd640 1 -- 172.21.15.116:0/3689749335 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3800cd80 con 0x7fde3c1006a0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.178+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c1a4540 con 0x7fde3c1006a0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.178+0000 7fde32ffd640 1 -- 172.21.15.116:0/3689749335 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde380108e0 con 0x7fde3c1006a0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.178+0000 7fde32ffd640 1 -- 172.21.15.116:0/3689749335 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38018920 con 0x7fde3c1006a0 2024-03-19T02:32:33.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.178+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a4a60 con 0x7fde3c1006a0 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.180+0000 7fde32ffd640 1 -- 172.21.15.116:0/3689749335 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fde38010a90 con 0x7fde3c1006a0 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.180+0000 7fde32ffd640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 0x7fde280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.180+0000 7fde46bce740 1 Processor -- start 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 0x7fde280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde32ffd640 1 -- 172.21.15.116:0/3689749335 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fde380cf490 con 0x7fde3c1006a0 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 -- start start 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefbe290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefc0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558abef184b0 con 0x558abefb52d0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558abeec0d70 con 0x558abefbe7d0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde46bce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558abef14550 con 0x558abefb5bf0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 0x7fde280a4e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde3c069530 tx=0x7fde2c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41d82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefbe290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41d82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41862/0 (socket says 172.21.15.116:41862) 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41d82640 1 -- 172.21.15.116:0/3033695066 learned_addr learned my addr 172.21.15.116:0/3033695066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde42583640 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefc0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 -- 172.21.15.116:0/3033695066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 msgr2=0x558abefc0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefc0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 -- 172.21.15.116:0/3033695066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 msgr2=0x558abefb56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41581640 1 -- 172.21.15.116:0/3033695066 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558abeee3210 con 0x558abefb5bf0 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.181+0000 7fde41d82640 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.182+0000 7fde41581640 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefbe290 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0041e0 tx=0x7fde2c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b418e5ab4cbbec00 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.182+0000 7fde27fff640 1 -- 172.21.15.116:0/3033695066 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00f480 con 0x558abefb5bf0 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.182+0000 7fde27fff640 1 -- 172.21.15.116:0/3033695066 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde2c00fad0 con 0x558abefb5bf0 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.182+0000 7fde27fff640 1 -- 172.21.15.116:0/3033695066 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c01d9f0 con 0x558abefb5bf0 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- 172.21.15.116:0/3033695066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 msgr2=0x558abefbe290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefbe290 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0041e0 tx=0x7fde2c014000 comp rx=0 tx=0).stop 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- 172.21.15.116:0/3033695066 shutdown_connections 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefc0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefbe290 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 --2- 172.21.15.116:0/3033695066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefb56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- 172.21.15.116:0/3033695066 >> 172.21.15.116:0/3033695066 conn(0x558abefafa00 msgr2=0x558abefafde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- 172.21.15.116:0/3033695066 shutdown_connections 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- 172.21.15.116:0/3033695066 wait complete. 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.183+0000 7fde46bce740 1 -- start start 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 Processor -- start 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefbc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558abefc15c0 con 0x558abefb5bf0 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558abeed4310 con 0x558abefbe7d0 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558abefc1740 con 0x558abefb52d0 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde41581640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde41581640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41872/0 (socket says 172.21.15.116:41872) 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde41d82640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde41581640 1 -- 172.21.15.116:0/1067675914 learned_addr learned my addr 172.21.15.116:0/1067675914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde46bce740 10 client.0 _ll_register_callbacks cb 0x558abeec82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.184+0000 7fde42583640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefbc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 -- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 msgr2=0x558abefbb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 -- 172.21.15.116:0/1067675914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 msgr2=0x558abefbc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefbc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 -- 172.21.15.116:0/1067675914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558abec2c3f0 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde42583640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefbc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde41581640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009b70 tx=0x7fde2c003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6078c887b1b40dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde257fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c033060 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c00fad0 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fde2c01d470 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fde2c03a050 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde46bce740 10 client.17040 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.185+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558abecf67d0 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde2c00fad0 con 0x558abefb5bf0 2024-03-19T02:32:33.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fde2c01a070 con 0x558abefb5bf0 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 10 client.17040 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 10 client.17040 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 20 client.17040 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abefceb00 0x558abefd0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 4 client.17040 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558abef763e0 con 0x558abefceb00 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.186+0000 7fde41d82640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abefceb00 0x558abefd0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.187+0000 7fde257fa640 10 client.17040 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.187+0000 7fde41d82640 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abefceb00 0x558abefd0ef0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3002daf1d6ddd480 server_cookie=6c4b3345c8fb19ea in_seq=0 out_seq=0 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.187+0000 7fde257fa640 1 -- 172.21.15.116:0/1067675914 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558abef763e0 con 0x558abefceb00 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.187+0000 7fde257fa640 10 client.17040 handle_command_reply: tid=0 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 client.17040 shutdown 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 msgr2=0x558abefbbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009b70 tx=0x7fde2c003e60 comp rx=0 tx=0).stop 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 shutdown_connections 2024-03-19T02:32:33.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abefceb00 0x558abefd0ef0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefbc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde257fa640 0 client.17040 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefbbdc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 --2- 172.21.15.116:0/1067675914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 >> 172.21.15.116:0/1067675914 conn(0x558abefafa00 msgr2=0x558abefb1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 shutdown_connections 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 1 -- 172.21.15.116:0/1067675914 wait complete. 2024-03-19T02:32:33.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.188+0000 7fde46bce740 20 client.17040 trim_cache size 0 max 16384 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 Processor -- start 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 -- start start 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1aeb90 con 0x558abefb5bf0 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b06d0 con 0x558abefbe7d0 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde46bce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1af260 con 0x558abefb52d0 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde41581640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde41581640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41884/0 (socket says 172.21.15.116:41884) 2024-03-19T02:32:33.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.202+0000 7fde41581640 1 -- 172.21.15.116:0/2358130149 learned_addr learned my addr 172.21.15.116:0/2358130149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41d82640 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde42583640 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 -- 172.21.15.116:0/2358130149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 msgr2=0x558abefbdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 -- 172.21.15.116:0/2358130149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 msgr2=0x558abefd3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 -- 172.21.15.116:0/2358130149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558abefba870 con 0x558abefb5bf0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde41581640 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde2c011ca0 tx=0x7fde2c003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bbd8f75e572b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde277fe640 1 -- 172.21.15.116:0/2358130149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c033070 con 0x558abefb5bf0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde277fe640 1 -- 172.21.15.116:0/2358130149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde2c003a80 con 0x558abefb5bf0 2024-03-19T02:32:33.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.203+0000 7fde277fe640 1 -- 172.21.15.116:0/2358130149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00f770 con 0x558abefb5bf0 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- 172.21.15.116:0/2358130149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 msgr2=0x558abefca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde2c011ca0 tx=0x7fde2c003f50 comp rx=0 tx=0).stop 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- 172.21.15.116:0/2358130149 shutdown_connections 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb5bf0 0x558abefca220 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 --2- 172.21.15.116:0/2358130149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb52d0 0x558abefbdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- 172.21.15.116:0/2358130149 >> 172.21.15.116:0/2358130149 conn(0x558abefafa00 msgr2=0x558abefafde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- 172.21.15.116:0/2358130149 shutdown_connections 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- 172.21.15.116:0/2358130149 wait complete. 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 -- start start 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.204+0000 7fde46bce740 1 Processor -- start 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefd14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1b0000 con 0x558abefb52d0 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1af930 con 0x558abefbe7d0 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1acb70 con 0x558abefb5bf0 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 10 client.0 _ll_register_callbacks cb 0x7fde380973a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41900/0 (socket says 172.21.15.116:41900) 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 -- 172.21.15.116:0/3653531478 learned_addr learned my addr 172.21.15.116:0/3653531478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde42583640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41581640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefd14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 -- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 msgr2=0x558abefd14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefd14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 -- 172.21.15.116:0/3653531478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 msgr2=0x558abefd19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 -- 172.21.15.116:0/3653531478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558abefd21f0 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde41d82640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde38097c80 tx=0x7fde3801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fdd9ce69d80ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde257fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde380ae070 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fde380ccae0 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c003a80 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.205+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fde380a3f10 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 10 client.17046 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558abf038e80 con 0x558abefb52d0 2024-03-19T02:32:33.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde2c003a80 con 0x558abefb52d0 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fde380afb50 con 0x558abefb52d0 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 10 client.17046 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 10 client.17046 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 20 client.17046 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abf039550 0x558abf03b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 4 client.17046 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558abef259f0 con 0x558abf039550 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde41581640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abf039550 0x558abf03b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde257fa640 10 client.17046 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.206+0000 7fde41581640 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abf039550 0x558abf03b970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=610d2a394a6b0b45 server_cookie=cc7ea437eaa78270 in_seq=0 out_seq=0 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde257fa640 1 -- 172.21.15.116:0/3653531478 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558abef259f0 con 0x558abf039550 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde257fa640 10 client.17046 handle_command_reply: tid=0 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 client.17046 shutdown 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 msgr2=0x558abefd4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde38097c80 tx=0x7fde3801a040 comp rx=0 tx=0).stop 2024-03-19T02:32:33.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 shutdown_connections 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558abf039550 0x558abf03b970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558abefbe7d0 0x558abefd19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde257fa640 0 client.17046 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558abefb5bf0 0x558abefd14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 --2- 172.21.15.116:0/3653531478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558abefb52d0 0x558abefd4120 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 >> 172.21.15.116:0/3653531478 conn(0x558abefafa00 msgr2=0x558abefaeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 shutdown_connections 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 1 -- 172.21.15.116:0/3653531478 wait complete. 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde46bce740 20 client.17046 trim_cache size 0 max 16384 2024-03-19T02:32:33.214 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 msgr2=0x7fde280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 0x7fde280a4e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde3c069530 tx=0x7fde2c005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 msgr2=0x7fde3c198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e9f0 tx=0x7fde3800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 shutdown_connections 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fde280a2970 0x7fde280a4e30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde3c1066b0 0x7fde3c1a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde3c101060 0x7fde3c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.207+0000 7fde43fe6640 1 --2- 172.21.15.116:0/3689749335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde3c1006a0 0x7fde3c198d50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.208+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 >> 172.21.15.116:0/3689749335 conn(0x7fde3c0778d0 msgr2=0x7fde3c102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.208+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 shutdown_connections 2024-03-19T02:32:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.208+0000 7fde43fe6640 1 -- 172.21.15.116:0/3689749335 wait complete. 2024-03-19T02:32:33.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:32:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 -- 172.21.15.116:0/3801102064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d30101060 msgr2=0x7f7d3010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 --2- 172.21.15.116:0/3801102064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d30101060 0x7f7d3010d570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800bda0 tx=0x7f7d2802dc50 comp rx=0 tx=0).stop 2024-03-19T02:32:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 -- 172.21.15.116:0/3801102064 shutdown_connections 2024-03-19T02:32:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 --2- 172.21.15.116:0/3801102064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d30101060 0x7f7d3010d570 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 --2- 172.21.15.116:0/3801102064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 --2- 172.21.15.116:0/3801102064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d301066b0 0x7f7d30106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 -- 172.21.15.116:0/3801102064 >> 172.21.15.116:0/3801102064 conn(0x7f7d300778d0 msgr2=0x7f7d30077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 -- 172.21.15.116:0/3801102064 shutdown_connections 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.328+0000 7f7d35b68640 1 -- 172.21.15.116:0/3801102064 wait complete. 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 Processor -- start 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 -- start start 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d30101060 0x7f7d301962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 0x7f7d3019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3010f860 con 0x7f7d30101060 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3010f6e0 con 0x7f7d301006a0 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d35b68640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3010f9e0 con 0x7f7d301066b0 2024-03-19T02:32:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d30101060 0x7f7d301962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 0x7f7d3019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.329+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 0x7f7d3019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50464/0 (socket says 172.21.15.116:50464) 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2ffff640 1 -- 172.21.15.116:0/973384052 learned_addr learned my addr 172.21.15.116:0/973384052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/973384052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 msgr2=0x7f7d3019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 0x7f7d3019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d30101060 msgr2=0x7f7d301962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d30101060 0x7f7d301962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/973384052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3019ad90 con 0x7f7d301006a0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30195da0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005fd0 tx=0x7f7d2400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2cff9640 1 -- 172.21.15.116:0/973384052 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400cd00 con 0x7f7d301006a0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3019b080 con 0x7f7d301006a0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2cff9640 1 -- 172.21.15.116:0/973384052 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d240035f0 con 0x7f7d301006a0 2024-03-19T02:32:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d2cff9640 1 -- 172.21.15.116:0/973384052 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24010690 con 0x7f7d301006a0 2024-03-19T02:32:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.330+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d301a2910 con 0x7f7d301006a0 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d2cff9640 1 -- 172.21.15.116:0/973384052 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7d240037a0 con 0x7f7d301006a0 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d3882a740 1 Processor -- start 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d3882a740 1 -- start start 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c2b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c34290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.333+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d3882a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5576c1b8e4b0 con 0x5576c1c347d0 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d3882a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5576c1b36d70 con 0x5576c1c2bbf0 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d3882a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5576c1b8a550 con 0x5576c1c2b2d0 2024-03-19T02:32:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c34290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c2b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c34290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38652/0 (socket says 172.21.15.116:38652) 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41934/0 (socket says 172.21.15.116:41934) 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 -- 172.21.15.116:0/1131496957 learned_addr learned my addr 172.21.15.116:0/1131496957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c2b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50472/0 (socket says 172.21.15.116:50472) 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2cff9640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 0x7f7d180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 -- 172.21.15.116:0/1131496957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 msgr2=0x5576c1c2b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c2b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 -- 172.21.15.116:0/1131496957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 msgr2=0x5576c1c34290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c34290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 -- 172.21.15.116:0/1131496957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576c1b59210 con 0x5576c1c347d0 2024-03-19T02:32:33.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2ffff640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 0x7f7d180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2cff9640 1 -- 172.21.15.116:0/973384052 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7d240cf3c0 con 0x7f7d301006a0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d2effd640 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009ca0 tx=0x7f7d20011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410898d5b2b2b785 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d15ffb640 1 -- 172.21.15.116:0/1131496957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2001e070 con 0x5576c1c347d0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d15ffb640 1 -- 172.21.15.116:0/1131496957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d20019ae0 con 0x5576c1c347d0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.334+0000 7f7d15ffb640 1 -- 172.21.15.116:0/1131496957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000e950 con 0x5576c1c347d0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- 172.21.15.116:0/1131496957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 msgr2=0x5576c1c36bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d2ffff640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 0x7f7d180a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d28005f00 tx=0x7f7d2800b710 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009ca0 tx=0x7f7d20011810 comp rx=0 tx=0).stop 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- 172.21.15.116:0/1131496957 shutdown_connections 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c347d0 0x5576c1c36bc0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c34290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1131496957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c2b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- 172.21.15.116:0/1131496957 >> 172.21.15.116:0/1131496957 conn(0x5576c1c25a00 msgr2=0x5576c1c25de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- 172.21.15.116:0/1131496957 shutdown_connections 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- 172.21.15.116:0/1131496957 wait complete. 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.335+0000 7f7d3882a740 1 -- start start 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 Processor -- start 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c347d0 0x5576c1c32250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5576c1c375c0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5576c1b4a310 con 0x5576c1c347d0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5576c1c37740 con 0x5576c1c2b2d0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d3882a740 10 client.0 _ll_register_callbacks cb 0x5576c1b3e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41950/0 (socket says 172.21.15.116:41950) 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 -- 172.21.15.116:0/479002091 learned_addr learned my addr 172.21.15.116:0/479002091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2ffff640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c347d0 0x5576c1c32250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 -- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 msgr2=0x5576c1c317d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c317d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 -- 172.21.15.116:0/479002091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c347d0 msgr2=0x5576c1c32250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c347d0 0x5576c1c32250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.336+0000 7f7d2effd640 1 -- 172.21.15.116:0/479002091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576c18a23f0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d2effd640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009c70 tx=0x7f7d20005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5629485807f5506a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2001e050 con 0x5576c1c2bbf0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7d2000e8d0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d20019ae0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7d20033050 con 0x5576c1c2bbf0 2024-03-19T02:32:33.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 10 client.17058 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5576c196c7d0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d20019ae0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7d20004850 con 0x5576c1c2bbf0 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 10 client.17058 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 10 client.17058 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 20 client.17058 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1c44a10 0x5576c1c46e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 4 client.17058 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576c1bec3e0 con 0x5576c1c44a10 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.337+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1c44a10 0x5576c1c46e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.338+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1c44a10 0x5576c1c46e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c1c74bdbff70d8da server_cookie=5bffa03966e32216 in_seq=0 out_seq=0 2024-03-19T02:32:33.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.338+0000 7f7d0effd640 10 client.17058 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.338+0000 7f7d0effd640 1 -- 172.21.15.116:0/479002091 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576c1bec3e0 con 0x5576c1c44a10 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.338+0000 7f7d0effd640 10 client.17058 handle_command_reply: tid=0 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 client.17058 shutdown 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 msgr2=0x5576c1c31d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009c70 tx=0x7f7d20005270 comp rx=0 tx=0).stop 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 shutdown_connections 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c347d0 0x5576c1c32250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1c44a10 0x5576c1c46e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2bbf0 0x5576c1c31d10 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 --2- 172.21.15.116:0/479002091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c2b2d0 0x5576c1c317d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d0effd640 0 client.17058 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 >> 172.21.15.116:0/479002091 conn(0x5576c1c25a00 msgr2=0x5576c1c27750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 shutdown_connections 2024-03-19T02:32:33.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 1 -- 172.21.15.116:0/479002091 wait complete. 2024-03-19T02:32:33.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.339+0000 7f7d3882a740 20 client.17058 trim_cache size 0 max 16384 2024-03-19T02:32:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 Processor -- start 2024-03-19T02:32:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 -- start start 2024-03-19T02:32:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c40360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c24310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ac950 con 0x5576c1c2b2d0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ae490 con 0x5576c1c2bbf0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d3882a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ad020 con 0x5576c1c347d0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41962/0 (socket says 172.21.15.116:41962) 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/1554080486 learned_addr learned my addr 172.21.15.116:0/1554080486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d2effd640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c40360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.353+0000 7f7d2ffff640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c24310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/1554080486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 msgr2=0x5576c1c24310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c24310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/1554080486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 msgr2=0x5576c1c40360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c40360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/1554080486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576c1c47170 con 0x5576c1c2b2d0 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2effd640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c40360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7d2409b630 tx=0x7f7d240d18d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6804daec0d6fbf29 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d157fa640 1 -- 172.21.15.116:0/1554080486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2409b470 con 0x5576c1c2b2d0 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d157fa640 1 -- 172.21.15.116:0/1554080486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d24009510 con 0x5576c1c2b2d0 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.354+0000 7f7d157fa640 1 -- 172.21.15.116:0/1554080486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2409b000 con 0x5576c1c2b2d0 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- 172.21.15.116:0/1554080486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 msgr2=0x5576c1c310b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7d2409b630 tx=0x7f7d240d18d0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- 172.21.15.116:0/1554080486 shutdown_connections 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c24310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c40360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- 172.21.15.116:0/1554080486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c310b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- 172.21.15.116:0/1554080486 >> 172.21.15.116:0/1554080486 conn(0x5576c1c25a00 msgr2=0x5576c1c32020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- 172.21.15.116:0/1554080486 shutdown_connections 2024-03-19T02:32:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- 172.21.15.116:0/1554080486 wait complete. 2024-03-19T02:32:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- start start 2024-03-19T02:32:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 Processor -- start 2024-03-19T02:32:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c37f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c38490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301addc0 con 0x5576c1c2b2d0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301ad6f0 con 0x5576c1c2bbf0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d301aa930 con 0x5576c1c347d0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.355+0000 7f7d3882a740 10 client.0 _ll_register_callbacks cb 0x7f7d2001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c38490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41972/0 (socket says 172.21.15.116:41972) 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/2210588059 learned_addr learned my addr 172.21.15.116:0/2210588059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2effd640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c37f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 msgr2=0x5576c1c38490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c38490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/2210588059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 msgr2=0x5576c1c37f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c37f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 -- 172.21.15.116:0/2210588059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576c1c38bd0 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2effd640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c37f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d2f7fe640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7d2409b3a0 tx=0x7f7d2409b080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d797f073120f526 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2409b1e0 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7d24003b10 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d24009510 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7d2409b6a0 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d3882a740 10 client.17064 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.356+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5576c1caab20 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d24009510 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7d240a39b0 con 0x5576c1c2b2d0 2024-03-19T02:32:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 10 client.17064 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 10 client.17064 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 20 client.17064 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1cab1e0 0x5576c1cad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 4 client.17064 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576c1b9b9f0 con 0x5576c1cab1e0 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d2effd640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1cab1e0 0x5576c1cad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d0effd640 10 client.17064 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.357+0000 7f7d2effd640 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1cab1e0 0x5576c1cad5d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=850be4560b4a56e0 server_cookie=1dc0ef96545fa52d in_seq=0 out_seq=0 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d0effd640 1 -- 172.21.15.116:0/2210588059 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576c1b9b9f0 con 0x5576c1cab1e0 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d0effd640 10 client.17064 handle_command_reply: tid=0 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 client.17064 shutdown 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 msgr2=0x5576c1c3cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7d2409b3a0 tx=0x7f7d2409b080 comp rx=0 tx=0).stop 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 shutdown_connections 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5576c1c347d0 0x5576c1c38490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5576c1c2bbf0 0x5576c1c37f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5576c1cab1e0 0x5576c1cad5d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 --2- 172.21.15.116:0/2210588059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576c1c2b2d0 0x5576c1c3cc00 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 >> 172.21.15.116:0/2210588059 conn(0x5576c1c25a00 msgr2=0x5576c1c26a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d0effd640 0 client.17064 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 shutdown_connections 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 1 -- 172.21.15.116:0/2210588059 wait complete. 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d3882a740 20 client.17064 trim_cache size 0 max 16384 2024-03-19T02:32:33.365 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 msgr2=0x7f7d180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 0x7f7d180a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d28005f00 tx=0x7f7d2800b710 comp rx=0 tx=0).stop 2024-03-19T02:32:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 msgr2=0x7f7d30195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.358+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30195da0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d24005fd0 tx=0x7f7d2400ee50 comp rx=0 tx=0).stop 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 shutdown_connections 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d180a2970 0x7f7d180a4e30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d301066b0 0x7f7d3019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d30101060 0x7f7d301962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 --2- 172.21.15.116:0/973384052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d301006a0 0x7f7d30195da0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 >> 172.21.15.116:0/973384052 conn(0x7f7d300778d0 msgr2=0x7f7d30102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 shutdown_connections 2024-03-19T02:32:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.359+0000 7f7d35b68640 1 -- 172.21.15.116:0/973384052 wait complete. 2024-03-19T02:32:33.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- 172.21.15.116:0/3399470236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 msgr2=0x7fccc4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/3399470236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc4106a90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009930 tx=0x7fccc002bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- 172.21.15.116:0/3399470236 shutdown_connections 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/3399470236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4101060 0x7fccc410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/3399470236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc41006a0 0x7fccc4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/3399470236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc4106a90 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- 172.21.15.116:0/3399470236 >> 172.21.15.116:0/3399470236 conn(0x7fccc40778d0 msgr2=0x7fccc4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- 172.21.15.116:0/3399470236 shutdown_connections 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- 172.21.15.116:0/3399470236 wait complete. 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 Processor -- start 2024-03-19T02:32:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 -- start start 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc41006a0 0x7fccc4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 0x7fccc41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.488+0000 7fcccc7f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccc7f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4111a70 con 0x7fccc41066b0 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccc7f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41118f0 con 0x7fccc41006a0 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccc7f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4111bf0 con 0x7fccc4101060 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccad90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccad90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41984/0 (socket says 172.21.15.116:41984) 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccca58f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc41006a0 0x7fccc4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccad90640 1 -- 172.21.15.116:0/4123948290 learned_addr learned my addr 172.21.15.116:0/4123948290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 0x7fccc41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 -- 172.21.15.116:0/4123948290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc41006a0 msgr2=0x7fccc4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc41006a0 0x7fccc4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 -- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 msgr2=0x7fccc419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 -- 172.21.15.116:0/4123948290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc419cf60 con 0x7fccc4101060 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccad90640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 0x7fccc41984b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fccb400e9f0 tx=0x7fccb400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccbb7fe640 1 -- 172.21.15.116:0/4123948290 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb400ccc0 con 0x7fccc4101060 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc419d250 con 0x7fccc4101060 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccbb7fe640 1 -- 172.21.15.116:0/4123948290 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccb40108e0 con 0x7fccc4101060 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fccbb7fe640 1 -- 172.21.15.116:0/4123948290 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4018880 con 0x7fccc4101060 2024-03-19T02:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.489+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41a4ae0 con 0x7fccc4101060 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fccbb7fe640 1 -- 172.21.15.116:0/4123948290 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fccb4010a90 con 0x7fccc4101060 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 Processor -- start 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 -- start start 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fccbb7fe640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 0x7fccb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8d1d4b0 con 0x55e9f8dc37d0 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8cc5d70 con 0x55e9f8dbabf0 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.492+0000 7fcccf3da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8d19550 con 0x55e9f8dba2d0 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fcccad90640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 0x7fccb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccbb7fe640 1 -- 172.21.15.116:0/4123948290 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fccb40cf550 con 0x7fccc4101060 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50522/0 (socket says 172.21.15.116:50522) 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38714/0 (socket says 172.21.15.116:38714) 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fcccad90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 -- 172.21.15.116:0/632862891 learned_addr learned my addr 172.21.15.116:0/632862891 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 -- 172.21.15.116:0/632862891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 msgr2=0x55e9f8dc3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 -- 172.21.15.116:0/632862891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 msgr2=0x55e9f8dc5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 -- 172.21.15.116:0/632862891 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f8ce8210 con 0x55e9f8dba2d0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fcccad90640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 0x7fccb00a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0049d0 tx=0x7fccbc0116d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccca58f640 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036040 tx=0x7fccc0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f09b3880fe9744 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccb8ff9640 1 -- 172.21.15.116:0/632862891 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0045020 con 0x55e9f8dba2d0 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccb8ff9640 1 -- 172.21.15.116:0/632862891 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccc002c7e0 con 0x55e9f8dba2d0 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.493+0000 7fccb8ff9640 1 -- 172.21.15.116:0/632862891 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0005430 con 0x55e9f8dba2d0 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 -- 172.21.15.116:0/632862891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 msgr2=0x55e9f8dba6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036040 tx=0x7fccc0009560 comp rx=0 tx=0).stop 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 -- 172.21.15.116:0/632862891 shutdown_connections 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 --2- 172.21.15.116:0/632862891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dba6b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 -- 172.21.15.116:0/632862891 >> 172.21.15.116:0/632862891 conn(0x55e9f8db4a00 msgr2=0x55e9f8db4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 -- 172.21.15.116:0/632862891 shutdown_connections 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.494+0000 7fcccf3da740 1 -- 172.21.15.116:0/632862891 wait complete. 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 -- start start 2024-03-19T02:32:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 Processor -- start 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8dc6560 con 0x55e9f8dbabf0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8cd9310 con 0x55e9f8dc37d0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e9f8dc66e0 con 0x55e9f8dba2d0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccf3da740 10 client.0 _ll_register_callbacks cb 0x55e9f8ccd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42002/0 (socket says 172.21.15.116:42002) 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2417429351 learned_addr learned my addr 172.21.15.116:0/2417429351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccca58f640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fcccad90640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2417429351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 msgr2=0x55e9f8dc07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 msgr2=0x55e9f8dc1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2417429351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f8a313f0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cc9d0 tx=0x7fccb4010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa3d68c7c264c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.495+0000 7fccadffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb401c3f0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fccb4007400 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccc002c7e0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fccb40ba0d0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 10 client.17076 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e9f8afb7d0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccc002c7e0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fccb40cf190 con 0x55e9f8dbabf0 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 10 client.17076 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 10 client.17076 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 20 client.17076 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8dd3a90 0x55e9f8dd5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 4 client.17076 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f8d7b3e0 con 0x55e9f8dd3a90 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.496+0000 7fccca58f640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8dd3a90 0x55e9f8dd5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.497+0000 7fccadffb640 10 client.17076 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.497+0000 7fccca58f640 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8dd3a90 0x55e9f8dd5e80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=71b1efbd6b1f0225 server_cookie=4caca1bf80ad151a in_seq=0 out_seq=0 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fccadffb640 1 -- 172.21.15.116:0/2417429351 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9f8d7b3e0 con 0x55e9f8dd3a90 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fccadffb640 10 client.17076 handle_command_reply: tid=0 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 client.17076 shutdown 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 msgr2=0x55e9f8dc0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cc9d0 tx=0x7fccb4010ec0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 shutdown_connections 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8dd3a90 0x55e9f8dd5e80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8dc1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fccadffb640 0 client.17076 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc0d00 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2417429351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 >> 172.21.15.116:0/2417429351 conn(0x55e9f8db4a00 msgr2=0x55e9f8db6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 shutdown_connections 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 1 -- 172.21.15.116:0/2417429351 wait complete. 2024-03-19T02:32:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.498+0000 7fcccf3da740 20 client.17076 trim_cache size 0 max 16384 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 Processor -- start 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 -- start start 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dcf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41ac880 con 0x55e9f8dc37d0 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41ae3c0 con 0x55e9f8dba2d0 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccf3da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41acf50 con 0x55e9f8dbabf0 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccad90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50542/0 (socket says 172.21.15.116:50542) 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fccc9d8e640 1 -- 172.21.15.116:0/766324857 learned_addr learned my addr 172.21.15.116:0/766324857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fccca58f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dcf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.514+0000 7fcccad90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42008/0 (socket says 172.21.15.116:42008) 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 -- 172.21.15.116:0/766324857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dba2d0 msgr2=0x55e9f8dcf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dcf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 -- 172.21.15.116:0/766324857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 msgr2=0x55e9f8ddb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 -- 172.21.15.116:0/766324857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f8dcf840 con 0x55e9f8dbabf0 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fcccad90640 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fccb40982b0 tx=0x7fccb4009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36ef38d7cb3db216 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccaffff640 1 -- 172.21.15.116:0/766324857 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb409f980 con 0x55e9f8dbabf0 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccaffff640 1 -- 172.21.15.116:0/766324857 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccb40a3fd0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.515+0000 7fccaffff640 1 -- 172.21.15.116:0/766324857 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb40a3da0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- 172.21.15.116:0/766324857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 msgr2=0x55e9f8dc1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fccb40982b0 tx=0x7fccb4009700 comp rx=0 tx=0).stop 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- 172.21.15.116:0/766324857 shutdown_connections 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dc37d0 0x55e9f8ddb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc1060 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 --2- 172.21.15.116:0/766324857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dcf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- 172.21.15.116:0/766324857 >> 172.21.15.116:0/766324857 conn(0x55e9f8db4a00 msgr2=0x55e9f8db3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- 172.21.15.116:0/766324857 shutdown_connections 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- 172.21.15.116:0/766324857 wait complete. 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 -- start start 2024-03-19T02:32:33.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.516+0000 7fcccf3da740 1 Processor -- start 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8e38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41adcf0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41ad620 con 0x55e9f8dc37d0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41aa860 con 0x55e9f8dba2d0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 10 client.0 _ll_register_callbacks cb 0x7fccb409b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42016/0 (socket says 172.21.15.116:42016) 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2508023008 learned_addr learned my addr 172.21.15.116:0/2508023008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccca58f640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccad90640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8e38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2508023008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 msgr2=0x55e9f8dc7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 msgr2=0x55e9f8e38d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8e38d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 -- 172.21.15.116:0/2508023008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f8e392c0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccca58f640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccc9d8e640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fccb409b5e0 tx=0x7fccb4005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6a791186bbf887 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccadffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb40a3d40 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fccb409bb40 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccb40a3fd0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fccb40cf3a0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 10 client.17082 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.517+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e9f8e3a460 con 0x55e9f8dbabf0 2024-03-19T02:32:33.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccb40a3fd0 con 0x55e9f8dbabf0 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fccb409cb60 con 0x55e9f8dbabf0 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 10 client.17082 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 10 client.17082 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 20 client.17082 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8e3a8e0 0x55e9f8e3ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 4 client.17082 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9f8d2a9f0 con 0x55e9f8e3a8e0 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fccca58f640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8e3a8e0 0x55e9f8e3ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fccadffb640 10 client.17082 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.518+0000 7fccca58f640 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8e3a8e0 0x55e9f8e3ccd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d118e1b42cc3a74 server_cookie=a6451eaaeeaf162c in_seq=0 out_seq=0 2024-03-19T02:32:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fccadffb640 1 -- 172.21.15.116:0/2508023008 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9f8d2a9f0 con 0x55e9f8e3a8e0 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fccadffb640 10 client.17082 handle_command_reply: tid=0 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 client.17082 shutdown 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 msgr2=0x55e9f8dc7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fccb409b5e0 tx=0x7fccb4005e30 comp rx=0 tx=0).stop 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 shutdown_connections 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e9f8e3a8e0 0x55e9f8e3ccd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e9f8dc37d0 0x55e9f8e38d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9f8dbabf0 0x55e9f8dc7b70 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 --2- 172.21.15.116:0/2508023008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9f8dba2d0 0x55e9f8dc7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 >> 172.21.15.116:0/2508023008 conn(0x55e9f8db4a00 msgr2=0x55e9f8db5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fccadffb640 0 client.17082 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 shutdown_connections 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 1 -- 172.21.15.116:0/2508023008 wait complete. 2024-03-19T02:32:33.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccf3da740 20 client.17082 trim_cache size 0 max 16384 2024-03-19T02:32:33.528 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:33.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 msgr2=0x7fccb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 0x7fccb00a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0049d0 tx=0x7fccbc0116d0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 msgr2=0x7fccc41984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.519+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 0x7fccc41984b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fccb400e9f0 tx=0x7fccb400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 shutdown_connections 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fccb00a2970 0x7fccb00a4e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccc41066b0 0x7fccc419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccc4101060 0x7fccc41984b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 --2- 172.21.15.116:0/4123948290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc41006a0 0x7fccc4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 >> 172.21.15.116:0/4123948290 conn(0x7fccc40778d0 msgr2=0x7fccc4103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 shutdown_connections 2024-03-19T02:32:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.520+0000 7fcccc7f3640 1 -- 172.21.15.116:0/4123948290 wait complete. 2024-03-19T02:32:33.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:32:33.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 -- 172.21.15.116:0/2516882995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c101790 msgr2=0x7f228c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 --2- 172.21.15.116:0/2516882995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c101790 0x7f228c101c10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f227c009930 tx=0x7f227c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:33.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 -- 172.21.15.116:0/2516882995 shutdown_connections 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 --2- 172.21.15.116:0/2516882995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c102150 0x7f228c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 --2- 172.21.15.116:0/2516882995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c101790 0x7f228c101c10 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 --2- 172.21.15.116:0/2516882995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c1077a0 0x7f228c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.645+0000 7f2293a12640 1 -- 172.21.15.116:0/2516882995 >> 172.21.15.116:0/2516882995 conn(0x7f228c0fd2a0 msgr2=0x7f228c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- 172.21.15.116:0/2516882995 shutdown_connections 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- 172.21.15.116:0/2516882995 wait complete. 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 Processor -- start 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- start start 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c101790 0x7f228c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c1077a0 0x7f228c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f228c111a60 con 0x7f228c102150 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1118e0 con 0x7f228c101790 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.646+0000 7f2293a12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f228c111be0 con 0x7f228c1077a0 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42026/0 (socket says 172.21.15.116:42026) 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 -- 172.21.15.116:0/2308411671 learned_addr learned my addr 172.21.15.116:0/2308411671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f22917ae640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c101790 0x7f228c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2291faf640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c1077a0 0x7f228c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 -- 172.21.15.116:0/2308411671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c1077a0 msgr2=0x7f228c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c1077a0 0x7f228c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 -- 172.21.15.116:0/2308411671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c101790 msgr2=0x7f228c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c101790 0x7f228c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 -- 172.21.15.116:0/2308411671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c19cfb0 con 0x7f228c102150 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2290fad640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f227c009230 tx=0x7f227c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f22827fc640 1 -- 172.21.15.116:0/2308411671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c043020 con 0x7f228c102150 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f22827fc640 1 -- 172.21.15.116:0/2308411671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f227c02ca70 con 0x7f228c102150 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228c19d240 con 0x7f228c102150 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f22827fc640 1 -- 172.21.15.116:0/2308411671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c003a50 con 0x7f228c102150 2024-03-19T02:32:33.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.647+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228c1a4b30 con 0x7f228c102150 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f22827fc640 1 -- 172.21.15.116:0/2308411671 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f227c046020 con 0x7f228c102150 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 Processor -- start 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 -- start start 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e61bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556f71db94b0 con 0x556f71e5f7d0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556f71d61d70 con 0x556f71e56bf0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2296602740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556f71db5550 con 0x556f71e562d0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2290fad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e61bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2291faf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f22917ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2291faf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50570/0 (socket says 172.21.15.116:50570) 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2290fad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e61bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42036/0 (socket says 172.21.15.116:42036) 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f22827fc640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 0x7f22780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.650+0000 7f2291faf640 1 -- 172.21.15.116:0/12693524 learned_addr learned my addr 172.21.15.116:0/12693524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f2291faf640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 0x7f22780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 -- 172.21.15.116:0/12693524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 msgr2=0x556f71e566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 -- 172.21.15.116:0/12693524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 msgr2=0x556f71e61bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e61bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 -- 172.21.15.116:0/12693524 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f71d84210 con 0x556f71e56bf0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22827fc640 1 -- 172.21.15.116:0/2308411671 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f227c0f10c0 con 0x7f228c102150 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f2291faf640 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22917ae640 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5f290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2288009c20 tx=0x7f2288011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e903310a013c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f2291faf640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 0x7f22780a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2284009fd0 tx=0x7f2284004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22777fe640 1 -- 172.21.15.116:0/12693524 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288004050 con 0x556f71e56bf0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22777fe640 1 -- 172.21.15.116:0/12693524 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22880054d0 con 0x556f71e56bf0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.651+0000 7f22777fe640 1 -- 172.21.15.116:0/12693524 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288004c60 con 0x556f71e56bf0 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- 172.21.15.116:0/12693524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 msgr2=0x556f71e5f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5f290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2288009c20 tx=0x7f2288011810 comp rx=0 tx=0).stop 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- 172.21.15.116:0/12693524 shutdown_connections 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e61bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5f290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 --2- 172.21.15.116:0/12693524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- 172.21.15.116:0/12693524 >> 172.21.15.116:0/12693524 conn(0x556f71e50a00 msgr2=0x556f71e50de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- 172.21.15.116:0/12693524 shutdown_connections 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- 172.21.15.116:0/12693524 wait complete. 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 -- start start 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.652+0000 7f2296602740 1 Processor -- start 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e5f7d0 0x556f71e5d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556f71e625c0 con 0x556f71e562d0 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556f71d75310 con 0x556f71e56bf0 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556f71e62740 con 0x556f71e5f7d0 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2296602740 10 client.0 _ll_register_callbacks cb 0x556f71d692d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42042/0 (socket says 172.21.15.116:42042) 2024-03-19T02:32:33.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2290fad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 -- 172.21.15.116:0/1103342259 learned_addr learned my addr 172.21.15.116:0/1103342259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2291faf640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e5f7d0 0x556f71e5d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 -- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e5f7d0 msgr2=0x556f71e5d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e5f7d0 0x556f71e5d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 -- 172.21.15.116:0/1103342259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 msgr2=0x556f71e5cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 -- 172.21.15.116:0/1103342259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f71acd3f0 con 0x556f71e562d0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2290fad640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f22917ae640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f228801b040 tx=0x7f22880038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2587cdad5b2d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.653+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288029070 con 0x556f71e562d0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2288003ca0 con 0x556f71e562d0 2024-03-19T02:32:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22880054d0 con 0x556f71e562d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2288035050 con 0x556f71e562d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 10 client.17094 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f71b977d0 con 0x556f71e562d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22880054d0 con 0x556f71e562d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2288023430 con 0x556f71e562d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 10 client.17094 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 10 client.17094 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 20 client.17094 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71e6f9d0 0x556f71e71dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 4 client.17094 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f71e173e0 con 0x556f71e6f9d0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.654+0000 7f2290fad640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71e6f9d0 0x556f71e71dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.655+0000 7f2274ff9640 10 client.17094 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.655+0000 7f2290fad640 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71e6f9d0 0x556f71e71dc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=214614fc7affebeb server_cookie=acbd5b59547df2c0 in_seq=0 out_seq=0 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.655+0000 7f2274ff9640 1 -- 172.21.15.116:0/1103342259 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f71e173e0 con 0x556f71e6f9d0 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.655+0000 7f2274ff9640 10 client.17094 handle_command_reply: tid=0 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.655+0000 7f2296602740 1 client.17094 shutdown 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 msgr2=0x556f71e5c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f228801b040 tx=0x7f22880038a0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 shutdown_connections 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71e6f9d0 0x556f71e71dc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e5f7d0 0x556f71e5d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 --2- 172.21.15.116:0/1103342259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e562d0 0x556f71e5c760 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2274ff9640 0 client.17094 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:33.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 >> 172.21.15.116:0/1103342259 conn(0x556f71e50a00 msgr2=0x556f71e52750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 shutdown_connections 2024-03-19T02:32:33.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 1 -- 172.21.15.116:0/1103342259 wait complete. 2024-03-19T02:32:33.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.656+0000 7f2296602740 20 client.17094 trim_cache size 0 max 16384 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 Processor -- start 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 -- start start 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e6b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1aeb60 con 0x556f71e5f7d0 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1b06a0 con 0x556f71e56bf0 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2296602740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1af230 con 0x556f71e562d0 2024-03-19T02:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f22917ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e6b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42044/0 (socket says 172.21.15.116:42044) 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2290fad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 -- 172.21.15.116:0/1271052961 learned_addr learned my addr 172.21.15.116:0/1271052961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 -- 172.21.15.116:0/1271052961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 msgr2=0x556f71e6b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e6b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 -- 172.21.15.116:0/1271052961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 msgr2=0x556f71e5d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 -- 172.21.15.116:0/1271052961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f71e723f0 con 0x556f71e5f7d0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.670+0000 7f2291faf640 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2284003fd0 tx=0x7f2284009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f160b2acc165f6bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2276ffd640 1 -- 172.21.15.116:0/1271052961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284013170 con 0x556f71e5f7d0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2276ffd640 1 -- 172.21.15.116:0/1271052961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2284013310 con 0x556f71e5f7d0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2276ffd640 1 -- 172.21.15.116:0/1271052961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284022df0 con 0x556f71e5f7d0 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 -- 172.21.15.116:0/1271052961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 msgr2=0x556f71e77620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2284003fd0 tx=0x7f2284009be0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 -- 172.21.15.116:0/1271052961 shutdown_connections 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e77620 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e5d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 --2- 172.21.15.116:0/1271052961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e6b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 -- 172.21.15.116:0/1271052961 >> 172.21.15.116:0/1271052961 conn(0x556f71e50a00 msgr2=0x556f71e4fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 -- 172.21.15.116:0/1271052961 shutdown_connections 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.671+0000 7f2296602740 1 -- 172.21.15.116:0/1271052961 wait complete. 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 -- start start 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 Processor -- start 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e67a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1affd0 con 0x556f71e5f7d0 2024-03-19T02:32:33.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1af900 con 0x556f71e56bf0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f228c1acb40 con 0x556f71e562d0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2296602740 10 client.0 _ll_register_callbacks cb 0x7f227c0bd3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2291faf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2291faf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42048/0 (socket says 172.21.15.116:42048) 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2290fad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f2291faf640 1 -- 172.21.15.116:0/1791256372 learned_addr learned my addr 172.21.15.116:0/1791256372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.672+0000 7f22917ae640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e67a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 -- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 msgr2=0x556f71e67a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e67a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 -- 172.21.15.116:0/1791256372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 msgr2=0x556f71e62d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e62d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 -- 172.21.15.116:0/1791256372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f71e61bc0 con 0x556f71e5f7d0 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2290fad640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e62d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2291faf640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2284022930 tx=0x7f2284013170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c0b0a2ae17d7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284022650 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f2284022b70 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2284013310 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2284022d70 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2296602740 10 client.17100 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f71e50830 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.673+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2284013310 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f2284029070 con 0x556f71e5f7d0 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 10 client.17100 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 10 client.17100 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 20 client.17100 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71eda100 0x556f71edc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 4 client.17100 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f71dc69f0 con 0x556f71eda100 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f22917ae640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71eda100 0x556f71edc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2274ff9640 10 client.17100 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f22917ae640 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71eda100 0x556f71edc520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=103dce0dd24f0c5c server_cookie=4841a8c1fc575ecb in_seq=0 out_seq=0 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2274ff9640 1 -- 172.21.15.116:0/1791256372 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f71dc69f0 con 0x556f71eda100 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2274ff9640 10 client.17100 handle_command_reply: tid=0 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 1 client.17100 shutdown 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 msgr2=0x556f71e632d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.674+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2284022930 tx=0x7f2284013170 comp rx=0 tx=0).stop 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 shutdown_connections 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556f71eda100 0x556f71edc520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f71e5f7d0 0x556f71e632d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f71e56bf0 0x556f71e62d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 --2- 172.21.15.116:0/1791256372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f71e562d0 0x556f71e67a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 >> 172.21.15.116:0/1791256372 conn(0x556f71e50a00 msgr2=0x556f71e51a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2274ff9640 0 client.17100 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:32:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 shutdown_connections 2024-03-19T02:32:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 1 -- 172.21.15.116:0/1791256372 wait complete. 2024-03-19T02:32:33.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2296602740 20 client.17100 trim_cache size 0 max 16384 2024-03-19T02:32:33.682 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 msgr2=0x7f22780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 0x7f22780a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2284009fd0 tx=0x7f2284004ab0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 msgr2=0x7f228c198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f227c009230 tx=0x7f227c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 shutdown_connections 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22780a2970 0x7f22780a4e30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f228c1077a0 0x7f228c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228c102150 0x7f228c198500 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 --2- 172.21.15.116:0/2308411671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f228c101790 0x7f228c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 >> 172.21.15.116:0/2308411671 conn(0x7f228c0fd2a0 msgr2=0x7f228c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.675+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 shutdown_connections 2024-03-19T02:32:33.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.676+0000 7f2293a12640 1 -- 172.21.15.116:0/2308411671 wait complete. 2024-03-19T02:32:33.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:32:33.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 143597 3.25 MB/sec execute 116 sec latency 18.439 ms 2024-03-19T02:32:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- 172.21.15.116:0/1142829678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 msgr2=0x7f3914107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 --2- 172.21.15.116:0/1142829678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f3914107be0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3908009890 tx=0x7f390802bb70 comp rx=0 tx=0).stop 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- 172.21.15.116:0/1142829678 shutdown_connections 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 --2- 172.21.15.116:0/1142829678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141021b0 0x7f391410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 --2- 172.21.15.116:0/1142829678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141017f0 0x7f3914101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 --2- 172.21.15.116:0/1142829678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f3914107be0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- 172.21.15.116:0/1142829678 >> 172.21.15.116:0/1142829678 conn(0x7f39140fd2c0 msgr2=0x7f39140ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- 172.21.15.116:0/1142829678 shutdown_connections 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- 172.21.15.116:0/1142829678 wait complete. 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 Processor -- start 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.801+0000 7f391b012640 1 -- start start 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141017f0 0x7f3914197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141021b0 0x7f3914198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3914111a80 con 0x7f3914107800 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3914111900 con 0x7f39141017f0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3914111c00 con 0x7f39141021b0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42060/0 (socket says 172.21.15.116:42060) 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 -- 172.21.15.116:0/555743386 learned_addr learned my addr 172.21.15.116:0/555743386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f3918dae640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141017f0 0x7f3914197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f3913fff640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141021b0 0x7f3914198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 -- 172.21.15.116:0/555743386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141021b0 msgr2=0x7f3914198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141021b0 0x7f3914198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 -- 172.21.15.116:0/555743386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141017f0 msgr2=0x7f3914197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141017f0 0x7f3914197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 -- 172.21.15.116:0/555743386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391419cfd0 con 0x7f3914107800 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f39195af640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f390c00eb10 tx=0x7f390c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f3911ffb640 1 -- 172.21.15.116:0/555743386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c004b20 con 0x7f3914107800 2024-03-19T02:32:33.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f3911ffb640 1 -- 172.21.15.116:0/555743386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f390c004cc0 con 0x7f3914107800 2024-03-19T02:32:33.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391419d2c0 con 0x7f3914107800 2024-03-19T02:32:33.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.802+0000 7f3911ffb640 1 -- 172.21.15.116:0/555743386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c010690 con 0x7f3914107800 2024-03-19T02:32:33.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.803+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39141a4b50 con 0x7f3914107800 2024-03-19T02:32:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.804+0000 7f3911ffb640 1 -- 172.21.15.116:0/555743386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f390c005020 con 0x7f3914107800 2024-03-19T02:32:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f3911ffb640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 0x7f39040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 Processor -- start 2024-03-19T02:32:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f3918dae640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 0x7f39040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f3911ffb640 1 -- 172.21.15.116:0/555743386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f390c0cf730 con 0x7f3914107800 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 -- start start 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654eb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556f654454b0 con 0x556f654eb7d0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556f653edd70 con 0x556f654e22d0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.805+0000 7f391dc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556f65441550 con 0x556f654e2bf0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f3918dae640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 0x7f39040a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3908000c00 tx=0x7f3908005890 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f3913fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42068/0 (socket says 172.21.15.116:42068) 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 -- 172.21.15.116:0/2744127899 learned_addr learned my addr 172.21.15.116:0/2744127899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f3918dae640 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 -- 172.21.15.116:0/2744127899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 msgr2=0x556f654eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 -- 172.21.15.116:0/2744127899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 msgr2=0x556f654e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 -- 172.21.15.116:0/2744127899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f65410210 con 0x556f654eb7d0 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f39195af640 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f390c0a3c80 tx=0x7f390c097de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2131b4d408150c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f38feffd640 1 -- 172.21.15.116:0/2744127899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c09ba10 con 0x556f654eb7d0 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f38feffd640 1 -- 172.21.15.116:0/2744127899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f390c09fb60 con 0x556f654eb7d0 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.806+0000 7f38feffd640 1 -- 172.21.15.116:0/2744127899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0cf570 con 0x556f654eb7d0 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 -- 172.21.15.116:0/2744127899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 msgr2=0x556f654edbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f390c0a3c80 tx=0x7f390c097de0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 -- 172.21.15.116:0/2744127899 shutdown_connections 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654edbc0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 --2- 172.21.15.116:0/2744127899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 -- 172.21.15.116:0/2744127899 >> 172.21.15.116:0/2744127899 conn(0x556f654dca00 msgr2=0x556f654dcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 -- 172.21.15.116:0/2744127899 shutdown_connections 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.807+0000 7f391dc00740 1 -- 172.21.15.116:0/2744127899 wait complete. 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 -- start start 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 Processor -- start 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556f654ee5c0 con 0x556f654eb7d0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556f65401310 con 0x556f654e22d0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556f654ee740 con 0x556f654e2bf0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f391dc00740 10 client.0 _ll_register_callbacks cb 0x556f653f52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f3913fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f3918dae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f39195af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f3913fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654e8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50636/0 (socket says 172.21.15.116:50636) 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f3913fff640 1 -- 172.21.15.116:0/2102466690 learned_addr learned my addr 172.21.15.116:0/2102466690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f39195af640 1 -- 172.21.15.116:0/2102466690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 msgr2=0x556f654e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f39195af640 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654e8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f39195af640 1 -- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 msgr2=0x556f654e8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.808+0000 7f39195af640 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f39195af640 1 -- 172.21.15.116:0/2102466690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f651593f0 con 0x556f654eb7d0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f39195af640 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654e92d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x556f654ee190 tx=0x7f390c00efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5546f5bff66fb5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0cf580 con 0x556f654eb7d0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f390c0ba270 con 0x556f654eb7d0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390c09fb60 con 0x556f654eb7d0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f390c082050 con 0x556f654eb7d0 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 10 client.17112 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f652237d0 con 0x556f654eb7d0 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f390c09fb60 con 0x556f654eb7d0 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f390c0cf050 con 0x556f654eb7d0 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 10 client.17112 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.809+0000 7f391dc00740 20 client.17112 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f654fba30 0x556f654fde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f391dc00740 4 client.17112 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f654a33e0 con 0x556f654fba30 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f3918dae640 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f654fba30 0x556f654fde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f38f3fff640 10 client.17112 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.810+0000 7f3918dae640 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f654fba30 0x556f654fde20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5fe757131ab071 server_cookie=e0f8ea15e7c0639d in_seq=0 out_seq=0 2024-03-19T02:32:33.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f38f3fff640 1 -- 172.21.15.116:0/2102466690 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f654a33e0 con 0x556f654fba30 2024-03-19T02:32:33.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f38f3fff640 10 client.17112 handle_command_reply: tid=0 2024-03-19T02:32:33.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 client.17112 shutdown 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 msgr2=0x556f654e92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654e92d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x556f654ee190 tx=0x7f390c00efd0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 shutdown_connections 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f654fba30 0x556f654fde20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654eb7d0 0x556f654e92d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f38f3fff640 0 client.17112 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654e2bf0 0x556f654e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 --2- 172.21.15.116:0/2102466690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654e8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.811+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 >> 172.21.15.116:0/2102466690 conn(0x556f654dca00 msgr2=0x556f654de750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.812+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 shutdown_connections 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.812+0000 7f391dc00740 1 -- 172.21.15.116:0/2102466690 wait complete. 2024-03-19T02:32:33.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.812+0000 7f391dc00740 20 client.17112 trim_cache size 0 max 16384 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 Processor -- start 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 -- start start 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e2bf0 0x556f65500e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654e5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39141aec60 con 0x556f654e22d0 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39141b07a0 con 0x556f654e2bf0 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.827+0000 7f391dc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39141af330 con 0x556f654eb7d0 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42086/0 (socket says 172.21.15.116:42086) 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 -- 172.21.15.116:0/3033401803 learned_addr learned my addr 172.21.15.116:0/3033401803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f39195af640 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654e5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3913fff640 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e2bf0 0x556f65500e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 -- 172.21.15.116:0/3033401803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 msgr2=0x556f654e5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654e5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 -- 172.21.15.116:0/3033401803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e2bf0 msgr2=0x556f65500e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e2bf0 0x556f65500e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 -- 172.21.15.116:0/3033401803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f654ee640 con 0x556f654e22d0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f3918dae640 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f390802c0b0 tx=0x7f390803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863fc82cf1e13942 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f38fe7fc640 1 -- 172.21.15.116:0/3033401803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390805a070 con 0x556f654e22d0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f38fe7fc640 1 -- 172.21.15.116:0/3033401803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3908044040 con 0x556f654e22d0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.828+0000 7f38fe7fc640 1 -- 172.21.15.116:0/3033401803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908039040 con 0x556f654e22d0 2024-03-19T02:32:33.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- 172.21.15.116:0/3033401803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 msgr2=0x556f654de560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f390802c0b0 tx=0x7f390803e000 comp rx=0 tx=0).stop 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- 172.21.15.116:0/3033401803 shutdown_connections 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654e5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e2bf0 0x556f65500e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 --2- 172.21.15.116:0/3033401803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e22d0 0x556f654de560 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- 172.21.15.116:0/3033401803 >> 172.21.15.116:0/3033401803 conn(0x556f654dca00 msgr2=0x556f654dde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- 172.21.15.116:0/3033401803 shutdown_connections 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- 172.21.15.116:0/3033401803 wait complete. 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 -- start start 2024-03-19T02:32:33.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.829+0000 7f391dc00740 1 Processor -- start 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654fea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39141b00d0 con 0x556f654e2bf0 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39141afa00 con 0x556f654e22d0 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39141acc40 con 0x556f654eb7d0 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f391dc00740 10 client.0 _ll_register_callbacks cb 0x7f390c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3918dae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654fea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42096/0 (socket says 172.21.15.116:42096) 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 -- 172.21.15.116:0/2558939516 learned_addr learned my addr 172.21.15.116:0/2558939516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f39195af640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 -- 172.21.15.116:0/2558939516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 msgr2=0x556f654ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654ff510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 -- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 msgr2=0x556f654fea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654fea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 -- 172.21.15.116:0/2558939516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f65560cf0 con 0x556f654e2bf0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3918dae640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654fea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f3913fff640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x556f65441630 tx=0x7f3900012c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf236497dcb7ad30 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.830+0000 7f38f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900022020 con 0x556f654e2bf0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3900004220 con 0x556f654e2bf0 2024-03-19T02:32:33.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3908044040 con 0x556f654e2bf0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f390002a050 con 0x556f654e2bf0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 10 client.17118 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f65561ff0 con 0x556f654e2bf0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3908044040 con 0x556f654e2bf0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f39000038b0 con 0x556f654e2bf0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 10 client.17118 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 20 client.17118 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f655627c0 0x556f65564be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 4 client.17118 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f654529f0 con 0x556f655627c0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.831+0000 7f3918dae640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f655627c0 0x556f65564be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.832+0000 7f38f3fff640 10 client.17118 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.832+0000 7f3918dae640 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f655627c0 0x556f65564be0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285d4aee68506125 server_cookie=d393181c6ddef740 in_seq=0 out_seq=0 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f38f3fff640 1 -- 172.21.15.116:0/2558939516 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f654529f0 con 0x556f655627c0 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f38f3fff640 10 client.17118 handle_command_reply: tid=0 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 client.17118 shutdown 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 msgr2=0x556f654fefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x556f65441630 tx=0x7f3900012c80 comp rx=0 tx=0).stop 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 shutdown_connections 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556f655627c0 0x556f65564be0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556f654eb7d0 0x556f654ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556f654e2bf0 0x556f654fefb0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f38f3fff640 0 client.17118 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 --2- 172.21.15.116:0/2558939516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556f654e22d0 0x556f654fea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 >> 172.21.15.116:0/2558939516 conn(0x556f654dca00 msgr2=0x556f654de6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 shutdown_connections 2024-03-19T02:32:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.833+0000 7f391dc00740 1 -- 172.21.15.116:0/2558939516 wait complete. 2024-03-19T02:32:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391dc00740 20 client.17118 trim_cache size 0 max 16384 2024-03-19T02:32:33.840 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 msgr2=0x7f39040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 0x7f39040a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3908000c00 tx=0x7f3908005890 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 msgr2=0x7f391419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f390c00eb10 tx=0x7f390c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 shutdown_connections 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39040a2970 0x7f39040a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3914107800 0x7f391419c8a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39141021b0 0x7f3914198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 --2- 172.21.15.116:0/555743386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39141017f0 0x7f3914197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 >> 172.21.15.116:0/555743386 conn(0x7f39140fd2c0 msgr2=0x7f39140fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 shutdown_connections 2024-03-19T02:32:33.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:33.834+0000 7f391b012640 1 -- 172.21.15.116:0/555743386 wait complete. 2024-03-19T02:32:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:33 smithi116 ceph-mon[32263]: pgmap v145: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:33 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/854756645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:33 smithi162 ceph-mon[35761]: pgmap v145: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:33 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/854756645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:33 smithi154 ceph-mon[35693]: pgmap v145: 129 pgs: 129 active+clean; 122 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:33 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/854756645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:32:34.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 144265 3.23 MB/sec execute 117 sec latency 23.933 ms 2024-03-19T02:32:34.847 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:32:34.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:32:34.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:32:34.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 -- 172.21.15.116:0/3012852192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c100690 msgr2=0x7fb16c100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 --2- 172.21.15.116:0/3012852192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c100690 0x7fb16c100b10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009930 tx=0x7fb15c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 -- 172.21.15.116:0/3012852192 shutdown_connections 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 --2- 172.21.15.116:0/3012852192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c101050 0x7fb16c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 --2- 172.21.15.116:0/3012852192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c100690 0x7fb16c100b10 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 --2- 172.21.15.116:0/3012852192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c1066a0 0x7fb16c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 -- 172.21.15.116:0/3012852192 >> 172.21.15.116:0/3012852192 conn(0x7fb16c077820 msgr2=0x7fb16c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 -- 172.21.15.116:0/3012852192 shutdown_connections 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.960+0000 7fb173852640 1 -- 172.21.15.116:0/3012852192 wait complete. 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 Processor -- start 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 -- start start 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 0x7fb16c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c1066a0 0x7fb16c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c110960 con 0x7fb16c101050 2024-03-19T02:32:34.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1107e0 con 0x7fb16c100690 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb173852640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c110ae0 con 0x7fb16c1066a0 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb170ded640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 0x7fb16c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb170ded640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42122/0 (socket says 172.21.15.116:42122) 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb170ded640 1 -- 172.21.15.116:0/4284994699 learned_addr learned my addr 172.21.15.116:0/4284994699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.961+0000 7fb171def640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c1066a0 0x7fb16c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 -- 172.21.15.116:0/4284994699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c1066a0 msgr2=0x7fb16c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c1066a0 0x7fb16c19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 -- 172.21.15.116:0/4284994699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 msgr2=0x7fb16c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 0x7fb16c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 -- 172.21.15.116:0/4284994699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c19cfc0 con 0x7fb16c101050 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb1715ee640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 0x7fb16c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb170ded640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009230 tx=0x7fb15c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb1627fc640 1 -- 172.21.15.116:0/4284994699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c043020 con 0x7fb16c101050 2024-03-19T02:32:34.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb1627fc640 1 -- 172.21.15.116:0/4284994699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb15c02ca70 con 0x7fb16c101050 2024-03-19T02:32:34.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c19d250 con 0x7fb16c101050 2024-03-19T02:32:34.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb1627fc640 1 -- 172.21.15.116:0/4284994699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c003a50 con 0x7fb16c101050 2024-03-19T02:32:34.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.962+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c1a4b40 con 0x7fb16c101050 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.964+0000 7fb1627fc640 1 -- 172.21.15.116:0/4284994699 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb15c046020 con 0x7fb16c101050 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 Processor -- start 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 -- start start 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1627fc640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 0x7fb1580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4fca0 0x561b5eb58340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb58880 0x561b5eb5ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b5eab3160 con 0x561b5eb4f380 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b5ea5be10 con 0x561b5eb58880 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb17643f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b5eaaf200 con 0x561b5eb4fca0 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb171def640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 0x7fb1580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42130/0 (socket says 172.21.15.116:42130) 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1627fc640 1 -- 172.21.15.116:0/4284994699 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb15c0f10c0 con 0x7fb16c101050 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 -- 172.21.15.116:0/2210214067 learned_addr learned my addr 172.21.15.116:0/2210214067 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb170ded640 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4fca0 0x561b5eb58340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb171def640 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb58880 0x561b5eb5ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 -- 172.21.15.116:0/2210214067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4fca0 msgr2=0x561b5eb58340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4fca0 0x561b5eb58340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 -- 172.21.15.116:0/2210214067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb58880 msgr2=0x561b5eb5ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb58880 0x561b5eb5ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.965+0000 7fb1715ee640 1 -- 172.21.15.116:0/2210214067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b5e88f7e0 con 0x561b5eb4f380 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb171def640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 0x7fb1580a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb164005fd0 tx=0x7fb164009f10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb1680114f0 tx=0x7fb1680119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fa830deb7d679d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb1577fe640 1 -- 172.21.15.116:0/2210214067 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16801e070 con 0x561b5eb4f380 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb1577fe640 1 -- 172.21.15.116:0/2210214067 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb168011030 con 0x561b5eb4f380 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb1577fe640 1 -- 172.21.15.116:0/2210214067 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168010630 con 0x561b5eb4f380 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb17643f740 1 -- 172.21.15.116:0/2210214067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 msgr2=0x561b5eb4f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.966+0000 7fb17643f740 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb1680114f0 tx=0x7fb1680119b0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- 172.21.15.116:0/2210214067 shutdown_connections 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb58880 0x561b5eb5ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4fca0 0x561b5eb58340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- 172.21.15.116:0/2210214067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4f380 0x561b5eb4f760 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- 172.21.15.116:0/2210214067 >> 172.21.15.116:0/2210214067 conn(0x561b5eb49ab0 msgr2=0x561b5eb49e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- 172.21.15.116:0/2210214067 shutdown_connections 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- 172.21.15.116:0/2210214067 wait complete. 2024-03-19T02:32:34.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- start start 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 Processor -- start 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 0x561b5eb55810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb58880 0x561b5eb56290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b5eb5b670 con 0x561b5eb4fca0 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b5ea6f1a0 con 0x561b5eb4f380 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b5eb5b7f0 con 0x561b5eb58880 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.967+0000 7fb17643f740 10 client.0 _ll_register_callbacks cb 0x561b5e7c6c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42138/0 (socket says 172.21.15.116:42138) 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 -- 172.21.15.116:0/2037347356 learned_addr learned my addr 172.21.15.116:0/2037347356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 0x561b5eb55810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb171def640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb58880 0x561b5eb56290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 -- 172.21.15.116:0/2037347356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb58880 msgr2=0x561b5eb56290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb58880 0x561b5eb56290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 -- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 msgr2=0x561b5eb55810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 0x561b5eb55810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 -- 172.21.15.116:0/2037347356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b5e8eff70 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 0x561b5eb55810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb170ded640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0bd2d0 tx=0x7fb15c0bcf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e107961d6e05f457 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb154ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0c1720 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb15c0a9300 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb168011030 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb15c0c5d70 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.968+0000 7fb17643f740 10 client.17130 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561b5e787170 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb168011030 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb15c0ac1c0 con 0x561b5eb4fca0 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 10 client.17130 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:34.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 20 client.17130 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5eb68b30 0x561b5eb6afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 4 client.17130 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b5eac06a0 con 0x561b5eb68b30 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5eb68b30 0x561b5eb6afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb154ff9640 10 client.17130 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.969+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5eb68b30 0x561b5eb6afa0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69062000fe277743 server_cookie=9bb6627bd720571c in_seq=0 out_seq=0 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb154ff9640 1 -- 172.21.15.116:0/2037347356 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b5eac06a0 con 0x561b5eb68b30 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb154ff9640 10 client.17130 handle_command_reply: tid=0 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 client.17130 shutdown 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 msgr2=0x561b5eb55d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb15c0bd2d0 tx=0x7fb15c0bcf30 comp rx=0 tx=0).stop 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 shutdown_connections 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5eb68b30 0x561b5eb6afa0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb58880 0x561b5eb56290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb154ff9640 0 client.17130 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:34.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb4fca0 0x561b5eb55d50 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 --2- 172.21.15.116:0/2037347356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4f380 0x561b5eb55810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 >> 172.21.15.116:0/2037347356 conn(0x561b5eb49ab0 msgr2=0x561b5eb4b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:34.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 shutdown_connections 2024-03-19T02:32:34.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 1 -- 172.21.15.116:0/2037347356 wait complete. 2024-03-19T02:32:34.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.971+0000 7fb17643f740 20 client.17130 trim_cache size 0 max 16384 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 Processor -- start 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 -- start start 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb64410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb70880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1ac9e0 con 0x561b5eb58880 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1ae520 con 0x561b5eb4fca0 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb17643f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1ad0b0 con 0x561b5eb4f380 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb171def640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb70880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb170ded640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb64410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb171def640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb70880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42144/0 (socket says 172.21.15.116:42144) 2024-03-19T02:32:34.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50694/0 (socket says 172.21.15.116:50694) 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb170ded640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb64410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38898/0 (socket says 172.21.15.116:38898) 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.991+0000 7fb171def640 1 -- 172.21.15.116:0/2285656631 learned_addr learned my addr 172.21.15.116:0/2285656631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 -- 172.21.15.116:0/2285656631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 msgr2=0x561b5eb64410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb64410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 -- 172.21.15.116:0/2285656631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 msgr2=0x561b5eb70880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb70880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 -- 172.21.15.116:0/2285656631 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b5eb6b5f0 con 0x561b5eb4f380 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb1715ee640 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb16800ff40 tx=0x7fb1680096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc88b17db2735cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb156ffd640 1 -- 172.21.15.116:0/2285656631 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16802e070 con 0x561b5eb4f380 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb156ffd640 1 -- 172.21.15.116:0/2285656631 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb168029210 con 0x561b5eb4f380 2024-03-19T02:32:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.992+0000 7fb156ffd640 1 -- 172.21.15.116:0/2285656631 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16801e040 con 0x561b5eb4f380 2024-03-19T02:32:34.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- 172.21.15.116:0/2285656631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 msgr2=0x561b5eb57f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:34.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb16800ff40 tx=0x7fb1680096c0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- 172.21.15.116:0/2285656631 shutdown_connections 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb70880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb64410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 --2- 172.21.15.116:0/2285656631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb57f40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- 172.21.15.116:0/2285656631 >> 172.21.15.116:0/2285656631 conn(0x561b5eb49ab0 msgr2=0x561b5eb56060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- 172.21.15.116:0/2285656631 shutdown_connections 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- 172.21.15.116:0/2285656631 wait complete. 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 -- start start 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.993+0000 7fb17643f740 1 Processor -- start 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb5d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb5d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1ade50 con 0x561b5eb58880 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1ad780 con 0x561b5eb4fca0 2024-03-19T02:32:34.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1aa9c0 con 0x561b5eb4f380 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50702/0 (socket says 172.21.15.116:50702) 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb171def640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb5d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 -- 172.21.15.116:0/3875632283 learned_addr learned my addr 172.21.15.116:0/3875632283 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb17643f740 10 client.0 _ll_register_callbacks cb 0x7fb15c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb170ded640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb5d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 -- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 msgr2=0x561b5eb5d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb5d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 -- 172.21.15.116:0/3875632283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 msgr2=0x561b5eb5d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb5d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 -- 172.21.15.116:0/3875632283 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b5eb5df80 con 0x561b5eb4f380 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb1715ee640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb16801c040 tx=0x7fb168011300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c955eecf2c178f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.994+0000 7fb154ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168039060 con 0x561b5eb4f380 2024-03-19T02:32:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb168029210 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fb1680295d0 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb16802e030 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 10 client.26893 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561b5ebd0000 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb168029210 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb1680035f0 con 0x561b5eb4f380 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 10 client.26893 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 10 client.26893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 20 client.26893 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5ebd0520 0x561b5ebd2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 4 client.26893 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561b5eaba9f0 con 0x561b5ebd0520 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.995+0000 7fb170ded640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5ebd0520 0x561b5ebd2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.996+0000 7fb154ff9640 10 client.26893 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.996+0000 7fb170ded640 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5ebd0520 0x561b5ebd2910 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c28c0e3be7d2eea server_cookie=3693e2f873e5809d in_seq=0 out_seq=0 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb154ff9640 1 -- 172.21.15.116:0/3875632283 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561b5eaba9f0 con 0x561b5ebd0520 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb154ff9640 10 client.26893 handle_command_reply: tid=0 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 client.26893 shutdown 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 msgr2=0x561b5eb60df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb16801c040 tx=0x7fb168011300 comp rx=0 tx=0).stop 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 shutdown_connections 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b5ebd0520 0x561b5ebd2910 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b5eb58880 0x561b5eb5d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b5eb4fca0 0x561b5eb5d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 --2- 172.21.15.116:0/3875632283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b5eb4f380 0x561b5eb60df0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb154ff9640 0 client.26893 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 >> 172.21.15.116:0/3875632283 conn(0x561b5eb49ab0 msgr2=0x561b5eb482a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 shutdown_connections 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 1 -- 172.21.15.116:0/3875632283 wait complete. 2024-03-19T02:32:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.997+0000 7fb17643f740 20 client.26893 trim_cache size 0 max 16384 2024-03-19T02:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "610a777e-d55b-4b84-95c0-904c6f88be30", 2024-03-19T02:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 msgr2=0x7fb1580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 0x7fb1580a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb164005fd0 tx=0x7fb164009f10 comp rx=0 tx=0).stop 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 msgr2=0x7fb16c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009230 tx=0x7fb15c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 shutdown_connections 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb1580a2970 0x7fb1580a4e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb16c1066a0 0x7fb16c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb16c101050 0x7fb16c198510 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 --2- 172.21.15.116:0/4284994699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c100690 0x7fb16c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 >> 172.21.15.116:0/4284994699 conn(0x7fb16c077820 msgr2=0x7fb16c068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 shutdown_connections 2024-03-19T02:32:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:34.998+0000 7fb173852640 1 -- 172.21.15.116:0/4284994699 wait complete. 2024-03-19T02:32:35.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:32:35.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:32:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 -- 172.21.15.116:0/768319534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70241066a0 msgr2=0x7f7024106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 --2- 172.21.15.116:0/768319534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70241066a0 0x7f7024106a80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7014009930 tx=0x7f701402bc30 comp rx=0 tx=0).stop 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 -- 172.21.15.116:0/768319534 shutdown_connections 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 --2- 172.21.15.116:0/768319534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024101050 0x7f702410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 --2- 172.21.15.116:0/768319534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024100690 0x7f7024100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 --2- 172.21.15.116:0/768319534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70241066a0 0x7f7024106a80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 -- 172.21.15.116:0/768319534 >> 172.21.15.116:0/768319534 conn(0x7f7024077820 msgr2=0x7f7024077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 -- 172.21.15.116:0/768319534 shutdown_connections 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.126+0000 7f702929a640 1 -- 172.21.15.116:0/768319534 wait complete. 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.127+0000 7f702929a640 1 Processor -- start 2024-03-19T02:32:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.127+0000 7f702929a640 1 -- start start 2024-03-19T02:32:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.127+0000 7f702929a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.127+0000 7f702929a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 0x7f7024198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f702929a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70241066a0 0x7f702419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f702929a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7024110920 con 0x7f7024100690 2024-03-19T02:32:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f702929a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70241107a0 con 0x7f7024101050 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f702929a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7024110aa0 con 0x7f70241066a0 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42168/0 (socket says 172.21.15.116:42168) 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 -- 172.21.15.116:0/3179896216 learned_addr learned my addr 172.21.15.116:0/3179896216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f70227fc640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 0x7f7024198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f70237fe640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70241066a0 0x7f702419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 -- 172.21.15.116:0/3179896216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70241066a0 msgr2=0x7f702419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70241066a0 0x7f702419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 -- 172.21.15.116:0/3179896216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 msgr2=0x7f7024198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 0x7f7024198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 -- 172.21.15.116:0/3179896216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702419cfe0 con 0x7f7024100690 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f70227fc640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 0x7f7024198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f7022ffd640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7014009230 tx=0x7f701402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f700bfff640 1 -- 172.21.15.116:0/3179896216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014043020 con 0x7f7024100690 2024-03-19T02:32:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f700bfff640 1 -- 172.21.15.116:0/3179896216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f701402caf0 con 0x7f7024100690 2024-03-19T02:32:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.128+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702419d270 con 0x7f7024100690 2024-03-19T02:32:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.129+0000 7f700bfff640 1 -- 172.21.15.116:0/3179896216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014003ae0 con 0x7f7024100690 2024-03-19T02:32:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.129+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241a4b60 con 0x7f7024100690 2024-03-19T02:32:35.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.131+0000 7f700bfff640 1 -- 172.21.15.116:0/3179896216 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7014046020 con 0x7f7024100690 2024-03-19T02:32:35.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.131+0000 7f702bf67740 1 Processor -- start 2024-03-19T02:32:35.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 -- start start 2024-03-19T02:32:35.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f700bfff640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 0x7f700c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b515e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 0x55d98b51e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b51ef30 0x55d98b521320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d98b479500 con 0x55d98b51ef30 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d98b421be0 con 0x55d98b515a30 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f702bf67740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d98b4755a0 con 0x55d98b516350 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b515e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f70227fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 0x55d98b51e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f70227fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 0x55d98b51e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50722/0 (socket says 172.21.15.116:50722) 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f70227fc640 1 -- 172.21.15.116:0/4184713569 learned_addr learned my addr 172.21.15.116:0/4184713569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f700bfff640 1 -- 172.21.15.116:0/3179896216 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f70140f10d0 con 0x7f7024100690 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f70237fe640 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b51ef30 0x55d98b521320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f70237fe640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 0x7f700c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 -- 172.21.15.116:0/4184713569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 msgr2=0x55d98b51e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 0x55d98b51e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 -- 172.21.15.116:0/4184713569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b51ef30 msgr2=0x55d98b521320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b51ef30 0x55d98b521320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.132+0000 7f7022ffd640 1 -- 172.21.15.116:0/4184713569 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d98b443000 con 0x55d98b515a30 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f7022ffd640 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b515e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70140ee5b0 tx=0x7f70140b9820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbae9ee71bd5eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f70237fe640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 0x7f700c0a4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55d98b5112f0 tx=0x7f701c009600 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f70097fa640 1 -- 172.21.15.116:0/4184713569 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140879a0 con 0x55d98b515a30 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f70097fa640 1 -- 172.21.15.116:0/4184713569 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70140bdbc0 con 0x55d98b515a30 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f70097fa640 1 -- 172.21.15.116:0/4184713569 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014087490 con 0x55d98b515a30 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f702bf67740 1 -- 172.21.15.116:0/4184713569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 msgr2=0x55d98b515e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.133+0000 7f702bf67740 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b515e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70140ee5b0 tx=0x7f70140b9820 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- 172.21.15.116:0/4184713569 shutdown_connections 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b51ef30 0x55d98b521320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b516350 0x55d98b51e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- 172.21.15.116:0/4184713569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b515e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- 172.21.15.116:0/4184713569 >> 172.21.15.116:0/4184713569 conn(0x55d98b510160 msgr2=0x55d98b510540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- 172.21.15.116:0/4184713569 shutdown_connections 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- 172.21.15.116:0/4184713569 wait complete. 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- start start 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 Processor -- start 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b51bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b51ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d98b525ef0 con 0x55d98b516350 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d98b435160 con 0x55d98b515a30 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d98b526070 con 0x55d98b51ef30 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f70227fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b51bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f70227fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42172/0 (socket says 172.21.15.116:42172) 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f70227fc640 1 -- 172.21.15.116:0/1091968071 learned_addr learned my addr 172.21.15.116:0/1091968071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f702bf67740 10 client.0 _ll_register_callbacks cb 0x55d98b471a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.134+0000 7f70237fe640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b51ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 -- 172.21.15.116:0/1091968071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 msgr2=0x55d98b51ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b51ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 -- 172.21.15.116:0/1091968071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 msgr2=0x55d98b51c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 -- 172.21.15.116:0/1091968071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d98b2577d0 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f70227fc640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7022ffd640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b51bfb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70140bb770 tx=0x7f70140b9890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4d279d139482be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140bd4c0 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7014003c80 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70140bdbc0 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f70140c1610 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f702bf67740 10 client.26966 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d98b18d3f0 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70140bdbc0 con 0x55d98b515a30 2024-03-19T02:32:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.135+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f70140d6150 con 0x55d98b515a30 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 10 client.26966 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 10 client.26966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 20 client.26966 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b5332b0 0x55d98b5356a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 4 client.26966 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d98b486a40 con 0x55d98b5332b0 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f70227fc640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b5332b0 0x55d98b5356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f70227fc640 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b5332b0 0x55d98b5356a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22de6d25662622c2 server_cookie=5847c3313da221f6 in_seq=0 out_seq=0 2024-03-19T02:32:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.136+0000 7f7006ff5640 10 client.26966 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f7006ff5640 1 -- 172.21.15.116:0/1091968071 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d98b486a40 con 0x55d98b5332b0 2024-03-19T02:32:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f7006ff5640 10 client.26966 handle_command_reply: tid=0 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 client.26966 shutdown 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 msgr2=0x55d98b51bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b51bfb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70140bb770 tx=0x7f70140b9890 comp rx=0 tx=0).stop 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 shutdown_connections 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b5332b0 0x55d98b5356a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b51ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b51c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 --2- 172.21.15.116:0/1091968071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b515a30 0x55d98b51bfb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f7006ff5640 0 client.26966 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 >> 172.21.15.116:0/1091968071 conn(0x55d98b510160 msgr2=0x55d98b511eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 shutdown_connections 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.137+0000 7f702bf67740 1 -- 172.21.15.116:0/1091968071 wait complete. 2024-03-19T02:32:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.138+0000 7f702bf67740 20 client.26966 trim_cache size 0 max 16384 2024-03-19T02:32:35.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 Processor -- start 2024-03-19T02:32:35.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 -- start start 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 0x55d98b51c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b51ef30 0x55d98b53ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70241ac890 con 0x55d98b516350 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70241ae3d0 con 0x55d98b51ef30 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.156+0000 7f702bf67740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70241acf60 con 0x55d98b515a30 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 0x55d98b51c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42180/0 (socket says 172.21.15.116:42180) 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70237fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b51ef30 0x55d98b53ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 -- 172.21.15.116:0/1247713749 learned_addr learned my addr 172.21.15.116:0/1247713749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 -- 172.21.15.116:0/1247713749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 msgr2=0x55d98b51c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 0x55d98b51c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 -- 172.21.15.116:0/1247713749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b51ef30 msgr2=0x55d98b53ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b51ef30 0x55d98b53ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 -- 172.21.15.116:0/1247713749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d98b14d0f0 con 0x55d98b516350 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f7022ffd640 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 0x55d98b51c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f70227fc640 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x55d98b51b810 tx=0x7f7018005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253e279592399b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f7008ff9640 1 -- 172.21.15.116:0/1247713749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701800fc80 con 0x55d98b516350 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f7008ff9640 1 -- 172.21.15.116:0/1247713749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f701800fe20 con 0x55d98b516350 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.157+0000 7f7008ff9640 1 -- 172.21.15.116:0/1247713749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018022810 con 0x55d98b516350 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- 172.21.15.116:0/1247713749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 msgr2=0x55d98b52eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x55d98b51b810 tx=0x7f7018005f90 comp rx=0 tx=0).stop 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- 172.21.15.116:0/1247713749 shutdown_connections 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b51ef30 0x55d98b53ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b516350 0x55d98b52eb60 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 --2- 172.21.15.116:0/1247713749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b515a30 0x55d98b51c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- 172.21.15.116:0/1247713749 >> 172.21.15.116:0/1247713749 conn(0x55d98b510160 msgr2=0x55d98b511460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- 172.21.15.116:0/1247713749 shutdown_connections 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- 172.21.15.116:0/1247713749 wait complete. 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 -- start start 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.158+0000 7f702bf67740 1 Processor -- start 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b516350 0x55d98b525660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b5219e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70241add00 con 0x55d98b515a30 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70241ad630 con 0x55d98b516350 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70241aa870 con 0x55d98b51ef30 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 10 client.0 _ll_register_callbacks cb 0x7f701c009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42184/0 (socket says 172.21.15.116:42184) 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 -- 172.21.15.116:0/18595657 learned_addr learned my addr 172.21.15.116:0/18595657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f70237fe640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b5219e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f70227fc640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b516350 0x55d98b525660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 -- 172.21.15.116:0/18595657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 msgr2=0x55d98b5219e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b5219e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 -- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b516350 msgr2=0x55d98b525660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b516350 0x55d98b525660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 -- 172.21.15.116:0/18595657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d98b522120 con 0x55d98b515a30 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f70237fe640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b5219e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7022ffd640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7014005760 tx=0x7f70140bd350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e760f6f18069392 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7006ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140da060 con 0x55d98b515a30 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f70140b9b30 con 0x55d98b515a30 2024-03-19T02:32:35.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.159+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f701800fe20 con 0x55d98b515a30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f70140bd4c0 con 0x55d98b515a30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 10 client.17154 fetch_fsmap learned FSMap version 30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d98b522120 con 0x55d98b515a30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f701800fe20 con 0x55d98b515a30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f70140c1120 con 0x55d98b515a30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 10 client.17154 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 20 client.17154 populate_metadata read hostname 'smithi116' 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b59e960 0x55d98b5a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 4 client.17154 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d98b480d90 con 0x55d98b59e960 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f70227fc640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b59e960 0x55d98b5a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f7006ff5640 10 client.17154 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.160+0000 7f70227fc640 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b59e960 0x55d98b5a0d50 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a109c3f08c196765 server_cookie=688010dabda66218 in_seq=0 out_seq=0 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f7006ff5640 1 -- 172.21.15.116:0/18595657 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d98b480d90 con 0x55d98b59e960 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f7006ff5640 10 client.17154 handle_command_reply: tid=0 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 client.17154 shutdown 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 msgr2=0x55d98b52b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7014005760 tx=0x7f70140bd350 comp rx=0 tx=0).stop 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 shutdown_connections 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d98b59e960 0x55d98b5a0d50 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d98b51ef30 0x55d98b5219e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d98b516350 0x55d98b525660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 --2- 172.21.15.116:0/18595657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d98b515a30 0x55d98b52b540 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 >> 172.21.15.116:0/18595657 conn(0x55d98b510160 msgr2=0x55d98b51c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f7006ff5640 0 client.17154 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 shutdown_connections 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 1 -- 172.21.15.116:0/18595657 wait complete. 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.162+0000 7f702bf67740 20 client.17154 trim_cache size 0 max 16384 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "610a777e-d55b-4b84-95c0-904c6f88be30": { 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "610a777e-d55b-4b84-95c0-904c6f88be30", 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:32:35.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 msgr2=0x7f700c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 0x7f700c0a4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55d98b5112f0 tx=0x7f701c009600 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 msgr2=0x7f7024197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7014009230 tx=0x7f701402c710 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 shutdown_connections 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f700c0a29f0 0x7f700c0a4eb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70241066a0 0x7f702419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7024101050 0x7f7024198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 --2- 172.21.15.116:0/3179896216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024100690 0x7f7024197ff0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 >> 172.21.15.116:0/3179896216 conn(0x7f7024077820 msgr2=0x7f7024068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 shutdown_connections 2024-03-19T02:32:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:32:35.163+0000 7f702929a640 1 -- 172.21.15.116:0/3179896216 wait complete. 2024-03-19T02:32:35.174 INFO:tasks.cephfs.filesystem:scrub status for tag:610a777e-d55b-4b84-95c0-904c6f88be30 - {'path': '/', 'tag': '610a777e-d55b-4b84-95c0-904c6f88be30', 'options': 'recursive,force'} 2024-03-19T02:32:35.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 145074 3.26 MB/sec execute 118 sec latency 24.473 ms 2024-03-19T02:32:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: pgmap v146: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:32:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:32:36.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:32:36.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:32:36.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: pgmap v146: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: pgmap v146: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T02:32:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:32:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:32:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:32:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:32:36.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 145461 3.27 MB/sec execute 119 sec latency 19.329 ms 2024-03-19T02:32:37.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 145957 3.25 MB/sec execute 120 sec latency 15.177 ms 2024-03-19T02:32:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:37 smithi162 ceph-mon[35761]: pgmap v147: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:37 smithi154 ceph-mon[35693]: pgmap v147: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:37 smithi116 ceph-mon[32263]: pgmap v147: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:38.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 146502 3.24 MB/sec execute 121 sec latency 46.303 ms 2024-03-19T02:32:39.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 147163 3.24 MB/sec execute 122 sec latency 15.348 ms 2024-03-19T02:32:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:39 smithi162 ceph-mon[35761]: pgmap v148: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:39 smithi154 ceph-mon[35693]: pgmap v148: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:39 smithi116 ceph-mon[32263]: pgmap v148: 129 pgs: 129 active+clean; 123 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 84 op/s 2024-03-19T02:32:40.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 147853 3.23 MB/sec execute 123 sec latency 16.467 ms 2024-03-19T02:32:41.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 148739 3.27 MB/sec execute 124 sec latency 16.557 ms 2024-03-19T02:32:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:41 smithi162 ceph-mon[35761]: pgmap v149: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:41 smithi154 ceph-mon[35693]: pgmap v149: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:41 smithi116 ceph-mon[32263]: pgmap v149: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:32:42.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 149092 3.27 MB/sec execute 125 sec latency 62.694 ms 2024-03-19T02:32:43.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 149527 3.25 MB/sec execute 126 sec latency 16.480 ms 2024-03-19T02:32:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:43 smithi162 ceph-mon[35761]: pgmap v150: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-03-19T02:32:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:43 smithi154 ceph-mon[35693]: pgmap v150: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-03-19T02:32:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:43 smithi116 ceph-mon[32263]: pgmap v150: 129 pgs: 129 active+clean; 128 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-03-19T02:32:44.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 150109 3.24 MB/sec execute 127 sec latency 32.772 ms 2024-03-19T02:32:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:44 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:32:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:44 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:32:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:44 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:32:45.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 150793 3.24 MB/sec execute 128 sec latency 19.140 ms 2024-03-19T02:32:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:32:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:45 smithi162 ceph-mon[35761]: pgmap v151: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:32:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:32:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:45 smithi154 ceph-mon[35693]: pgmap v151: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:32:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:32:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:45 smithi116 ceph-mon[32263]: pgmap v151: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:32:46.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 151430 3.23 MB/sec execute 129 sec latency 26.263 ms 2024-03-19T02:32:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: 1 152283 3.26 MB/sec execute 130 sec latency 17.347 ms 2024-03-19T02:32:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:47 smithi162 ceph-mon[35761]: pgmap v152: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:47 smithi154 ceph-mon[35693]: pgmap v152: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:47 smithi116 ceph-mon[32263]: pgmap v152: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:48.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 152738 3.27 MB/sec execute 131 sec latency 18.663 ms 2024-03-19T02:32:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 153138 3.25 MB/sec execute 132 sec latency 16.891 ms 2024-03-19T02:32:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:49 smithi162 ceph-mon[35761]: pgmap v153: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:49 smithi154 ceph-mon[35693]: pgmap v153: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:49 smithi116 ceph-mon[32263]: pgmap v153: 129 pgs: 129 active+clean; 131 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:32:50.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 153727 3.24 MB/sec execute 133 sec latency 31.715 ms 2024-03-19T02:32:51.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 154425 3.24 MB/sec execute 134 sec latency 21.550 ms 2024-03-19T02:32:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:51 smithi162 ceph-mon[35761]: pgmap v154: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:32:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:51 smithi154 ceph-mon[35693]: pgmap v154: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:32:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:51 smithi116 ceph-mon[32263]: pgmap v154: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:32:52.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 155045 3.23 MB/sec execute 135 sec latency 20.535 ms 2024-03-19T02:32:53.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 155852 3.26 MB/sec execute 136 sec latency 16.201 ms 2024-03-19T02:32:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:53 smithi162 ceph-mon[35761]: pgmap v155: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:53 smithi154 ceph-mon[35693]: pgmap v155: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:53 smithi116 ceph-mon[32263]: pgmap v155: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:32:54.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 156335 3.27 MB/sec execute 137 sec latency 23.543 ms 2024-03-19T02:32:55.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 156709 3.25 MB/sec execute 138 sec latency 39.697 ms 2024-03-19T02:32:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:55 smithi162 ceph-mon[35761]: pgmap v156: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.3 MiB/s wr, 126 op/s 2024-03-19T02:32:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:55 smithi154 ceph-mon[35693]: pgmap v156: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.3 MiB/s wr, 126 op/s 2024-03-19T02:32:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:55 smithi116 ceph-mon[32263]: pgmap v156: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 2.3 MiB/s wr, 126 op/s 2024-03-19T02:32:56.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 157344 3.24 MB/sec execute 139 sec latency 16.367 ms 2024-03-19T02:32:57.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 158060 3.24 MB/sec execute 140 sec latency 19.535 ms 2024-03-19T02:32:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:57 smithi162 ceph-mon[35761]: pgmap v157: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:57 smithi154 ceph-mon[35693]: pgmap v157: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:57 smithi116 ceph-mon[32263]: pgmap v157: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:32:58.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 158647 3.23 MB/sec execute 141 sec latency 17.952 ms 2024-03-19T02:32:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:58 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:32:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:58 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:32:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:58 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:32:59.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 159402 3.26 MB/sec execute 142 sec latency 15.899 ms 2024-03-19T02:33:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:59 smithi162 ceph-mon[35761]: pgmap v158: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:33:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:59 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:33:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:59 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:33:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:32:59 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:33:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:59 smithi154 ceph-mon[35693]: pgmap v158: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:33:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:59 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:33:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:59 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:33:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:32:59 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:33:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:59 smithi116 ceph-mon[32263]: pgmap v158: 129 pgs: 129 active+clean; 135 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T02:33:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:59 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:33:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:59 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:33:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:32:59 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:33:00.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 159897 3.26 MB/sec execute 143 sec latency 38.661 ms 2024-03-19T02:33:01.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 160332 3.24 MB/sec execute 144 sec latency 47.690 ms 2024-03-19T02:33:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:01 smithi162 ceph-mon[35761]: pgmap v159: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:33:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:01 smithi154 ceph-mon[35693]: pgmap v159: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:33:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:01 smithi116 ceph-mon[32263]: pgmap v159: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-03-19T02:33:02.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 160930 3.24 MB/sec execute 145 sec latency 17.925 ms 2024-03-19T02:33:03.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 161677 3.24 MB/sec execute 146 sec latency 24.510 ms 2024-03-19T02:33:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:03 smithi162 ceph-mon[35761]: pgmap v160: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-03-19T02:33:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:03 smithi154 ceph-mon[35693]: pgmap v160: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-03-19T02:33:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:03 smithi116 ceph-mon[32263]: pgmap v160: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-03-19T02:33:04.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 162257 3.23 MB/sec execute 147 sec latency 18.660 ms 2024-03-19T02:33:05.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:33:05.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:33:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.287+0000 7f9c21d37640 1 -- 172.21.15.116:0/2882193796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c101080 msgr2=0x7f9c1c10d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.287+0000 7f9c21d37640 1 --2- 172.21.15.116:0/2882193796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c101080 0x7f9c1c10d540 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009900 tx=0x7f9c1402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 -- 172.21.15.116:0/2882193796 shutdown_connections 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 --2- 172.21.15.116:0/2882193796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c101080 0x7f9c1c10d540 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 --2- 172.21.15.116:0/2882193796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 --2- 172.21.15.116:0/2882193796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 -- 172.21.15.116:0/2882193796 >> 172.21.15.116:0/2882193796 conn(0x7f9c1c0777b0 msgr2=0x7f9c1c077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 -- 172.21.15.116:0/2882193796 shutdown_connections 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 -- 172.21.15.116:0/2882193796 wait complete. 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 Processor -- start 2024-03-19T02:33:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.288+0000 7f9c21d37640 1 -- start start 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c0fe160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c110960 con 0x7f9c1c1055c0 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1107e0 con 0x7f9c1c101080 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c21d37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c110ae0 con 0x7f9c1c1006e0 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41012/0 (socket says 172.21.15.116:41012) 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53276/0 (socket says 172.21.15.116:53276) 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c0fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 -- 172.21.15.116:0/3935660178 learned_addr learned my addr 172.21.15.116:0/3935660178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 -- 172.21.15.116:0/3935660178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 msgr2=0x7f9c1c0fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c0fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 -- 172.21.15.116:0/3935660178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 msgr2=0x7f9c1c0fe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 -- 172.21.15.116:0/3935660178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1c1a64a0 con 0x7f9c1c1055c0 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c0fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c1bfff640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009230 tx=0x7f9c1402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c18ff9640 1 -- 172.21.15.116:0/3935660178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14043020 con 0x7f9c1c1055c0 2024-03-19T02:33:05.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.289+0000 7f9c18ff9640 1 -- 172.21.15.116:0/3935660178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c14004e00 con 0x7f9c1c1055c0 2024-03-19T02:33:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.290+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1c1a6730 con 0x7f9c1c1055c0 2024-03-19T02:33:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.290+0000 7f9c18ff9640 1 -- 172.21.15.116:0/3935660178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14003830 con 0x7f9c1c1055c0 2024-03-19T02:33:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.290+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c1c1a6c80 con 0x7f9c1c1055c0 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c18ff9640 1 -- 172.21.15.116:0/3935660178 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9c14046020 con 0x7f9c1c1055c0 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 Processor -- start 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 -- start start 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637740e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x56163774c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616376a4500 con 0x561637741350 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56163764cbe0 con 0x561637749f30 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.292+0000 7f9c249fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5616376a05a0 con 0x561637740a30 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c18ff9640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41018/0 (socket says 172.21.15.116:41018) 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637740e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/2474885797 learned_addr learned my addr 172.21.15.116:0/2474885797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1affd640 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x56163774c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1bfff640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c18ff9640 1 -- 172.21.15.116:0/3935660178 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9c140f10c0 con 0x7f9c1c1055c0 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/2474885797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 msgr2=0x561637740e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637740e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/2474885797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 msgr2=0x56163774c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x56163774c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/2474885797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56163766e000 con 0x561637741350 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1affd640 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637740e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c011810 tx=0x7f9c0c011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541ad7198bd596bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c1bfff640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 0x7f9c040a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c14005940 tx=0x7f9c140887e0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c01ffb640 1 -- 172.21.15.116:0/2474885797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c003c30 con 0x561637741350 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c01ffb640 1 -- 172.21.15.116:0/2474885797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c0c00fe50 con 0x561637741350 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.293+0000 7f9c01ffb640 1 -- 172.21.15.116:0/2474885797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c004820 con 0x561637741350 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 -- 172.21.15.116:0/2474885797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 msgr2=0x5616377499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c011810 tx=0x7f9c0c011cd0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 -- 172.21.15.116:0/2474885797 shutdown_connections 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x56163774c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x5616377499f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 --2- 172.21.15.116:0/2474885797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637740e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 -- 172.21.15.116:0/2474885797 >> 172.21.15.116:0/2474885797 conn(0x56163773b160 msgr2=0x56163773b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 -- 172.21.15.116:0/2474885797 shutdown_connections 2024-03-19T02:33:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.294+0000 7f9c249fc740 1 -- 172.21.15.116:0/2474885797 wait complete. 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 -- start start 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 Processor -- start 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637746e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637741350 0x5616377473d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56163774cd20 con 0x561637749f30 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561637660160 con 0x561637741350 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56163774cea0 con 0x561637740a30 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c249fc740 10 client.0 _ll_register_callbacks cb 0x56163769ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41024/0 (socket says 172.21.15.116:41024) 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637741350 0x5616377473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.295+0000 7f9c1affd640 1 -- 172.21.15.116:0/3803796193 learned_addr learned my addr 172.21.15.116:0/3803796193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637746e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 -- 172.21.15.116:0/3803796193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 msgr2=0x561637746e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637746e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 -- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637741350 msgr2=0x5616377473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637741350 0x5616377473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 -- 172.21.15.116:0/3803796193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616374827d0 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9c10011d90 tx=0x7f9c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc91bdbe963c64d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9bfaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10004050 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9c100041f0 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0c00fe50 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9c100043f0 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c249fc740 10 client.17166 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5616373b83f0 con 0x561637749f30 2024-03-19T02:33:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.296+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c0c00fe50 con 0x561637749f30 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9c1001d030 con 0x561637749f30 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 10 client.17166 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 20 client.17166 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56163775a2f0 0x56163775c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 4 client.17166 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616376b1a40 con 0x56163775a2f0 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56163775a2f0 0x56163775c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9bfaffd640 10 client.17166 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.297+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56163775a2f0 0x56163775c700 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f34d88c3016f030 server_cookie=9cd3c82bb4a8cd21 in_seq=0 out_seq=0 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3803796193 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616376b1a40 con 0x56163775a2f0 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9bfaffd640 10 client.17166 handle_command_reply: tid=0 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 client.17166 shutdown 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 msgr2=0x561637747910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9c10011d90 tx=0x7f9c100046c0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 shutdown_connections 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56163775a2f0 0x56163775c700 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x561637747910 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637741350 0x5616377473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3803796193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x561637746e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9bfaffd640 0 client.17166 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 >> 172.21.15.116:0/3803796193 conn(0x56163773b160 msgr2=0x56163773ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 shutdown_connections 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 1 -- 172.21.15.116:0/3803796193 wait complete. 2024-03-19T02:33:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.299+0000 7f9c249fc740 20 client.17166 trim_cache size 0 max 16384 2024-03-19T02:33:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.317+0000 7f9c249fc740 1 Processor -- start 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.317+0000 7f9c249fc740 1 -- start start 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x5616377495c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x5616377476e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1b0c10 con 0x561637741350 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1b2750 con 0x561637749f30 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c249fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1b12e0 con 0x561637740a30 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41040/0 (socket says 172.21.15.116:41040) 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/3670457205 learned_addr learned my addr 172.21.15.116:0/3670457205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x5616377495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x5616377476e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/3670457205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 msgr2=0x5616377495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x5616377495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/3670457205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 msgr2=0x5616377476e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x5616377476e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 -- 172.21.15.116:0/3670457205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56163774c690 con 0x561637741350 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c011810 tx=0x7f9c0c0044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e638ca989c1a57de server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c017fa640 1 -- 172.21.15.116:0/3670457205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c01f040 con 0x561637741350 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c017fa640 1 -- 172.21.15.116:0/3670457205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c0c003a30 con 0x561637741350 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.318+0000 7f9c017fa640 1 -- 172.21.15.116:0/3670457205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c00fb20 con 0x561637741350 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- 172.21.15.116:0/3670457205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 msgr2=0x561637755b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c011810 tx=0x7f9c0c0044a0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- 172.21.15.116:0/3670457205 shutdown_connections 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637749f30 0x5616377476e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637741350 0x561637755b20 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3670457205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637740a30 0x5616377495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- 172.21.15.116:0/3670457205 >> 172.21.15.116:0/3670457205 conn(0x56163773b160 msgr2=0x561637745dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- 172.21.15.116:0/3670457205 shutdown_connections 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- 172.21.15.116:0/3670457205 wait complete. 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 -- start start 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.319+0000 7f9c249fc740 1 Processor -- start 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637740a30 0x56163774e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637741350 0x56163774e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1b2080 con 0x561637749f30 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1b19b0 con 0x561637740a30 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c1aebf0 con 0x561637741350 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c249fc740 10 client.0 _ll_register_callbacks cb 0x7f9c100118b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:05.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41044/0 (socket says 172.21.15.116:41044) 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 -- 172.21.15.116:0/3694294996 learned_addr learned my addr 172.21.15.116:0/3694294996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637741350 0x56163774e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637740a30 0x56163774e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 -- 172.21.15.116:0/3694294996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637741350 msgr2=0x56163774e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637741350 0x56163774e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 -- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637740a30 msgr2=0x56163774e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637740a30 0x56163774e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 -- 172.21.15.116:0/3694294996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616373780f0 con 0x561637749f30 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9bfaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9c1affd640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009b70 tx=0x7f9c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9443bbb704a54b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.320+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000e920 con 0x561637749f30 2024-03-19T02:33:05.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0c003a30 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f9c1000eac0 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9c10044050 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 10 client.17172 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5616377c0be0 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c0c003a30 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9c1001d070 con 0x561637749f30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 10 client.17172 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 20 client.17172 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5616377c11e0 0x5616377c3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 4 client.17172 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:33:05.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616376abd90 con 0x5616377c11e0 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.321+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5616377c11e0 0x5616377c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.322+0000 7f9c1b7fe640 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5616377c11e0 0x5616377c3600 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a7365ef12c25a7 server_cookie=1a78faf60f64c9ea in_seq=0 out_seq=0 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.322+0000 7f9bfaffd640 10 client.17172 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9bfaffd640 1 -- 172.21.15.116:0/3694294996 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5616376abd90 con 0x5616377c11e0 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9bfaffd640 10 client.17172 handle_command_reply: tid=0 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 client.17172 shutdown 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 msgr2=0x5616377bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009b70 tx=0x7f9c100046c0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 shutdown_connections 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5616377c11e0 0x5616377c3600 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561637749f30 0x5616377bf600 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561637741350 0x56163774e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 --2- 172.21.15.116:0/3694294996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561637740a30 0x56163774e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9bfaffd640 0 client.17172 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 >> 172.21.15.116:0/3694294996 conn(0x56163773b160 msgr2=0x5616377476e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 shutdown_connections 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 1 -- 172.21.15.116:0/3694294996 wait complete. 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c249fc740 20 client.17172 trim_cache size 0 max 16384 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:33:05.331 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 msgr2=0x7f9c040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 0x7f9c040a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c14005940 tx=0x7f9c140887e0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 msgr2=0x7f9c1c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.323+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009230 tx=0x7f9c1402c670 comp rx=0 tx=0).stop 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 shutdown_connections 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c040a2970 0x7f9c040a4e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c1c1055c0 0x7f9c1c0febe0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c1c101080 0x7f9c1c0fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 --2- 172.21.15.116:0/3935660178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c1c1006e0 0x7f9c1c0fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 >> 172.21.15.116:0/3935660178 conn(0x7f9c1c0777b0 msgr2=0x7f9c1c069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 shutdown_connections 2024-03-19T02:33:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.324+0000 7f9c21d37640 1 -- 172.21.15.116:0/3935660178 wait complete. 2024-03-19T02:33:05.336 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:33:05.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:33:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.442+0000 7fefae827640 1 -- 172.21.15.116:0/207285435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa81077c0 msgr2=0x7fefa8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.442+0000 7fefae827640 1 --2- 172.21.15.116:0/207285435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa81077c0 0x7fefa8107ba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fef98009930 tx=0x7fef9802bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- 172.21.15.116:0/207285435 shutdown_connections 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- 172.21.15.116:0/207285435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8102170 0x7fefa810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- 172.21.15.116:0/207285435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81017b0 0x7fefa8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- 172.21.15.116:0/207285435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa81077c0 0x7fefa8107ba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- 172.21.15.116:0/207285435 >> 172.21.15.116:0/207285435 conn(0x7fefa8078c20 msgr2=0x7fefa8079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- 172.21.15.116:0/207285435 shutdown_connections 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- 172.21.15.116:0/207285435 wait complete. 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 Processor -- start 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- start start 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa8102170 0x7fefa8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 0x7fefa819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8111a80 con 0x7fefa81017b0 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8111900 con 0x7fefa81077c0 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.443+0000 7fefae827640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8111c00 con 0x7fefa8102170 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41056/0 (socket says 172.21.15.116:41056) 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 -- 172.21.15.116:0/164590516 learned_addr learned my addr 172.21.15.116:0/164590516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefacdc4640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 0x7fefa819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa77fe640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa8102170 0x7fefa8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 -- 172.21.15.116:0/164590516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa8102170 msgr2=0x7fefa8198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa8102170 0x7fefa8198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 -- 172.21.15.116:0/164590516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 msgr2=0x7fefa819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 0x7fefa819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 -- 172.21.15.116:0/164590516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa819cff0 con 0x7fefa81017b0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefacdc4640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 0x7fefa819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa7fff640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fef98009230 tx=0x7fef9802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98043020 con 0x7fefa81017b0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa819d280 con 0x7fefa81017b0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9802ca70 con 0x7fefa81017b0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98003a50 con 0x7fefa81017b0 2024-03-19T02:33:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.444+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa81a4b70 con 0x7fefa81017b0 2024-03-19T02:33:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.445+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa8068a60 con 0x7fefa81017b0 2024-03-19T02:33:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.447+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fef98046020 con 0x7fefa81017b0 2024-03-19T02:33:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.447+0000 7fefa57fa640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 0x7fef900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.447+0000 7fefa77fe640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 0x7fef900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.447+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fef980f1450 con 0x7fefa81017b0 2024-03-19T02:33:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.448+0000 7fefa77fe640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 0x7fef900a4e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefa81033b0 tx=0x7fef9c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.451+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef9804d020 con 0x7fefa81017b0 2024-03-19T02:33:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.718+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefa8199350 con 0x7fefa81017b0 2024-03-19T02:33:05.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.719+0000 7fefa57fa640 1 -- 172.21.15.116:0/164590516 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fefa8199350 con 0x7fefa81017b0 2024-03-19T02:33:05.719 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:33:05.720 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:33:05.720 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 msgr2=0x7fef900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 0x7fef900a4e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefa81033b0 tx=0x7fef9c005f90 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 msgr2=0x7fefa8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fef98009230 tx=0x7fef9802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 shutdown_connections 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef900a2970 0x7fef900a4e30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa81077c0 0x7fefa819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefa8102170 0x7fefa8198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 --2- 172.21.15.116:0/164590516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa81017b0 0x7fefa8198000 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 >> 172.21.15.116:0/164590516 conn(0x7fefa8078c20 msgr2=0x7fefa8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 shutdown_connections 2024-03-19T02:33:05.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.721+0000 7fefae827640 1 -- 172.21.15.116:0/164590516 wait complete. 2024-03-19T02:33:05.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:33:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 162963 3.25 MB/sec execute 148 sec latency 16.406 ms 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 -- 172.21.15.116:0/652865115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d41006a0 msgr2=0x7ff0d4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 --2- 172.21.15.116:0/652865115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d41006a0 0x7ff0d4100b20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009930 tx=0x7ff0bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 -- 172.21.15.116:0/652865115 shutdown_connections 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 --2- 172.21.15.116:0/652865115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d4101060 0x7ff0d410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 --2- 172.21.15.116:0/652865115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d41006a0 0x7ff0d4100b20 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 --2- 172.21.15.116:0/652865115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 0x7ff0d4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 -- 172.21.15.116:0/652865115 >> 172.21.15.116:0/652865115 conn(0x7ff0d40778d0 msgr2=0x7ff0d4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 -- 172.21.15.116:0/652865115 shutdown_connections 2024-03-19T02:33:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 -- 172.21.15.116:0/652865115 wait complete. 2024-03-19T02:33:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.842+0000 7ff0d883d640 1 Processor -- start 2024-03-19T02:33:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 -- start start 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d41006a0 0x7ff0d4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 0x7ff0d419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d4110970 con 0x7ff0d4101060 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41107f0 con 0x7ff0d41066b0 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d883d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d4110af0 con 0x7ff0d41006a0 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41080/0 (socket says 172.21.15.116:41080) 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/2272599860 learned_addr learned my addr 172.21.15.116:0/2272599860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d259c640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d41006a0 0x7ff0d4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 0x7ff0d419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/2272599860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d41006a0 msgr2=0x7ff0d4197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d41006a0 0x7ff0d4197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/2272599860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 msgr2=0x7ff0d419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 0x7ff0d419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.843+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/2272599860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d419cf60 con 0x7ff0d4101060 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009230 tx=0x7ff0bc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0cbfff640 1 -- 172.21.15.116:0/2272599860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc043020 con 0x7ff0d4101060 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0cbfff640 1 -- 172.21.15.116:0/2272599860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0bc02ca70 con 0x7ff0d4101060 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d419d1f0 con 0x7ff0d4101060 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0cbfff640 1 -- 172.21.15.116:0/2272599860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc003a50 con 0x7ff0d4101060 2024-03-19T02:33:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.844+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d41a4ae0 con 0x7ff0d4101060 2024-03-19T02:33:05.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.846+0000 7ff0db421740 1 Processor -- start 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.846+0000 7ff0cbfff640 1 -- 172.21.15.116:0/2272599860 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff0bc046020 con 0x7ff0d4101060 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.846+0000 7ff0db421740 1 -- start start 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df755bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f4df6ad4b0 con 0x55f4df74abf0 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f4df655d70 con 0x55f4df74a2d0 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0db421740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f4df6a9550 con 0x55f4df7537d0 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41088/0 (socket says 172.21.15.116:41088) 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53344/0 (socket says 172.21.15.116:53344) 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d259c640 1 -- 172.21.15.116:0/2557807425 learned_addr learned my addr 172.21.15.116:0/2557807425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df755bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0cbfff640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 0x7ff0c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 -- 172.21.15.116:0/2557807425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 msgr2=0x55f4df755bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df755bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 -- 172.21.15.116:0/2557807425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 msgr2=0x55f4df753290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 -- 172.21.15.116:0/2557807425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4df678210 con 0x55f4df74a2d0 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d259c640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df755bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0cbfff640 1 -- 172.21.15.116:0/2272599860 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff0bc0f10a0 con 0x7ff0d4101060 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.847+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 0x7ff0c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009fd0 tx=0x7ff0cc011920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e701e0483c536be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0c8d99640 1 -- 172.21.15.116:0/2557807425 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc017070 con 0x55f4df74a2d0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 0x7ff0c00a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc022000 tx=0x7ff0cc023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0c8d99640 1 -- 172.21.15.116:0/2557807425 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0cc00fcb0 con 0x55f4df74a2d0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0c8d99640 1 -- 172.21.15.116:0/2557807425 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00ea40 con 0x55f4df74a2d0 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0db421740 1 -- 172.21.15.116:0/2557807425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 msgr2=0x55f4df74a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.848+0000 7ff0db421740 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009fd0 tx=0x7ff0cc011920 comp rx=0 tx=0).stop 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- 172.21.15.116:0/2557807425 shutdown_connections 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df755bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df753290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- 172.21.15.116:0/2557807425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74a2d0 0x55f4df74a6b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- 172.21.15.116:0/2557807425 >> 172.21.15.116:0/2557807425 conn(0x55f4df744a00 msgr2=0x55f4df744de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- 172.21.15.116:0/2557807425 shutdown_connections 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- 172.21.15.116:0/2557807425 wait complete. 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- start start 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 Processor -- start 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 0x55f4df750d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df7512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f4df7565c0 con 0x55f4df74a2d0 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f4df669310 con 0x55f4df74abf0 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f4df756740 con 0x55f4df7537d0 2024-03-19T02:33:05.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0db421740 10 client.0 _ll_register_callbacks cb 0x55f4df65d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.849+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41096/0 (socket says 172.21.15.116:41096) 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 -- 172.21.15.116:0/457751864 learned_addr learned my addr 172.21.15.116:0/457751864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 0x55f4df750d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df7512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 -- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 msgr2=0x55f4df7512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df7512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 -- 172.21.15.116:0/457751864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 msgr2=0x55f4df750d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 0x55f4df750d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 -- 172.21.15.116:0/457751864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4df3c13f0 con 0x55f4df74a2d0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 0x55f4df750d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0d259c640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40126e0 tx=0x7ff0c4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1063fadd66462efe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40121d0 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff0c4004050 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0cc00fcb0 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff0c4004250 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0db421740 10 client.17190 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.850+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f4df48b7d0 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0cc00fcb0 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff0c4003970 con 0x55f4df74a2d0 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 10 client.17190 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 10 client.17190 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 20 client.17190 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df763c00 0x55f4df766010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 4 client.17190 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4df70b3e0 con 0x55f4df763c00 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.851+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df763c00 0x55f4df766010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df763c00 0x55f4df766010 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=35ebd06141ccdb4b server_cookie=cfd1a010d7845d93 in_seq=0 out_seq=0 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0b9ffb640 10 client.17190 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/457751864 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4df70b3e0 con 0x55f4df763c00 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0b9ffb640 10 client.17190 handle_command_reply: tid=0 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 client.17190 shutdown 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 msgr2=0x55f4df750840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40126e0 tx=0x7ff0c4012ba0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 shutdown_connections 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df763c00 0x55f4df766010 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df7537d0 0x55f4df7512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df74abf0 0x55f4df750d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 --2- 172.21.15.116:0/457751864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df750840 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0b9ffb640 0 client.17190 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 >> 172.21.15.116:0/457751864 conn(0x55f4df744a00 msgr2=0x55f4df746750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 shutdown_connections 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.852+0000 7ff0db421740 1 -- 172.21.15.116:0/457751864 wait complete. 2024-03-19T02:33:05.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.853+0000 7ff0db421740 20 client.17190 trim_cache size 0 max 16384 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 Processor -- start 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 -- start start 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74a2d0 0x55f4df752f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df7510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41aead0 con 0x55f4df74abf0 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41b0610 con 0x55f4df7537d0 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0db421740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41af1a0 con 0x55f4df74a2d0 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41106/0 (socket says 172.21.15.116:41106) 2024-03-19T02:33:05.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/1943623929 learned_addr learned my addr 172.21.15.116:0/1943623929 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0d259c640 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74a2d0 0x55f4df752f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df7510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/1943623929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74a2d0 msgr2=0x55f4df752f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74a2d0 0x55f4df752f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/1943623929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 msgr2=0x55f4df7510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df7510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.868+0000 7ff0c9d9b640 1 -- 172.21.15.116:0/1943623929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4df749a70 con 0x55f4df74abf0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0ee5e0 tx=0x7ff0bc0f3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cc26bc50d81b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0bbfff640 1 -- 172.21.15.116:0/1943623929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0bd0f0 con 0x55f4df74abf0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0bbfff640 1 -- 172.21.15.116:0/1943623929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0bc0bdcb0 con 0x55f4df74abf0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0bbfff640 1 -- 172.21.15.116:0/1943623929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0bcf10 con 0x55f4df74abf0 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 -- 172.21.15.116:0/1943623929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 msgr2=0x55f4df7466c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0ee5e0 tx=0x7ff0bc0f3040 comp rx=0 tx=0).stop 2024-03-19T02:33:05.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 -- 172.21.15.116:0/1943623929 shutdown_connections 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df7510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74abf0 0x55f4df7466c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 --2- 172.21.15.116:0/1943623929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74a2d0 0x55f4df752f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.869+0000 7ff0db421740 1 -- 172.21.15.116:0/1943623929 >> 172.21.15.116:0/1943623929 conn(0x55f4df744a00 msgr2=0x55f4df743310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- 172.21.15.116:0/1943623929 shutdown_connections 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- 172.21.15.116:0/1943623929 wait complete. 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- start start 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 Processor -- start 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74abf0 0x55f4df757780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df757cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41aff40 con 0x55f4df74a2d0 2024-03-19T02:33:05.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41af870 con 0x55f4df7537d0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d41acab0 con 0x55f4df74abf0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0db421740 10 client.0 _ll_register_callbacks cb 0x7ff0c4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41112/0 (socket says 172.21.15.116:41112) 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 -- 172.21.15.116:0/3407650127 learned_addr learned my addr 172.21.15.116:0/3407650127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74abf0 0x55f4df757780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df757cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 -- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74abf0 msgr2=0x55f4df757780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74abf0 0x55f4df757780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 -- 172.21.15.116:0/3407650127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 msgr2=0x55f4df757cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df757cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d259c640 1 -- 172.21.15.116:0/3407650127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4df7c8f10 con 0x55f4df74a2d0 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.870+0000 7ff0d2d9d640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df757cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:33:05.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0d259c640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4003f00 tx=0x7ff0c4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6515ba7fd6b8d749 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4003f80 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7ff0c400f920 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0bc0bdcb0 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff0c400fb20 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 10 client.17196 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f4df7ca210 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0bc0bdcb0 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff0c4010430 con 0x55f4df74a2d0 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 10 client.17196 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 10 client.17196 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 20 client.17196 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:05.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df7ca990 0x55f4df7ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 4 client.17196 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.871+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4df6ba9f0 con 0x55f4df7ca990 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df7ca990 0x55f4df7ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0b9ffb640 10 client.17196 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0c9d9b640 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df7ca990 0x55f4df7ccdb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9cdb33ff02faf126 server_cookie=cd00e9063984d081 in_seq=0 out_seq=0 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0b9ffb640 1 -- 172.21.15.116:0/3407650127 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4df6ba9f0 con 0x55f4df7ca990 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0b9ffb640 10 client.17196 handle_command_reply: tid=0 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 client.17196 shutdown 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 msgr2=0x55f4df757240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4003f00 tx=0x7ff0c4003c20 comp rx=0 tx=0).stop 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 shutdown_connections 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f4df7ca990 0x55f4df7ccdb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4df7537d0 0x55f4df757cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f4df74abf0 0x55f4df757780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 --2- 172.21.15.116:0/3407650127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4df74a2d0 0x55f4df757240 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 >> 172.21.15.116:0/3407650127 conn(0x55f4df744a00 msgr2=0x55f4df743ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0b9ffb640 0 client.17196 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 shutdown_connections 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 1 -- 172.21.15.116:0/3407650127 wait complete. 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.872+0000 7ff0db421740 20 client.17196 trim_cache size 0 max 16384 2024-03-19T02:33:05.882 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 msgr2=0x7ff0c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 0x7ff0c00a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc022000 tx=0x7ff0cc023000 comp rx=0 tx=0).stop 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 msgr2=0x7ff0d41984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009230 tx=0x7ff0bc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 shutdown_connections 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff0c00a2970 0x7ff0c00a4e30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0d41066b0 0x7ff0d419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d4101060 0x7ff0d41984b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 --2- 172.21.15.116:0/2272599860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff0d41006a0 0x7ff0d4197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 >> 172.21.15.116:0/2272599860 conn(0x7ff0d40778d0 msgr2=0x7ff0d4068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 shutdown_connections 2024-03-19T02:33:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.873+0000 7ff0d883d640 1 -- 172.21.15.116:0/2272599860 wait complete. 2024-03-19T02:33:05.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:33:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 -- 172.21.15.116:0/364913108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc101060 msgr2=0x7f22fc10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 --2- 172.21.15.116:0/364913108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc101060 0x7f22fc10d570 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009cd0 tx=0x7f22f402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 -- 172.21.15.116:0/364913108 shutdown_connections 2024-03-19T02:33:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 --2- 172.21.15.116:0/364913108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc101060 0x7f22fc10d570 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 --2- 172.21.15.116:0/364913108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc1006a0 0x7f22fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 --2- 172.21.15.116:0/364913108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 -- 172.21.15.116:0/364913108 >> 172.21.15.116:0/364913108 conn(0x7f22fc0778d0 msgr2=0x7f22fc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 -- 172.21.15.116:0/364913108 shutdown_connections 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.996+0000 7f2303509640 1 -- 172.21.15.116:0/364913108 wait complete. 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 Processor -- start 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 -- start start 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 0x7f22fc199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc1a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc110970 con 0x7f22fc1006a0 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1107f0 con 0x7f22fc101060 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2303509640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc110af0 con 0x7f22fc1066b0 2024-03-19T02:33:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f23012a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f23012a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41128/0 (socket says 172.21.15.116:41128) 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f23012a5640 1 -- 172.21.15.116:0/3743242368 learned_addr learned my addr 172.21.15.116:0/3743242368 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2301aa6640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc1a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.997+0000 7f2300aa4640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 0x7f22fc199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 -- 172.21.15.116:0/3743242368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 msgr2=0x7f22fc1a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc1a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 -- 172.21.15.116:0/3743242368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 msgr2=0x7f22fc199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 0x7f22fc199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 -- 172.21.15.116:0/3743242368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc1a43d0 con 0x7f22fc1006a0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f2300aa4640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 0x7f22fc199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f2301aa6640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f23012a5640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9f0 tx=0x7f22f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f22f27fc640 1 -- 172.21.15.116:0/3743242368 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f800cd80 con 0x7f22fc1006a0 2024-03-19T02:33:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc1a45a0 con 0x7f22fc1006a0 2024-03-19T02:33:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f22f27fc640 1 -- 172.21.15.116:0/3743242368 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f80054d0 con 0x7f22fc1006a0 2024-03-19T02:33:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f22f27fc640 1 -- 172.21.15.116:0/3743242368 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8010490 con 0x7f22fc1006a0 2024-03-19T02:33:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:05.998+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc1a4a60 con 0x7f22fc1006a0 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.000+0000 7f22f27fc640 1 -- 172.21.15.116:0/3743242368 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f22f8005020 con 0x7f22fc1006a0 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 Processor -- start 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 -- start start 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x5648006763e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x5648006836a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 0x564800685fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5648005d94b0 con 0x564800676000 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564800581d70 con 0x564800683be0 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23061d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5648005d5550 con 0x564800676920 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f22f27fc640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 0x7f22e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2301aa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x5648006763e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x5648006836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2301aa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x5648006763e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41142/0 (socket says 172.21.15.116:41142) 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2301aa6640 1 -- 172.21.15.116:0/923120730 learned_addr learned my addr 172.21.15.116:0/923120730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2300aa4640 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 0x564800685fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2301aa6640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 0x7f22e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f22f27fc640 1 -- 172.21.15.116:0/3743242368 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22f80cf4d0 con 0x7f22fc1006a0 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 -- 172.21.15.116:0/923120730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 msgr2=0x564800685fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 0x564800685fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 -- 172.21.15.116:0/923120730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 msgr2=0x5648006763e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x5648006763e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f23012a5640 1 -- 172.21.15.116:0/923120730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648005a4210 con 0x564800676920 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.001+0000 7f2300aa4640 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 0x564800685fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.002+0000 7f23012a5640 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x5648006836a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22f80cc8a0 tx=0x7f22f806e550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c6d15a6099d9963 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.002+0000 7f2301aa6640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 0x7f22e80a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f22f4033000 tx=0x7f22f4005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.002+0000 7f22e77fe640 1 -- 172.21.15.116:0/923120730 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8005310 con 0x564800676920 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.002+0000 7f22e77fe640 1 -- 172.21.15.116:0/923120730 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f8005680 con 0x564800676920 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.002+0000 7f22e77fe640 1 -- 172.21.15.116:0/923120730 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80046c0 con 0x564800676920 2024-03-19T02:33:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- 172.21.15.116:0/923120730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 msgr2=0x5648006836a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x5648006836a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22f80cc8a0 tx=0x7f22f806e550 comp rx=0 tx=0).stop 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- 172.21.15.116:0/923120730 shutdown_connections 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800683be0 0x564800685fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x5648006836a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- 172.21.15.116:0/923120730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x5648006763e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- 172.21.15.116:0/923120730 >> 172.21.15.116:0/923120730 conn(0x564800670a00 msgr2=0x564800670de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- 172.21.15.116:0/923120730 shutdown_connections 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- 172.21.15.116:0/923120730 wait complete. 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 -- start start 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 Processor -- start 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x564800681270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.003+0000 7f23061d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 0x5648006817b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23061d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5648006869a0 con 0x564800683be0 2024-03-19T02:33:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23061d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564800595310 con 0x564800676000 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23061d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564800686b20 con 0x564800676920 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23061d1740 10 client.0 _ll_register_callbacks cb 0x5648005892d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f2300aa4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x564800681270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f2301aa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 0x5648006817b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53420/0 (socket says 172.21.15.116:53420) 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 -- 172.21.15.116:0/3948229584 learned_addr learned my addr 172.21.15.116:0/3948229584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f2301aa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 0x5648006817b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f2300aa4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x564800681270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45838/0 (socket says 172.21.15.116:45838) 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 -- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 msgr2=0x564800681270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x564800681270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 -- 172.21.15.116:0/3948229584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 msgr2=0x5648006817b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 0x5648006817b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 -- 172.21.15.116:0/3948229584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648002ed3f0 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23012a5640 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f22f809b4b0 tx=0x7f22f8086110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd6cb0f7e020011 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f22e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b4060 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f8005680 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.004+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22f80ccb20 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f22f8005310 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 10 client.27026 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5648003b77d0 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f8005680 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f22f80c8070 con 0x564800676000 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 10 client.27026 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 10 client.27026 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:33:06.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 20 client.27026 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564800693f30 0x564800696320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 4 client.27026 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648006373e0 con 0x564800693f30 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.005+0000 7f2300aa4640 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564800693f30 0x564800696320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f22e4ff9640 10 client.27026 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f2300aa4640 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564800693f30 0x564800696320 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7b35e80eb35687f7 server_cookie=4effa977a9a07aac in_seq=0 out_seq=0 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f22e4ff9640 1 -- 172.21.15.116:0/3948229584 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648006373e0 con 0x564800693f30 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f22e4ff9640 10 client.27026 handle_command_reply: tid=0 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 client.27026 shutdown 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 msgr2=0x564800680d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f22f809b4b0 tx=0x7f22f8086110 comp rx=0 tx=0).stop 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 shutdown_connections 2024-03-19T02:33:06.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564800693f30 0x564800696320 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800683be0 0x5648006817b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f22e4ff9640 0 client.27026 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800676920 0x564800681270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 --2- 172.21.15.116:0/3948229584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x564800680d30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.006+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 >> 172.21.15.116:0/3948229584 conn(0x564800670a00 msgr2=0x564800672750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.007+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 shutdown_connections 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.007+0000 7f23061d1740 1 -- 172.21.15.116:0/3948229584 wait complete. 2024-03-19T02:33:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.007+0000 7f23061d1740 20 client.27026 trim_cache size 0 max 16384 2024-03-19T02:33:06.015 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:05 smithi116 ceph-mon[32263]: pgmap v161: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.2 MiB/s wr, 143 op/s 2024-03-19T02:33:06.015 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:05 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/164590516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 Processor -- start 2024-03-19T02:33:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 -- start start 2024-03-19T02:33:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676920 0x564800680610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800699340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1ac990 con 0x564800676000 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1ae4d0 con 0x564800676920 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23061d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1ad060 con 0x564800683be0 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23012a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23012a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41154/0 (socket says 172.21.15.116:41154) 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f23012a5640 1 -- 172.21.15.116:0/1779398201 learned_addr learned my addr 172.21.15.116:0/1779398201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f2301aa6640 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800699340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.021+0000 7f2300aa4640 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676920 0x564800680610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 -- 172.21.15.116:0/1779398201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 msgr2=0x564800699340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800699340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 -- 172.21.15.116:0/1779398201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676920 msgr2=0x564800680610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676920 0x564800680610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 -- 172.21.15.116:0/1779398201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564800676740 con 0x564800676000 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f23012a5640 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22f8099a60 tx=0x7f22f80ba170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ead852ce9505031 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f22e6ffd640 1 -- 172.21.15.116:0/1779398201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b4050 con 0x564800676000 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f22e6ffd640 1 -- 172.21.15.116:0/1779398201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f8007e50 con 0x564800676000 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.022+0000 7f22e6ffd640 1 -- 172.21.15.116:0/1779398201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f809beb0 con 0x564800676000 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- 172.21.15.116:0/1779398201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 msgr2=0x56480066fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22f8099a60 tx=0x7f22f80ba170 comp rx=0 tx=0).stop 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- 172.21.15.116:0/1779398201 shutdown_connections 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800699340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676920 0x564800680610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 --2- 172.21.15.116:0/1779398201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676000 0x56480066fde0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- 172.21.15.116:0/1779398201 >> 172.21.15.116:0/1779398201 conn(0x564800670a00 msgr2=0x56480068f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- 172.21.15.116:0/1779398201 shutdown_connections 2024-03-19T02:33:06.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- 172.21.15.116:0/1779398201 wait complete. 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 -- start start 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 Processor -- start 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.023+0000 7f23061d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800696eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1ade00 con 0x564800676920 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1ad730 con 0x564800676000 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1aa970 con 0x564800683be0 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 10 client.0 _ll_register_callbacks cb 0x7f22ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f2300aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53444/0 (socket says 172.21.15.116:53444) 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f2300aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41166/0 (socket says 172.21.15.116:41166) 2024-03-19T02:33:06.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 -- 172.21.15.116:0/1386818455 learned_addr learned my addr 172.21.15.116:0/1386818455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f2301aa6640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800696eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 -- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 msgr2=0x564800696eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800696eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 -- 172.21.15.116:0/1386818455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 msgr2=0x564800696970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 -- 172.21.15.116:0/1386818455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648006976e0 con 0x564800676000 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f2300aa4640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23012a5640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22f809b740 tx=0x7f22f80cc950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b855926c12ac617d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f22e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b4040 con 0x564800676000 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f8007e50 con 0x564800676000 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f22f809ba10 con 0x564800676000 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f22f80cf310 con 0x564800676000 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.024+0000 7f23061d1740 10 client.27032 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5648006f9ec0 con 0x564800676000 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f8007e50 con 0x564800676000 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f22f80c8070 con 0x564800676000 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 10 client.27032 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 10 client.27032 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 20 client.27032 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5648006fa2f0 0x5648006fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 4 client.27032 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648005e69f0 con 0x5648006fa2f0 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f2300aa4640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5648006fa2f0 0x5648006fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f22e4ff9640 10 client.27032 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.025+0000 7f2300aa4640 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5648006fa2f0 0x5648006fc6e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e205c0cec757212b server_cookie=6e1fe547a78d965f in_seq=0 out_seq=0 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f22e4ff9640 1 -- 172.21.15.116:0/1386818455 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648005e69f0 con 0x5648006fa2f0 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f22e4ff9640 10 client.27032 handle_command_reply: tid=0 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 client.27032 shutdown 2024-03-19T02:33:06.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 msgr2=0x5648006995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22f809b740 tx=0x7f22f80cc950 comp rx=0 tx=0).stop 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 shutdown_connections 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564800683be0 0x564800696eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5648006fa2f0 0x5648006fc6e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564800676920 0x564800696970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 --2- 172.21.15.116:0/1386818455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564800676000 0x5648006995e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f22e4ff9640 0 client.27032 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 >> 172.21.15.116:0/1386818455 conn(0x564800670a00 msgr2=0x56480068edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 shutdown_connections 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 1 -- 172.21.15.116:0/1386818455 wait complete. 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f23061d1740 20 client.27032 trim_cache size 0 max 16384 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:06.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 msgr2=0x7f22e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 0x7f22e80a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f22f4033000 tx=0x7f22f4005c50 comp rx=0 tx=0).stop 2024-03-19T02:33:06.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 msgr2=0x7f22fc198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9f0 tx=0x7f22f800edf0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 shutdown_connections 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22e80a2970 0x7f22e80a4e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22fc1066b0 0x7f22fc1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22fc101060 0x7f22fc199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.026+0000 7f2303509640 1 --2- 172.21.15.116:0/3743242368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc1006a0 0x7f22fc198e10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.027+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 >> 172.21.15.116:0/3743242368 conn(0x7f22fc0778d0 msgr2=0x7f22fc102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.027+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 shutdown_connections 2024-03-19T02:33:06.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.027+0000 7f2303509640 1 -- 172.21.15.116:0/3743242368 wait complete. 2024-03-19T02:33:06.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:33:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 -- 172.21.15.116:0/3073976760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84102150 msgr2=0x7f7c8410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 --2- 172.21.15.116:0/3073976760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84102150 0x7f7c8410e660 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009cd0 tx=0x7f7c7c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 -- 172.21.15.116:0/3073976760 shutdown_connections 2024-03-19T02:33:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 --2- 172.21.15.116:0/3073976760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84102150 0x7f7c8410e660 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 --2- 172.21.15.116:0/3073976760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84101790 0x7f7c84101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 --2- 172.21.15.116:0/3073976760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c841077a0 0x7f7c84107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 -- 172.21.15.116:0/3073976760 >> 172.21.15.116:0/3073976760 conn(0x7f7c840fd2c0 msgr2=0x7f7c840ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 -- 172.21.15.116:0/3073976760 shutdown_connections 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.156+0000 7f7c896ff640 1 -- 172.21.15.116:0/3073976760 wait complete. 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 Processor -- start 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 -- start start 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84102150 0x7f7c84074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c841077a0 0x7f7c840745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84111a20 con 0x7f7c841077a0 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841118a0 con 0x7f7c84101790 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c896ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c84111ba0 con 0x7f7c84102150 2024-03-19T02:33:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c841077a0 0x7f7c840745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c7bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84102150 0x7f7c84074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53466/0 (socket says 172.21.15.116:53466) 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.157+0000 7f7c82ffd640 1 -- 172.21.15.116:0/1344780891 learned_addr learned my addr 172.21.15.116:0/1344780891 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 -- 172.21.15.116:0/1344780891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84102150 msgr2=0x7f7c84074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84102150 0x7f7c84074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 -- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c841077a0 msgr2=0x7f7c840745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c841077a0 0x7f7c840745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 -- 172.21.15.116:0/1344780891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c841a6480 con 0x7f7c84101790 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e9f0 tx=0x7f7c7400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c80ff9640 1 -- 172.21.15.116:0/1344780891 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400cd80 con 0x7f7c84101790 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c841a6770 con 0x7f7c84101790 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c80ff9640 1 -- 172.21.15.116:0/1344780891 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c740108e0 con 0x7f7c84101790 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c80ff9640 1 -- 172.21.15.116:0/1344780891 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74018920 con 0x7f7c84101790 2024-03-19T02:33:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.158+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c841a6cc0 con 0x7f7c84101790 2024-03-19T02:33:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:05 smithi162 ceph-mon[35761]: pgmap v161: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.2 MiB/s wr, 143 op/s 2024-03-19T02:33:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:05 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/164590516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c80ff9640 1 -- 172.21.15.116:0/1344780891 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7c74010a90 con 0x7f7c84101790 2024-03-19T02:33:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 Processor -- start 2024-03-19T02:33:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 -- start start 2024-03-19T02:33:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e66556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 0x5615e665e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 0x5615e6660bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5615e65b84b0 con 0x5615e6655bf0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615e6560d70 con 0x5615e665e7d0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615e65b4550 con 0x5615e66552d0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c80ff9640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 0x7f7c700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 0x5615e665e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e66556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.161+0000 7f7c7bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 0x5615e6660bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e66556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45888/0 (socket says 172.21.15.116:45888) 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c837fe640 1 -- 172.21.15.116:0/2144223284 learned_addr learned my addr 172.21.15.116:0/2144223284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c837fe640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 0x7f7c700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c80ff9640 1 -- 172.21.15.116:0/1344780891 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7c740cf470 con 0x7f7c84101790 2024-03-19T02:33:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 -- 172.21.15.116:0/2144223284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 msgr2=0x5615e66556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e66556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 -- 172.21.15.116:0/2144223284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 msgr2=0x5615e665e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 0x5615e665e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 -- 172.21.15.116:0/2144223284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e6583210 con 0x5615e665e7d0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 0x5615e665e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c837fe640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 0x7f7c700a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009a60 tx=0x7f7c7c005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 0x5615e6660bc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009bf0 tx=0x7f7c6c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c07de5ea14d44a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c797fa640 1 -- 172.21.15.116:0/2144223284 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c004050 con 0x5615e665e7d0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c797fa640 1 -- 172.21.15.116:0/2144223284 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c6c0054d0 con 0x5615e665e7d0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.162+0000 7f7c797fa640 1 -- 172.21.15.116:0/2144223284 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c004c60 con 0x5615e665e7d0 2024-03-19T02:33:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2144223284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 msgr2=0x5615e6660bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 0x5615e6660bc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009bf0 tx=0x7f7c6c011810 comp rx=0 tx=0).stop 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2144223284 shutdown_connections 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e665e7d0 0x5615e6660bc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e6655bf0 0x5615e665e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2144223284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e66556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2144223284 >> 172.21.15.116:0/2144223284 conn(0x5615e664fa00 msgr2=0x5615e664fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2144223284 shutdown_connections 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2144223284 wait complete. 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.163+0000 7f7c8c3c6740 1 -- start start 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 Processor -- start 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e665b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e6655bf0 0x5615e665bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5615e66615c0 con 0x5615e665e7d0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615e6574310 con 0x5615e6655bf0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615e6661740 con 0x5615e66552d0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c8c3c6740 10 client.0 _ll_register_callbacks cb 0x5615e65682d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41202/0 (socket says 172.21.15.116:41202) 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 -- 172.21.15.116:0/2309994443 learned_addr learned my addr 172.21.15.116:0/2309994443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e665b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e6655bf0 0x5615e665bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 -- 172.21.15.116:0/2309994443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 msgr2=0x5615e665b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e665b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 -- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e6655bf0 msgr2=0x5615e665bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e6655bf0 0x5615e665bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 -- 172.21.15.116:0/2309994443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e62cc3f0 con 0x5615e665e7d0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c837fe640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c037040 tx=0x7f7c7c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdd13d9571278e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.164+0000 7f7c6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c054060 con 0x5615e665e7d0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7c7c004250 con 0x5615e665e7d0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c6c0054d0 con 0x5615e665e7d0 2024-03-19T02:33:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7c7c039040 con 0x5615e665e7d0 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 10 client.17226 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5615e63967d0 con 0x5615e665e7d0 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c6c0054d0 con 0x5615e665e7d0 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7c7c04f970 con 0x5615e665e7d0 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 10 client.17226 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 10 client.17226 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 20 client.17226 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e666eb00 0x5615e6670f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 4 client.17226 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.165+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615e66163e0 con 0x5615e666eb00 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.166+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e666eb00 0x5615e6670f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.166+0000 7f7c6a7fc640 10 client.17226 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.166+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e666eb00 0x5615e6670f30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d4ecc7f4b6dc1aa server_cookie=adf004ae99c9658c in_seq=0 out_seq=0 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/2309994443 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615e66163e0 con 0x5615e666eb00 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c6a7fc640 10 client.17226 handle_command_reply: tid=0 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 client.17226 shutdown 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 msgr2=0x5615e665c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c037040 tx=0x7f7c7c0095c0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 shutdown_connections 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e666eb00 0x5615e6670f30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665c240 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e6655bf0 0x5615e665bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2309994443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e66552d0 0x5615e665b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c6a7fc640 0 client.17226 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 >> 172.21.15.116:0/2309994443 conn(0x5615e664fa00 msgr2=0x5615e6651750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 shutdown_connections 2024-03-19T02:33:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2309994443 wait complete. 2024-03-19T02:33:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.167+0000 7f7c8c3c6740 20 client.17226 trim_cache size 0 max 16384 2024-03-19T02:33:06.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 Processor -- start 2024-03-19T02:33:06.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 -- start start 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e666a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b0e30 con 0x5615e665e7d0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b2970 con 0x5615e66552d0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b1500 con 0x5615e6655bf0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c7bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e666a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53496/0 (socket says 172.21.15.116:53496) 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 -- 172.21.15.116:0/2228382549 learned_addr learned my addr 172.21.15.116:0/2228382549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 -- 172.21.15.116:0/2228382549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 msgr2=0x5615e666a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e666a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 -- 172.21.15.116:0/2228382549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 msgr2=0x5615e665d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c82ffd640 1 -- 172.21.15.116:0/2228382549 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e665a790 con 0x5615e66552d0 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.183+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e666a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.184+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c740cc8a0 tx=0x7f7c740ccf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92d480fd2fa21d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.184+0000 7f7c78ff9640 1 -- 172.21.15.116:0/2228382549 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c740a3ef0 con 0x5615e66552d0 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.184+0000 7f7c78ff9640 1 -- 172.21.15.116:0/2228382549 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c74099b00 con 0x5615e66552d0 2024-03-19T02:33:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.184+0000 7f7c78ff9640 1 -- 172.21.15.116:0/2228382549 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c740ccac0 con 0x5615e66552d0 2024-03-19T02:33:06.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2228382549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 msgr2=0x5615e664aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c740cc8a0 tx=0x7f7c740ccf90 comp rx=0 tx=0).stop 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2228382549 shutdown_connections 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e665d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e666a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/2228382549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e664aa70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2228382549 >> 172.21.15.116:0/2228382549 conn(0x5615e664fa00 msgr2=0x5615e665cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2228382549 shutdown_connections 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/2228382549 wait complete. 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- start start 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 Processor -- start 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e6666b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e6658070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b22a0 con 0x5615e665e7d0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841b1bd0 con 0x5615e66552d0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c841aee10 con 0x5615e6655bf0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c8c3c6740 10 client.0 _ll_register_callbacks cb 0x7f7c6c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41218/0 (socket says 172.21.15.116:41218) 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e6666b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c837fe640 1 -- 172.21.15.116:0/810250919 learned_addr learned my addr 172.21.15.116:0/810250919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.185+0000 7f7c7bfff640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e6658070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 -- 172.21.15.116:0/810250919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 msgr2=0x5615e6658070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e6658070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 -- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 msgr2=0x5615e6666b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e6666b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 -- 172.21.15.116:0/810250919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e6658de0 con 0x5615e665e7d0 2024-03-19T02:33:06.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e6666b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c837fe640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c04fd30 tx=0x7f7c7c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e107e75c197d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c054040 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f7c7c04f470 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c74099b00 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7c7c004b40 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c8c3c6740 10 client.17232 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5615e66d4d10 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.186+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c74099b00 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7c7c005020 con 0x5615e665e7d0 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 10 client.17232 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 10 client.17232 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 20 client.17232 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e66d5450 0x5615e66d7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 4 client.17232 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615e65c59f0 con 0x5615e66d5450 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e66d5450 0x5615e66d7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c82ffd640 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e66d5450 0x5615e66d7870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d54972bcbcf54ce7 server_cookie=2a304dd2e9cc33d in_seq=0 out_seq=0 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c6a7fc640 10 client.17232 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c6a7fc640 1 -- 172.21.15.116:0/810250919 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615e65c59f0 con 0x5615e66d5450 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c6a7fc640 10 client.17232 handle_command_reply: tid=0 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 1 client.17232 shutdown 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 msgr2=0x5615e66585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.187+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c04fd30 tx=0x7f7c7c0048f0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 shutdown_connections 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615e66d5450 0x5615e66d7870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615e665e7d0 0x5615e66585b0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c6a7fc640 0 client.17232 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:06.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615e6655bf0 0x5615e6658070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 --2- 172.21.15.116:0/810250919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615e66552d0 0x5615e6666b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 >> 172.21.15.116:0/810250919 conn(0x5615e664fa00 msgr2=0x5615e664eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 shutdown_connections 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 1 -- 172.21.15.116:0/810250919 wait complete. 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c8c3c6740 20 client.17232 trim_cache size 0 max 16384 2024-03-19T02:33:06.195 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 msgr2=0x7f7c700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 0x7f7c700a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009a60 tx=0x7f7c7c005c50 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 msgr2=0x7f7c84073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e9f0 tx=0x7f7c7400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 shutdown_connections 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7c700a2970 0x7f7c700a4e30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c841077a0 0x7f7c840745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c84102150 0x7f7c84074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 --2- 172.21.15.116:0/1344780891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c84101790 0x7f7c84073b20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 >> 172.21.15.116:0/1344780891 conn(0x7f7c840fd2c0 msgr2=0x7f7c840fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 shutdown_connections 2024-03-19T02:33:06.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.188+0000 7f7c896ff640 1 -- 172.21.15.116:0/1344780891 wait complete. 2024-03-19T02:33:06.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:33:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:05 smithi154 ceph-mon[35693]: pgmap v161: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.2 MiB/s wr, 143 op/s 2024-03-19T02:33:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:05 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/164590516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 -- 172.21.15.116:0/3264485092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 msgr2=0x7fe120105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 --2- 172.21.15.116:0/3264485092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe120105a60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009900 tx=0x7fe11c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 -- 172.21.15.116:0/3264485092 shutdown_connections 2024-03-19T02:33:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 --2- 172.21.15.116:0/3264485092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120101330 0x7fe12010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 --2- 172.21.15.116:0/3264485092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120100970 0x7fe120100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 --2- 172.21.15.116:0/3264485092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe120105a60 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 -- 172.21.15.116:0/3264485092 >> 172.21.15.116:0/3264485092 conn(0x7fe1200774b0 msgr2=0x7fe1200778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 -- 172.21.15.116:0/3264485092 shutdown_connections 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.315+0000 7fe127027640 1 -- 172.21.15.116:0/3264485092 wait complete. 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 Processor -- start 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 -- start start 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120101330 0x7fe1201983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe12019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe120110920 con 0x7fe120105680 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201107a0 con 0x7fe120101330 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe127027640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe120110aa0 con 0x7fe120100970 2024-03-19T02:33:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120101330 0x7fe1201983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45936/0 (socket says 172.21.15.116:45936) 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe1255c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe12019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.316+0000 7fe124dc3640 1 -- 172.21.15.116:0/1496152098 learned_addr learned my addr 172.21.15.116:0/1496152098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 -- 172.21.15.116:0/1496152098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120101330 msgr2=0x7fe1201983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120101330 0x7fe1201983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 -- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 msgr2=0x7fe12019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe12019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 -- 172.21.15.116:0/1496152098 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12019ce90 con 0x7fe120100970 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe1255c4640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe12019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe124dc3640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe11c005940 tx=0x7fe11c02c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe115ffb640 1 -- 172.21.15.116:0/1496152098 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c043020 con 0x7fe120100970 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12019d120 con 0x7fe120100970 2024-03-19T02:33:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe115ffb640 1 -- 172.21.15.116:0/1496152098 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe11c02ca70 con 0x7fe120100970 2024-03-19T02:33:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe115ffb640 1 -- 172.21.15.116:0/1496152098 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c003a80 con 0x7fe120100970 2024-03-19T02:33:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.317+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1201a4a10 con 0x7fe120100970 2024-03-19T02:33:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 Processor -- start 2024-03-19T02:33:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 -- start start 2024-03-19T02:33:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe115ffb640 1 -- 172.21.15.116:0/1496152098 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe11c046020 con 0x7fe120100970 2024-03-19T02:33:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde103e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556dbdd734b0 con 0x556dbde10000 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556dbdd1bd70 con 0x556dbde1dbe0 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe129c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556dbdd6f550 con 0x556dbde10920 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:53532/0 (socket says 172.21.15.116:53532) 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe1255c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.320+0000 7fe117fff640 1 -- 172.21.15.116:0/357692506 learned_addr learned my addr 172.21.15.116:0/357692506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 -- 172.21.15.116:0/357692506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 msgr2=0x556dbde1d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 -- 172.21.15.116:0/357692506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 msgr2=0x556dbde103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde103e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 -- 172.21.15.116:0/357692506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbdd3e210 con 0x556dbde1dbe0 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe117fff640 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe110009c90 tx=0x7fe110011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59df48adbcbd6f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe115ffb640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 0x7fe1080a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe106ffd640 1 -- 172.21.15.116:0/357692506 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11001e070 con 0x556dbde1dbe0 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe106ffd640 1 -- 172.21.15.116:0/357692506 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe110019ae0 con 0x556dbde1dbe0 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe106ffd640 1 -- 172.21.15.116:0/357692506 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11000ea40 con 0x556dbde1dbe0 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe115ffb640 1 -- 172.21.15.116:0/1496152098 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fe11c0f1190 con 0x7fe120100970 2024-03-19T02:33:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.321+0000 7fe1255c4640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 0x7fe1080a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 -- 172.21.15.116:0/357692506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 msgr2=0x556dbde1ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe1255c4640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 0x7fe1080a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe118009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe110009c90 tx=0x7fe110011810 comp rx=0 tx=0).stop 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 -- 172.21.15.116:0/357692506 shutdown_connections 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde1ffd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 --2- 172.21.15.116:0/357692506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 -- 172.21.15.116:0/357692506 >> 172.21.15.116:0/357692506 conn(0x556dbde0aa00 msgr2=0x556dbde0ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 -- 172.21.15.116:0/357692506 shutdown_connections 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.322+0000 7fe129c12740 1 -- 172.21.15.116:0/357692506 wait complete. 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 -- start start 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 Processor -- start 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10000 0x556dbde1ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 0x556dbde1b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556dbde209a0 con 0x556dbde1dbe0 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556dbdd2f310 con 0x556dbde10000 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556dbde20b20 con 0x556dbde10920 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.323+0000 7fe129c12740 10 client.0 _ll_register_callbacks cb 0x556dbdd232d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe1255c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 0x556dbde1b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45942/0 (socket says 172.21.15.116:45942) 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 -- 172.21.15.116:0/3574281119 learned_addr learned my addr 172.21.15.116:0/3574281119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe124dc3640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10000 0x556dbde1ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 -- 172.21.15.116:0/3574281119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10000 msgr2=0x556dbde1ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10000 0x556dbde1ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 -- 172.21.15.116:0/3574281119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 msgr2=0x556dbde1b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 0x556dbde1b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 -- 172.21.15.116:0/3574281119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbda873f0 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe1255c4640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 0x556dbde1b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe117fff640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe110005890 tx=0x7fe110019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2245175f4315ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe0fffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11001e040 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe110019ae0 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.324+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fe110004030 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe110033050 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 10 client.27001 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556dbdb517d0 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe110019ae0 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe11000e740 con 0x556dbde10920 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 10 client.27001 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 10 client.27001 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 20 client.27001 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde2de40 0x556dbde30230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 4 client.27001 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dbddd13e0 con 0x556dbde2de40 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.325+0000 7fe124dc3640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde2de40 0x556dbde30230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.326+0000 7fe124dc3640 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde2de40 0x556dbde30230 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9dc7525f9ac75d23 server_cookie=37c0ac27b5caf80d in_seq=0 out_seq=0 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.326+0000 7fe0fffff640 10 client.27001 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe0fffff640 1 -- 172.21.15.116:0/3574281119 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556dbddd13e0 con 0x556dbde2de40 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe0fffff640 10 client.27001 handle_command_reply: tid=0 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 client.27001 shutdown 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 msgr2=0x556dbde1b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe110005890 tx=0x7fe110019660 comp rx=0 tx=0).stop 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 shutdown_connections 2024-03-19T02:33:06.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde2de40 0x556dbde30230 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde1dbe0 0x556dbde1b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde1b140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 --2- 172.21.15.116:0/3574281119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10000 0x556dbde1ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe0fffff640 0 client.27001 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 >> 172.21.15.116:0/3574281119 conn(0x556dbde0aa00 msgr2=0x556dbde0c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 shutdown_connections 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 1 -- 172.21.15.116:0/3574281119 wait complete. 2024-03-19T02:33:06.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.327+0000 7fe129c12740 20 client.27001 trim_cache size 0 max 16384 2024-03-19T02:33:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 Processor -- start 2024-03-19T02:33:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 -- start start 2024-03-19T02:33:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 0x556dbde29780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 0x556dbde0c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201ac880 con 0x556dbde10000 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201ae3c0 con 0x556dbde10920 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe129c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201acf50 con 0x556dbde1dbe0 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41242/0 (socket says 172.21.15.116:41242) 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 0x556dbde29780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 -- 172.21.15.116:0/8561947 learned_addr learned my addr 172.21.15.116:0/8561947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe1255c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 0x556dbde0c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 -- 172.21.15.116:0/8561947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 msgr2=0x556dbde0c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 0x556dbde0c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 -- 172.21.15.116:0/8561947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 msgr2=0x556dbde29780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 0x556dbde29780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe124dc3640 1 -- 172.21.15.116:0/8561947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbde0f3c0 con 0x556dbde10000 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe1255c4640 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 0x556dbde0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.342+0000 7fe117fff640 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 0x556dbde29780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe124dc3640 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0bd4b0 tx=0x7fe11c0f4bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c225c285743646 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe1067fc640 1 -- 172.21.15.116:0/8561947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0b97c0 con 0x556dbde10000 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe1067fc640 1 -- 172.21.15.116:0/8561947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe11c02cc20 con 0x556dbde10000 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe1067fc640 1 -- 172.21.15.116:0/8561947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0bce80 con 0x556dbde10000 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 -- 172.21.15.116:0/8561947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 msgr2=0x556dbde1d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0bd4b0 tx=0x7fe11c0f4bc0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 -- 172.21.15.116:0/8561947 shutdown_connections 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde1dbe0 0x556dbde0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde10920 0x556dbde29780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.343+0000 7fe129c12740 1 --2- 172.21.15.116:0/8561947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde1d330 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- 172.21.15.116:0/8561947 >> 172.21.15.116:0/8561947 conn(0x556dbde0aa00 msgr2=0x556dbde0be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- 172.21.15.116:0/8561947 shutdown_connections 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- 172.21.15.116:0/8561947 wait complete. 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- start start 2024-03-19T02:33:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 Processor -- start 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde174b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde179f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201adcf0 con 0x556dbde10000 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201ad620 con 0x556dbde1dbe0 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1201aa860 con 0x556dbde10920 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe129c12740 10 client.0 _ll_register_callbacks cb 0x7fe11001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe124dc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41258/0 (socket says 172.21.15.116:41258) 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe124dc3640 1 -- 172.21.15.116:0/2599187850 learned_addr learned my addr 172.21.15.116:0/2599187850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe117fff640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde174b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.344+0000 7fe1255c4640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde179f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 -- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 msgr2=0x556dbde174b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde174b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 -- 172.21.15.116:0/2599187850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 msgr2=0x556dbde179f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde179f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 -- 172.21.15.116:0/2599187850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbde18220 con 0x556dbde10000 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe1255c4640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde179f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe124dc3640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0b9ef0 tx=0x7fe11c0c5d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba396f5f38c52c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0bd060 con 0x556dbde10000 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fe11c0b9b90 con 0x556dbde10000 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe11c02cc20 con 0x556dbde10000 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe11c0bd2f0 con 0x556dbde10000 2024-03-19T02:33:06.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe129c12740 10 client.17250 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556dbde93f70 con 0x556dbde10000 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe11c02cc20 con 0x556dbde10000 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.345+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe11c0c5830 con 0x556dbde10000 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 10 client.17250 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 10 client.17250 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 20 client.17250 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde94680 0x556dbde96a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 4 client.17250 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556dbdd809f0 con 0x556dbde94680 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe117fff640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde94680 0x556dbde96a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe0fffff640 10 client.17250 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe117fff640 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde94680 0x556dbde96a70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fef94da746d8b21 server_cookie=916cfc9acec6b685 in_seq=0 out_seq=0 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe0fffff640 1 -- 172.21.15.116:0/2599187850 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556dbdd809f0 con 0x556dbde94680 2024-03-19T02:33:06.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe0fffff640 10 client.17250 handle_command_reply: tid=0 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 client.17250 shutdown 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 msgr2=0x556dbde25fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0b9ef0 tx=0x7fe11c0c5d70 comp rx=0 tx=0).stop 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 shutdown_connections 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x556dbde94680 0x556dbde96a70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556dbde1dbe0 0x556dbde179f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556dbde10920 0x556dbde174b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe0fffff640 0 client.17250 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 --2- 172.21.15.116:0/2599187850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556dbde10000 0x556dbde25fa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.346+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 >> 172.21.15.116:0/2599187850 conn(0x556dbde0aa00 msgr2=0x556dbde1b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 shutdown_connections 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe129c12740 1 -- 172.21.15.116:0/2599187850 wait complete. 2024-03-19T02:33:06.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe129c12740 20 client.17250 trim_cache size 0 max 16384 2024-03-19T02:33:06.355 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 msgr2=0x7fe1080a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 0x7fe1080a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe118009500 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 msgr2=0x7fe120197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe11c005940 tx=0x7fe11c02c5f0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 shutdown_connections 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1080a29f0 0x7fe1080a4eb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120105680 0x7fe12019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe120101330 0x7fe1201983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 --2- 172.21.15.116:0/1496152098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe120100970 0x7fe120197ea0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 >> 172.21.15.116:0/1496152098 conn(0x7fe1200774b0 msgr2=0x7fe12006a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 shutdown_connections 2024-03-19T02:33:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.347+0000 7fe127027640 1 -- 172.21.15.116:0/1496152098 wait complete. 2024-03-19T02:33:06.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:33:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 -- 172.21.15.116:0/955385538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 msgr2=0x7f2500107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 --2- 172.21.15.116:0/955385538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f2500107b90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009930 tx=0x7f24fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 -- 172.21.15.116:0/955385538 shutdown_connections 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 --2- 172.21.15.116:0/955385538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2500102160 0x7f250010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 --2- 172.21.15.116:0/955385538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25001017a0 0x7f2500101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 --2- 172.21.15.116:0/955385538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f2500107b90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.473+0000 7f2507627640 1 -- 172.21.15.116:0/955385538 >> 172.21.15.116:0/955385538 conn(0x7f25000fd2d0 msgr2=0x7f25000ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- 172.21.15.116:0/955385538 shutdown_connections 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- 172.21.15.116:0/955385538 wait complete. 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 Processor -- start 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- start start 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 0x7f2500197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 0x7f2500198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2500111a30 con 0x7f25001077b0 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25001118b0 con 0x7f25001017a0 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2507627640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2500111bb0 con 0x7f2500102160 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.474+0000 7f2505bc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f25053c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 0x7f2500197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41268/0 (socket says 172.21.15.116:41268) 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 -- 172.21.15.116:0/1456562025 learned_addr learned my addr 172.21.15.116:0/1456562025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 0x7f2500198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 -- 172.21.15.116:0/1456562025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 msgr2=0x7f2500198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 0x7f2500198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 -- 172.21.15.116:0/1456562025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 msgr2=0x7f2500197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 0x7f2500197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 -- 172.21.15.116:0/1456562025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250019cfe0 con 0x7f25001077b0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 0x7f2500198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f25053c3640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 0x7f2500197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2505bc4640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f24f800eb10 tx=0x7f24f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f24f67fc640 1 -- 172.21.15.116:0/1456562025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8004b20 con 0x7f25001077b0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f24f67fc640 1 -- 172.21.15.116:0/1456562025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24f8004cc0 con 0x7f25001077b0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250019d2d0 con 0x7f25001077b0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f24f67fc640 1 -- 172.21.15.116:0/1456562025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8010690 con 0x7f25001077b0 2024-03-19T02:33:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.475+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25001a4b60 con 0x7f25001077b0 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.478+0000 7f24f67fc640 1 -- 172.21.15.116:0/1456562025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f24f8005020 con 0x7f25001077b0 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 Processor -- start 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 -- start start 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d026152290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f24f67fc640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 0x7f24ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 0x55d026154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d0260ac4b0 con 0x55d026149bf0 2024-03-19T02:33:06.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d026054d70 con 0x55d0261527d0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f250a212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0260a8550 con 0x55d0261492d0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f2505bc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 0x55d026154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45974/0 (socket says 172.21.15.116:45974) 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f2504bc2640 1 -- 172.21.15.116:0/1116238563 learned_addr learned my addr 172.21.15.116:0/1116238563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f25053c3640 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d026152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 0x7f24ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.479+0000 7f24f67fc640 1 -- 172.21.15.116:0/1456562025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f24f80cf730 con 0x7f25001077b0 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 -- 172.21.15.116:0/1116238563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 msgr2=0x55d0261496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 -- 172.21.15.116:0/1116238563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 msgr2=0x55d026154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 0x55d026154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 -- 172.21.15.116:0/1116238563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d026077210 con 0x55d026149bf0 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f2505bc4640 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 0x55d026154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 0x7f24ec0a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24f00097e0 tx=0x7f24f0009ca0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f25053c3640 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d026152290 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc0095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6febb23ef6597e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.480+0000 7f24eb7fe640 1 -- 172.21.15.116:0/1116238563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc045070 con 0x55d026149bf0 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f24eb7fe640 1 -- 172.21.15.116:0/1116238563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24fc02c7e0 con 0x55d026149bf0 2024-03-19T02:33:06.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f24eb7fe640 1 -- 172.21.15.116:0/1116238563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0039d0 con 0x55d026149bf0 2024-03-19T02:33:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 -- 172.21.15.116:0/1116238563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 msgr2=0x55d026152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d026152290 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009230 tx=0x7f24fc0095f0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 -- 172.21.15.116:0/1116238563 shutdown_connections 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261527d0 0x55d026154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d026152290 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 --2- 172.21.15.116:0/1116238563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 -- 172.21.15.116:0/1116238563 >> 172.21.15.116:0/1116238563 conn(0x55d026143a00 msgr2=0x55d026143de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.481+0000 7f250a212740 1 -- 172.21.15.116:0/1116238563 shutdown_connections 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 -- 172.21.15.116:0/1116238563 wait complete. 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 -- start start 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 Processor -- start 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d02614f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026150320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d026155560 con 0x55d026149bf0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d026068310 con 0x55d0261492d0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0261556e0 con 0x55d0261527d0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f250a212740 10 client.0 _ll_register_callbacks cb 0x55d02605c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41284/0 (socket says 172.21.15.116:41284) 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f2504bc2640 1 -- 172.21.15.116:0/1568688168 learned_addr learned my addr 172.21.15.116:0/1568688168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f25053c3640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d02614f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.482+0000 7f2505bc4640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 -- 172.21.15.116:0/1568688168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 msgr2=0x55d026150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 -- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 msgr2=0x55d02614f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d02614f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 -- 172.21.15.116:0/1568688168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d025dc03f0 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f25053c3640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d02614f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f2504bc2640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011d10 tx=0x7f24f0009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bead5c15d43eb123 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f24e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0027070 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f24f0003af0 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24fc02c7e0 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24f0033050 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f250a212740 10 client.17262 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d025e8a7d0 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.483+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24fc02c7e0 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f24f0016c70 con 0x55d026149bf0 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 10 client.17262 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 20 client.17262 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d026162ad0 0x55d026164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 4 client.17262 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d02610a3e0 con 0x55d026162ad0 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f25053c3640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d026162ad0 0x55d026164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f25053c3640 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d026162ad0 0x55d026164f60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55f9978092adeb6e server_cookie=236091e0a490b3fb in_seq=0 out_seq=0 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.484+0000 7f24e8ff9640 10 client.17262 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f24e8ff9640 1 -- 172.21.15.116:0/1568688168 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d02610a3e0 con 0x55d026162ad0 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f24e8ff9640 10 client.17262 handle_command_reply: tid=0 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 client.17262 shutdown 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 msgr2=0x55d02614fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011d10 tx=0x7f24f0009770 comp rx=0 tx=0).stop 2024-03-19T02:33:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 shutdown_connections 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d026162ad0 0x55d026164f60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f24e8ff9640 0 client.17262 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02614fde0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 --2- 172.21.15.116:0/1568688168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d02614f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 >> 172.21.15.116:0/1568688168 conn(0x55d026143a00 msgr2=0x55d026145750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 shutdown_connections 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 1 -- 172.21.15.116:0/1568688168 wait complete. 2024-03-19T02:33:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.485+0000 7f250a212740 20 client.17262 trim_cache size 0 max 16384 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 Processor -- start 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 -- start start 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d026151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25001ac910 con 0x55d026149bf0 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25001ae450 con 0x55d0261492d0 2024-03-19T02:33:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f250a212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25001acfe0 con 0x55d0261527d0 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.499+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41300/0 (socket says 172.21.15.116:41300) 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 -- 172.21.15.116:0/201789959 learned_addr learned my addr 172.21.15.116:0/201789959 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2505bc4640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f25053c3640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d026151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 -- 172.21.15.116:0/201789959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 msgr2=0x55d026142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 -- 172.21.15.116:0/201789959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 msgr2=0x55d026151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d026151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 -- 172.21.15.116:0/201789959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02614e850 con 0x55d026149bf0 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f25053c3640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d026151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2505bc4640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f2504bc2640 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f24f0023440 tx=0x7f24f0017980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98769bf4c419dbc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f24eaffd640 1 -- 172.21.15.116:0/201789959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f003c070 con 0x55d026149bf0 2024-03-19T02:33:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f24eaffd640 1 -- 172.21.15.116:0/201789959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24f0030070 con 0x55d026149bf0 2024-03-19T02:33:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.500+0000 7f24eaffd640 1 -- 172.21.15.116:0/201789959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0027040 con 0x55d026149bf0 2024-03-19T02:33:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- 172.21.15.116:0/201789959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 msgr2=0x55d02615e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f24f0023440 tx=0x7f24f0017980 comp rx=0 tx=0).stop 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- 172.21.15.116:0/201789959 shutdown_connections 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261527d0 0x55d026142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d026149bf0 0x55d02615e300 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 --2- 172.21.15.116:0/201789959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d0261492d0 0x55d026151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- 172.21.15.116:0/201789959 >> 172.21.15.116:0/201789959 conn(0x55d026143a00 msgr2=0x55d026143de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- 172.21.15.116:0/201789959 shutdown_connections 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- 172.21.15.116:0/201789959 wait complete. 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.501+0000 7f250a212740 1 -- start start 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 Processor -- start 2024-03-19T02:33:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d026149bf0 0x55d026156920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25001add80 con 0x55d0261527d0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25001ad6b0 con 0x55d026149bf0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25001aa8f0 con 0x55d0261492d0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f250a212740 10 client.0 _ll_register_callbacks cb 0x7f24f0017430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f25053c3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f25053c3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46000/0 (socket says 172.21.15.116:46000) 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41310/0 (socket says 172.21.15.116:41310) 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f25053c3640 1 -- 172.21.15.116:0/923126609 learned_addr learned my addr 172.21.15.116:0/923126609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2504bc2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d026149bf0 0x55d026156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 -- 172.21.15.116:0/923126609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 msgr2=0x55d0261563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 -- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d026149bf0 msgr2=0x55d026156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d026149bf0 0x55d026156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f2505bc4640 1 -- 172.21.15.116:0/923126609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0261c8060 con 0x55d0261527d0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.502+0000 7f25053c3640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f2505bc4640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f24f80ccb00 tx=0x7f24f8009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc53eb06a1bd414 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f809c3a0 con 0x55d0261527d0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f24f80cf390 con 0x55d0261527d0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24f0030070 con 0x55d0261527d0 2024-03-19T02:33:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f24f80cf590 con 0x55d0261527d0 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 10 client.17268 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d0261c9360 con 0x55d0261527d0 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24f0030070 con 0x55d0261527d0 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f24f80cf070 con 0x55d0261527d0 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 10 client.17268 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 20 client.17268 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0261c9a30 0x55d0261cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 4 client.17268 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.503+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0260b99f0 con 0x55d0261c9a30 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.504+0000 7f25053c3640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0261c9a30 0x55d0261cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.504+0000 7f24e8ff9640 10 client.17268 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.504+0000 7f25053c3640 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0261c9a30 0x55d0261cbe50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f56a260042250b84 server_cookie=a454d22de814bf67 in_seq=0 out_seq=0 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f24e8ff9640 1 -- 172.21.15.116:0/923126609 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0260b99f0 con 0x55d0261c9a30 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f24e8ff9640 10 client.17268 handle_command_reply: tid=0 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 client.17268 shutdown 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 msgr2=0x55d026156e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f24f80ccb00 tx=0x7f24f8009a70 comp rx=0 tx=0).stop 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 shutdown_connections 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d0261c9a30 0x55d0261cbe50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d0261527d0 0x55d026156e60 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d026149bf0 0x55d026156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 --2- 172.21.15.116:0/923126609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0261492d0 0x55d0261563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 >> 172.21.15.116:0/923126609 conn(0x55d026143a00 msgr2=0x55d026145630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f24e8ff9640 0 client.17268 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 shutdown_connections 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 1 -- 172.21.15.116:0/923126609 wait complete. 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.505+0000 7f250a212740 20 client.17268 trim_cache size 0 max 16384 2024-03-19T02:33:06.513 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:06.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 msgr2=0x7f24ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 0x7f24ec0a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24f00097e0 tx=0x7f24f0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 msgr2=0x7f250019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:06.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f24f800eb10 tx=0x7f24f800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 shutdown_connections 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24ec0a2970 0x7f24ec0a4e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25001077b0 0x7f250019c8b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2500102160 0x7f2500198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 --2- 172.21.15.116:0/1456562025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25001017a0 0x7f2500197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 >> 172.21.15.116:0/1456562025 conn(0x7f25000fd2d0 msgr2=0x7f25000fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 shutdown_connections 2024-03-19T02:33:06.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:06.506+0000 7f2507627640 1 -- 172.21.15.116:0/1456562025 wait complete. 2024-03-19T02:33:06.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 163491 3.26 MB/sec execute 149 sec latency 21.564 ms 2024-03-19T02:33:07.518 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:33:07.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:33:07.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:33:07.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.635+0000 7faa3a353640 1 -- 172.21.15.116:0/3745454015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa34101050 msgr2=0x7faa3410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.635+0000 7faa3a353640 1 --2- 172.21.15.116:0/3745454015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa34101050 0x7faa3410d560 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7faa2c009cd0 tx=0x7faa2c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- 172.21.15.116:0/3745454015 shutdown_connections 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- 172.21.15.116:0/3745454015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa34101050 0x7faa3410d560 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- 172.21.15.116:0/3745454015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34100690 0x7faa34100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- 172.21.15.116:0/3745454015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa341066a0 0x7faa34106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- 172.21.15.116:0/3745454015 >> 172.21.15.116:0/3745454015 conn(0x7faa34077820 msgr2=0x7faa34077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- 172.21.15.116:0/3745454015 shutdown_connections 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- 172.21.15.116:0/3745454015 wait complete. 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 Processor -- start 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- start start 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 0x7faa34196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 0x7faa341973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa34110960 con 0x7faa341066a0 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa341107e0 con 0x7faa34100690 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.636+0000 7faa3a353640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa34110ae0 con 0x7faa34101050 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 0x7faa34196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 0x7faa341973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 0x7faa34196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56958/0 (socket says 172.21.15.116:56958) 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 0x7faa341973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59346/0 (socket says 172.21.15.116:59346) 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50280/0 (socket says 172.21.15.116:50280) 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa33fff640 1 -- 172.21.15.116:0/2358355902 learned_addr learned my addr 172.21.15.116:0/2358355902 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 -- 172.21.15.116:0/2358355902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 msgr2=0x7faa341973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 0x7faa341973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 -- 172.21.15.116:0/2358355902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 msgr2=0x7faa34196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 0x7faa34196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 -- 172.21.15.116:0/2358355902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3419be90 con 0x7faa341066a0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa388f0640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c630 tx=0x7faa2c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa317fa640 1 -- 172.21.15.116:0/2358355902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c039040 con 0x7faa341066a0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa317fa640 1 -- 172.21.15.116:0/2358355902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c044070 con 0x7faa341066a0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3419c120 con 0x7faa341066a0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.637+0000 7faa317fa640 1 -- 172.21.15.116:0/2358355902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c004330 con 0x7faa341066a0 2024-03-19T02:33:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.638+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa34068980 con 0x7faa341066a0 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa317fa640 1 -- 172.21.15.116:0/2358355902 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faa2c0044d0 con 0x7faa341066a0 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 Processor -- start 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 -- start start 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb07760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb10880 0x55d62fb12c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d62fa6b160 con 0x55d62fb07380 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d62fa13e10 con 0x55d62fb10880 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa3d01b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d62fa67200 con 0x55d62fb07ca0 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb07760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa33fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa33fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59362/0 (socket says 172.21.15.116:59362) 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa33fff640 1 -- 172.21.15.116:0/3028250447 learned_addr learned my addr 172.21.15.116:0/3028250447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb07760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50296/0 (socket says 172.21.15.116:50296) 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa337fe640 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb10880 0x55d62fb12c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.640+0000 7faa317fa640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 0x7faa1c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 -- 172.21.15.116:0/3028250447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb10880 msgr2=0x55d62fb12c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb10880 0x55d62fb12c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 -- 172.21.15.116:0/3028250447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 msgr2=0x55d62fb07760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb07760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 -- 172.21.15.116:0/3028250447 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d62f8477e0 con 0x55d62fb07ca0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa388f0640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 0x7faa1c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa317fa640 1 -- 172.21.15.116:0/2358355902 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7faa2c0f1490 con 0x7faa341066a0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa33fff640 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faa2400df10 tx=0x7faa24004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d06e8c52d1d1b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa1a7fc640 1 -- 172.21.15.116:0/3028250447 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24003dd0 con 0x55d62fb07ca0 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa388f0640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 0x7faa1c0a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0b9ce0 tx=0x7faa2c0b9ba0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa1a7fc640 1 -- 172.21.15.116:0/3028250447 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2401c070 con 0x55d62fb07ca0 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.641+0000 7faa1a7fc640 1 -- 172.21.15.116:0/3028250447 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24016650 con 0x55d62fb07ca0 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- 172.21.15.116:0/3028250447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 msgr2=0x55d62fb10340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faa2400df10 tx=0x7faa24004410 comp rx=0 tx=0).stop 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- 172.21.15.116:0/3028250447 shutdown_connections 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb10880 0x55d62fb12c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb10340 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 --2- 172.21.15.116:0/3028250447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb07760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- 172.21.15.116:0/3028250447 >> 172.21.15.116:0/3028250447 conn(0x55d62fb01ab0 msgr2=0x55d62fb01e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- 172.21.15.116:0/3028250447 shutdown_connections 2024-03-19T02:33:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- 172.21.15.116:0/3028250447 wait complete. 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 -- start start 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.642+0000 7faa3d01b740 1 Processor -- start 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07380 0x55d62fb0d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb0ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d62fb13670 con 0x55d62fb10880 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d62fa271a0 con 0x55d62fb07380 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d62fb137f0 con 0x55d62fb07ca0 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 10 client.0 _ll_register_callbacks cb 0x55d62f77ec60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa33fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb0ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50312/0 (socket says 172.21.15.116:50312) 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 -- 172.21.15.116:0/1822887223 learned_addr learned my addr 172.21.15.116:0/1822887223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07380 0x55d62fb0d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 -- 172.21.15.116:0/1822887223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 msgr2=0x55d62fb0ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb0ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 -- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07380 msgr2=0x55d62fb0d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07380 0x55d62fb0d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 -- 172.21.15.116:0/1822887223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d62f8a7f70 con 0x55d62fb10880 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa33fff640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb0ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa337fe640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7faa28003830 tx=0x7faa28004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97dbdee4689fc2dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa137fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28005020 con 0x55d62fb10880 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7faa280051c0 con 0x55d62fb10880 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.643+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa2401c070 con 0x55d62fb10880 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faa2802a050 con 0x55d62fb10880 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 10 client.17280 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d62f73f170 con 0x55d62fb10880 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2401c070 con 0x55d62fb10880 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faa2800e750 con 0x55d62fb10880 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 10 client.17280 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 20 client.17280 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb20c50 0x55d62fb23060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 4 client.17280 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d62fa786a0 con 0x55d62fb20c50 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.644+0000 7faa33fff640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb20c50 0x55d62fb23060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.645+0000 7faa33fff640 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb20c50 0x55d62fb23060 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a2ab9ef2d9490a server_cookie=56ad5ac5c390eebe in_seq=0 out_seq=0 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.645+0000 7faa137fe640 10 client.17280 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.646+0000 7faa137fe640 1 -- 172.21.15.116:0/1822887223 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d62fa786a0 con 0x55d62fb20c50 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.646+0000 7faa137fe640 10 client.17280 handle_command_reply: tid=0 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.646+0000 7faa3d01b740 1 client.17280 shutdown 2024-03-19T02:33:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 msgr2=0x55d62fb0e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7faa28003830 tx=0x7faa28004140 comp rx=0 tx=0).stop 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 shutdown_connections 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb20c50 0x55d62fb23060 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb10880 0x55d62fb0e300 secure :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7faa28003830 tx=0x7faa28004140 comp rx=0 tx=0).stop 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb0ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1822887223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07380 0x55d62fb0d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 >> 172.21.15.116:0/1822887223 conn(0x55d62fb01ab0 msgr2=0x55d62fb03800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa137fe640 0 client.17280 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 shutdown_connections 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 1 -- 172.21.15.116:0/1822887223 wait complete. 2024-03-19T02:33:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.647+0000 7faa3d01b740 20 client.17280 trim_cache size 0 max 16384 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.666+0000 7faa3d01b740 1 Processor -- start 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.666+0000 7faa3d01b740 1 -- start start 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07380 0x55d62fb0d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 0x55d62fb1c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa341ac930 con 0x55d62fb28930 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa341ae470 con 0x55d62fb07ca0 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa3d01b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa341ad000 con 0x55d62fb07380 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50320/0 (socket says 172.21.15.116:50320) 2024-03-19T02:33:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 -- 172.21.15.116:0/270678626 learned_addr learned my addr 172.21.15.116:0/270678626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa33fff640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 0x55d62fb1c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07380 0x55d62fb0d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 -- 172.21.15.116:0/270678626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07380 msgr2=0x55d62fb0d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07380 0x55d62fb0d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 -- 172.21.15.116:0/270678626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 msgr2=0x55d62fb1c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 0x55d62fb1c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 -- 172.21.15.116:0/270678626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d62fb06b20 con 0x55d62fb28930 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa33fff640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 0x55d62fb1c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa337fe640 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7faa2800bbf0 tx=0x7faa28000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d82bf376fca7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa19ffb640 1 -- 172.21.15.116:0/270678626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa280035f0 con 0x55d62fb28930 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa19ffb640 1 -- 172.21.15.116:0/270678626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa280054d0 con 0x55d62fb28930 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.667+0000 7faa19ffb640 1 -- 172.21.15.116:0/270678626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28004900 con 0x55d62fb28930 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 -- 172.21.15.116:0/270678626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 msgr2=0x55d62fb0df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7faa2800bbf0 tx=0x7faa28000bc0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 -- 172.21.15.116:0/270678626 shutdown_connections 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb28930 0x55d62fb0df50 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb07ca0 0x55d62fb1c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 --2- 172.21.15.116:0/270678626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07380 0x55d62fb0d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 -- 172.21.15.116:0/270678626 >> 172.21.15.116:0/270678626 conn(0x55d62fb01ab0 msgr2=0x55d62fb0d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 -- 172.21.15.116:0/270678626 shutdown_connections 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.668+0000 7faa3d01b740 1 -- 172.21.15.116:0/270678626 wait complete. 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 -- start start 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 Processor -- start 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb14150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb28930 0x55d62fb14760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa341adda0 con 0x55d62fb07380 2024-03-19T02:33:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa341ad6d0 con 0x55d62fb28930 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa341aa910 con 0x55d62fb07ca0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa3d01b740 10 client.0 _ll_register_callbacks cb 0x7faa2800b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa33fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb14150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50332/0 (socket says 172.21.15.116:50332) 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa337fe640 1 -- 172.21.15.116:0/1785770926 learned_addr learned my addr 172.21.15.116:0/1785770926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa337fe640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb28930 0x55d62fb14760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa33fff640 1 -- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb28930 msgr2=0x55d62fb14760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa33fff640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb28930 0x55d62fb14760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa33fff640 1 -- 172.21.15.116:0/1785770926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 msgr2=0x55d62fb18de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.669+0000 7faa33fff640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa33fff640 1 -- 172.21.15.116:0/1785770926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d62fb14f60 con 0x55d62fb07ca0 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa337fe640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa33fff640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb14150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faa24000c00 tx=0x7faa2400dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d91deb61122d8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa137fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2402c620 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa280054d0 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7faa2402c7c0 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faa24005040 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa3d01b740 10 client.27043 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d62fb87070 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.670+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa280054d0 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faa24004c50 con 0x55d62fb07ca0 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 10 client.27043 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 10 client.27043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 20 client.27043 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb875f0 0x55d62fb899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 4 client.27043 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:33:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d62fa729f0 con 0x55d62fb875f0 2024-03-19T02:33:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa337fe640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb875f0 0x55d62fb899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa137fe640 10 client.27043 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.671+0000 7faa337fe640 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb875f0 0x55d62fb899e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16bdfb77ca9971ab server_cookie=5866b87f9e1bf4a1 in_seq=0 out_seq=0 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa137fe640 1 -- 172.21.15.116:0/1785770926 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d62fa729f0 con 0x55d62fb875f0 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa137fe640 10 client.27043 handle_command_reply: tid=0 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 client.27043 shutdown 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 msgr2=0x55d62fb14150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb14150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faa24000c00 tx=0x7faa2400dc00 comp rx=0 tx=0).stop 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 shutdown_connections 2024-03-19T02:33:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d62fb875f0 0x55d62fb899e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d62fb28930 0x55d62fb14760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d62fb07ca0 0x55d62fb14150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 --2- 172.21.15.116:0/1785770926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d62fb07380 0x55d62fb18de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 >> 172.21.15.116:0/1785770926 conn(0x55d62fb01ab0 msgr2=0x55d62fb281b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa137fe640 0 client.27043 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 shutdown_connections 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 1 -- 172.21.15.116:0/1785770926 wait complete. 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.674+0000 7faa3d01b740 20 client.27043 trim_cache size 0 max 16384 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:07.679 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:33:07.680 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "2da73709-35fc-4c1c-86b8-75f29d1d7b4a", 2024-03-19T02:33:07.680 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:33:07.680 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 msgr2=0x7faa1c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 0x7faa1c0a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0b9ce0 tx=0x7faa2c0b9ba0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 msgr2=0x7faa3419b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7faa2c02c630 tx=0x7faa2c035d60 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 shutdown_connections 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faa1c0a2970 0x7faa1c0a4e30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa341066a0 0x7faa3419b760 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa34101050 0x7faa341973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 --2- 172.21.15.116:0/2358355902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa34100690 0x7faa34196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 >> 172.21.15.116:0/2358355902 conn(0x7faa34077820 msgr2=0x7faa34102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 shutdown_connections 2024-03-19T02:33:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.675+0000 7faa3a353640 1 -- 172.21.15.116:0/2358355902 wait complete. 2024-03-19T02:33:07.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:33:07.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:33:07.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 163909 3.24 MB/sec execute 150 sec latency 45.697 ms 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 -- 172.21.15.116:0/3658633749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd224101050 msgr2=0x7fd22410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 --2- 172.21.15.116:0/3658633749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd224101050 0x7fd22410d560 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd21c009cd0 tx=0x7fd21c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 -- 172.21.15.116:0/3658633749 shutdown_connections 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 --2- 172.21.15.116:0/3658633749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd224101050 0x7fd22410d560 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 --2- 172.21.15.116:0/3658633749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224100690 0x7fd224100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 --2- 172.21.15.116:0/3658633749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2241066a0 0x7fd224106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 -- 172.21.15.116:0/3658633749 >> 172.21.15.116:0/3658633749 conn(0x7fd224077820 msgr2=0x7fd224077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 -- 172.21.15.116:0/3658633749 shutdown_connections 2024-03-19T02:33:07.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.804+0000 7fd22916f640 1 -- 172.21.15.116:0/3658633749 wait complete. 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.805+0000 7fd22916f640 1 Processor -- start 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.805+0000 7fd22916f640 1 -- start start 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.805+0000 7fd22916f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.805+0000 7fd22916f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 0x7fd2241031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.805+0000 7fd22916f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2241066a0 0x7fd2241036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22916f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd224110960 con 0x7fd2241066a0 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22916f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2241107e0 con 0x7fd224101050 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22916f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd224110ae0 con 0x7fd224100690 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2241066a0 0x7fd2241036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59402/0 (socket says 172.21.15.116:59402) 2024-03-19T02:33:07.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 0x7fd2241031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 -- 172.21.15.116:0/2947823956 learned_addr learned my addr 172.21.15.116:0/2947823956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 -- 172.21.15.116:0/2947823956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 msgr2=0x7fd2241031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 0x7fd2241031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 -- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2241066a0 msgr2=0x7fd2241036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2241066a0 0x7fd2241036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 -- 172.21.15.116:0/2947823956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2241a6510 con 0x7fd224100690 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22259c640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 0x7fd2241031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd222d9d640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd21400e9f0 tx=0x7fd21400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd20bfff640 1 -- 172.21.15.116:0/2947823956 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21400cd80 con 0x7fd224100690 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2241a6800 con 0x7fd224100690 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.807+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2241a6d50 con 0x7fd224100690 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.806+0000 7fd20bfff640 1 -- 172.21.15.116:0/2947823956 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2140054d0 con 0x7fd224100690 2024-03-19T02:33:07.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.807+0000 7fd20bfff640 1 -- 172.21.15.116:0/2947823956 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214010490 con 0x7fd224100690 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.809+0000 7fd20bfff640 1 -- 172.21.15.116:0/2947823956 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd214010770 con 0x7fd224100690 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.809+0000 7fd22be30740 1 Processor -- start 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.809+0000 7fd22be30740 1 -- start start 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3959f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a398320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558e5a2f0500 con 0x558e5a395f30 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e5a298be0 con 0x558e5a38d350 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22be30740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e5a2ec5a0 con 0x558e5a38ca30 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59408/0 (socket says 172.21.15.116:59408) 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 -- 172.21.15.116:0/4075363849 learned_addr learned my addr 172.21.15.116:0/4075363849 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22259c640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a398320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd20bfff640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 0x7fd20c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 -- 172.21.15.116:0/4075363849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 msgr2=0x558e5a3959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd222d9d640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd20bfff640 1 -- 172.21.15.116:0/2947823956 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd2140cf340 con 0x7fd224100690 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 -- 172.21.15.116:0/4075363849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 msgr2=0x558e5a398320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a398320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 -- 172.21.15.116:0/4075363849 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e5a2ba000 con 0x558e5a38ca30 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd222d9d640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.810+0000 7fd22359e640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 0x7fd20c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.811+0000 7fd22259c640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a398320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.811+0000 7fd22359e640 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004af0 tx=0x7fd21c0323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a15d7c46fba0f43c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.811+0000 7fd2097fa640 1 -- 172.21.15.116:0/4075363849 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c039040 con 0x558e5a38ca30 2024-03-19T02:33:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.811+0000 7fd2097fa640 1 -- 172.21.15.116:0/4075363849 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd21c003b70 con 0x558e5a38ca30 2024-03-19T02:33:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.811+0000 7fd2097fa640 1 -- 172.21.15.116:0/4075363849 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c034680 con 0x558e5a38ca30 2024-03-19T02:33:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- 172.21.15.116:0/4075363849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 msgr2=0x558e5a38ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd21c004af0 tx=0x7fd21c0323c0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22359e640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 0x7fd20c0a4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd21c04c000 tx=0x7fd21c045000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- 172.21.15.116:0/4075363849 shutdown_connections 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a398320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 --2- 172.21.15.116:0/4075363849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38ca30 0x558e5a38ce10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- 172.21.15.116:0/4075363849 >> 172.21.15.116:0/4075363849 conn(0x558e5a387160 msgr2=0x558e5a387540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- 172.21.15.116:0/4075363849 shutdown_connections 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- 172.21.15.116:0/4075363849 wait complete. 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 -- start start 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.812+0000 7fd22be30740 1 Processor -- start 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3933d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a395f30 0x558e5a393910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558e5a398d20 con 0x558e5a38ca30 2024-03-19T02:33:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e5a2ac160 con 0x558e5a38d350 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e5a398ea0 con 0x558e5a395f30 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22be30740 10 client.0 _ll_register_callbacks cb 0x558e5a2e8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50364/0 (socket says 172.21.15.116:50364) 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3933d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 -- 172.21.15.116:0/4024335402 learned_addr learned my addr 172.21.15.116:0/4024335402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22359e640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a395f30 0x558e5a393910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 -- 172.21.15.116:0/4024335402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a395f30 msgr2=0x558e5a393910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a395f30 0x558e5a393910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 -- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 msgr2=0x558e5a3933d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3933d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 -- 172.21.15.116:0/4024335402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e5a0ce7d0 con 0x558e5a38ca30 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd22259c640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3933d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd222d9d640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd214004cb0 tx=0x7fd21409b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b58cd9338dffff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.813+0000 7fd206ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2140b4060 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd21409ba80 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd21c003b70 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd2140cce60 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 10 client.17298 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e5a0043f0 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd21c003b70 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd2140b4290 con 0x558e5a38ca30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 10 client.17298 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 20 client.17298 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a3a6270 0x558e5a3a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 4 client.17298 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e5a2fda40 con 0x558e5a3a6270 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.814+0000 7fd22259c640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a3a6270 0x558e5a3a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.815+0000 7fd22259c640 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a3a6270 0x558e5a3a86a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d5bfe030f95a478 server_cookie=aa67cc5d5f97a783 in_seq=0 out_seq=0 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.815+0000 7fd206ff5640 10 client.17298 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.815+0000 7fd206ff5640 1 -- 172.21.15.116:0/4024335402 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e5a2fda40 con 0x558e5a3a6270 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.815+0000 7fd206ff5640 10 client.17298 handle_command_reply: tid=0 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.815+0000 7fd22be30740 1 client.17298 shutdown 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 msgr2=0x558e5a392e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd214004cb0 tx=0x7fd21409b040 comp rx=0 tx=0).stop 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 shutdown_connections 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a3a6270 0x558e5a3a86a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a395f30 0x558e5a393910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a38d350 0x558e5a3933d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 --2- 172.21.15.116:0/4024335402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a38ca30 0x558e5a392e90 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 >> 172.21.15.116:0/4024335402 conn(0x558e5a387160 msgr2=0x558e5a388eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd206ff5640 0 client.17298 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 shutdown_connections 2024-03-19T02:33:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 1 -- 172.21.15.116:0/4024335402 wait complete. 2024-03-19T02:33:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.816+0000 7fd22be30740 20 client.17298 trim_cache size 0 max 16384 2024-03-19T02:33:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.834+0000 7fd22be30740 1 Processor -- start 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.834+0000 7fd22be30740 1 -- start start 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a3955c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a3adfb0 0x558e5a3ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd21000fda0 con 0x558e5a395f30 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2100118e0 con 0x558e5a3adfb0 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22be30740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd210010470 con 0x558e5a38d350 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50368/0 (socket says 172.21.15.116:50368) 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 -- 172.21.15.116:0/2060303143 learned_addr learned my addr 172.21.15.116:0/2060303143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd222d9d640 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a3955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22359e640 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a3adfb0 0x558e5a3ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 -- 172.21.15.116:0/2060303143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 msgr2=0x558e5a3955c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a3955c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 -- 172.21.15.116:0/2060303143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a3adfb0 msgr2=0x558e5a3ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a3adfb0 0x558e5a3ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 -- 172.21.15.116:0/2060303143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e59fc40f0 con 0x558e5a395f30 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd22259c640 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x558e5a3882f0 tx=0x7fd218005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb5c28de31a4ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd208ff9640 1 -- 172.21.15.116:0/2060303143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2180120d0 con 0x558e5a395f30 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd208ff9640 1 -- 172.21.15.116:0/2060303143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd218026020 con 0x558e5a395f30 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.835+0000 7fd208ff9640 1 -- 172.21.15.116:0/2060303143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218022680 con 0x558e5a395f30 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- 172.21.15.116:0/2060303143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 msgr2=0x558e5a3a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x558e5a3882f0 tx=0x7fd218005f90 comp rx=0 tx=0).stop 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- 172.21.15.116:0/2060303143 shutdown_connections 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a3adfb0 0x558e5a3ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a395f30 0x558e5a3a1b50 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 --2- 172.21.15.116:0/2060303143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a3955c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- 172.21.15.116:0/2060303143 >> 172.21.15.116:0/2060303143 conn(0x558e5a387160 msgr2=0x558e5a392e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- 172.21.15.116:0/2060303143 shutdown_connections 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- 172.21.15.116:0/2060303143 wait complete. 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.836+0000 7fd22be30740 1 -- start start 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 Processor -- start 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a399990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a395f30 0x558e5a399ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd210011210 con 0x558e5a3adfb0 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd210010b40 con 0x558e5a395f30 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd21000dd80 con 0x558e5a38d350 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22be30740 10 client.0 _ll_register_callbacks cb 0x7fd21409b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a399990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50376/0 (socket says 172.21.15.116:50376) 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22359e640 1 -- 172.21.15.116:0/3500186607 learned_addr learned my addr 172.21.15.116:0/3500186607 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22259c640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a395f30 0x558e5a399ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 -- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a395f30 msgr2=0x558e5a399ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a395f30 0x558e5a399ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 -- 172.21.15.116:0/3500186607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 msgr2=0x558e5a39a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 -- 172.21.15.116:0/3500186607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e5a40b5e0 con 0x558e5a38d350 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd22359e640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd222d9d640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a399990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd214004c10 tx=0x7fd21409aca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=713102924c7d7233 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.837+0000 7fd206ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:33:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214005200 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd2140046c0 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd218026020 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd21409b620 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 10 client.27061 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e5a40c7f0 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd218026020 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd2140b4060 con 0x558e5a38d350 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 10 client.27061 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 10 client.27061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 20 client.27061 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a40cf40 0x558e5a40f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 4 client.27061 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558e5a2f7d90 con 0x558e5a40cf40 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.838+0000 7fd22259c640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a40cf40 0x558e5a40f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.839+0000 7fd206ff5640 10 client.27061 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.839+0000 7fd22259c640 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a40cf40 0x558e5a40f360 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c749b225d17a0c3f server_cookie=a9de4b0bdaab90e7 in_seq=0 out_seq=0 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd206ff5640 1 -- 172.21.15.116:0/3500186607 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558e5a2f7d90 con 0x558e5a40cf40 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd206ff5640 10 client.27061 handle_command_reply: tid=0 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 client.27061 shutdown 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 msgr2=0x558e5a399990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a399990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd214004c10 tx=0x7fd21409aca0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 shutdown_connections 2024-03-19T02:33:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e5a40cf40 0x558e5a40f360 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e5a3adfb0 0x558e5a39a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd206ff5640 0 client.27061 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e5a395f30 0x558e5a399ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 --2- 172.21.15.116:0/3500186607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e5a38d350 0x558e5a399990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 >> 172.21.15.116:0/3500186607 conn(0x558e5a387160 msgr2=0x558e5a391330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 shutdown_connections 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 1 -- 172.21.15.116:0/3500186607 wait complete. 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22be30740 20 client.27061 trim_cache size 0 max 16384 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:07.847 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: "2da73709-35fc-4c1c-86b8-75f29d1d7b4a": { 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "2da73709-35fc-4c1c-86b8-75f29d1d7b4a", 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:33:07.848 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 msgr2=0x7fd20c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 0x7fd20c0a4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd21c04c000 tx=0x7fd21c045000 comp rx=0 tx=0).stop 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 msgr2=0x7fd224102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd21400e9f0 tx=0x7fd21400edf0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 shutdown_connections 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd20c0a29f0 0x7fd20c0a4eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2241066a0 0x7fd2241036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd224101050 0x7fd2241031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 --2- 172.21.15.116:0/2947823956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd224100690 0x7fd224102c60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.840+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 >> 172.21.15.116:0/2947823956 conn(0x7fd224077820 msgr2=0x7fd224068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.841+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 shutdown_connections 2024-03-19T02:33:07.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:07.841+0000 7fd22916f640 1 -- 172.21.15.116:0/2947823956 wait complete. 2024-03-19T02:33:07.852 INFO:tasks.cephfs.filesystem:scrub status for tag:2da73709-35fc-4c1c-86b8-75f29d1d7b4a - {'path': '/', 'tag': '2da73709-35fc-4c1c-86b8-75f29d1d7b4a', 'options': 'recursive,force'} 2024-03-19T02:33:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:07 smithi162 ceph-mon[35761]: pgmap v162: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:07 smithi154 ceph-mon[35693]: pgmap v162: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:07 smithi116 ceph-mon[32263]: pgmap v162: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:08.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 164587 3.25 MB/sec execute 151 sec latency 20.583 ms 2024-03-19T02:33:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:33:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:33:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:33:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:09.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 165247 3.24 MB/sec execute 152 sec latency 20.665 ms 2024-03-19T02:33:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:09 smithi162 ceph-mon[35761]: pgmap v163: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:09 smithi154 ceph-mon[35693]: pgmap v163: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:09 smithi116 ceph-mon[32263]: pgmap v163: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 1.4 MiB/s wr, 99 op/s 2024-03-19T02:33:10.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 165926 3.25 MB/sec execute 153 sec latency 19.534 ms 2024-03-19T02:33:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:10 smithi162 ceph-mon[35761]: pgmap v164: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:33:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:10 smithi154 ceph-mon[35693]: pgmap v164: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:33:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:10 smithi116 ceph-mon[32263]: pgmap v164: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 2.1 MiB/s wr, 142 op/s 2024-03-19T02:33:11.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 166518 3.25 MB/sec execute 154 sec latency 16.918 ms 2024-03-19T02:33:12.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 167109 3.26 MB/sec execute 155 sec latency 19.243 ms 2024-03-19T02:33:13.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 167561 3.24 MB/sec execute 156 sec latency 43.506 ms 2024-03-19T02:33:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:13 smithi162 ceph-mon[35761]: pgmap v165: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-03-19T02:33:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:13 smithi154 ceph-mon[35693]: pgmap v165: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-03-19T02:33:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:13 smithi116 ceph-mon[32263]: pgmap v165: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.5 MiB/s wr, 100 op/s 2024-03-19T02:33:14.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 168225 3.25 MB/sec execute 157 sec latency 19.309 ms 2024-03-19T02:33:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:14 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:33:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:14 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:33:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:14 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:33:15.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 168832 3.23 MB/sec execute 158 sec latency 19.600 ms 2024-03-19T02:33:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:33:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:15 smithi162 ceph-mon[35761]: pgmap v166: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:33:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:33:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:15 smithi154 ceph-mon[35693]: pgmap v166: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:33:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:33:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:15 smithi116 ceph-mon[32263]: pgmap v166: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 140 op/s 2024-03-19T02:33:16.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 169503 3.25 MB/sec execute 159 sec latency 16.573 ms 2024-03-19T02:33:17.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 170079 3.25 MB/sec execute 160 sec latency 17.456 ms 2024-03-19T02:33:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:17 smithi162 ceph-mon[35761]: pgmap v167: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:17 smithi154 ceph-mon[35693]: pgmap v167: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:17 smithi116 ceph-mon[32263]: pgmap v167: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:18.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 170654 3.26 MB/sec execute 161 sec latency 17.098 ms 2024-03-19T02:33:19.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 171136 3.24 MB/sec execute 162 sec latency 35.085 ms 2024-03-19T02:33:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:19 smithi162 ceph-mon[35761]: pgmap v168: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:19 smithi154 ceph-mon[35693]: pgmap v168: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:19 smithi116 ceph-mon[32263]: pgmap v168: 129 pgs: 129 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:20.785 INFO:tasks.workunit.client.0.smithi116.stdout: 1 171772 3.25 MB/sec execute 163 sec latency 21.175 ms 2024-03-19T02:33:21.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 172449 3.23 MB/sec execute 164 sec latency 17.652 ms 2024-03-19T02:33:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:21 smithi162 ceph-mon[35761]: pgmap v169: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:21 smithi154 ceph-mon[35693]: pgmap v169: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:21 smithi116 ceph-mon[32263]: pgmap v169: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:22.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 173147 3.25 MB/sec execute 165 sec latency 60.014 ms 2024-03-19T02:33:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 173663 3.25 MB/sec execute 166 sec latency 16.793 ms 2024-03-19T02:33:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:23 smithi162 ceph-mon[35761]: pgmap v170: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:33:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:23 smithi154 ceph-mon[35693]: pgmap v170: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:33:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:23 smithi116 ceph-mon[32263]: pgmap v170: 129 pgs: 129 active+clean; 153 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:33:24.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 174230 3.25 MB/sec execute 167 sec latency 17.174 ms 2024-03-19T02:33:25.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 174735 3.24 MB/sec execute 168 sec latency 31.063 ms 2024-03-19T02:33:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:25 smithi162 ceph-mon[35761]: pgmap v171: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:25 smithi154 ceph-mon[35693]: pgmap v171: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:25 smithi116 ceph-mon[32263]: pgmap v171: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:26.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 175354 3.25 MB/sec execute 169 sec latency 19.328 ms 2024-03-19T02:33:27.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 176078 3.23 MB/sec execute 170 sec latency 21.898 ms 2024-03-19T02:33:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:27 smithi162 ceph-mon[35761]: pgmap v172: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:27 smithi154 ceph-mon[35693]: pgmap v172: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:27 smithi116 ceph-mon[32263]: pgmap v172: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:28.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 176801 3.25 MB/sec execute 171 sec latency 17.046 ms 2024-03-19T02:33:29.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 177362 3.26 MB/sec execute 172 sec latency 15.872 ms 2024-03-19T02:33:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:29 smithi162 ceph-mon[35761]: pgmap v173: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:29 smithi154 ceph-mon[35693]: pgmap v173: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:29 smithi116 ceph-mon[32263]: pgmap v173: 129 pgs: 129 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:33:30.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 177845 3.25 MB/sec execute 173 sec latency 14.961 ms 2024-03-19T02:33:31.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 178420 3.24 MB/sec execute 174 sec latency 42.582 ms 2024-03-19T02:33:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:31 smithi162 ceph-mon[35761]: pgmap v174: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:31 smithi154 ceph-mon[35693]: pgmap v174: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:31 smithi116 ceph-mon[32263]: pgmap v174: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:32.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 179061 3.25 MB/sec execute 175 sec latency 16.903 ms 2024-03-19T02:33:33.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 179729 3.24 MB/sec execute 176 sec latency 27.743 ms 2024-03-19T02:33:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:33 smithi162 ceph-mon[35761]: pgmap v175: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:33:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:33 smithi154 ceph-mon[35693]: pgmap v175: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:33:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:33 smithi116 ceph-mon[32263]: pgmap v175: 129 pgs: 129 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:33:34.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 180533 3.25 MB/sec execute 177 sec latency 17.554 ms 2024-03-19T02:33:35.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 180997 3.26 MB/sec execute 178 sec latency 16.907 ms 2024-03-19T02:33:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:35 smithi162 ceph-mon[35761]: pgmap v176: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:35 smithi154 ceph-mon[35693]: pgmap v176: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:35 smithi116 ceph-mon[32263]: pgmap v176: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:36.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 181436 3.25 MB/sec execute 179 sec latency 15.129 ms 2024-03-19T02:33:37.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 181984 3.24 MB/sec execute 180 sec latency 31.883 ms 2024-03-19T02:33:37.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:33:37.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 -- 172.21.15.116:0/2207486006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0102170 msgr2=0x7fdee010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 --2- 172.21.15.116:0/2207486006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0102170 0x7fdee010e680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fded8009cd0 tx=0x7fded802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 -- 172.21.15.116:0/2207486006 shutdown_connections 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 --2- 172.21.15.116:0/2207486006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee0102170 0x7fdee010e680 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 --2- 172.21.15.116:0/2207486006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee01017b0 0x7fdee0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 --2- 172.21.15.116:0/2207486006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 -- 172.21.15.116:0/2207486006 >> 172.21.15.116:0/2207486006 conn(0x7fdee00fd2c0 msgr2=0x7fdee00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 -- 172.21.15.116:0/2207486006 shutdown_connections 2024-03-19T02:33:37.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.965+0000 7fdee8012640 1 -- 172.21.15.116:0/2207486006 wait complete. 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 Processor -- start 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 -- start start 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0102170 0x7fdee0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdee0111a80 con 0x7fdee01017b0 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdee0111900 con 0x7fdee0102170 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee8012640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdee0111c00 con 0x7fdee01077c0 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43418/0 (socket says 172.21.15.116:43418) 2024-03-19T02:33:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0102170 0x7fdee0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 -- 172.21.15.116:0/3827322171 learned_addr learned my addr 172.21.15.116:0/3827322171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee65af640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 -- 172.21.15.116:0/3827322171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 msgr2=0x7fdee019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 -- 172.21.15.116:0/3827322171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0102170 msgr2=0x7fdee0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0102170 0x7fdee0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee5dae640 1 -- 172.21.15.116:0/3827322171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee019cfd0 con 0x7fdee01017b0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.966+0000 7fdee65af640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fdee5dae640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00e9f0 tx=0x7fdedc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fded6ffd640 1 -- 172.21.15.116:0/3827322171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00cd80 con 0x7fdee01017b0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fded6ffd640 1 -- 172.21.15.116:0/3827322171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdedc0054d0 con 0x7fdee01017b0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee019d2c0 con 0x7fdee01017b0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fded6ffd640 1 -- 172.21.15.116:0/3827322171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc010490 con 0x7fdee01017b0 2024-03-19T02:33:37.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.967+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee01a4b50 con 0x7fdee01017b0 2024-03-19T02:33:37.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.969+0000 7fded6ffd640 1 -- 172.21.15.116:0/3827322171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdedc005020 con 0x7fdee01017b0 2024-03-19T02:33:37.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.969+0000 7fdeeac01740 1 Processor -- start 2024-03-19T02:33:37.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.969+0000 7fdeeac01740 1 -- start start 2024-03-19T02:33:37.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.969+0000 7fded6ffd640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 0x7fdecc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c561fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c562b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee65af640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 0x7fdecc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650c5583500 con 0x5650c5628f30 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650c552bbe0 con 0x5650c5620350 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdeeac01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5650c557f5a0 con 0x5650c561fa30 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fded6ffd640 1 -- 172.21.15.116:0/3827322171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdedc0cf4c0 con 0x7fdee01017b0 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee65af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c562b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee65af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c562b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43430/0 (socket says 172.21.15.116:43430) 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee65af640 1 -- 172.21.15.116:0/3522871788 learned_addr learned my addr 172.21.15.116:0/3522871788 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:37.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee5dae640 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c561fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 -- 172.21.15.116:0/3522871788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 msgr2=0x5650c561fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c561fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 -- 172.21.15.116:0/3522871788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 msgr2=0x5650c562b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c562b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 -- 172.21.15.116:0/3522871788 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c554d000 con 0x5650c5620350 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee5dae640 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c561fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee65af640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 0x7fdecc0a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded8004af0 tx=0x7fded8032450 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdee55ad640 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56289f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fded0009bf0 tx=0x7fded0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af9de0d80db447c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.970+0000 7fdecbfff640 1 -- 172.21.15.116:0/3522871788 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0004050 con 0x5650c5620350 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdecbfff640 1 -- 172.21.15.116:0/3522871788 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded00054d0 con 0x5650c5620350 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdecbfff640 1 -- 172.21.15.116:0/3522871788 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0004d60 con 0x5650c5620350 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 -- 172.21.15.116:0/3522871788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 msgr2=0x5650c56289f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56289f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fded0009bf0 tx=0x7fded0011810 comp rx=0 tx=0).stop 2024-03-19T02:33:37.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 -- 172.21.15.116:0/3522871788 shutdown_connections 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c562b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56289f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 --2- 172.21.15.116:0/3522871788 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c561fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 -- 172.21.15.116:0/3522871788 >> 172.21.15.116:0/3522871788 conn(0x5650c561a160 msgr2=0x5650c561a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 -- 172.21.15.116:0/3522871788 shutdown_connections 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.971+0000 7fdeeac01740 1 -- 172.21.15.116:0/3522871788 wait complete. 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 -- start start 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 Processor -- start 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5625e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c56268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650c562bd20 con 0x5650c5620350 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650c553f160 con 0x5650c561fa30 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5650c562bea0 con 0x5650c5628f30 2024-03-19T02:33:37.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdeeac01740 10 client.0 _ll_register_callbacks cb 0x5650c557ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43434/0 (socket says 172.21.15.116:43434) 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdee55ad640 1 -- 172.21.15.116:0/1911047539 learned_addr learned my addr 172.21.15.116:0/1911047539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5625e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.972+0000 7fdee65af640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c56268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 -- 172.21.15.116:0/1911047539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 msgr2=0x5650c56268c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c56268c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 -- 172.21.15.116:0/1911047539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 msgr2=0x5650c5626380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 -- 172.21.15.116:0/1911047539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c53617d0 con 0x5650c561fa30 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee65af640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c56268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:37.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5625e40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0cc890 tx=0x7fdedc0053d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d835e76a17d21c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdec97fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0b4060 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded00054d0 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fdedc097e10 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdedc09b7a0 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdeeac01740 10 client.27131 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650c52973f0 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.973+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded00054d0 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdedc084ce0 con 0x5650c561fa30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 10 client.27131 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 10 client.27131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 20 client.27131 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c563b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 4 client.27131 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:37.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650c5590a40 con 0x5650c56390f0 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c563b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdec97fa640 10 client.27131 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.974+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c563b580 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a06ef49d2741be23 server_cookie=80a6da0a91d0b60b in_seq=0 out_seq=0 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdec97fa640 1 -- 172.21.15.116:0/1911047539 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650c5590a40 con 0x5650c56390f0 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdec97fa640 10 client.27131 handle_command_reply: tid=0 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 client.27131 shutdown 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 msgr2=0x5650c5625e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5625e40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0cc890 tx=0x7fdedc0053d0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 shutdown_connections 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c563b580 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c56268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5626380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdec97fa640 0 client.27131 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1911047539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5625e40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 >> 172.21.15.116:0/1911047539 conn(0x5650c561a160 msgr2=0x5650c561beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 shutdown_connections 2024-03-19T02:33:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 1 -- 172.21.15.116:0/1911047539 wait complete. 2024-03-19T02:33:37.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.975+0000 7fdeeac01740 20 client.27131 trim_cache size 0 max 16384 2024-03-19T02:33:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 Processor -- start 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 -- start start 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5628570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01aeb70 con 0x5650c5620350 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01b06b0 con 0x5650c561fa30 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdeeac01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01af240 con 0x5650c5628f30 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee65af640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43440/0 (socket says 172.21.15.116:43440) 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee65af640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47526/0 (socket says 172.21.15.116:47526) 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 -- 172.21.15.116:0/1986228429 learned_addr learned my addr 172.21.15.116:0/1986228429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5628570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 -- 172.21.15.116:0/1986228429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 msgr2=0x5650c5624fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 -- 172.21.15.116:0/1986228429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 msgr2=0x5650c5628570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5628570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee55ad640 1 -- 172.21.15.116:0/1986228429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c52570f0 con 0x5650c5620350 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.997+0000 7fdee65af640 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.998+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fded00049d0 tx=0x7fded0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614ad36669cff986 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.998+0000 7fdecb7fe640 1 -- 172.21.15.116:0/1986228429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded001e040 con 0x5650c5620350 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.998+0000 7fdecb7fe640 1 -- 172.21.15.116:0/1986228429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded002a070 con 0x5650c5620350 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.998+0000 7fdecb7fe640 1 -- 172.21.15.116:0/1986228429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded00038e0 con 0x5650c5620350 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- 172.21.15.116:0/1986228429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 msgr2=0x5650c5634ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fded00049d0 tx=0x7fded0005d30 comp rx=0 tx=0).stop 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- 172.21.15.116:0/1986228429 shutdown_connections 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c5628f30 0x5650c5624fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5620350 0x5650c5634ac0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1986228429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c561fa30 0x5650c5628570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- 172.21.15.116:0/1986228429 >> 172.21.15.116:0/1986228429 conn(0x5650c561a160 msgr2=0x5650c5625e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- 172.21.15.116:0/1986228429 shutdown_connections 2024-03-19T02:33:38.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- 172.21.15.116:0/1986228429 wait complete. 2024-03-19T02:33:38.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- start start 2024-03-19T02:33:38.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 Processor -- start 2024-03-19T02:33:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c56312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56226b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01affe0 con 0x5650c5628f30 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01af910 con 0x5650c5620350 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:37.999+0000 7fdeeac01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdee01acb50 con 0x5650c561fa30 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdeeac01740 10 client.0 _ll_register_callbacks cb 0x7fdedc00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43454/0 (socket says 172.21.15.116:43454) 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 -- 172.21.15.116:0/1580163864 learned_addr learned my addr 172.21.15.116:0/1580163864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c56312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee55ad640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56226b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 -- 172.21.15.116:0/1580163864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 msgr2=0x5650c56312e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c56312e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 -- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 msgr2=0x5650c56226b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56226b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 -- 172.21.15.116:0/1580163864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c5623420 con 0x5650c5628f30 2024-03-19T02:33:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c56312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdee65af640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fded802c630 tx=0x7fded8046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f53cf009127cb7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdec97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8039040 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fded8005440 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded002a070 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fded805e050 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdeeac01740 10 client.17322 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.000+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650c569f380 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded002a070 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fded803f050 con 0x5650c5628f30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 10 client.17322 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 20 client.17322 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c56a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 4 client.17322 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5650c558ad90 con 0x5650c56390f0 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c56a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdee5dae640 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c56a1b60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdfb295b10e7bc2 server_cookie=a927fda9dc600bd8 in_seq=0 out_seq=0 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.001+0000 7fdec97fa640 10 client.17322 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdec97fa640 1 -- 172.21.15.116:0/1580163864 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5650c558ad90 con 0x5650c56390f0 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdec97fa640 10 client.17322 handle_command_reply: tid=0 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 client.17322 shutdown 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 msgr2=0x5650c5622bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fded802c630 tx=0x7fded8046040 comp rx=0 tx=0).stop 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 shutdown_connections 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650c56390f0 0x5650c56a1b60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c5628f30 0x5650c5622bf0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650c5620350 0x5650c56226b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 --2- 172.21.15.116:0/1580163864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650c561fa30 0x5650c56312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdec97fa640 0 client.17322 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 >> 172.21.15.116:0/1580163864 conn(0x5650c561a160 msgr2=0x5650c5625e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 shutdown_connections 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 1 -- 172.21.15.116:0/1580163864 wait complete. 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdeeac01740 20 client.17322 trim_cache size 0 max 16384 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:33:38.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 msgr2=0x7fdecc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 0x7fdecc0a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fded8004af0 tx=0x7fded8032450 comp rx=0 tx=0).stop 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 msgr2=0x7fdee0197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00e9f0 tx=0x7fdedc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 shutdown_connections 2024-03-19T02:33:38.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdecc0a2970 0x7fdecc0a4e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdee01077c0 0x7fdee019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0102170 0x7fdee0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.003+0000 7fdee8012640 1 --2- 172.21.15.116:0/3827322171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdee01017b0 0x7fdee0197fe0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.004+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 >> 172.21.15.116:0/3827322171 conn(0x7fdee00fd2c0 msgr2=0x7fdee00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.004+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 shutdown_connections 2024-03-19T02:33:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.004+0000 7fdee8012640 1 -- 172.21.15.116:0/3827322171 wait complete. 2024-03-19T02:33:38.015 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:33:38.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:33:38.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.131+0000 7fcabebb3640 1 -- 172.21.15.116:0/2931760226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab810aae0 msgr2=0x7fcab8113230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.131+0000 7fcabebb3640 1 --2- 172.21.15.116:0/2931760226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab810aae0 0x7fcab8113230 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009900 tx=0x7fcaa402bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 -- 172.21.15.116:0/2931760226 shutdown_connections 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 --2- 172.21.15.116:0/2931760226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab8113770 0x7fcab8115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 --2- 172.21.15.116:0/2931760226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab810aae0 0x7fcab8113230 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 --2- 172.21.15.116:0/2931760226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab810a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 -- 172.21.15.116:0/2931760226 >> 172.21.15.116:0/2931760226 conn(0x7fcab806c910 msgr2=0x7fcab806cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 -- 172.21.15.116:0/2931760226 shutdown_connections 2024-03-19T02:33:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.132+0000 7fcabebb3640 1 -- 172.21.15.116:0/2931760226 wait complete. 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.133+0000 7fcabebb3640 1 Processor -- start 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.133+0000 7fcabebb3640 1 -- start start 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.133+0000 7fcabebb3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab81ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.133+0000 7fcabebb3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 0x7fcab81abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.133+0000 7fcabebb3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 0x7fcab81a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabebb3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcab8119150 con 0x7fcab8113770 2024-03-19T02:33:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabebb3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcab8118fd0 con 0x7fcab810a1c0 2024-03-19T02:33:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabd150640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 0x7fcab81a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabd150640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 0x7fcab81a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43478/0 (socket says 172.21.15.116:43478) 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabd150640 1 -- 172.21.15.116:0/3222207470 learned_addr learned my addr 172.21.15.116:0/3222207470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabebb3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcab81192d0 con 0x7fcab810aae0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 0x7fcab81abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabc94f640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab81ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 -- 172.21.15.116:0/3222207470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 msgr2=0x7fcab81ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab81ab990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 -- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 msgr2=0x7fcab81a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 0x7fcab81a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcab7fff640 1 -- 172.21.15.116:0/3222207470 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab81a75b0 con 0x7fcab810aae0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.134+0000 7fcabc94f640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab81ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcab7fff640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 0x7fcab81abed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4036040 tx=0x7fcaa40045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4043020 con 0x7fcab810aae0 2024-03-19T02:33:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab81a7840 con 0x7fcab810aae0 2024-03-19T02:33:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaa402cb70 con 0x7fcab810aae0 2024-03-19T02:33:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4003ae0 con 0x7fcab810aae0 2024-03-19T02:33:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.135+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab81b3900 con 0x7fcab810aae0 2024-03-19T02:33:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.137+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab8068610 con 0x7fcab810aae0 2024-03-19T02:33:38.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.139+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcaa4046020 con 0x7fcab810aae0 2024-03-19T02:33:38.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.140+0000 7fcab5ffb640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 0x7fcaa80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.140+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fcaa40f1830 con 0x7fcab810aae0 2024-03-19T02:33:38.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.140+0000 7fcabc94f640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 0x7fcaa80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.141+0000 7fcabc94f640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 0x7fcaa80a4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcab81a8420 tx=0x7fcaac005e70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.147+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcaa404d020 con 0x7fcab810aae0 2024-03-19T02:33:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:37 smithi162 ceph-mon[35761]: pgmap v177: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:37 smithi154 ceph-mon[35693]: pgmap v177: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:37 smithi116 ceph-mon[32263]: pgmap v177: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:38.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.394+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcab810d7e0 con 0x7fcab810aae0 2024-03-19T02:33:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.395+0000 7fcab5ffb640 1 -- 172.21.15.116:0/3222207470 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fcaa40be2c0 con 0x7fcab810aae0 2024-03-19T02:33:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:33:38.396 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:33:38.396 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 msgr2=0x7fcaa80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 0x7fcaa80a4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcab81a8420 tx=0x7fcaac005e70 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 msgr2=0x7fcab81abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 0x7fcab81abed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4036040 tx=0x7fcaa40045e0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 shutdown_connections 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcaa80a29f0 0x7fcaa80a4eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcab8113770 0x7fcab81a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcab810aae0 0x7fcab81abed0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 --2- 172.21.15.116:0/3222207470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcab810a1c0 0x7fcab81ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 >> 172.21.15.116:0/3222207470 conn(0x7fcab806c910 msgr2=0x7fcab8109310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 shutdown_connections 2024-03-19T02:33:38.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.397+0000 7fcabebb3640 1 -- 172.21.15.116:0/3222207470 wait complete. 2024-03-19T02:33:38.409 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:33:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 -- 172.21.15.116:0/3449837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 msgr2=0x7fa8f0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/3449837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0100b20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009930 tx=0x7fa8e402bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 -- 172.21.15.116:0/3449837494 shutdown_connections 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/3449837494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f0101060 0x7fa8f010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/3449837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0100b20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/3449837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f01066b0 0x7fa8f0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 -- 172.21.15.116:0/3449837494 >> 172.21.15.116:0/3449837494 conn(0x7fa8f00778d0 msgr2=0x7fa8f0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 -- 172.21.15.116:0/3449837494 shutdown_connections 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 -- 172.21.15.116:0/3449837494 wait complete. 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.523+0000 7fa8f50d1640 1 Processor -- start 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 -- start start 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f0101060 0x7fa8f01973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 0x7fa8f019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0110970 con 0x7fa8f01006a0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01107f0 con 0x7fa8f01066b0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8f50d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0110af0 con 0x7fa8f0101060 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43494/0 (socket says 172.21.15.116:43494) 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1804522309 learned_addr learned my addr 172.21.15.116:0/1804522309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f0101060 0x7fa8f01973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 0x7fa8f019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1804522309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f0101060 msgr2=0x7fa8f01973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f0101060 0x7fa8f01973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1804522309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 msgr2=0x7fa8f019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 0x7fa8f019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1804522309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f019be80 con 0x7fa8f01006a0 2024-03-19T02:33:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.524+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 0x7fa8f019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000e9f0 tx=0x7fa8e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8d7fff640 1 -- 172.21.15.116:0/1804522309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e000cd80 con 0x7fa8f01006a0 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f019c170 con 0x7fa8f01006a0 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8d7fff640 1 -- 172.21.15.116:0/1804522309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e00054d0 con 0x7fa8f01006a0 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f01a3a00 con 0x7fa8f01006a0 2024-03-19T02:33:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.525+0000 7fa8d7fff640 1 -- 172.21.15.116:0/1804522309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0021640 con 0x7fa8f01006a0 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.527+0000 7fa8d7fff640 1 -- 172.21.15.116:0/1804522309 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa8e0005020 con 0x7fa8f01006a0 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 Processor -- start 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 -- start start 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8d7fff640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 0x7fa8d80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5565c5df74b0 con 0x5565c5e9f9e0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5565c5d9fd70 con 0x5565c5e93da0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 0x7fa8d80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5565c5df3550 con 0x5565c5e94750 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8d7fff640 1 -- 172.21.15.116:0/1804522309 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa8e00cf460 con 0x7fa8f01006a0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43174/0 (socket says 172.21.15.116:43174) 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3571839444 learned_addr learned my addr 172.21.15.116:0/3571839444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.528+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3571839444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 msgr2=0x5565c5e9f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3571839444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 msgr2=0x5565c5ea1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3571839444 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c5dc2210 con 0x5565c5e93da0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 0x7fa8d80a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8005fd0 tx=0x7fa8e800feb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0097c40 tx=0x7fa8e0097bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=878440a7d952bd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8d57fa640 1 -- 172.21.15.116:0/3571839444 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00512e0 con 0x5565c5e93da0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8d57fa640 1 -- 172.21.15.116:0/3571839444 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e0005310 con 0x5565c5e93da0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.529+0000 7fa8d57fa640 1 -- 172.21.15.116:0/3571839444 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0004e60 con 0x5565c5e93da0 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3571839444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 msgr2=0x5565c5e94180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0097c40 tx=0x7fa8e0097bb0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3571839444 shutdown_connections 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3571839444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e94180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3571839444 >> 172.21.15.116:0/3571839444 conn(0x5565c5e8ea00 msgr2=0x5565c5e8ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3571839444 shutdown_connections 2024-03-19T02:33:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3571839444 wait complete. 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- start start 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 Processor -- start 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5565c5e95550 con 0x5565c5e9f9e0 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5565c5db3310 con 0x5565c5e93da0 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.530+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5565c5e956d0 con 0x5565c5e94750 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8f7d97740 10 client.0 _ll_register_callbacks cb 0x5565c5da72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8ef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8ef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43514/0 (socket says 172.21.15.116:43514) 2024-03-19T02:33:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8ef59e640 1 -- 172.21.15.116:0/3437884677 learned_addr learned my addr 172.21.15.116:0/3437884677 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 msgr2=0x5565c5e9b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3437884677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 msgr2=0x5565c5ea78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3437884677 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c5b0b3f0 con 0x5565c5e93da0 2024-03-19T02:33:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9b1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000e9c0 tx=0x7fa8e00048d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d412538b2e0fb133 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00b9060 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565c5e553e0 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa8e0099ae0 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa8e00b3030 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8f7d97740 10 client.27152 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.531+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5565c5bd57d0 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5565c5e553e0 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa8e00ccb50 con 0x5565c5e93da0 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 10 client.27152 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 10 client.27152 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:33:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 20 client.27152 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5ea3660 0x5565c5ea5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 4 client.27152 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565c5e049f0 con 0x5565c5ea3660 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5ea3660 0x5565c5ea5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8d2ff5640 10 client.27152 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.532+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5ea3660 0x5565c5ea5a70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ce8bff175806f9d server_cookie=741cc93acb4d1b67 in_seq=0 out_seq=0 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3437884677 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565c5e049f0 con 0x5565c5ea3660 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8d2ff5640 10 client.27152 handle_command_reply: tid=0 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 client.27152 shutdown 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 msgr2=0x5565c5e9b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9b1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000e9c0 tx=0x7fa8e00048d0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 shutdown_connections 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5ea3660 0x5565c5ea5a70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e9f9e0 0x5565c5ea78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e94750 0x5565c5e9b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3437884677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9b1b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8d2ff5640 0 client.27152 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 >> 172.21.15.116:0/3437884677 conn(0x5565c5e8ea00 msgr2=0x5565c5e90750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 shutdown_connections 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3437884677 wait complete. 2024-03-19T02:33:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.533+0000 7fa8f7d97740 20 client.27152 trim_cache size 0 max 16384 2024-03-19T02:33:38.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 Processor -- start 2024-03-19T02:33:38.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 -- start start 2024-03-19T02:33:38.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5e98240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01ac8d0 con 0x5565c5e94750 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01ae410 con 0x5565c5e93da0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01acfa0 con 0x5565c5e9f9e0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43516/0 (socket says 172.21.15.116:43516) 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8ee59c640 1 -- 172.21.15.116:0/1001615885 learned_addr learned my addr 172.21.15.116:0/1001615885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5e98240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1001615885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 msgr2=0x5565c5e98240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5e98240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1001615885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 msgr2=0x5565c5eb5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8eed9d640 1 -- 172.21.15.116:0/1001615885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c5ea2140 con 0x5565c5e93da0 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.550+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9f470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c6a0 tx=0x7fa8e009b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bee6c257c42a0672 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8d4ff9640 1 -- 172.21.15.116:0/1001615885 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00b9070 con 0x5565c5e93da0 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8d4ff9640 1 -- 172.21.15.116:0/1001615885 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e0004e60 con 0x5565c5e93da0 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8d4ff9640 1 -- 172.21.15.116:0/1001615885 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e009f930 con 0x5565c5e93da0 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 -- 172.21.15.116:0/1001615885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 msgr2=0x5565c5e9f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9f470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c6a0 tx=0x7fa8e009b640 comp rx=0 tx=0).stop 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 -- 172.21.15.116:0/1001615885 shutdown_connections 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5e98240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eb5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/1001615885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5e9f470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.551+0000 7fa8f7d97740 1 -- 172.21.15.116:0/1001615885 >> 172.21.15.116:0/1001615885 conn(0x5565c5e8ea00 msgr2=0x5565c5e906c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- 172.21.15.116:0/1001615885 shutdown_connections 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- 172.21.15.116:0/1001615885 wait complete. 2024-03-19T02:33:38.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- start start 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 Processor -- start 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5eae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5eaadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01add40 con 0x5565c5e94750 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01ad670 con 0x5565c5e93da0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01aa8b0 con 0x5565c5e9f9e0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8f7d97740 10 client.0 _ll_register_callbacks cb 0x7fa8e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8eed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5eae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43524/0 (socket says 172.21.15.116:43524) 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8ee59c640 1 -- 172.21.15.116:0/3652097560 learned_addr learned my addr 172.21.15.116:0/3652097560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.552+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5eaadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 msgr2=0x5565c5eaadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5eaadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3652097560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 msgr2=0x5565c5eaa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 -- 172.21.15.116:0/3652097560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c5eab640 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8ef59e640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5eaadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8eed9d640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5eae490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa8e009b4e0 tx=0x7fa8e00cf3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edbee3796798b3a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00b9070 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa8e0099ae0 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565c5dfed40 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa8e009bee0 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8f7d97740 10 client.27158 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5565c5eab640 con 0x5565c5e93da0 2024-03-19T02:33:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.553+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5565c5dfed40 con 0x5565c5e93da0 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa8e00a3af0 con 0x5565c5e93da0 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 10 client.27158 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 10 client.27158 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 20 client.27158 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5f13520 0x5565c5f15910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 4 client.27158 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565c5d7c6a0 con 0x5565c5f13520 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5f13520 0x5565c5f15910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8d2ff5640 10 client.27158 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8ee59c640 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5f13520 0x5565c5f15910 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de80f7ced01a038f server_cookie=631b0c1cfd060549 in_seq=0 out_seq=0 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8d2ff5640 1 -- 172.21.15.116:0/3652097560 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565c5d7c6a0 con 0x5565c5f13520 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8d2ff5640 10 client.27158 handle_command_reply: tid=0 2024-03-19T02:33:38.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 client.27158 shutdown 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 msgr2=0x5565c5eae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5eae490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa8e009b4e0 tx=0x7fa8e00cf3f0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 shutdown_connections 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5565c5f13520 0x5565c5f15910 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565c5e9f9e0 0x5565c5eaadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c5e94750 0x5565c5eaa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 --2- 172.21.15.116:0/3652097560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c5e93da0 0x5565c5eae490 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8d2ff5640 0 client.27158 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 >> 172.21.15.116:0/3652097560 conn(0x5565c5e8ea00 msgr2=0x5565c5e8fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 shutdown_connections 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.554+0000 7fa8f7d97740 1 -- 172.21.15.116:0/3652097560 wait complete. 2024-03-19T02:33:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f7d97740 20 client.27158 trim_cache size 0 max 16384 2024-03-19T02:33:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:38.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 msgr2=0x7fa8d80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 0x7fa8d80a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8005fd0 tx=0x7fa8e800feb0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 msgr2=0x7fa8f0196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000e9f0 tx=0x7fa8e000edf0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 shutdown_connections 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8d80a2970 0x7fa8d80a4e30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f01066b0 0x7fa8f019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8f0101060 0x7fa8f01973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 --2- 172.21.15.116:0/1804522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f01006a0 0x7fa8f0196e90 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 >> 172.21.15.116:0/1804522309 conn(0x7fa8f00778d0 msgr2=0x7fa8f0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 shutdown_connections 2024-03-19T02:33:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.555+0000 7fa8f50d1640 1 -- 172.21.15.116:0/1804522309 wait complete. 2024-03-19T02:33:38.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:33:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 -- 172.21.15.116:0/3330062152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c100070 msgr2=0x7fd86c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3330062152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c100070 0x7fd86c1004f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd854009860 tx=0x7fd85402bb70 comp rx=0 tx=0).stop 2024-03-19T02:33:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 -- 172.21.15.116:0/3330062152 shutdown_connections 2024-03-19T02:33:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3330062152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c10e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3330062152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c100070 0x7fd86c1004f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3330062152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c106810 0x7fd86c106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 -- 172.21.15.116:0/3330062152 >> 172.21.15.116:0/3330062152 conn(0x7fd86c075a20 msgr2=0x7fd86c075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.681+0000 7fd870ba9640 1 -- 172.21.15.116:0/3330062152 shutdown_connections 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 -- 172.21.15.116:0/3330062152 wait complete. 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 Processor -- start 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 -- start start 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c106810 0x7fd86c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c111aa0 con 0x7fd86c100070 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c111920 con 0x7fd86c106810 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.682+0000 7fd870ba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c111c20 con 0x7fd86c100a30 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43556/0 (socket says 172.21.15.116:43556) 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 -- 172.21.15.116:0/3293298115 learned_addr learned my addr 172.21.15.116:0/3293298115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd863fff640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c106810 0x7fd86c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 -- 172.21.15.116:0/3293298115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 msgr2=0x7fd86c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 -- 172.21.15.116:0/3293298115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c106810 msgr2=0x7fd86c19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c106810 0x7fd86c19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 -- 172.21.15.116:0/3293298115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c19cec0 con 0x7fd86c100070 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd863fff640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00e9a0 tx=0x7fd85c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd8637fe640 1 -- 172.21.15.116:0/3293298115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00cd80 con 0x7fd86c100070 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd8637fe640 1 -- 172.21.15.116:0/3293298115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd85c0108e0 con 0x7fd86c100070 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86c19d1b0 con 0x7fd86c100070 2024-03-19T02:33:38.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd8637fe640 1 -- 172.21.15.116:0/3293298115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c018950 con 0x7fd86c100070 2024-03-19T02:33:38.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.683+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86c1a4a40 con 0x7fd86c100070 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.685+0000 7fd8637fe640 1 -- 172.21.15.116:0/3293298115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd85c010a90 con 0x7fd86c100070 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 Processor -- start 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 -- start start 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd8637fe640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 0x7fd8580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fde290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 0x5569c8fe0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 0x7fd8580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5569c8f384b0 con 0x5569c8fde7d0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5569c8ee0d70 con 0x5569c8fd5bf0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd87378c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5569c8f34550 con 0x5569c8fd52d0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd8637fe640 1 -- 172.21.15.116:0/3293298115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd85c0cf550 con 0x7fd86c100070 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47640/0 (socket says 172.21.15.116:47640) 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd86a59c640 1 -- 172.21.15.116:0/1332038259 learned_addr learned my addr 172.21.15.116:0/1332038259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.686+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 0x5569c8fe0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd863fff640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fde290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 -- 172.21.15.116:0/1332038259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 msgr2=0x5569c8fde290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fde290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 -- 172.21.15.116:0/1332038259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 msgr2=0x5569c8fe0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 0x5569c8fe0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 -- 172.21.15.116:0/1332038259 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c8f03210 con 0x5569c8fd52d0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 0x5569c8fe0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 0x7fd8580a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd864005fd0 tx=0x7fd864013b60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c04a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1700b8318476ec0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd860ff9640 1 -- 172.21.15.116:0/1332038259 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c010e20 con 0x5569c8fd52d0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd860ff9640 1 -- 172.21.15.116:0/1332038259 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd85c09bf30 con 0x5569c8fd52d0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.687+0000 7fd860ff9640 1 -- 172.21.15.116:0/1332038259 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0ccba0 con 0x5569c8fd52d0 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- 172.21.15.116:0/1332038259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 msgr2=0x5569c8fd56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c04a430 comp rx=0 tx=0).stop 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- 172.21.15.116:0/1332038259 shutdown_connections 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fde7d0 0x5569c8fe0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fde290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 --2- 172.21.15.116:0/1332038259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fd52d0 0x5569c8fd56b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- 172.21.15.116:0/1332038259 >> 172.21.15.116:0/1332038259 conn(0x5569c8fcfa00 msgr2=0x5569c8fcfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- 172.21.15.116:0/1332038259 shutdown_connections 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- 172.21.15.116:0/1332038259 wait complete. 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 -- start start 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.688+0000 7fd87378c740 1 Processor -- start 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8fdc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5569c8fe1560 con 0x5569c8fd52d0 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5569c8ef4310 con 0x5569c8fd5bf0 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5569c8fe16e0 con 0x5569c8fde7d0 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 10 client.0 _ll_register_callbacks cb 0x5569c8ee82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43584/0 (socket says 172.21.15.116:43584) 2024-03-19T02:33:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 -- 172.21.15.116:0/2243984980 learned_addr learned my addr 172.21.15.116:0/2243984980 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd863fff640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8fdc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 -- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 msgr2=0x5569c8fdc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8fdc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 -- 172.21.15.116:0/2243984980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 msgr2=0x5569c8fdbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 -- 172.21.15.116:0/2243984980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c8c4c3f0 con 0x5569c8fd52d0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd86a59c640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0058c0 tx=0x7fd85c099b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239fdeb39b2ce23a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b4060 con 0x5569c8fd52d0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd85c0cf1b0 con 0x5569c8fd52d0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.689+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c09bf30 con 0x5569c8fd52d0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd85c0cf3b0 con 0x5569c8fd52d0 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 10 client.17358 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5569c8d167d0 con 0x5569c8fd52d0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd85c09bf30 con 0x5569c8fd52d0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd85c09cb60 con 0x5569c8fd52d0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 10 client.17358 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 10 client.17358 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 20 client.17358 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c8fee960 0x5569c8ff0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 4 client.17358 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569c8f963e0 con 0x5569c8fee960 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.690+0000 7fd863fff640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c8fee960 0x5569c8ff0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.691+0000 7fd851ffb640 10 client.17358 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.691+0000 7fd863fff640 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c8fee960 0x5569c8ff0d50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3e08c5166b909c2 server_cookie=23306013ed28332 in_seq=0 out_seq=0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.691+0000 7fd851ffb640 1 -- 172.21.15.116:0/2243984980 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569c8f963e0 con 0x5569c8fee960 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.691+0000 7fd851ffb640 10 client.17358 handle_command_reply: tid=0 2024-03-19T02:33:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 client.17358 shutdown 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 msgr2=0x5569c8fdb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0058c0 tx=0x7fd85c099b30 comp rx=0 tx=0).stop 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 shutdown_connections 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8fdc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c8fee960 0x5569c8ff0d50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 --2- 172.21.15.116:0/2243984980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8fdb7e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd851ffb640 0 client.17358 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 >> 172.21.15.116:0/2243984980 conn(0x5569c8fcfa00 msgr2=0x5569c8fd1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 shutdown_connections 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 1 -- 172.21.15.116:0/2243984980 wait complete. 2024-03-19T02:33:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.692+0000 7fd87378c740 20 client.17358 trim_cache size 0 max 16384 2024-03-19T02:33:38.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 Processor -- start 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 -- start start 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 0x5569c8fddf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1aea50 con 0x5569c8fd5bf0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1b0590 con 0x5569c8fd52d0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd87378c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1af120 con 0x5569c8fde7d0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd863fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd863fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43598/0 (socket says 172.21.15.116:43598) 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd86ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd863fff640 1 -- 172.21.15.116:0/3225642097 learned_addr learned my addr 172.21.15.116:0/3225642097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.707+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 0x5569c8fddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 -- 172.21.15.116:0/3225642097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 msgr2=0x5569c8ff3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 -- 172.21.15.116:0/3225642097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 msgr2=0x5569c8fdc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 -- 172.21.15.116:0/3225642097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c8fc9fb0 con 0x5569c8fd52d0 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd86a59c640 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 0x5569c8fddf10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c04a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a18f80172222a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd853fff640 1 -- 172.21.15.116:0/3225642097 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b4050 con 0x5569c8fd52d0 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd853fff640 1 -- 172.21.15.116:0/3225642097 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd85c09f980 con 0x5569c8fd52d0 2024-03-19T02:33:38.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.708+0000 7fd853fff640 1 -- 172.21.15.116:0/3225642097 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c09cb60 con 0x5569c8fd52d0 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- 172.21.15.116:0/3225642097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 msgr2=0x5569c8fddf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 0x5569c8fddf10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c04a430 comp rx=0 tx=0).stop 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- 172.21.15.116:0/3225642097 shutdown_connections 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd5bf0 0x5569c8fdc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 --2- 172.21.15.116:0/3225642097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd52d0 0x5569c8fddf10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- 172.21.15.116:0/3225642097 >> 172.21.15.116:0/3225642097 conn(0x5569c8fcfa00 msgr2=0x5569c8fce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- 172.21.15.116:0/3225642097 shutdown_connections 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- 172.21.15.116:0/3225642097 wait complete. 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.709+0000 7fd87378c740 1 -- start start 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 Processor -- start 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1afec0 con 0x5569c8fd52d0 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1af7f0 con 0x5569c8fd5bf0 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1aca30 con 0x5569c8fde7d0 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd87378c740 10 client.0 _ll_register_callbacks cb 0x7fd85c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd863fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd863fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43280/0 (socket says 172.21.15.116:43280) 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd863fff640 1 -- 172.21.15.116:0/1367836448 learned_addr learned my addr 172.21.15.116:0/1367836448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd86a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43608/0 (socket says 172.21.15.116:43608) 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.710+0000 7fd86ad9d640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 -- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 msgr2=0x5569c8ff18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 -- 172.21.15.116:0/1367836448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 msgr2=0x5569c8ff1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 -- 172.21.15.116:0/1367836448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c8ff20b0 con 0x5569c8fd52d0 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd863fff640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd86a59c640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c09cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6118d78c477c9e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:38.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0b4070 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fd85c09aee0 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c09f980 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd85c09bf30 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd87378c740 10 client.17364 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.711+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5569c8ff20b0 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd85c09f980 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd85c09a6f0 con 0x5569c8fd52d0 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 10 client.17364 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 10 client.17364 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 20 client.17364 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c905d430 0x5569c905f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 4 client.17364 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5569c8f459f0 con 0x5569c905d430 2024-03-19T02:33:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd863fff640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c905d430 0x5569c905f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd863fff640 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c905d430 0x5569c905f820 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=322119976daaa303 server_cookie=64a9952060562abf in_seq=0 out_seq=0 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.712+0000 7fd851ffb640 10 client.17364 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd851ffb640 1 -- 172.21.15.116:0/1367836448 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5569c8f459f0 con 0x5569c905d430 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd851ffb640 10 client.17364 handle_command_reply: tid=0 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 client.17364 shutdown 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 msgr2=0x5569c8ff3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd85c098200 tx=0x7fd85c09cd90 comp rx=0 tx=0).stop 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 shutdown_connections 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569c8fde7d0 0x5569c8ff18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5569c8fd5bf0 0x5569c8ff1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5569c905d430 0x5569c905f820 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 --2- 172.21.15.116:0/1367836448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569c8fd52d0 0x5569c8ff3fe0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 >> 172.21.15.116:0/1367836448 conn(0x5569c8fcfa00 msgr2=0x5569c8fce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd851ffb640 0 client.17364 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:33:38.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 shutdown_connections 2024-03-19T02:33:38.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 1 -- 172.21.15.116:0/1367836448 wait complete. 2024-03-19T02:33:38.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd87378c740 20 client.17364 trim_cache size 0 max 16384 2024-03-19T02:33:38.719 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:38.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 msgr2=0x7fd8580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 0x7fd8580a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd864005fd0 tx=0x7fd864013b60 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 msgr2=0x7fd86c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.713+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00e9a0 tx=0x7fd85c00ee60 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 shutdown_connections 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8580a2970 0x7fd8580a4e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86c106810 0x7fd86c19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd86c100a30 0x7fd86c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 --2- 172.21.15.116:0/3293298115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c100070 0x7fd86c197f20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 >> 172.21.15.116:0/3293298115 conn(0x7fd86c075a20 msgr2=0x7fd86c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 shutdown_connections 2024-03-19T02:33:38.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.714+0000 7fd870ba9640 1 -- 172.21.15.116:0/3293298115 wait complete. 2024-03-19T02:33:38.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:33:38.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 182656 3.24 MB/sec execute 181 sec latency 22.385 ms 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 -- 172.21.15.116:0/3427847957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c1066b0 msgr2=0x7fa81c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 --2- 172.21.15.116:0/3427847957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c1066b0 0x7fa81c106a90 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009930 tx=0x7fa80c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 -- 172.21.15.116:0/3427847957 shutdown_connections 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 --2- 172.21.15.116:0/3427847957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c101060 0x7fa81c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 --2- 172.21.15.116:0/3427847957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 --2- 172.21.15.116:0/3427847957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c1066b0 0x7fa81c106a90 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 -- 172.21.15.116:0/3427847957 >> 172.21.15.116:0/3427847957 conn(0x7fa81c0778d0 msgr2=0x7fa81c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 -- 172.21.15.116:0/3427847957 shutdown_connections 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.839+0000 7fa821b3b640 1 -- 172.21.15.116:0/3427847957 wait complete. 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 Processor -- start 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 -- start start 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 0x7fa81c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 0x7fa81c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c110930 con 0x7fa81c101060 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1107b0 con 0x7fa81c1006a0 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa821b3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c110ab0 con 0x7fa81c1066b0 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa81affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 0x7fa81c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43300/0 (socket says 172.21.15.116:43300) 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 0x7fa81c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.840+0000 7fa81b7fe640 1 -- 172.21.15.116:0/1016534496 learned_addr learned my addr 172.21.15.116:0/1016534496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 -- 172.21.15.116:0/1016534496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 msgr2=0x7fa81c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 0x7fa81c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 -- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 msgr2=0x7fa81c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 0x7fa81c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 -- 172.21.15.116:0/1016534496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c19cfa0 con 0x7fa81c1006a0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81affd640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 0x7fa81c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81bfff640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 0x7fa81c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009230 tx=0x7fa80c02c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa818ff9640 1 -- 172.21.15.116:0/1016534496 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c043020 con 0x7fa81c1006a0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c19d230 con 0x7fa81c1006a0 2024-03-19T02:33:38.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa818ff9640 1 -- 172.21.15.116:0/1016534496 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa80c02ca70 con 0x7fa81c1006a0 2024-03-19T02:33:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1a4b20 con 0x7fa81c1006a0 2024-03-19T02:33:38.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.841+0000 7fa818ff9640 1 -- 172.21.15.116:0/1016534496 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c003a50 con 0x7fa81c1006a0 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.843+0000 7fa818ff9640 1 -- 172.21.15.116:0/1016534496 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa80c046020 con 0x7fa81c1006a0 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 Processor -- start 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 -- start start 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714baac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649714194b0 con 0x5649714c3be0 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5649713c1d70 con 0x5649714bb000 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa824807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564971415550 con 0x5649714ba6e0 2024-03-19T02:33:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714baac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43622/0 (socket says 172.21.15.116:43622) 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81affd640 1 -- 172.21.15.116:0/3012141102 learned_addr learned my addr 172.21.15.116:0/3012141102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa818ff9640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 0x7fa8040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81bfff640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 0x7fa8040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 -- 172.21.15.116:0/3012141102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 msgr2=0x5649714baac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714baac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 -- 172.21.15.116:0/3012141102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 msgr2=0x5649714c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81b7fe640 1 -- 172.21.15.116:0/3012141102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649713e4210 con 0x5649714bb000 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81affd640 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa81bfff640 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714baac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.844+0000 7fa818ff9640 1 -- 172.21.15.116:0/1016534496 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa80c0f1090 con 0x7fa81c1006a0 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714c36a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0ee570 tx=0x7fa80c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f02e5ea92a7b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa81bfff640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 0x7fa8040a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8140094d0 tx=0x7fa814004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa801ffb640 1 -- 172.21.15.116:0/3012141102 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c0bdb80 con 0x5649714bb000 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa801ffb640 1 -- 172.21.15.116:0/3012141102 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa80c0c15d0 con 0x5649714bb000 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa801ffb640 1 -- 172.21.15.116:0/3012141102 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c0bb810 con 0x5649714bb000 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa824807740 1 -- 172.21.15.116:0/3012141102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 msgr2=0x5649714c36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.845+0000 7fa824807740 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714c36a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0ee570 tx=0x7fa80c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- 172.21.15.116:0/3012141102 shutdown_connections 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714c36a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- 172.21.15.116:0/3012141102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714baac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- 172.21.15.116:0/3012141102 >> 172.21.15.116:0/3012141102 conn(0x5649714b4e10 msgr2=0x5649714b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- 172.21.15.116:0/3012141102 shutdown_connections 2024-03-19T02:33:38.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- 172.21.15.116:0/3012141102 wait complete. 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- start start 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 Processor -- start 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.846+0000 7fa824807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649714c6790 con 0x5649714c3be0 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5649713d5310 con 0x5649714ba6e0 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649714c6910 con 0x5649714bb000 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 10 client.0 _ll_register_callbacks cb 0x5649713c92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43322/0 (socket says 172.21.15.116:43322) 2024-03-19T02:33:38.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 -- 172.21.15.116:0/730102601 learned_addr learned my addr 172.21.15.116:0/730102601 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81bfff640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 -- 172.21.15.116:0/730102601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 msgr2=0x5649714c11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 -- 172.21.15.116:0/730102601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 msgr2=0x5649714c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 -- 172.21.15.116:0/730102601 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56497112d3f0 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0f3040 tx=0x7fa80c0bcf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b40098ac564f2d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c0bd480 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c0c15d0 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa80c076d50 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa80c0bdb80 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 10 client.27188 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.847+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5649711f77d0 con 0x5649714ba6e0 2024-03-19T02:33:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa80c0c15d0 con 0x5649714ba6e0 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa80c0c5790 con 0x5649714ba6e0 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 10 client.27188 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 10 client.27188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 20 client.27188 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5649714d3b30 0x5649714d5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 4 client.27188 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649714773e0 con 0x5649714d3b30 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa81affd640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5649714d3b30 0x5649714d5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa81affd640 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5649714d3b30 0x5649714d5f20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=144e0a5dcddd6fa7 server_cookie=4fc87e05a8729686 in_seq=0 out_seq=0 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.848+0000 7fa7faffd640 10 client.27188 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa7faffd640 1 -- 172.21.15.116:0/730102601 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649714773e0 con 0x5649714d3b30 2024-03-19T02:33:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa7faffd640 10 client.27188 handle_command_reply: tid=0 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 client.27188 shutdown 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 msgr2=0x5649714c0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0f3040 tx=0x7fa80c0bcf10 comp rx=0 tx=0).stop 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 shutdown_connections 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5649714d3b30 0x5649714d5f20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 --2- 172.21.15.116:0/730102601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c0c90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa7faffd640 0 client.27188 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 >> 172.21.15.116:0/730102601 conn(0x5649714b4e10 msgr2=0x5649714b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 shutdown_connections 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 1 -- 172.21.15.116:0/730102601 wait complete. 2024-03-19T02:33:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.849+0000 7fa824807740 20 client.27188 trim_cache size 0 max 16384 2024-03-19T02:33:38.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 Processor -- start 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 -- start start 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ac970 con 0x5649714c3be0 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ae4b0 con 0x5649714bb000 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa824807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ad040 con 0x5649714ba6e0 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43652/0 (socket says 172.21.15.116:43652) 2024-03-19T02:33:38.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47710/0 (socket says 172.21.15.116:47710) 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81bfff640 1 -- 172.21.15.116:0/1042737066 learned_addr learned my addr 172.21.15.116:0/1042737066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.864+0000 7fa81affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43334/0 (socket says 172.21.15.116:43334) 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 -- 172.21.15.116:0/1042737066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 msgr2=0x5649714c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 -- 172.21.15.116:0/1042737066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 msgr2=0x5649714cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 -- 172.21.15.116:0/1042737066 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649714bae30 con 0x5649714bb000 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa81affd640 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x5649714d66a0 tx=0x7fa810005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771ce06b3531a400 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa8017fa640 1 -- 172.21.15.116:0/1042737066 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810024020 con 0x5649714bb000 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa8017fa640 1 -- 172.21.15.116:0/1042737066 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8100201b0 con 0x5649714bb000 2024-03-19T02:33:38.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.865+0000 7fa8017fa640 1 -- 172.21.15.116:0/1042737066 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810004a00 con 0x5649714bb000 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- 172.21.15.116:0/1042737066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 msgr2=0x5649714afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x5649714d66a0 tx=0x7fa810005f90 comp rx=0 tx=0).stop 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- 172.21.15.116:0/1042737066 shutdown_connections 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714bb000 0x5649714afd10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- 172.21.15.116:0/1042737066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714ba6e0 0x5649714c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- 172.21.15.116:0/1042737066 >> 172.21.15.116:0/1042737066 conn(0x5649714b4e10 msgr2=0x5649714c14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- 172.21.15.116:0/1042737066 shutdown_connections 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- 172.21.15.116:0/1042737066 wait complete. 2024-03-19T02:33:38.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 -- start start 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 Processor -- start 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.866+0000 7fa824807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1adde0 con 0x5649714c3be0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ad710 con 0x5649714ba6e0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1aa950 con 0x5649714bb000 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 10 client.0 _ll_register_callbacks cb 0x7fa810009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43664/0 (socket says 172.21.15.116:43664) 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 -- 172.21.15.116:0/827970470 learned_addr learned my addr 172.21.15.116:0/827970470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:38.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 -- 172.21.15.116:0/827970470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 msgr2=0x5649714c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 -- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 msgr2=0x5649714c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 -- 172.21.15.116:0/827970470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564971538de0 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa81bfff640 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa8140039e0 tx=0x7fa814004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6b4dc08bce57e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814004820 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8100201b0 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7fa814004b20 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa814034050 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 10 client.17382 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.867+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56497153a170 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8100201b0 con 0x5649714c3be0 2024-03-19T02:33:38.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa81401f040 con 0x5649714c3be0 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 10 client.17382 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 10 client.17382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 20 client.17382 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56497153a920 0x56497153cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 4 client.17382 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5649714269f0 con 0x56497153a920 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56497153a920 0x56497153cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa81b7fe640 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56497153a920 0x56497153cd40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7b675e9d91851cf server_cookie=b945393cfa4f8ddd in_seq=0 out_seq=0 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.868+0000 7fa7faffd640 10 client.17382 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa7faffd640 1 -- 172.21.15.116:0/827970470 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5649714269f0 con 0x56497153a920 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa7faffd640 10 client.17382 handle_command_reply: tid=0 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 client.17382 shutdown 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 msgr2=0x5649714c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa8140039e0 tx=0x7fa814004a40 comp rx=0 tx=0).stop 2024-03-19T02:33:38.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 shutdown_connections 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56497153a920 0x56497153cd40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649714c3be0 0x5649714c7eb0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649714bb000 0x5649714c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 --2- 172.21.15.116:0/827970470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5649714ba6e0 0x5649714c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa7faffd640 0 client.17382 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 >> 172.21.15.116:0/827970470 conn(0x5649714b4e10 msgr2=0x5649714bf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 shutdown_connections 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 1 -- 172.21.15.116:0/827970470 wait complete. 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa824807740 20 client.17382 trim_cache size 0 max 16384 2024-03-19T02:33:38.876 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 msgr2=0x7fa8040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 0x7fa8040a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8140094d0 tx=0x7fa814004ab0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 msgr2=0x7fa81c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.869+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009230 tx=0x7fa80c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 shutdown_connections 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa8040a2970 0x7fa8040a4e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa81c1066b0 0x7fa81c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa81c101060 0x7fa81c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 --2- 172.21.15.116:0/1016534496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa81c1006a0 0x7fa81c198000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 >> 172.21.15.116:0/1016534496 conn(0x7fa81c0778d0 msgr2=0x7fa81c068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 shutdown_connections 2024-03-19T02:33:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.870+0000 7fa821b3b640 1 -- 172.21.15.116:0/1016534496 wait complete. 2024-03-19T02:33:38.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 -- 172.21.15.116:0/1320058002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c1006a0 msgr2=0x7f3a7c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 --2- 172.21.15.116:0/1320058002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c100b20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009930 tx=0x7f3a7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 -- 172.21.15.116:0/1320058002 shutdown_connections 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 --2- 172.21.15.116:0/1320058002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c101060 0x7f3a7c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 --2- 172.21.15.116:0/1320058002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c100b20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 --2- 172.21.15.116:0/1320058002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 -- 172.21.15.116:0/1320058002 >> 172.21.15.116:0/1320058002 conn(0x7f3a7c0778d0 msgr2=0x7f3a7c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 -- 172.21.15.116:0/1320058002 shutdown_connections 2024-03-19T02:33:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:38.999+0000 7f3a824b2640 1 -- 172.21.15.116:0/1320058002 wait complete. 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 Processor -- start 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 -- start start 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c110970 con 0x7f3a7c101060 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1107f0 con 0x7f3a7c1006a0 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a824b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c110af0 con 0x7f3a7c1066b0 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a7b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a7b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43672/0 (socket says 172.21.15.116:43672) 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a7b7fe640 1 -- 172.21.15.116:0/2127860220 learned_addr learned my addr 172.21.15.116:0/2127860220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.000+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 -- 172.21.15.116:0/2127860220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 msgr2=0x7f3a7c103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c103740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 -- 172.21.15.116:0/2127860220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 msgr2=0x7f3a7c102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 -- 172.21.15.116:0/2127860220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1a64e0 con 0x7f3a7c101060 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009230 tx=0x7f3a7002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a797fa640 1 -- 172.21.15.116:0/2127860220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70043020 con 0x7f3a7c101060 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a797fa640 1 -- 172.21.15.116:0/2127860220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a7002ca70 con 0x7f3a7c101060 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1a6770 con 0x7f3a7c101060 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a797fa640 1 -- 172.21.15.116:0/2127860220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70003a50 con 0x7f3a7c101060 2024-03-19T02:33:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.001+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1a6cc0 con 0x7f3a7c101060 2024-03-19T02:33:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.003+0000 7f3a797fa640 1 -- 172.21.15.116:0/2127860220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3a70046020 con 0x7f3a7c101060 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 Processor -- start 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 -- start start 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a797fa640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 0x7f3a640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297cca0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cca2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56297cbfa4b0 con 0x56297cc972d0 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56297cba2d70 con 0x56297cca07d0 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a8517d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56297cbf6550 con 0x56297cc97bf0 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 0x7f3a640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43682/0 (socket says 172.21.15.116:43682) 2024-03-19T02:33:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 -- 172.21.15.116:0/2356610633 learned_addr learned my addr 172.21.15.116:0/2356610633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a797fa640 1 -- 172.21.15.116:0/2127860220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3a700f10b0 con 0x7f3a7c101060 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297cca0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cca2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 -- 172.21.15.116:0/2356610633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 msgr2=0x56297cca0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297cca0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 -- 172.21.15.116:0/2356610633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 msgr2=0x56297cca2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cca2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7bfff640 1 -- 172.21.15.116:0/2356610633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56297cbc5210 con 0x56297cc972d0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297cca0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.004+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 0x7f3a640a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a740099e0 tx=0x7f3a7400ff60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0114f0 tx=0x7f3a6c0118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fb053d1c3e702c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a627fc640 1 -- 172.21.15.116:0/2356610633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01e070 con 0x56297cc972d0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a627fc640 1 -- 172.21.15.116:0/2356610633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a6c00f930 con 0x56297cc972d0 2024-03-19T02:33:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a627fc640 1 -- 172.21.15.116:0/2356610633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c019960 con 0x56297cc972d0 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a8517d740 1 -- 172.21.15.116:0/2356610633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 msgr2=0x56297cc976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.005+0000 7f3a8517d740 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0114f0 tx=0x7f3a6c0118f0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 -- 172.21.15.116:0/2356610633 shutdown_connections 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cca2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297cca0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 --2- 172.21.15.116:0/2356610633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc976b0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 -- 172.21.15.116:0/2356610633 >> 172.21.15.116:0/2356610633 conn(0x56297cc91a00 msgr2=0x56297cc91de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 -- 172.21.15.116:0/2356610633 shutdown_connections 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 -- 172.21.15.116:0/2356610633 wait complete. 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 -- start start 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 Processor -- start 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.006+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cc9dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 0x56297cc9e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56297cca35c0 con 0x56297cca07d0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56297cbb6310 con 0x56297cc97bf0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56297cca3740 con 0x56297cc972d0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 10 client.0 _ll_register_callbacks cb 0x56297cbaa2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a80a4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 0x56297cc9e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cc9dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47754/0 (socket says 172.21.15.116:47754) 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 -- 172.21.15.116:0/648811448 learned_addr learned my addr 172.21.15.116:0/648811448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 -- 172.21.15.116:0/648811448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 msgr2=0x56297cc9dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cc9dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 -- 172.21.15.116:0/648811448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 msgr2=0x56297cc9e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 0x56297cc9e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 -- 172.21.15.116:0/648811448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56297c90e3f0 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 0x56297cc9e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009700 tx=0x7f3a6c009d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c06d0c5f6fb5d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01e040 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c00f930 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.007+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3a6c00fbe0 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c02f050 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 10 client.27151 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56297c9d87d0 con 0x56297cc972d0 2024-03-19T02:33:39.013 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:38 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3222207470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a6c00f930 con 0x56297cc972d0 2024-03-19T02:33:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6c02a030 con 0x56297cc972d0 2024-03-19T02:33:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 10 client.27151 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 10 client.27151 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:33:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 20 client.27151 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297ccb0aa0 0x56297ccb2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 4 client.27151 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56297cc583e0 con 0x56297ccb0aa0 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.008+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297ccb0aa0 0x56297ccb2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.009+0000 7f3a5b7fe640 10 client.27151 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.009+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297ccb0aa0 0x56297ccb2e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f02ff917fa7c335 server_cookie=324c8c0335c6b8c2 in_seq=0 out_seq=0 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/648811448 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56297cc583e0 con 0x56297ccb0aa0 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a5b7fe640 10 client.27151 handle_command_reply: tid=0 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 client.27151 shutdown 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 msgr2=0x56297cc9d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009700 tx=0x7f3a6c009d40 comp rx=0 tx=0).stop 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 shutdown_connections 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297ccb0aa0 0x56297ccb2e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cca07d0 0x56297cc9e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cc9dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 --2- 172.21.15.116:0/648811448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc972d0 0x56297cc9d7f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 >> 172.21.15.116:0/648811448 conn(0x56297cc91a00 msgr2=0x56297cc93750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a5b7fe640 0 client.27151 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 shutdown_connections 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.010+0000 7f3a8517d740 1 -- 172.21.15.116:0/648811448 wait complete. 2024-03-19T02:33:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.011+0000 7f3a8517d740 20 client.27151 trim_cache size 0 max 16384 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 Processor -- start 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 -- start start 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297ccac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cc9e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b0d50 con 0x56297cc972d0 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b2890 con 0x56297cca07d0 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.025+0000 7f3a8517d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b1420 con 0x56297cc97bf0 2024-03-19T02:33:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43690/0 (socket says 172.21.15.116:43690) 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 -- 172.21.15.116:0/879218790 learned_addr learned my addr 172.21.15.116:0/879218790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297ccac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cc9e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 -- 172.21.15.116:0/879218790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 msgr2=0x56297ccac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297ccac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 -- 172.21.15.116:0/879218790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 msgr2=0x56297cc9e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cc9e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 -- 172.21.15.116:0/879218790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56297cc8d2c0 con 0x56297cc972d0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297ccac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0058c0 tx=0x7f3a6c009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380c3e8bf78c7542 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a61ffb640 1 -- 172.21.15.116:0/879218790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01e070 con 0x56297cc972d0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a61ffb640 1 -- 172.21.15.116:0/879218790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a6c019b20 con 0x56297cc972d0 2024-03-19T02:33:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.026+0000 7f3a61ffb640 1 -- 172.21.15.116:0/879218790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c003850 con 0x56297cc972d0 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- 172.21.15.116:0/879218790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 msgr2=0x56297cc9ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0058c0 tx=0x7f3a6c009ee0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- 172.21.15.116:0/879218790 shutdown_connections 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cca07d0 0x56297cc9e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cc97bf0 0x56297ccac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 --2- 172.21.15.116:0/879218790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cc9ff20 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- 172.21.15.116:0/879218790 >> 172.21.15.116:0/879218790 conn(0x56297cc91a00 msgr2=0x56297cc9c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- 172.21.15.116:0/879218790 shutdown_connections 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- 172.21.15.116:0/879218790 wait complete. 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.027+0000 7f3a8517d740 1 -- start start 2024-03-19T02:33:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 Processor -- start 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cca40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cca07d0 0x56297cd14cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b21c0 con 0x56297cc972d0 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b1af0 con 0x56297cc97bf0 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1aed30 con 0x56297cca07d0 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a8517d740 10 client.0 _ll_register_callbacks cb 0x7f3a6c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43696/0 (socket says 172.21.15.116:43696) 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 -- 172.21.15.116:0/1640242486 learned_addr learned my addr 172.21.15.116:0/1640242486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cca40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a80a4f640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cca07d0 0x56297cd14cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 -- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cca07d0 msgr2=0x56297cd14cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cca07d0 0x56297cd14cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 -- 172.21.15.116:0/1640242486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 msgr2=0x56297cca40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cca40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.028+0000 7f3a7bfff640 1 -- 172.21.15.116:0/1640242486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56297cd15200 con 0x56297cc972d0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a7bfff640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c019730 tx=0x7f3a6c00fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26689e3523338504 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01e040 con 0x56297cc972d0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f3a6c0050d0 con 0x56297cc972d0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c019b20 con 0x56297cc972d0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c044050 con 0x56297cc972d0 2024-03-19T02:33:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 10 client.17400 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56297cd164e0 con 0x56297cc972d0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a6c019b20 con 0x56297cc972d0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3a6c02a040 con 0x56297cc972d0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 10 client.17400 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 20 client.17400 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297cd16b50 0x56297cd18f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 4 client.17400 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.029+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56297cc079f0 con 0x56297cd16b50 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297cd16b50 0x56297cd18f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a5b7fe640 10 client.17400 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a7b7fe640 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297cd16b50 0x56297cd18f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea51705c680d92ba server_cookie=dd2f5c2f2be2333 in_seq=0 out_seq=0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a5b7fe640 1 -- 172.21.15.116:0/1640242486 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56297cc079f0 con 0x56297cd16b50 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a5b7fe640 10 client.17400 handle_command_reply: tid=0 2024-03-19T02:33:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 client.17400 shutdown 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 msgr2=0x56297cca3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c019730 tx=0x7f3a6c00fa10 comp rx=0 tx=0).stop 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 shutdown_connections 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56297cd16b50 0x56297cd18f70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56297cca07d0 0x56297cd14cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56297cc97bf0 0x56297cca40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a5b7fe640 0 client.17400 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 --2- 172.21.15.116:0/1640242486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56297cc972d0 0x56297cca3b70 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.030+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 >> 172.21.15.116:0/1640242486 conn(0x56297cc91a00 msgr2=0x56297cc9bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 shutdown_connections 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a8517d740 1 -- 172.21.15.116:0/1640242486 wait complete. 2024-03-19T02:33:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a8517d740 20 client.17400 trim_cache size 0 max 16384 2024-03-19T02:33:39.038 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 msgr2=0x7f3a640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 0x7f3a640a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a740099e0 tx=0x7f3a7400ff60 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 msgr2=0x7f3a7c103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009230 tx=0x7f3a7002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 shutdown_connections 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3a640a2970 0x7f3a640a4e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a7c1066b0 0x7f3a7c103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a7c101060 0x7f3a7c103200 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 --2- 172.21.15.116:0/2127860220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a7c1006a0 0x7f3a7c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 >> 172.21.15.116:0/2127860220 conn(0x7f3a7c0778d0 msgr2=0x7f3a7c068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 shutdown_connections 2024-03-19T02:33:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.031+0000 7f3a824b2640 1 -- 172.21.15.116:0/2127860220 wait complete. 2024-03-19T02:33:39.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:33:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 -- 172.21.15.116:0/1425238799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 msgr2=0x7f6bf0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 --2- 172.21.15.116:0/1425238799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf0106a90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009930 tx=0x7f6be002bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 -- 172.21.15.116:0/1425238799 shutdown_connections 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 --2- 172.21.15.116:0/1425238799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 0x7f6bf010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 --2- 172.21.15.116:0/1425238799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 --2- 172.21.15.116:0/1425238799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf0106a90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 -- 172.21.15.116:0/1425238799 >> 172.21.15.116:0/1425238799 conn(0x7f6bf00778d0 msgr2=0x7f6bf0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 -- 172.21.15.116:0/1425238799 shutdown_connections 2024-03-19T02:33:39.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 -- 172.21.15.116:0/1425238799 wait complete. 2024-03-19T02:33:39.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.164+0000 7f6bf5110640 1 Processor -- start 2024-03-19T02:33:39.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 -- start start 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 0x7f6bf0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0111a70 con 0x7f6bf01066b0 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01118f0 con 0x7f6bf0101060 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bf5110640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0111bf0 con 0x7f6bf01006a0 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 0x7f6bf0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47786/0 (socket says 172.21.15.116:47786) 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 -- 172.21.15.116:0/2984197540 learned_addr learned my addr 172.21.15.116:0/2984197540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 -- 172.21.15.116:0/2984197540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 msgr2=0x7f6bf0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 0x7f6bf0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 -- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 msgr2=0x7f6bf019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 -- 172.21.15.116:0/2984197540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf019cf90 con 0x7f6bf01006a0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.165+0000 7f6beed9d640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009230 tx=0x7f6be002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.166+0000 7f6bd7fff640 1 -- 172.21.15.116:0/2984197540 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0043020 con 0x7f6bf01006a0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.166+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf019d220 con 0x7f6bf01006a0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.166+0000 7f6bd7fff640 1 -- 172.21.15.116:0/2984197540 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6be002ca70 con 0x7f6bf01006a0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.166+0000 7f6bd7fff640 1 -- 172.21.15.116:0/2984197540 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0003a50 con 0x7f6bf01006a0 2024-03-19T02:33:39.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.166+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf01a4b10 con 0x7f6bf01006a0 2024-03-19T02:33:39.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:38 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3222207470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:39.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.168+0000 7f6bd7fff640 1 -- 172.21.15.116:0/2984197540 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6be0046020 con 0x7f6bf01006a0 2024-03-19T02:33:39.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.168+0000 7f6bf7dda740 1 Processor -- start 2024-03-19T02:33:39.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.168+0000 7f6bf7dda740 1 -- start start 2024-03-19T02:33:39.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bd7fff640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 0x7f6bd80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d167e0e4b0 con 0x55d167eb47d0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d167db6d70 con 0x55d167eab2d0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d167e0a550 con 0x55d167eabbf0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6beed9d640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 0x7f6bd80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47788/0 (socket says 172.21.15.116:47788) 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bee59c640 1 -- 172.21.15.116:0/2988038272 learned_addr learned my addr 172.21.15.116:0/2988038272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bd7fff640 1 -- 172.21.15.116:0/2984197540 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f6be00f10c0 con 0x7f6bf01006a0 2024-03-19T02:33:39.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 -- 172.21.15.116:0/2988038272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 msgr2=0x55d167eb4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 -- 172.21.15.116:0/2988038272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 msgr2=0x55d167eab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 -- 172.21.15.116:0/2988038272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d167dd9210 con 0x55d167eb47d0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bee59c640 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6beed9d640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 0x7f6bd80a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6be00bd2d0 tx=0x7f6be008f320 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb6bc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6be8005fd0 tx=0x7f6be8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b37f984e31628f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.169+0000 7f6bd57fa640 1 -- 172.21.15.116:0/2988038272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8016070 con 0x55d167eb47d0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bd57fa640 1 -- 172.21.15.116:0/2988038272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6be800e860 con 0x55d167eb47d0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bd57fa640 1 -- 172.21.15.116:0/2988038272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8019040 con 0x55d167eb47d0 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2988038272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 msgr2=0x55d167eb6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb6bc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6be8005fd0 tx=0x7f6be8009500 comp rx=0 tx=0).stop 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2988038272 shutdown_connections 2024-03-19T02:33:39.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb6bc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eabbf0 0x55d167eb4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2988038272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2988038272 >> 172.21.15.116:0/2988038272 conn(0x55d167ea5a00 msgr2=0x55d167ea5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2988038272 shutdown_connections 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.170+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2988038272 wait complete. 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 -- start start 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 Processor -- start 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eab2d0 0x55d167eb1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167eb1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d167eb7560 con 0x55d167eb47d0 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d167dca310 con 0x55d167eabbf0 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d167eb76e0 con 0x55d167eab2d0 2024-03-19T02:33:39.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bf7dda740 10 client.0 _ll_register_callbacks cb 0x55d167dbe2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43746/0 (socket says 172.21.15.116:43746) 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167eb1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 -- 172.21.15.116:0/2053881443 learned_addr learned my addr 172.21.15.116:0/2053881443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eab2d0 0x55d167eb1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 -- 172.21.15.116:0/2053881443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eab2d0 msgr2=0x55d167eb1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eab2d0 0x55d167eb1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 -- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 msgr2=0x55d167eb1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167eb1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bef59e640 1 -- 172.21.15.116:0/2053881443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d167b223f0 con 0x55d167eb47d0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.171+0000 7f6bee59c640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167eb1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6be800e820 tx=0x7f6be8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78c522c3ca0045d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:39.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8016040 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f6be800ead0 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6be800e860 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6be8036050 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 10 client.17412 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d167bec7d0 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6be800e860 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6be801d3d0 con 0x55d167eb47d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 10 client.17412 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 20 client.17412 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167ec49d0 0x55d167ec6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 4 client.17412 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.172+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d167e6c3e0 con 0x55d167ec49d0 2024-03-19T02:33:39.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.173+0000 7f6bee59c640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167ec49d0 0x55d167ec6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.173+0000 7f6bd2ff5640 10 client.17412 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.173+0000 7f6bee59c640 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167ec49d0 0x55d167ec6dc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc27de12ca2dbf47 server_cookie=5889aa5809f7a6da in_seq=0 out_seq=0 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/2053881443 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d167e6c3e0 con 0x55d167ec49d0 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bd2ff5640 10 client.17412 handle_command_reply: tid=0 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 client.17412 shutdown 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 msgr2=0x55d167eb22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6be800e820 tx=0x7f6be8005bd0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 shutdown_connections 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167ec49d0 0x55d167ec6dc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eb47d0 0x55d167eb22c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167eb1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2053881443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eab2d0 0x55d167eb1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bd2ff5640 0 client.17412 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:39.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 >> 172.21.15.116:0/2053881443 conn(0x55d167ea5a00 msgr2=0x55d167ea7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.174+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 shutdown_connections 2024-03-19T02:33:39.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.175+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2053881443 wait complete. 2024-03-19T02:33:39.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.175+0000 7f6bf7dda740 20 client.17412 trim_cache size 0 max 16384 2024-03-19T02:33:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 Processor -- start 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 -- start start 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167ec0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167ecc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01aeb80 con 0x55d167eab2d0 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01b06c0 con 0x55d167eabbf0 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01af250 con 0x55d167eb47d0 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43762/0 (socket says 172.21.15.116:43762) 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 -- 172.21.15.116:0/2623363461 learned_addr learned my addr 172.21.15.116:0/2623363461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167ec0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 -- 172.21.15.116:0/2623363461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 msgr2=0x55d167ecc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167ecc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 -- 172.21.15.116:0/2623363461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 msgr2=0x55d167ec0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167ec0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 -- 172.21.15.116:0/2623363461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d167e7b880 con 0x55d167eab2d0 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.189+0000 7f6bef59e640 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006010 tx=0x7f6be8010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7a75fe4fc80904 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bd4ff9640 1 -- 172.21.15.116:0/2623363461 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8016040 con 0x55d167eab2d0 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bd4ff9640 1 -- 172.21.15.116:0/2623363461 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6be800e860 con 0x55d167eab2d0 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bd4ff9640 1 -- 172.21.15.116:0/2623363461 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be801daf0 con 0x55d167eab2d0 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2623363461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 msgr2=0x55d167eb3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6be8006010 tx=0x7f6be8010d10 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2623363461 shutdown_connections 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167ecc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eabbf0 0x55d167ec0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/2623363461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eab2d0 0x55d167eb3f70 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2623363461 >> 172.21.15.116:0/2623363461 conn(0x55d167ea5a00 msgr2=0x55d167ea4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2623363461 shutdown_connections 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.190+0000 7f6bf7dda740 1 -- 172.21.15.116:0/2623363461 wait complete. 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 -- start start 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 Processor -- start 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eb78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167f28b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01afff0 con 0x55d167eabbf0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01af920 con 0x55d167eab2d0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf01acb60 con 0x55d167eb47d0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bf7dda740 10 client.0 _ll_register_callbacks cb 0x7f6be8010890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167f28b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43764/0 (socket says 172.21.15.116:43764) 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bef59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eb78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.191+0000 7f6bee59c640 1 -- 172.21.15.116:0/1968741118 learned_addr learned my addr 172.21.15.116:0/1968741118 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 -- 172.21.15.116:0/1968741118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 msgr2=0x55d167f28b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167f28b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 -- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 msgr2=0x55d167eb78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eb78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 -- 172.21.15.116:0/1968741118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d167eb8730 con 0x55d167eabbf0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bef59e640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eb78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bee59c640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6be4004700 tx=0x7f6be4012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad93104c6decc2c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4004460 con 0x55d167eabbf0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f6be4005020 con 0x55d167eabbf0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6be800e860 con 0x55d167eabbf0 2024-03-19T02:33:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6be4030050 con 0x55d167eabbf0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bf7dda740 10 client.17418 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d167f2a220 con 0x55d167eabbf0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.192+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6be800e860 con 0x55d167eabbf0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6be4020040 con 0x55d167eabbf0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 10 client.17418 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 20 client.17418 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167f2a9d0 0x55d167f2cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 4 client.17418 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d167e1b9f0 con 0x55d167f2a9d0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bef59e640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167f2a9d0 0x55d167f2cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bef59e640 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167f2a9d0 0x55d167f2cdf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d821f8e65c335451 server_cookie=94622a6a2016e463 in_seq=0 out_seq=0 2024-03-19T02:33:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.193+0000 7f6bd2ff5640 10 client.17418 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bd2ff5640 1 -- 172.21.15.116:0/1968741118 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d167e1b9f0 con 0x55d167f2a9d0 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bd2ff5640 10 client.17418 handle_command_reply: tid=0 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 client.17418 shutdown 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 msgr2=0x55d167eb7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6be4004700 tx=0x7f6be4012ae0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 shutdown_connections 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d167f2a9d0 0x55d167f2cdf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d167eb47d0 0x55d167f28b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bd2ff5640 0 client.17418 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d167eabbf0 0x55d167eb7e30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 --2- 172.21.15.116:0/1968741118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d167eab2d0 0x55d167eb78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 >> 172.21.15.116:0/1968741118 conn(0x55d167ea5a00 msgr2=0x55d167ea4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 shutdown_connections 2024-03-19T02:33:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 1 -- 172.21.15.116:0/1968741118 wait complete. 2024-03-19T02:33:39.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.194+0000 7f6bf7dda740 20 client.17418 trim_cache size 0 max 16384 2024-03-19T02:33:39.201 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 msgr2=0x7f6bd80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 0x7f6bd80a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6be00bd2d0 tx=0x7f6be008f320 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 msgr2=0x7f6bf0197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009230 tx=0x7f6be002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 shutdown_connections 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6bd80a2970 0x7f6bd80a4e30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bf01066b0 0x7f6bf019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bf0101060 0x7f6bf0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 --2- 172.21.15.116:0/2984197540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bf01006a0 0x7f6bf0197ff0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 >> 172.21.15.116:0/2984197540 conn(0x7f6bf00778d0 msgr2=0x7f6bf0102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 shutdown_connections 2024-03-19T02:33:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:39.195+0000 7f6bf5110640 1 -- 172.21.15.116:0/2984197540 wait complete. 2024-03-19T02:33:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:38 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3222207470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:33:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 183280 3.23 MB/sec execute 182 sec latency 17.680 ms 2024-03-19T02:33:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:39 smithi162 ceph-mon[35761]: pgmap v178: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:40.207 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:33:40.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:33:40.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:33:40.233 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:39 smithi116 ceph-mon[32263]: pgmap v178: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:39 smithi154 ceph-mon[35693]: pgmap v178: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 -- 172.21.15.116:0/812694645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0960101080 msgr2=0x7f096010d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 --2- 172.21.15.116:0/812694645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0960101080 0x7f096010d540 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0958009ca0 tx=0x7f095802c180 comp rx=0 tx=0).stop 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 -- 172.21.15.116:0/812694645 shutdown_connections 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 --2- 172.21.15.116:0/812694645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0960101080 0x7f096010d540 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 --2- 172.21.15.116:0/812694645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09601006e0 0x7f0960100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 --2- 172.21.15.116:0/812694645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 0x7f09601059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 -- 172.21.15.116:0/812694645 >> 172.21.15.116:0/812694645 conn(0x7f09600777b0 msgr2=0x7f0960077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 -- 172.21.15.116:0/812694645 shutdown_connections 2024-03-19T02:33:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.321+0000 7f09679e8640 1 -- 172.21.15.116:0/812694645 wait complete. 2024-03-19T02:33:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 Processor -- start 2024-03-19T02:33:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 -- start start 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 0x7f09601984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 0x7f096019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110960 con 0x7f09601006e0 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09601107e0 con 0x7f09601055c0 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f09679e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110ae0 con 0x7f0960101080 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43776/0 (socket says 172.21.15.116:43776) 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0965784640 1 -- 172.21.15.116:0/81323668 learned_addr learned my addr 172.21.15.116:0/81323668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 0x7f09601984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0965f85640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 0x7f096019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 -- 172.21.15.116:0/81323668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 msgr2=0x7f096019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 0x7f096019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 -- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 msgr2=0x7f0960197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0964f83640 1 -- 172.21.15.116:0/81323668 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096019cf50 con 0x7f0960101080 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.322+0000 7f0965784640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f0964f83640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 0x7f09601984a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f095000e9f0 tx=0x7f095000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f09567fc640 1 -- 172.21.15.116:0/81323668 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095000ccc0 con 0x7f0960101080 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096019d240 con 0x7f0960101080 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f09567fc640 1 -- 172.21.15.116:0/81323668 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09500108e0 con 0x7f0960101080 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f09567fc640 1 -- 172.21.15.116:0/81323668 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950018880 con 0x7f0960101080 2024-03-19T02:33:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.323+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09601a4ad0 con 0x7f0960101080 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.325+0000 7f09567fc640 1 -- 172.21.15.116:0/81323668 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0950010a90 con 0x7f0960101080 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.325+0000 7f096a5cb740 1 Processor -- start 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 -- start start 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa91693490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fa915f7160 con 0x55fa916939d0 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fa9159fe10 con 0x55fa916930b0 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f096a5cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa915f3200 con 0x55fa916a0c90 2024-03-19T02:33:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f09567fc640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 0x7f094c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0964f83640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43780/0 (socket says 172.21.15.116:43780) 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965f85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa91693490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 -- 172.21.15.116:0/4193911477 learned_addr learned my addr 172.21.15.116:0/4193911477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965f85640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 0x7f094c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f09567fc640 1 -- 172.21.15.116:0/81323668 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f09500cf550 con 0x7f0960101080 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 -- 172.21.15.116:0/4193911477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 msgr2=0x55fa916a3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 -- 172.21.15.116:0/4193911477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 msgr2=0x55fa91693490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa91693490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 -- 172.21.15.116:0/4193911477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa913d37e0 con 0x55fa916939d0 2024-03-19T02:33:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965784640 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f095c0115c0 tx=0x7f095c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558efad465b09d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.326+0000 7f0965f85640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 0x7f094c0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0958004af0 tx=0x7f0958005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f094b7fe640 1 -- 172.21.15.116:0/4193911477 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c00f660 con 0x55fa916939d0 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f094b7fe640 1 -- 172.21.15.116:0/4193911477 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f095c00fcb0 con 0x55fa916939d0 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f094b7fe640 1 -- 172.21.15.116:0/4193911477 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c019c80 con 0x55fa916939d0 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 -- 172.21.15.116:0/4193911477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 msgr2=0x55fa916a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f095c0115c0 tx=0x7f095c011a80 comp rx=0 tx=0).stop 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 -- 172.21.15.116:0/4193911477 shutdown_connections 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a0750 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4193911477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa91693490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 -- 172.21.15.116:0/4193911477 >> 172.21.15.116:0/4193911477 conn(0x55fa9168dab0 msgr2=0x55fa9168de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 -- 172.21.15.116:0/4193911477 shutdown_connections 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.327+0000 7f096a5cb740 1 -- 172.21.15.116:0/4193911477 wait complete. 2024-03-19T02:33:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 -- start start 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 Processor -- start 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916939d0 0x55fa9169e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa9169e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fa916a3a50 con 0x55fa916930b0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fa915b31a0 con 0x55fa916a0c90 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa916a3bd0 con 0x55fa916939d0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f096a5cb740 10 client.0 _ll_register_callbacks cb 0x55fa9130ac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f0965784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43786/0 (socket says 172.21.15.116:43786) 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.328+0000 7f0965784640 1 -- 172.21.15.116:0/1222383603 learned_addr learned my addr 172.21.15.116:0/1222383603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965f85640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa9169e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0964f83640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916939d0 0x55fa9169e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 -- 172.21.15.116:0/1222383603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916939d0 msgr2=0x55fa9169e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916939d0 0x55fa9169e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 -- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 msgr2=0x55fa9169e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa9169e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 -- 172.21.15.116:0/1222383603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa91433f70 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965f85640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa9169e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0965784640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f095c009350 tx=0x7f095c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43117c61d5b52413 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0948ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c027060 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f095c00f870 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c00fcb0 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f095c02e050 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f096a5cb740 10 client.17430 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa912cb170 con 0x55fa916930b0 2024-03-19T02:33:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.329+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f095c00fcb0 con 0x55fa916930b0 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f095c0046c0 con 0x55fa916930b0 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 10 client.17430 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 10 client.17430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 20 client.17430 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa916b1040 0x55fa916b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 4 client.17430 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa916046a0 con 0x55fa916b1040 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f0964f83640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa916b1040 0x55fa916b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f0964f83640 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa916b1040 0x55fa916b3430 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9dc39df8b4978a7 server_cookie=501c9559c8b192ca in_seq=0 out_seq=0 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.330+0000 7f0948ff9640 10 client.17430 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f0948ff9640 1 -- 172.21.15.116:0/1222383603 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa916046a0 con 0x55fa916b1040 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f0948ff9640 10 client.17430 handle_command_reply: tid=0 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 client.17430 shutdown 2024-03-19T02:33:40.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 msgr2=0x55fa9169dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f095c009350 tx=0x7f095c019640 comp rx=0 tx=0).stop 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 shutdown_connections 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa916b1040 0x55fa916b3430 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa9169e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916939d0 0x55fa9169e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1222383603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916930b0 0x55fa9169dc60 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f0948ff9640 0 client.17430 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 >> 172.21.15.116:0/1222383603 conn(0x55fa9168dab0 msgr2=0x55fa9168f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 shutdown_connections 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.331+0000 7f096a5cb740 1 -- 172.21.15.116:0/1222383603 wait complete. 2024-03-19T02:33:40.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.332+0000 7f096a5cb740 20 client.17430 trim_cache size 0 max 16384 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 Processor -- start 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 -- start start 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916930b0 0x55fa916ac860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa916b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ac890 con 0x55fa916939d0 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ae3d0 con 0x55fa916a0c90 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.350+0000 7f096a5cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09601acf60 con 0x55fa916930b0 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43794/0 (socket says 172.21.15.116:43794) 2024-03-19T02:33:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 -- 172.21.15.116:0/4035297962 learned_addr learned my addr 172.21.15.116:0/4035297962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0965784640 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916930b0 0x55fa916ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0965f85640 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa916b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 -- 172.21.15.116:0/4035297962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916930b0 msgr2=0x55fa916ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916930b0 0x55fa916ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 -- 172.21.15.116:0/4035297962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 msgr2=0x55fa916b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa916b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 -- 172.21.15.116:0/4035297962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa91693800 con 0x55fa916939d0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f0964f83640 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f09500982b0 tx=0x7f09500c6280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b4627f768c6eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f094affd640 1 -- 172.21.15.116:0/4035297962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950026040 con 0x55fa916939d0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f094affd640 1 -- 172.21.15.116:0/4035297962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09500ccba0 con 0x55fa916939d0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.351+0000 7f094affd640 1 -- 172.21.15.116:0/4035297962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950018a20 con 0x55fa916939d0 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- 172.21.15.116:0/4035297962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 msgr2=0x55fa9169d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f09500982b0 tx=0x7f09500c6280 comp rx=0 tx=0).stop 2024-03-19T02:33:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- 172.21.15.116:0/4035297962 shutdown_connections 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916a0c90 0x55fa916b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa9169d410 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 --2- 172.21.15.116:0/4035297962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916930b0 0x55fa916ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- 172.21.15.116:0/4035297962 >> 172.21.15.116:0/4035297962 conn(0x55fa9168dab0 msgr2=0x55fa9168c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- 172.21.15.116:0/4035297962 shutdown_connections 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- 172.21.15.116:0/4035297962 wait complete. 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 -- start start 2024-03-19T02:33:40.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.352+0000 7f096a5cb740 1 Processor -- start 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa916a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f09601add00 con 0x55fa916939d0 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09601ad630 con 0x55fa916930b0 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f09601aa870 con 0x55fa916a0c90 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 10 client.0 _ll_register_callbacks cb 0x7f095c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43806/0 (socket says 172.21.15.116:43806) 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 -- 172.21.15.116:0/1471301975 learned_addr learned my addr 172.21.15.116:0/1471301975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0965784640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa916a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0965f85640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 -- 172.21.15.116:0/1471301975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 msgr2=0x55fa916a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 -- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 msgr2=0x55fa916a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa916a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 -- 172.21.15.116:0/1471301975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa916a4c50 con 0x55fa916939d0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0965f85640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0964f83640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f09500cf520 tx=0x7f0950097c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20eb8e0e0dc8df1d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0948ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095009b8e0 con 0x55fa916939d0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f0950018a20 con 0x55fa916939d0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09500ccba0 con 0x55fa916939d0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0950018c20 con 0x55fa916939d0 2024-03-19T02:33:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.353+0000 7f096a5cb740 10 client.17436 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa91716f50 con 0x55fa916939d0 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09500ccba0 con 0x55fa916939d0 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f09500c7070 con 0x55fa916939d0 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 10 client.17436 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 10 client.17436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 20 client.17436 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa91717630 0x55fa91719a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 4 client.17436 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa915fe9f0 con 0x55fa91717630 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f0965784640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa91717630 0x55fa91719a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f0948ff9640 10 client.17436 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.354+0000 7f0965784640 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa91717630 0x55fa91719a20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=306b7e1200965c2b server_cookie=1de0d00c2d72e96a in_seq=0 out_seq=0 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.355+0000 7f0948ff9640 1 -- 172.21.15.116:0/1471301975 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa915fe9f0 con 0x55fa91717630 2024-03-19T02:33:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.355+0000 7f0948ff9640 10 client.17436 handle_command_reply: tid=0 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.355+0000 7f096a5cb740 1 client.17436 shutdown 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 msgr2=0x55fa916a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f09500cf520 tx=0x7f0950097c90 comp rx=0 tx=0).stop 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 shutdown_connections 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fa91717630 0x55fa91719a20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa916a0c90 0x55fa916a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fa916939d0 0x55fa916a9300 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 --2- 172.21.15.116:0/1471301975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa916930b0 0x55fa916a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 >> 172.21.15.116:0/1471301975 conn(0x55fa9168dab0 msgr2=0x55fa9168cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f0948ff9640 0 client.17436 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 shutdown_connections 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 1 -- 172.21.15.116:0/1471301975 wait complete. 2024-03-19T02:33:40.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f096a5cb740 20 client.17436 trim_cache size 0 max 16384 2024-03-19T02:33:40.363 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:33:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "d827317e-7d0c-4f2f-b904-4a216890db96", 2024-03-19T02:33:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:33:40.363 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 msgr2=0x7f094c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 0x7f094c0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0958004af0 tx=0x7f0958005c50 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 msgr2=0x7f09601984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 0x7f09601984a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f095000e9f0 tx=0x7f095000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 shutdown_connections 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f094c0a2970 0x7f094c0a4e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09601055c0 0x7f096019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0960101080 0x7f09601984a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 --2- 172.21.15.116:0/81323668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09601006e0 0x7f0960197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 >> 172.21.15.116:0/81323668 conn(0x7f09600777b0 msgr2=0x7f0960069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 shutdown_connections 2024-03-19T02:33:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.356+0000 7f09679e8640 1 -- 172.21.15.116:0/81323668 wait complete. 2024-03-19T02:33:40.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:33:40.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 -- 172.21.15.116:0/2934408043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 msgr2=0x7f8fa810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 --2- 172.21.15.116:0/2934408043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa810e680 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009930 tx=0x7f8fa002bc30 comp rx=0 tx=0).stop 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 -- 172.21.15.116:0/2934408043 shutdown_connections 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 --2- 172.21.15.116:0/2934408043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa810e680 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 --2- 172.21.15.116:0/2934408043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8068a20 0x7f8fa8068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 --2- 172.21.15.116:0/2934408043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8107870 0x7f8fa8107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 -- 172.21.15.116:0/2934408043 >> 172.21.15.116:0/2934408043 conn(0x7f8fa8077820 msgr2=0x7f8fa8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 -- 172.21.15.116:0/2934408043 shutdown_connections 2024-03-19T02:33:40.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.521+0000 7f8fad40b640 1 -- 172.21.15.116:0/2934408043 wait complete. 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 Processor -- start 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 -- start start 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 0x7f8fa8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 0x7f8fa819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa8111a70 con 0x7f8fa80693e0 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81118f0 con 0x7f8fa8107870 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fad40b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa8111bf0 con 0x7f8fa8068a20 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8f9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 0x7f8fa8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8f9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43822/0 (socket says 172.21.15.116:43822) 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8f9e7fc640 1 -- 172.21.15.116:0/1161813738 learned_addr learned my addr 172.21.15.116:0/1161813738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.522+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 0x7f8fa819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 -- 172.21.15.116:0/1161813738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 msgr2=0x7f8fa8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 0x7f8fa8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 -- 172.21.15.116:0/1161813738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 msgr2=0x7f8fa819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 0x7f8fa819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 -- 172.21.15.116:0/1161813738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa819cf00 con 0x7f8fa80693e0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 0x7f8fa8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 0x7f8fa819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e9f0 tx=0x7f8f9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fa4ff9640 1 -- 172.21.15.116:0/1161813738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9000cd80 con 0x7f8fa80693e0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa819d1f0 con 0x7f8fa80693e0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fa4ff9640 1 -- 172.21.15.116:0/1161813738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f900108e0 con 0x7f8fa80693e0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fa4ff9640 1 -- 172.21.15.116:0/1161813738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90018920 con 0x7f8fa80693e0 2024-03-19T02:33:40.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.523+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa81a4a80 con 0x7f8fa80693e0 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.525+0000 7f8fa4ff9640 1 -- 172.21.15.116:0/1161813738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8f90010a90 con 0x7f8fa80693e0 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 Processor -- start 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 -- start start 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 0x5587b370be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b370e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587b3662500 con 0x5587b370c340 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5587b360abe0 con 0x5587b36ff080 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fafff7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5587b365e5a0 con 0x5587b36fe760 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa4ff9640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 0x7f8f940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 0x5587b370be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8f9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b370e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47890/0 (socket says 172.21.15.116:47890) 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 0x5587b370be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43544/0 (socket says 172.21.15.116:43544) 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.526+0000 7f8fa77fe640 1 -- 172.21.15.116:0/2841455621 learned_addr learned my addr 172.21.15.116:0/2841455621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 0x7f8f940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa4ff9640 1 -- 172.21.15.116:0/1161813738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8f900cf440 con 0x7f8fa80693e0 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 -- 172.21.15.116:0/2841455621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 msgr2=0x5587b370be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 0x5587b370be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 -- 172.21.15.116:0/2841455621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 msgr2=0x5587b370e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b370e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 -- 172.21.15.116:0/2841455621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b362c000 con 0x5587b36fe760 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0005c40 tx=0x7f8fa002cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637b73f56e07921b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 0x7f8f940a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006010 tx=0x7f8fa0049000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8f9d7fa640 1 -- 172.21.15.116:0/2841455621 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00043c0 con 0x5587b36fe760 2024-03-19T02:33:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8f9d7fa640 1 -- 172.21.15.116:0/2841455621 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fa0003e70 con 0x5587b36fe760 2024-03-19T02:33:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.527+0000 7f8f9d7fa640 1 -- 172.21.15.116:0/2841455621 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00439e0 con 0x5587b36fe760 2024-03-19T02:33:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.528+0000 7f8fafff7740 1 -- 172.21.15.116:0/2841455621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 msgr2=0x5587b36feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.528+0000 7f8fafff7740 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0005c40 tx=0x7f8fa002cd30 comp rx=0 tx=0).stop 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.528+0000 7f8fafff7740 1 -- 172.21.15.116:0/2841455621 shutdown_connections 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.528+0000 7f8fafff7740 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b370e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36ff080 0x5587b370be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 --2- 172.21.15.116:0/2841455621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36fe760 0x5587b36feb40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 -- 172.21.15.116:0/2841455621 >> 172.21.15.116:0/2841455621 conn(0x5587b36f9160 msgr2=0x5587b36f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 -- 172.21.15.116:0/2841455621 shutdown_connections 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 -- 172.21.15.116:0/2841455621 wait complete. 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 -- start start 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 Processor -- start 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3709940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.529+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3709e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fafff7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587b370f0a0 con 0x5587b36fe760 2024-03-19T02:33:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fafff7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5587b361e160 con 0x5587b370c340 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fafff7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5587b370f220 con 0x5587b36ff080 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fafff7740 10 client.0 _ll_register_callbacks cb 0x5587b365aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43830/0 (socket says 172.21.15.116:43830) 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1544805189 learned_addr learned my addr 172.21.15.116:0/1544805189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3709940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3709e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1544805189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 msgr2=0x5587b3709940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3709940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 msgr2=0x5587b3709e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3709e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1544805189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b34407d0 con 0x5587b36fe760 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800fe60 tx=0x7f8f98012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3ba89f6e89b47a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8f8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98003970 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8f98003b10 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa0003e70 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.530+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8f98025050 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 10 client.17448 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5587b33763f0 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fa0003e70 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8f98010550 con 0x5587b36fe760 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 10 client.17448 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 20 client.17448 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b371c7b0 0x5587b371eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 4 client.17448 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587b366fa40 con 0x5587b371c7b0 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.531+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b371c7b0 0x5587b371eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.532+0000 7f8f8e7fc640 10 client.17448 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.532+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b371c7b0 0x5587b371eba0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=440b673e0121de51 server_cookie=4f3792ce5631730c in_seq=0 out_seq=0 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1544805189 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587b366fa40 con 0x5587b371c7b0 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8f8e7fc640 10 client.17448 handle_command_reply: tid=0 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 client.17448 shutdown 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 msgr2=0x5587b3709400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800fe60 tx=0x7f8f98012820 comp rx=0 tx=0).stop 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 shutdown_connections 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b371c7b0 0x5587b371eba0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3709e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3709940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8f8e7fc640 0 client.17448 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1544805189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b3709400 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 >> 172.21.15.116:0/1544805189 conn(0x5587b36f9160 msgr2=0x5587b36faeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 shutdown_connections 2024-03-19T02:33:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 1 -- 172.21.15.116:0/1544805189 wait complete. 2024-03-19T02:33:40.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.533+0000 7f8fafff7740 20 client.17448 trim_cache size 0 max 16384 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.551+0000 7f8fafff7740 1 Processor -- start 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.551+0000 7f8fafff7740 1 -- start start 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36fe760 0x5587b36f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3717e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81ac8b0 con 0x5587b370c340 2024-03-19T02:33:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81ae3f0 con 0x5587b36fe760 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fafff7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81acf80 con 0x5587b36ff080 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43846/0 (socket says 172.21.15.116:43846) 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8f9e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3717e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 -- 172.21.15.116:0/3300644703 learned_addr learned my addr 172.21.15.116:0/3300644703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36fe760 0x5587b36f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 -- 172.21.15.116:0/3300644703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 msgr2=0x5587b3717e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3717e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 -- 172.21.15.116:0/3300644703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36fe760 msgr2=0x5587b36f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36fe760 0x5587b36f81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 -- 172.21.15.116:0/3300644703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b371ef10 con 0x5587b370c340 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3717e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.552+0000 7f8fa77fe640 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8fa002c320 tx=0x7f8fa00437c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01c3d435b870871 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8f9cff9640 1 -- 172.21.15.116:0/3300644703 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0043460 con 0x5587b370c340 2024-03-19T02:33:40.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8f9cff9640 1 -- 172.21.15.116:0/3300644703 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fa0043600 con 0x5587b370c340 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8f9cff9640 1 -- 172.21.15.116:0/3300644703 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0043d30 con 0x5587b370c340 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 -- 172.21.15.116:0/3300644703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 msgr2=0x5587b36faca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8fa002c320 tx=0x7f8fa00437c0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 -- 172.21.15.116:0/3300644703 shutdown_connections 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b370c340 0x5587b36faca0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3717e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.553+0000 7f8fafff7740 1 --2- 172.21.15.116:0/3300644703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b36fe760 0x5587b36f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- 172.21.15.116:0/3300644703 >> 172.21.15.116:0/3300644703 conn(0x5587b36f9160 msgr2=0x5587b36fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- 172.21.15.116:0/3300644703 shutdown_connections 2024-03-19T02:33:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- 172.21.15.116:0/3300644703 wait complete. 2024-03-19T02:33:40.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- start start 2024-03-19T02:33:40.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 Processor -- start 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3706b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3781aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81add20 con 0x5587b36fe760 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81ad650 con 0x5587b370c340 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa81aa890 con 0x5587b36ff080 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fafff7740 10 client.0 _ll_register_callbacks cb 0x7f8f9800f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43858/0 (socket says 172.21.15.116:43858) 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8f9e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3706b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3781aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.554+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1611544994 learned_addr learned my addr 172.21.15.116:0/1611544994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1611544994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 msgr2=0x5587b3706b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3706b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 msgr2=0x5587b3781aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3781aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 -- 172.21.15.116:0/1611544994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b33360f0 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fa6ffd640 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006010 tx=0x7f8f98012d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4fd720142da6de server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8f8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f980092a0 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9422+0+0 (secure 0 0 0) 0x7f8f98018040 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587b3669d90 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8f98037050 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fafff7740 10 client.17454 fetch_fsmap learned FSMap version 30 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5587b3782fc0 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.555+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5587b3669d90 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8f98005110 con 0x5587b36fe760 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 10 client.17454 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:33:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 20 client.17454 populate_metadata read hostname 'smithi116' 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b37836d0 0x5587b3785ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 4 client.17454 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587b35e7540 con 0x5587b37836d0 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b37836d0 0x5587b3785ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8f9e7fc640 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b37836d0 0x5587b3785ac0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25abae99bdf9aecf server_cookie=1a4864372435bb28 in_seq=0 out_seq=0 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.556+0000 7f8f8e7fc640 10 client.17454 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8f8e7fc640 1 -- 172.21.15.116:0/1611544994 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587b35e7540 con 0x5587b37836d0 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8f8e7fc640 10 client.17454 handle_command_reply: tid=0 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 client.17454 shutdown 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 msgr2=0x5587b37065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8f98006010 tx=0x7f8f98012d90 comp rx=0 tx=0).stop 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 shutdown_connections 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5587b37836d0 0x5587b3785ac0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5587b370c340 0x5587b3781aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8f8e7fc640 0 client.17454 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587b36ff080 0x5587b3706b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 --2- 172.21.15.116:0/1611544994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587b36fe760 0x5587b37065c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 >> 172.21.15.116:0/1611544994 conn(0x5587b36f9160 msgr2=0x5587b36fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 shutdown_connections 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 1 -- 172.21.15.116:0/1611544994 wait complete. 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.557+0000 7f8fafff7740 20 client.17454 trim_cache size 0 max 16384 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:33:40.563 INFO:teuthology.orchestra.run.smithi116.stdout: "d827317e-7d0c-4f2f-b904-4a216890db96": { 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "d827317e-7d0c-4f2f-b904-4a216890db96", 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:33:40.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 msgr2=0x7f8f940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 0x7f8f940a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006010 tx=0x7f8fa0049000 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 msgr2=0x7f8fa8198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e9f0 tx=0x7f8f9000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 shutdown_connections 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8f940a2970 0x7f8f940a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa8107870 0x7f8fa819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fa80693e0 0x7f8fa8198450 secure :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e9f0 tx=0x7f8f9000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 --2- 172.21.15.116:0/1161813738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fa8068a20 0x7f8fa8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 >> 172.21.15.116:0/1161813738 conn(0x7f8fa8077820 msgr2=0x7f8fa80fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 shutdown_connections 2024-03-19T02:33:40.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:33:40.558+0000 7f8fad40b640 1 -- 172.21.15.116:0/1161813738 wait complete. 2024-03-19T02:33:40.569 INFO:tasks.cephfs.filesystem:scrub status for tag:d827317e-7d0c-4f2f-b904-4a216890db96 - {'path': '/', 'tag': 'd827317e-7d0c-4f2f-b904-4a216890db96', 'options': 'recursive,force'} 2024-03-19T02:33:40.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 184108 3.25 MB/sec execute 183 sec latency 15.564 ms 2024-03-19T02:33:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:40 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:33:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:40 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:40 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:33:41.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:40 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:40 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:33:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:40 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:40 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:33:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:40 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:40 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:33:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:40 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:40 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:33:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:40 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:41.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 184490 3.26 MB/sec execute 184 sec latency 16.585 ms 2024-03-19T02:33:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:41 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:33:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:41 smithi162 ceph-mon[35761]: pgmap v179: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:33:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:41 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:33:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:41 smithi154 ceph-mon[35693]: pgmap v179: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:33:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:41 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:33:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:33:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:41 smithi116 ceph-mon[32263]: pgmap v179: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-03-19T02:33:42.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 184970 3.25 MB/sec execute 185 sec latency 15.994 ms 2024-03-19T02:33:43.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 185528 3.24 MB/sec execute 186 sec latency 44.077 ms 2024-03-19T02:33:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:43 smithi162 ceph-mon[35761]: pgmap v180: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:33:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:43 smithi154 ceph-mon[35693]: pgmap v180: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:33:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:43 smithi116 ceph-mon[32263]: pgmap v180: 129 pgs: 129 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-03-19T02:33:44.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 186234 3.24 MB/sec execute 187 sec latency 20.961 ms 2024-03-19T02:33:45.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 186857 3.23 MB/sec execute 188 sec latency 23.240 ms 2024-03-19T02:33:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:45 smithi162 ceph-mon[35761]: pgmap v181: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:33:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:45 smithi154 ceph-mon[35693]: pgmap v181: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:33:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:45 smithi116 ceph-mon[32263]: pgmap v181: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.2 MiB/s wr, 123 op/s 2024-03-19T02:33:46.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 187736 3.26 MB/sec execute 189 sec latency 17.001 ms 2024-03-19T02:33:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 188120 3.26 MB/sec execute 190 sec latency 19.642 ms 2024-03-19T02:33:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:47 smithi162 ceph-mon[35761]: pgmap v182: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:47 smithi154 ceph-mon[35693]: pgmap v182: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:47 smithi116 ceph-mon[32263]: pgmap v182: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:48.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 188597 3.25 MB/sec execute 191 sec latency 15.130 ms 2024-03-19T02:33:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 189182 3.24 MB/sec execute 192 sec latency 34.186 ms 2024-03-19T02:33:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:49 smithi162 ceph-mon[35761]: pgmap v183: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:49 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:33:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:49 smithi154 ceph-mon[35693]: pgmap v183: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:49 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:33:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:49 smithi116 ceph-mon[32263]: pgmap v183: 129 pgs: 129 active+clean; 167 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:33:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:49 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:33:50.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 189853 3.24 MB/sec execute 193 sec latency 20.666 ms 2024-03-19T02:33:51.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:33:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:33:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:33:51.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 190442 3.23 MB/sec execute 194 sec latency 24.549 ms 2024-03-19T02:33:52.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:51 smithi154 ceph-mon[35693]: pgmap v184: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:51 smithi162 ceph-mon[35761]: pgmap v184: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:51 smithi116 ceph-mon[32263]: pgmap v184: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:33:52.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 191295 3.25 MB/sec execute 195 sec latency 17.769 ms 2024-03-19T02:33:53.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 191781 3.26 MB/sec execute 196 sec latency 18.876 ms 2024-03-19T02:33:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:53 smithi162 ceph-mon[35761]: pgmap v185: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-03-19T02:33:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:53 smithi154 ceph-mon[35693]: pgmap v185: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-03-19T02:33:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:53 smithi116 ceph-mon[32263]: pgmap v185: 129 pgs: 129 active+clean; 170 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-03-19T02:33:54.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 192156 3.24 MB/sec execute 197 sec latency 39.623 ms 2024-03-19T02:33:55.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 192797 3.24 MB/sec execute 198 sec latency 16.601 ms 2024-03-19T02:33:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:55 smithi162 ceph-mon[35761]: pgmap v186: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:55 smithi154 ceph-mon[35693]: pgmap v186: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:55 smithi116 ceph-mon[32263]: pgmap v186: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.4 MiB/s wr, 123 op/s 2024-03-19T02:33:56.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 193465 3.24 MB/sec execute 199 sec latency 17.714 ms 2024-03-19T02:33:57.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 194127 3.24 MB/sec execute 200 sec latency 16.069 ms 2024-03-19T02:33:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:57 smithi162 ceph-mon[35761]: pgmap v187: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:57 smithi154 ceph-mon[35693]: pgmap v187: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:57 smithi116 ceph-mon[32263]: pgmap v187: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:33:58.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 194865 3.25 MB/sec execute 201 sec latency 21.559 ms 2024-03-19T02:33:59.786 INFO:tasks.workunit.client.0.smithi116.stdout: 1 195365 3.26 MB/sec execute 202 sec latency 22.475 ms 2024-03-19T02:34:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:59 smithi162 ceph-mon[35761]: pgmap v188: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:34:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:33:59 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:34:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:59 smithi154 ceph-mon[35693]: pgmap v188: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:34:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:33:59 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:34:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:59 smithi116 ceph-mon[32263]: pgmap v188: 129 pgs: 129 active+clean; 172 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:34:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:33:59 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:34:00.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 195754 3.24 MB/sec execute 203 sec latency 46.236 ms 2024-03-19T02:34:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:00 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:34:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:00 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:34:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:00 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:34:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:00 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:34:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:00 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:34:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:00 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:34:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:00 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:34:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:00 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:34:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:00 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:34:01.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 196345 3.24 MB/sec execute 204 sec latency 15.734 ms 2024-03-19T02:34:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:01 smithi162 ceph-mon[35761]: pgmap v189: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:01 smithi154 ceph-mon[35693]: pgmap v189: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:01 smithi116 ceph-mon[32263]: pgmap v189: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:02.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 197014 3.24 MB/sec execute 205 sec latency 17.878 ms 2024-03-19T02:34:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:02 smithi154 ceph-mon[35693]: pgmap v190: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:02 smithi116 ceph-mon[32263]: pgmap v190: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:02 smithi162 ceph-mon[35761]: pgmap v190: 129 pgs: 129 active+clean; 175 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:03.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 197603 3.23 MB/sec execute 206 sec latency 18.262 ms 2024-03-19T02:34:04.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 198401 3.25 MB/sec execute 207 sec latency 18.577 ms 2024-03-19T02:34:05.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:05 smithi154 ceph-mon[35693]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:34:05.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:05 smithi154 ceph-mon[35693]: pgmap v192: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:05 smithi116 ceph-mon[32263]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:34:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:05 smithi116 ceph-mon[32263]: pgmap v192: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 198904 3.26 MB/sec execute 208 sec latency 16.697 ms 2024-03-19T02:34:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:05 smithi162 ceph-mon[35761]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:34:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:05 smithi162 ceph-mon[35761]: pgmap v192: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:06.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 199321 3.24 MB/sec execute 209 sec latency 38.791 ms 2024-03-19T02:34:07.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 199924 3.24 MB/sec execute 210 sec latency 46.917 ms 2024-03-19T02:34:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:07 smithi162 ceph-mon[35761]: pgmap v193: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:07 smithi154 ceph-mon[35693]: pgmap v193: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:07 smithi116 ceph-mon[32263]: pgmap v193: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:08.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 200621 3.24 MB/sec execute 211 sec latency 19.134 ms 2024-03-19T02:34:09.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 201220 3.23 MB/sec execute 212 sec latency 17.428 ms 2024-03-19T02:34:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:09 smithi162 ceph-mon[35761]: pgmap v194: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:09 smithi154 ceph-mon[35693]: pgmap v194: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:09 smithi116 ceph-mon[32263]: pgmap v194: 129 pgs: 129 active+clean; 179 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-03-19T02:34:10.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:34:10.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:34:10.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 -- 172.21.15.116:0/1714598698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4741077a0 msgr2=0x7ff474107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 --2- 172.21.15.116:0/1714598698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4741077a0 0x7ff474107b80 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff464009930 tx=0x7ff46402bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 -- 172.21.15.116:0/1714598698 shutdown_connections 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 --2- 172.21.15.116:0/1714598698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4740693c0 0x7ff47410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 --2- 172.21.15.116:0/1714598698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 --2- 172.21.15.116:0/1714598698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4741077a0 0x7ff474107b80 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 -- 172.21.15.116:0/1714598698 >> 172.21.15.116:0/1714598698 conn(0x7ff474077820 msgr2=0x7ff474077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 -- 172.21.15.116:0/1714598698 shutdown_connections 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.682+0000 7ff47a697640 1 -- 172.21.15.116:0/1714598698 wait complete. 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 Processor -- start 2024-03-19T02:34:10.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 -- start start 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 0x7ff474198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4741077a0 0x7ff47419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff474111a20 con 0x7ff4740693c0 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741118a0 con 0x7ff4741077a0 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff47a697640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff474111ba0 con 0x7ff474068a00 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff473fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 0x7ff474198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff473fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58434/0 (socket says 172.21.15.116:58434) 2024-03-19T02:34:10.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff478c34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4741077a0 0x7ff47419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.683+0000 7ff473fff640 1 -- 172.21.15.116:0/174850807 learned_addr learned my addr 172.21.15.116:0/174850807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 -- 172.21.15.116:0/174850807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4741077a0 msgr2=0x7ff47419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4741077a0 0x7ff47419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 -- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 msgr2=0x7ff474198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 0x7ff474198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 -- 172.21.15.116:0/174850807 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47419cfd0 con 0x7ff474068a00 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff4737fe640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 0x7ff474198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff473fff640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff464009230 tx=0x7ff46402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff4717fa640 1 -- 172.21.15.116:0/174850807 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464043020 con 0x7ff474068a00 2024-03-19T02:34:10.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47419d260 con 0x7ff474068a00 2024-03-19T02:34:10.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff4717fa640 1 -- 172.21.15.116:0/174850807 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff46402ca70 con 0x7ff474068a00 2024-03-19T02:34:10.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff4717fa640 1 -- 172.21.15.116:0/174850807 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464003a50 con 0x7ff474068a00 2024-03-19T02:34:10.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.684+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4741a4b50 con 0x7ff474068a00 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 Processor -- start 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 -- start start 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba97069f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9709320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ba9661500 con 0x556ba9706f30 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556ba9609be0 con 0x556ba96fda30 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff47d283740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556ba965d5a0 con 0x556ba96fe350 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4717fa640 1 -- 172.21.15.116:0/174850807 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff464046020 con 0x7ff474068a00 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff478c34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9709320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff478c34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:42880/0 (socket says 172.21.15.116:42880) 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff478c34640 1 -- 172.21.15.116:0/1679507159 learned_addr learned my addr 172.21.15.116:0/1679507159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff473fff640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba97069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 -- 172.21.15.116:0/1679507159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 msgr2=0x556ba97069f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba97069f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 -- 172.21.15.116:0/1679507159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 msgr2=0x556ba96fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff4737fe640 1 -- 172.21.15.116:0/1679507159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba962b000 con 0x556ba9706f30 2024-03-19T02:34:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff473fff640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba97069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.687+0000 7ff478c34640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff4737fe640 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9709320 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff468009fa0 tx=0x7ff4680118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73d09d173ae7854 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff4717fa640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 0x7ff45c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff45a7fc640 1 -- 172.21.15.116:0/1679507159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468004030 con 0x556ba9706f30 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff45a7fc640 1 -- 172.21.15.116:0/1679507159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4680054d0 con 0x556ba9706f30 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff45a7fc640 1 -- 172.21.15.116:0/1679507159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468004c10 con 0x556ba9706f30 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff478c34640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 0x7ff45c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff4717fa640 1 -- 172.21.15.116:0/174850807 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7ff4640f1160 con 0x7ff474068a00 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 -- 172.21.15.116:0/1679507159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 msgr2=0x556ba9709320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9709320 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff468009fa0 tx=0x7ff4680118d0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff478c34640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 0x7ff45c0a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fd0 tx=0x7ff46c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 -- 172.21.15.116:0/1679507159 shutdown_connections 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9709320 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba97069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 --2- 172.21.15.116:0/1679507159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba96fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.688+0000 7ff47d283740 1 -- 172.21.15.116:0/1679507159 >> 172.21.15.116:0/1679507159 conn(0x556ba96f8160 msgr2=0x556ba96f8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- 172.21.15.116:0/1679507159 shutdown_connections 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- 172.21.15.116:0/1679507159 wait complete. 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- start start 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 Processor -- start 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba9703ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9704400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ba9709d20 con 0x556ba9706f30 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556ba961d160 con 0x556ba96fda30 2024-03-19T02:34:10.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556ba9709ea0 con 0x556ba96fe350 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff47d283740 10 client.0 _ll_register_callbacks cb 0x556ba9659a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff478c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff473fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba9703ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff478c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60860/0 (socket says 172.21.15.116:60860) 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff478c34640 1 -- 172.21.15.116:0/3636551721 learned_addr learned my addr 172.21.15.116:0/3636551721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.689+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9704400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 -- 172.21.15.116:0/3636551721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 msgr2=0x556ba9704400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9704400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 -- 172.21.15.116:0/3636551721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 msgr2=0x556ba9704940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 -- 172.21.15.116:0/3636551721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ba943f7d0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff478c34640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff4737fe640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9704400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff473fff640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba9703ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4640ee590 tx=0x7ff4640b9100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ee8b87985ae2fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff4537fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4640bce00 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4680054d0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7ff4640053d0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff4640bdba0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff47d283740 10 client.27278 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.690+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556ba93753f0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4680054d0 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff4640d6120 con 0x556ba96fda30 2024-03-19T02:34:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 10 client.27278 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 10 client.27278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 20 client.27278 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba97171b0 0x556ba9719600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 4 client.27278 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ba966ea40 con 0x556ba97171b0 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff4737fe640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba97171b0 0x556ba9719600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff4537fe640 10 client.27278 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.691+0000 7ff4737fe640 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba97171b0 0x556ba9719600 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9bdc1990afb003 server_cookie=e29695980e17c07f in_seq=0 out_seq=0 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff4537fe640 1 -- 172.21.15.116:0/3636551721 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ba966ea40 con 0x556ba97171b0 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff4537fe640 10 client.27278 handle_command_reply: tid=0 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 client.27278 shutdown 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 msgr2=0x556ba9703ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba9703ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4640ee590 tx=0x7ff4640b9100 comp rx=0 tx=0).stop 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 shutdown_connections 2024-03-19T02:34:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba97171b0 0x556ba9719600 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba9704940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff4537fe640 0 client.27278 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9704400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 --2- 172.21.15.116:0/3636551721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba9703ec0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 >> 172.21.15.116:0/3636551721 conn(0x556ba96f8160 msgr2=0x556ba96f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 shutdown_connections 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 1 -- 172.21.15.116:0/3636551721 wait complete. 2024-03-19T02:34:10.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.693+0000 7ff47d283740 20 client.27278 trim_cache size 0 max 16384 2024-03-19T02:34:10.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 Processor -- start 2024-03-19T02:34:10.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 -- start start 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba97065f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9712ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741aedc0 con 0x556ba9706f30 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741b0900 con 0x556ba96fda30 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff47d283740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741af490 con 0x556ba96fe350 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff473fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba97065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60864/0 (socket says 172.21.15.116:60864) 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 -- 172.21.15.116:0/1085204790 learned_addr learned my addr 172.21.15.116:0/1085204790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff4737fe640 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9712ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 -- 172.21.15.116:0/1085204790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 msgr2=0x556ba9712ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9712ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 -- 172.21.15.116:0/1085204790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 msgr2=0x556ba97065f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba97065f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff478c34640 1 -- 172.21.15.116:0/1085204790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba93350f0 con 0x556ba9706f30 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.713+0000 7ff473fff640 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba97065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff478c34640 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00fcc0 tx=0x7ff46c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f652808f6cea08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff459ffb640 1 -- 172.21.15.116:0/1085204790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c00fdb0 con 0x556ba9706f30 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff459ffb640 1 -- 172.21.15.116:0/1085204790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff46c003970 con 0x556ba9706f30 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff459ffb640 1 -- 172.21.15.116:0/1085204790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c004030 con 0x556ba9706f30 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff47d283740 1 -- 172.21.15.116:0/1085204790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 msgr2=0x556ba96f9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.714+0000 7ff47d283740 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00fcc0 tx=0x7ff46c016000 comp rx=0 tx=0).stop 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- 172.21.15.116:0/1085204790 shutdown_connections 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba9706f30 0x556ba96f9ca0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fe350 0x556ba9712ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- 172.21.15.116:0/1085204790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba96fda30 0x556ba97065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- 172.21.15.116:0/1085204790 >> 172.21.15.116:0/1085204790 conn(0x556ba96f8160 msgr2=0x556ba96f9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- 172.21.15.116:0/1085204790 shutdown_connections 2024-03-19T02:34:10.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- 172.21.15.116:0/1085204790 wait complete. 2024-03-19T02:34:10.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- start start 2024-03-19T02:34:10.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 Processor -- start 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fda30 0x556ba970f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 0x556ba970ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741b0230 con 0x556ba96fe350 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741afb60 con 0x556ba9706f30 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff4741acdc0 con 0x556ba96fda30 2024-03-19T02:34:10.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff47d283740 10 client.0 _ll_register_callbacks cb 0x7ff4640057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.715+0000 7ff473fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fda30 0x556ba970f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60874/0 (socket says 172.21.15.116:60874) 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 -- 172.21.15.116:0/2579229628 learned_addr learned my addr 172.21.15.116:0/2579229628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff478c34640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 0x556ba970ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 -- 172.21.15.116:0/2579229628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fda30 msgr2=0x556ba970f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fda30 0x556ba970f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 -- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 msgr2=0x556ba970ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 0x556ba970ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 -- 172.21.15.116:0/2579229628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ba970b590 con 0x556ba96fe350 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff478c34640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 0x556ba970ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4737fe640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff468009380 tx=0x7ff468005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb7232e0df4cd54 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4537fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:10.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468031060 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff46c003970 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7ff46800fd00 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff468038050 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff47d283740 10 client.17472 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556ba977d530 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.716+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff46c003970 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff46802b540 con 0x556ba96fe350 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 10 client.17472 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 20 client.17472 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:10.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba977dac0 0x556ba977feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 4 client.17472 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ba9668d90 con 0x556ba977dac0 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff473fff640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba977dac0 0x556ba977feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff473fff640 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba977dac0 0x556ba977feb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44e17c854be75ab2 server_cookie=a74912713f91e072 in_seq=0 out_seq=0 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.717+0000 7ff4537fe640 10 client.17472 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff4537fe640 1 -- 172.21.15.116:0/2579229628 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ba9668d90 con 0x556ba977dac0 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff4537fe640 10 client.17472 handle_command_reply: tid=0 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 client.17472 shutdown 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 msgr2=0x556ba970a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff468009380 tx=0x7ff468005850 comp rx=0 tx=0).stop 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 shutdown_connections 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556ba977dac0 0x556ba977feb0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ba9706f30 0x556ba970ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ba96fe350 0x556ba970a7b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff4537fe640 0 client.17472 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 --2- 172.21.15.116:0/2579229628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ba96fda30 0x556ba970f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 >> 172.21.15.116:0/2579229628 conn(0x556ba96f8160 msgr2=0x556ba96f71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 shutdown_connections 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 1 -- 172.21.15.116:0/2579229628 wait complete. 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.719+0000 7ff47d283740 20 client.17472 trim_cache size 0 max 16384 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:10.727 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:34:10.728 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:34:10.728 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:10.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 msgr2=0x7ff45c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 0x7ff45c0a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fd0 tx=0x7ff46c009500 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 msgr2=0x7ff474197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff464009230 tx=0x7ff46402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 shutdown_connections 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff45c0a2970 0x7ff45c0a4e30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4741077a0 0x7ff47419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4740693c0 0x7ff474198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 --2- 172.21.15.116:0/174850807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff474068a00 0x7ff474197fe0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 >> 172.21.15.116:0/174850807 conn(0x7ff474077820 msgr2=0x7ff4740fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 shutdown_connections 2024-03-19T02:34:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.720+0000 7ff47a697640 1 -- 172.21.15.116:0/174850807 wait complete. 2024-03-19T02:34:10.731 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:34:10.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:34:10.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 201956 3.25 MB/sec execute 213 sec latency 15.973 ms 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 -- 172.21.15.116:0/6690861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601077d0 msgr2=0x7f6860107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 --2- 172.21.15.116:0/6690861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601077d0 0x7f6860107bb0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f6850009930 tx=0x7f685002bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 -- 172.21.15.116:0/6690861 shutdown_connections 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 --2- 172.21.15.116:0/6690861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6860102180 0x7f686010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 --2- 172.21.15.116:0/6690861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68601017c0 0x7f6860101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 --2- 172.21.15.116:0/6690861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601077d0 0x7f6860107bb0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 -- 172.21.15.116:0/6690861 >> 172.21.15.116:0/6690861 conn(0x7f68600fd2b0 msgr2=0x7f68600ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 -- 172.21.15.116:0/6690861 shutdown_connections 2024-03-19T02:34:10.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 -- 172.21.15.116:0/6690861 wait complete. 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.836+0000 7f68655b3640 1 Processor -- start 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 -- start start 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6860102180 0x7f6860198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68601077d0 0x7f686019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6860111a90 con 0x7f68601017c0 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6860111910 con 0x7f68601077d0 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f68655b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6860111c10 con 0x7f6860102180 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60884/0 (socket says 172.21.15.116:60884) 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 -- 172.21.15.116:0/866811541 learned_addr learned my addr 172.21.15.116:0/866811541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685e7fc640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6860102180 0x7f6860198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685f7fe640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68601077d0 0x7f686019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 -- 172.21.15.116:0/866811541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6860102180 msgr2=0x7f6860198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6860102180 0x7f6860198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 -- 172.21.15.116:0/866811541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68601077d0 msgr2=0x7f686019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68601077d0 0x7f686019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 -- 172.21.15.116:0/866811541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686019d020 con 0x7f68601017c0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.837+0000 7f685effd640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f6850009230 tx=0x7f685002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.838+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850043020 con 0x7f68601017c0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.838+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686019d2b0 con 0x7f68601017c0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.838+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f685002ca70 con 0x7f68601017c0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.838+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850003a50 con 0x7f68601017c0 2024-03-19T02:34:10.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.838+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68601a4ba0 con 0x7f68601017c0 2024-03-19T02:34:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.839+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6860068610 con 0x7f68601017c0 2024-03-19T02:34:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.840+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6850046020 con 0x7f68601017c0 2024-03-19T02:34:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.840+0000 7f6847fff640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 0x7f68480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.840+0000 7f685e7fc640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 0x7f68480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.840+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7f68500f1170 con 0x7f68601017c0 2024-03-19T02:34:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.841+0000 7f685e7fc640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 0x7f68480a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f68600ffa10 tx=0x7f6854005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:10.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:10.846+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f685004d020 con 0x7f68601017c0 2024-03-19T02:34:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.082+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6860109d30 con 0x7f68601017c0 2024-03-19T02:34:11.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.083+0000 7f6847fff640 1 -- 172.21.15.116:0/866811541 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f68500bdcd0 con 0x7f68601017c0 2024-03-19T02:34:11.083 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:34:11.084 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:34:11.084 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 msgr2=0x7f68480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 0x7f68480a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f68600ffa10 tx=0x7f6854005f90 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 msgr2=0x7f6860198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f6850009230 tx=0x7f685002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 shutdown_connections 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68480a2970 0x7f68480a4e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68601077d0 0x7f686019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6860102180 0x7f6860198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 --2- 172.21.15.116:0/866811541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68601017c0 0x7f6860198030 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 >> 172.21.15.116:0/866811541 conn(0x7f68600fd2b0 msgr2=0x7f68600fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 shutdown_connections 2024-03-19T02:34:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.085+0000 7f68655b3640 1 -- 172.21.15.116:0/866811541 wait complete. 2024-03-19T02:34:11.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- 172.21.15.116:0/2578343243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc741006a0 msgr2=0x7efc74100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 --2- 172.21.15.116:0/2578343243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc741006a0 0x7efc74100b20 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009930 tx=0x7efc5c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- 172.21.15.116:0/2578343243 shutdown_connections 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 --2- 172.21.15.116:0/2578343243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc74101060 0x7efc7410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 --2- 172.21.15.116:0/2578343243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc741006a0 0x7efc74100b20 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 --2- 172.21.15.116:0/2578343243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741066b0 0x7efc74106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- 172.21.15.116:0/2578343243 >> 172.21.15.116:0/2578343243 conn(0x7efc740778d0 msgr2=0x7efc74077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- 172.21.15.116:0/2578343243 shutdown_connections 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- 172.21.15.116:0/2578343243 wait complete. 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 Processor -- start 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.214+0000 7efc7930c640 1 -- start start 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 0x7efc74197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc741066b0 0x7efc7419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc74110970 con 0x7efc74101060 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc741107f0 con 0x7efc741066b0 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc7930c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc74110af0 con 0x7efc741006a0 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60904/0 (socket says 172.21.15.116:60904) 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc6bfff640 1 -- 172.21.15.116:0/1169827346 learned_addr learned my addr 172.21.15.116:0/1169827346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 0x7efc74197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc737fe640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc741066b0 0x7efc7419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 -- 172.21.15.116:0/1169827346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc741066b0 msgr2=0x7efc7419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc741066b0 0x7efc7419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 -- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 msgr2=0x7efc74198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 -- 172.21.15.116:0/1169827346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7419cfb0 con 0x7efc741006a0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc6bfff640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.215+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 0x7efc74197fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efc6400e9f0 tx=0x7efc6400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.216+0000 7efc70ff9640 1 -- 172.21.15.116:0/1169827346 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6400ccc0 con 0x7efc741006a0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.216+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7419d2a0 con 0x7efc741006a0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.216+0000 7efc70ff9640 1 -- 172.21.15.116:0/1169827346 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc64005020 con 0x7efc741006a0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.216+0000 7efc70ff9640 1 -- 172.21.15.116:0/1169827346 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64004a00 con 0x7efc741006a0 2024-03-19T02:34:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.216+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc741a4b30 con 0x7efc741006a0 2024-03-19T02:34:11.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.218+0000 7efc70ff9640 1 -- 172.21.15.116:0/1169827346 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7efc64004ba0 con 0x7efc741006a0 2024-03-19T02:34:11.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.218+0000 7efc7bfcb740 1 Processor -- start 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.218+0000 7efc7bfcb740 1 -- start start 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d286b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d31290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2c8b4b0 con 0x55d8f2d317d0 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2c33d70 con 0x55d8f2d282d0 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2c87550 con 0x55d8f2d28bf0 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc70ff9640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 0x7efc600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc72ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d31290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60912/0 (socket says 172.21.15.116:60912) 2024-03-19T02:34:11.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc6bfff640 1 -- 172.21.15.116:0/3018906225 learned_addr learned my addr 172.21.15.116:0/3018906225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc737fe640 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d286b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc737fe640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 0x7efc600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc70ff9640 1 -- 172.21.15.116:0/1169827346 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efc640cf210 con 0x7efc741006a0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.219+0000 7efc6bfff640 1 -- 172.21.15.116:0/3018906225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 msgr2=0x55d8f2d31290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc6bfff640 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d31290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc6bfff640 1 -- 172.21.15.116:0/3018906225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 msgr2=0x55d8f2d286b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc6bfff640 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d286b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc6bfff640 1 -- 172.21.15.116:0/3018906225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f2c56210 con 0x55d8f2d317d0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc737fe640 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc6bfff640 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009430 tx=0x7efc5c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f80fcaabcaee35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc737fe640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 0x7efc600a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009990 tx=0x7efc6c0096a0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc697fa640 1 -- 172.21.15.116:0/3018906225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c045070 con 0x55d8f2d317d0 2024-03-19T02:34:11.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc697fa640 1 -- 172.21.15.116:0/3018906225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc5c02c7e0 con 0x55d8f2d317d0 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.220+0000 7efc697fa640 1 -- 172.21.15.116:0/3018906225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c003b90 con 0x55d8f2d317d0 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- 172.21.15.116:0/3018906225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 msgr2=0x55d8f2d33bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009430 tx=0x7efc5c02c650 comp rx=0 tx=0).stop 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- 172.21.15.116:0/3018906225 shutdown_connections 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d317d0 0x55d8f2d33bc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d31290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/3018906225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- 172.21.15.116:0/3018906225 >> 172.21.15.116:0/3018906225 conn(0x55d8f2d22a00 msgr2=0x55d8f2d22de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- 172.21.15.116:0/3018906225 shutdown_connections 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- 172.21.15.116:0/3018906225 wait complete. 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 -- start start 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.221+0000 7efc7bfcb740 1 Processor -- start 2024-03-19T02:34:11.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d282d0 0x55d8f2d2e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d2f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2d34560 con 0x55d8f2d28bf0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2c47310 con 0x55d8f2d317d0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8f2d346e0 con 0x55d8f2d282d0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 10 client.0 _ll_register_callbacks cb 0x55d8f2c3b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60918/0 (socket says 172.21.15.116:60918) 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 -- 172.21.15.116:0/1038601784 learned_addr learned my addr 172.21.15.116:0/1038601784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc737fe640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d2f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d282d0 0x55d8f2d2e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 -- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d282d0 msgr2=0x55d8f2d2e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d282d0 0x55d8f2d2e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 -- 172.21.15.116:0/1038601784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 msgr2=0x55d8f2d2f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d2f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 -- 172.21.15.116:0/1038601784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f299f3f0 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc6bfff640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7efc5c005940 tx=0x7efc5c005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33b55b8ac07fc76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c04e020 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efc5c003d00 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.222+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc5c02c7e0 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efc5c059050 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 10 client.17490 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d8f2a697d0 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc5c02c7e0 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efc5c03f440 con 0x55d8f2d28bf0 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 10 client.17490 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 10 client.17490 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 20 client.17490 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2d45de0 0x55d8f2d481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 4 client.17490 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8f2ce93e0 con 0x55d8f2d45de0 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.223+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2d45de0 0x55d8f2d481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc5a7fc640 10 client.17490 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc72ffd640 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2d45de0 0x55d8f2d481d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0e2648dfae87dad server_cookie=7379704805910b81 in_seq=0 out_seq=0 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc5a7fc640 1 -- 172.21.15.116:0/1038601784 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8f2ce93e0 con 0x55d8f2d45de0 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc5a7fc640 10 client.17490 handle_command_reply: tid=0 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 client.17490 shutdown 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 msgr2=0x55d8f2d2ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7efc5c005940 tx=0x7efc5c005de0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 shutdown_connections 2024-03-19T02:34:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2d45de0 0x55d8f2d481d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d2f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d2ed80 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/1038601784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d282d0 0x55d8f2d2e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 >> 172.21.15.116:0/1038601784 conn(0x55d8f2d22a00 msgr2=0x55d8f2d24750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.224+0000 7efc5a7fc640 0 client.17490 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.225+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 shutdown_connections 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.225+0000 7efc7bfcb740 1 -- 172.21.15.116:0/1038601784 wait complete. 2024-03-19T02:34:11.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.225+0000 7efc7bfcb740 20 client.17490 trim_cache size 0 max 16384 2024-03-19T02:34:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 Processor -- start 2024-03-19T02:34:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 -- start start 2024-03-19T02:34:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d41790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d24540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc741acad0 con 0x55d8f2d282d0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc741ae610 con 0x55d8f2d317d0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc741ad1a0 con 0x55d8f2d28bf0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d41790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc737fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d24540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60930/0 (socket says 172.21.15.116:60930) 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.240+0000 7efc72ffd640 1 -- 172.21.15.116:0/735339321 learned_addr learned my addr 172.21.15.116:0/735339321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 -- 172.21.15.116:0/735339321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 msgr2=0x55d8f2d41790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d41790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 -- 172.21.15.116:0/735339321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 msgr2=0x55d8f2d30f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 -- 172.21.15.116:0/735339321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f2cf8880 con 0x55d8f2d317d0 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc72ffd640 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc6bfff640 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d41790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc737fe640 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d24540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efc6c005c30 tx=0x7efc6c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f8a13a205f698f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc68ff9640 1 -- 172.21.15.116:0/735339321 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c012e20 con 0x55d8f2d317d0 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc68ff9640 1 -- 172.21.15.116:0/735339321 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc6c010950 con 0x55d8f2d317d0 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.241+0000 7efc68ff9640 1 -- 172.21.15.116:0/735339321 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c005480 con 0x55d8f2d317d0 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- 172.21.15.116:0/735339321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 msgr2=0x55d8f2d24540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d24540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efc6c005c30 tx=0x7efc6c009920 comp rx=0 tx=0).stop 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- 172.21.15.116:0/735339321 shutdown_connections 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d317d0 0x55d8f2d24540 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d41790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/735339321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d282d0 0x55d8f2d30f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- 172.21.15.116:0/735339321 >> 172.21.15.116:0/735339321 conn(0x55d8f2d22a00 msgr2=0x55d8f2d23e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- 172.21.15.116:0/735339321 shutdown_connections 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- 172.21.15.116:0/735339321 wait complete. 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 -- start start 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 Processor -- start 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d39db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.242+0000 7efc7bfcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d317d0 0x55d8f2daadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc741adf40 con 0x55d8f2d28bf0 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc741ad870 con 0x55d8f2d282d0 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc741aaad0 con 0x55d8f2d317d0 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 10 client.0 _ll_register_callbacks cb 0x7efc6c009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60938/0 (socket says 172.21.15.116:60938) 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc6bfff640 1 -- 172.21.15.116:0/2170738198 learned_addr learned my addr 172.21.15.116:0/2170738198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d39db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc737fe640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d317d0 0x55d8f2daadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 -- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d317d0 msgr2=0x55d8f2daadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d317d0 0x55d8f2daadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 -- 172.21.15.116:0/2170738198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 msgr2=0x55d8f2d3a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 -- 172.21.15.116:0/2170738198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f2dab310 con 0x55d8f2d282d0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc6bfff640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc72ffd640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d39db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc64097840 tx=0x7efc6406d010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6754add033a275a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6401c040 con 0x55d8f2d282d0 2024-03-19T02:34:11.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc6c010950 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efc640051d0 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efc6409f530 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 10 client.27308 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.243+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d8f2dac620 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc6c010950 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efc6409f160 con 0x55d8f2d282d0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 10 client.27308 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 10 client.27308 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 20 client.27308 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2daccf0 0x55d8f2daf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 4 client.27308 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8f2c989f0 con 0x55d8f2daccf0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc6bfff640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2daccf0 0x55d8f2daf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc5a7fc640 10 client.27308 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.244+0000 7efc6bfff640 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2daccf0 0x55d8f2daf110 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69d94cc7cb1bfcfc server_cookie=45e484ea241377ee in_seq=0 out_seq=0 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc5a7fc640 1 -- 172.21.15.116:0/2170738198 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8f2c989f0 con 0x55d8f2daccf0 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc5a7fc640 10 client.27308 handle_command_reply: tid=0 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 client.27308 shutdown 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 msgr2=0x55d8f2d39db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d39db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc64097840 tx=0x7efc6406d010 comp rx=0 tx=0).stop 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 shutdown_connections 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d8f2daccf0 0x55d8f2daf110 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8f2d317d0 0x55d8f2daadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8f2d28bf0 0x55d8f2d3a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc5a7fc640 0 client.27308 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 --2- 172.21.15.116:0/2170738198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8f2d282d0 0x55d8f2d39db0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 >> 172.21.15.116:0/2170738198 conn(0x55d8f2d22a00 msgr2=0x55d8f2d246c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 shutdown_connections 2024-03-19T02:34:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 1 -- 172.21.15.116:0/2170738198 wait complete. 2024-03-19T02:34:11.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7bfcb740 20 client.27308 trim_cache size 0 max 16384 2024-03-19T02:34:11.253 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 msgr2=0x7efc600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 0x7efc600a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009990 tx=0x7efc6c0096a0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 msgr2=0x7efc74197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 0x7efc74197fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efc6400e9f0 tx=0x7efc6400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 shutdown_connections 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efc600a2970 0x7efc600a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc741066b0 0x7efc7419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc74101060 0x7efc74198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 --2- 172.21.15.116:0/1169827346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc741006a0 0x7efc74197fc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.245+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 >> 172.21.15.116:0/1169827346 conn(0x7efc740778d0 msgr2=0x7efc74068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.246+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 shutdown_connections 2024-03-19T02:34:11.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.246+0000 7efc7930c640 1 -- 172.21.15.116:0/1169827346 wait complete. 2024-03-19T02:34:11.257 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:34:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.367+0000 7f35e2c7e640 1 -- 172.21.15.116:0/2745659431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc1006a0 msgr2=0x7f35dc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.367+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/2745659431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc1006a0 0x7f35dc100b20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009930 tx=0x7f35c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 -- 172.21.15.116:0/2745659431 shutdown_connections 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/2745659431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc101060 0x7f35dc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/2745659431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc1006a0 0x7f35dc100b20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/2745659431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 0x7f35dc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 -- 172.21.15.116:0/2745659431 >> 172.21.15.116:0/2745659431 conn(0x7f35dc0778d0 msgr2=0x7f35dc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 -- 172.21.15.116:0/2745659431 shutdown_connections 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 -- 172.21.15.116:0/2745659431 wait complete. 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 Processor -- start 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 -- start start 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc1006a0 0x7f35dc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.368+0000 7f35e2c7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 0x7f35dc19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e2c7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc110970 con 0x7f35dc101060 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e2c7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1107f0 con 0x7f35dc1066b0 2024-03-19T02:34:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e2c7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc110af0 con 0x7f35dc1006a0 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60950/0 (socket says 172.21.15.116:60950) 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 -- 172.21.15.116:0/3670799780 learned_addr learned my addr 172.21.15.116:0/3670799780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc1006a0 0x7f35dc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e121b640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 0x7f35dc19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 -- 172.21.15.116:0/3670799780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc1006a0 msgr2=0x7f35dc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc1006a0 0x7f35dc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 -- 172.21.15.116:0/3670799780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 msgr2=0x7f35dc19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 0x7f35dc19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 -- 172.21.15.116:0/3670799780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35dc19cf80 con 0x7f35dc101060 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009230 tx=0x7f35c802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35d9ffb640 1 -- 172.21.15.116:0/3670799780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8043020 con 0x7f35dc101060 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35d9ffb640 1 -- 172.21.15.116:0/3670799780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35c802ca70 con 0x7f35dc101060 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35dc19d210 con 0x7f35dc101060 2024-03-19T02:34:11.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.369+0000 7f35d9ffb640 1 -- 172.21.15.116:0/3670799780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8003a50 con 0x7f35dc101060 2024-03-19T02:34:11.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.370+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35dc1a4b00 con 0x7f35dc101060 2024-03-19T02:34:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35d9ffb640 1 -- 172.21.15.116:0/3670799780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f35c8046020 con 0x7f35dc101060 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 Processor -- start 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 -- start start 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557cd2ce44b0 con 0x557cd2d812d0 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557cd2c8cd70 con 0x557cd2d81bf0 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35e5866740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557cd2ce0550 con 0x557cd2d8a7d0 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.372+0000 7f35d9ffb640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 0x7f35cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60952/0 (socket says 172.21.15.116:60952) 2024-03-19T02:34:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43010/0 (socket says 172.21.15.116:43010) 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 -- 172.21.15.116:0/2041473627 learned_addr learned my addr 172.21.15.116:0/2041473627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35dbfff640 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 0x7f35cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35d9ffb640 1 -- 172.21.15.116:0/3670799780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7f35c80f1160 con 0x7f35dc101060 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 -- 172.21.15.116:0/2041473627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 msgr2=0x557cd2d8cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 -- 172.21.15.116:0/2041473627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 msgr2=0x557cd2d816b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 -- 172.21.15.116:0/2041473627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd2caf210 con 0x557cd2d81bf0 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e121b640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 0x7f35cc0a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35d40099e0 tx=0x7f35d4009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009c20 tx=0x7f35d0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb07d82ef422840 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35c6ffd640 1 -- 172.21.15.116:0/2041473627 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0004050 con 0x557cd2d81bf0 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35c6ffd640 1 -- 172.21.15.116:0/2041473627 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d00054d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.373+0000 7f35c6ffd640 1 -- 172.21.15.116:0/2041473627 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0004c30 con 0x557cd2d81bf0 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- 172.21.15.116:0/2041473627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 msgr2=0x557cd2d8a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009c20 tx=0x7f35d0011810 comp rx=0 tx=0).stop 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- 172.21.15.116:0/2041473627 shutdown_connections 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d81bf0 0x557cd2d8a290 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 --2- 172.21.15.116:0/2041473627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d816b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- 172.21.15.116:0/2041473627 >> 172.21.15.116:0/2041473627 conn(0x557cd2d7ba00 msgr2=0x557cd2d7bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- 172.21.15.116:0/2041473627 shutdown_connections 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- 172.21.15.116:0/2041473627 wait complete. 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.374+0000 7f35e5866740 1 -- start start 2024-03-19T02:34:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 Processor -- start 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2d88250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557cd2d8d5c0 con 0x557cd2d81bf0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557cd2ca0310 con 0x557cd2d8a7d0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557cd2d8d740 con 0x557cd2d812d0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e5866740 10 client.0 _ll_register_callbacks cb 0x557cd2c942d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60964/0 (socket says 172.21.15.116:60964) 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58566/0 (socket says 172.21.15.116:58566) 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 -- 172.21.15.116:0/3039080464 learned_addr learned my addr 172.21.15.116:0/3039080464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e121b640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2d88250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 -- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 msgr2=0x557cd2d877d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 -- 172.21.15.116:0/3039080464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 msgr2=0x557cd2d88250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2d88250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35dbfff640 1 -- 172.21.15.116:0/3039080464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd29f83f0 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.375+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35dbfff640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f35c80bd430 tx=0x7f35c808cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acf8443f0ad71ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80bce00 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7f35c80053d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35d00054d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35c80bdba0 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 10 client.17508 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557cd2ac27d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d00054d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35c80bd700 con 0x557cd2d81bf0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 10 client.17508 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 10 client.17508 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 20 client.17508 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2d9a9a0 0x557cd2d9cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 4 client.17508 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.376+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cd2d423e0 con 0x557cd2d9a9a0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.377+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2d9a9a0 0x557cd2d9cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.377+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2d9a9a0 0x557cd2d9cdd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5bbf8eb00b64b395 server_cookie=5babba77e010b351 in_seq=0 out_seq=0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.377+0000 7f35bbfff640 10 client.17508 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35bbfff640 1 -- 172.21.15.116:0/3039080464 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557cd2d423e0 con 0x557cd2d9a9a0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35bbfff640 10 client.17508 handle_command_reply: tid=0 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 client.17508 shutdown 2024-03-19T02:34:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 msgr2=0x557cd2d87d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f35c80bd430 tx=0x7f35c808cf70 comp rx=0 tx=0).stop 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 shutdown_connections 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2d88250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2d9a9a0 0x557cd2d9cdd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d87d10 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 --2- 172.21.15.116:0/3039080464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d812d0 0x557cd2d877d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 >> 172.21.15.116:0/3039080464 conn(0x557cd2d7ba00 msgr2=0x557cd2d7d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.378+0000 7f35bbfff640 0 client.17508 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.379+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 shutdown_connections 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.379+0000 7f35e5866740 1 -- 172.21.15.116:0/3039080464 wait complete. 2024-03-19T02:34:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.379+0000 7f35e5866740 20 client.17508 trim_cache size 0 max 16384 2024-03-19T02:34:11.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 Processor -- start 2024-03-19T02:34:11.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 -- start start 2024-03-19T02:34:11.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 0x557cd2d96360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2da26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1aed30 con 0x557cd2d812d0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1b0870 con 0x557cd2d8a7d0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e5866740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1af400 con 0x557cd2d81bf0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 0x557cd2d96360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60980/0 (socket says 172.21.15.116:60980) 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e0a1a640 1 -- 172.21.15.116:0/1334226998 learned_addr learned my addr 172.21.15.116:0/1334226998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.394+0000 7f35e121b640 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2da26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 -- 172.21.15.116:0/1334226998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 msgr2=0x557cd2da26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2da26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 -- 172.21.15.116:0/1334226998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 msgr2=0x557cd2d89f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 -- 172.21.15.116:0/1334226998 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd2d51880 con 0x557cd2d81bf0 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35dbfff640 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 0x557cd2d96360 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f35c80ee640 tx=0x7f35c80bb9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7656ea1daeeec84e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35c67fc640 1 -- 172.21.15.116:0/1334226998 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80bce00 con 0x557cd2d81bf0 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35c67fc640 1 -- 172.21.15.116:0/1334226998 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35c80eebd0 con 0x557cd2d81bf0 2024-03-19T02:34:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.395+0000 7f35c67fc640 1 -- 172.21.15.116:0/1334226998 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80053d0 con 0x557cd2d81bf0 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- 172.21.15.116:0/1334226998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 msgr2=0x557cd2d96360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 0x557cd2d96360 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f35c80ee640 tx=0x7f35c80bb9b0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- 172.21.15.116:0/1334226998 shutdown_connections 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d8a7d0 0x557cd2da26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d81bf0 0x557cd2d96360 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 --2- 172.21.15.116:0/1334226998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d812d0 0x557cd2d89f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- 172.21.15.116:0/1334226998 >> 172.21.15.116:0/1334226998 conn(0x557cd2d7ba00 msgr2=0x557cd2d7d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- 172.21.15.116:0/1334226998 shutdown_connections 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- 172.21.15.116:0/1334226998 wait complete. 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 -- start start 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.396+0000 7f35e5866740 1 Processor -- start 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 0x557cd2d92d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d8e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1b01a0 con 0x557cd2d81bf0 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1afad0 con 0x557cd2d812d0 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc1acd30 con 0x557cd2d8a7d0 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 10 client.0 _ll_register_callbacks cb 0x7f35c80057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d8e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 0x557cd2d92d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e121b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e121b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58592/0 (socket says 172.21.15.116:58592) 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e0a1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 0x557cd2d92d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43030/0 (socket says 172.21.15.116:43030) 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e121b640 1 -- 172.21.15.116:0/2992396936 learned_addr learned my addr 172.21.15.116:0/2992396936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 -- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 msgr2=0x557cd2d8e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 -- 172.21.15.116:0/2992396936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 msgr2=0x557cd2d92d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 0x557cd2d92d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 -- 172.21.15.116:0/2992396936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd2d8cbc0 con 0x557cd2d81bf0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e121b640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35dbfff640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d8e090 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x557cd2d7de70 tx=0x7f35c8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf2efec6502d1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80bce00 con 0x557cd2d81bf0 2024-03-19T02:34:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7f35c80c1160 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c80eebd0 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35c80bd270 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 10 client.17514 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.397+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557cd2d8cbc0 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35c80eebd0 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35c80dd0e0 con 0x557cd2d81bf0 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 10 client.17514 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 10 client.17514 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 20 client.17514 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2e00f40 0x557cd2e03330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 4 client.17514 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557cd2cf19f0 con 0x557cd2e00f40 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2e00f40 0x557cd2e03330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35bbfff640 10 client.17514 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.398+0000 7f35e0a1a640 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2e00f40 0x557cd2e03330 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6ef1f099e0101490 server_cookie=28556093d91a11ce in_seq=0 out_seq=0 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35bbfff640 1 -- 172.21.15.116:0/2992396936 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557cd2cf19f0 con 0x557cd2e00f40 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35bbfff640 10 client.17514 handle_command_reply: tid=0 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 client.17514 shutdown 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 msgr2=0x557cd2d8e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d8e090 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x557cd2d7de70 tx=0x7f35c8005b50 comp rx=0 tx=0).stop 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 shutdown_connections 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557cd2d8a7d0 0x557cd2d8e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557cd2e00f40 0x557cd2e03330 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cd2d81bf0 0x557cd2d8e090 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 --2- 172.21.15.116:0/2992396936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557cd2d812d0 0x557cd2d92d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35bbfff640 0 client.17514 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 >> 172.21.15.116:0/2992396936 conn(0x557cd2d7ba00 msgr2=0x557cd2d7a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 shutdown_connections 2024-03-19T02:34:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 1 -- 172.21.15.116:0/2992396936 wait complete. 2024-03-19T02:34:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.399+0000 7f35e5866740 20 client.17514 trim_cache size 0 max 16384 2024-03-19T02:34:11.406 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 msgr2=0x7f35cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 0x7f35cc0a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f35d40099e0 tx=0x7f35d4009500 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 msgr2=0x7f35dc1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009230 tx=0x7f35c802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 shutdown_connections 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35cc0a2970 0x7f35cc0a4e30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35dc1066b0 0x7f35dc19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35dc101060 0x7f35dc1984d0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 --2- 172.21.15.116:0/3670799780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35dc1006a0 0x7f35dc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 >> 172.21.15.116:0/3670799780 conn(0x7f35dc0778d0 msgr2=0x7f35dc068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 shutdown_connections 2024-03-19T02:34:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.400+0000 7f35e2c7e640 1 -- 172.21.15.116:0/3670799780 wait complete. 2024-03-19T02:34:11.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:34:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- 172.21.15.116:0/2368058294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 msgr2=0x7fb234106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 --2- 172.21.15.116:0/2368058294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb234106a90 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb230009930 tx=0x7fb23002bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- 172.21.15.116:0/2368058294 shutdown_connections 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 --2- 172.21.15.116:0/2368058294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb234101060 0x7fb23410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 --2- 172.21.15.116:0/2368058294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2341006a0 0x7fb234100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 --2- 172.21.15.116:0/2368058294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb234106a90 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- 172.21.15.116:0/2368058294 >> 172.21.15.116:0/2368058294 conn(0x7fb2340778d0 msgr2=0x7fb234077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- 172.21.15.116:0/2368058294 shutdown_connections 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- 172.21.15.116:0/2368058294 wait complete. 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 Processor -- start 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.522+0000 7fb23bea1640 1 -- start start 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 0x7fb234198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb234101060 0x7fb234199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb234110930 con 0x7fb2341066b0 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341107b0 con 0x7fb234101060 2024-03-19T02:34:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb234110ab0 con 0x7fb2341006a0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32784/0 (socket says 172.21.15.116:32784) 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 0x7fb234198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 -- 172.21.15.116:0/208885683 learned_addr learned my addr 172.21.15.116:0/208885683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23943c640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb234101060 0x7fb234199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 -- 172.21.15.116:0/208885683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 msgr2=0x7fb234198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 0x7fb234198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 -- 172.21.15.116:0/208885683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb234101060 msgr2=0x7fb234199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb234101060 0x7fb234199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 -- 172.21.15.116:0/208885683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2341a43b0 con 0x7fb2341066b0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb239c3d640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 0x7fb234198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23a43e640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00efd0 tx=0x7fb22c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb22affd640 1 -- 172.21.15.116:0/208885683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c003b70 con 0x7fb2341066b0 2024-03-19T02:34:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb22affd640 1 -- 172.21.15.116:0/208885683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb22c003d10 con 0x7fb2341066b0 2024-03-19T02:34:11.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2341a4580 con 0x7fb2341066b0 2024-03-19T02:34:11.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.523+0000 7fb22affd640 1 -- 172.21.15.116:0/208885683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c010980 con 0x7fb2341066b0 2024-03-19T02:34:11.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.524+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2341a4a50 con 0x7fb2341066b0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.526+0000 7fb22affd640 1 -- 172.21.15.116:0/208885683 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb22c004020 con 0x7fb2341066b0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.526+0000 7fb22affd640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 0x7fb2200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb239c3d640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 0x7fb2200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb22affd640 1 -- 172.21.15.116:0/208885683 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7fb22c0cf6b0 con 0x7fb2341066b0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 Processor -- start 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 -- start start 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fa46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fafbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3f074b0 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3eafd70 con 0x55cfe3fa42d0 2024-03-19T02:34:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3f03550 con 0x55cfe3fad7d0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb239c3d640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 0x7fb2200a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb230005ab0 tx=0x7fb230005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fa46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32794/0 (socket says 172.21.15.116:32794) 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23943c640 1 -- 172.21.15.116:0/3340222353 learned_addr learned my addr 172.21.15.116:0/3340222353 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.527+0000 7fb23a43e640 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fafbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 -- 172.21.15.116:0/3340222353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 msgr2=0x55cfe3fafbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fafbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 -- 172.21.15.116:0/3340222353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa42d0 msgr2=0x55cfe3fa46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fa46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 -- 172.21.15.116:0/3340222353 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe3ed2210 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb23943c640 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb224009c90 tx=0x7fb224011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc15fda39f6177 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb21ffff640 1 -- 172.21.15.116:0/3340222353 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22401e070 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb21ffff640 1 -- 172.21.15.116:0/3340222353 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb224019ae0 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.528+0000 7fb21ffff640 1 -- 172.21.15.116:0/3340222353 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22400e990 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 -- 172.21.15.116:0/3340222353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 msgr2=0x55cfe3fad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb224009c90 tx=0x7fb224011810 comp rx=0 tx=0).stop 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 -- 172.21.15.116:0/3340222353 shutdown_connections 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fafbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fad290 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/3340222353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fa46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 -- 172.21.15.116:0/3340222353 >> 172.21.15.116:0/3340222353 conn(0x55cfe3f9ea00 msgr2=0x55cfe3f9ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 -- 172.21.15.116:0/3340222353 shutdown_connections 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.529+0000 7fb23eb6f740 1 -- 172.21.15.116:0/3340222353 wait complete. 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 -- start start 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 Processor -- start 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3faad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3fb05c0 con 0x55cfe3fad7d0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3ec3310 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cfe3fb0740 con 0x55cfe3fa42d0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23eb6f740 10 client.0 _ll_register_callbacks cb 0x55cfe3eb72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23943c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3faad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32800/0 (socket says 172.21.15.116:32800) 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.530+0000 7fb23a43e640 1 -- 172.21.15.116:0/4096404907 learned_addr learned my addr 172.21.15.116:0/4096404907 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 -- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 msgr2=0x55cfe3faad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3faad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 -- 172.21.15.116:0/4096404907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 msgr2=0x55cfe3fab2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 -- 172.21.15.116:0/4096404907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe3c1b3f0 con 0x55cfe3fa42d0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb23a43e640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb239c3d640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb230009230 tx=0x7fb23002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f3f933390b6d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb21d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23005b060 con 0x55cfe3fa42d0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb224019ae0 con 0x55cfe3fa42d0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7fb23003a040 con 0x55cfe3fa42d0 2024-03-19T02:34:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb230062050 con 0x55cfe3fa42d0 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb23eb6f740 10 client.27280 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.531+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cfe3ce57d0 con 0x55cfe3fa42d0 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb224019ae0 con 0x55cfe3fa42d0 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb230003940 con 0x55cfe3fa42d0 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 10 client.27280 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 10 client.27280 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 20 client.27280 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe3fbdb70 0x55cfe3fbff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 4 client.27280 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfe3f653e0 con 0x55cfe3fbdb70 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23943c640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe3fbdb70 0x55cfe3fbff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb21d7fa640 10 client.27280 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.532+0000 7fb23943c640 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe3fbdb70 0x55cfe3fbff80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a431401a2a0c8b20 server_cookie=a799fa2cbd89cf78 in_seq=0 out_seq=0 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb21d7fa640 1 -- 172.21.15.116:0/4096404907 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cfe3f653e0 con 0x55cfe3fbdb70 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb21d7fa640 10 client.27280 handle_command_reply: tid=0 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 client.27280 shutdown 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 msgr2=0x55cfe3faa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb230009230 tx=0x7fb23002bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 shutdown_connections 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe3fbdb70 0x55cfe3fbff80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fab2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb21d7fa640 0 client.27280 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3faad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/4096404907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa830 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 >> 172.21.15.116:0/4096404907 conn(0x55cfe3f9ea00 msgr2=0x55cfe3fa0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 shutdown_connections 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 1 -- 172.21.15.116:0/4096404907 wait complete. 2024-03-19T02:34:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.533+0000 7fb23eb6f740 20 client.27280 trim_cache size 0 max 16384 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.549+0000 7fb23eb6f740 1 Processor -- start 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.549+0000 7fb23eb6f740 1 -- start start 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.549+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fc5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341acb70 con 0x55cfe3fa42d0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341ae6b0 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341ad240 con 0x55cfe3fad7d0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32802/0 (socket says 172.21.15.116:32802) 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 -- 172.21.15.116:0/608578795 learned_addr learned my addr 172.21.15.116:0/608578795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23943c640 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb23a43e640 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fc5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 -- 172.21.15.116:0/608578795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 msgr2=0x55cfe3fc5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fc5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 -- 172.21.15.116:0/608578795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 msgr2=0x55cfe3fb9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 -- 172.21.15.116:0/608578795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfe3fafbc0 con 0x55cfe3fa42d0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb239c3d640 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb2300096c0 tx=0x7fb23002c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8433722d8a4e9ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb21f7fe640 1 -- 172.21.15.116:0/608578795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23005b040 con 0x55cfe3fa42d0 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb21f7fe640 1 -- 172.21.15.116:0/608578795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb23003a040 con 0x55cfe3fa42d0 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.550+0000 7fb21f7fe640 1 -- 172.21.15.116:0/608578795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23003e980 con 0x55cfe3fa42d0 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- 172.21.15.116:0/608578795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 msgr2=0x55cfe3faa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb2300096c0 tx=0x7fb23002c910 comp rx=0 tx=0).stop 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- 172.21.15.116:0/608578795 shutdown_connections 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fc5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/608578795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fa42d0 0x55cfe3faa110 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- 172.21.15.116:0/608578795 >> 172.21.15.116:0/608578795 conn(0x55cfe3f9ea00 msgr2=0x55cfe3fa06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- 172.21.15.116:0/608578795 shutdown_connections 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- 172.21.15.116:0/608578795 wait complete. 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.551+0000 7fb23eb6f740 1 -- start start 2024-03-19T02:34:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 Processor -- start 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fb57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341adfe0 con 0x55cfe3fad7d0 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341ad910 con 0x55cfe3fa4bf0 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341aab70 con 0x55cfe3fa42d0 2024-03-19T02:34:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23eb6f740 10 client.0 _ll_register_callbacks cb 0x7fb230009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32808/0 (socket says 172.21.15.116:32808) 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb239c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fb57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 -- 172.21.15.116:0/404639194 learned_addr learned my addr 172.21.15.116:0/404639194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23943c640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 -- 172.21.15.116:0/404639194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 msgr2=0x55cfe3fb57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fb57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 -- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 msgr2=0x55cfe3fb5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb23a43e640 1 -- 172.21.15.116:0/404639194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfe3fb1400 con 0x55cfe3fad7d0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.552+0000 7fb239c3d640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fb57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23a43e640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb22c09e010 tx=0x7fb22c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ec6e7d5ee28f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0b9070 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb23003a040 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7fb22c0a3df0 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb22c09b8e0 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 10 client.17532 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cfe4023a00 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb23003a040 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb22c01f020 con 0x55cfe3fad7d0 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 10 client.17532 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 10 client.17532 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 20 client.17532 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe4024150 0x55cfe4026570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 4 client.17532 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.553+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfe3f149f0 con 0x55cfe4024150 2024-03-19T02:34:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb239c3d640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe4024150 0x55cfe4026570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb239c3d640 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe4024150 0x55cfe4026570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d87459f01bc45f7 server_cookie=cced25321698b925 in_seq=0 out_seq=0 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb21d7fa640 10 client.17532 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb21d7fa640 1 -- 172.21.15.116:0/404639194 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfe3f149f0 con 0x55cfe4024150 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb21d7fa640 10 client.17532 handle_command_reply: tid=0 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 client.17532 shutdown 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 msgr2=0x55cfe3fb0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb22c09e010 tx=0x7fb22c0098e0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 shutdown_connections 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cfe4024150 0x55cfe4026570 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfe3fad7d0 0x55cfe3fb0c00 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cfe3fa4bf0 0x55cfe3fb5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb21d7fa640 0 client.17532 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 --2- 172.21.15.116:0/404639194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cfe3fa42d0 0x55cfe3fb57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 >> 172.21.15.116:0/404639194 conn(0x55cfe3f9ea00 msgr2=0x55cfe3f9d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 shutdown_connections 2024-03-19T02:34:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 1 -- 172.21.15.116:0/404639194 wait complete. 2024-03-19T02:34:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.554+0000 7fb23eb6f740 20 client.17532 trim_cache size 0 max 16384 2024-03-19T02:34:11.562 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 msgr2=0x7fb2200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 0x7fb2200a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb230005ab0 tx=0x7fb230005fd0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 msgr2=0x7fb2341a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00efd0 tx=0x7fb22c00c6a0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 shutdown_connections 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2200a2970 0x7fb2200a4e30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2341066b0 0x7fb2341a3e70 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb234101060 0x7fb234199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 --2- 172.21.15.116:0/208885683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2341006a0 0x7fb234198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 >> 172.21.15.116:0/208885683 conn(0x7fb2340778d0 msgr2=0x7fb2341030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 shutdown_connections 2024-03-19T02:34:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.555+0000 7fb23bea1640 1 -- 172.21.15.116:0/208885683 wait complete. 2024-03-19T02:34:11.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 -- 172.21.15.116:0/1226207049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0010bd80 msgr2=0x7efd0010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 --2- 172.21.15.116:0/1226207049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0010bd80 0x7efd0010e220 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009ca0 tx=0x7efcf802c180 comp rx=0 tx=0).stop 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 -- 172.21.15.116:0/1226207049 shutdown_connections 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 --2- 172.21.15.116:0/1226207049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd0010bd80 0x7efd0010e220 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 --2- 172.21.15.116:0/1226207049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd00103360 0x7efd0010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 --2- 172.21.15.116:0/1226207049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00102a40 0x7efd00102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 -- 172.21.15.116:0/1226207049 >> 172.21.15.116:0/1226207049 conn(0x7efd000778d0 msgr2=0x7efd00077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 -- 172.21.15.116:0/1226207049 shutdown_connections 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 -- 172.21.15.116:0/1226207049 wait complete. 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.680+0000 7efd07be1640 1 Processor -- start 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 -- start start 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 0x7efd00198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 0x7efd0019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd00111a30 con 0x7efd00102a40 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd001118b0 con 0x7efd00103360 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd07be1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd00111bb0 con 0x7efd0010bd80 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32826/0 (socket says 172.21.15.116:32826) 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 -- 172.21.15.116:0/3921853303 learned_addr learned my addr 172.21.15.116:0/3921853303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0517c640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 0x7efd00198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0617e640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 0x7efd0019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 -- 172.21.15.116:0/3921853303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 msgr2=0x7efd0019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 0x7efd0019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 -- 172.21.15.116:0/3921853303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 msgr2=0x7efd00198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 0x7efd00198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 -- 172.21.15.116:0/3921853303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0019cf20 con 0x7efd00102a40 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0517c640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 0x7efd00198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0617e640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 0x7efd0019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.681+0000 7efd0597d640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00ea60 tx=0x7efcfc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.682+0000 7efcf6ffd640 1 -- 172.21.15.116:0/3921853303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00cde0 con 0x7efd00102a40 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.682+0000 7efcf6ffd640 1 -- 172.21.15.116:0/3921853303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcfc0035f0 con 0x7efd00102a40 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.682+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0019d210 con 0x7efd00102a40 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.682+0000 7efcf6ffd640 1 -- 172.21.15.116:0/3921853303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc010690 con 0x7efd00102a40 2024-03-19T02:34:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.682+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001a4aa0 con 0x7efd00102a40 2024-03-19T02:34:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.684+0000 7efcf6ffd640 1 -- 172.21.15.116:0/3921853303 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7efcfc0037a0 con 0x7efd00102a40 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.684+0000 7efd0a7c5740 1 Processor -- start 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 -- start start 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55957102f4b0 con 0x5595710ccbf0 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559570fd7d70 con 0x5595710cc2d0 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55957102b550 con 0x5595710d57d0 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efcf6ffd640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 0x7efcec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0597d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58690/0 (socket says 172.21.15.116:58690) 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 -- 172.21.15.116:0/1434782014 learned_addr learned my addr 172.21.15.116:0/1434782014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0617e640 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0617e640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 0x7efcec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efcf6ffd640 1 -- 172.21.15.116:0/3921853303 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efcfc0cf530 con 0x7efd00102a40 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 -- 172.21.15.116:0/1434782014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 msgr2=0x5595710cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 -- 172.21.15.116:0/1434782014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 msgr2=0x5595710d5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.685+0000 7efd0517c640 1 -- 172.21.15.116:0/1434782014 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559570ffa210 con 0x5595710d57d0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.686+0000 7efd0517c640 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efcf0011470 tx=0x7efcf0011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47816607629ea49d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.686+0000 7efd0617e640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 0x7efcec0a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcf8004af0 tx=0x7efcf8005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.686+0000 7efcebfff640 1 -- 172.21.15.116:0/1434782014 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf001e070 con 0x5595710d57d0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.686+0000 7efcebfff640 1 -- 172.21.15.116:0/1434782014 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf0004310 con 0x5595710d57d0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.686+0000 7efcebfff640 1 -- 172.21.15.116:0/1434782014 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf000f8a0 con 0x5595710d57d0 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1434782014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 msgr2=0x5595710d7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efcf0011470 tx=0x7efcf0011930 comp rx=0 tx=0).stop 2024-03-19T02:34:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1434782014 shutdown_connections 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d7bc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1434782014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1434782014 >> 172.21.15.116:0/1434782014 conn(0x5595710c6a00 msgr2=0x5595710c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1434782014 shutdown_connections 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1434782014 wait complete. 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 -- start start 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.687+0000 7efd0a7c5740 1 Processor -- start 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595710d85c0 con 0x5595710ccbf0 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559570feb310 con 0x5595710cc2d0 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5595710d8740 con 0x5595710d57d0 2024-03-19T02:34:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 10 client.0 _ll_register_callbacks cb 0x559570fdf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0617e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0597d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32840/0 (socket says 172.21.15.116:32840) 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0617e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58692/0 (socket says 172.21.15.116:58692) 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 -- 172.21.15.116:0/884882347 learned_addr learned my addr 172.21.15.116:0/884882347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 -- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 msgr2=0x5595710d32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 -- 172.21.15.116:0/884882347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 msgr2=0x5595710d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 -- 172.21.15.116:0/884882347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559570d433f0 con 0x5595710ccbf0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0597d640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0517c640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efcf00096e0 tx=0x7efcf0005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdcf1067d344d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0027020 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efcf00048d0 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.688+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcf0004310 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efcf0032050 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 10 client.17544 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559570e0d7d0 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf0004310 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efcf000f620 con 0x5595710ccbf0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 10 client.17544 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 10 client.17544 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 20 client.17544 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5595710e5aa0 0x5595710e7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 4 client.17544 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55957108d3e0 con 0x5595710e5aa0 2024-03-19T02:34:11.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.689+0000 7efd0597d640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5595710e5aa0 0x5595710e7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efd0597d640 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5595710e5aa0 0x5595710e7e90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a72783a689ccfa47 server_cookie=90c9cb1f9d4835aa in_seq=0 out_seq=0 2024-03-19T02:34:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efce97fa640 10 client.17544 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efce97fa640 1 -- 172.21.15.116:0/884882347 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55957108d3e0 con 0x5595710e5aa0 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efce97fa640 10 client.17544 handle_command_reply: tid=0 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efd0a7c5740 1 client.17544 shutdown 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 msgr2=0x5595710d2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.690+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efcf00096e0 tx=0x7efcf0005760 comp rx=0 tx=0).stop 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 shutdown_connections 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5595710e5aa0 0x5595710e7e90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710d57d0 0x5595710d32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710ccbf0 0x5595710d2da0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efce97fa640 0 client.17544 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/884882347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 >> 172.21.15.116:0/884882347 conn(0x5595710c6a00 msgr2=0x5595710c8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 shutdown_connections 2024-03-19T02:34:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 1 -- 172.21.15.116:0/884882347 wait complete. 2024-03-19T02:34:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.691+0000 7efd0a7c5740 20 client.17544 trim_cache size 0 max 16384 2024-03-19T02:34:11.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 Processor -- start 2024-03-19T02:34:11.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 -- start start 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710e1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 0x5595710ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd001aed80 con 0x5595710d57d0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd001b08c0 con 0x5595710cc2d0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd001af450 con 0x5595710ccbf0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0617e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 0x5595710ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43106/0 (socket says 172.21.15.116:43106) 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 -- 172.21.15.116:0/330770193 learned_addr learned my addr 172.21.15.116:0/330770193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0617e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 0x5595710ed6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32854/0 (socket says 172.21.15.116:32854) 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0517c640 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710e1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 -- 172.21.15.116:0/330770193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 msgr2=0x5595710e1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710e1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 -- 172.21.15.116:0/330770193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 msgr2=0x5595710ed6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 0x5595710ed6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0597d640 1 -- 172.21.15.116:0/330770193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595710cca20 con 0x5595710cc2d0 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.707+0000 7efd0517c640 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710e1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0597d640 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00d010 tx=0x7efcfc0d1980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cd3c7dcf47fa52b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efceb7fe640 1 -- 172.21.15.116:0/330770193 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc09be00 con 0x5595710cc2d0 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efceb7fe640 1 -- 172.21.15.116:0/330770193 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcfc09b700 con 0x5595710cc2d0 2024-03-19T02:34:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efceb7fe640 1 -- 172.21.15.116:0/330770193 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc097d70 con 0x5595710cc2d0 2024-03-19T02:34:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 -- 172.21.15.116:0/330770193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 msgr2=0x5595710d4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00d010 tx=0x7efcfc0d1980 comp rx=0 tx=0).stop 2024-03-19T02:34:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 -- 172.21.15.116:0/330770193 shutdown_connections 2024-03-19T02:34:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710d57d0 0x5595710ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710e1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/330770193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710cc2d0 0x5595710d4f90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 -- 172.21.15.116:0/330770193 >> 172.21.15.116:0/330770193 conn(0x5595710c6a00 msgr2=0x5595710c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 -- 172.21.15.116:0/330770193 shutdown_connections 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.708+0000 7efd0a7c5740 1 -- 172.21.15.116:0/330770193 wait complete. 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 -- start start 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 Processor -- start 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710d9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710d57d0 0x5595710d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd001b01f0 con 0x5595710cc2d0 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd001afb20 con 0x5595710d57d0 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efd001acd80 con 0x5595710ccbf0 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0a7c5740 10 client.0 _ll_register_callbacks cb 0x7efcf001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0597d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0597d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32862/0 (socket says 172.21.15.116:32862) 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0597d640 1 -- 172.21.15.116:0/1022742777 learned_addr learned my addr 172.21.15.116:0/1022742777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0617e640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710d57d0 0x5595710d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.709+0000 7efd0517c640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710d9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 -- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 msgr2=0x5595710d9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710d9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 -- 172.21.15.116:0/1022742777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710d57d0 msgr2=0x5595710d9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710d57d0 0x5595710d9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 -- 172.21.15.116:0/1022742777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55957114ad10 con 0x5595710cc2d0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0597d640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009920 tx=0x7efcfc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3a7c9782acfaa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc09f850 con 0x5595710cc2d0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9598+0+0 (secure 0 0 0) 0x7efcfc0cca70 con 0x5595710cc2d0 2024-03-19T02:34:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcfc09b700 con 0x5595710cc2d0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efcfc0a3ea0 con 0x5595710cc2d0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0a7c5740 10 client.17550 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55957114bf50 con 0x5595710cc2d0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.710+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcfc09b700 con 0x5595710cc2d0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7efcfc0b42d0 con 0x5595710cc2d0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 10 client.17550 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 10 client.17550 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 20 client.17550 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55957114c560 0x55957114e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 4 client.17550 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55957103c9f0 con 0x55957114c560 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0517c640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55957114c560 0x55957114e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efce97fa640 10 client.17550 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0517c640 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55957114c560 0x55957114e950 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52ce5ac7881f053 server_cookie=52f4d10841bd21a6 in_seq=0 out_seq=0 2024-03-19T02:34:11.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efce97fa640 1 -- 172.21.15.116:0/1022742777 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55957103c9f0 con 0x55957114c560 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efce97fa640 10 client.17550 handle_command_reply: tid=0 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 1 client.17550 shutdown 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 msgr2=0x5595710d91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.711+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009920 tx=0x7efcfc00ef90 comp rx=0 tx=0).stop 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 shutdown_connections 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55957114c560 0x55957114e950 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595710d57d0 0x5595710d9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595710ccbf0 0x5595710d9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 --2- 172.21.15.116:0/1022742777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595710cc2d0 0x5595710d91d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efce97fa640 0 client.17550 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 >> 172.21.15.116:0/1022742777 conn(0x5595710c6a00 msgr2=0x5595710c7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 shutdown_connections 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 1 -- 172.21.15.116:0/1022742777 wait complete. 2024-03-19T02:34:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd0a7c5740 20 client.17550 trim_cache size 0 max 16384 2024-03-19T02:34:11.719 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 msgr2=0x7efcec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 0x7efcec0a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcf8004af0 tx=0x7efcf8005c50 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 msgr2=0x7efd00197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00ea60 tx=0x7efcfc00ef20 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 shutdown_connections 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efcec0a2970 0x7efcec0a4e30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd0010bd80 0x7efd0019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd00103360 0x7efd00198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 --2- 172.21.15.116:0/3921853303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd00102a40 0x7efd00197f30 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 >> 172.21.15.116:0/3921853303 conn(0x7efd000778d0 msgr2=0x7efd000fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 shutdown_connections 2024-03-19T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.712+0000 7efd07be1640 1 -- 172.21.15.116:0/3921853303 wait complete. 2024-03-19T02:34:11.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:34:11.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 202465 3.25 MB/sec execute 214 sec latency 21.805 ms 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 -- 172.21.15.116:0/422672380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc148111500 msgr2=0x7fc1481190d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 --2- 172.21.15.116:0/422672380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc148111500 0x7fc1481190d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc140009ca0 tx=0x7fc14002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 -- 172.21.15.116:0/422672380 shutdown_connections 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 --2- 172.21.15.116:0/422672380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc148111500 0x7fc1481190d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 --2- 172.21.15.116:0/422672380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810cdf0 0x7fc148110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 --2- 172.21.15.116:0/422672380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc14810c440 0x7fc14810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 -- 172.21.15.116:0/422672380 >> 172.21.15.116:0/422672380 conn(0x7fc14806c8e0 msgr2=0x7fc14810a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 -- 172.21.15.116:0/422672380 shutdown_connections 2024-03-19T02:34:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 -- 172.21.15.116:0/422672380 wait complete. 2024-03-19T02:34:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.844+0000 7fc14f84b640 1 Processor -- start 2024-03-19T02:34:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 -- start start 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 0x7fc1481a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 0x7fc1481a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc14811c450 con 0x7fc1481a27d0 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc14811c2d0 con 0x7fc148111500 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14f84b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc14811c5d0 con 0x7fc14810c440 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 0x7fc1481a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32890/0 (socket says 172.21.15.116:32890) 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14dde8640 1 -- 172.21.15.116:0/521707141 learned_addr learned my addr 172.21.15.116:0/521707141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14cde6640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 0x7fc1481a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 -- 172.21.15.116:0/521707141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 msgr2=0x7fc1481a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 0x7fc1481a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 -- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 msgr2=0x7fc1481a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 -- 172.21.15.116:0/521707141 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1481a3360 con 0x7fc14810c440 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14cde6640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 0x7fc1481a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14dde8640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.845+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 0x7fc1481a8610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e9f0 tx=0x7fc14400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.846+0000 7fc13e7fc640 1 -- 172.21.15.116:0/521707141 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14400ccc0 con 0x7fc14810c440 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.846+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1481a3650 con 0x7fc14810c440 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.846+0000 7fc13e7fc640 1 -- 172.21.15.116:0/521707141 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1440108e0 con 0x7fc14810c440 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.846+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481af830 con 0x7fc14810c440 2024-03-19T02:34:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.846+0000 7fc13e7fc640 1 -- 172.21.15.116:0/521707141 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144018820 con 0x7fc14810c440 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 Processor -- start 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc13e7fc640 1 -- 172.21.15.116:0/521707141 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc144010a90 con 0x7fc14810c440 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 -- start start 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d669a44b0 con 0x560d66a41bf0 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560d6694cd70 con 0x560d66a412d0 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.848+0000 7fc15243c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d669a0550 con 0x560d66a4a7d0 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32902/0 (socket says 172.21.15.116:32902) 2024-03-19T02:34:11.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 -- 172.21.15.116:0/3015356684 learned_addr learned my addr 172.21.15.116:0/3015356684 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14cde6640 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc13e7fc640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 0x7fc1340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 -- 172.21.15.116:0/3015356684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 msgr2=0x560d66a4cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 -- 172.21.15.116:0/3015356684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 msgr2=0x560d66a416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 -- 172.21.15.116:0/3015356684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d6696f210 con 0x560d66a41bf0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14dde8640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 0x7fc1340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc13e7fc640 1 -- 172.21.15.116:0/521707141 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc1440cf530 con 0x7fc14810c440 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.849+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc1440cc810 tx=0x7fc1440c6200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be134bba51572392 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc14dde8640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 0x7fc1340a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc140033000 tx=0x7fc14002e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc1337fe640 1 -- 172.21.15.116:0/3015356684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14409b630 con 0x560d66a41bf0 2024-03-19T02:34:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc1337fe640 1 -- 172.21.15.116:0/3015356684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc144026050 con 0x560d66a41bf0 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc1337fe640 1 -- 172.21.15.116:0/3015356684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1440a2c50 con 0x560d66a41bf0 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 -- 172.21.15.116:0/3015356684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 msgr2=0x560d66a4a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc1440cc810 tx=0x7fc1440c6200 comp rx=0 tx=0).stop 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 -- 172.21.15.116:0/3015356684 shutdown_connections 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4a290 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 --2- 172.21.15.116:0/3015356684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 -- 172.21.15.116:0/3015356684 >> 172.21.15.116:0/3015356684 conn(0x560d66a3ba00 msgr2=0x560d66a3bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 -- 172.21.15.116:0/3015356684 shutdown_connections 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.850+0000 7fc15243c740 1 -- 172.21.15.116:0/3015356684 wait complete. 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 -- start start 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 Processor -- start 2024-03-19T02:34:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a47800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a41bf0 0x560d66a47d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d66a4d500 con 0x560d66a4a7d0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560d66960310 con 0x560d66a412d0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d66a4d680 con 0x560d66a41bf0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc15243c740 10 client.0 _ll_register_callbacks cb 0x560d669542d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32906/0 (socket says 172.21.15.116:32906) 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc14dde8640 1 -- 172.21.15.116:0/1507371210 learned_addr learned my addr 172.21.15.116:0/1507371210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a47800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.851+0000 7fc14cde6640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a41bf0 0x560d66a47d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 -- 172.21.15.116:0/1507371210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a41bf0 msgr2=0x560d66a47d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a41bf0 0x560d66a47d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 -- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 msgr2=0x560d66a47800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a47800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 -- 172.21.15.116:0/1507371210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d666b83f0 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a47800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc14dde8640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1400096a0 tx=0x7fc140005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3140a1064f4d3af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc130ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140039040 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc140041cf0 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc144026050 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc14005b050 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc15243c740 10 client.17562 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.852+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560d667827d0 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc144026050 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc14003f040 con 0x560d66a4a7d0 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 10 client.17562 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 20 client.17562 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66a5ab80 0x560d66a5cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 4 client.17562 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d66a023e0 con 0x560d66a5ab80 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66a5ab80 0x560d66a5cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66a5ab80 0x560d66a5cfe0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73a8cc2bcf47479a server_cookie=b5611b5e9f160c0e in_seq=0 out_seq=0 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.853+0000 7fc130ff9640 10 client.17562 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.854+0000 7fc130ff9640 1 -- 172.21.15.116:0/1507371210 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d66a023e0 con 0x560d66a5ab80 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.854+0000 7fc130ff9640 10 client.17562 handle_command_reply: tid=0 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.854+0000 7fc15243c740 1 client.17562 shutdown 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 msgr2=0x560d66a48280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1400096a0 tx=0x7fc140005d80 comp rx=0 tx=0).stop 2024-03-19T02:34:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 shutdown_connections 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66a5ab80 0x560d66a5cfe0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a48280 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a41bf0 0x560d66a47d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 --2- 172.21.15.116:0/1507371210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a47800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 >> 172.21.15.116:0/1507371210 conn(0x560d66a3ba00 msgr2=0x560d66a3d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc130ff9640 0 client.17562 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 shutdown_connections 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 1 -- 172.21.15.116:0/1507371210 wait complete. 2024-03-19T02:34:11.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.855+0000 7fc15243c740 20 client.17562 trim_cache size 0 max 16384 2024-03-19T02:34:11.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 Processor -- start 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 -- start start 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a412d0 0x560d66a49f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a41bf0 0x560d66a562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481b96a0 con 0x560d66a4a7d0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481bb1e0 con 0x560d66a41bf0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc15243c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481b9d70 con 0x560d66a412d0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc14dde8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32918/0 (socket says 172.21.15.116:32918) 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc14dde8640 1 -- 172.21.15.116:0/2718575066 learned_addr learned my addr 172.21.15.116:0/2718575066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc14cde6640 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a41bf0 0x560d66a562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.871+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a412d0 0x560d66a49f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 -- 172.21.15.116:0/2718575066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a412d0 msgr2=0x560d66a49f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a412d0 0x560d66a49f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 -- 172.21.15.116:0/2718575066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a41bf0 msgr2=0x560d66a562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a41bf0 0x560d66a562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 -- 172.21.15.116:0/2718575066 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d66a567e0 con 0x560d66a4a7d0 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc14dde8640 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc140009dc0 tx=0x7fc140005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b4f92601cf20f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc132ffd640 1 -- 172.21.15.116:0/2718575066 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140063020 con 0x560d66a4a7d0 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc132ffd640 1 -- 172.21.15.116:0/2718575066 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc140046030 con 0x560d66a4a7d0 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.872+0000 7fc132ffd640 1 -- 172.21.15.116:0/2718575066 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140039070 con 0x560d66a4a7d0 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- 172.21.15.116:0/2718575066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 msgr2=0x560d66a3d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc140009dc0 tx=0x7fc140005bf0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- 172.21.15.116:0/2718575066 shutdown_connections 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a4a7d0 0x560d66a3d560 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a41bf0 0x560d66a562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- 172.21.15.116:0/2718575066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a412d0 0x560d66a49f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- 172.21.15.116:0/2718575066 >> 172.21.15.116:0/2718575066 conn(0x560d66a3ba00 msgr2=0x560d66a3ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- 172.21.15.116:0/2718575066 shutdown_connections 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- 172.21.15.116:0/2718575066 wait complete. 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- start start 2024-03-19T02:34:11.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 Processor -- start 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a4e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481bab10 con 0x560d66a41bf0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481ba440 con 0x560d66a412d0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481b76a0 con 0x560d66a4a7d0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.873+0000 7fc15243c740 10 client.0 _ll_register_callbacks cb 0x7fc138005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32930/0 (socket says 172.21.15.116:32930) 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 -- 172.21.15.116:0/2839475666 learned_addr learned my addr 172.21.15.116:0/2839475666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a4e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14dde8640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 -- 172.21.15.116:0/2839475666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 msgr2=0x560d66a4eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 -- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 msgr2=0x560d66a4e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a4e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 -- 172.21.15.116:0/2839475666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d66abfee0 con 0x560d66a41bf0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc14cde6640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc1481a3ad0 tx=0x7fc13800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf8cb3e25ee8992 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc130ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138023060 con 0x560d66a41bf0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc138004250 con 0x560d66a41bf0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc140046030 con 0x560d66a41bf0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc13802a050 con 0x560d66a41bf0 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc15243c740 10 client.17568 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560d66ac1180 con 0x560d66a41bf0 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.874+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc140046030 con 0x560d66a41bf0 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc138010430 con 0x560d66a41bf0 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 10 client.17568 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 20 client.17568 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66ac1960 0x560d66ac3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 4 client.17568 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d669b19f0 con 0x560d66ac1960 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66ac1960 0x560d66ac3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc130ff9640 10 client.17568 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.875+0000 7fc14d5e7640 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66ac1960 0x560d66ac3d80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ea003073c0be6 server_cookie=8a2cfec7f7645717 in_seq=0 out_seq=0 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc130ff9640 1 -- 172.21.15.116:0/2839475666 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d669b19f0 con 0x560d66ac1960 2024-03-19T02:34:11.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc130ff9640 10 client.17568 handle_command_reply: tid=0 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 client.17568 shutdown 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 msgr2=0x560d66a4e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc1481a3ad0 tx=0x7fc13800f790 comp rx=0 tx=0).stop 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 shutdown_connections 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560d66ac1960 0x560d66ac3d80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc130ff9640 0 client.17568 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d66a4a7d0 0x560d66a4eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d66a41bf0 0x560d66a4e630 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 --2- 172.21.15.116:0/2839475666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d66a412d0 0x560d66a4e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 >> 172.21.15.116:0/2839475666 conn(0x560d66a3ba00 msgr2=0x560d66a3d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 shutdown_connections 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.876+0000 7fc15243c740 1 -- 172.21.15.116:0/2839475666 wait complete. 2024-03-19T02:34:11.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc15243c740 20 client.17568 trim_cache size 0 max 16384 2024-03-19T02:34:11.884 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 msgr2=0x7fc1340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 0x7fc1340a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc140033000 tx=0x7fc14002e040 comp rx=0 tx=0).stop 2024-03-19T02:34:11.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 msgr2=0x7fc1481a8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 0x7fc1481a8610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e9f0 tx=0x7fc14400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 shutdown_connections 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1340a2970 0x7fc1340a4e30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1481a27d0 0x7fc1481a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc148111500 0x7fc1481a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 --2- 172.21.15.116:0/521707141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc14810c440 0x7fc1481a8610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 >> 172.21.15.116:0/521707141 conn(0x7fc14806c8e0 msgr2=0x7fc14810f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 shutdown_connections 2024-03-19T02:34:11.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:11.877+0000 7fc14f84b640 1 -- 172.21.15.116:0/521707141 wait complete. 2024-03-19T02:34:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:11 smithi116 ceph-mon[32263]: pgmap v195: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-03-19T02:34:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:11 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/866811541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:11 smithi116 ceph-mon[32263]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:34:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:11 smithi162 ceph-mon[35761]: pgmap v195: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-03-19T02:34:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:11 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/866811541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:11 smithi162 ceph-mon[35761]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:34:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:11 smithi154 ceph-mon[35693]: pgmap v195: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-03-19T02:34:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:11 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/866811541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:11 smithi154 ceph-mon[35693]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:34:12.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 202878 3.24 MB/sec execute 215 sec latency 41.126 ms 2024-03-19T02:34:12.890 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:34:12.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:34:12.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:34:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 -- 172.21.15.116:0/3990444792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 msgr2=0x7fc25c100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 --2- 172.21.15.116:0/3990444792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c100160 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fc250009930 tx=0x7fc25002bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 -- 172.21.15.116:0/3990444792 shutdown_connections 2024-03-19T02:34:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 --2- 172.21.15.116:0/3990444792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 --2- 172.21.15.116:0/3990444792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c100160 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 --2- 172.21.15.116:0/3990444792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 0x7fc25c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 -- 172.21.15.116:0/3990444792 >> 172.21.15.116:0/3990444792 conn(0x7fc25c076c40 msgr2=0x7fc25c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 -- 172.21.15.116:0/3990444792 shutdown_connections 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.007+0000 7fc261597640 1 -- 172.21.15.116:0/3990444792 wait complete. 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 Processor -- start 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 -- start start 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 0x7fc25c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c111b40 con 0x7fc25c0ffce0 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1119c0 con 0x7fc25c1067c0 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc261597640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c111cc0 con 0x7fc25c1006a0 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 0x7fc25c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58766/0 (socket says 172.21.15.116:58766) 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25a7fc640 1 -- 172.21.15.116:0/4054684636 learned_addr learned my addr 172.21.15.116:0/4054684636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.008+0000 7fc25affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32944/0 (socket says 172.21.15.116:32944) 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 -- 172.21.15.116:0/4054684636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 msgr2=0x7fc25c19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 0x7fc25c19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 -- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 msgr2=0x7fc25c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 -- 172.21.15.116:0/4054684636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25c19d0b0 con 0x7fc25c1006a0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc250009230 tx=0x7fc25002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc243fff640 1 -- 172.21.15.116:0/4054684636 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250043020 con 0x7fc25c1006a0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25c19d340 con 0x7fc25c1006a0 2024-03-19T02:34:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc243fff640 1 -- 172.21.15.116:0/4054684636 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc25002ca70 con 0x7fc25c1006a0 2024-03-19T02:34:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc243fff640 1 -- 172.21.15.116:0/4054684636 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250003a50 con 0x7fc25c1006a0 2024-03-19T02:34:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.009+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25c1a4c30 con 0x7fc25c1006a0 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.011+0000 7fc243fff640 1 -- 172.21.15.116:0/4054684636 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc250046020 con 0x7fc25c1006a0 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 Processor -- start 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 -- start start 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc243fff640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 0x7fc2440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90950 0x5590b4a9d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590b49f4160 con 0x5590b4a9dc90 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5590b499ce10 con 0x5590b4a90030 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc264186740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590b49f0200 con 0x5590b4a90950 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 0x7fc2440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc243fff640 1 -- 172.21.15.116:0/4054684636 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc2500f1110 con 0x7fc25c1006a0 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43222/0 (socket says 172.21.15.116:43222) 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 -- 172.21.15.116:0/3892808302 learned_addr learned my addr 172.21.15.116:0/3892808302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90950 0x5590b4a9d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 -- 172.21.15.116:0/3892808302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90950 msgr2=0x5590b4a9d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90950 0x5590b4a9d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 -- 172.21.15.116:0/3892808302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 msgr2=0x5590b4aa0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25affd640 1 -- 172.21.15.116:0/3892808302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b47d07e0 con 0x5590b4a90030 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.012+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 0x7fc2440a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc254005fd0 tx=0x7fc254013b60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc25affd640 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009cf0 tx=0x7fc24c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edf118d2276868e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc2417fa640 1 -- 172.21.15.116:0/3892808302 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01e070 con 0x5590b4a90030 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc2417fa640 1 -- 172.21.15.116:0/3892808302 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c009de0 con 0x5590b4a90030 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc2417fa640 1 -- 172.21.15.116:0/3892808302 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c019450 con 0x5590b4a90030 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 -- 172.21.15.116:0/3892808302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 msgr2=0x5590b4a90410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009cf0 tx=0x7fc24c011810 comp rx=0 tx=0).stop 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 -- 172.21.15.116:0/3892808302 shutdown_connections 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90950 0x5590b4a9d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 --2- 172.21.15.116:0/3892808302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90030 0x5590b4a90410 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 -- 172.21.15.116:0/3892808302 >> 172.21.15.116:0/3892808302 conn(0x5590b4a8aab0 msgr2=0x5590b4a8ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.013+0000 7fc264186740 1 -- 172.21.15.116:0/3892808302 shutdown_connections 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 -- 172.21.15.116:0/3892808302 wait complete. 2024-03-19T02:34:13.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 -- start start 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 Processor -- start 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4a9b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4a9b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590b4aa0ad0 con 0x5590b4a90950 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5590b49b01a0 con 0x5590b4a9dc90 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5590b4aa0c50 con 0x5590b4a90030 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc264186740 10 client.0 _ll_register_callbacks cb 0x5590b4707c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc25affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc25a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4a9b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc25affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58784/0 (socket says 172.21.15.116:58784) 2024-03-19T02:34:13.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc25affd640 1 -- 172.21.15.116:0/3694009013 learned_addr learned my addr 172.21.15.116:0/3694009013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.014+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4a9b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 -- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 msgr2=0x5590b4a9b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4a9b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 -- 172.21.15.116:0/3694009013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 msgr2=0x5590b4a9b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4a9b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 -- 172.21.15.116:0/3694009013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b4830f70 con 0x5590b4a90030 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4a9b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc25affd640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000c00 tx=0x7fc24c0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6379ea102552a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc23eff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01e040 con 0x5590b4a90030 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc24c005510 con 0x5590b4a90030 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24c009de0 con 0x5590b4a90030 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc24c030050 con 0x5590b4a90030 2024-03-19T02:34:13.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc264186740 10 client.27334 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5590b46c8170 con 0x5590b4a90030 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.015+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c009de0 con 0x5590b4a90030 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc24c021070 con 0x5590b4a90030 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 10 client.27334 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 10 client.27334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 20 client.27334 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4aadf90 0x5590b4ab0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 4 client.27334 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590b4a016a0 con 0x5590b4aadf90 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4aadf90 0x5590b4ab0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4aadf90 0x5590b4ab0380 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d00ca7c09d9c44f server_cookie=77353a8032546d2b in_seq=0 out_seq=0 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.016+0000 7fc23eff5640 10 client.27334 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc23eff5640 1 -- 172.21.15.116:0/3694009013 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590b4a016a0 con 0x5590b4aadf90 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc23eff5640 10 client.27334 handle_command_reply: tid=0 2024-03-19T02:34:13.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 client.27334 shutdown 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 msgr2=0x5590b4a9abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000c00 tx=0x7fc24c0052f0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 shutdown_connections 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4aadf90 0x5590b4ab0380 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4a9b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4a9b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc23eff5640 0 client.27334 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 --2- 172.21.15.116:0/3694009013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9abe0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 >> 172.21.15.116:0/3694009013 conn(0x5590b4a8aab0 msgr2=0x5590b4a8c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 shutdown_connections 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 1 -- 172.21.15.116:0/3694009013 wait complete. 2024-03-19T02:34:13.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.017+0000 7fc264186740 20 client.27334 trim_cache size 0 max 16384 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 Processor -- start 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 -- start start 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90950 0x5590b4a96330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.036+0000 7fc264186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1ac9d0 con 0x5590b4a9dc90 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc264186740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1ae510 con 0x5590b4a90950 2024-03-19T02:34:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc264186740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1ad0a0 con 0x5590b4a90030 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32972/0 (socket says 172.21.15.116:32972) 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 -- 172.21.15.116:0/2464373095 learned_addr learned my addr 172.21.15.116:0/2464373095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25affd640 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90950 0x5590b4a96330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 -- 172.21.15.116:0/2464373095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 msgr2=0x5590b4a9d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 -- 172.21.15.116:0/2464373095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90950 msgr2=0x5590b4a96330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90950 0x5590b4a96330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 -- 172.21.15.116:0/2464373095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590b4a90780 con 0x5590b4a9dc90 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc254012370 tx=0x7fc25401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9be38b468037cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc240ff9640 1 -- 172.21.15.116:0/2464373095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254004b20 con 0x5590b4a9dc90 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc240ff9640 1 -- 172.21.15.116:0/2464373095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc254004cc0 con 0x5590b4a9dc90 2024-03-19T02:34:13.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.037+0000 7fc240ff9640 1 -- 172.21.15.116:0/2464373095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254004280 con 0x5590b4a9dc90 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- 172.21.15.116:0/2464373095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 msgr2=0x5590b4aa98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fc254012370 tx=0x7fc25401f000 comp rx=0 tx=0).stop 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- 172.21.15.116:0/2464373095 shutdown_connections 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a9dc90 0x5590b4aa98e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a90950 0x5590b4a96330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 --2- 172.21.15.116:0/2464373095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4a9d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- 172.21.15.116:0/2464373095 >> 172.21.15.116:0/2464373095 conn(0x5590b4a8aab0 msgr2=0x5590b4a892a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- 172.21.15.116:0/2464373095 shutdown_connections 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- 172.21.15.116:0/2464373095 wait complete. 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 -- start start 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.038+0000 7fc264186740 1 Processor -- start 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4aa1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4aa2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1ade40 con 0x5590b4a90950 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1ad770 con 0x5590b4a9dc90 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c1aa9d0 con 0x5590b4a90030 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 10 client.0 _ll_register_callbacks cb 0x7fc24c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4aa1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32988/0 (socket says 172.21.15.116:32988) 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4aa1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58798/0 (socket says 172.21.15.116:58798) 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 -- 172.21.15.116:0/2934667563 learned_addr learned my addr 172.21.15.116:0/2934667563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25b7fe640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4aa2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 -- 172.21.15.116:0/2934667563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 msgr2=0x5590b4aa1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4aa1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 -- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 msgr2=0x5590b4aa2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4aa2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 -- 172.21.15.116:0/2934667563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590b4b133e0 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc25a7fc640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc2500b9900 tx=0x7fc2500dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265fbe5016109dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc23eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500c15f0 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc2500b9b10 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.039+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc254004cc0 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc2500c5c40 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 10 client.17586 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5590b4b14760 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc254004cc0 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc2500bd700 con 0x5590b4a90950 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 10 client.17586 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 20 client.17586 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4b14e30 0x5590b4b17220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 4 client.17586 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:34:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5590b49fb9f0 con 0x5590b4b14e30 2024-03-19T02:34:13.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.040+0000 7fc25affd640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4b14e30 0x5590b4b17220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.041+0000 7fc25affd640 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4b14e30 0x5590b4b17220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7faf007675d490f8 server_cookie=dd2697fa2cf0913f in_seq=0 out_seq=0 2024-03-19T02:34:13.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.041+0000 7fc23eff5640 10 client.17586 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc23eff5640 1 -- 172.21.15.116:0/2934667563 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5590b49fb9f0 con 0x5590b4b14e30 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc23eff5640 10 client.17586 handle_command_reply: tid=0 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 client.17586 shutdown 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 msgr2=0x5590b4aa1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc2500b9900 tx=0x7fc2500dd040 comp rx=0 tx=0).stop 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 shutdown_connections 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5590b4b14e30 0x5590b4b17220 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590b4a9dc90 0x5590b4aa2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b4a90950 0x5590b4aa1fc0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 --2- 172.21.15.116:0/2934667563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5590b4a90030 0x5590b4aa1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc23eff5640 0 client.17586 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 >> 172.21.15.116:0/2934667563 conn(0x5590b4a8aab0 msgr2=0x5590b4a89b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 shutdown_connections 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 1 -- 172.21.15.116:0/2934667563 wait complete. 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.042+0000 7fc264186740 20 client.17586 trim_cache size 0 max 16384 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "dda4e632-aaea-4297-89f4-51a6481eff1c", 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:34:13.048 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 msgr2=0x7fc2440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 0x7fc2440a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc254005fd0 tx=0x7fc254013b60 comp rx=0 tx=0).stop 2024-03-19T02:34:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 msgr2=0x7fc25c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc250009230 tx=0x7fc25002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 shutdown_connections 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2440a2970 0x7fc2440a4e30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc25c1067c0 0x7fc25c19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc25c1006a0 0x7fc25c198600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 --2- 172.21.15.116:0/4054684636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc25c0ffce0 0x7fc25c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 >> 172.21.15.116:0/4054684636 conn(0x7fc25c076c40 msgr2=0x7fc25c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 shutdown_connections 2024-03-19T02:34:13.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.043+0000 7fc261597640 1 -- 172.21.15.116:0/4054684636 wait complete. 2024-03-19T02:34:13.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:34:13.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 -- 172.21.15.116:0/1630993461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 msgr2=0x7fc670114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 --2- 172.21.15.116:0/1630993461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc670114d20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc660009be0 tx=0x7fc66002bef0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 -- 172.21.15.116:0/1630993461 shutdown_connections 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 --2- 172.21.15.116:0/1630993461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc670114d20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 --2- 172.21.15.116:0/1630993461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc670108660 0x7fc670108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 --2- 172.21.15.116:0/1630993461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc67010df60 0x7fc67010e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 -- 172.21.15.116:0/1630993461 >> 172.21.15.116:0/1630993461 conn(0x7fc6700773d0 msgr2=0x7fc6700797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 -- 172.21.15.116:0/1630993461 shutdown_connections 2024-03-19T02:34:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.175+0000 7fc674c4b640 1 -- 172.21.15.116:0/1630993461 wait complete. 2024-03-19T02:34:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 Processor -- start 2024-03-19T02:34:13.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 -- start start 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc670108660 0x7fc67019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc67019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc670118110 con 0x7fc67010df60 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc670117f90 con 0x7fc670108660 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc674c4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc670118290 con 0x7fc670109020 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33008/0 (socket says 172.21.15.116:33008) 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3905995205 learned_addr learned my addr 172.21.15.116:0/3905995205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc670108660 0x7fc67019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66dd9b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc67019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3905995205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 msgr2=0x7fc67019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc67019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3905995205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc670108660 msgr2=0x7fc67019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc670108660 0x7fc67019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.176+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3905995205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6701a3650 con 0x7fc67010df60 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc660036040 tx=0x7fc660033a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc65f7fe640 1 -- 172.21.15.116:0/3905995205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660043020 con 0x7fc67010df60 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6701a38e0 con 0x7fc67010df60 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc65f7fe640 1 -- 172.21.15.116:0/3905995205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc660033e60 con 0x7fc67010df60 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc65f7fe640 1 -- 172.21.15.116:0/3905995205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660034430 con 0x7fc67010df60 2024-03-19T02:34:13.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.177+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6701ab1d0 con 0x7fc67010df60 2024-03-19T02:34:13.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.179+0000 7fc65f7fe640 1 -- 172.21.15.116:0/3905995205 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc660046020 con 0x7fc67010df60 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc65f7fe640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 0x7fc6580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 Processor -- start 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc65f7fe640 1 -- 172.21.15.116:0/3905995205 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc6600f16e0 con 0x7fc67010df60 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 0x7fc6580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 -- start start 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff59350 0x55cb1ff619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb1febc500 con 0x55cb1ff59350 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cb1fe64be0 con 0x55cb1ff58a30 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc677836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cb1feb85a0 con 0x55cb1ff61f30 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 0x7fc6580a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc6680096c0 tx=0x7fc668009260 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff59350 0x55cb1ff619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58826/0 (socket says 172.21.15.116:58826) 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.180+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3921826481 learned_addr learned my addr 172.21.15.116:0/3921826481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3921826481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 msgr2=0x55cb1ff58e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff58e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3921826481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff59350 msgr2=0x55cb1ff619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff59350 0x55cb1ff619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3921826481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1fe86000 con 0x55cb1ff61f30 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc6600bda00 tx=0x7fc6600bd5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dbd2265e1478a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc65cff9640 1 -- 172.21.15.116:0/3921826481 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600be170 con 0x55cb1ff61f30 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc65cff9640 1 -- 172.21.15.116:0/3921826481 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6600bda70 con 0x55cb1ff61f30 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.181+0000 7fc65cff9640 1 -- 172.21.15.116:0/3921826481 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600041d0 con 0x55cb1ff61f30 2024-03-19T02:34:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- 172.21.15.116:0/3921826481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 msgr2=0x55cb1ff64320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc6600bda00 tx=0x7fc6600bd5b0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- 172.21.15.116:0/3921826481 shutdown_connections 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff61f30 0x55cb1ff64320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff59350 0x55cb1ff619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 --2- 172.21.15.116:0/3921826481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff58e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- 172.21.15.116:0/3921826481 >> 172.21.15.116:0/3921826481 conn(0x55cb1ff53160 msgr2=0x55cb1ff53540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- 172.21.15.116:0/3921826481 shutdown_connections 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- 172.21.15.116:0/3921826481 wait complete. 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 -- start start 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 Processor -- start 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.182+0000 7fc677836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb1ff64ae0 con 0x55cb1ff61f30 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cb1fe78160 con 0x55cb1ff59350 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cb1ff64c60 con 0x55cb1ff58a30 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 10 client.0 _ll_register_callbacks cb 0x55cb1feb4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33014/0 (socket says 172.21.15.116:33014) 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58840/0 (socket says 172.21.15.116:58840) 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43272/0 (socket says 172.21.15.116:43272) 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3702639295 learned_addr learned my addr 172.21.15.116:0/3702639295 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3702639295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 msgr2=0x55cb1ff5ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 msgr2=0x55cb1ff5f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 -- 172.21.15.116:0/3702639295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1fc9a7d0 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66dd9b640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc660009370 tx=0x7fc660004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8899fa3a4a2012 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600eecd0 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc6600c5f90 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.183+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6600bda70 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc6600be170 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 10 client.17598 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cb1fbd03f0 con 0x55cb1ff61f30 2024-03-19T02:34:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6600bda70 con 0x55cb1ff61f30 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc6600f13c0 con 0x55cb1ff61f30 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 10 client.17598 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 20 client.17598 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ff71d20 0x55cb1ff74110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 4 client.17598 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb1fec9a40 con 0x55cb1ff71d20 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.184+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ff71d20 0x55cb1ff74110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc66e59c640 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ff71d20 0x55cb1ff74110 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f7400cdd41e544 server_cookie=42954b6224bc3b39 in_seq=0 out_seq=0 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc655ffb640 10 client.17598 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc655ffb640 1 -- 172.21.15.116:0/3702639295 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cb1fec9a40 con 0x55cb1ff71d20 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc655ffb640 10 client.17598 handle_command_reply: tid=0 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc677836740 1 client.17598 shutdown 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 msgr2=0x55cb1ff5f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.185+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fc660009370 tx=0x7fc660004400 comp rx=0 tx=0).stop 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 shutdown_connections 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ff71d20 0x55cb1ff74110 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5f8a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc655ffb640 0 client.17598 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 --2- 172.21.15.116:0/3702639295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff58a30 0x55cb1ff5ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 >> 172.21.15.116:0/3702639295 conn(0x55cb1ff53160 msgr2=0x55cb1ff54eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 shutdown_connections 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 1 -- 172.21.15.116:0/3702639295 wait complete. 2024-03-19T02:34:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.186+0000 7fc677836740 20 client.17598 trim_cache size 0 max 16384 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.203+0000 7fc677836740 1 Processor -- start 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.203+0000 7fc677836740 1 -- start start 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff61f30 0x55cb1ff79800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b51f0 con 0x55cb1ff58a30 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b6d30 con 0x55cb1ff61f30 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc677836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b58c0 con 0x55cb1ff59350 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33018/0 (socket says 172.21.15.116:33018) 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 -- 172.21.15.116:0/429116462 learned_addr learned my addr 172.21.15.116:0/429116462 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66dd9b640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff61f30 0x55cb1ff79800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 -- 172.21.15.116:0/429116462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 msgr2=0x55cb1ff5f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 -- 172.21.15.116:0/429116462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff61f30 msgr2=0x55cb1ff79800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff61f30 0x55cb1ff79800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 -- 172.21.15.116:0/429116462 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1fb900f0 con 0x55cb1ff58a30 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66dd9b640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:13.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc66e59c640 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x55cb1ff74770 tx=0x7fc668014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1da9a98c682c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.204+0000 7fc657fff640 1 -- 172.21.15.116:0/429116462 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66802a3b0 con 0x55cb1ff58a30 2024-03-19T02:34:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc657fff640 1 -- 172.21.15.116:0/429116462 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc66801fd00 con 0x55cb1ff58a30 2024-03-19T02:34:13.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc657fff640 1 -- 172.21.15.116:0/429116462 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc668005310 con 0x55cb1ff58a30 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 -- 172.21.15.116:0/429116462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 msgr2=0x55cb1ff6d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x55cb1ff74770 tx=0x7fc668014000 comp rx=0 tx=0).stop 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 -- 172.21.15.116:0/429116462 shutdown_connections 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff61f30 0x55cb1ff79800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 --2- 172.21.15.116:0/429116462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff58a30 0x55cb1ff6d790 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 -- 172.21.15.116:0/429116462 >> 172.21.15.116:0/429116462 conn(0x55cb1ff53160 msgr2=0x55cb1ff51950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 -- 172.21.15.116:0/429116462 shutdown_connections 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.205+0000 7fc677836740 1 -- 172.21.15.116:0/429116462 wait complete. 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 -- start start 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 Processor -- start 2024-03-19T02:34:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff69dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b6660 con 0x55cb1ff61f30 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b5f90 con 0x55cb1ff58a30 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6701b31f0 con 0x55cb1ff59350 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc677836740 10 client.0 _ll_register_callbacks cb 0x7fc6640097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33030/0 (socket says 172.21.15.116:33030) 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 -- 172.21.15.116:0/4188083221 learned_addr learned my addr 172.21.15.116:0/4188083221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66e59c640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff69dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 -- 172.21.15.116:0/4188083221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 msgr2=0x55cb1ff5b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 -- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 msgr2=0x55cb1ff69dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff69dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.206+0000 7fc66ed9d640 1 -- 172.21.15.116:0/4188083221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1ff5bf60 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc66ed9d640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc6600bd770 tx=0x7fc6600044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd0e8cb458536a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600bd3d0 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc6600c2d70 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc66801fd00 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc6600bda70 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 10 client.17604 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cb1ff5bf60 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc66801fd00 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc6600eec40 con 0x55cb1ff61f30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 10 client.17604 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 20 client.17604 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.207+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ffd7e90 0x55cb1ffda2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.208+0000 7fc677836740 4 client.17604 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.208+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cb1fec3d90 con 0x55cb1ffd7e90 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.208+0000 7fc66e59c640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ffd7e90 0x55cb1ffda2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.208+0000 7fc655ffb640 10 client.17604 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.208+0000 7fc66e59c640 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ffd7e90 0x55cb1ffda2b0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1bbdd4102ab99e server_cookie=55ff9ba7a88b5560 in_seq=0 out_seq=0 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc655ffb640 1 -- 172.21.15.116:0/4188083221 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cb1fec3d90 con 0x55cb1ffd7e90 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc655ffb640 10 client.17604 handle_command_reply: tid=0 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 client.17604 shutdown 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 msgr2=0x55cb1ff5ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc6600bd770 tx=0x7fc6600044b0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 shutdown_connections 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cb1ffd7e90 0x55cb1ffda2b0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb1ff61f30 0x55cb1ff5ba20 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb1ff59350 0x55cb1ff5b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc655ffb640 0 client.17604 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 --2- 172.21.15.116:0/4188083221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1ff58a30 0x55cb1ff69dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.209+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 >> 172.21.15.116:0/4188083221 conn(0x55cb1ff53160 msgr2=0x55cb1ff521e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 shutdown_connections 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc677836740 1 -- 172.21.15.116:0/4188083221 wait complete. 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc677836740 20 client.17604 trim_cache size 0 max 16384 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "dda4e632-aaea-4297-89f4-51a6481eff1c": { 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "dda4e632-aaea-4297-89f4-51a6481eff1c", 2024-03-19T02:34:13.215 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:34:13.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:34:13.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:34:13.216 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 msgr2=0x7fc6580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 0x7fc6580a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc6680096c0 tx=0x7fc668009260 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 msgr2=0x7fc6701a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc660036040 tx=0x7fc660033a60 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 shutdown_connections 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc6580a2970 0x7fc6580a4e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc67010df60 0x7fc6701a2f20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc670109020 0x7fc67019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 --2- 172.21.15.116:0/3905995205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc670108660 0x7fc67019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 >> 172.21.15.116:0/3905995205 conn(0x7fc6700773d0 msgr2=0x7fc67010ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 shutdown_connections 2024-03-19T02:34:13.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:13.210+0000 7fc674c4b640 1 -- 172.21.15.116:0/3905995205 wait complete. 2024-03-19T02:34:13.222 INFO:tasks.cephfs.filesystem:scrub status for tag:dda4e632-aaea-4297-89f4-51a6481eff1c - {'path': '/', 'tag': 'dda4e632-aaea-4297-89f4-51a6481eff1c', 'options': 'recursive,force'} 2024-03-19T02:34:13.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 203470 3.23 MB/sec execute 216 sec latency 27.349 ms 2024-03-19T02:34:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: pgmap v197: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.2 MiB/s wr, 145 op/s 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: pgmap v197: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.2 MiB/s wr, 145 op/s 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: pgmap v197: 129 pgs: 129 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.2 MiB/s wr, 145 op/s 2024-03-19T02:34:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:34:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:34:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:34:14.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:14.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 204090 3.23 MB/sec execute 217 sec latency 65.099 ms 2024-03-19T02:34:15.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 204740 3.23 MB/sec execute 218 sec latency 16.615 ms 2024-03-19T02:34:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:15 smithi162 ceph-mon[35761]: pgmap v198: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:15 smithi154 ceph-mon[35693]: pgmap v198: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:15 smithi116 ceph-mon[32263]: pgmap v198: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:16.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 205496 3.25 MB/sec execute 219 sec latency 16.182 ms 2024-03-19T02:34:17.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 205999 3.25 MB/sec execute 220 sec latency 24.859 ms 2024-03-19T02:34:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:17 smithi162 ceph-mon[35761]: pgmap v199: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:17 smithi154 ceph-mon[35693]: pgmap v199: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:17 smithi116 ceph-mon[32263]: pgmap v199: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:18.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 206386 3.24 MB/sec execute 221 sec latency 42.743 ms 2024-03-19T02:34:19.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 206940 3.23 MB/sec execute 222 sec latency 49.953 ms 2024-03-19T02:34:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:19 smithi162 ceph-mon[35761]: pgmap v200: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:19 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:34:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:19 smithi154 ceph-mon[35693]: pgmap v200: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:19 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:34:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:19 smithi116 ceph-mon[32263]: pgmap v200: 129 pgs: 129 active+clean; 184 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-03-19T02:34:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:19 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:34:20.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 207604 3.23 MB/sec execute 223 sec latency 16.427 ms 2024-03-19T02:34:21.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:34:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:34:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:34:21.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 208217 3.23 MB/sec execute 224 sec latency 34.622 ms 2024-03-19T02:34:22.157 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:21 smithi154 ceph-mon[35693]: pgmap v201: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:34:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:21 smithi162 ceph-mon[35761]: pgmap v201: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:34:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:21 smithi116 ceph-mon[32263]: pgmap v201: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:34:22.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 209031 3.24 MB/sec execute 225 sec latency 16.997 ms 2024-03-19T02:34:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 209539 3.25 MB/sec execute 226 sec latency 21.190 ms 2024-03-19T02:34:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:23 smithi162 ceph-mon[35761]: pgmap v202: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-03-19T02:34:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:23 smithi154 ceph-mon[35693]: pgmap v202: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-03-19T02:34:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:23 smithi116 ceph-mon[32263]: pgmap v202: 129 pgs: 129 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-03-19T02:34:24.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 209960 3.24 MB/sec execute 227 sec latency 33.087 ms 2024-03-19T02:34:25.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 210566 3.23 MB/sec execute 228 sec latency 16.924 ms 2024-03-19T02:34:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:25 smithi162 ceph-mon[35761]: pgmap v203: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:34:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:25 smithi154 ceph-mon[35693]: pgmap v203: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:34:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:25 smithi116 ceph-mon[32263]: pgmap v203: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:34:26.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 211315 3.23 MB/sec execute 229 sec latency 18.601 ms 2024-03-19T02:34:27.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 211882 3.23 MB/sec execute 230 sec latency 20.483 ms 2024-03-19T02:34:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:27 smithi162 ceph-mon[35761]: pgmap v204: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:27 smithi154 ceph-mon[35693]: pgmap v204: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:27 smithi116 ceph-mon[32263]: pgmap v204: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:28.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 212599 3.24 MB/sec execute 231 sec latency 22.168 ms 2024-03-19T02:34:29.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 213161 3.25 MB/sec execute 232 sec latency 16.136 ms 2024-03-19T02:34:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:29 smithi162 ceph-mon[35761]: pgmap v205: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:29 smithi154 ceph-mon[35693]: pgmap v205: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:29 smithi116 ceph-mon[32263]: pgmap v205: 129 pgs: 129 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:34:30.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 213608 3.24 MB/sec execute 233 sec latency 44.638 ms 2024-03-19T02:34:31.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 214245 3.24 MB/sec execute 234 sec latency 17.943 ms 2024-03-19T02:34:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:31 smithi162 ceph-mon[35761]: pgmap v206: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-19T02:34:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:31 smithi154 ceph-mon[35693]: pgmap v206: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-19T02:34:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:31 smithi116 ceph-mon[32263]: pgmap v206: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.5 MiB/s wr, 125 op/s 2024-03-19T02:34:32.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 214930 3.23 MB/sec execute 235 sec latency 22.812 ms 2024-03-19T02:34:33.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 215633 3.24 MB/sec execute 236 sec latency 16.410 ms 2024-03-19T02:34:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:33 smithi162 ceph-mon[35761]: pgmap v207: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:33 smithi154 ceph-mon[35693]: pgmap v207: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:33 smithi116 ceph-mon[32263]: pgmap v207: 129 pgs: 129 active+clean; 195 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:34.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 216202 3.24 MB/sec execute 237 sec latency 65.638 ms 2024-03-19T02:34:35.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 216746 3.25 MB/sec execute 238 sec latency 21.520 ms 2024-03-19T02:34:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:35 smithi162 ceph-mon[35761]: pgmap v208: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:34:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:35 smithi154 ceph-mon[35693]: pgmap v208: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:34:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:35 smithi116 ceph-mon[32263]: pgmap v208: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-19T02:34:36.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 217230 3.24 MB/sec execute 239 sec latency 32.834 ms 2024-03-19T02:34:37.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 217895 3.24 MB/sec execute 240 sec latency 20.058 ms 2024-03-19T02:34:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:37 smithi162 ceph-mon[35761]: pgmap v209: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:37 smithi154 ceph-mon[35693]: pgmap v209: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:37 smithi116 ceph-mon[32263]: pgmap v209: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:38.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 218521 3.23 MB/sec execute 241 sec latency 19.305 ms 2024-03-19T02:34:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 219192 3.24 MB/sec execute 242 sec latency 18.566 ms 2024-03-19T02:34:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:39 smithi162 ceph-mon[35761]: pgmap v210: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:39 smithi154 ceph-mon[35693]: pgmap v210: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:39 smithi116 ceph-mon[32263]: pgmap v210: 129 pgs: 129 active+clean; 198 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:34:40.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 219791 3.24 MB/sec execute 243 sec latency 16.299 ms 2024-03-19T02:34:41.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 220363 3.25 MB/sec execute 244 sec latency 17.008 ms 2024-03-19T02:34:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:41 smithi162 ceph-mon[35761]: pgmap v211: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.3 MiB/s wr, 127 op/s 2024-03-19T02:34:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:41 smithi154 ceph-mon[35693]: pgmap v211: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.3 MiB/s wr, 127 op/s 2024-03-19T02:34:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:41 smithi116 ceph-mon[32263]: pgmap v211: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.3 MiB/s wr, 127 op/s 2024-03-19T02:34:42.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 220856 3.24 MB/sec execute 245 sec latency 45.728 ms 2024-03-19T02:34:43.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:34:43.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:34:43.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 -- 172.21.15.116:0/1711455541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 msgr2=0x7f94e410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/1711455541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e410d560 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009cd0 tx=0x7f94dc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 -- 172.21.15.116:0/1711455541 shutdown_connections 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/1711455541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e410d560 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/1711455541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e4100690 0x7f94e4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/1711455541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e41066a0 0x7f94e4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 -- 172.21.15.116:0/1711455541 >> 172.21.15.116:0/1711455541 conn(0x7f94e4077820 msgr2=0x7f94e4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 -- 172.21.15.116:0/1711455541 shutdown_connections 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.336+0000 7f94ea4ac640 1 -- 172.21.15.116:0/1711455541 wait complete. 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 Processor -- start 2024-03-19T02:34:43.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 -- start start 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 0x7f94e4198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e41066a0 0x7f94e41a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4110960 con 0x7f94e4101050 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41107e0 con 0x7f94e4100690 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94ea4ac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4110ae0 con 0x7f94e41066a0 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.337+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46988/0 (socket says 172.21.15.116:46988) 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 -- 172.21.15.116:0/158007973 learned_addr learned my addr 172.21.15.116:0/158007973 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e3fff640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 0x7f94e4198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e8a49640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e41066a0 0x7f94e41a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 -- 172.21.15.116:0/158007973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e41066a0 msgr2=0x7f94e41a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e41066a0 0x7f94e41a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 -- 172.21.15.116:0/158007973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 msgr2=0x7f94e4198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 0x7f94e4198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 -- 172.21.15.116:0/158007973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e41a43a0 con 0x7f94e4101050 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e3fff640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 0x7f94e4198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e37fe640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f94d800e9f0 tx=0x7f94d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e17fa640 1 -- 172.21.15.116:0/158007973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d800cd80 con 0x7f94e4101050 2024-03-19T02:34:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e41a4570 con 0x7f94e4101050 2024-03-19T02:34:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e17fa640 1 -- 172.21.15.116:0/158007973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d80054d0 con 0x7f94e4101050 2024-03-19T02:34:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94e17fa640 1 -- 172.21.15.116:0/158007973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8010490 con 0x7f94e4101050 2024-03-19T02:34:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.338+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41a4ac0 con 0x7f94e4101050 2024-03-19T02:34:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.340+0000 7f94e17fa640 1 -- 172.21.15.116:0/158007973 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f94d8005020 con 0x7f94e4101050 2024-03-19T02:34:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e17fa640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 0x7f94cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 Processor -- start 2024-03-19T02:34:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 -- start start 2024-03-19T02:34:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e3fff640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 0x7f94cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e17fa640 1 -- 172.21.15.116:0/158007973 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f94d80cf5f0 con 0x7f94e4101050 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b6a30 0x55ca590b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590c2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca5901a500 con 0x55ca590b7350 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca58fc2be0 con 0x55ca590b6a30 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94ed175740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ca590165a0 con 0x55ca590bff30 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b6a30 0x55ca590b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46994/0 (socket says 172.21.15.116:46994) 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e37fe640 1 -- 172.21.15.116:0/3845349438 learned_addr learned my addr 172.21.15.116:0/3845349438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.341+0000 7f94e8a49640 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 -- 172.21.15.116:0/3845349438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 msgr2=0x55ca590c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 -- 172.21.15.116:0/3845349438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b6a30 msgr2=0x55ca590b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b6a30 0x55ca590b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 -- 172.21.15.116:0/3845349438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca58fe4000 con 0x55ca590b7350 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e3fff640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 0x7f94cc0a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94d4011840 tx=0x7f94d4011c40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94e37fe640 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f94d80a3ae0 tx=0x7f94d809b290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf4b886252dabe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94ca7fc640 1 -- 172.21.15.116:0/3845349438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d80b4070 con 0x55ca590b7350 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94ca7fc640 1 -- 172.21.15.116:0/3845349438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d809bf70 con 0x55ca590b7350 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.342+0000 7f94ca7fc640 1 -- 172.21.15.116:0/3845349438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8097ee0 con 0x55ca590b7350 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- 172.21.15.116:0/3845349438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 msgr2=0x55ca590bf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f94d80a3ae0 tx=0x7f94d809b290 comp rx=0 tx=0).stop 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- 172.21.15.116:0/3845349438 shutdown_connections 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b7350 0x55ca590bf9f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 --2- 172.21.15.116:0/3845349438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b6a30 0x55ca590b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- 172.21.15.116:0/3845349438 >> 172.21.15.116:0/3845349438 conn(0x55ca590b1160 msgr2=0x55ca590b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- 172.21.15.116:0/3845349438 shutdown_connections 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- 172.21.15.116:0/3845349438 wait complete. 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 -- start start 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 Processor -- start 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.343+0000 7f94ed175740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590b6a30 0x55ca590bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca590c2d20 con 0x55ca590bff30 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca58fd6160 con 0x55ca590b7350 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ca590c2ea0 con 0x55ca590b6a30 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 10 client.0 _ll_register_callbacks cb 0x55ca59012a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e8a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590b6a30 0x55ca590bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e8a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47008/0 (socket says 172.21.15.116:47008) 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e8a49640 1 -- 172.21.15.116:0/2115735423 learned_addr learned my addr 172.21.15.116:0/2115735423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 -- 172.21.15.116:0/2115735423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590b6a30 msgr2=0x55ca590bcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590b6a30 0x55ca590bcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 -- 172.21.15.116:0/2115735423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 msgr2=0x55ca590bda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 -- 172.21.15.116:0/2115735423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca58df87d0 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e8a49640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd4f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f94d8099b20 tx=0x7f94d80048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca965a0aff8d0879 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d80b8070 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d809bf70 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.344+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f94d80cf1f0 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f94d80b4040 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 10 client.27428 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca58d2e3f0 con 0x55ca590b7350 2024-03-19T02:34:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d809bf70 con 0x55ca590b7350 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f94d80a3c10 con 0x55ca590b7350 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 10 client.27428 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 20 client.27428 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca590d0190 0x55ca590d2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 4 client.27428 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca59027a40 con 0x55ca590d0190 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.345+0000 7f94e3fff640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca590d0190 0x55ca590d2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.346+0000 7f94c37fe640 10 client.27428 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.346+0000 7f94e3fff640 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca590d0190 0x55ca590d2580 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa31cef7cf528e76 server_cookie=e9c247647cb4d277 in_seq=0 out_seq=0 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94c37fe640 1 -- 172.21.15.116:0/2115735423 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca59027a40 con 0x55ca590d0190 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94c37fe640 10 client.27428 handle_command_reply: tid=0 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 client.27428 shutdown 2024-03-19T02:34:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 msgr2=0x55ca590bd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd4f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f94d8099b20 tx=0x7f94d80048f0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 shutdown_connections 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca590d0190 0x55ca590d2580 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590bff30 0x55ca590bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94c37fe640 0 client.27428 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd4f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 --2- 172.21.15.116:0/2115735423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590b6a30 0x55ca590bcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 >> 172.21.15.116:0/2115735423 conn(0x55ca590b1160 msgr2=0x55ca590b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 shutdown_connections 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 1 -- 172.21.15.116:0/2115735423 wait complete. 2024-03-19T02:34:43.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.347+0000 7f94ed175740 20 client.27428 trim_cache size 0 max 16384 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 Processor -- start 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 -- start start 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41aec30 con 0x55ca590b6a30 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41b0770 con 0x55ca590b7350 2024-03-19T02:34:43.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94ed175740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41af300 con 0x55ca590bff30 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47022/0 (socket says 172.21.15.116:47022) 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 -- 172.21.15.116:0/996752970 learned_addr learned my addr 172.21.15.116:0/996752970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e8a49640 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 -- 172.21.15.116:0/996752970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 msgr2=0x55ca590cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 -- 172.21.15.116:0/996752970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 msgr2=0x55ca590bd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e3fff640 1 -- 172.21.15.116:0/996752970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca58cee0f0 con 0x55ca590b6a30 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.366+0000 7f94e37fe640 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94e3fff640 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f94d400f790 tx=0x7f94d401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe6e047d5846a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94c9ffb640 1 -- 172.21.15.116:0/996752970 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d401a040 con 0x55ca590b6a30 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94c9ffb640 1 -- 172.21.15.116:0/996752970 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d4004050 con 0x55ca590b6a30 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94c9ffb640 1 -- 172.21.15.116:0/996752970 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4003800 con 0x55ca590b6a30 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94ed175740 1 -- 172.21.15.116:0/996752970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 msgr2=0x55ca590bf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94ed175740 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f94d400f790 tx=0x7f94d401f000 comp rx=0 tx=0).stop 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.367+0000 7f94ed175740 1 -- 172.21.15.116:0/996752970 shutdown_connections 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590bd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- 172.21.15.116:0/996752970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590bf6e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- 172.21.15.116:0/996752970 >> 172.21.15.116:0/996752970 conn(0x55ca590b1160 msgr2=0x55ca590cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- 172.21.15.116:0/996752970 shutdown_connections 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- 172.21.15.116:0/996752970 wait complete. 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- start start 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 Processor -- start 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590d2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41b00a0 con 0x55ca590b6a30 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41af9d0 con 0x55ca590b7350 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41acc30 con 0x55ca590bff30 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94ed175740 10 client.0 _ll_register_callbacks cb 0x7f94d80b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:43.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47036/0 (socket says 172.21.15.116:47036) 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.368+0000 7f94e3fff640 1 -- 172.21.15.116:0/2722321380 learned_addr learned my addr 172.21.15.116:0/2722321380 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e8a49640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590d2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 -- 172.21.15.116:0/2722321380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 msgr2=0x55ca590d2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590d2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 -- 172.21.15.116:0/2722321380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 msgr2=0x55ca590d5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 -- 172.21.15.116:0/2722321380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca590d3410 con 0x55ca590b7350 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e3fff640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94e37fe640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590d2990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f94d8098240 tx=0x7f94d8099d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b06aefb5b7c03e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:34:43.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d80b8040 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d4004050 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f94d80cf3f0 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f94d80a3d90 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94ed175740 10 client.27434 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca590d3410 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.369+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94d4004050 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f94d8097640 con 0x55ca590b7350 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 10 client.27434 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 10 client.27434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 20 client.27434 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca59136200 0x55ca59138620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 4 client.27434 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca59021d90 con 0x55ca59136200 2024-03-19T02:34:43.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94e3fff640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca59136200 0x55ca59138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94c37fe640 10 client.27434 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94e3fff640 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca59136200 0x55ca59138620 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da3ee93d479026d server_cookie=2b10e5d5187f1e0b in_seq=0 out_seq=0 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94c37fe640 1 -- 172.21.15.116:0/2722321380 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ca59021d90 con 0x55ca59136200 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94c37fe640 10 client.27434 handle_command_reply: tid=0 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.370+0000 7f94ed175740 1 client.27434 shutdown 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 msgr2=0x55ca590d2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590d2990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f94d8098240 tx=0x7f94d8099d80 comp rx=0 tx=0).stop 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 shutdown_connections 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ca59136200 0x55ca59138620 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ca590bff30 0x55ca590d2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94c37fe640 0 client.27434 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:43.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca590b7350 0x55ca590d2990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 --2- 172.21.15.116:0/2722321380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca590b6a30 0x55ca590d5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 >> 172.21.15.116:0/2722321380 conn(0x55ca590b1160 msgr2=0x55ca590b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 shutdown_connections 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 1 -- 172.21.15.116:0/2722321380 wait complete. 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ed175740 20 client.27434 trim_cache size 0 max 16384 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:34:43.381 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:43.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 msgr2=0x7f94cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 0x7f94cc0a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94d4011840 tx=0x7f94d4011c40 comp rx=0 tx=0).stop 2024-03-19T02:34:43.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 msgr2=0x7f94e4199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f94d800e9f0 tx=0x7f94d800edf0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 shutdown_connections 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94cc0a2970 0x7f94cc0a4e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94e41066a0 0x7f94e41a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94e4101050 0x7f94e4199320 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 --2- 172.21.15.116:0/158007973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94e4100690 0x7f94e4198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.371+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 >> 172.21.15.116:0/158007973 conn(0x7f94e4077820 msgr2=0x7f94e4102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.372+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 shutdown_connections 2024-03-19T02:34:43.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.372+0000 7f94ea4ac640 1 -- 172.21.15.116:0/158007973 wait complete. 2024-03-19T02:34:43.384 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:34:43.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:34:43.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 -- 172.21.15.116:0/1486498149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e01033b0 msgr2=0x7f19e010b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/1486498149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e01033b0 0x7f19e010b890 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009900 tx=0x7f19d402bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 -- 172.21.15.116:0/1486498149 shutdown_connections 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/1486498149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e010be60 0x7f19e010e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/1486498149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e01033b0 0x7f19e010b890 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/1486498149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e0102a90 0x7f19e0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 -- 172.21.15.116:0/1486498149 >> 172.21.15.116:0/1486498149 conn(0x7f19e0077820 msgr2=0x7f19e0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 -- 172.21.15.116:0/1486498149 shutdown_connections 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 -- 172.21.15.116:0/1486498149 wait complete. 2024-03-19T02:34:43.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.495+0000 7f19e5fb2640 1 Processor -- start 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 -- start start 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 0x7f19e0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e01033b0 0x7f19e0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0111b10 con 0x7f19e010be60 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0111990 con 0x7f19e01033b0 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19e5fb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0111c90 con 0x7f19e0102a90 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 0x7f19e0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47048/0 (socket says 172.21.15.116:47048) 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19dffff640 1 -- 172.21.15.116:0/84694398 learned_addr learned my addr 172.21.15.116:0/84694398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19deffd640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e01033b0 0x7f19e0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 -- 172.21.15.116:0/84694398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e01033b0 msgr2=0x7f19e0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e01033b0 0x7f19e0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 -- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 msgr2=0x7f19e019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 -- 172.21.15.116:0/84694398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e019d040 con 0x7f19e0102a90 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19dffff640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.496+0000 7f19df7fe640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 0x7f19e0198050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19d000e9f0 tx=0x7f19d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.497+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d000ccc0 con 0x7f19e0102a90 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.497+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e019d330 con 0x7f19e0102a90 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.497+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19d00108e0 con 0x7f19e0102a90 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.497+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0018880 con 0x7f19e0102a90 2024-03-19T02:34:43.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.497+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a4bc0 con 0x7f19e0102a90 2024-03-19T02:34:43.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.498+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e0068610 con 0x7f19e0102a90 2024-03-19T02:34:43.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.499+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f19d0010a90 con 0x7f19e0102a90 2024-03-19T02:34:43.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.499+0000 7f19dcff9640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 0x7f19c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.500+0000 7f19deffd640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 0x7f19c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.500+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f19d00cf5b0 con 0x7f19e0102a90 2024-03-19T02:34:43.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.500+0000 7f19deffd640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 0x7f19c80a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006010 tx=0x7f19d4005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.504+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f19d009b8e0 con 0x7f19e0102a90 2024-03-19T02:34:43.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.755+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19e0106f50 con 0x7f19e0102a90 2024-03-19T02:34:43.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.756+0000 7f19dcff9640 1 -- 172.21.15.116:0/84694398 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f19d0084e20 con 0x7f19e0102a90 2024-03-19T02:34:43.757 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:34:43.757 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:34:43.757 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 msgr2=0x7f19c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 0x7f19c80a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006010 tx=0x7f19d4005e40 comp rx=0 tx=0).stop 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 msgr2=0x7f19e0198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 0x7f19e0198050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19d000e9f0 tx=0x7f19d000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 shutdown_connections 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19c80a2970 0x7f19c80a4e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19e010be60 0x7f19e019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19e01033b0 0x7f19e0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 --2- 172.21.15.116:0/84694398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19e0102a90 0x7f19e0198050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 >> 172.21.15.116:0/84694398 conn(0x7f19e0077820 msgr2=0x7f19e00fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 shutdown_connections 2024-03-19T02:34:43.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.758+0000 7f19e5fb2640 1 -- 172.21.15.116:0/84694398 wait complete. 2024-03-19T02:34:43.770 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:34:43.787 INFO:tasks.workunit.client.0.smithi116.stdout: 1 221471 3.24 MB/sec execute 246 sec latency 15.903 ms 2024-03-19T02:34:43.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 -- 172.21.15.116:0/996136279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 msgr2=0x7f4a1810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/996136279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a1810d570 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009cd0 tx=0x7f4a1002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 -- 172.21.15.116:0/996136279 shutdown_connections 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/996136279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a1810d570 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/996136279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/996136279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 0x7f4a18106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 -- 172.21.15.116:0/996136279 >> 172.21.15.116:0/996136279 conn(0x7f4a180778d0 msgr2=0x7f4a18077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 -- 172.21.15.116:0/996136279 shutdown_connections 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.886+0000 7f4a1dddf640 1 -- 172.21.15.116:0/996136279 wait complete. 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 Processor -- start 2024-03-19T02:34:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 -- start start 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 0x7f4a1819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a18110970 con 0x7f4a18101060 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a181107f0 con 0x7f4a181066b0 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.887+0000 7f4a1dddf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a18110af0 con 0x7f4a181006a0 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47058/0 (socket says 172.21.15.116:47058) 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 -- 172.21.15.116:0/999614990 learned_addr learned my addr 172.21.15.116:0/999614990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a17fff640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 0x7f4a1819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 -- 172.21.15.116:0/999614990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 msgr2=0x7f4a18197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 -- 172.21.15.116:0/999614990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 msgr2=0x7f4a1819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 0x7f4a1819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 -- 172.21.15.116:0/999614990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1819cfd0 con 0x7f4a18101060 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a177fe640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00e9f0 tx=0x7f4a0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a14ff9640 1 -- 172.21.15.116:0/999614990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c00cd80 con 0x7f4a18101060 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1819d2c0 con 0x7f4a18101060 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a181a4b50 con 0x7f4a18101060 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a14ff9640 1 -- 172.21.15.116:0/999614990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c0054d0 con 0x7f4a18101060 2024-03-19T02:34:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.888+0000 7f4a14ff9640 1 -- 172.21.15.116:0/999614990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c010490 con 0x7f4a18101060 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a14ff9640 1 -- 172.21.15.116:0/999614990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4a0c005020 con 0x7f4a18101060 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a209ca740 1 Processor -- start 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a209ca740 1 -- start start 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.891+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 0x559d31ce5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a209ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559d31c3d4b0 con 0x559d31ce37d0 2024-03-19T02:34:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a209ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d31be5d70 con 0x559d31cdabf0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a209ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559d31c39550 con 0x559d31cda2d0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 0x559d31ce5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56530/0 (socket says 172.21.15.116:56530) 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a17fff640 1 -- 172.21.15.116:0/1396524995 learned_addr learned my addr 172.21.15.116:0/1396524995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a177fe640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37248/0 (socket says 172.21.15.116:37248) 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a14ff9640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 0x7f4a000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a17fff640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 0x7f4a000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a14ff9640 1 -- 172.21.15.116:0/999614990 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f4a0c0cf400 con 0x7f4a18101060 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 -- 172.21.15.116:0/1396524995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 msgr2=0x559d31cda6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 -- 172.21.15.116:0/1396524995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 msgr2=0x559d31ce3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 -- 172.21.15.116:0/1396524995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d31c08210 con 0x559d31ce37d0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a177fe640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a17fff640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 0x559d31ce5bc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0cc770 tx=0x7f4a0c097260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ecaf9d1e4836000 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.892+0000 7f49fdffb640 1 -- 172.21.15.116:0/1396524995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c09afe0 con 0x559d31ce37d0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a17fff640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 0x7f4a000a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009a60 tx=0x7f4a1002c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f49fdffb640 1 -- 172.21.15.116:0/1396524995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c0bb050 con 0x559d31ce37d0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f49fdffb640 1 -- 172.21.15.116:0/1396524995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c09b680 con 0x559d31ce37d0 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 -- 172.21.15.116:0/1396524995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 msgr2=0x559d31ce5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 0x559d31ce5bc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0cc770 tx=0x7f4a0c097260 comp rx=0 tx=0).stop 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 -- 172.21.15.116:0/1396524995 shutdown_connections 2024-03-19T02:34:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31ce37d0 0x559d31ce5bc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 --2- 172.21.15.116:0/1396524995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cda6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 -- 172.21.15.116:0/1396524995 >> 172.21.15.116:0/1396524995 conn(0x559d31cd4a00 msgr2=0x559d31cd4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 -- 172.21.15.116:0/1396524995 shutdown_connections 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.893+0000 7f4a209ca740 1 -- 172.21.15.116:0/1396524995 wait complete. 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 -- start start 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 Processor -- start 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31ce37d0 0x559d31ce11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559d31ce6500 con 0x559d31cda2d0 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d31bf9310 con 0x559d31cdabf0 2024-03-19T02:34:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559d31ce6680 con 0x559d31ce37d0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a209ca740 10 client.0 _ll_register_callbacks cb 0x559d31bed2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47078/0 (socket says 172.21.15.116:47078) 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a177fe640 1 -- 172.21.15.116:0/202722129 learned_addr learned my addr 172.21.15.116:0/202722129 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a17fff640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31ce37d0 0x559d31ce11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.894+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 -- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31ce37d0 msgr2=0x559d31ce11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31ce37d0 0x559d31ce11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 -- 172.21.15.116:0/202722129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 msgr2=0x559d31ce0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 -- 172.21.15.116:0/202722129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d319513f0 con 0x559d31cda2d0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a177fe640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4a08012d20 tx=0x7f4a0800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc706ea68145e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f49f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a080120d0 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f4a08010430 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a0c0bb050 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4a08012270 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a209ca740 10 client.17640 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d31a1b7d0 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.895+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c0bb050 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4a0801c730 con 0x559d31cda2d0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 10 client.17640 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 10 client.17640 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 20 client.17640 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31cf3ba0 0x559d31cf6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 4 client.17640 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d31c9b3e0 con 0x559d31cf3ba0 2024-03-19T02:34:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31cf3ba0 0x559d31cf6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f49f6ffd640 10 client.17640 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.896+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31cf3ba0 0x559d31cf6020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6081c2f9cc49b3db server_cookie=1f78c023633a16b2 in_seq=0 out_seq=0 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f49f6ffd640 1 -- 172.21.15.116:0/202722129 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559d31c9b3e0 con 0x559d31cf3ba0 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f49f6ffd640 10 client.17640 handle_command_reply: tid=0 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 client.17640 shutdown 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 msgr2=0x559d31ce0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4a08012d20 tx=0x7f4a0800f790 comp rx=0 tx=0).stop 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 shutdown_connections 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31cf3ba0 0x559d31cf6020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31ce37d0 0x559d31ce11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31cdabf0 0x559d31ce0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f49f6ffd640 0 client.17640 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.897+0000 7f4a209ca740 1 --2- 172.21.15.116:0/202722129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cda2d0 0x559d31ce0770 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.898+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 >> 172.21.15.116:0/202722129 conn(0x559d31cd4a00 msgr2=0x559d31cd6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.898+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 shutdown_connections 2024-03-19T02:34:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.898+0000 7f4a209ca740 1 -- 172.21.15.116:0/202722129 wait complete. 2024-03-19T02:34:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.898+0000 7f4a209ca740 20 client.17640 trim_cache size 0 max 16384 2024-03-19T02:34:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 Processor -- start 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 -- start start 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cef300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31cfb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0400ff60 con 0x559d31cdabf0 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a04011aa0 con 0x559d31ce37d0 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a209ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a04010630 con 0x559d31cda2d0 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47084/0 (socket says 172.21.15.116:47084) 2024-03-19T02:34:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2764776858 learned_addr learned my addr 172.21.15.116:0/2764776858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a177fe640 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a17fff640 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31cfb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2764776858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 msgr2=0x559d31cef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2764776858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 msgr2=0x559d31cfb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31cfb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.913+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2764776858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d31cd4830 con 0x559d31cdabf0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0ccd70 tx=0x7f4a0c09bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58a30b410e0e706 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f49fd7fa640 1 -- 172.21.15.116:0/2764776858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c097920 con 0x559d31cdabf0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f49fd7fa640 1 -- 172.21.15.116:0/2764776858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c0bb040 con 0x559d31cdabf0 2024-03-19T02:34:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f49fd7fa640 1 -- 172.21.15.116:0/2764776858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c09b1c0 con 0x559d31cdabf0 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f4a209ca740 1 -- 172.21.15.116:0/2764776858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 msgr2=0x559d31ce1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.914+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c0ccd70 tx=0x7f4a0c09bef0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- 172.21.15.116:0/2764776858 shutdown_connections 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31cfb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce1010 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2764776858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31cef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- 172.21.15.116:0/2764776858 >> 172.21.15.116:0/2764776858 conn(0x559d31cd4a00 msgr2=0x559d31cd66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- 172.21.15.116:0/2764776858 shutdown_connections 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- 172.21.15.116:0/2764776858 wait complete. 2024-03-19T02:34:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- start start 2024-03-19T02:34:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 Processor -- start 2024-03-19T02:34:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31ce7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31ce7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a040113d0 con 0x559d31cdabf0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a04010d00 con 0x559d31ce37d0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0400df60 con 0x559d31cda2d0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a209ca740 10 client.0 _ll_register_callbacks cb 0x7f4a08009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47086/0 (socket says 172.21.15.116:47086) 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.915+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2164635867 learned_addr learned my addr 172.21.15.116:0/2164635867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a17fff640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31ce7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a177fe640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31ce7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 msgr2=0x559d31ce7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31ce7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2164635867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 msgr2=0x559d31ce7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31ce7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 -- 172.21.15.116:0/2164635867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d31d59160 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a16ffd640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00cfd0 tx=0x7f4a0c09b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afcbdd6ea3404ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f49f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c09b1c0 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f4a0c0cc9f0 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a0c0bb040 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4a0c097920 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a209ca740 10 client.17646 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d31d5a400 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.916+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a0c0bb040 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4a0c0a3990 con 0x559d31cdabf0 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 10 client.17646 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 10 client.17646 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 20 client.17646 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31d5a700 0x559d31d5caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 4 client.17646 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d31c4a9f0 con 0x559d31d5a700 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a177fe640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31d5a700 0x559d31d5caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f49f6ffd640 10 client.17646 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a177fe640 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31d5a700 0x559d31d5caf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d4f163d993e29449 server_cookie=bff6c7bbc58dc271 in_seq=0 out_seq=0 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f49f6ffd640 1 -- 172.21.15.116:0/2164635867 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d31c4a9f0 con 0x559d31d5a700 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f49f6ffd640 10 client.17646 handle_command_reply: tid=0 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.917+0000 7f4a209ca740 1 client.17646 shutdown 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 msgr2=0x559d31ce79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00cfd0 tx=0x7f4a0c09b4d0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 shutdown_connections 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x559d31d5a700 0x559d31d5caf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d31ce37d0 0x559d31ce7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d31cdabf0 0x559d31ce79c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 --2- 172.21.15.116:0/2164635867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559d31cda2d0 0x559d31ce7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f49f6ffd640 0 client.17646 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 >> 172.21.15.116:0/2164635867 conn(0x559d31cd4a00 msgr2=0x559d31cd3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 shutdown_connections 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 1 -- 172.21.15.116:0/2164635867 wait complete. 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a209ca740 20 client.17646 trim_cache size 0 max 16384 2024-03-19T02:34:43.926 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 msgr2=0x7f4a000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 0x7f4a000a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009a60 tx=0x7f4a1002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 msgr2=0x7f4a18198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00e9f0 tx=0x7f4a0c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 shutdown_connections 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4a000a2970 0x7f4a000a4e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a181066b0 0x7f4a1819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18101060 0x7f4a18198520 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 --2- 172.21.15.116:0/999614990 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4a181006a0 0x7f4a18197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.918+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 >> 172.21.15.116:0/999614990 conn(0x7f4a180778d0 msgr2=0x7f4a18068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.919+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 shutdown_connections 2024-03-19T02:34:43.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:43.919+0000 7f4a1dddf640 1 -- 172.21.15.116:0/999614990 wait complete. 2024-03-19T02:34:43.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 -- 172.21.15.116:0/3790205596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81066b0 msgr2=0x7fa6c8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/3790205596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81066b0 0x7fa6c8106a90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009930 tx=0x7fa6bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 -- 172.21.15.116:0/3790205596 shutdown_connections 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/3790205596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c8101060 0x7fa6c810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/3790205596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c81006a0 0x7fa6c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/3790205596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81066b0 0x7fa6c8106a90 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 -- 172.21.15.116:0/3790205596 >> 172.21.15.116:0/3790205596 conn(0x7fa6c80778d0 msgr2=0x7fa6c8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 -- 172.21.15.116:0/3790205596 shutdown_connections 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.088+0000 7fa6cf15b640 1 -- 172.21.15.116:0/3790205596 wait complete. 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 Processor -- start 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 -- start start 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 0x7fa6c81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c81066b0 0x7fa6c819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8111a70 con 0x7fa6c81006a0 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81118f0 con 0x7fa6c8101060 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cf15b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8111bf0 con 0x7fa6c81066b0 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47108/0 (socket says 172.21.15.116:47108) 2024-03-19T02:34:44.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2883912578 learned_addr learned my addr 172.21.15.116:0/2883912578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 0x7fa6c81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c81066b0 0x7fa6c819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2883912578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c81066b0 msgr2=0x7fa6c819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c81066b0 0x7fa6c819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2883912578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 msgr2=0x7fa6c81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 0x7fa6c81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2883912578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c819cf80 con 0x7fa6c81006a0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.089+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 0x7fa6c81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009230 tx=0x7fa6bc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6c5ffb640 1 -- 172.21.15.116:0/2883912578 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc043020 con 0x7fa6c81006a0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c819d210 con 0x7fa6c81006a0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6c5ffb640 1 -- 172.21.15.116:0/2883912578 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6bc02ca70 con 0x7fa6c81006a0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6c5ffb640 1 -- 172.21.15.116:0/2883912578 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc003a50 con 0x7fa6c81006a0 2024-03-19T02:34:44.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.090+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c81a4b00 con 0x7fa6c81006a0 2024-03-19T02:34:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c5ffb640 1 -- 172.21.15.116:0/2883912578 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa6bc046020 con 0x7fa6c81006a0 2024-03-19T02:34:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 Processor -- start 2024-03-19T02:34:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 -- start start 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c5ffb640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 0x7fa6b80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139dabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3139324b0 con 0x55d3139cfbf0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d3138dad70 con 0x55d3139d87d0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d31392e550 con 0x55d3139cf2d0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 0x7fa6b80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47124/0 (socket says 172.21.15.116:47124) 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1057052251 learned_addr learned my addr 172.21.15.116:0/1057052251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6c5ffb640 1 -- 172.21.15.116:0/2883912578 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fa6bc0f1060 con 0x7fa6c81006a0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.093+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139dabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1057052251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 msgr2=0x55d3139cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1057052251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 msgr2=0x55d3139dabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139dabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1057052251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3138fd210 con 0x55d3139cfbf0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 0x7fa6b80a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0005fd0 tx=0x7fa6c0009b00 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40114f0 tx=0x7fa6b40119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be43ef58c3208b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6b2ffd640 1 -- 172.21.15.116:0/1057052251 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b401e070 con 0x55d3139cfbf0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6b2ffd640 1 -- 172.21.15.116:0/1057052251 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6b40035f0 con 0x55d3139cfbf0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.094+0000 7fa6b2ffd640 1 -- 172.21.15.116:0/1057052251 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b400f740 con 0x55d3139cfbf0 2024-03-19T02:34:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1057052251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 msgr2=0x55d3139d8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40114f0 tx=0x7fa6b40119b0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1057052251 shutdown_connections 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139dabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d8290 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1057052251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1057052251 >> 172.21.15.116:0/1057052251 conn(0x55d3139c9a00 msgr2=0x55d3139c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1057052251 shutdown_connections 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1057052251 wait complete. 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 -- start start 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.095+0000 7fa6d1e21740 1 Processor -- start 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 0x55d3139d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3139db5c0 con 0x55d3139cf2d0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d3138ee310 con 0x55d3139d87d0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d3139db740 con 0x55d3139cfbf0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6d1e21740 10 client.0 _ll_register_callbacks cb 0x55d3138e22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47134/0 (socket says 172.21.15.116:47134) 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6cd6f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 0x55d3139d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2491528141 learned_addr learned my addr 172.21.15.116:0/2491528141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 msgr2=0x55d3139d5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 0x55d3139d5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2491528141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 msgr2=0x55d3139d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139d62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 -- 172.21.15.116:0/2491528141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3136463f0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 0x55d3139d5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0b9ee0 tx=0x7fa6bc0b97b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b5f24f2f9833b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.096+0000 7fa6a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0ca8b0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fa6bc0053d0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6b40035f0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa6bc0bdc10 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 10 client.17658 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d3137107d0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6b40035f0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa6bc0d61f0 con 0x55d3139cf2d0 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 10 client.17658 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 10 client.17658 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 20 client.17658 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d3139e8bf0 0x55d3139eb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 4 client.17658 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3139903e0 con 0x55d3139e8bf0 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.097+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d3139e8bf0 0x55d3139eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.098+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d3139e8bf0 0x55d3139eb060 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=70bf3f6574baffbc server_cookie=a5e40cd2267de20e in_seq=0 out_seq=0 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.098+0000 7fa6a7fff640 10 client.17658 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6a7fff640 1 -- 172.21.15.116:0/2491528141 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3139903e0 con 0x55d3139e8bf0 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6a7fff640 10 client.17658 handle_command_reply: tid=0 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 client.17658 shutdown 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 msgr2=0x55d3139d5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0b9ee0 tx=0x7fa6bc0b97b0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 shutdown_connections 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139d87d0 0x55d3139d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cfbf0 0x55d3139d5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d3139e8bf0 0x55d3139eb060 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/2491528141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cf2d0 0x55d3139d5840 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 >> 172.21.15.116:0/2491528141 conn(0x55d3139c9a00 msgr2=0x55d3139cb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6a7fff640 0 client.17658 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 shutdown_connections 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 1 -- 172.21.15.116:0/2491528141 wait complete. 2024-03-19T02:34:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.099+0000 7fa6d1e21740 20 client.17658 trim_cache size 0 max 16384 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.113+0000 7fa6d1e21740 1 Processor -- start 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.113+0000 7fa6d1e21740 1 -- start start 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cf2d0 0x55d3139d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 0x55d3139f0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81aec20 con 0x55d3139cfbf0 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81b0760 con 0x55d3139cf2d0 2024-03-19T02:34:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81af2f0 con 0x55d3139d87d0 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47140/0 (socket says 172.21.15.116:47140) 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cf2d0 0x55d3139d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 0x55d3139f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1012498202 learned_addr learned my addr 172.21.15.116:0/1012498202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1012498202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 msgr2=0x55d3139f0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 0x55d3139f0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1012498202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cf2d0 msgr2=0x55d3139d7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cf2d0 0x55d3139d7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 -- 172.21.15.116:0/1012498202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3139d1400 con 0x55d3139cfbf0 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 0x55d3139f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40114c0 tx=0x7fa6b4005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8dceb867c98c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6b27fc640 1 -- 172.21.15.116:0/1012498202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b401e040 con 0x55d3139cfbf0 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6b27fc640 1 -- 172.21.15.116:0/1012498202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6b402f070 con 0x55d3139cfbf0 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.114+0000 7fa6b27fc640 1 -- 172.21.15.116:0/1012498202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b402a430 con 0x55d3139cfbf0 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1012498202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 msgr2=0x55d3139d60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40114c0 tx=0x7fa6b4005870 comp rx=0 tx=0).stop 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1012498202 shutdown_connections 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139d87d0 0x55d3139f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139cfbf0 0x55d3139d60e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/1012498202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cf2d0 0x55d3139d7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1012498202 >> 172.21.15.116:0/1012498202 conn(0x55d3139c9a00 msgr2=0x55d3139cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1012498202 shutdown_connections 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.115+0000 7fa6d1e21740 1 -- 172.21.15.116:0/1012498202 wait complete. 2024-03-19T02:34:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 -- start start 2024-03-19T02:34:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 Processor -- start 2024-03-19T02:34:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cfbf0 0x55d3139dc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81b0090 con 0x55d3139d87d0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81af9c0 con 0x55d3139cfbf0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c81acc20 con 0x55d3139cf2d0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6d1e21740 10 client.0 _ll_register_callbacks cb 0x7fa6bc0b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47154/0 (socket says 172.21.15.116:47154) 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 -- 172.21.15.116:0/3886992655 learned_addr learned my addr 172.21.15.116:0/3886992655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6ccef7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6c7fff640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cfbf0 0x55d3139dc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 -- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 msgr2=0x55d3139e0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139e0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 -- 172.21.15.116:0/3886992655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cfbf0 msgr2=0x55d3139dc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cfbf0 0x55d3139dc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6cd6f8640 1 -- 172.21.15.116:0/3886992655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d313a4df60 con 0x55d3139d87d0 2024-03-19T02:34:44.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.116+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6cd6f8640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000fec0 tx=0x7fa6c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68076ef15fed5943 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c002f060 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fa6c00046c0 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6b402f070 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa6c0036050 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 10 client.17664 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d313a4f090 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6b402f070 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa6c0029070 con 0x55d3139d87d0 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 10 client.17664 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 10 client.17664 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 20 client.17664 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d313a4f840 0x55d313a51c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 4 client.17664 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.117+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d31393f9f0 con 0x55d313a4f840 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d313a4f840 0x55d313a51c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6a7fff640 10 client.17664 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6ccef7640 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d313a4f840 0x55d313a51c60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d80843a4ad046ac8 server_cookie=141bf1501c93e03 in_seq=0 out_seq=0 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6a7fff640 1 -- 172.21.15.116:0/3886992655 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d31393f9f0 con 0x55d313a4f840 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6a7fff640 10 client.17664 handle_command_reply: tid=0 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 client.17664 shutdown 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 msgr2=0x55d3139dc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000fec0 tx=0x7fa6c0004ab0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 shutdown_connections 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55d313a4f840 0x55d313a51c60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3139d87d0 0x55d3139dc880 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6a7fff640 0 client.17664 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3139cfbf0 0x55d3139dc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 --2- 172.21.15.116:0/3886992655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d3139cf2d0 0x55d3139e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 >> 172.21.15.116:0/3886992655 conn(0x55d3139c9a00 msgr2=0x55d3139cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 shutdown_connections 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.118+0000 7fa6d1e21740 1 -- 172.21.15.116:0/3886992655 wait complete. 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6d1e21740 20 client.17664 trim_cache size 0 max 16384 2024-03-19T02:34:44.126 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 msgr2=0x7fa6b80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 0x7fa6b80a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0005fd0 tx=0x7fa6c0009b00 comp rx=0 tx=0).stop 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 msgr2=0x7fa6c8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009230 tx=0x7fa6bc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 shutdown_connections 2024-03-19T02:34:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6b80a29f0 0x7fa6b80a4eb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6c81066b0 0x7fa6c819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c8101060 0x7fa6c81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 --2- 172.21.15.116:0/2883912578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6c81006a0 0x7fa6c8197f90 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 >> 172.21.15.116:0/2883912578 conn(0x7fa6c80778d0 msgr2=0x7fa6c8103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 shutdown_connections 2024-03-19T02:34:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.119+0000 7fa6cf15b640 1 -- 172.21.15.116:0/2883912578 wait complete. 2024-03-19T02:34:44.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:43 smithi116 ceph-mon[32263]: pgmap v212: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:44.130 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:43 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/84694398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:44.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:34:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:43 smithi162 ceph-mon[35761]: pgmap v212: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:43 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/84694398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:43 smithi154 ceph-mon[35693]: pgmap v212: 129 pgs: 129 active+clean; 200 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2024-03-19T02:34:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:43 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/84694398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 -- 172.21.15.116:0/360740911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 msgr2=0x7fc3f4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/360740911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4100b20 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009930 tx=0x7fc3dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 -- 172.21.15.116:0/360740911 shutdown_connections 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/360740911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/360740911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4100b20 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/360740911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 0x7fc3f4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 -- 172.21.15.116:0/360740911 >> 172.21.15.116:0/360740911 conn(0x7fc3f40778d0 msgr2=0x7fc3f4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 -- 172.21.15.116:0/360740911 shutdown_connections 2024-03-19T02:34:44.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.256+0000 7fc3f8edc640 1 -- 172.21.15.116:0/360740911 wait complete. 2024-03-19T02:34:44.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 Processor -- start 2024-03-19T02:34:44.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 -- start start 2024-03-19T02:34:44.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 0x7fc3f419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4110970 con 0x7fc3f41006a0 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f41107f0 con 0x7fc3f4101060 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f8edc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4110af0 con 0x7fc3f41066b0 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47186/0 (socket says 172.21.15.116:47186) 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f259c640 1 -- 172.21.15.116:0/3236382836 learned_addr learned my addr 172.21.15.116:0/3236382836 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 0x7fc3f419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 -- 172.21.15.116:0/3236382836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 msgr2=0x7fc3f419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 0x7fc3f419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 -- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 msgr2=0x7fc3f4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3ebfff640 1 -- 172.21.15.116:0/3236382836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f419cfd0 con 0x7fc3f4101060 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.257+0000 7fc3f259c640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f4198520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009230 tx=0x7fc3dc0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3eb7fe640 1 -- 172.21.15.116:0/3236382836 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc043020 con 0x7fc3f4101060 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f419d260 con 0x7fc3f4101060 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f41a4b50 con 0x7fc3f4101060 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3eb7fe640 1 -- 172.21.15.116:0/3236382836 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3dc004c50 con 0x7fc3f4101060 2024-03-19T02:34:44.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.258+0000 7fc3eb7fe640 1 -- 172.21.15.116:0/3236382836 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0035f0 con 0x7fc3f4101060 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.260+0000 7fc3eb7fe640 1 -- 172.21.15.116:0/3236382836 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc3dc046020 con 0x7fc3f4101060 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 Processor -- start 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 -- start start 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3eb7fe640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 0x7fc3e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab2a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620baa844b0 con 0x5620bab2a7d0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5620baa2cd70 con 0x5620bab21bf0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5620baa80550 con 0x5620bab212d0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab2a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 0x7fc3e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56654/0 (socket says 172.21.15.116:56654) 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f259c640 1 -- 172.21.15.116:0/1398381479 learned_addr learned my addr 172.21.15.116:0/1398381479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3eb7fe640 1 -- 172.21.15.116:0/3236382836 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc3dc0f08c0 con 0x7fc3f4101060 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f259c640 1 -- 172.21.15.116:0/1398381479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 msgr2=0x5620bab2a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.261+0000 7fc3f259c640 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab2a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3f259c640 1 -- 172.21.15.116:0/1398381479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 msgr2=0x5620bab2cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3f259c640 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3f259c640 1 -- 172.21.15.116:0/1398381479 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620baa4f210 con 0x5620bab212d0 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3f259c640 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40115f0 tx=0x7fc3e4011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b41519d25a52f442 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 0x7fc3e00a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x5620bab1cb90 tx=0x7fc3ec009bf0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3e8ff9640 1 -- 172.21.15.116:0/1398381479 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400f8f0 con 0x5620bab212d0 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3e8ff9640 1 -- 172.21.15.116:0/1398381479 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3e401a070 con 0x5620bab212d0 2024-03-19T02:34:44.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.262+0000 7fc3e8ff9640 1 -- 172.21.15.116:0/1398381479 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4016dd0 con 0x5620bab212d0 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1398381479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 msgr2=0x5620bab216b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc3e40115f0 tx=0x7fc3e4011ab0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1398381479 shutdown_connections 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab2a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1398381479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab216b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1398381479 >> 172.21.15.116:0/1398381479 conn(0x5620bab1ba00 msgr2=0x5620bab1bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1398381479 shutdown_connections 2024-03-19T02:34:44.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1398381479 wait complete. 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.263+0000 7fc3fbba4740 1 -- start start 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 Processor -- start 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab27760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab27ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab281e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620bab2d5c0 con 0x5620bab2a7d0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5620baa40310 con 0x5620bab21bf0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5620bab2d740 con 0x5620bab212d0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3fbba4740 10 client.0 _ll_register_callbacks cb 0x5620baa342d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab281e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab27760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab27ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab27760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56670/0 (socket says 172.21.15.116:56670) 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f259c640 1 -- 172.21.15.116:0/886669336 learned_addr learned my addr 172.21.15.116:0/886669336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 -- 172.21.15.116:0/886669336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 msgr2=0x5620bab27760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab27760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 -- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 msgr2=0x5620bab27ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab27ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 -- 172.21.15.116:0/886669336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620ba7983f0 con 0x5620bab2a7d0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab27ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab281e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec01f520 tx=0x7fc3ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f44c06118876a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.264+0000 7fc3d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec026070 con 0x5620bab2a7d0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc3ec004b20 con 0x5620bab2a7d0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3e401a070 con 0x5620bab2a7d0 2024-03-19T02:34:44.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec004d20 con 0x5620bab2a7d0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 10 client.17676 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5620ba8627d0 con 0x5620bab2a7d0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3e401a070 con 0x5620bab2a7d0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc3ec01d040 con 0x5620bab2a7d0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 10 client.17676 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 10 client.17676 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 20 client.17676 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620bab3abb0 0x5620bab3d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 4 client.17676 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620baae23e0 con 0x5620bab3abb0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.265+0000 7fc3f259c640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620bab3abb0 0x5620bab3d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.266+0000 7fc3f259c640 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620bab3abb0 0x5620bab3d010 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60f21b02e9cfc1e8 server_cookie=1f2e6b4dc454502f in_seq=0 out_seq=0 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.266+0000 7fc3d9ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:44.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/886669336 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620baae23e0 con 0x5620bab3abb0 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3d9ffb640 10 client.17676 handle_command_reply: tid=0 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 client.17676 shutdown 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 msgr2=0x5620bab281e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab281e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec01f520 tx=0x7fc3ec004ab0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 shutdown_connections 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620bab3abb0 0x5620bab3d010 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab281e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab21bf0 0x5620bab27ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/886669336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab212d0 0x5620bab27760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 >> 172.21.15.116:0/886669336 conn(0x5620bab1ba00 msgr2=0x5620bab1d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3d9ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 shutdown_connections 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 1 -- 172.21.15.116:0/886669336 wait complete. 2024-03-19T02:34:44.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.267+0000 7fc3fbba4740 20 client.17676 trim_cache size 0 max 16384 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.281+0000 7fc3fbba4740 1 Processor -- start 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.281+0000 7fc3fbba4740 1 -- start start 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab29e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab2a7d0 0x5620bab268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d400ff60 con 0x5620bab21bf0 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d4011aa0 con 0x5620bab212d0 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d4010630 con 0x5620bab2a7d0 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47222/0 (socket says 172.21.15.116:47222) 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 -- 172.21.15.116:0/1616374333 learned_addr learned my addr 172.21.15.116:0/1616374333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3f259c640 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab29e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab2a7d0 0x5620bab268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 -- 172.21.15.116:0/1616374333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab2a7d0 msgr2=0x5620bab268c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab2a7d0 0x5620bab268c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 -- 172.21.15.116:0/1616374333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 msgr2=0x5620bab29e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab29e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 -- 172.21.15.116:0/1616374333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620bab3d690 con 0x5620bab21bf0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0b9620 tx=0x7fc3dc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c806b85bc24b7a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3dbfff640 1 -- 172.21.15.116:0/1616374333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0bd350 con 0x5620bab21bf0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3dbfff640 1 -- 172.21.15.116:0/1616374333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3dc0d5d10 con 0x5620bab21bf0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.282+0000 7fc3dbfff640 1 -- 172.21.15.116:0/1616374333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0b92c0 con 0x5620bab21bf0 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1616374333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 msgr2=0x5620bab363c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0b9620 tx=0x7fc3dc009260 comp rx=0 tx=0).stop 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1616374333 shutdown_connections 2024-03-19T02:34:44.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab2a7d0 0x5620bab268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab21bf0 0x5620bab363c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/1616374333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab29e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1616374333 >> 172.21.15.116:0/1616374333 conn(0x5620bab1ba00 msgr2=0x5620bab1a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1616374333 shutdown_connections 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.283+0000 7fc3fbba4740 1 -- 172.21.15.116:0/1616374333 wait complete. 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 -- start start 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 Processor -- start 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab2e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d40113d0 con 0x5620bab2a7d0 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d4010d00 con 0x5620bab212d0 2024-03-19T02:34:44.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d400df60 con 0x5620bab21bf0 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3fbba4740 10 client.0 _ll_register_callbacks cb 0x7fc3ec009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3f2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab2e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56682/0 (socket says 172.21.15.116:56682) 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 -- 172.21.15.116:0/350117755 learned_addr learned my addr 172.21.15.116:0/350117755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3f259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab2e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37414/0 (socket says 172.21.15.116:37414) 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3f2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47226/0 (socket says 172.21.15.116:47226) 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 -- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 msgr2=0x5620bab2e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab2e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 -- 172.21.15.116:0/350117755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 msgr2=0x5620bab2f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 -- 172.21.15.116:0/350117755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620bab170a0 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3f2d9d640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3ebfff640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc004650 tx=0x7fc3dc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=944343bf471ef77b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.284+0000 7fc3d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0db500 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7fc3dc0bceb0 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3dc0d5d10 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc3dc0c53f0 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 10 client.27436 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5620baba1350 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3dc0d5d10 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc3dc0c4f60 con 0x5620bab21bf0 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 10 client.27436 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 10 client.27436 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 20 client.27436 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620baba1820 0x5620baba3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 4 client.27436 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620baa919f0 con 0x5620baba1820 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.285+0000 7fc3f259c640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620baba1820 0x5620baba3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3f259c640 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620baba1820 0x5620baba3c10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc66ce779162ebba server_cookie=38eae0d624f37f30 in_seq=0 out_seq=0 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3d9ffb640 10 client.27436 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3d9ffb640 1 -- 172.21.15.116:0/350117755 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620baa919f0 con 0x5620baba1820 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3d9ffb640 10 client.27436 handle_command_reply: tid=0 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 client.27436 shutdown 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 msgr2=0x5620bab2eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc004650 tx=0x7fc3dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 shutdown_connections 2024-03-19T02:34:44.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5620baba1820 0x5620baba3c10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620bab2a7d0 0x5620bab2f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620bab21bf0 0x5620bab2eb50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 --2- 172.21.15.116:0/350117755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620bab212d0 0x5620bab2e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3d9ffb640 0 client.27436 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 >> 172.21.15.116:0/350117755 conn(0x5620bab1ba00 msgr2=0x5620bab1cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 shutdown_connections 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 1 -- 172.21.15.116:0/350117755 wait complete. 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.286+0000 7fc3fbba4740 20 client.27436 trim_cache size 0 max 16384 2024-03-19T02:34:44.294 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:44.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 msgr2=0x7fc3e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 0x7fc3e00a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x5620bab1cb90 tx=0x7fc3ec009bf0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 msgr2=0x7fc3f4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f4198520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009230 tx=0x7fc3dc0045e0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 shutdown_connections 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3e00a2970 0x7fc3e00a4e30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f41066b0 0x7fc3f419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f4101060 0x7fc3f4198520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 --2- 172.21.15.116:0/3236382836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f41006a0 0x7fc3f4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 >> 172.21.15.116:0/3236382836 conn(0x7fc3f40778d0 msgr2=0x7fc3f4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 shutdown_connections 2024-03-19T02:34:44.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.287+0000 7fc3f8edc640 1 -- 172.21.15.116:0/3236382836 wait complete. 2024-03-19T02:34:44.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:34:44.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- 172.21.15.116:0/298407057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 msgr2=0x7faf18100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/298407057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100140 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7faf08009930 tx=0x7faf0802bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- 172.21.15.116:0/298407057 shutdown_connections 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/298407057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf18100680 0x7faf1810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/298407057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100140 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/298407057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf181077e0 0x7faf18107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- 172.21.15.116:0/298407057 >> 172.21.15.116:0/298407057 conn(0x7faf18076c10 msgr2=0x7faf180fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- 172.21.15.116:0/298407057 shutdown_connections 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- 172.21.15.116:0/298407057 wait complete. 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 Processor -- start 2024-03-19T02:34:44.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.415+0000 7faf1f7b9640 1 -- start start 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf18100680 0x7faf181013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf181077e0 0x7faf18101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf18111aa0 con 0x7faf180ffcc0 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf18111920 con 0x7faf18100680 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf18111c20 con 0x7faf181077e0 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47236/0 (socket says 172.21.15.116:47236) 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 -- 172.21.15.116:0/4114846489 learned_addr learned my addr 172.21.15.116:0/4114846489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1cd54640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf18100680 0x7faf181013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf181077e0 0x7faf18101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 -- 172.21.15.116:0/4114846489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf181077e0 msgr2=0x7faf18101920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf181077e0 0x7faf18101920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 -- 172.21.15.116:0/4114846489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf18100680 msgr2=0x7faf181013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf18100680 0x7faf181013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 -- 172.21.15.116:0/4114846489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf181a6520 con 0x7faf180ffcc0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1d555640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faf1400e9f0 tx=0x7faf1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf0e7fc640 1 -- 172.21.15.116:0/4114846489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1400ccc0 con 0x7faf180ffcc0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf0e7fc640 1 -- 172.21.15.116:0/4114846489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf140108e0 con 0x7faf180ffcc0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf181a6810 con 0x7faf180ffcc0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.416+0000 7faf0e7fc640 1 -- 172.21.15.116:0/4114846489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf140187c0 con 0x7faf180ffcc0 2024-03-19T02:34:44.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.417+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf181a6d80 con 0x7faf180ffcc0 2024-03-19T02:34:44.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 Processor -- start 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf0e7fc640 1 -- 172.21.15.116:0/4114846489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faf14010a90 con 0x7faf180ffcc0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 -- start start 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 0x557b4bbfabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4bb524b0 con 0x557b4bbefbf0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557b4bafad70 con 0x557b4bbf87d0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.419+0000 7faf223a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557b4bb4e550 con 0x557b4bbef2d0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1d555640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1d555640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47244/0 (socket says 172.21.15.116:47244) 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1d555640 1 -- 172.21.15.116:0/4002243770 learned_addr learned my addr 172.21.15.116:0/4002243770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1cd54640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 0x557b4bbfabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf0e7fc640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 0x7faf040a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 -- 172.21.15.116:0/4002243770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 msgr2=0x557b4bbfabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 0x557b4bbfabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 -- 172.21.15.116:0/4002243770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 msgr2=0x557b4bbf8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 -- 172.21.15.116:0/4002243770 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4bb1d210 con 0x557b4bbef2d0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1d555640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1cd54640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 0x557b4bbfabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf0e7fc640 1 -- 172.21.15.116:0/4114846489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7faf140cf4a0 con 0x7faf180ffcc0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 0x7faf040a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbef6b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf10009f10 tx=0x7faf10011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221532ca331eb2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.420+0000 7faf037fe640 1 -- 172.21.15.116:0/4002243770 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf10004030 con 0x557b4bbef2d0 2024-03-19T02:34:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf037fe640 1 -- 172.21.15.116:0/4002243770 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf10004830 con 0x557b4bbef2d0 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf037fe640 1 -- 172.21.15.116:0/4002243770 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1001c640 con 0x557b4bbef2d0 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 -- 172.21.15.116:0/4002243770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 msgr2=0x557b4bbef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf1dd56640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 0x7faf040a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf10020000 tx=0x7faf10021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbef6b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf10009f10 tx=0x7faf10011810 comp rx=0 tx=0).stop 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 -- 172.21.15.116:0/4002243770 shutdown_connections 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbf87d0 0x557b4bbfabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 --2- 172.21.15.116:0/4002243770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbef6b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.421+0000 7faf223a1740 1 -- 172.21.15.116:0/4002243770 >> 172.21.15.116:0/4002243770 conn(0x557b4bbe9a00 msgr2=0x557b4bbe9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.422+0000 7faf223a1740 1 -- 172.21.15.116:0/4002243770 shutdown_connections 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.422+0000 7faf223a1740 1 -- 172.21.15.116:0/4002243770 wait complete. 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.422+0000 7faf223a1740 1 -- start start 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.422+0000 7faf223a1740 1 Processor -- start 2024-03-19T02:34:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbefbf0 0x557b4bbf5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbf62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4bbfb5c0 con 0x557b4bbf87d0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557b4bb0e310 con 0x557b4bbefbf0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557b4bbfb740 con 0x557b4bbef2d0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56716/0 (socket says 172.21.15.116:56716) 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 -- 172.21.15.116:0/620777146 learned_addr learned my addr 172.21.15.116:0/620777146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf223a1740 10 client.0 _ll_register_callbacks cb 0x557b4bb022d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1cd54640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbefbf0 0x557b4bbf5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1dd56640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbf62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 -- 172.21.15.116:0/620777146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbefbf0 msgr2=0x557b4bbf5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbefbf0 0x557b4bbf5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 -- 172.21.15.116:0/620777146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 msgr2=0x557b4bbf62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbf62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 -- 172.21.15.116:0/620777146 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b4b8663f0 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf1d555640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf14098190 tx=0x7faf140097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2649ead164877dfc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.423+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14010e00 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7faf140cf120 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf10004830 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf1409bec0 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 10 client.27448 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b4b9307d0 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf10004830 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faf1409ac10 con 0x557b4bbef2d0 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 10 client.27448 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 10 client.27448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 20 client.27448 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc0aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 4 client.27448 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b4bbb03e0 con 0x557b4bc08ad0 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.424+0000 7faf1cd54640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc0aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.425+0000 7faf00ff9640 10 client.27448 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.425+0000 7faf1cd54640 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc0aec0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d3bf310c0b665f7 server_cookie=2b2866b6263d822d in_seq=0 out_seq=0 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.425+0000 7faf00ff9640 1 -- 172.21.15.116:0/620777146 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b4bbb03e0 con 0x557b4bc08ad0 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.425+0000 7faf00ff9640 10 client.27448 handle_command_reply: tid=0 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.425+0000 7faf223a1740 1 client.27448 shutdown 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 msgr2=0x557b4bbf5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf14098190 tx=0x7faf140097a0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 shutdown_connections 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc0aec0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbf62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf00ff9640 0 client.27448 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbefbf0 0x557b4bbf5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 --2- 172.21.15.116:0/620777146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbef2d0 0x557b4bbf5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 >> 172.21.15.116:0/620777146 conn(0x557b4bbe9a00 msgr2=0x557b4bbeb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 shutdown_connections 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 1 -- 172.21.15.116:0/620777146 wait complete. 2024-03-19T02:34:44.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.426+0000 7faf223a1740 20 client.27448 trim_cache size 0 max 16384 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 Processor -- start 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 -- start start 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bc04360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 0x557b4bc10790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf181b0ea0 con 0x557b4bbefbf0 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf181b29e0 con 0x557b4bbef2d0 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.442+0000 7faf223a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf181b1570 con 0x557b4bbf87d0 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1dd56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 0x557b4bc10790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1d555640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bc04360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47256/0 (socket says 172.21.15.116:47256) 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1dd56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 0x557b4bc10790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56720/0 (socket says 172.21.15.116:56720) 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 -- 172.21.15.116:0/3466457983 learned_addr learned my addr 172.21.15.116:0/3466457983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 -- 172.21.15.116:0/3466457983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 msgr2=0x557b4bc10790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 0x557b4bc10790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 -- 172.21.15.116:0/3466457983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 msgr2=0x557b4bc04360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bc04360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 -- 172.21.15.116:0/3466457983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4bbefa20 con 0x557b4bbefbf0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf1cd54640 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faf08036040 tx=0x7faf0802c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6380f275c113ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf02ffd640 1 -- 172.21.15.116:0/3466457983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf08051070 con 0x557b4bbefbf0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf02ffd640 1 -- 172.21.15.116:0/3466457983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0802cc90 con 0x557b4bbefbf0 2024-03-19T02:34:44.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.443+0000 7faf02ffd640 1 -- 172.21.15.116:0/3466457983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf080055d0 con 0x557b4bbefbf0 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- 172.21.15.116:0/3466457983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 msgr2=0x557b4bbf60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faf08036040 tx=0x7faf0802c3d0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- 172.21.15.116:0/3466457983 shutdown_connections 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbf87d0 0x557b4bc10790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbefbf0 0x557b4bbf60d0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 --2- 172.21.15.116:0/3466457983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bc04360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- 172.21.15.116:0/3466457983 >> 172.21.15.116:0/3466457983 conn(0x557b4bbe9a00 msgr2=0x557b4bbe8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- 172.21.15.116:0/3466457983 shutdown_connections 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- 172.21.15.116:0/3466457983 wait complete. 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.444+0000 7faf223a1740 1 -- start start 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 Processor -- start 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bbfc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbefbf0 0x557b4bbfc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf181b2310 con 0x557b4bbf87d0 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf181b1c40 con 0x557b4bbef2d0 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf181aeea0 con 0x557b4bbefbf0 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf223a1740 10 client.0 _ll_register_callbacks cb 0x7faf140d2d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47260/0 (socket says 172.21.15.116:47260) 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 -- 172.21.15.116:0/1736458447 learned_addr learned my addr 172.21.15.116:0/1736458447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1cd54640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbefbf0 0x557b4bbfc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1d555640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bbfc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 -- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbefbf0 msgr2=0x557b4bbfc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbefbf0 0x557b4bbfc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 -- 172.21.15.116:0/1736458447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 msgr2=0x557b4bbfc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bbfc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.445+0000 7faf1dd56640 1 -- 172.21.15.116:0/1736458447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b4bc6e010 con 0x557b4bbf87d0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf1dd56640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faf10011d10 tx=0x7faf10009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af9742f5290f616 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf10003970 con 0x557b4bbf87d0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7faf1001ca40 con 0x557b4bbf87d0 2024-03-19T02:34:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf0802cc90 con 0x557b4bbf87d0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf10041050 con 0x557b4bbf87d0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 10 client.17700 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b4bc6f2b0 con 0x557b4bbf87d0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0802cc90 con 0x557b4bbf87d0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faf1002f050 con 0x557b4bbf87d0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 10 client.17700 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 10 client.17700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.446+0000 7faf223a1740 20 client.17700 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc71890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf223a1740 4 client.17700 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b4bb5f9f0 con 0x557b4bc08ad0 2024-03-19T02:34:44.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf1d555640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc71890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf00ff9640 10 client.17700 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf1d555640 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc71890 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e0b2ae9c632413d server_cookie=a7af402da6828ed8 in_seq=0 out_seq=0 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf00ff9640 1 -- 172.21.15.116:0/1736458447 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b4bb5f9f0 con 0x557b4bc08ad0 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf00ff9640 10 client.17700 handle_command_reply: tid=0 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.447+0000 7faf223a1740 1 client.17700 shutdown 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 msgr2=0x557b4bbfcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faf10011d10 tx=0x7faf10009ab0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 shutdown_connections 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557b4bc08ad0 0x557b4bc71890 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4bbf87d0 0x557b4bbfcf10 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf00ff9640 0 client.17700 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557b4bbefbf0 0x557b4bbfc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 --2- 172.21.15.116:0/1736458447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b4bbef2d0 0x557b4bbfc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 >> 172.21.15.116:0/1736458447 conn(0x557b4bbe9a00 msgr2=0x557b4bbeaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 shutdown_connections 2024-03-19T02:34:44.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 1 -- 172.21.15.116:0/1736458447 wait complete. 2024-03-19T02:34:44.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf223a1740 20 client.17700 trim_cache size 0 max 16384 2024-03-19T02:34:44.455 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 msgr2=0x7faf040a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 0x7faf040a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faf10020000 tx=0x7faf10021000 comp rx=0 tx=0).stop 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 msgr2=0x7faf18100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.448+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faf1400e9f0 tx=0x7faf1400edf0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 shutdown_connections 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf040a29f0 0x7faf040a4eb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf181077e0 0x7faf18101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf18100680 0x7faf181013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 --2- 172.21.15.116:0/4114846489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf180ffcc0 0x7faf18100ea0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 >> 172.21.15.116:0/4114846489 conn(0x7faf18076c10 msgr2=0x7faf181047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 shutdown_connections 2024-03-19T02:34:44.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.449+0000 7faf1f7b9640 1 -- 172.21.15.116:0/4114846489 wait complete. 2024-03-19T02:34:44.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:34:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 -- 172.21.15.116:0/1783199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434100dd0 msgr2=0x7f9434101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 --2- 172.21.15.116:0/1783199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434100dd0 0x7f9434101250 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9428009930 tx=0x7f942802bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 -- 172.21.15.116:0/1783199542 shutdown_connections 2024-03-19T02:34:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 --2- 172.21.15.116:0/1783199542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434101790 0x7f943410d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 --2- 172.21.15.116:0/1783199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434100dd0 0x7f9434101250 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 --2- 172.21.15.116:0/1783199542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 0x7f9434106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 -- 172.21.15.116:0/1783199542 >> 172.21.15.116:0/1783199542 conn(0x7f9434076c50 msgr2=0x7f94340fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 -- 172.21.15.116:0/1783199542 shutdown_connections 2024-03-19T02:34:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.574+0000 7f9438a34640 1 -- 172.21.15.116:0/1783199542 wait complete. 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 Processor -- start 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 -- start start 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 0x7f94341981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 0x7f943419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9434110b80 con 0x7f9434101790 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9434110a00 con 0x7f94341068c0 2024-03-19T02:34:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9438a34640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9434110d00 con 0x7f9434100dd0 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f9431d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.575+0000 7f943259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 0x7f94341981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9432d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 0x7f943419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47280/0 (socket says 172.21.15.116:47280) 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f943259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 0x7f94341981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56742/0 (socket says 172.21.15.116:56742) 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 -- 172.21.15.116:0/4160349609 learned_addr learned my addr 172.21.15.116:0/4160349609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 -- 172.21.15.116:0/4160349609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 msgr2=0x7f94341981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 0x7f94341981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 -- 172.21.15.116:0/4160349609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 msgr2=0x7f943419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 0x7f943419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 -- 172.21.15.116:0/4160349609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943419d1c0 con 0x7f9434101790 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9431d9b640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9428009230 tx=0x7f942802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f94237fe640 1 -- 172.21.15.116:0/4160349609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428043020 con 0x7f9434101790 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943419d450 con 0x7f9434101790 2024-03-19T02:34:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f94237fe640 1 -- 172.21.15.116:0/4160349609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f942802ca70 con 0x7f9434101790 2024-03-19T02:34:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f94237fe640 1 -- 172.21.15.116:0/4160349609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428003a50 con 0x7f9434101790 2024-03-19T02:34:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.576+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94341a4d40 con 0x7f9434101790 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f94237fe640 1 -- 172.21.15.116:0/4160349609 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9428046020 con 0x7f9434101790 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 Processor -- start 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 -- start start 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee21080bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee20fd84b0 con 0x55ee2107e7d0 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ee20f80d70 con 0x55ee210752d0 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943b61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ee20fd4550 con 0x55ee21075bf0 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f9431d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee21080bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f9432d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f9432d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37470/0 (socket says 172.21.15.116:37470) 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f9431d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee21080bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47284/0 (socket says 172.21.15.116:47284) 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f9432d9d640 1 -- 172.21.15.116:0/2957094179 learned_addr learned my addr 172.21.15.116:0/2957094179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.579+0000 7f943259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 -- 172.21.15.116:0/2957094179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 msgr2=0x55ee2107e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 -- 172.21.15.116:0/2957094179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 msgr2=0x55ee21080bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f94237fe640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 0x7f941c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee21080bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 -- 172.21.15.116:0/2957094179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee20fa3210 con 0x55ee210752d0 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f943259c640 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f94237fe640 1 -- 172.21.15.116:0/4160349609 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f94280f1110 con 0x7f9434101790 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 0x7f941c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9432d9d640 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f942c009fd0 tx=0x7f942c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=815034ac6c258de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9420ff9640 1 -- 172.21.15.116:0/2957094179 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c004250 con 0x55ee210752d0 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9420ff9640 1 -- 172.21.15.116:0/2957094179 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f942c003db0 con 0x55ee210752d0 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.580+0000 7f9420ff9640 1 -- 172.21.15.116:0/2957094179 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c01bb80 con 0x55ee210752d0 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- 172.21.15.116:0/2957094179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 msgr2=0x55ee210756b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f9432d9d640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 0x7f941c0a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f942c020000 tx=0x7f942c021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f942c009fd0 tx=0x7f942c0118d0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- 172.21.15.116:0/2957094179 shutdown_connections 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee21080bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 --2- 172.21.15.116:0/2957094179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee210752d0 0x55ee210756b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- 172.21.15.116:0/2957094179 >> 172.21.15.116:0/2957094179 conn(0x55ee2106fa00 msgr2=0x55ee2106fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- 172.21.15.116:0/2957094179 shutdown_connections 2024-03-19T02:34:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- 172.21.15.116:0/2957094179 wait complete. 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.581+0000 7f943b61b740 1 -- start start 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 Processor -- start 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee2107c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee210815c0 con 0x55ee210752d0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ee20f94310 con 0x55ee2107e7d0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ee21081740 con 0x55ee21075bf0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943b61b740 10 client.0 _ll_register_callbacks cb 0x55ee20f882d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47290/0 (socket says 172.21.15.116:47290) 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 -- 172.21.15.116:0/1229850396 learned_addr learned my addr 172.21.15.116:0/1229850396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f9431d9b640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f9432d9d640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee2107c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 -- 172.21.15.116:0/1229850396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 msgr2=0x55ee2107bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 -- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 msgr2=0x55ee2107c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee2107c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 -- 172.21.15.116:0/1229850396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee20cec3f0 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f943259c640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f94240126e0 tx=0x7f9424012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b9afd0baa6d35a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.582+0000 7f9419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424012190 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f9424004020 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f942c003db0 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9424025050 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 10 client.17712 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ee20db67d0 con 0x55ee210752d0 2024-03-19T02:34:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f942c003db0 con 0x55ee210752d0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9424003710 con 0x55ee210752d0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 10 client.17712 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 20 client.17712 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee2108eba0 0x55ee21091060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 4 client.17712 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.583+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee210363e0 con 0x55ee2108eba0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f9431d9b640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee2108eba0 0x55ee21091060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f9431d9b640 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee2108eba0 0x55ee21091060 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4656bd0bb7d0d692 server_cookie=88431312b6285229 in_seq=0 out_seq=0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f9419ffb640 10 client.17712 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f9419ffb640 1 -- 172.21.15.116:0/1229850396 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee210363e0 con 0x55ee2108eba0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f9419ffb640 10 client.17712 handle_command_reply: tid=0 2024-03-19T02:34:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.584+0000 7f943b61b740 1 client.17712 shutdown 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 msgr2=0x55ee2107b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f94240126e0 tx=0x7f9424012ba0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 shutdown_connections 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee2108eba0 0x55ee21091060 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee2107c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f9419ffb640 0 client.17712 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 --2- 172.21.15.116:0/1229850396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107b7d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 >> 172.21.15.116:0/1229850396 conn(0x55ee2106fa00 msgr2=0x55ee21071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 shutdown_connections 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 1 -- 172.21.15.116:0/1229850396 wait complete. 2024-03-19T02:34:44.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.585+0000 7f943b61b740 20 client.17712 trim_cache size 0 max 16384 2024-03-19T02:34:44.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 Processor -- start 2024-03-19T02:34:44.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 -- start start 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee21096930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94341aec40 con 0x55ee210752d0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94341b0780 con 0x55ee2107e7d0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943b61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94341af310 con 0x55ee21075bf0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47304/0 (socket says 172.21.15.116:47304) 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 -- 172.21.15.116:0/1977448703 learned_addr learned my addr 172.21.15.116:0/1977448703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f9432d9d640 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee21096930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f9431d9b640 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 -- 172.21.15.116:0/1977448703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 msgr2=0x55ee2107c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 -- 172.21.15.116:0/1977448703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 msgr2=0x55ee21096930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee21096930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 -- 172.21.15.116:0/1977448703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee210705d0 con 0x55ee210752d0 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.600+0000 7f943259c640 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9424001eb0 tx=0x7f9424005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf368cd22fed8d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f941bfff640 1 -- 172.21.15.116:0/1977448703 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424012190 con 0x55ee210752d0 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f941bfff640 1 -- 172.21.15.116:0/1977448703 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f942400fdd0 con 0x55ee210752d0 2024-03-19T02:34:44.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f941bfff640 1 -- 172.21.15.116:0/1977448703 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94240053c0 con 0x55ee210752d0 2024-03-19T02:34:44.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 -- 172.21.15.116:0/1977448703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 msgr2=0x55ee2107df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9424001eb0 tx=0x7f9424005db0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 -- 172.21.15.116:0/1977448703 shutdown_connections 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee2107e7d0 0x55ee21096930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee21075bf0 0x55ee2107c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 --2- 172.21.15.116:0/1977448703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee210752d0 0x55ee2107df00 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 -- 172.21.15.116:0/1977448703 >> 172.21.15.116:0/1977448703 conn(0x55ee2106fa00 msgr2=0x55ee2106e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 -- 172.21.15.116:0/1977448703 shutdown_connections 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.601+0000 7f943b61b740 1 -- 172.21.15.116:0/1977448703 wait complete. 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 -- start start 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 Processor -- start 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee21075bf0 0x55ee21078ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94341b00b0 con 0x55ee2107e7d0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94341af9e0 con 0x55ee21075bf0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94341acc40 con 0x55ee210752d0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943b61b740 10 client.0 _ll_register_callbacks cb 0x7f9424009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f9432d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f9432d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47306/0 (socket says 172.21.15.116:47306) 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56782/0 (socket says 172.21.15.116:56782) 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f9432d9d640 1 -- 172.21.15.116:0/3863274944 learned_addr learned my addr 172.21.15.116:0/3863274944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f9431d9b640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee21075bf0 0x55ee21078ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 -- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee21075bf0 msgr2=0x55ee21078ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee21075bf0 0x55ee21078ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 -- 172.21.15.116:0/3863274944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 msgr2=0x55ee210f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f943259c640 1 -- 172.21.15.116:0/3863274944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee210f44a0 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.602+0000 7f9432d9d640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943259c640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9424001eb0 tx=0x7f9424009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5869c099ac21dbce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424003800 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f942400fdd0 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f942400f920 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f942400fb60 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 10 client.27472 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ee210f5640 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f942400fdd0 con 0x55ee210752d0 2024-03-19T02:34:44.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f942401caf0 con 0x55ee210752d0 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 10 client.27472 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 10 client.27472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.603+0000 7f943b61b740 20 client.27472 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee210f5c70 0x55ee210f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f943b61b740 4 client.27472 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee20fe59f0 con 0x55ee210f5c70 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f9431d9b640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee210f5c70 0x55ee210f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f9419ffb640 10 client.27472 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.604+0000 7f9431d9b640 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee210f5c70 0x55ee210f8090 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2015eab47da35eb server_cookie=9d5dcbd8fc6f64b0 in_seq=0 out_seq=0 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9419ffb640 1 -- 172.21.15.116:0/3863274944 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee20fe59f0 con 0x55ee210f5c70 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9419ffb640 10 client.27472 handle_command_reply: tid=0 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 client.27472 shutdown 2024-03-19T02:34:44.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 msgr2=0x55ee21078960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9424001eb0 tx=0x7f9424009e30 comp rx=0 tx=0).stop 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 shutdown_connections 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ee210f5c70 0x55ee210f8090 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2107e7d0 0x55ee210f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9419ffb640 0 client.27472 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ee21075bf0 0x55ee21078ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 --2- 172.21.15.116:0/3863274944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ee210752d0 0x55ee21078960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 >> 172.21.15.116:0/3863274944 conn(0x55ee2106fa00 msgr2=0x55ee2106eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 shutdown_connections 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 1 -- 172.21.15.116:0/3863274944 wait complete. 2024-03-19T02:34:44.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f943b61b740 20 client.27472 trim_cache size 0 max 16384 2024-03-19T02:34:44.613 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:34:44.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 msgr2=0x7f941c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 0x7f941c0a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f942c020000 tx=0x7f942c021000 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 msgr2=0x7f9434198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.605+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9428009230 tx=0x7f942802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 shutdown_connections 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f941c0a2970 0x7f941c0a4e30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94341068c0 0x7f943419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9434101790 0x7f9434198710 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 --2- 172.21.15.116:0/4160349609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9434100dd0 0x7f94341981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 >> 172.21.15.116:0/4160349609 conn(0x7f9434076c50 msgr2=0x7f9434100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 shutdown_connections 2024-03-19T02:34:44.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:44.606+0000 7f9438a34640 1 -- 172.21.15.116:0/4160349609 wait complete. 2024-03-19T02:34:44.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 222109 3.23 MB/sec execute 247 sec latency 16.496 ms 2024-03-19T02:34:45.617 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:34:45.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:34:45.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 -- 172.21.15.116:0/1823053572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724100690 msgr2=0x7f2724100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 --2- 172.21.15.116:0/1823053572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724100690 0x7f2724100b10 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2718009930 tx=0x7f271802bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 -- 172.21.15.116:0/1823053572 shutdown_connections 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 --2- 172.21.15.116:0/1823053572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724101050 0x7f272410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 --2- 172.21.15.116:0/1823053572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724100690 0x7f2724100b10 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 --2- 172.21.15.116:0/1823053572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 0x7f2724106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 -- 172.21.15.116:0/1823053572 >> 172.21.15.116:0/1823053572 conn(0x7f2724077820 msgr2=0x7f2724077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 -- 172.21.15.116:0/1823053572 shutdown_connections 2024-03-19T02:34:45.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.732+0000 7f272a259640 1 -- 172.21.15.116:0/1823053572 wait complete. 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 Processor -- start 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 -- start start 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 0x7f2724197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 0x7f272419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2724110960 con 0x7f2724101050 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27241107e0 con 0x7f27241066a0 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f272a259640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2724110ae0 con 0x7f2724100690 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47320/0 (socket says 172.21.15.116:47320) 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f27237fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 0x7f2724197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 -- 172.21.15.116:0/2348945502 learned_addr learned my addr 172.21.15.116:0/2348945502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2723fff640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 0x7f272419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 -- 172.21.15.116:0/2348945502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 msgr2=0x7f2724197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 0x7f2724197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 -- 172.21.15.116:0/2348945502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 msgr2=0x7f272419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 0x7f272419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f2722ffd640 1 -- 172.21.15.116:0/2348945502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272419cfe0 con 0x7f2724101050 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.733+0000 7f27237fe640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 0x7f2724197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f2722ffd640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2718009230 tx=0x7f271802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f2720ff9640 1 -- 172.21.15.116:0/2348945502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718043020 con 0x7f2724101050 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272419d270 con 0x7f2724101050 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f2720ff9640 1 -- 172.21.15.116:0/2348945502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f271802ca70 con 0x7f2724101050 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f2720ff9640 1 -- 172.21.15.116:0/2348945502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718003a50 con 0x7f2724101050 2024-03-19T02:34:45.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.734+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27241a4b60 con 0x7f2724101050 2024-03-19T02:34:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.736+0000 7f2720ff9640 1 -- 172.21.15.116:0/2348945502 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2718046020 con 0x7f2724101050 2024-03-19T02:34:45.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.736+0000 7f272ce43740 1 Processor -- start 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 -- start start 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2720ff9640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 0x7f270c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8dc8160 con 0x55f1a8e64ca0 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8d70e10 con 0x55f1a8e6d880 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f272ce43740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8dc4200 con 0x55f1a8e64380 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2723fff640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 0x7f270c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2720ff9640 1 -- 172.21.15.116:0/2348945502 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f27180f1110 con 0x7f2724101050 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56806/0 (socket says 172.21.15.116:56806) 2024-03-19T02:34:45.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 -- 172.21.15.116:0/1025073871 learned_addr learned my addr 172.21.15.116:0/1025073871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2723fff640 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 -- 172.21.15.116:0/1025073871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 msgr2=0x55f1a8e6fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 -- 172.21.15.116:0/1025073871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 msgr2=0x55f1a8e6d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 -- 172.21.15.116:0/1025073871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1a8ba47e0 con 0x55f1a8e64380 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f2723fff640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 0x7f270c0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f271c005fd0 tx=0x7f271c009e60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.737+0000 7f27237fe640 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27140114f0 tx=0x7f27140118f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22816cd234199165 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f2709ffb640 1 -- 172.21.15.116:0/1025073871 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271401e070 con 0x55f1a8e64380 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f2709ffb640 1 -- 172.21.15.116:0/1025073871 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f271400f930 con 0x55f1a8e64380 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f2709ffb640 1 -- 172.21.15.116:0/1025073871 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714019980 con 0x55f1a8e64380 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 -- 172.21.15.116:0/1025073871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 msgr2=0x55f1a8e64760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27140114f0 tx=0x7f27140118f0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 -- 172.21.15.116:0/1025073871 shutdown_connections 2024-03-19T02:34:45.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 --2- 172.21.15.116:0/1025073871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64380 0x55f1a8e64760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 -- 172.21.15.116:0/1025073871 >> 172.21.15.116:0/1025073871 conn(0x55f1a8e5eab0 msgr2=0x55f1a8e5ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 -- 172.21.15.116:0/1025073871 shutdown_connections 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.738+0000 7f272ce43740 1 -- 172.21.15.116:0/1025073871 wait complete. 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 -- start start 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 Processor -- start 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e6a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8e74a80 con 0x55f1a8e64ca0 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8d841a0 con 0x55f1a8e64380 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1a8e74c00 con 0x55f1a8e6d880 2024-03-19T02:34:45.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f272ce43740 10 client.0 _ll_register_callbacks cb 0x55f1a8adbc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f27237fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e6a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f2723fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47334/0 (socket says 172.21.15.116:47334) 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.739+0000 7f2722ffd640 1 -- 172.21.15.116:0/127677102 learned_addr learned my addr 172.21.15.116:0/127677102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 -- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 msgr2=0x55f1a8e6a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e6a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 -- 172.21.15.116:0/127677102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 msgr2=0x55f1a8e6ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 -- 172.21.15.116:0/127677102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1a8c04f70 con 0x55f1a8e6d880 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f27237fe640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e6a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2722ffd640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2723fff640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6b2b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f271c009270 tx=0x7f271c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40de756c9666850 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:45.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c012e30 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f271400f930 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f271c021620 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f271c021820 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f272ce43740 10 client.27484 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1a8a9c170 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.740+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f271400f930 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f271c020050 con 0x55f1a8e6d880 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 10 client.27484 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 10 client.27484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 20 client.27484 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8e81f80 0x55f1a8e843d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 4 client.27484 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:45.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1a8dd56a0 con 0x55f1a8e81f80 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f27237fe640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8e81f80 0x55f1a8e843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f2702ffd640 10 client.27484 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.741+0000 7f27237fe640 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8e81f80 0x55f1a8e843d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530d919cf370ca2c server_cookie=cb019d62820446b2 in_seq=0 out_seq=0 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f2702ffd640 1 -- 172.21.15.116:0/127677102 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1a8dd56a0 con 0x55f1a8e81f80 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f2702ffd640 10 client.27484 handle_command_reply: tid=0 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 client.27484 shutdown 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 msgr2=0x55f1a8e6b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6b2b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f271c009270 tx=0x7f271c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 shutdown_connections 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8e81f80 0x55f1a8e843d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8e6b2b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e6ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 --2- 172.21.15.116:0/127677102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e6a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f2702ffd640 0 client.27484 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 >> 172.21.15.116:0/127677102 conn(0x55f1a8e5eab0 msgr2=0x55f1a8e60800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 shutdown_connections 2024-03-19T02:34:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 1 -- 172.21.15.116:0/127677102 wait complete. 2024-03-19T02:34:45.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.742+0000 7f272ce43740 20 client.27484 trim_cache size 0 max 16384 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.761+0000 7f272ce43740 1 Processor -- start 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.761+0000 7f272ce43740 1 -- start start 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e69770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e7d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27241aca70 con 0x55f1a8e64380 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27241ae5b0 con 0x55f1a8e6d880 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f272ce43740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27241ad140 con 0x55f1a8e64ca0 2024-03-19T02:34:45.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47344/0 (socket says 172.21.15.116:47344) 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 -- 172.21.15.116:0/606508402 learned_addr learned my addr 172.21.15.116:0/606508402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f2723fff640 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e7d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f2722ffd640 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e69770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 -- 172.21.15.116:0/606508402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64ca0 msgr2=0x55f1a8e69770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e69770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 -- 172.21.15.116:0/606508402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 msgr2=0x55f1a8e7d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e7d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 -- 172.21.15.116:0/606508402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1a8e7dd60 con 0x55f1a8e64380 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27237fe640 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f27140058c0 tx=0x7f2714014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c731cd55107b3662 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27097fa640 1 -- 172.21.15.116:0/606508402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271402f070 con 0x55f1a8e64380 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.762+0000 7f27097fa640 1 -- 172.21.15.116:0/606508402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f271402a2b0 con 0x55f1a8e64380 2024-03-19T02:34:45.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f27097fa640 1 -- 172.21.15.116:0/606508402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271401e040 con 0x55f1a8e64380 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 -- 172.21.15.116:0/606508402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 msgr2=0x55f1a8e6cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f27140058c0 tx=0x7f2714014000 comp rx=0 tx=0).stop 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 -- 172.21.15.116:0/606508402 shutdown_connections 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e6d880 0x55f1a8e7d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e69770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 --2- 172.21.15.116:0/606508402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64380 0x55f1a8e6cf60 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 -- 172.21.15.116:0/606508402 >> 172.21.15.116:0/606508402 conn(0x55f1a8e5eab0 msgr2=0x55f1a8e6b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 -- 172.21.15.116:0/606508402 shutdown_connections 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.763+0000 7f272ce43740 1 -- 172.21.15.116:0/606508402 wait complete. 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 -- start start 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 Processor -- start 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e67720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27241adee0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27241ad810 con 0x55f1a8e64380 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f27241aaa70 con 0x55f1a8e6d880 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f272ce43740 10 client.0 _ll_register_callbacks cb 0x7f271c0135a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2723fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f27237fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e67720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47352/0 (socket says 172.21.15.116:47352) 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 -- 172.21.15.116:0/734585540 learned_addr learned my addr 172.21.15.116:0/734585540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2723fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56838/0 (socket says 172.21.15.116:56838) 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 -- 172.21.15.116:0/734585540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 msgr2=0x55f1a8ee72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 -- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 msgr2=0x55f1a8e67720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.764+0000 7f2722ffd640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e67720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2722ffd640 1 -- 172.21.15.116:0/734585540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1a8e68530 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f27237fe640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e67720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2723fff640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2722ffd640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f27180b9180 tx=0x7f27180bd3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e2448a907bffde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27180d6060 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f27180053d0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f271402a2b0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f27180c15f0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 10 client.17736 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1a8ee89b0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f271402a2b0 con 0x55f1a8e64ca0 2024-03-19T02:34:45.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f27180bd700 con 0x55f1a8e64ca0 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 10 client.17736 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 20 client.17736 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8ee90e0 0x55f1a8eeb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 4 client.17736 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.765+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1a8dcf9f0 con 0x55f1a8ee90e0 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.766+0000 7f27237fe640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8ee90e0 0x55f1a8eeb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.766+0000 7f2702ffd640 10 client.17736 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.766+0000 7f27237fe640 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8ee90e0 0x55f1a8eeb500 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b80a204c619cf7f server_cookie=9880fb3715d82adf in_seq=0 out_seq=0 2024-03-19T02:34:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f2702ffd640 1 -- 172.21.15.116:0/734585540 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1a8dcf9f0 con 0x55f1a8ee90e0 2024-03-19T02:34:45.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f2702ffd640 10 client.17736 handle_command_reply: tid=0 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 client.17736 shutdown 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 msgr2=0x55f1a8e67c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f27180b9180 tx=0x7f27180bd3a0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 shutdown_connections 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1a8ee90e0 0x55f1a8eeb500 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1a8e6d880 0x55f1a8ee72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1a8e64ca0 0x55f1a8e67c60 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f2702ffd640 0 client.17736 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 --2- 172.21.15.116:0/734585540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1a8e64380 0x55f1a8e67720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.768+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 >> 172.21.15.116:0/734585540 conn(0x55f1a8e5eab0 msgr2=0x55f1a8e6a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 shutdown_connections 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272ce43740 1 -- 172.21.15.116:0/734585540 wait complete. 2024-03-19T02:34:45.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272ce43740 20 client.17736 trim_cache size 0 max 16384 2024-03-19T02:34:45.775 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:45.775 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:34:45.775 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "02bf4bb0-96c3-4d5d-8a2d-5d36ed94aa4d", 2024-03-19T02:34:45.775 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:34:45.775 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 msgr2=0x7f270c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 0x7f270c0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f271c005fd0 tx=0x7f271c009e60 comp rx=0 tx=0).stop 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 msgr2=0x7f2724198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f2718009230 tx=0x7f271802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 shutdown_connections 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f270c0a2970 0x7f270c0a4e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27241066a0 0x7f272419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2724101050 0x7f2724198530 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 --2- 172.21.15.116:0/2348945502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2724100690 0x7f2724197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 >> 172.21.15.116:0/2348945502 conn(0x7f2724077820 msgr2=0x7f2724068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 shutdown_connections 2024-03-19T02:34:45.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.769+0000 7f272a259640 1 -- 172.21.15.116:0/2348945502 wait complete. 2024-03-19T02:34:45.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:34:45.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:34:45.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 222834 3.24 MB/sec execute 248 sec latency 16.886 ms 2024-03-19T02:34:45.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 -- 172.21.15.116:0/3314505144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81077d0 msgr2=0x7f97e8107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/3314505144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81077d0 0x7f97e8107bb0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009930 tx=0x7f97d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:34:45.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 -- 172.21.15.116:0/3314505144 shutdown_connections 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/3314505144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e8102180 0x7f97e810e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/3314505144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e81017c0 0x7f97e8101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/3314505144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81077d0 0x7f97e8107bb0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 -- 172.21.15.116:0/3314505144 >> 172.21.15.116:0/3314505144 conn(0x7f97e80fd250 msgr2=0x7f97e80ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 -- 172.21.15.116:0/3314505144 shutdown_connections 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.897+0000 7f97ee4dc640 1 -- 172.21.15.116:0/3314505144 wait complete. 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 Processor -- start 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 -- start start 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 0x7f97e8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e81077d0 0x7f97e819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97e8111a90 con 0x7f97e81017c0 2024-03-19T02:34:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e8111910 con 0x7f97e8102180 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97ee4dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97e8111c10 con 0x7f97e81077d0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47370/0 (socket says 172.21.15.116:47370) 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97e7fff640 1 -- 172.21.15.116:0/1965237823 learned_addr learned my addr 172.21.15.116:0/1965237823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97e77fe640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 0x7f97e8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.898+0000 7f97eca79640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e81077d0 0x7f97e819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 -- 172.21.15.116:0/1965237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e81077d0 msgr2=0x7f97e819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e81077d0 0x7f97e819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 -- 172.21.15.116:0/1965237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 msgr2=0x7f97e8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 0x7f97e8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 -- 172.21.15.116:0/1965237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e819d010 con 0x7f97e81017c0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e77fe640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 0x7f97e8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e7fff640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009230 tx=0x7f97d802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e57fa640 1 -- 172.21.15.116:0/1965237823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8043020 con 0x7f97e81017c0 2024-03-19T02:34:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e57fa640 1 -- 172.21.15.116:0/1965237823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97d802ca70 con 0x7f97e81017c0 2024-03-19T02:34:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e819d2a0 con 0x7f97e81017c0 2024-03-19T02:34:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97e57fa640 1 -- 172.21.15.116:0/1965237823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8003a50 con 0x7f97e81017c0 2024-03-19T02:34:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.899+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e81a4b90 con 0x7f97e81017c0 2024-03-19T02:34:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.901+0000 7f97e57fa640 1 -- 172.21.15.116:0/1965237823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f97d8046020 con 0x7f97e81017c0 2024-03-19T02:34:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.901+0000 7f97f11a2740 1 Processor -- start 2024-03-19T02:34:45.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 -- start start 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cfa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5613a2c52500 con 0x5613a2ceea30 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613a2bfabe0 con 0x5613a2cef350 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97f11a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613a2c4e5a0 con 0x5613a2cf7f30 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47382/0 (socket says 172.21.15.116:47382) 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 -- 172.21.15.116:0/3384040541 learned_addr learned my addr 172.21.15.116:0/3384040541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e77fe640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cfa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 -- 172.21.15.116:0/3384040541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 msgr2=0x5613a2cfa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cfa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 -- 172.21.15.116:0/3384040541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 msgr2=0x5613a2cf79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e57fa640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 0x7f97d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97eca79640 1 -- 172.21.15.116:0/3384040541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a2c1c000 con 0x5613a2ceea30 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e7fff640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e77fe640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cfa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.902+0000 7f97e57fa640 1 -- 172.21.15.116:0/1965237823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f97d80f1120 con 0x7f97e81017c0 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.903+0000 7f97eca79640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 0x7f97d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.903+0000 7f97eca79640 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009f50 tx=0x7f97e00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8073706e36bd8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.903+0000 7f97ce7fc640 1 -- 172.21.15.116:0/3384040541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0004250 con 0x5613a2ceea30 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.903+0000 7f97ce7fc640 1 -- 172.21.15.116:0/3384040541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97e000fe50 con 0x5613a2ceea30 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.903+0000 7f97ce7fc640 1 -- 172.21.15.116:0/3384040541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e001ba80 con 0x5613a2ceea30 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- 172.21.15.116:0/3384040541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 msgr2=0x5613a2ceee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97eca79640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 0x7f97d00a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f97e0012fd0 tx=0x7f97e0021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009f50 tx=0x7f97e00118d0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- 172.21.15.116:0/3384040541 shutdown_connections 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cfa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- 172.21.15.116:0/3384040541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2ceee10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- 172.21.15.116:0/3384040541 >> 172.21.15.116:0/3384040541 conn(0x5613a2ce9160 msgr2=0x5613a2ce9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- 172.21.15.116:0/3384040541 shutdown_connections 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- 172.21.15.116:0/3384040541 wait complete. 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 -- start start 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 Processor -- start 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2ceea30 0x5613a2cf4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.904+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5613a2cfad20 con 0x5613a2cf7f30 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613a2c0e160 con 0x5613a2cef350 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613a2cfaea0 con 0x5613a2ceea30 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 10 client.0 _ll_register_callbacks cb 0x5613a2c4aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2ceea30 0x5613a2cf4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47384/0 (socket says 172.21.15.116:47384) 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 -- 172.21.15.116:0/1556678778 learned_addr learned my addr 172.21.15.116:0/1556678778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97e77fe640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 -- 172.21.15.116:0/1556678778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2ceea30 msgr2=0x5613a2cf4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2ceea30 0x5613a2cf4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 -- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 msgr2=0x5613a2cf5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 -- 172.21.15.116:0/1556678778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a2a307d0 con 0x5613a2cf7f30 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97eca79640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f97e0003f90 tx=0x7f97e001b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d43a3d5f2914f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0033030 con 0x5613a2cf7f30 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f97e0004260 con 0x5613a2cf7f30 2024-03-19T02:34:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.905+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e000fe50 con 0x5613a2cf7f30 2024-03-19T02:34:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.908+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f97e003d050 con 0x5613a2cf7f30 2024-03-19T02:34:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.908+0000 7f97f11a2740 10 client.17748 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.908+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5613a29663f0 con 0x5613a2cf7f30 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97e000fe50 con 0x5613a2cf7f30 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f97e0025940 con 0x5613a2cf7f30 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 10 client.17748 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 10 client.17748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 20 client.17748 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d08140 0x5613a2d0a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 4 client.17748 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613a2c5fa40 con 0x5613a2d08140 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.909+0000 7f97e7fff640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d08140 0x5613a2d0a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.910+0000 7f97e7fff640 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d08140 0x5613a2d0a530 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ad8bdc4ebc47033 server_cookie=282d55594fc7e33c in_seq=0 out_seq=0 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.910+0000 7f97c77fe640 10 client.17748 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97c77fe640 1 -- 172.21.15.116:0/1556678778 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613a2c5fa40 con 0x5613a2d08140 2024-03-19T02:34:45.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97c77fe640 10 client.17748 handle_command_reply: tid=0 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 client.17748 shutdown 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 msgr2=0x5613a2cf58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f97e0003f90 tx=0x7f97e001b430 comp rx=0 tx=0).stop 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 shutdown_connections 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d08140 0x5613a2d0a530 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cf7f30 0x5613a2cf58c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cef350 0x5613a2cf5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97c77fe640 0 client.17748 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 --2- 172.21.15.116:0/1556678778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2ceea30 0x5613a2cf4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 >> 172.21.15.116:0/1556678778 conn(0x5613a2ce9160 msgr2=0x5613a2ceaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 shutdown_connections 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 1 -- 172.21.15.116:0/1556678778 wait complete. 2024-03-19T02:34:45.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.911+0000 7f97f11a2740 20 client.17748 trim_cache size 0 max 16384 2024-03-19T02:34:45.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.930+0000 7f97f11a2740 1 Processor -- start 2024-03-19T02:34:45.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.930+0000 7f97f11a2740 1 -- start start 2024-03-19T02:34:45.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 0x5613a2cf3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cf7f30 0x5613a2cf56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81aed80 con 0x5613a2ceea30 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81b08c0 con 0x5613a2cf7f30 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97f11a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81af450 con 0x5613a2cef350 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47392/0 (socket says 172.21.15.116:47392) 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 0x5613a2cf3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e7fff640 1 -- 172.21.15.116:0/2777130281 learned_addr learned my addr 172.21.15.116:0/2777130281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97eca79640 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cf7f30 0x5613a2cf56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 -- 172.21.15.116:0/2777130281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cf7f30 msgr2=0x5613a2cf56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cf7f30 0x5613a2cf56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 -- 172.21.15.116:0/2777130281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 msgr2=0x5613a2cf7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 -- 172.21.15.116:0/2777130281 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a29260f0 con 0x5613a2cef350 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e7fff640 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97e77fe640 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 0x5613a2cf3e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5613a2cf46a0 tx=0x7f97dc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32516707b3415594 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97cdffb640 1 -- 172.21.15.116:0/2777130281 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc022070 con 0x5613a2cef350 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97cdffb640 1 -- 172.21.15.116:0/2777130281 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97dc0035f0 con 0x5613a2cef350 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.931+0000 7f97cdffb640 1 -- 172.21.15.116:0/2777130281 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00f740 con 0x5613a2cef350 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 -- 172.21.15.116:0/2777130281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 msgr2=0x5613a2cf3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 0x5613a2cf3e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5613a2cf46a0 tx=0x7f97dc005f90 comp rx=0 tx=0).stop 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 -- 172.21.15.116:0/2777130281 shutdown_connections 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2cf7f30 0x5613a2cf56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cef350 0x5613a2cf3e70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 --2- 172.21.15.116:0/2777130281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2ceea30 0x5613a2cf7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 -- 172.21.15.116:0/2777130281 >> 172.21.15.116:0/2777130281 conn(0x5613a2ce9160 msgr2=0x5613a2ce7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 -- 172.21.15.116:0/2777130281 shutdown_connections 2024-03-19T02:34:45.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.932+0000 7f97f11a2740 1 -- 172.21.15.116:0/2777130281 wait complete. 2024-03-19T02:34:45.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 -- start start 2024-03-19T02:34:45.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 Processor -- start 2024-03-19T02:34:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 0x5613a2cf1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cf2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81b01f0 con 0x5613a2cef350 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81afb20 con 0x5613a2ceea30 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97e81acd80 con 0x5613a2cf7f30 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97f11a2740 10 client.0 _ll_register_callbacks cb 0x7f97d80f4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47406/0 (socket says 172.21.15.116:47406) 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97e7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 0x5613a2cf1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97e77fe640 1 -- 172.21.15.116:0/422055581 learned_addr learned my addr 172.21.15.116:0/422055581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.933+0000 7f97eca79640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cf2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 -- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 msgr2=0x5613a2cf1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 0x5613a2cf1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 -- 172.21.15.116:0/422055581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 msgr2=0x5613a2cf1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 -- 172.21.15.116:0/422055581 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a2d6d390 con 0x5613a2cf7f30 2024-03-19T02:34:45.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97e77fe640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97e7fff640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 0x5613a2cf1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97eca79640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cf2400 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e0025de0 tx=0x7f97e00094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f373375e54af5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0033040 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97dc0035f0 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9510+0+0 (secure 0 0 0) 0x7f97e0009b30 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f97e0054050 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97f11a2740 10 client.27508 fetch_fsmap learned FSMap version 30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5613a2d6e600 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97dc0035f0 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.934+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f97e00256f0 con 0x5613a2cf7f30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 10 client.27508 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 10 client.27508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:34:45.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 20 client.27508 populate_metadata read hostname 'smithi116' 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d6ece0 0x5613a2d71100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 4 client.27508 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5613a2cf44f0 con 0x5613a2d6ece0 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97e7fff640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d6ece0 0x5613a2d71100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97c77fe640 10 client.27508 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97e7fff640 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d6ece0 0x5613a2d71100 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae62d007f906fa61 server_cookie=6577785b77214d46 in_seq=0 out_seq=0 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97c77fe640 1 -- 172.21.15.116:0/422055581 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5613a2cf44f0 con 0x5613a2d6ece0 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97c77fe640 10 client.27508 handle_command_reply: tid=0 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.935+0000 7f97f11a2740 1 client.27508 shutdown 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 msgr2=0x5613a2cf2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cf2400 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e0025de0 tx=0x7f97e00094a0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 shutdown_connections 2024-03-19T02:34:45.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5613a2d6ece0 0x5613a2d71100 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613a2cf7f30 0x5613a2cf2400 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613a2cef350 0x5613a2cf1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 --2- 172.21.15.116:0/422055581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a2ceea30 0x5613a2cf1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 >> 172.21.15.116:0/422055581 conn(0x5613a2ce9160 msgr2=0x5613a2ce81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97c77fe640 0 client.27508 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 shutdown_connections 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 1 -- 172.21.15.116:0/422055581 wait complete. 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97f11a2740 20 client.27508 trim_cache size 0 max 16384 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:34:45.943 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: "02bf4bb0-96c3-4d5d-8a2d-5d36ed94aa4d": { 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "02bf4bb0-96c3-4d5d-8a2d-5d36ed94aa4d", 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:34:45.944 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:34:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 msgr2=0x7f97d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 0x7f97d00a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f97e0012fd0 tx=0x7f97e0021000 comp rx=0 tx=0).stop 2024-03-19T02:34:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 msgr2=0x7f97e8198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.936+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009230 tx=0x7f97d802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 shutdown_connections 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f97d00a2970 0x7f97d00a4e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97e81077d0 0x7f97e819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8102180 0x7f97e8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 --2- 172.21.15.116:0/1965237823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97e81017c0 0x7f97e8198020 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 >> 172.21.15.116:0/1965237823 conn(0x7f97e80fd250 msgr2=0x7f97e80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 shutdown_connections 2024-03-19T02:34:45.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:34:45.937+0000 7f97ee4dc640 1 -- 172.21.15.116:0/1965237823 wait complete. 2024-03-19T02:34:45.949 INFO:tasks.cephfs.filesystem:scrub status for tag:02bf4bb0-96c3-4d5d-8a2d-5d36ed94aa4d - {'path': '/', 'tag': '02bf4bb0-96c3-4d5d-8a2d-5d36ed94aa4d', 'options': 'recursive,force'} 2024-03-19T02:34:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:45 smithi162 ceph-mon[35761]: pgmap v213: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-03-19T02:34:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:45 smithi154 ceph-mon[35693]: pgmap v213: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-03-19T02:34:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:45 smithi116 ceph-mon[32263]: pgmap v213: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 2.2 MiB/s wr, 126 op/s 2024-03-19T02:34:46.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 223297 3.24 MB/sec execute 249 sec latency 30.075 ms 2024-03-19T02:34:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:34:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:34:47.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:34:47.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:46 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:46 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:46 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:34:47.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 223868 3.24 MB/sec execute 250 sec latency 18.362 ms 2024-03-19T02:34:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:47 smithi162 ceph-mon[35761]: pgmap v214: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:47 smithi154 ceph-mon[35693]: pgmap v214: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:47 smithi116 ceph-mon[32263]: pgmap v214: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:48.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 224323 3.23 MB/sec execute 251 sec latency 30.821 ms 2024-03-19T02:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 224937 3.24 MB/sec execute 252 sec latency 22.524 ms 2024-03-19T02:34:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:49 smithi162 ceph-mon[35761]: pgmap v215: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:49 smithi154 ceph-mon[35693]: pgmap v215: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:49 smithi116 ceph-mon[32263]: pgmap v215: 129 pgs: 129 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 86 op/s 2024-03-19T02:34:50.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 225609 3.23 MB/sec execute 253 sec latency 18.893 ms 2024-03-19T02:34:51.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 226310 3.24 MB/sec execute 254 sec latency 16.855 ms 2024-03-19T02:34:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:51 smithi162 ceph-mon[35761]: pgmap v216: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 127 op/s 2024-03-19T02:34:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:51 smithi154 ceph-mon[35693]: pgmap v216: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 127 op/s 2024-03-19T02:34:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:51 smithi116 ceph-mon[32263]: pgmap v216: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 127 op/s 2024-03-19T02:34:52.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 226869 3.24 MB/sec execute 255 sec latency 16.458 ms 2024-03-19T02:34:53.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 227438 3.24 MB/sec execute 256 sec latency 20.782 ms 2024-03-19T02:34:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:53 smithi154 ceph-mon[35693]: pgmap v217: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:53 smithi116 ceph-mon[32263]: pgmap v217: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:53 smithi162 ceph-mon[35761]: pgmap v217: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:34:54.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 227884 3.23 MB/sec execute 257 sec latency 87.114 ms 2024-03-19T02:34:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:54 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:34:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:54 smithi154 ceph-mon[35693]: pgmap v218: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:54 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:34:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:54 smithi116 ceph-mon[32263]: pgmap v218: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:54 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:34:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:54 smithi162 ceph-mon[35761]: pgmap v218: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.4 MiB/s wr, 126 op/s 2024-03-19T02:34:55.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 228560 3.24 MB/sec execute 258 sec latency 16.960 ms 2024-03-19T02:34:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:34:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:34:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:34:56.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 229205 3.23 MB/sec execute 259 sec latency 16.474 ms 2024-03-19T02:34:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:56 smithi154 ceph-mon[35693]: pgmap v219: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:34:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:56 smithi116 ceph-mon[32263]: pgmap v219: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:34:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:56 smithi162 ceph-mon[35761]: pgmap v219: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:34:57.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 229951 3.24 MB/sec execute 260 sec latency 17.620 ms 2024-03-19T02:34:58.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 230444 3.24 MB/sec execute 261 sec latency 15.677 ms 2024-03-19T02:34:59.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 231027 3.24 MB/sec execute 262 sec latency 16.337 ms 2024-03-19T02:35:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:34:59 smithi162 ceph-mon[35761]: pgmap v220: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:35:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:34:59 smithi154 ceph-mon[35693]: pgmap v220: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:35:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:34:59 smithi116 ceph-mon[32263]: pgmap v220: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-03-19T02:35:00.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 231491 3.23 MB/sec execute 263 sec latency 43.291 ms 2024-03-19T02:35:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:00 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:35:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:00 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:35:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:00 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:35:01.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 232101 3.24 MB/sec execute 264 sec latency 18.752 ms 2024-03-19T02:35:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:01 smithi162 ceph-mon[35761]: pgmap v221: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:35:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:01 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:35:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:01 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:35:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:01 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:35:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:01 smithi154 ceph-mon[35693]: pgmap v221: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:35:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:01 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:35:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:01 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:35:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:01 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:35:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:01 smithi116 ceph-mon[32263]: pgmap v221: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:35:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:01 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:35:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:01 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:35:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:01 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:35:02.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 232781 3.23 MB/sec execute 265 sec latency 18.304 ms 2024-03-19T02:35:03.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 233504 3.24 MB/sec execute 266 sec latency 17.270 ms 2024-03-19T02:35:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:03 smithi162 ceph-mon[35761]: pgmap v222: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:35:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:03 smithi154 ceph-mon[35693]: pgmap v222: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:35:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:03 smithi116 ceph-mon[32263]: pgmap v222: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:35:04.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 233976 3.24 MB/sec execute 267 sec latency 19.280 ms 2024-03-19T02:35:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 234529 3.24 MB/sec execute 268 sec latency 19.395 ms 2024-03-19T02:35:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:05 smithi162 ceph-mon[35761]: pgmap v223: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.5 MiB/s wr, 141 op/s 2024-03-19T02:35:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:05 smithi162 ceph-mon[35761]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:35:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:05 smithi154 ceph-mon[35693]: pgmap v223: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.5 MiB/s wr, 141 op/s 2024-03-19T02:35:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:05 smithi154 ceph-mon[35693]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:35:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:05 smithi116 ceph-mon[32263]: pgmap v223: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.5 MiB/s wr, 141 op/s 2024-03-19T02:35:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:05 smithi116 ceph-mon[32263]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:35:06.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 235036 3.23 MB/sec execute 269 sec latency 33.602 ms 2024-03-19T02:35:07.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 235532 3.24 MB/sec execute 270 sec latency 271.398 ms 2024-03-19T02:35:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:07 smithi162 ceph-mon[35761]: pgmap v225: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:07 smithi154 ceph-mon[35693]: pgmap v225: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:07 smithi116 ceph-mon[32263]: pgmap v225: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:08.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 235532 3.22 MB/sec execute 271 sec latency 1271.483 ms 2024-03-19T02:35:09.788 INFO:tasks.workunit.client.0.smithi116.stdout: 1 235666 3.21 MB/sec execute 272 sec latency 1961.055 ms 2024-03-19T02:35:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:09 smithi162 ceph-mon[35761]: pgmap v226: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:09 smithi154 ceph-mon[35693]: pgmap v226: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:09 smithi116 ceph-mon[32263]: pgmap v226: 129 pgs: 129 active+clean; 219 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:35:10.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 236211 3.20 MB/sec execute 273 sec latency 19.973 ms 2024-03-19T02:35:11.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 236768 3.20 MB/sec execute 274 sec latency 29.096 ms 2024-03-19T02:35:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:11 smithi162 ceph-mon[35761]: pgmap v227: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.6 MiB/s wr, 111 op/s 2024-03-19T02:35:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:11 smithi162 ceph-mon[35761]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T02:35:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:11 smithi154 ceph-mon[35693]: pgmap v227: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.6 MiB/s wr, 111 op/s 2024-03-19T02:35:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:11 smithi154 ceph-mon[35693]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T02:35:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:11 smithi116 ceph-mon[32263]: pgmap v227: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.6 MiB/s wr, 111 op/s 2024-03-19T02:35:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:11 smithi116 ceph-mon[32263]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T02:35:12.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 237479 3.21 MB/sec execute 275 sec latency 17.146 ms 2024-03-19T02:35:13.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 238006 3.22 MB/sec execute 276 sec latency 19.980 ms 2024-03-19T02:35:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:13 smithi162 ceph-mon[35761]: pgmap v229: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 788 KiB/s wr, 51 op/s 2024-03-19T02:35:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:13 smithi154 ceph-mon[35693]: pgmap v229: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 788 KiB/s wr, 51 op/s 2024-03-19T02:35:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:13 smithi116 ceph-mon[32263]: pgmap v229: 129 pgs: 129 active+clean; 216 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 788 KiB/s wr, 51 op/s 2024-03-19T02:35:14.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 238389 3.21 MB/sec execute 277 sec latency 29.685 ms 2024-03-19T02:35:15.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 238983 3.20 MB/sec execute 278 sec latency 17.568 ms 2024-03-19T02:35:15.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:35:15.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 -- 172.21.15.116:0/2172798177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 msgr2=0x7f8c4c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 --2- 172.21.15.116:0/2172798177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c107ba0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009930 tx=0x7f8c3c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 -- 172.21.15.116:0/2172798177 shutdown_connections 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 --2- 172.21.15.116:0/2172798177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 0x7f8c4c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 --2- 172.21.15.116:0/2172798177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 0x7f8c4c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 --2- 172.21.15.116:0/2172798177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c107ba0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 -- 172.21.15.116:0/2172798177 >> 172.21.15.116:0/2172798177 conn(0x7f8c4c078be0 msgr2=0x7f8c4c078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 -- 172.21.15.116:0/2172798177 shutdown_connections 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.058+0000 7f8c519f4640 1 -- 172.21.15.116:0/2172798177 wait complete. 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 Processor -- start 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 -- start start 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 0x7f8c4c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 0x7f8c4c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c111a40 con 0x7f8c4c1077c0 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1118c0 con 0x7f8c4c1017b0 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c519f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c111bc0 con 0x7f8c4c102170 2024-03-19T02:35:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50058/0 (socket says 172.21.15.116:50058) 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 0x7f8c4c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c4b7fe640 1 -- 172.21.15.116:0/39495081 learned_addr learned my addr 172.21.15.116:0/39495081 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.059+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 0x7f8c4c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 -- 172.21.15.116:0/39495081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 msgr2=0x7f8c4c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 0x7f8c4c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 -- 172.21.15.116:0/39495081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 msgr2=0x7f8c4c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 0x7f8c4c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 -- 172.21.15.116:0/39495081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c19cff0 con 0x7f8c4c1077c0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c4b7fe640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400eb10 tx=0x7f8c4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c509f2640 1 -- 172.21.15.116:0/39495081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44004b20 con 0x7f8c4c1077c0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c19d2e0 con 0x7f8c4c1077c0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c509f2640 1 -- 172.21.15.116:0/39495081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c44004cc0 con 0x7f8c4c1077c0 2024-03-19T02:35:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c509f2640 1 -- 172.21.15.116:0/39495081 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44010690 con 0x7f8c4c1077c0 2024-03-19T02:35:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.060+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c1a4b70 con 0x7f8c4c1077c0 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c509f2640 1 -- 172.21.15.116:0/39495081 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8c44005020 con 0x7f8c4c1077c0 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 Processor -- start 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 -- start start 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b7486e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b748ff30 0x5564b7492320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5564b73ea500 con 0x5564b7487350 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5564b7392be0 con 0x5564b7486a30 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c545e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564b73e65a0 con 0x5564b748ff30 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b7486e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50072/0 (socket says 172.21.15.116:50072) 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c4affd640 1 -- 172.21.15.116:0/952064584 learned_addr learned my addr 172.21.15.116:0/952064584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b748ff30 0x5564b7492320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.063+0000 7f8c509f2640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 0x7f8c340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/952064584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b748ff30 msgr2=0x5564b7492320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b748ff30 0x5564b7492320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/952064584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 msgr2=0x5564b748f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/952064584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564b73b4000 con 0x5564b7486a30 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4affd640 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4b7fe640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 0x7f8c340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c509f2640 1 -- 172.21.15.116:0/39495081 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f8c440cf790 con 0x7f8c4c1077c0 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b7486e10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c103b10 tx=0x7f8c40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eec8274b7215f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c31ffb640 1 -- 172.21.15.116:0/952064584 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40012ca0 con 0x5564b7486a30 2024-03-19T02:35:16.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c4b7fe640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 0x7f8c340a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c440ccbb0 tx=0x7f8c4406a820 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c31ffb640 1 -- 172.21.15.116:0/952064584 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c40012e40 con 0x5564b7486a30 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.064+0000 7f8c31ffb640 1 -- 172.21.15.116:0/952064584 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c400049e0 con 0x5564b7486a30 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- 172.21.15.116:0/952064584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 msgr2=0x5564b7486e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b7486e10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c103b10 tx=0x7f8c40004730 comp rx=0 tx=0).stop 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- 172.21.15.116:0/952064584 shutdown_connections 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b748ff30 0x5564b7492320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 --2- 172.21.15.116:0/952064584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b7486e10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- 172.21.15.116:0/952064584 >> 172.21.15.116:0/952064584 conn(0x5564b7481160 msgr2=0x5564b7481540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- 172.21.15.116:0/952064584 shutdown_connections 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- 172.21.15.116:0/952064584 wait complete. 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 -- start start 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.065+0000 7f8c545e5740 1 Processor -- start 2024-03-19T02:35:16.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b748d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b748d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5564b7492d20 con 0x5564b7486a30 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5564b73a6160 con 0x5564b748ff30 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5564b7492ea0 con 0x5564b7487350 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50076/0 (socket says 172.21.15.116:50076) 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/928017304 learned_addr learned my addr 172.21.15.116:0/928017304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 10 client.0 _ll_register_callbacks cb 0x5564b73e2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4affd640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b748d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b748d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/928017304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 msgr2=0x5564b748d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b748d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 msgr2=0x5564b748d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b748d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/928017304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564b71c87d0 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8c40012e00 tx=0x7f8c40005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4f2abcafc49798 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c2affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40010800 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f8c400109a0 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.066+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5564b73f7a40 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8c40036050 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 10 client.17766 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564b70fe3f0 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5564b73f7a40 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8c400304f0 con 0x5564b7486a30 2024-03-19T02:35:16.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 10 client.17766 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 20 client.17766 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b74a86f0 0x5564b74aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 4 client.17766 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564b73f1d90 con 0x5564b74a86f0 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.067+0000 7f8c4affd640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b74a86f0 0x5564b74aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.068+0000 7f8c2affd640 10 client.17766 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.068+0000 7f8c4affd640 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b74a86f0 0x5564b74aaae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d758f2b6c406e02 server_cookie=9f0163c2ac760e03 in_seq=0 out_seq=0 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c2affd640 1 -- 172.21.15.116:0/928017304 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5564b73f1d90 con 0x5564b74a86f0 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c2affd640 10 client.17766 handle_command_reply: tid=0 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 client.17766 shutdown 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 msgr2=0x5564b748cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8c40012e00 tx=0x7f8c40005520 comp rx=0 tx=0).stop 2024-03-19T02:35:16.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 shutdown_connections 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b74a86f0 0x5564b74aaae0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b748d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b748d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 --2- 172.21.15.116:0/928017304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7486a30 0x5564b748cef0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c2affd640 0 client.17766 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 >> 172.21.15.116:0/928017304 conn(0x5564b7481160 msgr2=0x5564b7482eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 shutdown_connections 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 1 -- 172.21.15.116:0/928017304 wait complete. 2024-03-19T02:35:16.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.069+0000 7f8c545e5740 20 client.17766 trim_cache size 0 max 16384 2024-03-19T02:35:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 Processor -- start 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 -- start start 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7486a30 0x5564b74a4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b74b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1acb60 con 0x5564b7487350 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1ae6a0 con 0x5564b748ff30 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c545e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1ad230 con 0x5564b7486a30 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50086/0 (socket says 172.21.15.116:50086) 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7486a30 0x5564b74a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c4affd640 1 -- 172.21.15.116:0/1696516343 learned_addr learned my addr 172.21.15.116:0/1696516343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.088+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b74b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 -- 172.21.15.116:0/1696516343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7486a30 msgr2=0x5564b74a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7486a30 0x5564b74a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 -- 172.21.15.116:0/1696516343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 msgr2=0x5564b74b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b74b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 -- 172.21.15.116:0/1696516343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564b70be0f0 con 0x5564b7487350 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c4affd640 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005ab0 tx=0x7f8c3c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e517730738cb375 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c317fa640 1 -- 172.21.15.116:0/1696516343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c04f020 con 0x5564b7487350 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c317fa640 1 -- 172.21.15.116:0/1696516343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c3c02cc90 con 0x5564b7487350 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.089+0000 7f8c317fa640 1 -- 172.21.15.116:0/1696516343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c005590 con 0x5564b7487350 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- 172.21.15.116:0/1696516343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 msgr2=0x5564b748c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005ab0 tx=0x7f8c3c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- 172.21.15.116:0/1696516343 shutdown_connections 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b748ff30 0x5564b74b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b7487350 0x5564b748c6c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 --2- 172.21.15.116:0/1696516343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7486a30 0x5564b74a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- 172.21.15.116:0/1696516343 >> 172.21.15.116:0/1696516343 conn(0x5564b7481160 msgr2=0x5564b748d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- 172.21.15.116:0/1696516343 shutdown_connections 2024-03-19T02:35:16.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- 172.21.15.116:0/1696516343 wait complete. 2024-03-19T02:35:16.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 -- start start 2024-03-19T02:35:16.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 Processor -- start 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.090+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b749bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b749c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1adfd0 con 0x5564b748ff30 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1ad900 con 0x5564b7486a30 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c1aab60 con 0x5564b7487350 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 10 client.0 _ll_register_callbacks cb 0x7f8c3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b749c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50090/0 (socket says 172.21.15.116:50090) 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/3543339355 learned_addr learned my addr 172.21.15.116:0/3543339355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b749bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/3543339355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 msgr2=0x5564b749c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b749c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 msgr2=0x5564b749bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b749bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 -- 172.21.15.116:0/3543339355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564b750ceb0 con 0x5564b748ff30 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c4a7fc640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8c40030940 tx=0x7f8c40005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0ec608819e549c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c2affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4000be20 con 0x5564b748ff30 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f8c4001b070 con 0x5564b748ff30 2024-03-19T02:35:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c02cc90 con 0x5564b748ff30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8c400302c0 con 0x5564b748ff30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 10 client.17772 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.091+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5564b750e1a0 con 0x5564b748ff30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c3c02cc90 con 0x5564b748ff30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8c400035f0 con 0x5564b748ff30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 10 client.17772 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 10 client.17772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 20 client.17772 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b750e800 0x5564b7510c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 4 client.17772 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564b736f540 con 0x5564b750e800 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.092+0000 7f8c4affd640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b750e800 0x5564b7510c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c2affd640 10 client.17772 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c4affd640 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b750e800 0x5564b7510c20 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77031fc7802c087d server_cookie=b241a6c71f54ce8f in_seq=0 out_seq=0 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c2affd640 1 -- 172.21.15.116:0/3543339355 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5564b736f540 con 0x5564b750e800 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c2affd640 10 client.17772 handle_command_reply: tid=0 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 client.17772 shutdown 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 msgr2=0x5564b750c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8c40030940 tx=0x7f8c40005520 comp rx=0 tx=0).stop 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 shutdown_connections 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5564b750e800 0x5564b7510c20 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5564b748ff30 0x5564b750c8c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5564b7487350 0x5564b749c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c2affd640 0 client.17772 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 --2- 172.21.15.116:0/3543339355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5564b7486a30 0x5564b749bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 >> 172.21.15.116:0/3543339355 conn(0x5564b7481160 msgr2=0x5564b748d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 shutdown_connections 2024-03-19T02:35:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 1 -- 172.21.15.116:0/3543339355 wait complete. 2024-03-19T02:35:16.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.093+0000 7f8c545e5740 20 client.17772 trim_cache size 0 max 16384 2024-03-19T02:35:16.101 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:16.101 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:35:16.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:35:16.101 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 msgr2=0x7f8c340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 0x7f8c340a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c440ccbb0 tx=0x7f8c4406a820 comp rx=0 tx=0).stop 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 msgr2=0x7f8c4c19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400eb10 tx=0x7f8c4400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 shutdown_connections 2024-03-19T02:35:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8c340a2970 0x7f8c340a4e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c4c1077c0 0x7f8c4c19c8c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c4c102170 0x7f8c4c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 --2- 172.21.15.116:0/39495081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c1017b0 0x7f8c4c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 >> 172.21.15.116:0/39495081 conn(0x7f8c4c078be0 msgr2=0x7f8c4c1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 shutdown_connections 2024-03-19T02:35:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.094+0000 7f8c519f4640 1 -- 172.21.15.116:0/39495081 wait complete. 2024-03-19T02:35:16.106 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:35:16.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:35:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:15 smithi162 ceph-mon[35761]: pgmap v230: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 -- 172.21.15.116:0/2316017313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b601010e0 msgr2=0x7f2b6010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 --2- 172.21.15.116:0/2316017313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b601010e0 0x7f2b6010d5f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009cd0 tx=0x7f2b5802c0e0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 -- 172.21.15.116:0/2316017313 shutdown_connections 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 --2- 172.21.15.116:0/2316017313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b601010e0 0x7f2b6010d5f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 --2- 172.21.15.116:0/2316017313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b60100720 0x7f2b60100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 --2- 172.21.15.116:0/2316017313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 0x7f2b60106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 -- 172.21.15.116:0/2316017313 >> 172.21.15.116:0/2316017313 conn(0x7f2b60077820 msgr2=0x7f2b60077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 -- 172.21.15.116:0/2316017313 shutdown_connections 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 -- 172.21.15.116:0/2316017313 wait complete. 2024-03-19T02:35:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.217+0000 7f2b685a9640 1 Processor -- start 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 -- start start 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b601010e0 0x7f2b60198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 0x7f2b6019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b601109b0 con 0x7f2b60100720 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60110830 con 0x7f2b601010e0 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b685a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60110b30 con 0x7f2b60106730 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50098/0 (socket says 172.21.15.116:50098) 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 -- 172.21.15.116:0/4243897619 learned_addr learned my addr 172.21.15.116:0/4243897619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66b46640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 0x7f2b6019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b65b44640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b601010e0 0x7f2b60198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 -- 172.21.15.116:0/4243897619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 msgr2=0x7f2b6019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 0x7f2b6019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 -- 172.21.15.116:0/4243897619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b601010e0 msgr2=0x7f2b60198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b601010e0 0x7f2b60198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 -- 172.21.15.116:0/4243897619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6019cfb0 con 0x7f2b60100720 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.218+0000 7f2b66345640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00e9f0 tx=0x7f2b5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.219+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c00cd80 con 0x7f2b60100720 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.219+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6019d2a0 con 0x7f2b60100720 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.219+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b5c0108e0 con 0x7f2b60100720 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.219+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c018920 con 0x7f2b60100720 2024-03-19T02:35:16.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.219+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b60102d10 con 0x7f2b60100720 2024-03-19T02:35:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.220+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b60101e60 con 0x7f2b60100720 2024-03-19T02:35:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.221+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2b5c010a90 con 0x7f2b60100720 2024-03-19T02:35:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.222+0000 7f2b577fe640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 0x7f2b4c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.222+0000 7f2b65b44640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 0x7f2b4c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.222+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f2b5c0cf4d0 con 0x7f2b60100720 2024-03-19T02:35:16.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.223+0000 7f2b65b44640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 0x7f2b4c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2b601028f0 tx=0x7f2b50005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.227+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b5c09b800 con 0x7f2b60100720 2024-03-19T02:35:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:15 smithi154 ceph-mon[35693]: pgmap v230: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:15 smithi116 ceph-mon[32263]: pgmap v230: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.493+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b60068610 con 0x7f2b60100720 2024-03-19T02:35:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.494+0000 7f2b577fe640 1 -- 172.21.15.116:0/4243897619 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f2b5c09f8a0 con 0x7f2b60100720 2024-03-19T02:35:16.495 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:35:16.495 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:35:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 msgr2=0x7f2b4c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 0x7f2b4c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2b601028f0 tx=0x7f2b50005f90 comp rx=0 tx=0).stop 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 msgr2=0x7f2b60197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00e9f0 tx=0x7f2b5c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 shutdown_connections 2024-03-19T02:35:16.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b4c0a2970 0x7f2b4c0a4e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b60106730 0x7f2b6019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b601010e0 0x7f2b60198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.496+0000 7f2b685a9640 1 --2- 172.21.15.116:0/4243897619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b60100720 0x7f2b60197fc0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.497+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 >> 172.21.15.116:0/4243897619 conn(0x7f2b60077820 msgr2=0x7f2b60068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.497+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 shutdown_connections 2024-03-19T02:35:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.497+0000 7f2b685a9640 1 -- 172.21.15.116:0/4243897619 wait complete. 2024-03-19T02:35:16.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:35:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 -- 172.21.15.116:0/1599153467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc1066b0 msgr2=0x7fb4cc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 --2- 172.21.15.116:0/1599153467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc106a90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009930 tx=0x7fb4bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 -- 172.21.15.116:0/1599153467 shutdown_connections 2024-03-19T02:35:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 --2- 172.21.15.116:0/1599153467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc101060 0x7fb4cc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 --2- 172.21.15.116:0/1599153467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 --2- 172.21.15.116:0/1599153467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc106a90 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 -- 172.21.15.116:0/1599153467 >> 172.21.15.116:0/1599153467 conn(0x7fb4cc0778d0 msgr2=0x7fb4cc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 -- 172.21.15.116:0/1599153467 shutdown_connections 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.620+0000 7fb4d171b640 1 -- 172.21.15.116:0/1599153467 wait complete. 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 Processor -- start 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 -- start start 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc110970 con 0x7fb4cc101060 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1107f0 con 0x7fb4cc1006a0 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4d171b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc110af0 con 0x7fb4cc1066b0 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50106/0 (socket says 172.21.15.116:50106) 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56804/0 (socket says 172.21.15.116:56804) 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 -- 172.21.15.116:0/869781286 learned_addr learned my addr 172.21.15.116:0/869781286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 -- 172.21.15.116:0/869781286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc1066b0 msgr2=0x7fb4cc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 -- 172.21.15.116:0/869781286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 msgr2=0x7fb4cc198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 -- 172.21.15.116:0/869781286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4cc1a43b0 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.621+0000 7fb4c3fff640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e9f0 tx=0x7fb4b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.622+0000 7fb4c8ff9640 1 -- 172.21.15.116:0/869781286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b400cd80 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.622+0000 7fb4c8ff9640 1 -- 172.21.15.116:0/869781286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4b40108e0 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.622+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4cc1a4580 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.622+0000 7fb4c8ff9640 1 -- 172.21.15.116:0/869781286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4018920 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.622+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4cc1a4af0 con 0x7fb4cc101060 2024-03-19T02:35:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.624+0000 7fb4c8ff9640 1 -- 172.21.15.116:0/869781286 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb4b4010a90 con 0x7fb4cc101060 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.624+0000 7fb4d43e5740 1 Processor -- start 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 -- start start 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e70c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4c8ff9640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 0x7fb4b80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e717bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56166e66f4b0 con 0x56166e70cbf0 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56166e617d70 con 0x56166e7157d0 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56166e66b550 con 0x56166e70c2d0 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4c3fff640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 0x7fb4b80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e70c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50118/0 (socket says 172.21.15.116:50118) 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4caffd640 1 -- 172.21.15.116:0/4064469834 learned_addr learned my addr 172.21.15.116:0/4064469834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e717bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.625+0000 7fb4c8ff9640 1 -- 172.21.15.116:0/869781286 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fb4b40cf4c0 con 0x7fb4cc101060 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 -- 172.21.15.116:0/4064469834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 msgr2=0x56166e70c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e70c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 -- 172.21.15.116:0/4064469834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 msgr2=0x56166e717bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e717bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 -- 172.21.15.116:0/4064469834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56166e63a210 con 0x56166e70cbf0 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e70c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4c3fff640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 0x7fb4b80a4eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4b409b780 tx=0x7fb4b406e590 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4caffd640 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc036040 tx=0x7fb4bc009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8366d30799f21d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4c17fa640 1 -- 172.21.15.116:0/4064469834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc045020 con 0x56166e70cbf0 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4c17fa640 1 -- 172.21.15.116:0/4064469834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4bc02c7e0 con 0x56166e70cbf0 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.626+0000 7fb4c17fa640 1 -- 172.21.15.116:0/4064469834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc005390 con 0x56166e70cbf0 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- 172.21.15.116:0/4064469834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 msgr2=0x56166e715290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc036040 tx=0x7fb4bc009560 comp rx=0 tx=0).stop 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- 172.21.15.116:0/4064469834 shutdown_connections 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e717bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70cbf0 0x56166e715290 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/4064469834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e70c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- 172.21.15.116:0/4064469834 >> 172.21.15.116:0/4064469834 conn(0x56166e706a00 msgr2=0x56166e706de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- 172.21.15.116:0/4064469834 shutdown_connections 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- 172.21.15.116:0/4064469834 wait complete. 2024-03-19T02:35:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 -- start start 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.627+0000 7fb4d43e5740 1 Processor -- start 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e7127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e712d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56166e718560 con 0x56166e7157d0 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56166e62b310 con 0x56166e70cbf0 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56166e7186e0 con 0x56166e70c2d0 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 10 client.0 _ll_register_callbacks cb 0x56166e61f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50122/0 (socket says 172.21.15.116:50122) 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/3112008598 learned_addr learned my addr 172.21.15.116:0/3112008598 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e7127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e712d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 msgr2=0x56166e7127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e7127f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/3112008598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 msgr2=0x56166e712d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e712d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/3112008598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56166e3833f0 con 0x56166e7157d0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4011fd0 tx=0x7fb4c4009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4435232fe85ec6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4004b20 con 0x56166e7157d0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fb4c4004cc0 con 0x56166e7157d0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.628+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc02c7e0 con 0x56166e7157d0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb4c4003970 con 0x56166e7157d0 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 10 client.17790 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56166e44d7d0 con 0x56166e7157d0 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4bc02c7e0 con 0x56166e7157d0 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb4c4010430 con 0x56166e7157d0 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 10 client.17790 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 10 client.17790 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 20 client.17790 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e725bd0 0x56166e728010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 4 client.17790 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56166e6cd3e0 con 0x56166e725bd0 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.629+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e725bd0 0x56166e728010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.630+0000 7fb4b27fc640 10 client.17790 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.630+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e725bd0 0x56166e728010 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=42c9567355712158 server_cookie=86215948d6563d95 in_seq=0 out_seq=0 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4b27fc640 1 -- 172.21.15.116:0/3112008598 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56166e6cd3e0 con 0x56166e725bd0 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4b27fc640 10 client.17790 handle_command_reply: tid=0 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 client.17790 shutdown 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 msgr2=0x56166e713270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4011fd0 tx=0x7fb4c4009df0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 shutdown_connections 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e725bd0 0x56166e728010 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e713270 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e712d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3112008598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e7127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 >> 172.21.15.116:0/3112008598 conn(0x56166e706a00 msgr2=0x56166e708750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4b27fc640 0 client.17790 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 shutdown_connections 2024-03-19T02:35:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3112008598 wait complete. 2024-03-19T02:35:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.631+0000 7fb4d43e5740 20 client.17790 trim_cache size 0 max 16384 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.647+0000 7fb4d43e5740 1 Processor -- start 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.647+0000 7fb4d43e5740 1 -- start start 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e714f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e721300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1aec10 con 0x56166e7157d0 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b0750 con 0x56166e70cbf0 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1af2e0 con 0x56166e70c2d0 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e721300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e721300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50134/0 (socket says 172.21.15.116:50134) 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56832/0 (socket says 172.21.15.116:56832) 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/3933914529 learned_addr learned my addr 172.21.15.116:0/3933914529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e714f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 -- 172.21.15.116:0/3933914529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 msgr2=0x56166e714f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e714f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 -- 172.21.15.116:0/3933914529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 msgr2=0x56166e721300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e721300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 -- 172.21.15.116:0/3933914529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56166e706830 con 0x56166e70cbf0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4caffd640 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005ab0 tx=0x7fb4bc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8248cbe4e9632ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.648+0000 7fb4c0ff9640 1 -- 172.21.15.116:0/3933914529 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc050070 con 0x56166e70cbf0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4c0ff9640 1 -- 172.21.15.116:0/3933914529 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4bc02cc90 con 0x56166e70cbf0 2024-03-19T02:35:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4c0ff9640 1 -- 172.21.15.116:0/3933914529 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc045040 con 0x56166e70cbf0 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3933914529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 msgr2=0x56166e711950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc005ab0 tx=0x7fb4bc000bc0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3933914529 shutdown_connections 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e7157d0 0x56166e721300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e70cbf0 0x56166e711950 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/3933914529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70c2d0 0x56166e714f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3933914529 >> 172.21.15.116:0/3933914529 conn(0x56166e706a00 msgr2=0x56166e707e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3933914529 shutdown_connections 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.649+0000 7fb4d43e5740 1 -- 172.21.15.116:0/3933914529 wait complete. 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 -- start start 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 Processor -- start 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 0x56166e7198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e719e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1b0080 con 0x56166e70c2d0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1af9b0 con 0x56166e7157d0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1acc10 con 0x56166e70cbf0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4d43e5740 10 client.0 _ll_register_callbacks cb 0x7fb4c4011ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50148/0 (socket says 172.21.15.116:50148) 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4cb7fe640 1 -- 172.21.15.116:0/2470349443 learned_addr learned my addr 172.21.15.116:0/2470349443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 0x56166e7198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e719e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 -- 172.21.15.116:0/2470349443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 msgr2=0x56166e719e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e719e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 -- 172.21.15.116:0/2470349443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 msgr2=0x56166e719380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4caffd640 1 -- 172.21.15.116:0/2470349443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56166e78b020 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.650+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4caffd640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 0x56166e7198c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02c170 tx=0x7fb4bc0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a79c3ef56788b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc050040 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fb4bc004240 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc02cc90 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb4bc062050 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 10 client.27550 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56166e78c300 con 0x56166e70cbf0 2024-03-19T02:35:16.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb4bc02cc90 con 0x56166e70cbf0 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb4bc0047e0 con 0x56166e70cbf0 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 10 client.27550 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 10 client.27550 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 20 client.27550 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.651+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e78c680 0x56166e78ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 4 client.27550 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56166e67c9f0 con 0x56166e78c680 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e78c680 0x56166e78ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4b27fc640 10 client.27550 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4cb7fe640 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e78c680 0x56166e78ea70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6d8499d8030fb905 server_cookie=14f13d33000d20e0 in_seq=0 out_seq=0 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4b27fc640 1 -- 172.21.15.116:0/2470349443 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56166e67c9f0 con 0x56166e78c680 2024-03-19T02:35:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4b27fc640 10 client.27550 handle_command_reply: tid=0 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 client.27550 shutdown 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 msgr2=0x56166e7198c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 0x56166e7198c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc02c170 tx=0x7fb4bc0058d0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 shutdown_connections 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56166e78c680 0x56166e78ea70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56166e7157d0 0x56166e719e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4b27fc640 0 client.27550 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56166e70cbf0 0x56166e7198c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 --2- 172.21.15.116:0/2470349443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56166e70c2d0 0x56166e719380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 >> 172.21.15.116:0/2470349443 conn(0x56166e706a00 msgr2=0x56166e707d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 shutdown_connections 2024-03-19T02:35:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.652+0000 7fb4d43e5740 1 -- 172.21.15.116:0/2470349443 wait complete. 2024-03-19T02:35:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d43e5740 20 client.27550 trim_cache size 0 max 16384 2024-03-19T02:35:16.662 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 msgr2=0x7fb4b80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 0x7fb4b80a4eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4b409b780 tx=0x7fb4b406e590 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 msgr2=0x7fb4cc199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e9f0 tx=0x7fb4b400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 shutdown_connections 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4b80a29f0 0x7fb4b80a4eb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4cc1066b0 0x7fb4cc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4cc101060 0x7fb4cc199330 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 --2- 172.21.15.116:0/869781286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4cc1006a0 0x7fb4cc198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 >> 172.21.15.116:0/869781286 conn(0x7fb4cc0778d0 msgr2=0x7fb4cc102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 shutdown_connections 2024-03-19T02:35:16.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.653+0000 7fb4d171b640 1 -- 172.21.15.116:0/869781286 wait complete. 2024-03-19T02:35:16.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:35:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 -- 172.21.15.116:0/3528287422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c1006a0 msgr2=0x7faf7c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 --2- 172.21.15.116:0/3528287422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c1006a0 0x7faf7c100b20 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7faf70009930 tx=0x7faf7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 -- 172.21.15.116:0/3528287422 shutdown_connections 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 --2- 172.21.15.116:0/3528287422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c101060 0x7faf7c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 --2- 172.21.15.116:0/3528287422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c1006a0 0x7faf7c100b20 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 --2- 172.21.15.116:0/3528287422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 -- 172.21.15.116:0/3528287422 >> 172.21.15.116:0/3528287422 conn(0x7faf7c0778d0 msgr2=0x7faf7c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 -- 172.21.15.116:0/3528287422 shutdown_connections 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.779+0000 7faf81085640 1 -- 172.21.15.116:0/3528287422 wait complete. 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 Processor -- start 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 -- start start 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c110970 con 0x7faf7c101060 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1107f0 con 0x7faf7c1006a0 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf81085640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c110af0 con 0x7faf7c1066b0 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50168/0 (socket says 172.21.15.116:50168) 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56868/0 (socket says 172.21.15.116:56868) 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7b59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.780+0000 7faf7a59c640 1 -- 172.21.15.116:0/589638631 learned_addr learned my addr 172.21.15.116:0/589638631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 -- 172.21.15.116:0/589638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 msgr2=0x7faf7c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 -- 172.21.15.116:0/589638631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 msgr2=0x7faf7c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 -- 172.21.15.116:0/589638631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c19cfd0 con 0x7faf7c101060 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7b59e640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf7a59c640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faf70009230 tx=0x7faf7002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf63fff640 1 -- 172.21.15.116:0/589638631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70043020 con 0x7faf7c101060 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7c19d260 con 0x7faf7c101060 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf63fff640 1 -- 172.21.15.116:0/589638631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf7002ca70 con 0x7faf7c101060 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf63fff640 1 -- 172.21.15.116:0/589638631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70003a50 con 0x7faf7c101060 2024-03-19T02:35:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.781+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7c1a4b50 con 0x7faf7c101060 2024-03-19T02:35:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.783+0000 7faf63fff640 1 -- 172.21.15.116:0/589638631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faf70046020 con 0x7faf7c101060 2024-03-19T02:35:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf63fff640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 0x7faf640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 Processor -- start 2024-03-19T02:35:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 -- start start 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 0x7faf640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf63fff640 1 -- 172.21.15.116:0/589638631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faf700f1110 con 0x7faf7c101060 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041792d0 0x560c041796b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04184bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c040dc4b0 con 0x560c04179bf0 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c04084d70 con 0x560c041827d0 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf83c7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c040d8550 con 0x560c041792d0 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50184/0 (socket says 172.21.15.116:50184) 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7a59c640 1 -- 172.21.15.116:0/2701809198 learned_addr learned my addr 172.21.15.116:0/2701809198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041792d0 0x560c041796b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.784+0000 7faf7b59e640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04184bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 0x7faf640a4e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009f90 tx=0x7faf6c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 -- 172.21.15.116:0/2701809198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041792d0 msgr2=0x560c041796b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041792d0 0x560c041796b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 -- 172.21.15.116:0/2701809198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 msgr2=0x560c04184bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04184bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 -- 172.21.15.116:0/2701809198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c040a7210 con 0x560c04179bf0 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7b59e640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04184bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf7a59c640 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7faf700b9900 tx=0x7faf7002c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6ebad4e19ea7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf617fa640 1 -- 172.21.15.116:0/2701809198 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700bd4a0 con 0x560c04179bf0 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf617fa640 1 -- 172.21.15.116:0/2701809198 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf700c15f0 con 0x560c04179bf0 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.785+0000 7faf617fa640 1 -- 172.21.15.116:0/2701809198 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700b9ca0 con 0x560c04179bf0 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- 172.21.15.116:0/2701809198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 msgr2=0x560c04182290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7faf700b9900 tx=0x7faf7002c740 comp rx=0 tx=0).stop 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- 172.21.15.116:0/2701809198 shutdown_connections 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04184bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04182290 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- 172.21.15.116:0/2701809198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041792d0 0x560c041796b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- 172.21.15.116:0/2701809198 >> 172.21.15.116:0/2701809198 conn(0x560c04173a00 msgr2=0x560c04173de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- 172.21.15.116:0/2701809198 shutdown_connections 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- 172.21.15.116:0/2701809198 wait complete. 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 -- start start 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 Processor -- start 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0417f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.786+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c0417fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c04189790 con 0x560c041827d0 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c04098310 con 0x560c041792d0 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c04189910 con 0x560c04179bf0 2024-03-19T02:35:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 10 client.0 _ll_register_callbacks cb 0x560c0408c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c0417fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50194/0 (socket says 172.21.15.116:50194) 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0417f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7b59e640 1 -- 172.21.15.116:0/85988842 learned_addr learned my addr 172.21.15.116:0/85988842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 -- 172.21.15.116:0/85988842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 msgr2=0x560c0417f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0417f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 -- 172.21.15.116:0/85988842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 msgr2=0x560c041802a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 -- 172.21.15.116:0/85988842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c03df03f0 con 0x560c04179bf0 2024-03-19T02:35:16.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7b59e640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf7a59c640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c0417fd60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faf70009900 tx=0x7faf70005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e7a75e1da128a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700b9c80 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf700c15f0 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faf700c1170 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf700c13b0 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 10 client.27562 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.787+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c03eba7d0 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf700c15f0 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faf700eebd0 con 0x560c04179bf0 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 10 client.27562 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 10 client.27562 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:35:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 20 client.27562 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c04196b40 0x560c04198f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 4 client.27562 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c0413a3e0 con 0x560c04196b40 2024-03-19T02:35:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.788+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c04196b40 0x560c04198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 239539 3.20 MB/sec execute 279 sec latency 27.159 ms 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c04196b40 0x560c04198f30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=366a949844d9b8b3 server_cookie=e66f0833ed4eb989 in_seq=0 out_seq=0 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf5eff5640 10 client.27562 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf5eff5640 1 -- 172.21.15.116:0/85988842 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c0413a3e0 con 0x560c04196b40 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf5eff5640 10 client.27562 handle_command_reply: tid=0 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 client.27562 shutdown 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 msgr2=0x560c0417fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c0417fd60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faf70009900 tx=0x7faf70005b50 comp rx=0 tx=0).stop 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 shutdown_connections 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c04196b40 0x560c04198f30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041827d0 0x560c041802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c0417fd60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf5eff5640 0 client.27562 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 --2- 172.21.15.116:0/85988842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0417f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 >> 172.21.15.116:0/85988842 conn(0x560c04173a00 msgr2=0x560c04175750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 shutdown_connections 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.789+0000 7faf83c7a740 1 -- 172.21.15.116:0/85988842 wait complete. 2024-03-19T02:35:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.790+0000 7faf83c7a740 20 client.27562 trim_cache size 0 max 16384 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 Processor -- start 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 -- start start 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 0x560c04192480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c041800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04172a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1ac910 con 0x560c041792d0 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1ae450 con 0x560c041827d0 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf83c7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1acfe0 con 0x560c04179bf0 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 0x560c04192480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c041800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c041800c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51170/0 (socket says 172.21.15.116:51170) 2024-03-19T02:35:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7a59c640 1 -- 172.21.15.116:0/184926898 learned_addr learned my addr 172.21.15.116:0/184926898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7b59e640 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04172a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 -- 172.21.15.116:0/184926898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 msgr2=0x560c041800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c041800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 -- 172.21.15.116:0/184926898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 msgr2=0x560c04172a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04172a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 -- 172.21.15.116:0/184926898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c04149880 con 0x560c041792d0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.805+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 0x560c04192480 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7faf700bb750 tx=0x7faf6c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20395090a4cbdf9f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf60ff9640 1 -- 172.21.15.116:0/184926898 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c033070 con 0x560c041792d0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf60ff9640 1 -- 172.21.15.116:0/184926898 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf6c003c40 con 0x560c041792d0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf60ff9640 1 -- 172.21.15.116:0/184926898 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c027670 con 0x560c041792d0 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 -- 172.21.15.116:0/184926898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 msgr2=0x560c04192480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 0x560c04192480 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7faf700bb750 tx=0x7faf6c01c000 comp rx=0 tx=0).stop 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 -- 172.21.15.116:0/184926898 shutdown_connections 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041827d0 0x560c04172a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c04179bf0 0x560c041800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 --2- 172.21.15.116:0/184926898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c041792d0 0x560c04192480 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 -- 172.21.15.116:0/184926898 >> 172.21.15.116:0/184926898 conn(0x560c04173a00 msgr2=0x560c04172310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 -- 172.21.15.116:0/184926898 shutdown_connections 2024-03-19T02:35:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.806+0000 7faf83c7a740 1 -- 172.21.15.116:0/184926898 wait complete. 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 -- start start 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 Processor -- start 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0418ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041827d0 0x560c0418a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1add80 con 0x560c04179bf0 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1ad6b0 con 0x560c041792d0 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c1aa910 con 0x560c041827d0 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf83c7a740 10 client.0 _ll_register_callbacks cb 0x7faf74009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50214/0 (socket says 172.21.15.116:50214) 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 -- 172.21.15.116:0/1744156410 learned_addr learned my addr 172.21.15.116:0/1744156410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0418ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7b59e640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041827d0 0x560c0418a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 -- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041827d0 msgr2=0x560c0418a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041827d0 0x560c0418a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 -- 172.21.15.116:0/1744156410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 msgr2=0x560c0418ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0418ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7a59c640 1 -- 172.21.15.116:0/1744156410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c0418a890 con 0x560c04179bf0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.807+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0418ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf7a59c640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faf70009900 tx=0x7faf70009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9114a6ad8170ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf700a3020 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faf700b9330 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf6c003c40 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf700bce00 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 10 client.17814 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c0418a890 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf6c003c40 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faf700bd700 con 0x560c04179bf0 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 10 client.17814 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 10 client.17814 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 20 client.17814 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c041fcde0 0x560c041ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 4 client.17814 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.808+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c040e99f0 con 0x560c041fcde0 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c041fcde0 0x560c041ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf7ad9d640 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c041fcde0 0x560c041ff200 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b454152a2725624d server_cookie=ccdd0accd46f68e3 in_seq=0 out_seq=0 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf5eff5640 10 client.17814 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf5eff5640 1 -- 172.21.15.116:0/1744156410 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c040e99f0 con 0x560c041fcde0 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf5eff5640 10 client.17814 handle_command_reply: tid=0 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 client.17814 shutdown 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 msgr2=0x560c04189e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faf70009900 tx=0x7faf70009560 comp rx=0 tx=0).stop 2024-03-19T02:35:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 shutdown_connections 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c041827d0 0x560c0418a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x560c041fcde0 0x560c041ff200 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c04179bf0 0x560c04189e10 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 --2- 172.21.15.116:0/1744156410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c041792d0 0x560c0418ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf5eff5640 0 client.17814 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 >> 172.21.15.116:0/1744156410 conn(0x560c04173a00 msgr2=0x560c04174c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 shutdown_connections 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 1 -- 172.21.15.116:0/1744156410 wait complete. 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.809+0000 7faf83c7a740 20 client.17814 trim_cache size 0 max 16384 2024-03-19T02:35:16.822 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 msgr2=0x7faf640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 0x7faf640a4e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009f90 tx=0x7faf6c005f90 comp rx=0 tx=0).stop 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 msgr2=0x7faf7c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7faf70009230 tx=0x7faf7002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 shutdown_connections 2024-03-19T02:35:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf640a2970 0x7faf640a4e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf7c1066b0 0x7faf7c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf7c101060 0x7faf7c198520 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 --2- 172.21.15.116:0/589638631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf7c1006a0 0x7faf7c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 >> 172.21.15.116:0/589638631 conn(0x7faf7c0778d0 msgr2=0x7faf7c0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 shutdown_connections 2024-03-19T02:35:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.810+0000 7faf81085640 1 -- 172.21.15.116:0/589638631 wait complete. 2024-03-19T02:35:16.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 -- 172.21.15.116:0/3276851444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc103360 msgr2=0x7f1dfc10b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3276851444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc103360 0x7f1dfc10b840 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009900 tx=0x7f1df002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 -- 172.21.15.116:0/3276851444 shutdown_connections 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3276851444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3276851444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc103360 0x7f1dfc10b840 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3276851444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc102a40 0x7f1dfc102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 -- 172.21.15.116:0/3276851444 >> 172.21.15.116:0/3276851444 conn(0x7f1dfc0778d0 msgr2=0x7f1dfc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 -- 172.21.15.116:0/3276851444 shutdown_connections 2024-03-19T02:35:16.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.941+0000 7f1e024ea640 1 -- 172.21.15.116:0/3276851444 wait complete. 2024-03-19T02:35:16.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 Processor -- start 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 -- start start 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103360 0x7f1dfc100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc111a30 con 0x7f1dfc102a40 2024-03-19T02:35:16.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1118b0 con 0x7f1dfc10bd80 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e024ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc111bb0 con 0x7f1dfc103360 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e00a87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50226/0 (socket says 172.21.15.116:50226) 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1e00a87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56936/0 (socket says 172.21.15.116:56936) 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1dfbfff640 1 -- 172.21.15.116:0/3902453109 learned_addr learned my addr 172.21.15.116:0/3902453109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.942+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103360 0x7f1dfc100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 -- 172.21.15.116:0/3902453109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103360 msgr2=0x7f1dfc100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103360 0x7f1dfc100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 -- 172.21.15.116:0/3902453109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 msgr2=0x7f1dfc101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 -- 172.21.15.116:0/3902453109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc1a64d0 con 0x7f1dfc102a40 2024-03-19T02:35:16.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1e00a87640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1df97fa640 1 -- 172.21.15.116:0/3902453109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00cd80 con 0x7f1dfc102a40 2024-03-19T02:35:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc1a67c0 con 0x7f1dfc102a40 2024-03-19T02:35:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1df97fa640 1 -- 172.21.15.116:0/3902453109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dec0054d0 con 0x7f1dfc102a40 2024-03-19T02:35:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1df97fa640 1 -- 172.21.15.116:0/3902453109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec010490 con 0x7f1dfc102a40 2024-03-19T02:35:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.943+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1a6ce0 con 0x7f1dfc102a40 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.945+0000 7f1df97fa640 1 -- 172.21.15.116:0/3902453109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1dec005020 con 0x7f1dfc102a40 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 Processor -- start 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 -- start start 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1df97fa640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 0x7f1de40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e39327e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3932877d0 0x55e393289bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e3931e14b0 con 0x55e3932877d0 2024-03-19T02:35:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e00a87640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 0x7f1de40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e393189d70 con 0x55e39327e2d0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1e051b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e3931dd550 con 0x55e39327ebf0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.946+0000 7f1df97fa640 1 -- 172.21.15.116:0/3902453109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f1dec0cf5d0 con 0x7f1dfc102a40 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51212/0 (socket says 172.21.15.116:51212) 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/405546311 learned_addr learned my addr 172.21.15.116:0/405546311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e39327e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1e00a87640 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3932877d0 0x55e393289bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/405546311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 msgr2=0x55e39327e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e39327e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/405546311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3932877d0 msgr2=0x55e393289bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3932877d0 0x55e393289bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/405546311 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3931ac210 con 0x55e39327ebf0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1e00a87640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 0x7f1de40a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009fd0 tx=0x7f1df4011f10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009900 tx=0x7f1df00095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e181cf8c9cbe9d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1de27fc640 1 -- 172.21.15.116:0/405546311 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0004040 con 0x55e39327ebf0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1de27fc640 1 -- 172.21.15.116:0/405546311 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df00054f0 con 0x55e39327ebf0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.947+0000 7f1de27fc640 1 -- 172.21.15.116:0/405546311 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df002ce20 con 0x55e39327ebf0 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- 172.21.15.116:0/405546311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 msgr2=0x55e393287290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009900 tx=0x7f1df00095f0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- 172.21.15.116:0/405546311 shutdown_connections 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3932877d0 0x55e393289bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327ebf0 0x55e393287290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 --2- 172.21.15.116:0/405546311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e39327e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- 172.21.15.116:0/405546311 >> 172.21.15.116:0/405546311 conn(0x55e393278a00 msgr2=0x55e393278de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- 172.21.15.116:0/405546311 shutdown_connections 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- 172.21.15.116:0/405546311 wait complete. 2024-03-19T02:35:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.948+0000 7f1e051b3740 1 -- start start 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 Processor -- start 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e3932847e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e393285260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e39328a560 con 0x55e39327ebf0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e39319d310 con 0x55e39327e2d0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e39328a6e0 con 0x55e3932877d0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e051b3740 10 client.0 _ll_register_callbacks cb 0x55e3931912d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50258/0 (socket says 172.21.15.116:50258) 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e3932847e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/927946854 learned_addr learned my addr 172.21.15.116:0/927946854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1e00a87640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e393285260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/927946854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 msgr2=0x55e393285260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e393285260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 msgr2=0x55e3932847e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e3932847e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/927946854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e392ef53f0 con 0x55e39327ebf0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e3932847e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1df0004650 tx=0x7f1df00346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c8d02a06925018 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.949+0000 7f1ddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0034cf0 con 0x55e39327ebf0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f1df003f430 con 0x55e39327ebf0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df00054f0 con 0x55e39327ebf0 2024-03-19T02:35:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1df005a050 con 0x55e39327ebf0 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 10 client.17826 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e392fbf7d0 con 0x55e39327ebf0 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df00054f0 con 0x55e39327ebf0 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1df003fca0 con 0x55e39327ebf0 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 10 client.17826 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 10 client.17826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 20 client.17826 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e393299d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 4 client.17826 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e39323f3e0 con 0x55e393297960 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.950+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e393299d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.951+0000 7f1ddb7fe640 10 client.17826 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.951+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e393299d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f165b0add0d1e60c server_cookie=82c83a0b4396b9df in_seq=0 out_seq=0 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/927946854 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e39323f3e0 con 0x55e393297960 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1ddb7fe640 10 client.17826 handle_command_reply: tid=0 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 client.17826 shutdown 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 msgr2=0x55e393284d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1df0004650 tx=0x7f1df00346f0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 shutdown_connections 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e393299d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e393285260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1ddb7fe640 0 client.17826 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393284d20 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 --2- 172.21.15.116:0/927946854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e3932847e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 >> 172.21.15.116:0/927946854 conn(0x55e393278a00 msgr2=0x55e39327a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 shutdown_connections 2024-03-19T02:35:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 1 -- 172.21.15.116:0/927946854 wait complete. 2024-03-19T02:35:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.952+0000 7f1e051b3740 20 client.17826 trim_cache size 0 max 16384 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 Processor -- start 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 -- start start 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e393286f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e39329f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b0d60 con 0x55e39327ebf0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b28a0 con 0x55e39327e2d0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e051b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b1430 con 0x55e3932877d0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50274/0 (socket says 172.21.15.116:50274) 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e393286f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/183237945 learned_addr learned my addr 172.21.15.116:0/183237945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1e00a87640 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e39329f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/183237945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 msgr2=0x55e39329f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e39329f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/183237945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 msgr2=0x55e393286f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e393286f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/183237945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e39328a690 con 0x55e39327ebf0 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e393286f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f1df002c210 tx=0x7f1df002ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28efc49e86ece05 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.966+0000 7f1de1ffb640 1 -- 172.21.15.116:0/183237945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005d50 con 0x55e39327ebf0 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1de1ffb640 1 -- 172.21.15.116:0/183237945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df0005070 con 0x55e39327ebf0 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1de1ffb640 1 -- 172.21.15.116:0/183237945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df002c6b0 con 0x55e39327ebf0 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 -- 172.21.15.116:0/183237945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 msgr2=0x55e393293300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f1df002c210 tx=0x7f1df002ceb0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 -- 172.21.15.116:0/183237945 shutdown_connections 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e3932877d0 0x55e39329f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393293300 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 --2- 172.21.15.116:0/183237945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e39327e2d0 0x55e393286f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 -- 172.21.15.116:0/183237945 >> 172.21.15.116:0/183237945 conn(0x55e393278a00 msgr2=0x55e393277310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 -- 172.21.15.116:0/183237945 shutdown_connections 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.967+0000 7f1e051b3740 1 -- 172.21.15.116:0/183237945 wait complete. 2024-03-19T02:35:16.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 -- start start 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 Processor -- start 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 0x55e39328fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e3932877d0 0x55e393281810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b21d0 con 0x55e39327ebf0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1b1b00 con 0x55e3932877d0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1aed60 con 0x55e39327e2d0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e051b3740 10 client.0 _ll_register_callbacks cb 0x7f1dec009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 0x55e39328fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50286/0 (socket says 172.21.15.116:50286) 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/1438025666 learned_addr learned my addr 172.21.15.116:0/1438025666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1e00a87640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e3932877d0 0x55e393281810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/1438025666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 msgr2=0x55e39328fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 0x55e39328fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e3932877d0 msgr2=0x55e393281810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e3932877d0 0x55e393281810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfb7fe640 1 -- 172.21.15.116:0/1438025666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3932fc9e0 con 0x55e39327ebf0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.968+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 0x55e39328fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1dfb7fe640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005d00 tx=0x7f1df0034850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cab90eb14c2d2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005d50 con 0x55e39327ebf0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f1df0005220 con 0x55e39327ebf0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df0005070 con 0x55e39327ebf0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1df0005420 con 0x55e39327ebf0 2024-03-19T02:35:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 10 client.17832 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e3932fdb10 con 0x55e39327ebf0 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df0005070 con 0x55e39327ebf0 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1df0047690 con 0x55e39327ebf0 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 10 client.17832 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 10 client.17832 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 20 client.17832 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e3932fff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 4 client.17832 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3931ee9f0 con 0x55e393297960 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.969+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e3932fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:16.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1dfbfff640 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e3932fff00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=446ba6615d4307d1 server_cookie=c7a183d45b28f48a in_seq=0 out_seq=0 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1ddb7fe640 10 client.17832 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1ddb7fe640 1 -- 172.21.15.116:0/1438025666 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3931ee9f0 con 0x55e393297960 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1ddb7fe640 10 client.17832 handle_command_reply: tid=0 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 client.17832 shutdown 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 msgr2=0x55e393281190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005d00 tx=0x7f1df0034850 comp rx=0 tx=0).stop 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 shutdown_connections 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e393297960 0x55e3932fff00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e3932877d0 0x55e393281810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e39327ebf0 0x55e393281190 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1ddb7fe640 0 client.17832 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 --2- 172.21.15.116:0/1438025666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e39327e2d0 0x55e39328fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 >> 172.21.15.116:0/1438025666 conn(0x55e393278a00 msgr2=0x55e393285030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 shutdown_connections 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 1 -- 172.21.15.116:0/1438025666 wait complete. 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.970+0000 7f1e051b3740 20 client.17832 trim_cache size 0 max 16384 2024-03-19T02:35:16.979 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 msgr2=0x7f1de40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 0x7f1de40a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009fd0 tx=0x7f1df4011f10 comp rx=0 tx=0).stop 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 msgr2=0x7f1dfc100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00edf0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 shutdown_connections 2024-03-19T02:35:16.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de40a2970 0x7f1de40a4e30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10bd80 0x7f1dfc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dfc103360 0x7f1dfc100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 --2- 172.21.15.116:0/3902453109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dfc102a40 0x7f1dfc100690 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 >> 172.21.15.116:0/3902453109 conn(0x7f1dfc0778d0 msgr2=0x7f1dfc0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 shutdown_connections 2024-03-19T02:35:16.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:16.971+0000 7f1e024ea640 1 -- 172.21.15.116:0/3902453109 wait complete. 2024-03-19T02:35:16.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.096+0000 7faec1619640 1 -- 172.21.15.116:0/3563546791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 msgr2=0x7faebc10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.096+0000 7faec1619640 1 --2- 172.21.15.116:0/3563546791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc10e670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009cd0 tx=0x7faeb402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 -- 172.21.15.116:0/3563546791 shutdown_connections 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 --2- 172.21.15.116:0/3563546791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc10e670 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 --2- 172.21.15.116:0/3563546791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1017a0 0x7faebc101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 --2- 172.21.15.116:0/3563546791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1077b0 0x7faebc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 -- 172.21.15.116:0/3563546791 >> 172.21.15.116:0/3563546791 conn(0x7faebc0fd2b0 msgr2=0x7faebc0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 -- 172.21.15.116:0/3563546791 shutdown_connections 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 -- 172.21.15.116:0/3563546791 wait complete. 2024-03-19T02:35:17.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 Processor -- start 2024-03-19T02:35:17.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.097+0000 7faec1619640 1 -- start start 2024-03-19T02:35:17.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1077b0 0x7faebc0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faebc111a70 con 0x7faebc1017a0 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faebc1118f0 con 0x7faebc1077b0 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faebc111bf0 con 0x7faebc102160 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50298/0 (socket says 172.21.15.116:50298) 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 -- 172.21.15.116:0/3943965182 learned_addr learned my addr 172.21.15.116:0/3943965182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1077b0 0x7faebc0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 -- 172.21.15.116:0/3943965182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 msgr2=0x7faebc074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 -- 172.21.15.116:0/3943965182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1077b0 msgr2=0x7faebc0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1077b0 0x7faebc0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 -- 172.21.15.116:0/3943965182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc1a6450 con 0x7faebc1017a0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faebaffd640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e9f0 tx=0x7faeac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faea3fff640 1 -- 172.21.15.116:0/3943965182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac00cd80 con 0x7faebc1017a0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.098+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc1a6740 con 0x7faebc1017a0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.099+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc1a6ce0 con 0x7faebc1017a0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.099+0000 7faea3fff640 1 -- 172.21.15.116:0/3943965182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faeac0108e0 con 0x7faebc1017a0 2024-03-19T02:35:17.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.099+0000 7faea3fff640 1 -- 172.21.15.116:0/3943965182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac018920 con 0x7faebc1017a0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 Processor -- start 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faea3fff640 1 -- 172.21.15.116:0/3943965182 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faeac010430 con 0x7faebc1017a0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 -- start start 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b2c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b35290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5648a0a8f4b0 con 0x5648a0b2c2d0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5648a0a37d70 con 0x5648a0b2cbf0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faec4206740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5648a0a8b550 con 0x5648a0b357d0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.101+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b2c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51252/0 (socket says 172.21.15.116:51252) 2024-03-19T02:35:17.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 -- 172.21.15.116:0/3037715370 learned_addr learned my addr 172.21.15.116:0/3037715370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faebaffd640 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b35290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faea3fff640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 0x7faea40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faea3fff640 1 -- 172.21.15.116:0/3943965182 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faeac0cf3f0 con 0x7faebc1017a0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 -- 172.21.15.116:0/3037715370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 msgr2=0x5648a0b35290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b35290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 -- 172.21.15.116:0/3037715370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 msgr2=0x5648a0b2c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b2c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faeba7fc640 1 -- 172.21.15.116:0/3037715370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a0a5a210 con 0x5648a0b357d0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.102+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b2c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.103+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 0x7faea40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.103+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faeb00114f0 tx=0x7faeb00119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bbf34e9fd59b9d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.103+0000 7faea17fa640 1 -- 172.21.15.116:0/3037715370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb0004020 con 0x5648a0b357d0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.103+0000 7faea17fa640 1 -- 172.21.15.116:0/3037715370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faeb00041c0 con 0x5648a0b357d0 2024-03-19T02:35:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.103+0000 7faea17fa640 1 -- 172.21.15.116:0/3037715370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb000f880 con 0x5648a0b357d0 2024-03-19T02:35:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- 172.21.15.116:0/3037715370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 msgr2=0x5648a0b37bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 0x7faea40a4e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faeb4005cc0 tx=0x7faeb402c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faeb00114f0 tx=0x7faeb00119b0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- 172.21.15.116:0/3037715370 shutdown_connections 2024-03-19T02:35:17.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b37bc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b35290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- 172.21.15.116:0/3037715370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b2c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- 172.21.15.116:0/3037715370 >> 172.21.15.116:0/3037715370 conn(0x5648a0b26a00 msgr2=0x5648a0b26de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- 172.21.15.116:0/3037715370 shutdown_connections 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- 172.21.15.116:0/3037715370 wait complete. 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- start start 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 Processor -- start 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b32c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5648a0b385c0 con 0x5648a0b357d0 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5648a0a4b310 con 0x5648a0b2cbf0 2024-03-19T02:35:17.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5648a0b38740 con 0x5648a0b2c2d0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.104+0000 7faec4206740 10 client.0 _ll_register_callbacks cb 0x5648a0a3f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faeba7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b32c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51268/0 (socket says 172.21.15.116:51268) 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50320/0 (socket says 172.21.15.116:50320) 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebaffd640 1 -- 172.21.15.116:0/3410883749 learned_addr learned my addr 172.21.15.116:0/3410883749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 -- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 msgr2=0x5648a0b32730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 -- 172.21.15.116:0/3410883749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 msgr2=0x5648a0b32c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b32c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 -- 172.21.15.116:0/3410883749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a07a33f0 con 0x5648a0b357d0 2024-03-19T02:35:17.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebaffd640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7faeb402c630 tx=0x7faeb4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b598860f0ab334cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7fae9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4052060 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faeb4004040 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb00041c0 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faeb4059050 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faec4206740 10 client.17844 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.105+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5648a086d7d0 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faeb00041c0 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faeb404d830 con 0x5648a0b357d0 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 10 client.17844 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 10 client.17844 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:35:17.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 20 client.17844 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0b45b20 0x5648a0b47f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 4 client.17844 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648a0aed3e0 con 0x5648a0b45b20 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faebaffd640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0b45b20 0x5648a0b47f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7faebaffd640 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0b45b20 0x5648a0b47f30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4578daf16cfa2c2b server_cookie=d4c53b126e40f596 in_seq=0 out_seq=0 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.106+0000 7fae9eff5640 10 client.17844 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.107+0000 7fae9eff5640 1 -- 172.21.15.116:0/3410883749 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648a0aed3e0 con 0x5648a0b45b20 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.107+0000 7fae9eff5640 10 client.17844 handle_command_reply: tid=0 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.107+0000 7faec4206740 1 client.17844 shutdown 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 msgr2=0x5648a0b331b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7faeb402c630 tx=0x7faeb4005c50 comp rx=0 tx=0).stop 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 shutdown_connections 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0b45b20 0x5648a0b47f30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b357d0 0x5648a0b331b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7fae9eff5640 0 client.17844 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b32c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 --2- 172.21.15.116:0/3410883749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2c2d0 0x5648a0b32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 >> 172.21.15.116:0/3410883749 conn(0x5648a0b26a00 msgr2=0x5648a0b28750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 shutdown_connections 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 1 -- 172.21.15.116:0/3410883749 wait complete. 2024-03-19T02:35:17.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.108+0000 7faec4206740 20 client.17844 trim_cache size 0 max 16384 2024-03-19T02:35:17.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 Processor -- start 2024-03-19T02:35:17.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 -- start start 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 0x5648a0b4d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b357d0 0x5648a0b31000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faea800ff10 con 0x5648a0b2c2d0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faea8011a50 con 0x5648a0b357d0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faec4206740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faea80105e0 con 0x5648a0b2cbf0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faeba7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 0x5648a0b4d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50334/0 (socket says 172.21.15.116:50334) 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faeba7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 0x5648a0b4d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51282/0 (socket says 172.21.15.116:51282) 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b357d0 0x5648a0b31000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 -- 172.21.15.116:0/4007487700 learned_addr learned my addr 172.21.15.116:0/4007487700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 -- 172.21.15.116:0/4007487700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 msgr2=0x5648a0b4d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 0x5648a0b4d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 -- 172.21.15.116:0/4007487700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b357d0 msgr2=0x5648a0b31000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b357d0 0x5648a0b31000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.124+0000 7faebaffd640 1 -- 172.21.15.116:0/4007487700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a0b26830 con 0x5648a0b2c2d0 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faebaffd640 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7faeac09b6b0 tx=0x7faeac0d1900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c99183bdbdb3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faea0ff9640 1 -- 172.21.15.116:0/4007487700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac0bb040 con 0x5648a0b2c2d0 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faea0ff9640 1 -- 172.21.15.116:0/4007487700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faeac0a3e10 con 0x5648a0b2c2d0 2024-03-19T02:35:17.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faea0ff9640 1 -- 172.21.15.116:0/4007487700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac097d90 con 0x5648a0b2c2d0 2024-03-19T02:35:17.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 -- 172.21.15.116:0/4007487700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 msgr2=0x5648a0b413c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7faeac09b6b0 tx=0x7faeac0d1900 comp rx=0 tx=0).stop 2024-03-19T02:35:17.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 -- 172.21.15.116:0/4007487700 shutdown_connections 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b357d0 0x5648a0b31000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b2cbf0 0x5648a0b4d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.125+0000 7faec4206740 1 --2- 172.21.15.116:0/4007487700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b413c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- 172.21.15.116:0/4007487700 >> 172.21.15.116:0/4007487700 conn(0x5648a0b26a00 msgr2=0x5648a0b26de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- 172.21.15.116:0/4007487700 shutdown_connections 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- 172.21.15.116:0/4007487700 wait complete. 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- start start 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 Processor -- start 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b3dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b38a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faea8011380 con 0x5648a0b2c2d0 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faea8010cb0 con 0x5648a0b2cbf0 2024-03-19T02:35:17.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faea800df10 con 0x5648a0b357d0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faec4206740 10 client.0 _ll_register_callbacks cb 0x7faeb0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faebaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50344/0 (socket says 172.21.15.116:50344) 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faebaffd640 1 -- 172.21.15.116:0/3616080703 learned_addr learned my addr 172.21.15.116:0/3616080703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b3dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.126+0000 7faebb7fe640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b38a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 -- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 msgr2=0x5648a0b38a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b38a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 -- 172.21.15.116:0/3616080703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 msgr2=0x5648a0b3dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b3dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 -- 172.21.15.116:0/3616080703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a0b39250 con 0x5648a0b2c2d0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faebaffd640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7faeac0980f0 tx=0x7faeac097300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7983913373dc72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:17.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac0b4060 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7faeac097d90 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeac0a3e10 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faeac0bb070 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 10 client.17850 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5648a0bab910 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faeac0a3e10 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7faeac097470 con 0x5648a0b2c2d0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 10 client.17850 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 10 client.17850 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 20 client.17850 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0babfd0 0x5648a0bae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 4 client.17850 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.127+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648a0a9c9f0 con 0x5648a0babfd0 2024-03-19T02:35:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0babfd0 0x5648a0bae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faeba7fc640 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0babfd0 0x5648a0bae3f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6570b326fe426e92 server_cookie=8840ba5a821484f7 in_seq=0 out_seq=0 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7fae9eff5640 10 client.17850 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7fae9eff5640 1 -- 172.21.15.116:0/3616080703 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648a0a9c9f0 con 0x5648a0babfd0 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7fae9eff5640 10 client.17850 handle_command_reply: tid=0 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 client.17850 shutdown 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 msgr2=0x5648a0b3d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7faeac0980f0 tx=0x7faeac097300 comp rx=0 tx=0).stop 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 shutdown_connections 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5648a0babfd0 0x5648a0bae3f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648a0b357d0 0x5648a0b38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648a0b2cbf0 0x5648a0b3dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 --2- 172.21.15.116:0/3616080703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648a0b2c2d0 0x5648a0b3d590 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 >> 172.21.15.116:0/3616080703 conn(0x5648a0b26a00 msgr2=0x5648a0b30f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7fae9eff5640 0 client.17850 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:17.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 shutdown_connections 2024-03-19T02:35:17.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.128+0000 7faec4206740 1 -- 172.21.15.116:0/3616080703 wait complete. 2024-03-19T02:35:17.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec4206740 20 client.17850 trim_cache size 0 max 16384 2024-03-19T02:35:17.136 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 msgr2=0x7faea40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 0x7faea40a4e60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faeb4005cc0 tx=0x7faeb402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 msgr2=0x7faebc073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e9f0 tx=0x7faeac00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 shutdown_connections 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faea40a29a0 0x7faea40a4e60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faebc1077b0 0x7faebc0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faebc102160 0x7faebc074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 --2- 172.21.15.116:0/3943965182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faebc1017a0 0x7faebc073b20 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 >> 172.21.15.116:0/3943965182 conn(0x7faebc0fd2b0 msgr2=0x7faebc0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 shutdown_connections 2024-03-19T02:35:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.129+0000 7faec1619640 1 -- 172.21.15.116:0/3943965182 wait complete. 2024-03-19T02:35:17.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:16 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4243897619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:17.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:35:17.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:16 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4243897619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 -- 172.21.15.116:0/1321685159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0008340 msgr2=0x7f08e00a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/1321685159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0008340 0x7f08e00a2410 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009900 tx=0x7f08d802bc10 comp rx=0 tx=0).stop 2024-03-19T02:35:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 -- 172.21.15.116:0/1321685159 shutdown_connections 2024-03-19T02:35:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/1321685159 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 0x7f08e00a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/1321685159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0008340 0x7f08e00a2410 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/1321685159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0007a20 0x7f08e0007e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 -- 172.21.15.116:0/1321685159 >> 172.21.15.116:0/1321685159 conn(0x7f08e000b740 msgr2=0x7f08e000bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 -- 172.21.15.116:0/1321685159 shutdown_connections 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.255+0000 7f08ee2d0640 1 -- 172.21.15.116:0/1321685159 wait complete. 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 Processor -- start 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 -- start start 2024-03-19T02:35:17.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 0x7f08e0134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 0x7f08e012e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08e00a8170 con 0x7f08e0007a20 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08e00a7ff0 con 0x7f08e0008340 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ee2d0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08e00a82f0 con 0x7f08e00a2950 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ed2ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ed2ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50364/0 (socket says 172.21.15.116:50364) 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ed2ce640 1 -- 172.21.15.116:0/2351344654 learned_addr learned my addr 172.21.15.116:0/2351344654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08ecacd640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 0x7f08e0134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.256+0000 7f08edacf640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 0x7f08e012e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 -- 172.21.15.116:0/2351344654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 msgr2=0x7f08e012e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 0x7f08e012e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 -- 172.21.15.116:0/2351344654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 msgr2=0x7f08e0134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 0x7f08e0134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 -- 172.21.15.116:0/2351344654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e012ed10 con 0x7f08e0007a20 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ecacd640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 0x7f08e0134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f08e80507f0 tx=0x7f08e807aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08de7fc640 1 -- 172.21.15.116:0/2351344654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8085070 con 0x7f08e0007a20 2024-03-19T02:35:17.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08de7fc640 1 -- 172.21.15.116:0/2351344654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08e8078c00 con 0x7f08e0007a20 2024-03-19T02:35:17.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e012f000 con 0x7f08e0007a20 2024-03-19T02:35:17.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08de7fc640 1 -- 172.21.15.116:0/2351344654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e80690d0 con 0x7f08e0007a20 2024-03-19T02:35:17.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.257+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e013b2a0 con 0x7f08e0007a20 2024-03-19T02:35:17.263 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:16 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4243897619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.259+0000 7f08de7fc640 1 -- 172.21.15.116:0/2351344654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f08e8088070 con 0x7f08e0007a20 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 Processor -- start 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 -- start start 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4e2f4b0 con 0x55c2a4ed59b0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08de7fc640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4dd7d70 con 0x55c2a4eccdd0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08f21f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4e2b550 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08edacf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08ed2ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08edacf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51318/0 (socket says 172.21.15.116:51318) 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50380/0 (socket says 172.21.15.116:50380) 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08edacf640 1 -- 172.21.15.116:0/1992657209 learned_addr learned my addr 172.21.15.116:0/1992657209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.260+0000 7f08edacf640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08de7fc640 1 -- 172.21.15.116:0/2351344654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f08e813a7e0 con 0x7f08e0007a20 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 -- 172.21.15.116:0/1992657209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 msgr2=0x55c2a4ed5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 -- 172.21.15.116:0/1992657209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 msgr2=0x55c2a4ed7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 -- 172.21.15.116:0/1992657209 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a4dfa210 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 0x7f08d40a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08e400f850 tx=0x7f08e4018000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08edacf640 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08e4011a30 tx=0x7f08e4011ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6f36932d133f889 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08d37fe640 1 -- 172.21.15.116:0/1992657209 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4004030 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08d37fe640 1 -- 172.21.15.116:0/1992657209 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08e4009a20 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.261+0000 7f08d37fe640 1 -- 172.21.15.116:0/1992657209 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e40039e0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- 172.21.15.116:0/1992657209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 msgr2=0x55c2a4ecc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08e4011a30 tx=0x7f08e4011ef0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- 172.21.15.116:0/1992657209 shutdown_connections 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1992657209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ecc890 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- 172.21.15.116:0/1992657209 >> 172.21.15.116:0/1992657209 conn(0x55c2a4ec6be0 msgr2=0x55c2a4ec6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- 172.21.15.116:0/1992657209 shutdown_connections 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- 172.21.15.116:0/1992657209 wait complete. 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.262+0000 7f08f21f3740 1 -- start start 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 Processor -- start 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4ed8740 con 0x55c2a4ed59b0 2024-03-19T02:35:17.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4deb310 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c2a4ed88c0 con 0x55c2a4eccdd0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08f21f3740 10 client.0 _ll_register_callbacks cb 0x55c2a4ddf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08edacf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08edacf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50382/0 (socket says 172.21.15.116:50382) 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08edacf640 1 -- 172.21.15.116:0/1830141738 learned_addr learned my addr 172.21.15.116:0/1830141738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1830141738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 msgr2=0x55c2a4ed2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1830141738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 msgr2=0x55c2a4ed3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1830141738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a4b433f0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08edacf640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.263+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed2980 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f08e8137af0 tx=0x7f08e80fbf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c12fc5cd4a0a6ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e810dea0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08e4009a20 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f08e810d0d0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f08e8137f70 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 10 client.27674 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2a4c0d7d0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08e4009a20 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f08e80d1890 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 10 client.27674 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 10 client.27674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 20 client.27674 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4ee5c90 0x55c2a4ee80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 4 client.27674 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.264+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2a4e8d3e0 con 0x55c2a4ee5c90 2024-03-19T02:35:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.265+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4ee5c90 0x55c2a4ee80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.265+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4ee5c90 0x55c2a4ee80e0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1580f13f344fc6 server_cookie=9784cdba41ac4e60 in_seq=0 out_seq=0 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.265+0000 7f08d0ff9640 10 client.27674 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1830141738 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2a4e8d3e0 con 0x55c2a4ee5c90 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08d0ff9640 10 client.27674 handle_command_reply: tid=0 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 client.27674 shutdown 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 msgr2=0x55c2a4ed2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed2980 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f08e8137af0 tx=0x7f08e80fbf40 comp rx=0 tx=0).stop 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 shutdown_connections 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4ee5c90 0x55c2a4ee80e0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ed3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08d0ff9640 0 client.27674 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1830141738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed2980 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 >> 172.21.15.116:0/1830141738 conn(0x55c2a4ec6be0 msgr2=0x55c2a4ec8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 shutdown_connections 2024-03-19T02:35:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 1 -- 172.21.15.116:0/1830141738 wait complete. 2024-03-19T02:35:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.266+0000 7f08f21f3740 20 client.27674 trim_cache size 0 max 16384 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 Processor -- start 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 -- start start 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ee1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ec8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08e0145240 con 0x55c2a4eccdd0 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08e0146d80 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08f21f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08e0145910 con 0x55c2a4ed59b0 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50386/0 (socket says 172.21.15.116:50386) 2024-03-19T02:35:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 -- 172.21.15.116:0/2250594791 learned_addr learned my addr 172.21.15.116:0/2250594791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ee1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08edacf640 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ec8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 -- 172.21.15.116:0/2250594791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ed59b0 msgr2=0x55c2a4ec8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ec8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 -- 172.21.15.116:0/2250594791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 msgr2=0x55c2a4ee1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.281+0000 7f08ecacd640 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ee1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.282+0000 7f08ecacd640 1 -- 172.21.15.116:0/2250594791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a4ee1a50 con 0x55c2a4eccdd0 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.282+0000 7f08ecacd640 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009900 tx=0x7f08d802c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffb972d3d17c72d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.282+0000 7f08d2ffd640 1 -- 172.21.15.116:0/2250594791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d804dce0 con 0x55c2a4eccdd0 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.282+0000 7f08d2ffd640 1 -- 172.21.15.116:0/2250594791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08d802cb90 con 0x55c2a4eccdd0 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.282+0000 7f08d2ffd640 1 -- 172.21.15.116:0/2250594791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d804d5c0 con 0x55c2a4eccdd0 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- 172.21.15.116:0/2250594791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 msgr2=0x55c2a4ed3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009900 tx=0x7f08d802c310 comp rx=0 tx=0).stop 2024-03-19T02:35:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- 172.21.15.116:0/2250594791 shutdown_connections 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ed59b0 0x55c2a4ec8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed3220 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 --2- 172.21.15.116:0/2250594791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ee1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- 172.21.15.116:0/2250594791 >> 172.21.15.116:0/2250594791 conn(0x55c2a4ec6be0 msgr2=0x55c2a4ec8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- 172.21.15.116:0/2250594791 shutdown_connections 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- 172.21.15.116:0/2250594791 wait complete. 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 -- start start 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.283+0000 7f08f21f3740 1 Processor -- start 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ed59b0 0x55c2a4f49f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08e01466b0 con 0x55c2a4eccdd0 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08e0145fe0 con 0x55c2a4ed59b0 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08e0143240 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08ecacd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50388/0 (socket says 172.21.15.116:50388) 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08ed2ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08ecacd640 1 -- 172.21.15.116:0/1348545100 learned_addr learned my addr 172.21.15.116:0/1348545100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08f21f3740 10 client.0 _ll_register_callbacks cb 0x7f08e81065f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.284+0000 7f08edacf640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ed59b0 0x55c2a4f49f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ed59b0 msgr2=0x55c2a4f49f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ed59b0 0x55c2a4f49f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1348545100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 msgr2=0x55c2a4ed9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 -- 172.21.15.116:0/1348545100 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a4f4a590 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08ed2ce640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed8cf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08e802d7b0 tx=0x7f08e80675c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa6374e12591a801 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e810f1f0 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d802cb90 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f08e810ed10 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f08e8102d50 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08f21f3740 10 client.27622 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2a4f4b850 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.285+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08d802cb90 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f08e813a340 con 0x55c2a4ecc4b0 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 10 client.27622 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 10 client.27622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 20 client.27622 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4f4be10 0x55c2a4f4e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 4 client.27622 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2a4e3c9f0 con 0x55c2a4f4be10 2024-03-19T02:35:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4f4be10 0x55c2a4f4e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08d0ff9640 10 client.27622 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.286+0000 7f08ecacd640 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4f4be10 0x55c2a4f4e230 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2272e11d3e679004 server_cookie=ef1d14cf8011f9e3 in_seq=0 out_seq=0 2024-03-19T02:35:17.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08d0ff9640 1 -- 172.21.15.116:0/1348545100 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2a4e3c9f0 con 0x55c2a4f4be10 2024-03-19T02:35:17.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08d0ff9640 10 client.27622 handle_command_reply: tid=0 2024-03-19T02:35:17.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 client.27622 shutdown 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 msgr2=0x55c2a4ed8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed8cf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08e802d7b0 tx=0x7f08e80675c0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 shutdown_connections 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c2a4f4be10 0x55c2a4f4e230 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2a4ed59b0 0x55c2a4f49f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2a4eccdd0 0x55c2a4ed9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08d0ff9640 0 client.27622 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 --2- 172.21.15.116:0/1348545100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c2a4ecc4b0 0x55c2a4ed8cf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 >> 172.21.15.116:0/1348545100 conn(0x55c2a4ec6be0 msgr2=0x55c2a4ec5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 shutdown_connections 2024-03-19T02:35:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 1 -- 172.21.15.116:0/1348545100 wait complete. 2024-03-19T02:35:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08f21f3740 20 client.27622 trim_cache size 0 max 16384 2024-03-19T02:35:17.301 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 msgr2=0x7f08d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 0x7f08d40a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08e400f850 tx=0x7f08e4018000 comp rx=0 tx=0).stop 2024-03-19T02:35:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 msgr2=0x7f08e01343e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.288+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f08e80507f0 tx=0x7f08e807aee0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 shutdown_connections 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08d40a2970 0x7f08d40a4e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08e00a2950 0x7f08e012e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e0008340 0x7f08e0134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 --2- 172.21.15.116:0/2351344654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08e0007a20 0x7f08e01343e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 >> 172.21.15.116:0/2351344654 conn(0x7f08e000b740 msgr2=0x7f08e0094570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 shutdown_connections 2024-03-19T02:35:17.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:17.289+0000 7f08ee2d0640 1 -- 172.21.15.116:0/2351344654 wait complete. 2024-03-19T02:35:17.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 240103 3.20 MB/sec execute 280 sec latency 22.441 ms 2024-03-19T02:35:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:17 smithi162 ceph-mon[35761]: pgmap v231: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:17 smithi154 ceph-mon[35693]: pgmap v231: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:17 smithi116 ceph-mon[32263]: pgmap v231: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:18.306 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:35:18.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:35:18.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- 172.21.15.116:0/846010622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100d00 msgr2=0x7f00801098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 --2- 172.21.15.116:0/846010622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100d00 0x7f00801098a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f0074009900 tx=0x7f007402bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- 172.21.15.116:0/846010622 shutdown_connections 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 --2- 172.21.15.116:0/846010622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0080109fd0 0x7f00801109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 --2- 172.21.15.116:0/846010622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100d00 0x7f00801098a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 --2- 172.21.15.116:0/846010622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080100350 0x7f0080100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- 172.21.15.116:0/846010622 >> 172.21.15.116:0/846010622 conn(0x7f0080075a60 msgr2=0x7f0080075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- 172.21.15.116:0/846010622 shutdown_connections 2024-03-19T02:35:18.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- 172.21.15.116:0/846010622 wait complete. 2024-03-19T02:35:18.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 Processor -- start 2024-03-19T02:35:18.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.423+0000 7f008643d640 1 -- start start 2024-03-19T02:35:18.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 0x7f008019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 0x7f008019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0080113ca0 con 0x7f0080100350 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0080113b20 con 0x7f008019eaa0 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f008643d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0080113e20 con 0x7f0080109fd0 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52578/0 (socket says 172.21.15.116:52578) 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 -- 172.21.15.116:0/226602104 learned_addr learned my addr 172.21.15.116:0/226602104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007f7fe640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 0x7f008019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f00849da640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 0x7f008019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 -- 172.21.15.116:0/226602104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 msgr2=0x7f008019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 0x7f008019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 -- 172.21.15.116:0/226602104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 msgr2=0x7f008019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 0x7f008019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 -- 172.21.15.116:0/226602104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008019f590 con 0x7f0080100350 2024-03-19T02:35:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007f7fe640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 0x7f008019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f00849da640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 0x7f008019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.424+0000 7f007ffff640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f007000e9c0 tx=0x7f007000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.425+0000 7f007d7fa640 1 -- 172.21.15.116:0/226602104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007000cd80 con 0x7f0080100350 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.425+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008019f880 con 0x7f0080100350 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.425+0000 7f007d7fa640 1 -- 172.21.15.116:0/226602104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00700035f0 con 0x7f0080100350 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.425+0000 7f007d7fa640 1 -- 172.21.15.116:0/226602104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070010690 con 0x7f0080100350 2024-03-19T02:35:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.425+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00801a7090 con 0x7f0080100350 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f007d7fa640 1 -- 172.21.15.116:0/226602104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f00700037a0 con 0x7f0080100350 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 Processor -- start 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 -- start start 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f007d7fa640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 0x7f00680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3e8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3eac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e36a343160 con 0x55e36a3df380 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e36a2ebe10 con 0x55e36a3e8880 2024-03-19T02:35:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.427+0000 7f0089022740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e36a33f200 con 0x55e36a3dfca0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3e8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3df760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52586/0 (socket says 172.21.15.116:52586) 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00849da640 1 -- 172.21.15.116:0/3771262726 learned_addr learned my addr 172.21.15.116:0/3771262726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007d7fa640 1 -- 172.21.15.116:0/226602104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f00700cf5d0 con 0x7f0080100350 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007ffff640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 0x7f00680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00849da640 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 -- 172.21.15.116:0/3771262726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 msgr2=0x55e36a3eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 -- 172.21.15.116:0/3771262726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 msgr2=0x55e36a3df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 -- 172.21.15.116:0/3771262726 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e36a11f7e0 con 0x55e36a3dfca0 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00849da640 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:35:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007ffff640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 0x7f00680a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f00700cc9a0 tx=0x7f0070066a60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3e8340 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f007402c170 tx=0x7f0074009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dec81475c851c11e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00667fc640 1 -- 172.21.15.116:0/3771262726 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074045070 con 0x55e36a3dfca0 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00667fc640 1 -- 172.21.15.116:0/3771262726 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f007402c7e0 con 0x55e36a3dfca0 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.428+0000 7f00667fc640 1 -- 172.21.15.116:0/3771262726 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074003b10 con 0x55e36a3dfca0 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- 172.21.15.116:0/3771262726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 msgr2=0x55e36a3e8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3e8340 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f007402c170 tx=0x7f0074009610 comp rx=0 tx=0).stop 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- 172.21.15.116:0/3771262726 shutdown_connections 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3e8340 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 --2- 172.21.15.116:0/3771262726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- 172.21.15.116:0/3771262726 >> 172.21.15.116:0/3771262726 conn(0x55e36a3d9ab0 msgr2=0x55e36a3d9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- 172.21.15.116:0/3771262726 shutdown_connections 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- 172.21.15.116:0/3771262726 wait complete. 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.429+0000 7f0089022740 1 -- start start 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 Processor -- start 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3df380 0x55e36a3e5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3e8880 0x55e36a3e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e36a3eb610 con 0x55e36a3dfca0 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e36a2ff1a0 con 0x55e36a3df380 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e36a3eb790 con 0x55e36a3e8880 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f0089022740 10 client.0 _ll_register_callbacks cb 0x55e36a056c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52600/0 (socket says 172.21.15.116:52600) 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 -- 172.21.15.116:0/1872846390 learned_addr learned my addr 172.21.15.116:0/1872846390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f00849da640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3df380 0x55e36a3e5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f00849da640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3e8880 0x55e36a3e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 -- 172.21.15.116:0/1872846390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3e8880 msgr2=0x55e36a3e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3e8880 0x55e36a3e61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 -- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3df380 msgr2=0x55e36a3e5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3df380 0x55e36a3e5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.430+0000 7f007f7fe640 1 -- 172.21.15.116:0/1872846390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e36a17ff70 con 0x55e36a3dfca0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f007f7fe640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0074005c40 tx=0x7f007403f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80b6260e6a6983 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007404e070 con 0x55e36a3dfca0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007402c7e0 con 0x55e36a3dfca0 2024-03-19T02:35:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f007403fc70 con 0x55e36a3dfca0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0074059050 con 0x55e36a3dfca0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 10 client.17880 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e36a017170 con 0x55e36a3dfca0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f007402c7e0 con 0x55e36a3dfca0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0074034440 con 0x55e36a3dfca0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 10 client.17880 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 10 client.17880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.431+0000 7f0089022740 20 client.17880 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a3f8880 0x55e36a3fac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f0089022740 4 client.17880 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e36a3506a0 con 0x55e36a3f8880 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f00849da640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a3f8880 0x55e36a3fac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f005f7fe640 10 client.17880 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.432+0000 7f00849da640 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a3f8880 0x55e36a3fac70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8580483a3c0ce33f server_cookie=3295b2a4b9b0c9e2 in_seq=0 out_seq=0 2024-03-19T02:35:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.433+0000 7f005f7fe640 1 -- 172.21.15.116:0/1872846390 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e36a3506a0 con 0x55e36a3f8880 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.433+0000 7f005f7fe640 10 client.17880 handle_command_reply: tid=0 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.433+0000 7f0089022740 1 client.17880 shutdown 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 msgr2=0x55e36a3e5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0074005c40 tx=0x7f007403f6f0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 shutdown_connections 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a3f8880 0x55e36a3fac70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3e8880 0x55e36a3e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3dfca0 0x55e36a3e5cb0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 --2- 172.21.15.116:0/1872846390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3df380 0x55e36a3e5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 >> 172.21.15.116:0/1872846390 conn(0x55e36a3d9ab0 msgr2=0x55e36a3db800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f005f7fe640 0 client.17880 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 shutdown_connections 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 1 -- 172.21.15.116:0/1872846390 wait complete. 2024-03-19T02:35:18.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.434+0000 7f0089022740 20 client.17880 trim_cache size 0 max 16384 2024-03-19T02:35:18.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 Processor -- start 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 -- start start 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3df380 0x55e36a3e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3dfca0 0x55e36a3fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00801b0ea0 con 0x55e36a3e8880 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00801b29e0 con 0x55e36a3dfca0 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f0089022740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00801b1570 con 0x55e36a3df380 2024-03-19T02:35:18.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52614/0 (socket says 172.21.15.116:52614) 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 -- 172.21.15.116:0/2967181671 learned_addr learned my addr 172.21.15.116:0/2967181671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f007f7fe640 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3dfca0 0x55e36a3fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3df380 0x55e36a3e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 -- 172.21.15.116:0/2967181671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3df380 msgr2=0x55e36a3e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3df380 0x55e36a3e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 -- 172.21.15.116:0/2967181671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3dfca0 msgr2=0x55e36a3fd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.454+0000 7f00849da640 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3dfca0 0x55e36a3fd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.455+0000 7f00849da640 1 -- 172.21.15.116:0/2967181671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e36a3dfad0 con 0x55e36a3e8880 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.455+0000 7f00849da640 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x55e36a3da620 tx=0x7f0078009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e685f1b72ff135 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.455+0000 7f0065ffb640 1 -- 172.21.15.116:0/2967181671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007802a070 con 0x55e36a3e8880 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.455+0000 7f0065ffb640 1 -- 172.21.15.116:0/2967181671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0078024230 con 0x55e36a3e8880 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.455+0000 7f0065ffb640 1 -- 172.21.15.116:0/2967181671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007801a040 con 0x55e36a3e8880 2024-03-19T02:35:18.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- 172.21.15.116:0/2967181671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 msgr2=0x55e36a3e2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x55e36a3da620 tx=0x7f0078009500 comp rx=0 tx=0).stop 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- 172.21.15.116:0/2967181671 shutdown_connections 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3e8880 0x55e36a3e2a10 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3dfca0 0x55e36a3fd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- 172.21.15.116:0/2967181671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3df380 0x55e36a3e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- 172.21.15.116:0/2967181671 >> 172.21.15.116:0/2967181671 conn(0x55e36a3d9ab0 msgr2=0x55e36a3f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- 172.21.15.116:0/2967181671 shutdown_connections 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- 172.21.15.116:0/2967181671 wait complete. 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 -- start start 2024-03-19T02:35:18.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 Processor -- start 2024-03-19T02:35:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3fda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.456+0000 7f0089022740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f0089022740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00801b2310 con 0x55e36a3df380 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f0089022740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00801b1c40 con 0x55e36a3e8880 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f0089022740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00801aeea0 con 0x55e36a3dfca0 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f0089022740 10 client.0 _ll_register_callbacks cb 0x7f0078009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f00849da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52622/0 (socket says 172.21.15.116:52622) 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f00849da640 1 -- 172.21.15.116:0/3183235426 learned_addr learned my addr 172.21.15.116:0/3183235426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3fda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f00849da640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 -- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 msgr2=0x55e36a3fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3fbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 -- 172.21.15.116:0/3183235426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 msgr2=0x55e36a3fd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 -- 172.21.15.116:0/3183235426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e36a3fc1c0 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f00849da640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f007f7fe640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3fda90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0074000c00 tx=0x7f0074034650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13f1d7ff85708477 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f005f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007404e040 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0078024230 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.457+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f00740349b0 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0074066050 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 10 client.27640 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e36a45f400 con 0x55e36a3dfca0 2024-03-19T02:35:18.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0078024230 con 0x55e36a3dfca0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0074034bd0 con 0x55e36a3dfca0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 10 client.27640 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 10 client.27640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 20 client.27640 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a45fbb0 0x55e36a461fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 4 client.27640 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e36a34a9f0 con 0x55e36a45fbb0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.458+0000 7f00849da640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a45fbb0 0x55e36a461fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.459+0000 7f00849da640 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a45fbb0 0x55e36a461fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d81ffeb2de36d server_cookie=fced30667c5a53f in_seq=0 out_seq=0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.459+0000 7f005f7fe640 10 client.27640 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f005f7fe640 1 -- 172.21.15.116:0/3183235426 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e36a34a9f0 con 0x55e36a45fbb0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f005f7fe640 10 client.27640 handle_command_reply: tid=0 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 client.27640 shutdown 2024-03-19T02:35:18.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 msgr2=0x55e36a3fda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3fda90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0074000c00 tx=0x7f0074034650 comp rx=0 tx=0).stop 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 shutdown_connections 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e36a45fbb0 0x55e36a461fd0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e36a3e8880 0x55e36a3fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e36a3dfca0 0x55e36a3fda90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f005f7fe640 0 client.27640 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 --2- 172.21.15.116:0/3183235426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e36a3df380 0x55e36a3fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 >> 172.21.15.116:0/3183235426 conn(0x55e36a3d9ab0 msgr2=0x55e36a3d82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 shutdown_connections 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 1 -- 172.21.15.116:0/3183235426 wait complete. 2024-03-19T02:35:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.460+0000 7f0089022740 20 client.27640 trim_cache size 0 max 16384 2024-03-19T02:35:18.467 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:35:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "879b21b7-5464-4e14-bac0-10b8ce26ffed", 2024-03-19T02:35:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:35:18.467 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 msgr2=0x7f00680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 0x7f00680a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f00700cc9a0 tx=0x7f0070066a60 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 msgr2=0x7f008019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f007000e9c0 tx=0x7f007000ee80 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 shutdown_connections 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f00680a2970 0x7f00680a4e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008019eaa0 0x7f008019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0080109fd0 0x7f008019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 --2- 172.21.15.116:0/226602104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0080100350 0x7f008019a1e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 >> 172.21.15.116:0/226602104 conn(0x7f0080075a60 msgr2=0x7f0080107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 shutdown_connections 2024-03-19T02:35:18.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.461+0000 7f008643d640 1 -- 172.21.15.116:0/226602104 wait complete. 2024-03-19T02:35:18.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:35:18.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 -- 172.21.15.116:0/2813865084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 msgr2=0x7f62d010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 --2- 172.21.15.116:0/2813865084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d010d560 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009cd0 tx=0x7f62c802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 -- 172.21.15.116:0/2813865084 shutdown_connections 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 --2- 172.21.15.116:0/2813865084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d010d560 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 --2- 172.21.15.116:0/2813865084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d0100690 0x7f62d0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 --2- 172.21.15.116:0/2813865084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d01066a0 0x7f62d0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 -- 172.21.15.116:0/2813865084 >> 172.21.15.116:0/2813865084 conn(0x7f62d0077820 msgr2=0x7f62d0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 -- 172.21.15.116:0/2813865084 shutdown_connections 2024-03-19T02:35:18.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.585+0000 7f62d5080640 1 -- 172.21.15.116:0/2813865084 wait complete. 2024-03-19T02:35:18.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 Processor -- start 2024-03-19T02:35:18.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 -- start start 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 0x7f62d019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0110960 con 0x7f62d0101050 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01107e0 con 0x7f62d0100690 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.586+0000 7f62d5080640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0110ae0 con 0x7f62d01066a0 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 0x7f62d019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47374/0 (socket says 172.21.15.116:47374) 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 0x7f62d019c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52662/0 (socket says 172.21.15.116:52662) 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 -- 172.21.15.116:0/2881730587 learned_addr learned my addr 172.21.15.116:0/2881730587 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62c659c640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 -- 172.21.15.116:0/2881730587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 msgr2=0x7f62d019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 0x7f62d019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 -- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 msgr2=0x7f62d0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 -- 172.21.15.116:0/2881730587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d019d020 con 0x7f62d0100690 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.587+0000 7f62ced9d640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f62c000e9f0 tx=0x7f62c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.588+0000 7f62ccd99640 1 -- 172.21.15.116:0/2881730587 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c000cd80 con 0x7f62d0100690 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.588+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d019d310 con 0x7f62d0100690 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.588+0000 7f62ccd99640 1 -- 172.21.15.116:0/2881730587 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c00108e0 con 0x7f62d0100690 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.588+0000 7f62ccd99640 1 -- 172.21.15.116:0/2881730587 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0018920 con 0x7f62d0100690 2024-03-19T02:35:18.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.588+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01a4ba0 con 0x7f62d0100690 2024-03-19T02:35:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.590+0000 7f62ccd99640 1 -- 172.21.15.116:0/2881730587 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f62c0010a90 con 0x7f62d0100690 2024-03-19T02:35:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.590+0000 7f62d7c5f740 1 Processor -- start 2024-03-19T02:35:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.590+0000 7f62d7c5f740 1 -- start start 2024-03-19T02:35:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a32f30 0x55ce90a35320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ce9098d500 con 0x55ce90a2a350 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ce90935be0 con 0x55ce90a29a30 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce909895a0 con 0x55ce90a32f30 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ccd99640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 0x7f62bc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47386/0 (socket says 172.21.15.116:47386) 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 -- 172.21.15.116:0/3678032337 learned_addr learned my addr 172.21.15.116:0/3678032337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62c659c640 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a32f30 0x55ce90a35320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 0x7f62bc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ccd99640 1 -- 172.21.15.116:0/2881730587 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f62c00cf450 con 0x7f62d0100690 2024-03-19T02:35:18.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 -- 172.21.15.116:0/3678032337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a32f30 msgr2=0x55ce90a35320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a32f30 0x55ce90a35320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 -- 172.21.15.116:0/3678032337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 msgr2=0x55ce90a29e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 -- 172.21.15.116:0/3678032337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce90957000 con 0x55ce90a2a350 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62ced9d640 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a329f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f62c00cc7c0 tx=0x7f62c001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7068d1c221ddd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62c559a640 1 -- 172.21.15.116:0/3678032337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0010de0 con 0x55ce90a2a350 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.591+0000 7f62cf59e640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 0x7f62bc0a4eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009a60 tx=0x7f62c802c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62c559a640 1 -- 172.21.15.116:0/3678032337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c009b780 con 0x55ce90a2a350 2024-03-19T02:35:18.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62c559a640 1 -- 172.21.15.116:0/3678032337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c00cca40 con 0x55ce90a2a350 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 -- 172.21.15.116:0/3678032337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 msgr2=0x55ce90a329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a329f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f62c00cc7c0 tx=0x7f62c001f000 comp rx=0 tx=0).stop 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 -- 172.21.15.116:0/3678032337 shutdown_connections 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a32f30 0x55ce90a35320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a329f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/3678032337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a29e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 -- 172.21.15.116:0/3678032337 >> 172.21.15.116:0/3678032337 conn(0x55ce90a24160 msgr2=0x55ce90a24540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.592+0000 7f62d7c5f740 1 -- 172.21.15.116:0/3678032337 shutdown_connections 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 -- 172.21.15.116:0/3678032337 wait complete. 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 -- start start 2024-03-19T02:35:18.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 Processor -- start 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a2fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a30350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ce90a35cc0 con 0x55ce90a32f30 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ce90949160 con 0x55ce90a29a30 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce90a35e40 con 0x55ce90a2a350 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62d7c5f740 10 client.0 _ll_register_callbacks cb 0x55ce90985a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a2fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52640/0 (socket says 172.21.15.116:52640) 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62cf59e640 1 -- 172.21.15.116:0/1617911559 learned_addr learned my addr 172.21.15.116:0/1617911559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.593+0000 7f62c659c640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a30350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 -- 172.21.15.116:0/1617911559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 msgr2=0x55ce90a30350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a30350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 -- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 msgr2=0x55ce90a2fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a2fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 -- 172.21.15.116:0/1617911559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9076b7d0 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62cf59e640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005cc0 tx=0x7f62c80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8bd186caa460b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8052060 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f62c8004450 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c009b780 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f62c8059050 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62d7c5f740 10 client.17898 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce906a13f0 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c009b780 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.594+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f62c803f070 con 0x55ce90a32f30 2024-03-19T02:35:18.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 10 client.17898 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 10 client.17898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 20 client.17898 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90a45580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 4 client.17898 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce9099aa40 con 0x55ce90a43100 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62ced9d640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90a45580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62ced9d640 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90a45580 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcab9734d7fca30a server_cookie=620c623badda8484 in_seq=0 out_seq=0 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.595+0000 7f62b67fc640 10 client.17898 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62b67fc640 1 -- 172.21.15.116:0/1617911559 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce9099aa40 con 0x55ce90a43100 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62b67fc640 10 client.17898 handle_command_reply: tid=0 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 client.17898 shutdown 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 msgr2=0x55ce90a30890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005cc0 tx=0x7f62c80095c0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 shutdown_connections 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90a45580 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62b67fc640 0 client.17898 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a32f30 0x55ce90a30890 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a30350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1617911559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a29a30 0x55ce90a2fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 >> 172.21.15.116:0/1617911559 conn(0x55ce90a24160 msgr2=0x55ce90a25eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 shutdown_connections 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1617911559 wait complete. 2024-03-19T02:35:18.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.596+0000 7f62d7c5f740 20 client.17898 trim_cache size 0 max 16384 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 Processor -- start 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 -- start start 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a25590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01acb10 con 0x55ce90a29a30 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01ae650 con 0x55ce90a32f30 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01ad1e0 con 0x55ce90a2a350 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52654/0 (socket says 172.21.15.116:52654) 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62ced9d640 1 -- 172.21.15.116:0/1914889202 learned_addr learned my addr 172.21.15.116:0/1914889202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62c659c640 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a25590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.614+0000 7f62cf59e640 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 -- 172.21.15.116:0/1914889202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 msgr2=0x55ce90a25590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a25590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 -- 172.21.15.116:0/1914889202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 msgr2=0x55ce90a3ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a3ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 -- 172.21.15.116:0/1914889202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce906610f0 con 0x55ce90a29a30 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62ced9d640 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f62c009b480 tx=0x7f62c009f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57a521b8d639c62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62c4d99640 1 -- 172.21.15.116:0/1914889202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c00af030 con 0x55ce90a29a30 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62c4d99640 1 -- 172.21.15.116:0/1914889202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c009b780 con 0x55ce90a29a30 2024-03-19T02:35:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.615+0000 7f62c4d99640 1 -- 172.21.15.116:0/1914889202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c00a39e0 con 0x55ce90a29a30 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1914889202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 msgr2=0x55ce90a32540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f62c009b480 tx=0x7f62c009f9a0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1914889202 shutdown_connections 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a2a350 0x55ce90a25590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/1914889202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a29a30 0x55ce90a32540 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1914889202 >> 172.21.15.116:0/1914889202 conn(0x55ce90a24160 msgr2=0x55ce90a2ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1914889202 shutdown_connections 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- 172.21.15.116:0/1914889202 wait complete. 2024-03-19T02:35:18.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- start start 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 Processor -- start 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a29a30 0x55ce90a2c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a2d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01adf80 con 0x55ce90a2a350 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01ad8b0 con 0x55ce90a32f30 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.616+0000 7f62d7c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62d01aab10 con 0x55ce90a29a30 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62d7c5f740 10 client.0 _ll_register_callbacks cb 0x7f62c8032820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52670/0 (socket says 172.21.15.116:52670) 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a29a30 0x55ce90a2c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 -- 172.21.15.116:0/2380323925 learned_addr learned my addr 172.21.15.116:0/2380323925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:18.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62cf59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a2d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 -- 172.21.15.116:0/2380323925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a29a30 msgr2=0x55ce90a2c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a29a30 0x55ce90a2c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 -- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 msgr2=0x55ce90a2d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a2d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 -- 172.21.15.116:0/2380323925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce90aa8480 con 0x55ce90a2a350 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62c659c640 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x55ce90a252f0 tx=0x7f62b8012c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa45c0358988c2d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b8023060 con 0x55ce90a2a350 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f62b8004230 con 0x55ce90a2a350 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c009b780 con 0x55ce90a2a350 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f62b802a050 con 0x55ce90a2a350 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62d7c5f740 10 client.17904 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.617+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce90aa9700 con 0x55ce90a2a350 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c009b780 con 0x55ce90a2a350 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f62b8003a10 con 0x55ce90a2a350 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 10 client.17904 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 10 client.17904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 20 client.17904 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90aabf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 4 client.17904 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ce90994d90 con 0x55ce90a43100 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62ced9d640 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90aabf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62b67fc640 10 client.17904 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.618+0000 7f62ced9d640 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90aabf20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70062b0c09d1509f server_cookie=f2e8ea50e73ef9b7 in_seq=0 out_seq=0 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62b67fc640 1 -- 172.21.15.116:0/2380323925 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ce90994d90 con 0x55ce90a43100 2024-03-19T02:35:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62b67fc640 10 client.17904 handle_command_reply: tid=0 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 client.17904 shutdown 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 msgr2=0x55ce90a2cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x55ce90a252f0 tx=0x7f62b8012c50 comp rx=0 tx=0).stop 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 shutdown_connections 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ce90a43100 0x55ce90aabf20 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce90a32f30 0x55ce90a2d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62b67fc640 0 client.17904 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce90a2a350 0x55ce90a2cdd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 --2- 172.21.15.116:0/2380323925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce90a29a30 0x55ce90a2c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 >> 172.21.15.116:0/2380323925 conn(0x55ce90a24160 msgr2=0x55ce90a2e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 shutdown_connections 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 1 -- 172.21.15.116:0/2380323925 wait complete. 2024-03-19T02:35:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.619+0000 7f62d7c5f740 20 client.17904 trim_cache size 0 max 16384 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "879b21b7-5464-4e14-bac0-10b8ce26ffed": { 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "879b21b7-5464-4e14-bac0-10b8ce26ffed", 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:35:18.626 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 msgr2=0x7f62bc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 0x7f62bc0a4eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009a60 tx=0x7f62c802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 msgr2=0x7f62d0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f62c000e9f0 tx=0x7f62c000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 shutdown_connections 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62bc0a29f0 0x7f62bc0a4eb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d01066a0 0x7f62d019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d0101050 0x7f62d0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 --2- 172.21.15.116:0/2881730587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d0100690 0x7f62d0198030 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 >> 172.21.15.116:0/2881730587 conn(0x7f62d0077820 msgr2=0x7f62d0068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 shutdown_connections 2024-03-19T02:35:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:18.620+0000 7f62d5080640 1 -- 172.21.15.116:0/2881730587 wait complete. 2024-03-19T02:35:18.631 INFO:tasks.cephfs.filesystem:scrub status for tag:879b21b7-5464-4e14-bac0-10b8ce26ffed - {'path': '/', 'tag': '879b21b7-5464-4e14-bac0-10b8ce26ffed', 'options': 'recursive,force'} 2024-03-19T02:35:18.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 240977 3.21 MB/sec execute 281 sec latency 16.111 ms 2024-03-19T02:35:19.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 241354 3.21 MB/sec execute 282 sec latency 26.119 ms 2024-03-19T02:35:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:35:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:35:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:35:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:19 smithi162 ceph-mon[35761]: pgmap v232: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:20.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:19 smithi154 ceph-mon[35693]: pgmap v232: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:19 smithi116 ceph-mon[32263]: pgmap v232: 129 pgs: 129 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-03-19T02:35:20.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 241754 3.20 MB/sec execute 283 sec latency 17.705 ms 2024-03-19T02:35:21.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 242307 3.20 MB/sec execute 284 sec latency 36.200 ms 2024-03-19T02:35:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:21 smithi162 ceph-mon[35761]: pgmap v233: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-03-19T02:35:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:21 smithi154 ceph-mon[35693]: pgmap v233: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-03-19T02:35:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:21 smithi116 ceph-mon[32263]: pgmap v233: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-03-19T02:35:22.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 242889 3.20 MB/sec execute 285 sec latency 25.080 ms 2024-03-19T02:35:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 243500 3.19 MB/sec execute 286 sec latency 23.580 ms 2024-03-19T02:35:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:23 smithi162 ceph-mon[35761]: pgmap v234: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-03-19T02:35:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:23 smithi154 ceph-mon[35693]: pgmap v234: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-03-19T02:35:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:23 smithi116 ceph-mon[32263]: pgmap v234: 129 pgs: 129 active+clean; 225 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-03-19T02:35:24.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 244158 3.20 MB/sec execute 287 sec latency 53.733 ms 2024-03-19T02:35:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:24 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:35:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:24 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:35:25.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:24 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:35:25.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 244601 3.20 MB/sec execute 288 sec latency 17.957 ms 2024-03-19T02:35:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:35:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:25 smithi162 ceph-mon[35761]: pgmap v235: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:35:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:25 smithi154 ceph-mon[35693]: pgmap v235: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:35:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:25 smithi116 ceph-mon[32263]: pgmap v235: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:26.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 245150 3.20 MB/sec execute 289 sec latency 20.740 ms 2024-03-19T02:35:27.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 245613 3.19 MB/sec execute 290 sec latency 34.517 ms 2024-03-19T02:35:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:27 smithi154 ceph-mon[35693]: pgmap v236: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:27 smithi116 ceph-mon[32263]: pgmap v236: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:27 smithi162 ceph-mon[35761]: pgmap v236: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:28.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 246241 3.20 MB/sec execute 291 sec latency 24.470 ms 2024-03-19T02:35:29.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 246719 3.19 MB/sec execute 292 sec latency 37.188 ms 2024-03-19T02:35:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:29 smithi154 ceph-mon[35693]: pgmap v237: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:29 smithi116 ceph-mon[32263]: pgmap v237: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:29 smithi162 ceph-mon[35761]: pgmap v237: 129 pgs: 129 active+clean; 228 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:35:30.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 247337 3.19 MB/sec execute 293 sec latency 20.325 ms 2024-03-19T02:35:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:30 smithi154 ceph-mon[35693]: pgmap v238: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:35:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:30 smithi116 ceph-mon[32263]: pgmap v238: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:35:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:30 smithi162 ceph-mon[35761]: pgmap v238: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-03-19T02:35:31.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 248096 3.20 MB/sec execute 294 sec latency 17.587 ms 2024-03-19T02:35:32.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 248562 3.20 MB/sec execute 295 sec latency 20.637 ms 2024-03-19T02:35:33.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 248941 3.19 MB/sec execute 296 sec latency 29.760 ms 2024-03-19T02:35:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:33 smithi162 ceph-mon[35761]: pgmap v239: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:35:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:33 smithi154 ceph-mon[35693]: pgmap v239: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:35:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:33 smithi116 ceph-mon[32263]: pgmap v239: 129 pgs: 129 active+clean; 230 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:35:34.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 249493 3.19 MB/sec execute 297 sec latency 21.763 ms 2024-03-19T02:35:35.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 250034 3.19 MB/sec execute 298 sec latency 29.784 ms 2024-03-19T02:35:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:35 smithi162 ceph-mon[35761]: pgmap v240: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:35:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:35 smithi154 ceph-mon[35693]: pgmap v240: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:35:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:35 smithi116 ceph-mon[32263]: pgmap v240: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:35:36.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 250665 3.18 MB/sec execute 299 sec latency 20.976 ms 2024-03-19T02:35:37.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 251493 3.19 MB/sec execute 300 sec latency 18.086 ms 2024-03-19T02:35:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:37 smithi162 ceph-mon[35761]: pgmap v241: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:37 smithi154 ceph-mon[35693]: pgmap v241: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:37 smithi116 ceph-mon[32263]: pgmap v241: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:38.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 251885 3.20 MB/sec execute 301 sec latency 20.670 ms 2024-03-19T02:35:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 252293 3.19 MB/sec execute 302 sec latency 36.898 ms 2024-03-19T02:35:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:39 smithi162 ceph-mon[35761]: pgmap v242: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:39 smithi154 ceph-mon[35693]: pgmap v242: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:39 smithi116 ceph-mon[32263]: pgmap v242: 129 pgs: 129 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:40.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 252752 3.18 MB/sec execute 303 sec latency 33.287 ms 2024-03-19T02:35:41.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 253376 3.19 MB/sec execute 304 sec latency 27.813 ms 2024-03-19T02:35:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:41 smithi162 ceph-mon[35761]: pgmap v243: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:41 smithi154 ceph-mon[35693]: pgmap v243: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:41 smithi116 ceph-mon[32263]: pgmap v243: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:35:42.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 253934 3.18 MB/sec execute 305 sec latency 20.405 ms 2024-03-19T02:35:43.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 254488 3.18 MB/sec execute 306 sec latency 19.601 ms 2024-03-19T02:35:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:43 smithi162 ceph-mon[35761]: pgmap v244: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:35:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:43 smithi154 ceph-mon[35693]: pgmap v244: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:35:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:43 smithi116 ceph-mon[32263]: pgmap v244: 129 pgs: 129 active+clean; 240 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:35:44.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 255198 3.19 MB/sec execute 307 sec latency 19.470 ms 2024-03-19T02:35:45.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 255696 3.19 MB/sec execute 308 sec latency 17.990 ms 2024-03-19T02:35:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:45 smithi162 ceph-mon[35761]: pgmap v245: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:35:46.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:45 smithi154 ceph-mon[35693]: pgmap v245: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:35:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:45 smithi116 ceph-mon[32263]: pgmap v245: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:35:46.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 256096 3.18 MB/sec execute 309 sec latency 32.716 ms 2024-03-19T02:35:47.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 256672 3.18 MB/sec execute 310 sec latency 22.425 ms 2024-03-19T02:35:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:47 smithi162 ceph-mon[35761]: pgmap v246: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:47 smithi154 ceph-mon[35693]: pgmap v246: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:47 smithi116 ceph-mon[32263]: pgmap v246: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:48.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:35:48.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:35:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 -- 172.21.15.116:0/4273676515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54100690 msgr2=0x7fbf54100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 --2- 172.21.15.116:0/4273676515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54100690 0x7fbf54100b10 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009930 tx=0x7fbf3c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 -- 172.21.15.116:0/4273676515 shutdown_connections 2024-03-19T02:35:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 --2- 172.21.15.116:0/4273676515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf54101050 0x7fbf5410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 --2- 172.21.15.116:0/4273676515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54100690 0x7fbf54100b10 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 --2- 172.21.15.116:0/4273676515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 0x7fbf54106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 -- 172.21.15.116:0/4273676515 >> 172.21.15.116:0/4273676515 conn(0x7fbf54077820 msgr2=0x7fbf54077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 -- 172.21.15.116:0/4273676515 shutdown_connections 2024-03-19T02:35:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.740+0000 7fbf58831640 1 -- 172.21.15.116:0/4273676515 wait complete. 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 Processor -- start 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 -- start start 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf54100690 0x7fbf541980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 0x7fbf5419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf54110980 con 0x7fbf54101050 2024-03-19T02:35:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf54110800 con 0x7fbf541066a0 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf58831640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf54110b00 con 0x7fbf54100690 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35040/0 (socket says 172.21.15.116:35040) 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf4bfff640 1 -- 172.21.15.116:0/2161781355 learned_addr learned my addr 172.21.15.116:0/2161781355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.741+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 0x7fbf5419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 -- 172.21.15.116:0/2161781355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf54100690 msgr2=0x7fbf541980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf54100690 0x7fbf541980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 -- 172.21.15.116:0/2161781355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 msgr2=0x7fbf5419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 0x7fbf5419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 -- 172.21.15.116:0/2161781355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5419d090 con 0x7fbf54101050 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009230 tx=0x7fbf3c02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4b7fe640 1 -- 172.21.15.116:0/2161781355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c043020 con 0x7fbf54101050 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5419d320 con 0x7fbf54101050 2024-03-19T02:35:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4b7fe640 1 -- 172.21.15.116:0/2161781355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf3c02caf0 con 0x7fbf54101050 2024-03-19T02:35:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf4b7fe640 1 -- 172.21.15.116:0/2161781355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c003ae0 con 0x7fbf54101050 2024-03-19T02:35:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.742+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf541a4c10 con 0x7fbf54101050 2024-03-19T02:35:48.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.744+0000 7fbf4b7fe640 1 -- 172.21.15.116:0/2161781355 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fbf3c046020 con 0x7fbf54101050 2024-03-19T02:35:48.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.744+0000 7fbf5b418740 1 Processor -- start 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 -- start start 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf4b7fe640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 0x7fbf400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996f08f30 0x562996f0b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562996e63500 con 0x562996f00350 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562996e0bbe0 con 0x562996f08f30 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5b418740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562996e5f5a0 con 0x562996effa30 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 0x7fbf400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf4b7fe640 1 -- 172.21.15.116:0/2161781355 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fbf3c0f1240 con 0x7fbf54101050 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58514/0 (socket says 172.21.15.116:58514) 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 -- 172.21.15.116:0/423396040 learned_addr learned my addr 172.21.15.116:0/423396040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996f08f30 0x562996f0b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 -- 172.21.15.116:0/423396040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996f08f30 msgr2=0x562996f0b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996f08f30 0x562996f0b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 -- 172.21.15.116:0/423396040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 msgr2=0x562996f089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf5259c640 1 -- 172.21.15.116:0/423396040 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562996e2d000 con 0x562996effa30 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.745+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 0x7fbf400a4e30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c005fd0 tx=0x7fbf4c011bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.746+0000 7fbf5259c640 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbf440097a0 tx=0x7fbf44009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0f117b464d4ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.746+0000 7fbf48ff9640 1 -- 172.21.15.116:0/423396040 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44015020 con 0x562996effa30 2024-03-19T02:35:48.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.746+0000 7fbf48ff9640 1 -- 172.21.15.116:0/423396040 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf44011420 con 0x562996effa30 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.746+0000 7fbf48ff9640 1 -- 172.21.15.116:0/423396040 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf440105e0 con 0x562996effa30 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- 172.21.15.116:0/423396040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 msgr2=0x562996effe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbf440097a0 tx=0x7fbf44009c60 comp rx=0 tx=0).stop 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- 172.21.15.116:0/423396040 shutdown_connections 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996f08f30 0x562996f0b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- 172.21.15.116:0/423396040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996effa30 0x562996effe10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- 172.21.15.116:0/423396040 >> 172.21.15.116:0/423396040 conn(0x562996efa160 msgr2=0x562996efa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- 172.21.15.116:0/423396040 shutdown_connections 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- 172.21.15.116:0/423396040 wait complete. 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- start start 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 Processor -- start 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f06320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562996f0bd20 con 0x562996f08f30 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562996e1f160 con 0x562996effa30 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562996f0bea0 con 0x562996f00350 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5b418740 10 client.0 _ll_register_callbacks cb 0x562996e5ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35056/0 (socket says 172.21.15.116:35056) 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f06320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43644/0 (socket says 172.21.15.116:43644) 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.747+0000 7fbf52d9d640 1 -- 172.21.15.116:0/4195077545 learned_addr learned my addr 172.21.15.116:0/4195077545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 -- 172.21.15.116:0/4195077545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 msgr2=0x562996f06320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f06320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 -- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 msgr2=0x562996f05de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 -- 172.21.15.116:0/4195077545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562996c417d0 con 0x562996f08f30 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf5259c640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00ff10 tx=0x7fbf4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12270b51ba148ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c024070 con 0x562996f08f30 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf44011420 con 0x562996f08f30 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fbf4c028600 con 0x562996f08f30 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbf4c01a040 con 0x562996f08f30 2024-03-19T02:35:48.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf5b418740 10 client.17916 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562996b773f0 con 0x562996f08f30 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.748+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf44011420 con 0x562996f08f30 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fbf4c01d030 con 0x562996f08f30 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 10 client.17916 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 10 client.17916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 20 client.17916 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:48.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f19190 0x562996f1b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 4 client.17916 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562996e70a40 con 0x562996f19190 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5259c640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f19190 0x562996f1b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf39ffb640 10 client.17916 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.749+0000 7fbf5259c640 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f19190 0x562996f1b5e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f90362aeb69120d server_cookie=904182143882354b in_seq=0 out_seq=0 2024-03-19T02:35:48.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf39ffb640 1 -- 172.21.15.116:0/4195077545 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562996e70a40 con 0x562996f19190 2024-03-19T02:35:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf39ffb640 10 client.17916 handle_command_reply: tid=0 2024-03-19T02:35:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 client.17916 shutdown 2024-03-19T02:35:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 msgr2=0x562996f06860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00ff10 tx=0x7fbf4c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 shutdown_connections 2024-03-19T02:35:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f19190 0x562996f1b5e0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f06860 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f06320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf39ffb640 0 client.17916 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 --2- 172.21.15.116:0/4195077545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f05de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 >> 172.21.15.116:0/4195077545 conn(0x562996efa160 msgr2=0x562996efbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 shutdown_connections 2024-03-19T02:35:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 1 -- 172.21.15.116:0/4195077545 wait complete. 2024-03-19T02:35:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.750+0000 7fbf5b418740 20 client.17916 trim_cache size 0 max 16384 2024-03-19T02:35:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 Processor -- start 2024-03-19T02:35:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 -- start start 2024-03-19T02:35:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f08510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f08f30 0x562996efbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541aecf0 con 0x562996f00350 2024-03-19T02:35:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541b0830 con 0x562996effa30 2024-03-19T02:35:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.768+0000 7fbf5b418740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541af3c0 con 0x562996f08f30 2024-03-19T02:35:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35070/0 (socket says 172.21.15.116:35070) 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 -- 172.21.15.116:0/3771105921 learned_addr learned my addr 172.21.15.116:0/3771105921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f08510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f08f30 0x562996efbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 -- 172.21.15.116:0/3771105921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f08f30 msgr2=0x562996efbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f08f30 0x562996efbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 -- 172.21.15.116:0/3771105921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 msgr2=0x562996f08510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f08510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 -- 172.21.15.116:0/3771105921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562996b370f0 con 0x562996f00350 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf4bfff640 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c005ec0 tx=0x7fbf3c041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ec389a8d411bed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf3bfff640 1 -- 172.21.15.116:0/3771105921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0bd5d0 con 0x562996f00350 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf3bfff640 1 -- 172.21.15.116:0/3771105921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf3c0c1720 con 0x562996f00350 2024-03-19T02:35:48.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.769+0000 7fbf3bfff640 1 -- 172.21.15.116:0/3771105921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0b9d90 con 0x562996f00350 2024-03-19T02:35:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- 172.21.15.116:0/3771105921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 msgr2=0x562996f14ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c005ec0 tx=0x7fbf3c041040 comp rx=0 tx=0).stop 2024-03-19T02:35:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- 172.21.15.116:0/3771105921 shutdown_connections 2024-03-19T02:35:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f08f30 0x562996efbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f00350 0x562996f14ac0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- 172.21.15.116:0/3771105921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f08510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- 172.21.15.116:0/3771105921 >> 172.21.15.116:0/3771105921 conn(0x562996efa160 msgr2=0x562996efb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- 172.21.15.116:0/3771105921 shutdown_connections 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- 172.21.15.116:0/3771105921 wait complete. 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 -- start start 2024-03-19T02:35:48.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 Processor -- start 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f112e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.770+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f025b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541b0160 con 0x562996f08f30 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541afa90 con 0x562996effa30 2024-03-19T02:35:48.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf541accf0 con 0x562996f00350 2024-03-19T02:35:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 10 client.0 _ll_register_callbacks cb 0x7fbf4c0114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35076/0 (socket says 172.21.15.116:35076) 2024-03-19T02:35:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf4bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f025b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58538/0 (socket says 172.21.15.116:58538) 2024-03-19T02:35:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 -- 172.21.15.116:0/2533097123 learned_addr learned my addr 172.21.15.116:0/2533097123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 -- 172.21.15.116:0/2533097123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 msgr2=0x562996f025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f025b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 -- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 msgr2=0x562996f112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f112e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 -- 172.21.15.116:0/2533097123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562996f03190 con 0x562996f08f30 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5259c640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf52d9d640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c028dd0 tx=0x7fbf4c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867a3f46edb9f502 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:48.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c024040 con 0x562996f08f30 2024-03-19T02:35:48.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf3c0c1720 con 0x562996f08f30 2024-03-19T02:35:48.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fbf4c028600 con 0x562996f08f30 2024-03-19T02:35:48.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbf4c046050 con 0x562996f08f30 2024-03-19T02:35:48.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 10 client.17922 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:48.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.771+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562996f837b0 con 0x562996f08f30 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf3c0c1720 con 0x562996f08f30 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fbf4c01d040 con 0x562996f08f30 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 10 client.17922 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 10 client.17922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 20 client.17922 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:48.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f83fb0 0x562996f863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 4 client.17922 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:35:48.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562996e6ad90 con 0x562996f83fb0 2024-03-19T02:35:48.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5259c640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f83fb0 0x562996f863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf5259c640 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f83fb0 0x562996f863d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466eb41782296423 server_cookie=c6bbb1ebfe508e8f in_seq=0 out_seq=0 2024-03-19T02:35:48.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.772+0000 7fbf39ffb640 10 client.17922 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf39ffb640 1 -- 172.21.15.116:0/2533097123 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562996e6ad90 con 0x562996f83fb0 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf39ffb640 10 client.17922 handle_command_reply: tid=0 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 client.17922 shutdown 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 msgr2=0x562996f02af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c028dd0 tx=0x7fbf4c003ba0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 shutdown_connections 2024-03-19T02:35:48.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562996f83fb0 0x562996f863d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562996f08f30 0x562996f02af0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562996f00350 0x562996f025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf39ffb640 0 client.17922 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:48.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 --2- 172.21.15.116:0/2533097123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562996effa30 0x562996f112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 >> 172.21.15.116:0/2533097123 conn(0x562996efa160 msgr2=0x562996efbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 shutdown_connections 2024-03-19T02:35:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 1 -- 172.21.15.116:0/2533097123 wait complete. 2024-03-19T02:35:48.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.774+0000 7fbf5b418740 20 client.17922 trim_cache size 0 max 16384 2024-03-19T02:35:48.797 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:35:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:35:48.797 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:48.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 msgr2=0x7fbf400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 0x7fbf400a4e30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c005fd0 tx=0x7fbf4c011bc0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 msgr2=0x7fbf54198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009230 tx=0x7fbf3c02c710 comp rx=0 tx=0).stop 2024-03-19T02:35:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 shutdown_connections 2024-03-19T02:35:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbf400a2970 0x7fbf400a4e30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf541066a0 0x7fbf5419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf54101050 0x7fbf54198630 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 --2- 172.21.15.116:0/2161781355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf54100690 0x7fbf541980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 >> 172.21.15.116:0/2161781355 conn(0x7fbf54077820 msgr2=0x7fbf54102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 shutdown_connections 2024-03-19T02:35:48.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.775+0000 7fbf58831640 1 -- 172.21.15.116:0/2161781355 wait complete. 2024-03-19T02:35:48.802 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:35:48.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:35:48.803 INFO:tasks.workunit.client.0.smithi116.stdout: 1 257199 3.18 MB/sec execute 311 sec latency 34.693 ms 2024-03-19T02:35:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 -- 172.21.15.116:0/1167413174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6450105600 msgr2=0x7f64501059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 --2- 172.21.15.116:0/1167413174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6450105600 0x7f64501059e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f644c009900 tx=0x7f644c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 -- 172.21.15.116:0/1167413174 shutdown_connections 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 --2- 172.21.15.116:0/1167413174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64501010c0 0x7f645010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 --2- 172.21.15.116:0/1167413174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 --2- 172.21.15.116:0/1167413174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6450105600 0x7f64501059e0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 -- 172.21.15.116:0/1167413174 >> 172.21.15.116:0/1167413174 conn(0x7f6450077850 msgr2=0x7f6450077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 -- 172.21.15.116:0/1167413174 shutdown_connections 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.913+0000 7f6458630640 1 -- 172.21.15.116:0/1167413174 wait complete. 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 Processor -- start 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 -- start start 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6450105600 0x7f645019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6450110960 con 0x7f64501010c0 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64501107e0 con 0x7f6450100720 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6458630640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6450110ae0 con 0x7f6450105600 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6455bcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6455bcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35094/0 (socket says 172.21.15.116:35094) 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.914+0000 7f6455bcb640 1 -- 172.21.15.116:0/1925266082 learned_addr learned my addr 172.21.15.116:0/1925266082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f6456bcd640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6450105600 0x7f645019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 -- 172.21.15.116:0/1925266082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6450105600 msgr2=0x7f645019b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6450105600 0x7f645019b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 -- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 msgr2=0x7f6450197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 -- 172.21.15.116:0/1925266082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645019beb0 con 0x7f6450100720 2024-03-19T02:35:48.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f6455bcb640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.915+0000 7f64563cc640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450196ec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f644c036040 tx=0x7f644c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.916+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c043020 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.916+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f644c02dbe0 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.916+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c0038e0 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.916+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645019c140 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.916+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6450069b60 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.917+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6450067d80 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.918+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f644c046020 con 0x7f6450100720 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.918+0000 7f64477fe640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 0x7f643c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.919+0000 7f6455bcb640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 0x7f643c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.919+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f644c0f1030 con 0x7f6450100720 2024-03-19T02:35:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.919+0000 7f6455bcb640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 0x7f643c0a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6450077f10 tx=0x7f6440010040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:48.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:48.924+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f644c04d020 con 0x7f6450100720 2024-03-19T02:35:49.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.169+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64501086d0 con 0x7f6450100720 2024-03-19T02:35:49.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.170+0000 7f64477fe640 1 -- 172.21.15.116:0/1925266082 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f644c0bda60 con 0x7f6450100720 2024-03-19T02:35:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:35:49.171 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:35:49.171 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 msgr2=0x7f643c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 0x7f643c0a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6450077f10 tx=0x7f6440010040 comp rx=0 tx=0).stop 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 msgr2=0x7f6450196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450196ec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f644c036040 tx=0x7f644c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 shutdown_connections 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f643c0a2970 0x7f643c0a4e30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6450105600 0x7f645019b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64501010c0 0x7f6450197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 --2- 172.21.15.116:0/1925266082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6450100720 0x7f6450196ec0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 >> 172.21.15.116:0/1925266082 conn(0x7f6450077850 msgr2=0x7f6450103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 shutdown_connections 2024-03-19T02:35:49.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.172+0000 7f6458630640 1 -- 172.21.15.116:0/1925266082 wait complete. 2024-03-19T02:35:49.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:35:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 -- 172.21.15.116:0/700759750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 msgr2=0x7f0634100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 --2- 172.21.15.116:0/700759750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634100b20 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0624009930 tx=0x7f062402bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 -- 172.21.15.116:0/700759750 shutdown_connections 2024-03-19T02:35:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 --2- 172.21.15.116:0/700759750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 0x7f063410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 --2- 172.21.15.116:0/700759750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634100b20 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 --2- 172.21.15.116:0/700759750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 0x7f0634106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 -- 172.21.15.116:0/700759750 >> 172.21.15.116:0/700759750 conn(0x7f06340778d0 msgr2=0x7f0634077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 -- 172.21.15.116:0/700759750 shutdown_connections 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.302+0000 7f063b8f7640 1 -- 172.21.15.116:0/700759750 wait complete. 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 Processor -- start 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 -- start start 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 0x7f0634199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 0x7f06341a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0634110970 con 0x7f06341006a0 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06341107f0 con 0x7f06341066b0 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f063b8f7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0634110af0 con 0x7f0634101060 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35114/0 (socket says 172.21.15.116:35114) 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 -- 172.21.15.116:0/67877397 learned_addr learned my addr 172.21.15.116:0/67877397 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0638e92640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 0x7f0634199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639e94640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 0x7f06341a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 -- 172.21.15.116:0/67877397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 msgr2=0x7f0634199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 0x7f0634199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 -- 172.21.15.116:0/67877397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 msgr2=0x7f06341a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 0x7f06341a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.303+0000 7f0639693640 1 -- 172.21.15.116:0/67877397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06341a43b0 con 0x7f06341006a0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f0639693640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f063000e9f0 tx=0x7f063000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f062a7fc640 1 -- 172.21.15.116:0/67877397 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063000cd80 con 0x7f06341006a0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f062a7fc640 1 -- 172.21.15.116:0/67877397 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06300108e0 con 0x7f06341006a0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06341a4580 con 0x7f06341006a0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f062a7fc640 1 -- 172.21.15.116:0/67877397 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630018920 con 0x7f06341006a0 2024-03-19T02:35:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.304+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341a4aa0 con 0x7f06341006a0 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.306+0000 7f062a7fc640 1 -- 172.21.15.116:0/67877397 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0630010a90 con 0x7f06341006a0 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.306+0000 7f063e5b6740 1 Processor -- start 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 -- start start 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 0x56010bbbfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56010bb174b0 con 0x56010bbb42d0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56010babfd70 con 0x56010bbbd7d0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f063e5b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56010bb13550 con 0x56010bbb4bf0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f062a7fc640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 0x7f06200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0638e92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 0x56010bbbfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35126/0 (socket says 172.21.15.116:35126) 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 -- 172.21.15.116:0/286327912 learned_addr learned my addr 172.21.15.116:0/286327912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f062a7fc640 1 -- 172.21.15.116:0/67877397 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f06300cf4f0 con 0x7f06341006a0 2024-03-19T02:35:49.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639693640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 0x7f06200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 -- 172.21.15.116:0/286327912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 msgr2=0x56010bbbd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 -- 172.21.15.116:0/286327912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 msgr2=0x56010bbbfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 0x56010bbbfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639e94640 1 -- 172.21.15.116:0/286327912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010bae2210 con 0x56010bbb42d0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0639693640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.307+0000 7f0638e92640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 0x56010bbbfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f0639e94640 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f062c0099e0 tx=0x7f062c011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67aa27d2099376e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f061f7fe640 1 -- 172.21.15.116:0/286327912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c022020 con 0x56010bbb42d0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f061f7fe640 1 -- 172.21.15.116:0/286327912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f062c0114c0 con 0x56010bbb42d0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f061f7fe640 1 -- 172.21.15.116:0/286327912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c01dbc0 con 0x56010bbb42d0 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f063e5b6740 1 -- 172.21.15.116:0/286327912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 msgr2=0x56010bbb46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.308+0000 7f0639e94640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 0x7f06200a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f062c023000 tx=0x7f062c01f000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f062c0099e0 tx=0x7f062c011e00 comp rx=0 tx=0).stop 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 -- 172.21.15.116:0/286327912 shutdown_connections 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbbd7d0 0x56010bbbfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- 172.21.15.116:0/286327912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb42d0 0x56010bbb46b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 -- 172.21.15.116:0/286327912 >> 172.21.15.116:0/286327912 conn(0x56010bbaea00 msgr2=0x56010bbaede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 -- 172.21.15.116:0/286327912 shutdown_connections 2024-03-19T02:35:49.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 -- 172.21.15.116:0/286327912 wait complete. 2024-03-19T02:35:49.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 -- start start 2024-03-19T02:35:49.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 Processor -- start 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbba7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.309+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbbd7d0 0x56010bbbb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56010bbc05c0 con 0x56010bbb4bf0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56010bad3310 con 0x56010bbb42d0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56010bbc0740 con 0x56010bbbd7d0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 10 client.0 _ll_register_callbacks cb 0x56010bac72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0638e92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbba7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0638e92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35132/0 (socket says 172.21.15.116:35132) 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0638e92640 1 -- 172.21.15.116:0/1383275618 learned_addr learned my addr 172.21.15.116:0/1383275618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639e94640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbbd7d0 0x56010bbbb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 -- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbbd7d0 msgr2=0x56010bbbb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbbd7d0 0x56010bbbb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 -- 172.21.15.116:0/1383275618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 msgr2=0x56010bbbad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 -- 172.21.15.116:0/1383275618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010b82b3f0 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0638e92640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f0639693640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbba7e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06300cc830 tx=0x7f063009b270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cd64156395ee835 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f061cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06300a3f10 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f062c0114c0 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.310+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f06300cf150 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06300cf350 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 10 client.27746 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56010b8f57d0 con 0x56010bbb42d0 2024-03-19T02:35:49.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f062c0114c0 con 0x56010bbb42d0 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0630097600 con 0x56010bbb42d0 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 10 client.27746 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 10 client.27746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 20 client.27746 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bbcda40 0x56010bbcfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 4 client.27746 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56010bb753e0 con 0x56010bbcda40 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.311+0000 7f0638e92640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bbcda40 0x56010bbcfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f061cff9640 10 client.27746 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f0638e92640 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bbcda40 0x56010bbcfe90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=160ebf376c1d04d2 server_cookie=353ef5ba1bf1efae in_seq=0 out_seq=0 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f061cff9640 1 -- 172.21.15.116:0/1383275618 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56010bb753e0 con 0x56010bbcda40 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f061cff9640 10 client.27746 handle_command_reply: tid=0 2024-03-19T02:35:49.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 client.27746 shutdown 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 msgr2=0x56010bbba7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbba7e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06300cc830 tx=0x7f063009b270 comp rx=0 tx=0).stop 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 shutdown_connections 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bbcda40 0x56010bbcfe90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbbd7d0 0x56010bbbb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f061cff9640 0 client.27746 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbb4bf0 0x56010bbbad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.312+0000 7f063e5b6740 1 --2- 172.21.15.116:0/1383275618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbba7e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.313+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 >> 172.21.15.116:0/1383275618 conn(0x56010bbaea00 msgr2=0x56010bbb0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.313+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 shutdown_connections 2024-03-19T02:35:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.313+0000 7f063e5b6740 1 -- 172.21.15.116:0/1383275618 wait complete. 2024-03-19T02:35:49.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.313+0000 7f063e5b6740 20 client.27746 trim_cache size 0 max 16384 2024-03-19T02:35:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 Processor -- start 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 -- start start 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbd5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06341aec80 con 0x56010bbbd7d0 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06341b07c0 con 0x56010bbb42d0 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f063e5b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06341af350 con 0x56010bbb4bf0 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0639e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbd5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0639693640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58596/0 (socket says 172.21.15.116:58596) 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 -- 172.21.15.116:0/3103004773 learned_addr learned my addr 172.21.15.116:0/3103004773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 -- 172.21.15.116:0/3103004773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 msgr2=0x56010bbc9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 -- 172.21.15.116:0/3103004773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 msgr2=0x56010bbd5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbd5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 -- 172.21.15.116:0/3103004773 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010bbaf5d0 con 0x56010bbb4bf0 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0639693640 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.328+0000 7f0638e92640 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0624036040 tx=0x7f0624005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ea5d3b2b773198 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f061effd640 1 -- 172.21.15.116:0/3103004773 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062402c7e0 con 0x56010bbb4bf0 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f061effd640 1 -- 172.21.15.116:0/3103004773 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0624005040 con 0x56010bbb4bf0 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f061effd640 1 -- 172.21.15.116:0/3103004773 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06240048e0 con 0x56010bbb4bf0 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 -- 172.21.15.116:0/3103004773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 msgr2=0x56010bbbb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0624036040 tx=0x7f0624005850 comp rx=0 tx=0).stop 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 -- 172.21.15.116:0/3103004773 shutdown_connections 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbd5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbbb080 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 --2- 172.21.15.116:0/3103004773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 -- 172.21.15.116:0/3103004773 >> 172.21.15.116:0/3103004773 conn(0x56010bbaea00 msgr2=0x56010bbadba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 -- 172.21.15.116:0/3103004773 shutdown_connections 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.329+0000 7f063e5b6740 1 -- 172.21.15.116:0/3103004773 wait complete. 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 -- start start 2024-03-19T02:35:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 Processor -- start 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbb7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06341b00f0 con 0x56010bbbd7d0 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06341afa20 con 0x56010bbb42d0 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06341acc80 con 0x56010bbb4bf0 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f063e5b6740 10 client.0 _ll_register_callbacks cb 0x7f063000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f0639e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f0639e94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35150/0 (socket says 172.21.15.116:35150) 2024-03-19T02:35:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f0639693640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f0639e94640 1 -- 172.21.15.116:0/559017675 learned_addr learned my addr 172.21.15.116:0/559017675 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.330+0000 7f0638e92640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbb7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 -- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 msgr2=0x56010bbb7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbb7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 -- 172.21.15.116:0/559017675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 msgr2=0x56010bbb7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 -- 172.21.15.116:0/559017675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56010bbb7c30 con 0x56010bbb42d0 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0638e92640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbb7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639e94640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f0639693640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc5b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f063009b7b0 tx=0x7f0630097ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d674337d73f3873e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f061cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:35:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063009b360 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0624005040 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f06300afd60 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0630010430 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f063e5b6740 10 client.27752 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56010bc33c00 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.331+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0624005040 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f063009a6e0 con 0x56010bbb42d0 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 10 client.27752 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 10 client.27752 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 20 client.27752 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bc34140 0x56010bc36530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 4 client.27752 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56010bb249f0 con 0x56010bc34140 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f0638e92640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bc34140 0x56010bc36530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f0638e92640 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bc34140 0x56010bc36530 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1cb325c3781c541a server_cookie=20608fb8c5759957 in_seq=0 out_seq=0 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f061cff9640 10 client.27752 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f061cff9640 1 -- 172.21.15.116:0/559017675 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56010bb249f0 con 0x56010bc34140 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f061cff9640 10 client.27752 handle_command_reply: tid=0 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.332+0000 7f063e5b6740 1 client.27752 shutdown 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 msgr2=0x56010bbc5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc5b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f063009b7b0 tx=0x7f0630097ac0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 shutdown_connections 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56010bc34140 0x56010bc36530 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56010bbbd7d0 0x56010bbb7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56010bbb4bf0 0x56010bbb7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 --2- 172.21.15.116:0/559017675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56010bbb42d0 0x56010bbc5b80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f061cff9640 0 client.27752 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 >> 172.21.15.116:0/559017675 conn(0x56010bbaea00 msgr2=0x56010bbafd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 shutdown_connections 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 1 -- 172.21.15.116:0/559017675 wait complete. 2024-03-19T02:35:49.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063e5b6740 20 client.27752 trim_cache size 0 max 16384 2024-03-19T02:35:49.346 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 msgr2=0x7f06200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 0x7f06200a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f062c023000 tx=0x7f062c01f000 comp rx=0 tx=0).stop 2024-03-19T02:35:49.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 msgr2=0x7f0634198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f063000e9f0 tx=0x7f063000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 shutdown_connections 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06200a2970 0x7f06200a4e30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341066b0 0x7f06341a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0634101060 0x7f0634199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 --2- 172.21.15.116:0/67877397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06341006a0 0x7f0634198df0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 >> 172.21.15.116:0/67877397 conn(0x7f06340778d0 msgr2=0x7f0634102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 shutdown_connections 2024-03-19T02:35:49.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.333+0000 7f063b8f7640 1 -- 172.21.15.116:0/67877397 wait complete. 2024-03-19T02:35:49.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:35:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- 172.21.15.116:0/415499823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 msgr2=0x7f38a010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 --2- 172.21.15.116:0/415499823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a010e220 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3898009ca0 tx=0x7f389802c180 comp rx=0 tx=0).stop 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- 172.21.15.116:0/415499823 shutdown_connections 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 --2- 172.21.15.116:0/415499823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a010e220 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 --2- 172.21.15.116:0/415499823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 0x7f38a010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 --2- 172.21.15.116:0/415499823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 0x7f38a0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- 172.21.15.116:0/415499823 >> 172.21.15.116:0/415499823 conn(0x7f38a00778d0 msgr2=0x7f38a0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- 172.21.15.116:0/415499823 shutdown_connections 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- 172.21.15.116:0/415499823 wait complete. 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 Processor -- start 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.461+0000 7f38a7962640 1 -- start start 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 0x7f38a0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 0x7f38a01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0111a90 con 0x7f38a010bd80 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0111910 con 0x7f38a0103360 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a7962640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0111c10 con 0x7f38a0102a40 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35164/0 (socket says 172.21.15.116:35164) 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 -- 172.21.15.116:0/206721823 learned_addr learned my addr 172.21.15.116:0/206721823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a56fe640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 0x7f38a0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a4efd640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 0x7f38a01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 -- 172.21.15.116:0/206721823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 msgr2=0x7f38a0197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 0x7f38a0197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 -- 172.21.15.116:0/206721823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 msgr2=0x7f38a01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 0x7f38a01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 -- 172.21.15.116:0/206721823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a019cfa0 con 0x7f38a010bd80 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38a5eff640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f389802c790 tx=0x7f3898035ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.462+0000 7f38967fc640 1 -- 172.21.15.116:0/206721823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898039040 con 0x7f38a010bd80 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.463+0000 7f38967fc640 1 -- 172.21.15.116:0/206721823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3898044070 con 0x7f38a010bd80 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.463+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a019d230 con 0x7f38a010bd80 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.463+0000 7f38967fc640 1 -- 172.21.15.116:0/206721823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898032850 con 0x7f38a010bd80 2024-03-19T02:35:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.463+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a01a4b20 con 0x7f38a010bd80 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.465+0000 7f38967fc640 1 -- 172.21.15.116:0/206721823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f38980329f0 con 0x7f38a010bd80 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.465+0000 7f38aa62d740 1 Processor -- start 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.465+0000 7f38aa62d740 1 -- start start 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed793e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed86be0 0x555a9ed88fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555a9ecdc4b0 con 0x555a9ed79920 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555a9ec84d70 con 0x555a9ed86be0 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38aa62d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a9ecd8550 con 0x555a9ed79000 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35174/0 (socket says 172.21.15.116:35174) 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 -- 172.21.15.116:0/1032218081 learned_addr learned my addr 172.21.15.116:0/1032218081 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed793e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed86be0 0x555a9ed88fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38967fc640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 0x7f388c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 -- 172.21.15.116:0/1032218081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 msgr2=0x555a9ed793e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed793e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 -- 172.21.15.116:0/1032218081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed86be0 msgr2=0x555a9ed88fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed86be0 0x555a9ed88fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 -- 172.21.15.116:0/1032218081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a9eca7210 con 0x555a9ed79920 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a5eff640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 0x7f388c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38967fc640 1 -- 172.21.15.116:0/206721823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f38980f14b0 con 0x7f38a010bd80 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f389c009d20 tx=0x7f389c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19fca087c9572ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.466+0000 7f388b7fe640 1 -- 172.21.15.116:0/1032218081 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c01e070 con 0x555a9ed79920 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f388b7fe640 1 -- 172.21.15.116:0/1032218081 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f389c009e10 con 0x555a9ed79920 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f388b7fe640 1 -- 172.21.15.116:0/1032218081 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c018870 con 0x555a9ed79920 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38a5eff640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 0x7f388c0a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f38980eea80 tx=0x7f38980b8040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 -- 172.21.15.116:0/1032218081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 msgr2=0x555a9ed866a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f389c009d20 tx=0x7f389c011810 comp rx=0 tx=0).stop 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 -- 172.21.15.116:0/1032218081 shutdown_connections 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed86be0 0x555a9ed88fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79920 0x555a9ed866a0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1032218081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed793e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 -- 172.21.15.116:0/1032218081 >> 172.21.15.116:0/1032218081 conn(0x555a9ed73a00 msgr2=0x555a9ed73de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 -- 172.21.15.116:0/1032218081 shutdown_connections 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.467+0000 7f38aa62d740 1 -- 172.21.15.116:0/1032218081 wait complete. 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 -- start start 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 Processor -- start 2024-03-19T02:35:49.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed83c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed84170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555a9ed899a0 con 0x555a9ed86be0 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555a9ec98310 con 0x555a9ed79920 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a9ed89b20 con 0x555a9ed79000 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38aa62d740 10 client.0 _ll_register_callbacks cb 0x555a9ec8c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35184/0 (socket says 172.21.15.116:35184) 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38a4efd640 1 -- 172.21.15.116:0/2391203045 learned_addr learned my addr 172.21.15.116:0/2391203045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38a56fe640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed84170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.468+0000 7f38a4efd640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed83c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 -- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 msgr2=0x555a9ed83c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed83c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 -- 172.21.15.116:0/2391203045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 msgr2=0x555a9ed84170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed84170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 -- 172.21.15.116:0/2391203045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a9e9f03f0 con 0x555a9ed86be0 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a56fe640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed84170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38a4efd640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f389001aef0 tx=0x7f3890008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2c26dda6a84cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389001a0d0 con 0x555a9ed86be0 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f389001a270 con 0x555a9ed86be0 2024-03-19T02:35:49.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f389c009e10 con 0x555a9ed86be0 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f389002b050 con 0x555a9ed86be0 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38aa62d740 10 client.17958 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a9eaba7d0 con 0x555a9ed86be0 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f389c009e10 con 0x555a9ed86be0 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.469+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3890026420 con 0x555a9ed86be0 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 10 client.17958 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 10 client.17958 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 20 client.17958 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9ed97290 0x555a9ed99680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 4 client.17958 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a9ed3a3e0 con 0x555a9ed97290 2024-03-19T02:35:49.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38a56fe640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9ed97290 0x555a9ed99680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f3888ff9640 10 client.17958 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.470+0000 7f38a56fe640 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9ed97290 0x555a9ed99680 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=94931ab039306c77 server_cookie=52529cb41212c312 in_seq=0 out_seq=0 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f3888ff9640 1 -- 172.21.15.116:0/2391203045 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a9ed3a3e0 con 0x555a9ed97290 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f3888ff9640 10 client.17958 handle_command_reply: tid=0 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 client.17958 shutdown 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 msgr2=0x555a9ed846b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f389001aef0 tx=0x7f3890008040 comp rx=0 tx=0).stop 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 shutdown_connections 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9ed97290 0x555a9ed99680 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed846b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f3888ff9640 0 client.17958 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed84170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 --2- 172.21.15.116:0/2391203045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79000 0x555a9ed83c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 >> 172.21.15.116:0/2391203045 conn(0x555a9ed73a00 msgr2=0x555a9ed75750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 shutdown_connections 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 1 -- 172.21.15.116:0/2391203045 wait complete. 2024-03-19T02:35:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.471+0000 7f38aa62d740 20 client.17958 trim_cache size 0 max 16384 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 Processor -- start 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 -- start start 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed9ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed86be0 0x555a9ed8edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ac9c0 con 0x555a9ed79000 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ae500 con 0x555a9ed79920 2024-03-19T02:35:49.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38aa62d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ad090 con 0x555a9ed86be0 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35190/0 (socket says 172.21.15.116:35190) 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a4efd640 1 -- 172.21.15.116:0/4164638651 learned_addr learned my addr 172.21.15.116:0/4164638651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed9ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a4efd640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed86be0 0x555a9ed8edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 -- 172.21.15.116:0/4164638651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed86be0 msgr2=0x555a9ed8edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed86be0 0x555a9ed8edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 -- 172.21.15.116:0/4164638651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 msgr2=0x555a9ed927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 -- 172.21.15.116:0/4164638651 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a9ed92cf0 con 0x555a9ed79920 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a4efd640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.485+0000 7f38a56fe640 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed9ece0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f389c01c040 tx=0x7f389c009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486a474d51365422 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f388affd640 1 -- 172.21.15.116:0/4164638651 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c01e040 con 0x555a9ed79920 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f388affd640 1 -- 172.21.15.116:0/4164638651 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f389c02a740 con 0x555a9ed79920 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f388affd640 1 -- 172.21.15.116:0/4164638651 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c005620 con 0x555a9ed79920 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 -- 172.21.15.116:0/4164638651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 msgr2=0x555a9ed9ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed9ece0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f389c01c040 tx=0x7f389c009fa0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 -- 172.21.15.116:0/4164638651 shutdown_connections 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed86be0 0x555a9ed8edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79920 0x555a9ed9ece0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 --2- 172.21.15.116:0/4164638651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed79000 0x555a9ed927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 -- 172.21.15.116:0/4164638651 >> 172.21.15.116:0/4164638651 conn(0x555a9ed73a00 msgr2=0x555a9ed83bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 -- 172.21.15.116:0/4164638651 shutdown_connections 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.486+0000 7f38aa62d740 1 -- 172.21.15.116:0/4164638651 wait complete. 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 -- start start 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 Processor -- start 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79000 0x555a9ed80980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79920 0x555a9ed80ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ade30 con 0x555a9ed86be0 2024-03-19T02:35:49.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01ad760 con 0x555a9ed79000 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01aa9c0 con 0x555a9ed79920 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38aa62d740 10 client.0 _ll_register_callbacks cb 0x7f3890009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38a4efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35198/0 (socket says 172.21.15.116:35198) 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38a4efd640 1 -- 172.21.15.116:0/1398173476 learned_addr learned my addr 172.21.15.116:0/1398173476 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79000 0x555a9ed80980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.487+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79920 0x555a9ed80ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 -- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79920 msgr2=0x555a9ed80ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79920 0x555a9ed80ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 -- 172.21.15.116:0/1398173476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79000 msgr2=0x555a9ed80980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79000 0x555a9ed80980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 -- 172.21.15.116:0/1398173476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a9edfc470 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38a4efd640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3890009950 tx=0x7f3890009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5206d603357ec09a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389001a2e0 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f38900269c0 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f389c02a740 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3890026bc0 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 10 client.17964 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a9edfd6f0 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f389c02a740 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3890018b00 con 0x555a9ed86be0 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 10 client.17964 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 10 client.17964 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 20 client.17964 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9edfdd50 0x555a9ee00140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 4 client.17964 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.488+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a9ece99f0 con 0x555a9edfdd50 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9edfdd50 0x555a9ee00140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f3888ff9640 10 client.17964 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38a56fe640 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9edfdd50 0x555a9ee00140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=33761c115081c9a5 server_cookie=8d9f3510e82f66ba in_seq=0 out_seq=0 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f3888ff9640 1 -- 172.21.15.116:0/1398173476 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a9ece99f0 con 0x555a9edfdd50 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f3888ff9640 10 client.17964 handle_command_reply: tid=0 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 client.17964 shutdown 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 msgr2=0x555a9ed81400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3890009950 tx=0x7f3890009e00 comp rx=0 tx=0).stop 2024-03-19T02:35:49.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 shutdown_connections 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x555a9edfdd50 0x555a9ee00140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a9ed86be0 0x555a9ed81400 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a9ed79920 0x555a9ed80ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f3888ff9640 0 client.17964 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 --2- 172.21.15.116:0/1398173476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a9ed79000 0x555a9ed80980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 >> 172.21.15.116:0/1398173476 conn(0x555a9ed73a00 msgr2=0x555a9ed84480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 shutdown_connections 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.489+0000 7f38aa62d740 1 -- 172.21.15.116:0/1398173476 wait complete. 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38aa62d740 20 client.17964 trim_cache size 0 max 16384 2024-03-19T02:35:49.499 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:49.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 msgr2=0x7f388c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 0x7f388c0a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f38980eea80 tx=0x7f38980b8040 comp rx=0 tx=0).stop 2024-03-19T02:35:49.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 msgr2=0x7f38a019c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f389802c790 tx=0x7f3898035ea0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 shutdown_connections 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f388c0a2970 0x7f388c0a4e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38a010bd80 0x7f38a019c870 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38a0103360 0x7f38a01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 --2- 172.21.15.116:0/206721823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38a0102a40 0x7f38a0197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 >> 172.21.15.116:0/206721823 conn(0x7f38a00778d0 msgr2=0x7f38a00fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 shutdown_connections 2024-03-19T02:35:49.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.490+0000 7f38a7962640 1 -- 172.21.15.116:0/206721823 wait complete. 2024-03-19T02:35:49.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:35:49.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 -- 172.21.15.116:0/709748324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901066b0 msgr2=0x7f6b90106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 --2- 172.21.15.116:0/709748324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901066b0 0x7f6b90106a90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009930 tx=0x7f6b8002bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 -- 172.21.15.116:0/709748324 shutdown_connections 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 --2- 172.21.15.116:0/709748324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b90101060 0x7f6b9010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 --2- 172.21.15.116:0/709748324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b901006a0 0x7f6b90100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 --2- 172.21.15.116:0/709748324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901066b0 0x7f6b90106a90 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 -- 172.21.15.116:0/709748324 >> 172.21.15.116:0/709748324 conn(0x7f6b900778d0 msgr2=0x7f6b90077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 -- 172.21.15.116:0/709748324 shutdown_connections 2024-03-19T02:35:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 -- 172.21.15.116:0/709748324 wait complete. 2024-03-19T02:35:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.617+0000 7f6b9488c640 1 Processor -- start 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 -- start start 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101060 0x7f6b90199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901066b0 0x7f6b901a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110970 con 0x7f6b901006a0 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901107f0 con 0x7f6b901066b0 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110af0 con 0x7f6b90101060 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35206/0 (socket says 172.21.15.116:35206) 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 -- 172.21.15.116:0/3237814426 learned_addr learned my addr 172.21.15.116:0/3237814426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b87fff640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101060 0x7f6b90199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901066b0 0x7f6b901a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 -- 172.21.15.116:0/3237814426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101060 msgr2=0x7f6b90199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101060 0x7f6b90199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 -- 172.21.15.116:0/3237814426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901066b0 msgr2=0x7f6b901a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901066b0 0x7f6b901a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 -- 172.21.15.116:0/3237814426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b901a43f0 con 0x7f6b901006a0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009230 tx=0x7f6b8002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b877fe640 1 -- 172.21.15.116:0/3237814426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80043020 con 0x7f6b901006a0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b877fe640 1 -- 172.21.15.116:0/3237814426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b8002ca70 con 0x7f6b901006a0 2024-03-19T02:35:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b901a45c0 con 0x7f6b901006a0 2024-03-19T02:35:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.618+0000 7f6b877fe640 1 -- 172.21.15.116:0/3237814426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80003970 con 0x7f6b901006a0 2024-03-19T02:35:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.619+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b901a4ab0 con 0x7f6b901006a0 2024-03-19T02:35:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.621+0000 7f6b877fe640 1 -- 172.21.15.116:0/3237814426 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6b80046020 con 0x7f6b901006a0 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.621+0000 7f6b9747e740 1 Processor -- start 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.621+0000 7f6b9747e740 1 -- start start 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.621+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 0x55eead5f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead5f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b9747e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eead54a4b0 con 0x55eead5f4be0 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b9747e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eead4f2d70 con 0x55eead5e7920 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b9747e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eead546550 con 0x55eead5e7000 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b877fe640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 0x7f6b7c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 0x55eead5f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58686/0 (socket says 172.21.15.116:58686) 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8ed9d640 1 -- 172.21.15.116:0/2629395614 learned_addr learned my addr 172.21.15.116:0/2629395614 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b87fff640 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead5f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 0x7f6b7c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b877fe640 1 -- 172.21.15.116:0/3237814426 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f6b800f11a0 con 0x7f6b901006a0 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 -- 172.21.15.116:0/2629395614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 msgr2=0x55eead5e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 -- 172.21.15.116:0/2629395614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 msgr2=0x55eead5f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead5f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 -- 172.21.15.116:0/2629395614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eead515210 con 0x55eead5e7920 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.622+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 0x55eead5f46a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6b800ee620 tx=0x7f6b8002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d773b7987e698912 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 0x7f6b7c0a4e30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fd0 tx=0x7f6b88004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b84ff9640 1 -- 172.21.15.116:0/2629395614 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b800c1680 con 0x55eead5e7920 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b84ff9640 1 -- 172.21.15.116:0/2629395614 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b800bdc30 con 0x55eead5e7920 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b84ff9640 1 -- 172.21.15.116:0/2629395614 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b800bd790 con 0x55eead5e7920 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 -- 172.21.15.116:0/2629395614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 msgr2=0x55eead5f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 0x55eead5f46a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6b800ee620 tx=0x7f6b8002bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 -- 172.21.15.116:0/2629395614 shutdown_connections 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead5f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7920 0x55eead5f46a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2629395614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7000 0x55eead5e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 -- 172.21.15.116:0/2629395614 >> 172.21.15.116:0/2629395614 conn(0x55eead5e1a00 msgr2=0x55eead5e1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.623+0000 7f6b9747e740 1 -- 172.21.15.116:0/2629395614 shutdown_connections 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 -- 172.21.15.116:0/2629395614 wait complete. 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 -- start start 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 Processor -- start 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead5f2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eead4f2d70 con 0x55eead5e7920 2024-03-19T02:35:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eead506310 con 0x55eead5e7000 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eead5f7760 con 0x55eead5f4be0 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b9747e740 10 client.0 _ll_register_callbacks cb 0x55eead4fa2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35230/0 (socket says 172.21.15.116:35230) 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 -- 172.21.15.116:0/4028063291 learned_addr learned my addr 172.21.15.116:0/4028063291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead5f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 -- 172.21.15.116:0/4028063291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 msgr2=0x55eead5f2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead5f2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 -- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 msgr2=0x55eead5f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.624+0000 7f6b87fff640 1 -- 172.21.15.116:0/4028063291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eead25e3f0 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b87fff640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f6b780126e0 tx=0x7f6b78012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9cd2a43122e441 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800f670 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f6b7800f810 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b800bdc30 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6b7800fa10 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 10 client.17976 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55eead3287d0 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b800bdc30 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6b78010c10 con 0x55eead5e7920 2024-03-19T02:35:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 10 client.17976 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 10 client.17976 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 20 client.17976 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead604c70 0x55eead6070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 4 client.17976 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.625+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eead5a83e0 con 0x55eead604c70 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.626+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead604c70 0x55eead6070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.626+0000 7f6b75ffb640 10 client.17976 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.626+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead604c70 0x55eead6070c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=116b7cfb8a0726b8 server_cookie=f375284b53801430 in_seq=0 out_seq=0 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b75ffb640 1 -- 172.21.15.116:0/4028063291 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eead5a83e0 con 0x55eead604c70 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b75ffb640 10 client.17976 handle_command_reply: tid=0 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 client.17976 shutdown 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 msgr2=0x55eead5f21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f6b780126e0 tx=0x7f6b78012ae0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 shutdown_connections 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead604c70 0x55eead6070c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead5f2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5f21c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b75ffb640 0 client.17976 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 --2- 172.21.15.116:0/4028063291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.627+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 >> 172.21.15.116:0/4028063291 conn(0x55eead5e1a00 msgr2=0x55eead5e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.628+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 shutdown_connections 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.628+0000 7f6b9747e740 1 -- 172.21.15.116:0/4028063291 wait complete. 2024-03-19T02:35:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.628+0000 7f6b9747e740 20 client.17976 trim_cache size 0 max 16384 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.641+0000 7f6b9747e740 1 Processor -- start 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.641+0000 7f6b9747e740 1 -- start start 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7920 0x55eead6003c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901aca40 con 0x55eead5f4be0 2024-03-19T02:35:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901ae580 con 0x55eead5e7000 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b9747e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901ad110 con 0x55eead5e7920 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35232/0 (socket says 172.21.15.116:35232) 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 -- 172.21.15.116:0/2014490025 learned_addr learned my addr 172.21.15.116:0/2014490025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b87fff640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7920 0x55eead6003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 -- 172.21.15.116:0/2014490025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7920 msgr2=0x55eead6003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7920 0x55eead6003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 -- 172.21.15.116:0/2014490025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 msgr2=0x55eead5f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 -- 172.21.15.116:0/2014490025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eead5f01f0 con 0x55eead5f4be0 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6b88012390 tx=0x7f6b88015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6200c624bd4074 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b77fff640 1 -- 172.21.15.116:0/2014490025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88004d20 con 0x55eead5f4be0 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b77fff640 1 -- 172.21.15.116:0/2014490025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b88005020 con 0x55eead5f4be0 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.642+0000 7f6b77fff640 1 -- 172.21.15.116:0/2014490025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88010e60 con 0x55eead5f4be0 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- 172.21.15.116:0/2014490025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 msgr2=0x55eead60c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6b88012390 tx=0x7f6b88015070 comp rx=0 tx=0).stop 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- 172.21.15.116:0/2014490025 shutdown_connections 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5f4be0 0x55eead60c9d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5e7920 0x55eead6003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 --2- 172.21.15.116:0/2014490025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- 172.21.15.116:0/2014490025 >> 172.21.15.116:0/2014490025 conn(0x55eead5e1a00 msgr2=0x55eead5dca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- 172.21.15.116:0/2014490025 shutdown_connections 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- 172.21.15.116:0/2014490025 wait complete. 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.643+0000 7f6b9747e740 1 -- start start 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 Processor -- start 2024-03-19T02:35:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead66a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901adeb0 con 0x55eead5e7920 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901ad7e0 con 0x55eead5e7000 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901aaa40 con 0x55eead5f4be0 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b9747e740 10 client.0 _ll_register_callbacks cb 0x7f6b78009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35238/0 (socket says 172.21.15.116:35238) 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 -- 172.21.15.116:0/1761647482 learned_addr learned my addr 172.21.15.116:0/1761647482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b8ed9d640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead66a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 -- 172.21.15.116:0/1761647482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 msgr2=0x55eead66a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead66a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 -- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 msgr2=0x55eead5ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 -- 172.21.15.116:0/1761647482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eead66a5c0 con 0x55eead5e7920 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b87fff640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6b78012420 tx=0x7f6b7800fe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f53d84575ea8390 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.644+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7802f060 con 0x55eead5e7920 2024-03-19T02:35:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b88005020 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f6b7801c8a0 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6b7801cae0 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 10 client.17982 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55eead66b8f0 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b88005020 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6b78003710 con 0x55eead5e7920 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 10 client.17982 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 10 client.17982 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 20 client.17982 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead66c0a0 0x55eead66e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 4 client.17982 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eead5579f0 con 0x55eead66c0a0 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.645+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead66c0a0 0x55eead66e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b8e59c640 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead66c0a0 0x55eead66e4c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea5e0eaee0b78e20 server_cookie=bf41d123770e7022 in_seq=0 out_seq=0 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b75ffb640 10 client.17982 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b75ffb640 1 -- 172.21.15.116:0/1761647482 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eead5579f0 con 0x55eead66c0a0 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b75ffb640 10 client.17982 handle_command_reply: tid=0 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 client.17982 shutdown 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 msgr2=0x55eead5ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f6b78012420 tx=0x7f6b7800fe60 comp rx=0 tx=0).stop 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 shutdown_connections 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55eead66c0a0 0x55eead66e4c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eead5f4be0 0x55eead66a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b75ffb640 0 client.17982 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eead5e7920 0x55eead5ef570 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 --2- 172.21.15.116:0/1761647482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eead5e7000 0x55eead5ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 >> 172.21.15.116:0/1761647482 conn(0x55eead5e1a00 msgr2=0x55eead5e0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 shutdown_connections 2024-03-19T02:35:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 1 -- 172.21.15.116:0/1761647482 wait complete. 2024-03-19T02:35:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.646+0000 7f6b9747e740 20 client.17982 trim_cache size 0 max 16384 2024-03-19T02:35:49.655 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 msgr2=0x7f6b7c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 0x7f6b7c0a4e30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fd0 tx=0x7f6b88004ab0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 msgr2=0x7f6b90198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009230 tx=0x7f6b8002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 shutdown_connections 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b7c0a2970 0x7f6b7c0a4e30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b901066b0 0x7f6b901a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101060 0x7f6b90199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 --2- 172.21.15.116:0/3237814426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b901006a0 0x7f6b90198e30 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 >> 172.21.15.116:0/3237814426 conn(0x7f6b900778d0 msgr2=0x7f6b901030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 shutdown_connections 2024-03-19T02:35:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.647+0000 7f6b9488c640 1 -- 172.21.15.116:0/3237814426 wait complete. 2024-03-19T02:35:49.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:35:49.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 -- 172.21.15.116:0/4001913906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4910102170 msgr2=0x7f491010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 --2- 172.21.15.116:0/4001913906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4910102170 0x7f491010e680 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f4908009cd0 tx=0x7f490802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 -- 172.21.15.116:0/4001913906 shutdown_connections 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 --2- 172.21.15.116:0/4001913906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4910102170 0x7f491010e680 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 --2- 172.21.15.116:0/4001913906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49101017b0 0x7f4910101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 --2- 172.21.15.116:0/4001913906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101077c0 0x7f4910107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 -- 172.21.15.116:0/4001913906 >> 172.21.15.116:0/4001913906 conn(0x7f49100fd2e0 msgr2=0x7f49100ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 -- 172.21.15.116:0/4001913906 shutdown_connections 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.771+0000 7f49171b9640 1 -- 172.21.15.116:0/4001913906 wait complete. 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 Processor -- start 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 -- start start 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 0x7f4910197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4910102170 0x7f49101984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4910111a80 con 0x7f49101077c0 2024-03-19T02:35:49.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4910111900 con 0x7f49101017b0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f49171b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4910111c00 con 0x7f4910102170 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4914f55640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 0x7f4910197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f490ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4910102170 0x7f49101984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35258/0 (socket says 172.21.15.116:35258) 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4914f55640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 0x7f4910197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:43868/0 (socket says 172.21.15.116:43868) 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 -- 172.21.15.116:0/585511536 learned_addr learned my addr 172.21.15.116:0/585511536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 -- 172.21.15.116:0/585511536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4910102170 msgr2=0x7f49101984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4910102170 0x7f49101984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 -- 172.21.15.116:0/585511536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 msgr2=0x7f4910197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 0x7f4910197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 -- 172.21.15.116:0/585511536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491019cf70 con 0x7f49101077c0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.772+0000 7f4915756640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4908004af0 tx=0x7f4908035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.773+0000 7f490dffb640 1 -- 172.21.15.116:0/585511536 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908039040 con 0x7f49101077c0 2024-03-19T02:35:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.773+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f491019d200 con 0x7f49101077c0 2024-03-19T02:35:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.773+0000 7f490dffb640 1 -- 172.21.15.116:0/585511536 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4908044070 con 0x7f49101077c0 2024-03-19T02:35:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.773+0000 7f490dffb640 1 -- 172.21.15.116:0/585511536 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49080042a0 con 0x7f49101077c0 2024-03-19T02:35:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.773+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49101a4af0 con 0x7f49101077c0 2024-03-19T02:35:49.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.775+0000 7f490dffb640 1 -- 172.21.15.116:0/585511536 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4908003970 con 0x7f49101077c0 2024-03-19T02:35:49.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.775+0000 7f4919da2740 1 Processor -- start 2024-03-19T02:35:49.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.775+0000 7f4919da2740 1 -- start start 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f490dffb640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 0x7f49000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f490ffff640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 0x7f49000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce316290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce318bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce2704b0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f490dffb640 1 -- 172.21.15.116:0/585511536 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f49080f14a0 con 0x7f49101077c0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce218d70 con 0x55e8ce3167d0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4919da2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce26c550 con 0x55e8ce30dbf0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35272/0 (socket says 172.21.15.116:35272) 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 -- 172.21.15.116:0/1079727592 learned_addr learned my addr 172.21.15.116:0/1079727592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f490ffff640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce316290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4915756640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce318bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 -- 172.21.15.116:0/1079727592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 msgr2=0x55e8ce316290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce316290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 -- 172.21.15.116:0/1079727592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 msgr2=0x55e8ce318bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce318bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 -- 172.21.15.116:0/1079727592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ce23b210 con 0x55e8ce30d2d0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4915756640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce318bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f490ffff640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 0x7f49000a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49100ff940 tx=0x7f48fc011920 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.776+0000 7f4914f55640 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4904009bf0 tx=0x7f4904011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28fba2fa9d4e494 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f48faffd640 1 -- 172.21.15.116:0/1079727592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904009e60 con 0x55e8ce30d2d0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f48faffd640 1 -- 172.21.15.116:0/1079727592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4904004230 con 0x55e8ce30d2d0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f48faffd640 1 -- 172.21.15.116:0/1079727592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904019640 con 0x55e8ce30d2d0 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 -- 172.21.15.116:0/1079727592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 msgr2=0x55e8ce30d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4904009bf0 tx=0x7f4904011810 comp rx=0 tx=0).stop 2024-03-19T02:35:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 -- 172.21.15.116:0/1079727592 shutdown_connections 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce318bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce316290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 --2- 172.21.15.116:0/1079727592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce30d6b0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 -- 172.21.15.116:0/1079727592 >> 172.21.15.116:0/1079727592 conn(0x55e8ce307a00 msgr2=0x55e8ce307de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 -- 172.21.15.116:0/1079727592 shutdown_connections 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.777+0000 7f4919da2740 1 -- 172.21.15.116:0/1079727592 wait complete. 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 -- start start 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 Processor -- start 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce313cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce314200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce3195c0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce22c310 con 0x55e8ce3167d0 2024-03-19T02:35:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e8ce319740 con 0x55e8ce30dbf0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4919da2740 10 client.0 _ll_register_callbacks cb 0x55e8ce2202d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35276/0 (socket says 172.21.15.116:35276) 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f4914f55640 1 -- 172.21.15.116:0/2749685447 learned_addr learned my addr 172.21.15.116:0/2749685447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.778+0000 7f490ffff640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce313cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4915756640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce314200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 -- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 msgr2=0x55e8ce313cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce313cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 -- 172.21.15.116:0/2749685447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 msgr2=0x55e8ce314200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce314200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 -- 172.21.15.116:0/2749685447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8cdf843f0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4914f55640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4904003ac0 tx=0x7f4904003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2753e3594e048b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f48effff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904027070 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f49040043e0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4904004230 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4904033050 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4919da2740 10 client.17994 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e8ce04e7d0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.779+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4904004230 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4904018070 con 0x55e8ce30d2d0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 10 client.17994 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 10 client.17994 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 20 client.17994 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce3269c0 0x55e8ce328db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 4 client.17994 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8ce2ce3e0 con 0x55e8ce3269c0 2024-03-19T02:35:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f490ffff640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce3269c0 0x55e8ce328db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f48effff640 10 client.17994 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.780+0000 7f490ffff640 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce3269c0 0x55e8ce328db0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57e17a5e48cd1b8f server_cookie=dc3b31f6f3b148ca in_seq=0 out_seq=0 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.781+0000 7f48effff640 1 -- 172.21.15.116:0/2749685447 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8ce2ce3e0 con 0x55e8ce3269c0 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.781+0000 7f48effff640 10 client.17994 handle_command_reply: tid=0 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.781+0000 7f4919da2740 1 client.17994 shutdown 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 msgr2=0x55e8ce313780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4904003ac0 tx=0x7f4904003af0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 shutdown_connections 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce3269c0 0x55e8ce328db0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce314200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f48effff640 0 client.17994 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce313cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 --2- 172.21.15.116:0/2749685447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce313780 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 >> 172.21.15.116:0/2749685447 conn(0x55e8ce307a00 msgr2=0x55e8ce309750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 shutdown_connections 2024-03-19T02:35:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 1 -- 172.21.15.116:0/2749685447 wait complete. 2024-03-19T02:35:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.782+0000 7f4919da2740 20 client.17994 trim_cache size 0 max 16384 2024-03-19T02:35:49.789 INFO:tasks.workunit.client.0.smithi116.stdout: 1 257785 3.17 MB/sec execute 312 sec latency 21.812 ms 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 Processor -- start 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 -- start start 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce30dbf0 0x55e8ce322360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49101acb30 con 0x55e8ce3167d0 2024-03-19T02:35:49.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49101ae670 con 0x55e8ce30dbf0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.796+0000 7f4919da2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49101ad200 con 0x55e8ce30d2d0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4914f55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35286/0 (socket says 172.21.15.116:35286) 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4914f55640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58778/0 (socket says 172.21.15.116:58778) 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 -- 172.21.15.116:0/1677934480 learned_addr learned my addr 172.21.15.116:0/1677934480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f490ffff640 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce30dbf0 0x55e8ce322360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 -- 172.21.15.116:0/1677934480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 msgr2=0x55e8ce306de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 -- 172.21.15.116:0/1677934480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce30dbf0 msgr2=0x55e8ce322360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce30dbf0 0x55e8ce322360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 -- 172.21.15.116:0/1677934480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ce2dd880 con 0x55e8ce3167d0 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4914f55640 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f4915756640 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f49080b9d30 tx=0x7f49080329c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d099129620c587d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f48fa7fc640 1 -- 172.21.15.116:0/1677934480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908034700 con 0x55e8ce3167d0 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f48fa7fc640 1 -- 172.21.15.116:0/1677934480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49080098d0 con 0x55e8ce3167d0 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.797+0000 7f48fa7fc640 1 -- 172.21.15.116:0/1677934480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49080bbc60 con 0x55e8ce3167d0 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- 172.21.15.116:0/1677934480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 msgr2=0x55e8ce314020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f49080b9d30 tx=0x7f49080329c0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- 172.21.15.116:0/1677934480 shutdown_connections 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce3167d0 0x55e8ce314020 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce30dbf0 0x55e8ce322360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 --2- 172.21.15.116:0/1677934480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30d2d0 0x55e8ce306de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- 172.21.15.116:0/1677934480 >> 172.21.15.116:0/1677934480 conn(0x55e8ce307a00 msgr2=0x55e8ce3096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- 172.21.15.116:0/1677934480 shutdown_connections 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- 172.21.15.116:0/1677934480 wait complete. 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 -- start start 2024-03-19T02:35:49.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.798+0000 7f4919da2740 1 Processor -- start 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce319f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce31a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49101adfa0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49101ad8d0 con 0x55e8ce3167d0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49101aab30 con 0x55e8ce30dbf0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 10 client.0 _ll_register_callbacks cb 0x7f4908005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35296/0 (socket says 172.21.15.116:35296) 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 -- 172.21.15.116:0/2730785924 learned_addr learned my addr 172.21.15.116:0/2730785924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f490ffff640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce319f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4915756640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce31a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 -- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 msgr2=0x55e8ce319f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce319f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 -- 172.21.15.116:0/2730785924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 msgr2=0x55e8ce31a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce31a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 -- 172.21.15.116:0/2730785924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ce31ac70 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4914f55640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f490401ae60 tx=0x7f4904005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6297e85e3e4da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f48effff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904027030 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f4904009da0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.799+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e8ce27d9f0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4904005280 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 10 client.18000 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e8ce38cba0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e8ce27d9f0 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4904018070 con 0x55e8ce30d2d0 2024-03-19T02:35:49.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 10 client.18000 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 10 client.18000 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 20 client.18000 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce38d2c0 0x55e8ce38f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 4 client.18000 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8ce277d40 con 0x55e8ce38d2c0 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f490ffff640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce38d2c0 0x55e8ce38f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.800+0000 7f48effff640 10 client.18000 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f490ffff640 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce38d2c0 0x55e8ce38f6e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d08c9a903bc7758 server_cookie=954f1a1c233c44ca in_seq=0 out_seq=0 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f48effff640 1 -- 172.21.15.116:0/2730785924 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8ce277d40 con 0x55e8ce38d2c0 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f48effff640 10 client.18000 handle_command_reply: tid=0 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 client.18000 shutdown 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 msgr2=0x55e8ce31eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f490401ae60 tx=0x7f4904005f10 comp rx=0 tx=0).stop 2024-03-19T02:35:49.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 shutdown_connections 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e8ce38d2c0 0x55e8ce38f6e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e8ce3167d0 0x55e8ce31a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e8ce30dbf0 0x55e8ce319f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 --2- 172.21.15.116:0/2730785924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8ce30d2d0 0x55e8ce31eb70 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 >> 172.21.15.116:0/2730785924 conn(0x55e8ce307a00 msgr2=0x55e8ce308a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f48effff640 0 client.18000 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 shutdown_connections 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 1 -- 172.21.15.116:0/2730785924 wait complete. 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.801+0000 7f4919da2740 20 client.18000 trim_cache size 0 max 16384 2024-03-19T02:35:49.808 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:49.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 msgr2=0x7f49000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 0x7f49000a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49100ff940 tx=0x7f48fc011920 comp rx=0 tx=0).stop 2024-03-19T02:35:49.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 msgr2=0x7f491019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4908004af0 tx=0x7f4908035ce0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 shutdown_connections 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f49000a2970 0x7f49000a4e30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49101077c0 0x7f491019c840 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4910102170 0x7f49101984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 --2- 172.21.15.116:0/585511536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49101017b0 0x7f4910197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 >> 172.21.15.116:0/585511536 conn(0x7f49100fd2e0 msgr2=0x7f49100fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 shutdown_connections 2024-03-19T02:35:49.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.802+0000 7f49171b9640 1 -- 172.21.15.116:0/585511536 wait complete. 2024-03-19T02:35:49.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:35:49.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 -- 172.21.15.116:0/1623509830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784101f90 msgr2=0x7fa78410e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 --2- 172.21.15.116:0/1623509830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784101f90 0x7fa78410e450 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009ca0 tx=0x7fa77c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 -- 172.21.15.116:0/1623509830 shutdown_connections 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 --2- 172.21.15.116:0/1623509830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784101f90 0x7fa78410e450 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 --2- 172.21.15.116:0/1623509830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7841015d0 0x7fa784101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 --2- 172.21.15.116:0/1623509830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa784107690 0x7fa784107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 -- 172.21.15.116:0/1623509830 >> 172.21.15.116:0/1623509830 conn(0x7fa7840fd0c0 msgr2=0x7fa7840ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 -- 172.21.15.116:0/1623509830 shutdown_connections 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.926+0000 7fa78a39d640 1 -- 172.21.15.116:0/1623509830 wait complete. 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 Processor -- start 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 -- start start 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7841015d0 0x7fa78419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 0x7fa78419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa784111880 con 0x7fa784107690 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa784111700 con 0x7fa7841015d0 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78a39d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa784111a00 con 0x7fa784101f90 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa789b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa789b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35302/0 (socket says 172.21.15.116:35302) 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa789b9c640 1 -- 172.21.15.116:0/3447740151 learned_addr learned my addr 172.21.15.116:0/3447740151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa788b9a640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 0x7fa78419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.927+0000 7fa78939b640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7841015d0 0x7fa78419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 -- 172.21.15.116:0/3447740151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 msgr2=0x7fa78419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 0x7fa78419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 -- 172.21.15.116:0/3447740151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7841015d0 msgr2=0x7fa78419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7841015d0 0x7fa78419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 -- 172.21.15.116:0/3447740151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa784198200 con 0x7fa784107690 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa788b9a640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 0x7fa78419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa789b9c640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fa77c02c700 tx=0x7fa77c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa77a7fc640 1 -- 172.21.15.116:0/3447740151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c039040 con 0x7fa784107690 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa784198490 con 0x7fa784107690 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa77a7fc640 1 -- 172.21.15.116:0/3447740151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa77c044070 con 0x7fa784107690 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa77a7fc640 1 -- 172.21.15.116:0/3447740151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0042a0 con 0x7fa784107690 2024-03-19T02:35:49.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.928+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7841a4790 con 0x7fa784107690 2024-03-19T02:35:49.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.930+0000 7fa77a7fc640 1 -- 172.21.15.116:0/3447740151 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa77c004440 con 0x7fa784107690 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 Processor -- start 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 -- start start 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380bb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e23801e4b0 con 0x55e2380bbdd0 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e237fc6d70 con 0x55e2380bb4b0 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e23801a550 con 0x55e2380c49b0 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa77a7fc640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 0x7fa7700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78939b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78939b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35304/0 (socket says 172.21.15.116:35304) 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa78939b640 1 -- 172.21.15.116:0/265123635 learned_addr learned my addr 172.21.15.116:0/265123635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa788b9a640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa788b9a640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa789b9c640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 0x7fa7700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.931+0000 7fa77a7fc640 1 -- 172.21.15.116:0/3447740151 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fa77c0f14d0 con 0x7fa784107690 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 -- 172.21.15.116:0/265123635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 msgr2=0x55e2380bb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380bb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 -- 172.21.15.116:0/265123635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 msgr2=0x55e2380c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 -- 172.21.15.116:0/265123635 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e237fe9210 con 0x55e2380c49b0 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa78939b640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa789b9c640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 0x7fa7700a4e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0eeaa0 tx=0x7fa77c02c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa788b9a640 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c6da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa784106d20 tx=0x7fa774005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d68c7cb1686e4c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa76f7fe640 1 -- 172.21.15.116:0/265123635 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774005020 con 0x55e2380c49b0 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa76f7fe640 1 -- 172.21.15.116:0/265123635 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa77400f850 con 0x55e2380c49b0 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.932+0000 7fa76f7fe640 1 -- 172.21.15.116:0/265123635 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774004ae0 con 0x55e2380c49b0 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- 172.21.15.116:0/265123635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 msgr2=0x55e2380c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c6da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa784106d20 tx=0x7fa774005850 comp rx=0 tx=0).stop 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- 172.21.15.116:0/265123635 shutdown_connections 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c6da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/265123635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380bb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- 172.21.15.116:0/265123635 >> 172.21.15.116:0/265123635 conn(0x55e2380b5be0 msgr2=0x55e2380b5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- 172.21.15.116:0/265123635 shutdown_connections 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- 172.21.15.116:0/265123635 wait complete. 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 -- start start 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.933+0000 7fa78e1ec740 1 Processor -- start 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e2380c77a0 con 0x55e2380bbdd0 2024-03-19T02:35:49.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e237fda310 con 0x55e2380bb4b0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2380c7920 con 0x55e2380c49b0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78e1ec740 10 client.0 _ll_register_callbacks cb 0x55e237fce2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35308/0 (socket says 172.21.15.116:35308) 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 -- 172.21.15.116:0/1710937075 learned_addr learned my addr 172.21.15.116:0/1710937075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa788b9a640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa788b9a640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 -- 172.21.15.116:0/1710937075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 msgr2=0x55e2380c24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 -- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 msgr2=0x55e2380c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380c1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 -- 172.21.15.116:0/1710937075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e237d323f0 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa788b9a640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa78939b640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa780012740 tx=0x7fa780012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c04c3ca69bc595 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa76cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78000f920 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.934+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fa78000fac0 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa77400f850 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa78000fcc0 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 10 client.18012 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e237dfc7d0 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa77400f850 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa78001c420 con 0x55e2380bbdd0 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 10 client.18012 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 10 client.18012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 20 client.18012 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e2380d72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 4 client.18012 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e23807c3e0 con 0x55e2380d4e70 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.935+0000 7fa788b9a640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e2380d72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.936+0000 7fa788b9a640 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e2380d72f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d136102223e576ea server_cookie=3a18b372eef87722 in_seq=0 out_seq=0 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.936+0000 7fa76cff9640 10 client.18012 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa76cff9640 1 -- 172.21.15.116:0/1710937075 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e23807c3e0 con 0x55e2380d4e70 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa76cff9640 10 client.18012 handle_command_reply: tid=0 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 client.18012 shutdown 2024-03-19T02:35:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 msgr2=0x55e2380c1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa780012740 tx=0x7fa780012c00 comp rx=0 tx=0).stop 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 shutdown_connections 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e2380d72f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa76cff9640 0 client.18012 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380c49b0 0x55e2380c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380bbdd0 0x55e2380c1f70 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/1710937075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bb4b0 0x55e2380c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 >> 172.21.15.116:0/1710937075 conn(0x55e2380b5be0 msgr2=0x55e2380b7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 shutdown_connections 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 1 -- 172.21.15.116:0/1710937075 wait complete. 2024-03-19T02:35:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.937+0000 7fa78e1ec740 20 client.18012 trim_cache size 0 max 16384 2024-03-19T02:35:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 Processor -- start 2024-03-19T02:35:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 -- start start 2024-03-19T02:35:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841ae920 con 0x55e2380c49b0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841b0460 con 0x55e2380bbdd0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841aeff0 con 0x55e2380bb4b0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa788b9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa788b9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35314/0 (socket says 172.21.15.116:35314) 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa788b9a640 1 -- 172.21.15.116:0/643369117 learned_addr learned my addr 172.21.15.116:0/643369117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa788b9a640 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa78939b640 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.951+0000 7fa788b9a640 1 -- 172.21.15.116:0/643369117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 msgr2=0x55e2380c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa788b9a640 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa788b9a640 1 -- 172.21.15.116:0/643369117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 msgr2=0x55e2380d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa788b9a640 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa788b9a640 1 -- 172.21.15.116:0/643369117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2380bff80 con 0x55e2380c49b0 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa788b9a640 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa7740057c0 tx=0x7fa7740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b77846a79898b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa76effd640 1 -- 172.21.15.116:0/643369117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77400f910 con 0x55e2380c49b0 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa76effd640 1 -- 172.21.15.116:0/643369117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa77402e800 con 0x55e2380c49b0 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.952+0000 7fa76effd640 1 -- 172.21.15.116:0/643369117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7740041c0 con 0x55e2380c49b0 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- 172.21.15.116:0/643369117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 msgr2=0x55e2380b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa7740057c0 tx=0x7fa7740046c0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- 172.21.15.116:0/643369117 shutdown_connections 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380b4b80 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/643369117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- 172.21.15.116:0/643369117 >> 172.21.15.116:0/643369117 conn(0x55e2380b5be0 msgr2=0x55e2380b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- 172.21.15.116:0/643369117 shutdown_connections 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- 172.21.15.116:0/643369117 wait complete. 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- start start 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 Processor -- start 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841afd90 con 0x55e2380c49b0 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841af6c0 con 0x55e2380bbdd0 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.953+0000 7fa78e1ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7841ac920 con 0x55e2380bb4b0 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78e1ec740 10 client.0 _ll_register_callbacks cb 0x7fa780009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:49.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa788b9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa788b9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35318/0 (socket says 172.21.15.116:35318) 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa788b9a640 1 -- 172.21.15.116:0/778773186 learned_addr learned my addr 172.21.15.116:0/778773186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa788b9a640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 -- 172.21.15.116:0/778773186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 msgr2=0x55e2380c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 -- 172.21.15.116:0/778773186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 msgr2=0x55e2380c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 -- 172.21.15.116:0/778773186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e23813e930 con 0x55e2380bbdd0 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa788b9a640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78939b640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380c8d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa78001c910 tx=0x7fa780009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad017ffdc71ac67 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa76cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78000fe60 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e23802b9f0 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7fa78001c630 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.954+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa7800050c0 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 10 client.27821 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e23813fc30 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e23802b9f0 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa780003b90 con 0x55e2380bbdd0 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 10 client.27821 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 10 client.27821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 20 client.27821 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e238142440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 4 client.27821 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e238025d40 con 0x55e2380d4e70 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa788b9a640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e238142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa788b9a640 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e238142440 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad53d6ee813aeb6c server_cookie=4b53f1ad3af0b6df in_seq=0 out_seq=0 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.955+0000 7fa76cff9640 10 client.27821 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa76cff9640 1 -- 172.21.15.116:0/778773186 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e238025d40 con 0x55e2380d4e70 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa76cff9640 10 client.27821 handle_command_reply: tid=0 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 client.27821 shutdown 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 msgr2=0x55e2380c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380c8d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa78001c910 tx=0x7fa780009fd0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 shutdown_connections 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2380d4e70 0x55e238142440 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2380c49b0 0x55e2380c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2380bbdd0 0x55e2380c8d00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa76cff9640 0 client.27821 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 --2- 172.21.15.116:0/778773186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2380bb4b0 0x55e2380c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.956+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 >> 172.21.15.116:0/778773186 conn(0x55e2380b5be0 msgr2=0x55e2380b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 shutdown_connections 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78e1ec740 1 -- 172.21.15.116:0/778773186 wait complete. 2024-03-19T02:35:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78e1ec740 20 client.27821 trim_cache size 0 max 16384 2024-03-19T02:35:49.966 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:35:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 msgr2=0x7fa7700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 0x7fa7700a4e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0eeaa0 tx=0x7fa77c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 msgr2=0x7fa784197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fa77c02c700 tx=0x7fa77c035d60 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 shutdown_connections 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7700a2970 0x7fa7700a4e30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa784107690 0x7fa784197a80 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa784101f90 0x7fa78419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 --2- 172.21.15.116:0/3447740151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7841015d0 0x7fa78419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 >> 172.21.15.116:0/3447740151 conn(0x7fa7840fd0c0 msgr2=0x7fa7840fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 shutdown_connections 2024-03-19T02:35:49.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:49.957+0000 7fa78a39d640 1 -- 172.21.15.116:0/3447740151 wait complete. 2024-03-19T02:35:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:49 smithi162 ceph-mon[35761]: pgmap v247: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:49 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1925266082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:49 smithi154 ceph-mon[35693]: pgmap v247: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:49 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1925266082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:49 smithi116 ceph-mon[32263]: pgmap v247: 129 pgs: 129 active+clean; 243 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-03-19T02:35:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:49 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1925266082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:35:50.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 258608 3.18 MB/sec execute 313 sec latency 17.935 ms 2024-03-19T02:35:50.969 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:35:50.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:35:50.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 -- 172.21.15.116:0/110694298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 msgr2=0x7ff634100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 --2- 172.21.15.116:0/110694298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634100aa0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff624009930 tx=0x7ff62402bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 -- 172.21.15.116:0/110694298 shutdown_connections 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 --2- 172.21.15.116:0/110694298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff63410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 --2- 172.21.15.116:0/110694298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634100aa0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 --2- 172.21.15.116:0/110694298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 0x7ff634106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 -- 172.21.15.116:0/110694298 >> 172.21.15.116:0/110694298 conn(0x7ff6340fc130 msgr2=0x7ff6340fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 -- 172.21.15.116:0/110694298 shutdown_connections 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.085+0000 7ff63a59d640 1 -- 172.21.15.116:0/110694298 wait complete. 2024-03-19T02:35:51.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 Processor -- start 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 -- start start 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff634199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 0x7ff6341a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341108f0 con 0x7ff634100620 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff634110770 con 0x7ff634100fe0 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63a59d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff634110a70 con 0x7ff634106630 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff634199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35334/0 (socket says 172.21.15.116:35334) 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 -- 172.21.15.116:0/417110856 learned_addr learned my addr 172.21.15.116:0/417110856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff639d9c640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 0x7ff6341a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 -- 172.21.15.116:0/417110856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 msgr2=0x7ff6341a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 0x7ff6341a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 -- 172.21.15.116:0/417110856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 msgr2=0x7ff634199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff634199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff63959b640 1 -- 172.21.15.116:0/417110856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6341a4380 con 0x7ff634100620 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.086+0000 7ff638d9a640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff634199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff63959b640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff63000e9f0 tx=0x7ff63000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff62a7fc640 1 -- 172.21.15.116:0/417110856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63000cd80 con 0x7ff634100620 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6341a4550 con 0x7ff634100620 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff62a7fc640 1 -- 172.21.15.116:0/417110856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6300054d0 con 0x7ff634100620 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff62a7fc640 1 -- 172.21.15.116:0/417110856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630010490 con 0x7ff634100620 2024-03-19T02:35:51.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.087+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6341a4a20 con 0x7ff634100620 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.089+0000 7ff62a7fc640 1 -- 172.21.15.116:0/417110856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff630005020 con 0x7ff634100620 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.089+0000 7ff63e3ef740 1 Processor -- start 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.089+0000 7ff63e3ef740 1 -- start start 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcde520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff62a7fc640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 0x7ff6200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bce0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558e0bc39160 con 0x558e0bcd5e80 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e0bbe1e10 con 0x558e0bcdea60 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e0bc35200 con 0x558e0bcd5560 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff63959b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcde520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff638d9a640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 0x7ff6200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58870/0 (socket says 172.21.15.116:58870) 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 -- 172.21.15.116:0/3385280039 learned_addr learned my addr 172.21.15.116:0/3385280039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff62a7fc640 1 -- 172.21.15.116:0/417110856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7ff6300cf530 con 0x7ff634100620 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bce0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 -- 172.21.15.116:0/3385280039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 msgr2=0x558e0bce0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bce0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 -- 172.21.15.116:0/3385280039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 msgr2=0x558e0bcde520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcde520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 -- 172.21.15.116:0/3385280039 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0ba157e0 con 0x558e0bcd5560 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff62c005f40 tx=0x7ff62c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4effdb564333c07d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff638d9a640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 0x7ff6200a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff624005ab0 tx=0x7ff624005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.090+0000 7ff61f7fe640 1 -- 172.21.15.116:0/3385280039 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c004b20 con 0x558e0bcd5560 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff61f7fe640 1 -- 172.21.15.116:0/3385280039 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff62c00fd30 con 0x558e0bcd5560 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff61f7fe640 1 -- 172.21.15.116:0/3385280039 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0056c0 con 0x558e0bcd5560 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3385280039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 msgr2=0x558e0bcd5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff62c005f40 tx=0x7ff62c009b80 comp rx=0 tx=0).stop 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3385280039 shutdown_connections 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bce0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcde520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3385280039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcd5940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3385280039 >> 172.21.15.116:0/3385280039 conn(0x558e0bccfc40 msgr2=0x558e0bcd0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3385280039 shutdown_connections 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.091+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3385280039 wait complete. 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 -- start start 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 Processor -- start 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcdba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bcdc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558e0bce17f0 con 0x558e0bcd5e80 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e0bbf51a0 con 0x558e0bcdea60 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e0bce1970 con 0x558e0bcd5560 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63e3ef740 10 client.0 _ll_register_callbacks cb 0x558e0b94cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff63959b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcdba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35340/0 (socket says 172.21.15.116:35340) 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff638d9a640 1 -- 172.21.15.116:0/70773394 learned_addr learned my addr 172.21.15.116:0/70773394 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.092+0000 7ff639d9c640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bcdc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 -- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 msgr2=0x558e0bcdc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bcdc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 -- 172.21.15.116:0/70773394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 msgr2=0x558e0bcdbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 -- 172.21.15.116:0/70773394 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0ba75f70 con 0x558e0bcd5560 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff638d9a640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63959b640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcdba30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6300cc8a0 tx=0x7ff63001ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21bf18ac0addcba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630005310 con 0x558e0bcd5560 2024-03-19T02:35:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff62c00fd30 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7ff6300ccb20 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff630005680 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63e3ef740 10 client.27781 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e0b90d170 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff62c00fd30 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.093+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff630097e20 con 0x558e0bcd5560 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 10 client.27781 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 10 client.27781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 20 client.27781 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bceecd0 0x558e0bcf10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 4 client.27781 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e0bc466a0 con 0x558e0bceecd0 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff638d9a640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bceecd0 0x558e0bcf10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff638d9a640 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bceecd0 0x558e0bcf10c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35937ed7768bab0c server_cookie=dd8d7e4ff28ce6df in_seq=0 out_seq=0 2024-03-19T02:35:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.094+0000 7ff61cff9640 10 client.27781 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff61cff9640 1 -- 172.21.15.116:0/70773394 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e0bc466a0 con 0x558e0bceecd0 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff61cff9640 10 client.27781 handle_command_reply: tid=0 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 client.27781 shutdown 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 msgr2=0x558e0bcdba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcdba30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6300cc8a0 tx=0x7ff63001ded0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 shutdown_connections 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bceecd0 0x558e0bcf10c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bcdc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bcdbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/70773394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcdba30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff61cff9640 0 client.27781 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.095+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 >> 172.21.15.116:0/70773394 conn(0x558e0bccfc40 msgr2=0x558e0bcd19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.096+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 shutdown_connections 2024-03-19T02:35:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.096+0000 7ff63e3ef740 1 -- 172.21.15.116:0/70773394 wait complete. 2024-03-19T02:35:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.096+0000 7ff63e3ef740 20 client.27781 trim_cache size 0 max 16384 2024-03-19T02:35:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.114+0000 7ff63e3ef740 1 Processor -- start 2024-03-19T02:35:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 -- start start 2024-03-19T02:35:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcea590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcd5e80 0x558e0bcceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341aead0 con 0x558e0bcdea60 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341b0610 con 0x558e0bcd5e80 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341af1a0 con 0x558e0bcd5560 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcd5e80 0x558e0bcceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35356/0 (socket says 172.21.15.116:35356) 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 -- 172.21.15.116:0/3021637419 learned_addr learned my addr 172.21.15.116:0/3021637419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff63959b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 -- 172.21.15.116:0/3021637419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 msgr2=0x558e0bcea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 -- 172.21.15.116:0/3021637419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcd5e80 msgr2=0x558e0bcceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcd5e80 0x558e0bcceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 -- 172.21.15.116:0/3021637419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0bceaad0 con 0x558e0bcdea60 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff639d9c640 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff62c005f40 tx=0x7ff62c0097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f5771c4ce857d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.115+0000 7ff61effd640 1 -- 172.21.15.116:0/3021637419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c02d070 con 0x558e0bcdea60 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff61effd640 1 -- 172.21.15.116:0/3021637419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff62c00fd00 con 0x558e0bcdea60 2024-03-19T02:35:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff61effd640 1 -- 172.21.15.116:0/3021637419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c004a00 con 0x558e0bcdea60 2024-03-19T02:35:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3021637419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 msgr2=0x558e0bcf6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff62c005f40 tx=0x7ff62c0097a0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3021637419 shutdown_connections 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcdea60 0x558e0bcf6990 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcd5e80 0x558e0bcceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/3021637419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bcea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3021637419 >> 172.21.15.116:0/3021637419 conn(0x558e0bccfc40 msgr2=0x558e0bcce470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3021637419 shutdown_connections 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.116+0000 7ff63e3ef740 1 -- 172.21.15.116:0/3021637419 wait complete. 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 -- start start 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 Processor -- start 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bce27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bd53fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341aff40 con 0x558e0bcd5e80 2024-03-19T02:35:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341af870 con 0x558e0bcdea60 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341acad0 con 0x558e0bcd5560 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63e3ef740 10 client.0 _ll_register_callbacks cb 0x7ff63001df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35370/0 (socket says 172.21.15.116:35370) 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff63959b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bce27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 -- 172.21.15.116:0/4263321593 learned_addr learned my addr 172.21.15.116:0/4263321593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff639d9c640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bd53fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 -- 172.21.15.116:0/4263321593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 msgr2=0x558e0bce27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bce27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 -- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 msgr2=0x558e0bd53fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bd53fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.117+0000 7ff638d9a640 1 -- 172.21.15.116:0/4263321593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0bce35d0 con 0x558e0bcd5e80 2024-03-19T02:35:51.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff638d9a640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff624009930 tx=0x7ff62402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ca552704134afd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624003800 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff62c00fd00 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7ff6240039a0 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff624064050 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 10 client.18036 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e0bd556a0 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff62c00fd00 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff62403f440 con 0x558e0bcd5e80 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 10 client.18036 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 10 client.18036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 20 client.18036 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bd55dd0 0x558e0bd581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 4 client.18036 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.118+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558e0bc409f0 con 0x558e0bd55dd0 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.119+0000 7ff63959b640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bd55dd0 0x558e0bd581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.119+0000 7ff63959b640 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bd55dd0 0x558e0bd581f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=992172c261671c57 server_cookie=f3fc3c3f19c1739d in_seq=0 out_seq=0 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.119+0000 7ff61cff9640 10 client.18036 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff61cff9640 1 -- 172.21.15.116:0/4263321593 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558e0bc409f0 con 0x558e0bd55dd0 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff61cff9640 10 client.18036 handle_command_reply: tid=0 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 client.18036 shutdown 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 msgr2=0x558e0bce2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff624009930 tx=0x7ff62402bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 shutdown_connections 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558e0bd55dd0 0x558e0bd581f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0bcdea60 0x558e0bd53fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e0bcd5e80 0x558e0bce2d30 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 --2- 172.21.15.116:0/4263321593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e0bcd5560 0x558e0bce27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff61cff9640 0 client.18036 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 >> 172.21.15.116:0/4263321593 conn(0x558e0bccfc40 msgr2=0x558e0bcced00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 shutdown_connections 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 1 -- 172.21.15.116:0/4263321593 wait complete. 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.120+0000 7ff63e3ef740 20 client.18036 trim_cache size 0 max 16384 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "69a31e20-c544-40df-844e-439301717c3c", 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:35:51.127 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 msgr2=0x7ff6200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 0x7ff6200a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff624005ab0 tx=0x7ff624005fd0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 msgr2=0x7ff634198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff63000e9f0 tx=0x7ff63000edf0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 shutdown_connections 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff6200a2970 0x7ff6200a4e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff634106630 0x7ff6341a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff634100fe0 0x7ff634199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 --2- 172.21.15.116:0/417110856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634100620 0x7ff634198dc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 >> 172.21.15.116:0/417110856 conn(0x7ff6340fc130 msgr2=0x7ff6340fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 shutdown_connections 2024-03-19T02:35:51.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.121+0000 7ff63a59d640 1 -- 172.21.15.116:0/417110856 wait complete. 2024-03-19T02:35:51.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:35:51.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 -- 172.21.15.116:0/136335031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f0107870 msgr2=0x7f79f0107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/136335031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f0107870 0x7f79f0107c50 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009930 tx=0x7f79e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 -- 172.21.15.116:0/136335031 shutdown_connections 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/136335031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f00693e0 0x7f79f010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/136335031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0068a20 0x7f79f0068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/136335031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f0107870 0x7f79f0107c50 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 -- 172.21.15.116:0/136335031 >> 172.21.15.116:0/136335031 conn(0x7f79f0077820 msgr2=0x7f79f0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 -- 172.21.15.116:0/136335031 shutdown_connections 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.245+0000 7f79f5cbf640 1 -- 172.21.15.116:0/136335031 wait complete. 2024-03-19T02:35:51.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 Processor -- start 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 -- start start 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f0068a20 0x7f79f01006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107870 0x7f79f0101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79f0111a70 con 0x7f79f00693e0 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01118f0 con 0x7f79f0107870 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79f5cbf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f79f0111bf0 con 0x7f79f0068a20 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35390/0 (socket says 172.21.15.116:35390) 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79eeffd640 1 -- 172.21.15.116:0/2678337820 learned_addr learned my addr 172.21.15.116:0/2678337820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f0068a20 0x7f79f01006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.246+0000 7f79effff640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107870 0x7f79f0101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 -- 172.21.15.116:0/2678337820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f0068a20 msgr2=0x7f79f01006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f0068a20 0x7f79f01006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 -- 172.21.15.116:0/2678337820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107870 msgr2=0x7f79f0101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107870 0x7f79f0101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 -- 172.21.15.116:0/2678337820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f01a64d0 con 0x7f79f00693e0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79eeffd640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f79e400e9f0 tx=0x7f79e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79ecff9640 1 -- 172.21.15.116:0/2678337820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e400cd80 con 0x7f79f00693e0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f01a67c0 con 0x7f79f00693e0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79ecff9640 1 -- 172.21.15.116:0/2678337820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79e40108e0 con 0x7f79f00693e0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79ecff9640 1 -- 172.21.15.116:0/2678337820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4018920 con 0x7f79f00693e0 2024-03-19T02:35:51.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.247+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f01a6d10 con 0x7f79f00693e0 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ecff9640 1 -- 172.21.15.116:0/2678337820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f79e4010a90 con 0x7f79f00693e0 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 Processor -- start 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 -- start start 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879199f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28791c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d287874500 con 0x55d287910a30 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d28781cbe0 con 0x55d287911350 2024-03-19T02:35:51.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79f8988740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2878705a0 con 0x55d287919f30 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35398/0 (socket says 172.21.15.116:35398) 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79effff640 1 -- 172.21.15.116:0/1596122278 learned_addr learned my addr 172.21.15.116:0/1596122278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879199f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ecff9640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 0x7f79d80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79eeffd640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28791c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79effff640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 0x7f79d80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ecff9640 1 -- 172.21.15.116:0/2678337820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f79e40cf4a0 con 0x7f79f00693e0 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 -- 172.21.15.116:0/1596122278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 msgr2=0x55d28791c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28791c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 -- 172.21.15.116:0/1596122278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 msgr2=0x55d287910e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79ef7fe640 1 -- 172.21.15.116:0/1596122278 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d28783e000 con 0x55d287911350 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.250+0000 7f79effff640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.251+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879199f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009230 tx=0x7f79e0009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18dbebf0394ea02c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.251+0000 7f79effff640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 0x7f79d80a4e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f79e80049d0 tx=0x7f79e8004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.251+0000 7f79d5ffb640 1 -- 172.21.15.116:0/1596122278 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e002c780 con 0x55d287911350 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.251+0000 7f79d5ffb640 1 -- 172.21.15.116:0/1596122278 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79e002c920 con 0x55d287911350 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.251+0000 7f79d5ffb640 1 -- 172.21.15.116:0/1596122278 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00046c0 con 0x55d287911350 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- 172.21.15.116:0/1596122278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 msgr2=0x55d2879199f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879199f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009230 tx=0x7f79e0009560 comp rx=0 tx=0).stop 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- 172.21.15.116:0/1596122278 shutdown_connections 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28791c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879199f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 --2- 172.21.15.116:0/1596122278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287910e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- 172.21.15.116:0/1596122278 >> 172.21.15.116:0/1596122278 conn(0x55d28790b160 msgr2=0x55d28790b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- 172.21.15.116:0/1596122278 shutdown_connections 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- 172.21.15.116:0/1596122278 wait complete. 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 -- start start 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.252+0000 7f79f8988740 1 Processor -- start 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 0x55d287916f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287919f30 0x55d2879179d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d28791ccc0 con 0x55d287919f30 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d287830160 con 0x55d287911350 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d28791ce40 con 0x55d287910a30 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 10 client.0 _ll_register_callbacks cb 0x55d28786ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:44050/0 (socket says 172.21.15.116:44050) 2024-03-19T02:35:51.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 -- 172.21.15.116:0/805854146 learned_addr learned my addr 172.21.15.116:0/805854146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 0x55d287916f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79effff640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287919f30 0x55d2879179d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 -- 172.21.15.116:0/805854146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 msgr2=0x55d287916f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 0x55d287916f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 -- 172.21.15.116:0/805854146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287919f30 msgr2=0x55d2879179d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287919f30 0x55d2879179d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 -- 172.21.15.116:0/805854146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2876527d0 con 0x55d287911350 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 0x55d287916f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79eeffd640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79e400e9c0 tx=0x7f79e406b400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534437d97a4dcd61 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4026040 con 0x55d287911350 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f79e40860b0 con 0x55d287911350 2024-03-19T02:35:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.253+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e002c920 con 0x55d287911350 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f79e40a3f60 con 0x55d287911350 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 10 client.27851 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d2875883f0 con 0x55d287911350 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79e002c920 con 0x55d287911350 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f79e409acc0 con 0x55d287911350 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 10 client.27851 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 10 client.27851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 20 client.27851 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d28792a0c0 0x55d28792c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 4 client.27851 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d287881a40 con 0x55d28792a0c0 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.254+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d28792a0c0 0x55d28792c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.255+0000 7f79ceffd640 10 client.27851 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.255+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d28792a0c0 0x55d28792c4d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bd2feae83524eba server_cookie=869fe557bc96676d in_seq=0 out_seq=0 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.255+0000 7f79ceffd640 1 -- 172.21.15.116:0/805854146 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d287881a40 con 0x55d28792a0c0 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.255+0000 7f79ceffd640 10 client.27851 handle_command_reply: tid=0 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.255+0000 7f79f8988740 1 client.27851 shutdown 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 msgr2=0x55d287917490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79e400e9c0 tx=0x7f79e406b400 comp rx=0 tx=0).stop 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 shutdown_connections 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d28792a0c0 0x55d28792c4d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287919f30 0x55d2879179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287917490 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 --2- 172.21.15.116:0/805854146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287910a30 0x55d287916f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79ceffd640 0 client.27851 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 >> 172.21.15.116:0/805854146 conn(0x55d28790b160 msgr2=0x55d28790ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 shutdown_connections 2024-03-19T02:35:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 1 -- 172.21.15.116:0/805854146 wait complete. 2024-03-19T02:35:51.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.256+0000 7f79f8988740 20 client.27851 trim_cache size 0 max 16384 2024-03-19T02:35:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 Processor -- start 2024-03-19T02:35:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 -- start start 2024-03-19T02:35:51.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287915eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d287925a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01b0e60 con 0x55d287910a30 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01b29a0 con 0x55d287911350 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79f8988740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01b1530 con 0x55d287919f30 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35426/0 (socket says 172.21.15.116:35426) 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 -- 172.21.15.116:0/2259279148 learned_addr learned my addr 172.21.15.116:0/2259279148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 -- 172.21.15.116:0/2259279148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 msgr2=0x55d287925a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79effff640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d287925a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79eeffd640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287915eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d287925a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 -- 172.21.15.116:0/2259279148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 msgr2=0x55d287915eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287915eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 -- 172.21.15.116:0/2259279148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d28791c690 con 0x55d287910a30 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79effff640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d287925a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79eeffd640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287915eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000c00 tx=0x7f79e002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a385795f91b3a907 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.274+0000 7f79d57fa640 1 -- 172.21.15.116:0/2259279148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e002c900 con 0x55d287910a30 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79d57fa640 1 -- 172.21.15.116:0/2259279148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79e00035f0 con 0x55d287910a30 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79d57fa640 1 -- 172.21.15.116:0/2259279148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00416c0 con 0x55d287910a30 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 -- 172.21.15.116:0/2259279148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 msgr2=0x55d287919680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000c00 tx=0x7f79e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 -- 172.21.15.116:0/2259279148 shutdown_connections 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d287925a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d287915eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 --2- 172.21.15.116:0/2259279148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287919680 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 -- 172.21.15.116:0/2259279148 >> 172.21.15.116:0/2259279148 conn(0x55d28790b160 msgr2=0x55d28790cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 -- 172.21.15.116:0/2259279148 shutdown_connections 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.275+0000 7f79f8988740 1 -- 172.21.15.116:0/2259279148 wait complete. 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 -- start start 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 Processor -- start 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28798f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01b22d0 con 0x55d287910a30 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01b1c00 con 0x55d287911350 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f79f01aee60 con 0x55d287919f30 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79f8988740 10 client.0 _ll_register_callbacks cb 0x7f79e409b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:35:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35432/0 (socket says 172.21.15.116:35432) 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 -- 172.21.15.116:0/89871162 learned_addr learned my addr 172.21.15.116:0/89871162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 -- 172.21.15.116:0/89871162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 msgr2=0x55d28798f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28798f3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 -- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 msgr2=0x55d2879143c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879143c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.276+0000 7f79ef7fe640 1 -- 172.21.15.116:0/89871162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2875480f0 con 0x55d287910a30 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ef7fe640 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f79e002c980 tx=0x7f79e0009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a220b04617a979ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e005a030 con 0x55d287910a30 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9510+0+0 (secure 0 0 0) 0x7f79e00041b0 con 0x55d287910a30 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e00035f0 con 0x55d287910a30 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f79e00043b0 con 0x55d287910a30 2024-03-19T02:35:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 10 client.18054 fetch_fsmap learned FSMap version 30 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d2879907a0 con 0x55d287910a30 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79e00035f0 con 0x55d287910a30 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f79e0041890 con 0x55d287910a30 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 10 client.18054 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 10 client.18054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 20 client.18054 populate_metadata read hostname 'smithi116' 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d287990b20 0x55d287992f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 4 client.18054 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.277+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d28787bd90 con 0x55d287990b20 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.278+0000 7f79eeffd640 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d287990b20 0x55d287992f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.278+0000 7f79ceffd640 10 client.18054 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.278+0000 7f79eeffd640 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d287990b20 0x55d287992f40 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=272df8ec59ab63b4 server_cookie=c568a8869e80e55c in_seq=0 out_seq=0 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79ceffd640 1 -- 172.21.15.116:0/89871162 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d28787bd90 con 0x55d287990b20 2024-03-19T02:35:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79ceffd640 10 client.18054 handle_command_reply: tid=0 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 client.18054 shutdown 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 msgr2=0x55d287913e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f79e002c980 tx=0x7f79e0009650 comp rx=0 tx=0).stop 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 shutdown_connections 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d287990b20 0x55d287992f40 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d287919f30 0x55d28798f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d287911350 0x55d2879143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 --2- 172.21.15.116:0/89871162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d287910a30 0x55d287913e80 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 >> 172.21.15.116:0/89871162 conn(0x55d28790b160 msgr2=0x55d287916f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79ceffd640 0 client.18054 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 shutdown_connections 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 1 -- 172.21.15.116:0/89871162 wait complete. 2024-03-19T02:35:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.279+0000 7f79f8988740 20 client.18054 trim_cache size 0 max 16384 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "69a31e20-c544-40df-844e-439301717c3c": { 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "69a31e20-c544-40df-844e-439301717c3c", 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:35:51.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:35:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:35:51.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 msgr2=0x7f79d80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 0x7f79d80a4e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f79e80049d0 tx=0x7f79e8004ab0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 msgr2=0x7f79f0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f79e400e9f0 tx=0x7f79e400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 shutdown_connections 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f79d80a2970 0x7f79d80a4e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107870 0x7f79f0101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f00693e0 0x7f79f0100c20 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 --2- 172.21.15.116:0/2678337820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79f0068a20 0x7f79f01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 >> 172.21.15.116:0/2678337820 conn(0x7f79f0077820 msgr2=0x7f79f00fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 shutdown_connections 2024-03-19T02:35:51.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:35:51.280+0000 7f79f5cbf640 1 -- 172.21.15.116:0/2678337820 wait complete. 2024-03-19T02:35:51.295 INFO:tasks.cephfs.filesystem:scrub status for tag:69a31e20-c544-40df-844e-439301717c3c - {'path': '/', 'tag': '69a31e20-c544-40df-844e-439301717c3c', 'options': 'recursive,force'} 2024-03-19T02:35:51.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 258975 3.19 MB/sec execute 314 sec latency 27.839 ms 2024-03-19T02:35:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:51 smithi162 ceph-mon[35761]: pgmap v248: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-03-19T02:35:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:51 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:35:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:35:52.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:51 smithi154 ceph-mon[35693]: pgmap v248: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-03-19T02:35:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:51 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:35:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:35:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:51 smithi116 ceph-mon[32263]: pgmap v248: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-03-19T02:35:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:51 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:35:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:35:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:52.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 259416 3.18 MB/sec execute 315 sec latency 18.204 ms 2024-03-19T02:35:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:52 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:35:53.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:52 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:52 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:35:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:52 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:52 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:35:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:52 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:35:53.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 259948 3.18 MB/sec execute 316 sec latency 26.884 ms 2024-03-19T02:35:54.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:53 smithi162 ceph-mon[35761]: pgmap v249: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:35:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:53 smithi154 ceph-mon[35693]: pgmap v249: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:35:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:53 smithi116 ceph-mon[32263]: pgmap v249: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.4 MiB/s wr, 81 op/s 2024-03-19T02:35:54.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 260576 3.18 MB/sec execute 317 sec latency 25.614 ms 2024-03-19T02:35:55.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 261147 3.17 MB/sec execute 318 sec latency 30.551 ms 2024-03-19T02:35:56.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:55 smithi162 ceph-mon[35761]: pgmap v250: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.3 MiB/s wr, 120 op/s 2024-03-19T02:35:56.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:55 smithi154 ceph-mon[35693]: pgmap v250: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.3 MiB/s wr, 120 op/s 2024-03-19T02:35:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:55 smithi116 ceph-mon[32263]: pgmap v250: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.3 MiB/s wr, 120 op/s 2024-03-19T02:35:56.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 261905 3.18 MB/sec execute 319 sec latency 16.850 ms 2024-03-19T02:35:57.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 262343 3.18 MB/sec execute 320 sec latency 17.640 ms 2024-03-19T02:35:58.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:57 smithi162 ceph-mon[35761]: pgmap v251: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:57 smithi154 ceph-mon[35693]: pgmap v251: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:57 smithi116 ceph-mon[32263]: pgmap v251: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:35:58.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 262887 3.18 MB/sec execute 321 sec latency 17.495 ms 2024-03-19T02:35:59.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 263379 3.17 MB/sec execute 322 sec latency 29.676 ms 2024-03-19T02:36:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:59 smithi162 ceph-mon[35761]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:36:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:35:59 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:36:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:59 smithi154 ceph-mon[35693]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:36:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:35:59 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:36:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:59 smithi116 ceph-mon[32263]: pgmap v252: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-03-19T02:36:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:35:59 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:36:00.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 264005 3.18 MB/sec execute 323 sec latency 27.273 ms 2024-03-19T02:36:01.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:36:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:36:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:36:01.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 264492 3.17 MB/sec execute 324 sec latency 40.509 ms 2024-03-19T02:36:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:01 smithi162 ceph-mon[35761]: pgmap v253: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.4 MiB/s wr, 121 op/s 2024-03-19T02:36:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:01 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:36:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:01 smithi154 ceph-mon[35693]: pgmap v253: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.4 MiB/s wr, 121 op/s 2024-03-19T02:36:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:01 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:36:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:01 smithi116 ceph-mon[32263]: pgmap v253: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.4 MiB/s wr, 121 op/s 2024-03-19T02:36:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:01 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:36:02.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 265018 3.17 MB/sec execute 325 sec latency 23.480 ms 2024-03-19T02:36:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:02 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:36:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:02 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:36:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:02 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:36:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:02 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:36:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:02 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:36:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:02 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:36:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:02 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:36:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:02 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:36:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:02 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:36:03.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 265825 3.18 MB/sec execute 326 sec latency 18.077 ms 2024-03-19T02:36:04.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:03 smithi154 ceph-mon[35693]: pgmap v254: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:36:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:03 smithi116 ceph-mon[32263]: pgmap v254: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:36:04.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:03 smithi162 ceph-mon[35761]: pgmap v254: 129 pgs: 129 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:36:04.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 266289 3.18 MB/sec execute 327 sec latency 18.492 ms 2024-03-19T02:36:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 266662 3.17 MB/sec execute 328 sec latency 31.706 ms 2024-03-19T02:36:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:05 smithi154 ceph-mon[35693]: pgmap v255: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-03-19T02:36:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:05 smithi154 ceph-mon[35693]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T02:36:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:05 smithi116 ceph-mon[32263]: pgmap v255: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-03-19T02:36:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:05 smithi116 ceph-mon[32263]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T02:36:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:05 smithi162 ceph-mon[35761]: pgmap v255: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-03-19T02:36:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:05 smithi162 ceph-mon[35761]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T02:36:06.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 267240 3.17 MB/sec execute 329 sec latency 15.878 ms 2024-03-19T02:36:07.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:06 smithi154 ceph-mon[35693]: pgmap v257: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:06 smithi116 ceph-mon[32263]: pgmap v257: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:07.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:06 smithi162 ceph-mon[35761]: pgmap v257: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:07.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 267836 3.17 MB/sec execute 330 sec latency 31.124 ms 2024-03-19T02:36:08.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 268433 3.17 MB/sec execute 331 sec latency 20.897 ms 2024-03-19T02:36:09.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 269323 3.18 MB/sec execute 332 sec latency 16.978 ms 2024-03-19T02:36:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:09 smithi162 ceph-mon[35761]: pgmap v258: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:10.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:09 smithi154 ceph-mon[35693]: pgmap v258: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:09 smithi116 ceph-mon[32263]: pgmap v258: 129 pgs: 129 active+clean; 258 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-19T02:36:10.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 269634 3.18 MB/sec execute 333 sec latency 18.657 ms 2024-03-19T02:36:11.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 270068 3.17 MB/sec execute 334 sec latency 20.101 ms 2024-03-19T02:36:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:11 smithi162 ceph-mon[35761]: pgmap v259: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:36:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:11 smithi162 ceph-mon[35761]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T02:36:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:11 smithi154 ceph-mon[35693]: pgmap v259: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:36:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:11 smithi154 ceph-mon[35693]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T02:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:11 smithi116 ceph-mon[32263]: pgmap v259: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.0 MiB/s wr, 118 op/s 2024-03-19T02:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:11 smithi116 ceph-mon[32263]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T02:36:12.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 270543 3.17 MB/sec execute 335 sec latency 40.477 ms 2024-03-19T02:36:13.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 271132 3.17 MB/sec execute 336 sec latency 28.444 ms 2024-03-19T02:36:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:13 smithi162 ceph-mon[35761]: pgmap v261: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 1.2 MiB/s wr, 62 op/s 2024-03-19T02:36:14.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:13 smithi154 ceph-mon[35693]: pgmap v261: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 1.2 MiB/s wr, 62 op/s 2024-03-19T02:36:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:13 smithi116 ceph-mon[32263]: pgmap v261: 129 pgs: 129 active+clean; 261 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 1.2 MiB/s wr, 62 op/s 2024-03-19T02:36:14.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 271685 3.16 MB/sec execute 337 sec latency 21.062 ms 2024-03-19T02:36:15.790 INFO:tasks.workunit.client.0.smithi116.stdout: 1 272221 3.16 MB/sec execute 338 sec latency 18.108 ms 2024-03-19T02:36:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:15 smithi162 ceph-mon[35761]: pgmap v262: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-03-19T02:36:16.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:15 smithi154 ceph-mon[35693]: pgmap v262: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-03-19T02:36:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:15 smithi116 ceph-mon[32263]: pgmap v262: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-03-19T02:36:16.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 272932 3.17 MB/sec execute 339 sec latency 16.357 ms 2024-03-19T02:36:17.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 273447 3.17 MB/sec execute 340 sec latency 21.042 ms 2024-03-19T02:36:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:17 smithi162 ceph-mon[35761]: pgmap v263: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:18.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:17 smithi154 ceph-mon[35693]: pgmap v263: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:17 smithi116 ceph-mon[32263]: pgmap v263: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:18.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 273849 3.16 MB/sec execute 341 sec latency 38.207 ms 2024-03-19T02:36:19.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 274439 3.16 MB/sec execute 342 sec latency 30.519 ms 2024-03-19T02:36:20.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:19 smithi162 ceph-mon[35761]: pgmap v264: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:19 smithi154 ceph-mon[35693]: pgmap v264: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:19 smithi116 ceph-mon[32263]: pgmap v264: 129 pgs: 129 active+clean; 263 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-03-19T02:36:20.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 275016 3.16 MB/sec execute 343 sec latency 24.310 ms 2024-03-19T02:36:21.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:36:21.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:36:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2746489901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 msgr2=0x7f1df8068e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2746489901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8068e80 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009930 tx=0x7f1de402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:21.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2746489901 shutdown_connections 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2746489901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 0x7f1df810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2746489901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8068e80 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2746489901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 0x7f1df8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2746489901 >> 172.21.15.116:0/2746489901 conn(0x7f1df8077820 msgr2=0x7f1df8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2746489901 shutdown_connections 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.415+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2746489901 wait complete. 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 Processor -- start 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 -- start start 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 0x7f1df8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 0x7f1df819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8111a60 con 0x7f1df8068a00 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81118e0 con 0x7f1df81077a0 2024-03-19T02:36:21.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dff0ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8111be0 con 0x7f1df80693c0 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56564/0 (socket says 172.21.15.116:56564) 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dfce9b640 1 -- 172.21.15.116:0/2183728835 learned_addr learned my addr 172.21.15.116:0/2183728835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1df7fff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 0x7f1df8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.416+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 0x7f1df819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 -- 172.21.15.116:0/2183728835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 msgr2=0x7f1df8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 0x7f1df8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 -- 172.21.15.116:0/2183728835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 msgr2=0x7f1df819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 0x7f1df819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 -- 172.21.15.116:0/2183728835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df819cfd0 con 0x7f1df8068a00 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1df5ffb640 1 -- 172.21.15.116:0/2183728835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00cd80 con 0x7f1df8068a00 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1df5ffb640 1 -- 172.21.15.116:0/2183728835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dec0108e0 con 0x7f1df8068a00 2024-03-19T02:36:21.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df819d2c0 con 0x7f1df8068a00 2024-03-19T02:36:21.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1df5ffb640 1 -- 172.21.15.116:0/2183728835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec018920 con 0x7f1df8068a00 2024-03-19T02:36:21.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.417+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df81a4b50 con 0x7f1df8068a00 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.419+0000 7f1df5ffb640 1 -- 172.21.15.116:0/2183728835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1dec010a90 con 0x7f1df8068a00 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 Processor -- start 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1df5ffb640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 0x7f1de80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 -- start start 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1df5ffb640 1 -- 172.21.15.116:0/2183728835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f1dec0cf4f0 con 0x7f1df8068a00 2024-03-19T02:36:21.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 0x7f1de80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f841e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 0x56234f84a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f84af30 0x56234f84d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56234f7a5500 con 0x56234f841a30 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56234f74dbe0 con 0x56234f842350 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1e01dca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56234f7a15a0 con 0x56234f84af30 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f841e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f841e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56568/0 (socket says 172.21.15.116:56568) 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1dfce9b640 1 -- 172.21.15.116:0/2542873308 learned_addr learned my addr 172.21.15.116:0/2542873308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 0x56234f84a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.420+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f84af30 0x56234f84d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 -- 172.21.15.116:0/2542873308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f84af30 msgr2=0x56234f84d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f84af30 0x56234f84d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 -- 172.21.15.116:0/2542873308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 msgr2=0x56234f841e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f841e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 -- 172.21.15.116:0/2542873308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56234f76f000 con 0x56234f842350 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 0x7f1de80a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005fd0 tx=0x7f1df0011bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1df7fff640 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 0x56234f84a9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b59e1bc567ecdf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1de2ffd640 1 -- 172.21.15.116:0/2542873308 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de404c070 con 0x56234f842350 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1de2ffd640 1 -- 172.21.15.116:0/2542873308 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1de402c7e0 con 0x56234f842350 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.421+0000 7f1de2ffd640 1 -- 172.21.15.116:0/2542873308 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de4005190 con 0x56234f842350 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- 172.21.15.116:0/2542873308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 msgr2=0x56234f84a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 0x56234f84a9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4009560 comp rx=0 tx=0).stop 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- 172.21.15.116:0/2542873308 shutdown_connections 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f84af30 0x56234f84d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f842350 0x56234f84a9f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 --2- 172.21.15.116:0/2542873308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f841e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- 172.21.15.116:0/2542873308 >> 172.21.15.116:0/2542873308 conn(0x56234f83c160 msgr2=0x56234f83c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- 172.21.15.116:0/2542873308 shutdown_connections 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- 172.21.15.116:0/2542873308 wait complete. 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 -- start start 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.422+0000 7f1e01dca740 1 Processor -- start 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 0x56234f847e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f848900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56234f84dcc0 con 0x56234f842350 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56234f761160 con 0x56234f84af30 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56234f84de40 con 0x56234f841a30 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1e01dca740 10 client.0 _ll_register_callbacks cb 0x56234f79da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:21.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 0x56234f847e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f848900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 0x56234f847e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57616/0 (socket says 172.21.15.116:57616) 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfce9b640 1 -- 172.21.15.116:0/1987869778 learned_addr learned my addr 172.21.15.116:0/1987869778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56580/0 (socket says 172.21.15.116:56580) 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1987869778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 msgr2=0x56234f847e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 0x56234f847e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1987869778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 msgr2=0x56234f8483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1987869778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56234f5837d0 con 0x56234f84af30 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f848900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x56234f83d2f0 tx=0x7f1df0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0fa66a2ab197a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.423+0000 7f1dd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:36:21.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df001a040 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f1df001f3b0 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1de402c7e0 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1df001f5b0 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 10 client.27863 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56234f4b93f0 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1de402c7e0 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1df001d030 con 0x56234f84af30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 10 client.27863 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 10 client.27863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 20 client.27863 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f85b210 0x56234f85d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 4 client.27863 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56234f7b2a40 con 0x56234f85b210 2024-03-19T02:36:21.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.424+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f85b210 0x56234f85d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.425+0000 7f1dd7fff640 10 client.27863 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.425+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f85b210 0x56234f85d620 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d89c9926c03c6c server_cookie=7d6dcdd37213893 in_seq=0 out_seq=0 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1987869778 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56234f7b2a40 con 0x56234f85b210 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1dd7fff640 10 client.27863 handle_command_reply: tid=0 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 client.27863 shutdown 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 msgr2=0x56234f848900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f848900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x56234f83d2f0 tx=0x7f1df0004ab0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 shutdown_connections 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f85b210 0x56234f85d620 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f848900 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1dd7fff640 0 client.27863 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f842350 0x56234f8483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1987869778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f841a30 0x56234f847e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 >> 172.21.15.116:0/1987869778 conn(0x56234f83c160 msgr2=0x56234f83deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 shutdown_connections 2024-03-19T02:36:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 1 -- 172.21.15.116:0/1987869778 wait complete. 2024-03-19T02:36:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.426+0000 7f1e01dca740 20 client.27863 trim_cache size 0 max 16384 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 Processor -- start 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 -- start start 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f841a30 0x56234f84a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f856a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81ac9a0 con 0x56234f84af30 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81ae4e0 con 0x56234f841a30 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.444+0000 7f1e01dca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81ad070 con 0x56234f842350 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56588/0 (socket says 172.21.15.116:56588) 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f856a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1714168520 learned_addr learned my addr 172.21.15.116:0/1714168520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f841a30 0x56234f84a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1714168520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 msgr2=0x56234f856a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f856a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1714168520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f841a30 msgr2=0x56234f84a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f841a30 0x56234f84a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 -- 172.21.15.116:0/1714168520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56234f4790f0 con 0x56234f84af30 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f856a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005c40 tx=0x7f1df0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e71eda099897b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1de27fc640 1 -- 172.21.15.116:0/1714168520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df001a070 con 0x56234f84af30 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1de27fc640 1 -- 172.21.15.116:0/1714168520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df000fd50 con 0x56234f84af30 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.445+0000 7f1de27fc640 1 -- 172.21.15.116:0/1714168520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df001fbf0 con 0x56234f84af30 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- 172.21.15.116:0/1714168520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 msgr2=0x56234f847000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005c40 tx=0x7f1df0009580 comp rx=0 tx=0).stop 2024-03-19T02:36:21.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- 172.21.15.116:0/1714168520 shutdown_connections 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f84af30 0x56234f847000 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f856a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1714168520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f841a30 0x56234f84a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- 172.21.15.116:0/1714168520 >> 172.21.15.116:0/1714168520 conn(0x56234f83c160 msgr2=0x56234f83d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- 172.21.15.116:0/1714168520 shutdown_connections 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- 172.21.15.116:0/1714168520 wait complete. 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 -- start start 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.446+0000 7f1e01dca740 1 Processor -- start 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f844710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f844c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81ade10 con 0x56234f841a30 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81ad740 con 0x56234f84af30 2024-03-19T02:36:21.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81aa9a0 con 0x56234f842350 2024-03-19T02:36:21.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 10 client.0 _ll_register_callbacks cb 0x7f1df00114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:21.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56598/0 (socket says 172.21.15.116:56598) 2024-03-19T02:36:21.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 -- 172.21.15.116:0/1254919205 learned_addr learned my addr 172.21.15.116:0/1254919205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfd69c640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f844c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f844710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 -- 172.21.15.116:0/1254919205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 msgr2=0x56234f844710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f844710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 -- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 msgr2=0x56234f844c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f844c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 -- 172.21.15.116:0/1254919205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56234f845450 con 0x56234f841a30 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f844710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dfce9b640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1dec09b520 tx=0x7f1dec01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735e65d0a764de5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0ae020 con 0x56234f841a30 2024-03-19T02:36:21.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f1dec0ccae0 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.447+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56234f7acd90 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1dec0a3f10 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 10 client.18072 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56234f8c1560 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56234f7acd90 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1dec0afb50 con 0x56234f841a30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 10 client.18072 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 10 client.18072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 20 client.18072 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f8c1c80 0x56234f8c4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 4 client.18072 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:36:21.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56234f72a540 con 0x56234f8c1c80 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.448+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f8c1c80 0x56234f8c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1dd7fff640 10 client.18072 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1df7fff640 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f8c1c80 0x56234f8c4070 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33e16f945b43f6f1 server_cookie=39c8ef2c0679654a in_seq=0 out_seq=0 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1dd7fff640 1 -- 172.21.15.116:0/1254919205 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56234f72a540 con 0x56234f8c1c80 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1dd7fff640 10 client.18072 handle_command_reply: tid=0 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 client.18072 shutdown 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 msgr2=0x56234f8532f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1dec09b520 tx=0x7f1dec01a040 comp rx=0 tx=0).stop 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 shutdown_connections 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56234f8c1c80 0x56234f8c4070 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56234f84af30 0x56234f844c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56234f842350 0x56234f844710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1dd7fff640 0 client.18072 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 --2- 172.21.15.116:0/1254919205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56234f841a30 0x56234f8532f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 >> 172.21.15.116:0/1254919205 conn(0x56234f83c160 msgr2=0x56234f83d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 shutdown_connections 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 1 -- 172.21.15.116:0/1254919205 wait complete. 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.449+0000 7f1e01dca740 20 client.18072 trim_cache size 0 max 16384 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:36:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:36:21.459 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 msgr2=0x7f1de80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 0x7f1de80a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1df0005fd0 tx=0x7f1df0011bc0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 msgr2=0x7f1df8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e9f0 tx=0x7f1dec00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 shutdown_connections 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1de80a2970 0x7f1de80a4e30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1df81077a0 0x7f1df819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1df80693c0 0x7f1df8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 --2- 172.21.15.116:0/2183728835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1df8068a00 0x7f1df8197fe0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 >> 172.21.15.116:0/2183728835 conn(0x7f1df8077820 msgr2=0x7f1df80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 shutdown_connections 2024-03-19T02:36:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.450+0000 7f1dff0ff640 1 -- 172.21.15.116:0/2183728835 wait complete. 2024-03-19T02:36:21.461 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:36:21.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:36:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 -- 172.21.15.116:0/4197446636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc4100720 msgr2=0x7f6fc4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/4197446636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc4100720 0x7f6fc4100ba0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009930 tx=0x7f6fac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 -- 172.21.15.116:0/4197446636 shutdown_connections 2024-03-19T02:36:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/4197446636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc41010e0 0x7f6fc410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/4197446636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc4100720 0x7f6fc4100ba0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/4197446636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 0x7f6fc4106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 -- 172.21.15.116:0/4197446636 >> 172.21.15.116:0/4197446636 conn(0x7f6fc4077820 msgr2=0x7f6fc4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 -- 172.21.15.116:0/4197446636 shutdown_connections 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.572+0000 7f6fc91c6640 1 -- 172.21.15.116:0/4197446636 wait complete. 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 Processor -- start 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 -- start start 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 0x7f6fc4102db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 0x7f6fc4103830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc41109f0 con 0x7f6fc41010e0 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4110870 con 0x7f6fc4100720 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc91c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4110b70 con 0x7f6fc4106730 2024-03-19T02:36:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 0x7f6fc4102db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56622/0 (socket says 172.21.15.116:56622) 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fbbfff640 1 -- 172.21.15.116:0/3654057687 learned_addr learned my addr 172.21.15.116:0/3654057687 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.573+0000 7f6fc359e640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 0x7f6fc4103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 -- 172.21.15.116:0/3654057687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 msgr2=0x7f6fc4103830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 0x7f6fc4103830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 -- 172.21.15.116:0/3654057687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 msgr2=0x7f6fc4102db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 0x7f6fc4102db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 -- 172.21.15.116:0/3654057687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc41a6500 con 0x7f6fc41010e0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc2d9d640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 0x7f6fc4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fbbfff640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009230 tx=0x7f6fac02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac043020 con 0x7f6fc41010e0 2024-03-19T02:36:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc41a6790 con 0x7f6fc41010e0 2024-03-19T02:36:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fac02ca70 con 0x7f6fc41010e0 2024-03-19T02:36:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac003a50 con 0x7f6fc41010e0 2024-03-19T02:36:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.574+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc41a6d30 con 0x7f6fc41010e0 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.575+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc4068610 con 0x7f6fc41010e0 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.576+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6fac046020 con 0x7f6fc41010e0 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.577+0000 7f6fc0d99640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 0x7f6fb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.577+0000 7f6fc2d9d640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 0x7f6fb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.577+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f6fac0f1110 con 0x7f6fc41010e0 2024-03-19T02:36:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.577+0000 7f6fc2d9d640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 0x7f6fb00a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009f40 tx=0x7f6fb4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.582+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6fac04d020 con 0x7f6fc41010e0 2024-03-19T02:36:21.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 275593 3.16 MB/sec execute 344 sec latency 19.091 ms 2024-03-19T02:36:21.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.831+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fc4109cc0 con 0x7f6fc41010e0 2024-03-19T02:36:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.831+0000 7f6fc0d99640 1 -- 172.21.15.116:0/3654057687 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f6fac0bdba0 con 0x7f6fc41010e0 2024-03-19T02:36:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:36:21.832 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:36:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:36:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.833+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 msgr2=0x7f6fb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.833+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 0x7f6fb00a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009f40 tx=0x7f6fb4005f90 comp rx=0 tx=0).stop 2024-03-19T02:36:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.833+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 msgr2=0x7f6fc41032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.833+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009230 tx=0x7f6fac02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 shutdown_connections 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fb00a2970 0x7f6fb00a4e30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fc4106730 0x7f6fc4103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc41010e0 0x7f6fc41032f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 --2- 172.21.15.116:0/3654057687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc4100720 0x7f6fc4102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 >> 172.21.15.116:0/3654057687 conn(0x7f6fc4077820 msgr2=0x7f6fc4068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 shutdown_connections 2024-03-19T02:36:21.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.834+0000 7f6fc91c6640 1 -- 172.21.15.116:0/3654057687 wait complete. 2024-03-19T02:36:21.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 -- 172.21.15.116:0/2638184540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 msgr2=0x7f1180100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 --2- 172.21.15.116:0/2638184540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180100b20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1174009930 tx=0x7f117402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 -- 172.21.15.116:0/2638184540 shutdown_connections 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 --2- 172.21.15.116:0/2638184540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f118010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 --2- 172.21.15.116:0/2638184540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180100b20 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 --2- 172.21.15.116:0/2638184540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 0x7f1180106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 -- 172.21.15.116:0/2638184540 >> 172.21.15.116:0/2638184540 conn(0x7f11800778d0 msgr2=0x7f1180077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 -- 172.21.15.116:0/2638184540 shutdown_connections 2024-03-19T02:36:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.958+0000 7f1185b3b640 1 -- 172.21.15.116:0/2638184540 wait complete. 2024-03-19T02:36:21.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 Processor -- start 2024-03-19T02:36:21.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 -- start start 2024-03-19T02:36:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f11801984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 0x7f118019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1180110970 con 0x7f11801006a0 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11801107f0 con 0x7f1180101060 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f1185b3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1180110af0 con 0x7f11801066b0 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56632/0 (socket says 172.21.15.116:56632) 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 -- 172.21.15.116:0/192928995 learned_addr learned my addr 172.21.15.116:0/192928995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117effd640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f11801984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117ffff640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 0x7f118019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 -- 172.21.15.116:0/192928995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 msgr2=0x7f118019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 0x7f118019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 -- 172.21.15.116:0/192928995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 msgr2=0x7f11801984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f11801984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117f7fe640 1 -- 172.21.15.116:0/192928995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118019cf80 con 0x7f11801006a0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.959+0000 7f117effd640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f11801984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f117f7fe640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f117000e9f0 tx=0x7f117000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f117cff9640 1 -- 172.21.15.116:0/192928995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117000cd80 con 0x7f11801006a0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118019d270 con 0x7f11801006a0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f117cff9640 1 -- 172.21.15.116:0/192928995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11700108e0 con 0x7f11801006a0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f117cff9640 1 -- 172.21.15.116:0/192928995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170018920 con 0x7f11801006a0 2024-03-19T02:36:21.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.960+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11801a4b00 con 0x7f11801006a0 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.962+0000 7f117cff9640 1 -- 172.21.15.116:0/192928995 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1170010a90 con 0x7f11801006a0 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.962+0000 7f117cff9640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 0x7f11680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.962+0000 7f1188802740 1 Processor -- start 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.962+0000 7f117effd640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 0x7f11680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.962+0000 7f1188802740 1 -- start start 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117cff9640 1 -- 172.21.15.116:0/192928995 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f11700cf4f0 con 0x7f11801006a0 2024-03-19T02:36:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e1564290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e1566bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3e14be4b0 con 0x55f3e155b2d0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3e1466d70 con 0x55f3e15647d0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f1188802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f3e14ba550 con 0x55f3e155bbf0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56644/0 (socket says 172.21.15.116:56644) 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 -- 172.21.15.116:0/935371655 learned_addr learned my addr 172.21.15.116:0/935371655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117ffff640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e1566bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117effd640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e1564290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117effd640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 0x7f11680a4e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1174005ab0 tx=0x7f1174005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 -- 172.21.15.116:0/935371655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 msgr2=0x55f3e1564290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e1564290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 -- 172.21.15.116:0/935371655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 msgr2=0x55f3e1566bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e1566bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 -- 172.21.15.116:0/935371655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e1489210 con 0x55f3e155b2d0 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117ffff640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e1566bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.963+0000 7f117f7fe640 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f11700981f0 tx=0x7f117009b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5048d50b7a13d0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1165ffb640 1 -- 172.21.15.116:0/935371655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11700a3f10 con 0x55f3e155b2d0 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1165ffb640 1 -- 172.21.15.116:0/935371655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11700ccae0 con 0x55f3e155b2d0 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1165ffb640 1 -- 172.21.15.116:0/935371655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117009ac70 con 0x55f3e155b2d0 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 -- 172.21.15.116:0/935371655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 msgr2=0x55f3e155b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f11700981f0 tx=0x7f117009b2b0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 -- 172.21.15.116:0/935371655 shutdown_connections 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e1566bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e1564290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 --2- 172.21.15.116:0/935371655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e155b6b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 -- 172.21.15.116:0/935371655 >> 172.21.15.116:0/935371655 conn(0x55f3e1555a00 msgr2=0x55f3e1555de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 -- 172.21.15.116:0/935371655 shutdown_connections 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.964+0000 7f1188802740 1 -- 172.21.15.116:0/935371655 wait complete. 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 -- start start 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 Processor -- start 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e1561860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e15622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3e1567500 con 0x55f3e155bbf0 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3e147a310 con 0x55f3e15647d0 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f3e1567680 con 0x55f3e155b2d0 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f1188802740 10 client.0 _ll_register_callbacks cb 0x55f3e146e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f117effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f117effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56648/0 (socket says 172.21.15.116:56648) 2024-03-19T02:36:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f117effd640 1 -- 172.21.15.116:0/559046403 learned_addr learned my addr 172.21.15.116:0/559046403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f117f7fe640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e1561860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.965+0000 7f117ffff640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e15622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 -- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 msgr2=0x55f3e1561860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e1561860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 -- 172.21.15.116:0/559046403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 msgr2=0x55f3e15622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e15622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 -- 172.21.15.116:0/559046403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e11d23f0 con 0x55f3e155bbf0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117f7fe640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e1561860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f117effd640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1174036040 tx=0x7f117402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548ecedd05244037 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f115effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174051070 con 0x55f3e155bbf0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f1174003800 con 0x55f3e155bbf0 2024-03-19T02:36:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11700ccae0 con 0x55f3e155bbf0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f117405e050 con 0x55f3e155bbf0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f1188802740 10 client.18090 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f3e129c7d0 con 0x55f3e155bbf0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.966+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11700ccae0 con 0x55f3e155bbf0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f117403fb10 con 0x55f3e155bbf0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 10 client.18090 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 10 client.18090 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 20 client.18090 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e1574ae0 0x55f3e1576ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 4 client.18090 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3e151c3e0 con 0x55f3e1574ae0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f117f7fe640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e1574ae0 0x55f3e1576ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f117f7fe640 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e1574ae0 0x55f3e1576ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=88392cc622729a82 server_cookie=b87bd7c9333b472c in_seq=0 out_seq=0 2024-03-19T02:36:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.967+0000 7f115effd640 10 client.18090 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f115effd640 1 -- 172.21.15.116:0/559046403 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3e151c3e0 con 0x55f3e1574ae0 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f115effd640 10 client.18090 handle_command_reply: tid=0 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 client.18090 shutdown 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 msgr2=0x55f3e1561da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1174036040 tx=0x7f117402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 shutdown_connections 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e1574ae0 0x55f3e1576ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e15622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155bbf0 0x55f3e1561da0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 --2- 172.21.15.116:0/559046403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e1561860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f115effd640 0 client.18090 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 >> 172.21.15.116:0/559046403 conn(0x55f3e1555a00 msgr2=0x55f3e1557750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 shutdown_connections 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.968+0000 7f1188802740 1 -- 172.21.15.116:0/559046403 wait complete. 2024-03-19T02:36:21.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.969+0000 7f1188802740 20 client.18090 trim_cache size 0 max 16384 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 Processor -- start 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 -- start start 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e15702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e157c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11801aec30 con 0x55f3e155b2d0 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11801b0770 con 0x55f3e15647d0 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f1188802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11801af300 con 0x55f3e155bbf0 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56660/0 (socket says 172.21.15.116:56660) 2024-03-19T02:36:21.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 -- 172.21.15.116:0/4186688483 learned_addr learned my addr 172.21.15.116:0/4186688483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117effd640 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e15702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117ffff640 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e157c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 -- 172.21.15.116:0/4186688483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 msgr2=0x55f3e15702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e15702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 -- 172.21.15.116:0/4186688483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 msgr2=0x55f3e157c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e157c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 -- 172.21.15.116:0/4186688483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e15565d0 con 0x55f3e155b2d0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f117f7fe640 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f11700ccdb0 tx=0x7f1170097bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15b61b5c76dc7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f11657fa640 1 -- 172.21.15.116:0/4186688483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11700ccae0 con 0x55f3e155b2d0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.984+0000 7f11657fa640 1 -- 172.21.15.116:0/4186688483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11700a3f10 con 0x55f3e155b2d0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f11657fa640 1 -- 172.21.15.116:0/4186688483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117009af60 con 0x55f3e155b2d0 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- 172.21.15.116:0/4186688483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 msgr2=0x55f3e1576ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f11700ccdb0 tx=0x7f1170097bf0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- 172.21.15.116:0/4186688483 shutdown_connections 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e15647d0 0x55f3e157c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155bbf0 0x55f3e15702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 --2- 172.21.15.116:0/4186688483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e155b2d0 0x55f3e1576ed0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- 172.21.15.116:0/4186688483 >> 172.21.15.116:0/4186688483 conn(0x55f3e1555a00 msgr2=0x55f3e155cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- 172.21.15.116:0/4186688483 shutdown_connections 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- 172.21.15.116:0/4186688483 wait complete. 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.985+0000 7f1188802740 1 -- start start 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 Processor -- start 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e156cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e155bbf0 0x55f3e1567eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11801b00a0 con 0x55f3e15647d0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11801af9d0 con 0x55f3e155bbf0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11801acc30 con 0x55f3e155b2d0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f1188802740 10 client.0 _ll_register_callbacks cb 0x7f1174009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56664/0 (socket says 172.21.15.116:56664) 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 -- 172.21.15.116:0/177397862 learned_addr learned my addr 172.21.15.116:0/177397862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e155bbf0 0x55f3e1567eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117f7fe640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e156cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 -- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 msgr2=0x55f3e156cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e156cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 -- 172.21.15.116:0/177397862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e155bbf0 msgr2=0x55f3e1567eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e155bbf0 0x55f3e1567eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.986+0000 7f117ffff640 1 -- 172.21.15.116:0/177397862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e1568c00 con 0x55f3e15647d0 2024-03-19T02:36:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f117ffff640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1178012cd0 tx=0x7f117800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f93757bd5bf2e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178022060 con 0x55f3e15647d0 2024-03-19T02:36:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f1178003b70 con 0x55f3e15647d0 2024-03-19T02:36:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11700a3f10 con 0x55f3e15647d0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1178029050 con 0x55f3e15647d0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 10 client.18096 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f3e15dae60 con 0x55f3e15647d0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11700a3f10 con 0x55f3e15647d0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f11780104a0 con 0x55f3e15647d0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 10 client.18096 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 10 client.18096 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 20 client.18096 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e15db610 0x55f3e15dda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 4 client.18096 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.987+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3e14cb9f0 con 0x55f3e15db610 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f117f7fe640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e15db610 0x55f3e15dda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f115effd640 10 client.18096 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f117f7fe640 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e15db610 0x55f3e15dda30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb818e01ff21e6b2 server_cookie=75441b36bf9ec36e in_seq=0 out_seq=0 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f115effd640 1 -- 172.21.15.116:0/177397862 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3e14cb9f0 con 0x55f3e15db610 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f115effd640 10 client.18096 handle_command_reply: tid=0 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 client.18096 shutdown 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 msgr2=0x55f3e15683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1178012cd0 tx=0x7f117800f790 comp rx=0 tx=0).stop 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 shutdown_connections 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f3e15db610 0x55f3e15dda30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3e15647d0 0x55f3e15683f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3e155bbf0 0x55f3e1567eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f115effd640 0 client.18096 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 --2- 172.21.15.116:0/177397862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3e155b2d0 0x55f3e156cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 >> 172.21.15.116:0/177397862 conn(0x55f3e1555a00 msgr2=0x55f3e1561820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 shutdown_connections 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 1 -- 172.21.15.116:0/177397862 wait complete. 2024-03-19T02:36:21.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.988+0000 7f1188802740 20 client.18096 trim_cache size 0 max 16384 2024-03-19T02:36:21.997 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 msgr2=0x7f11680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 0x7f11680a4e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1174005ab0 tx=0x7f1174005fd0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 msgr2=0x7f1180197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f117000e9f0 tx=0x7f117000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 shutdown_connections 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11680a2970 0x7f11680a4e30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11801066b0 0x7f118019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1180101060 0x7f11801984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 --2- 172.21.15.116:0/192928995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11801006a0 0x7f1180197f90 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 >> 172.21.15.116:0/192928995 conn(0x7f11800778d0 msgr2=0x7f1180068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 shutdown_connections 2024-03-19T02:36:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:21.989+0000 7f1185b3b640 1 -- 172.21.15.116:0/192928995 wait complete. 2024-03-19T02:36:22.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:36:22.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 -- 172.21.15.116:0/3447791473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44300fff90 msgr2=0x7f4430100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 --2- 172.21.15.116:0/3447791473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44300fff90 0x7f4430100370 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f4420009930 tx=0x7f442002bc80 comp rx=0 tx=0).stop 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 -- 172.21.15.116:0/3447791473 shutdown_connections 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 --2- 172.21.15.116:0/3447791473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 0x7f443010e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 --2- 172.21.15.116:0/3447791473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4430100940 0x7f4430106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 --2- 172.21.15.116:0/3447791473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44300fff90 0x7f4430100370 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 -- 172.21.15.116:0/3447791473 >> 172.21.15.116:0/3447791473 conn(0x7f4430076c50 msgr2=0x7f44300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 -- 172.21.15.116:0/3447791473 shutdown_connections 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 -- 172.21.15.116:0/3447791473 wait complete. 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.116+0000 7f4435b05640 1 Processor -- start 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 -- start start 2024-03-19T02:36:22.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44300fff90 0x7f4430100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 0x7f4430101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4430111a60 con 0x7f4430101980 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44301118e0 con 0x7f4430106ba0 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f4435b05640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4430111be0 con 0x7f44300fff90 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56692/0 (socket says 172.21.15.116:56692) 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 -- 172.21.15.116:0/3185520064 learned_addr learned my addr 172.21.15.116:0/3185520064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442f7fe640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44300fff90 0x7f4430100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442effd640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 0x7f4430101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 -- 172.21.15.116:0/3185520064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44300fff90 msgr2=0x7f4430100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44300fff90 0x7f4430100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 -- 172.21.15.116:0/3185520064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 msgr2=0x7f4430101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 0x7f4430101440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 -- 172.21.15.116:0/3185520064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44301a68d0 con 0x7f4430101980 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.117+0000 7f442ffff640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f442800eb10 tx=0x7f442800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.118+0000 7f442cff9640 1 -- 172.21.15.116:0/3185520064 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428004b20 con 0x7f4430101980 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.118+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44301a6bc0 con 0x7f4430101980 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.118+0000 7f442cff9640 1 -- 172.21.15.116:0/3185520064 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4428004cc0 con 0x7f4430101980 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.118+0000 7f442cff9640 1 -- 172.21.15.116:0/3185520064 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428010690 con 0x7f4430101980 2024-03-19T02:36:22.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.118+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44301a70c0 con 0x7f4430101980 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f442cff9640 1 -- 172.21.15.116:0/3185520064 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4428005020 con 0x7f4430101980 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f44387c5740 1 Processor -- start 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f44387c5740 1 -- start start 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f442cff9640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 0x7f44180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f44387c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f44387c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x56209830a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.120+0000 7f44387c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x56209830cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f44387c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620982644b0 con 0x5620983012d0 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f44387c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56209820cd70 con 0x56209830a7d0 2024-03-19T02:36:22.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f44387c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562098260550 con 0x562098301bf0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442effd640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 0x7f44180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56696/0 (socket says 172.21.15.116:56696) 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 -- 172.21.15.116:0/1854654680 learned_addr learned my addr 172.21.15.116:0/1854654680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442cff9640 1 -- 172.21.15.116:0/3185520064 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f44280cf730 con 0x7f4430101980 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442effd640 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x56209830a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442ffff640 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x56209830cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 -- 172.21.15.116:0/1854654680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 msgr2=0x56209830a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x56209830a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 -- 172.21.15.116:0/1854654680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 msgr2=0x56209830cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x56209830cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 -- 172.21.15.116:0/1854654680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56209822f210 con 0x5620983012d0 2024-03-19T02:36:22.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442f7fe640 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4420036040 tx=0x7f4420009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e46e127e5b5e474 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f442effd640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 0x7f44180a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4424009710 tx=0x7f4424009af0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f4415ffb640 1 -- 172.21.15.116:0/1854654680 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420045020 con 0x5620983012d0 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f4415ffb640 1 -- 172.21.15.116:0/1854654680 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f442002c7d0 con 0x5620983012d0 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.121+0000 7f4415ffb640 1 -- 172.21.15.116:0/1854654680 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44200053b0 con 0x5620983012d0 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 -- 172.21.15.116:0/1854654680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 msgr2=0x5620983016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4420036040 tx=0x7f4420009560 comp rx=0 tx=0).stop 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 -- 172.21.15.116:0/1854654680 shutdown_connections 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x56209830cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x56209830a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 --2- 172.21.15.116:0/1854654680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620983012d0 0x5620983016b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 -- 172.21.15.116:0/1854654680 >> 172.21.15.116:0/1854654680 conn(0x5620982fba00 msgr2=0x5620982fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 -- 172.21.15.116:0/1854654680 shutdown_connections 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.122+0000 7f44387c5740 1 -- 172.21.15.116:0/1854654680 wait complete. 2024-03-19T02:36:22.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 -- start start 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 Processor -- start 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620983012d0 0x5620983077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x562098307ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56209830d560 con 0x56209830a7d0 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562098220310 con 0x5620983012d0 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56209830d6e0 con 0x562098301bf0 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f44387c5740 10 client.0 _ll_register_callbacks cb 0x5620982142d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620983012d0 0x5620983077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x562098307ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56698/0 (socket says 172.21.15.116:56698) 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 -- 172.21.15.116:0/1616387304 learned_addr learned my addr 172.21.15.116:0/1616387304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 -- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 msgr2=0x562098307ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x562098307ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 -- 172.21.15.116:0/1616387304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620983012d0 msgr2=0x5620983077a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620983012d0 0x5620983077a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.123+0000 7f442ffff640 1 -- 172.21.15.116:0/1616387304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562097f783f0 con 0x56209830a7d0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f442ffff640 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f44280ccb00 tx=0x7f442801def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d269ff91106e7b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442809b310 con 0x56209830a7d0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f442809bc10 con 0x56209830a7d0 2024-03-19T02:36:22.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442002c7d0 con 0x56209830a7d0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4428086670 con 0x56209830a7d0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 10 client.18108 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5620980427d0 con 0x56209830a7d0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f442002c7d0 con 0x56209830a7d0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f442809cce0 con 0x56209830a7d0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 10 client.18108 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 10 client.18108 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 20 client.18108 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56209831ae00 0x56209831d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 4 client.18108 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.124+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620982c23e0 con 0x56209831ae00 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.125+0000 7f442f7fe640 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56209831ae00 0x56209831d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.125+0000 7f440effd640 10 client.18108 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:22.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.125+0000 7f442f7fe640 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56209831ae00 0x56209831d1f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=733d5f2b37728387 server_cookie=9047e507fefdabda in_seq=0 out_seq=0 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f440effd640 1 -- 172.21.15.116:0/1616387304 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620982c23e0 con 0x56209831ae00 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f440effd640 10 client.18108 handle_command_reply: tid=0 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 client.18108 shutdown 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 msgr2=0x562098308220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f44280ccb00 tx=0x7f442801def0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 shutdown_connections 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56209831ae00 0x56209831d1f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x562098308220 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562098301bf0 0x562098307ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 --2- 172.21.15.116:0/1616387304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5620983012d0 0x5620983077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 >> 172.21.15.116:0/1616387304 conn(0x5620982fba00 msgr2=0x5620982fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f440effd640 0 client.18108 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:22.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 shutdown_connections 2024-03-19T02:36:22.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 1 -- 172.21.15.116:0/1616387304 wait complete. 2024-03-19T02:36:22.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.126+0000 7f44387c5740 20 client.18108 trim_cache size 0 max 16384 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 Processor -- start 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 -- start start 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x562098309ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x5620983227c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b0e20 con 0x562098301bf0 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b2960 con 0x56209830a7d0 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44387c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b14f0 con 0x5620983012d0 2024-03-19T02:36:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x562098309ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56714/0 (socket says 172.21.15.116:56714) 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 -- 172.21.15.116:0/256990392 learned_addr learned my addr 172.21.15.116:0/256990392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442ffff640 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x5620983227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 -- 172.21.15.116:0/256990392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 msgr2=0x562098309ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x562098309ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 -- 172.21.15.116:0/256990392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 msgr2=0x5620983227c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x5620983227c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 -- 172.21.15.116:0/256990392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620982fb830 con 0x562098301bf0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f442effd640 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4424013040 tx=0x7f4424021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3915599fed09861a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.141+0000 7f44157fa640 1 -- 172.21.15.116:0/256990392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442400fa50 con 0x562098301bf0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44157fa640 1 -- 172.21.15.116:0/256990392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f442400fbf0 con 0x562098301bf0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44157fa640 1 -- 172.21.15.116:0/256990392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424010690 con 0x562098301bf0 2024-03-19T02:36:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 -- 172.21.15.116:0/256990392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 msgr2=0x562098316300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4424013040 tx=0x7f4424021000 comp rx=0 tx=0).stop 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 -- 172.21.15.116:0/256990392 shutdown_connections 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56209830a7d0 0x5620983227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562098301bf0 0x562098316300 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 --2- 172.21.15.116:0/256990392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x562098309ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 -- 172.21.15.116:0/256990392 >> 172.21.15.116:0/256990392 conn(0x5620982fba00 msgr2=0x5620982faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 -- 172.21.15.116:0/256990392 shutdown_connections 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.142+0000 7f44387c5740 1 -- 172.21.15.116:0/256990392 wait complete. 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 -- start start 2024-03-19T02:36:22.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 Processor -- start 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x56209830e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 0x56209830e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x56209830ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b2290 con 0x56209830a7d0 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b1bc0 con 0x562098301bf0 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f44301aee20 con 0x5620983012d0 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f44387c5740 10 client.0 _ll_register_callbacks cb 0x7f4428004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x56209830ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x56209830e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x56209830ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56718/0 (socket says 172.21.15.116:56718) 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 0x56209830e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442ffff640 1 -- 172.21.15.116:0/576301104 learned_addr learned my addr 172.21.15.116:0/576301104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 -- 172.21.15.116:0/576301104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 msgr2=0x56209830e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 0x56209830e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 -- 172.21.15.116:0/576301104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 msgr2=0x56209830ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x56209830ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442f7fe640 1 -- 172.21.15.116:0/576301104 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56209837fdf0 con 0x5620983012d0 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.143+0000 7f442effd640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 0x56209830e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f442f7fe640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x56209830e1d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4420005ab0 tx=0x7f44200041f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=887b8808aa54037f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420045040 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442400fbf0 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f4420054ab0 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f442005c050 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 10 client.27859 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5620983810f0 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f442400fbf0 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f442002cad0 con 0x5620983012d0 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 10 client.27859 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 10 client.27859 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 20 client.27859 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562098381430 0x562098383820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 4 client.27859 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.144+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620982719f0 con 0x562098381430 2024-03-19T02:36:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f442effd640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562098381430 0x562098383820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f442effd640 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562098381430 0x562098383820 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=946d9f470036958 server_cookie=93125622e5633185 in_seq=0 out_seq=0 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f440effd640 10 client.27859 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f440effd640 1 -- 172.21.15.116:0/576301104 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620982719f0 con 0x562098381430 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f440effd640 10 client.27859 handle_command_reply: tid=0 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 client.27859 shutdown 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 msgr2=0x56209830e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x56209830e1d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4420005ab0 tx=0x7f44200041f0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 shutdown_connections 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x562098381430 0x562098383820 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56209830a7d0 0x56209830ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f440effd640 0 client.27859 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562098301bf0 0x56209830e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 --2- 172.21.15.116:0/576301104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5620983012d0 0x56209830e1d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 >> 172.21.15.116:0/576301104 conn(0x5620982fba00 msgr2=0x562098307760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 shutdown_connections 2024-03-19T02:36:22.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.145+0000 7f44387c5740 1 -- 172.21.15.116:0/576301104 wait complete. 2024-03-19T02:36:22.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f44387c5740 20 client.27859 trim_cache size 0 max 16384 2024-03-19T02:36:22.153 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 msgr2=0x7f44180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 0x7f44180a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4424009710 tx=0x7f4424009af0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 msgr2=0x7f44301a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f442800eb10 tx=0x7f442800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 shutdown_connections 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f44180a29a0 0x7f44180a4e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430101980 0x7f44301a61d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4430106ba0 0x7f4430101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 --2- 172.21.15.116:0/3185520064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44300fff90 0x7f4430100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 >> 172.21.15.116:0/3185520064 conn(0x7f4430076c50 msgr2=0x7f44300fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 shutdown_connections 2024-03-19T02:36:22.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.146+0000 7f4435b05640 1 -- 172.21.15.116:0/3185520064 wait complete. 2024-03-19T02:36:22.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:21 smithi116 ceph-mon[32263]: pgmap v265: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-03-19T02:36:22.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:21 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3654057687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:22.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:36:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:21 smithi162 ceph-mon[35761]: pgmap v265: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-03-19T02:36:22.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:21 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3654057687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:21 smithi154 ceph-mon[35693]: pgmap v265: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-03-19T02:36:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:21 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3654057687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 -- 172.21.15.116:0/1060716559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 msgr2=0x7fcef410b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/1060716559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef410b840 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009900 tx=0x7fcedc02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 -- 172.21.15.116:0/1060716559 shutdown_connections 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/1060716559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef410bd80 0x7fcef410e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/1060716559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef410b840 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/1060716559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef4102a40 0x7fcef4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 -- 172.21.15.116:0/1060716559 >> 172.21.15.116:0/1060716559 conn(0x7fcef40778d0 msgr2=0x7fcef4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 -- 172.21.15.116:0/1060716559 shutdown_connections 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.269+0000 7fcef8b3b640 1 -- 172.21.15.116:0/1060716559 wait complete. 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 Processor -- start 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 -- start start 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef4102a40 0x7fcef4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef410bd80 0x7fcef419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4111a30 con 0x7fcef4103360 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41118b0 con 0x7fcef410bd80 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.270+0000 7fcef8b3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4111bb0 con 0x7fcef4102a40 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fcef2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef410bd80 0x7fcef419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56724/0 (socket says 172.21.15.116:56724) 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fcef259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef4102a40 0x7fcef4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 -- 172.21.15.116:0/3216786167 learned_addr learned my addr 172.21.15.116:0/3216786167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 -- 172.21.15.116:0/3216786167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef4102a40 msgr2=0x7fcef4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef4102a40 0x7fcef4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 -- 172.21.15.116:0/3216786167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef410bd80 msgr2=0x7fcef419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef410bd80 0x7fcef419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 -- 172.21.15.116:0/3216786167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef419cf40 con 0x7fcef4103360 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceebfff640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fcedc005940 tx=0x7fcedc02c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceeb7fe640 1 -- 172.21.15.116:0/3216786167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc043020 con 0x7fcef4103360 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef419d1d0 con 0x7fcef4103360 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceeb7fe640 1 -- 172.21.15.116:0/3216786167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcedc004e00 con 0x7fcef4103360 2024-03-19T02:36:22.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fceeb7fe640 1 -- 172.21.15.116:0/3216786167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc003970 con 0x7fcef4103360 2024-03-19T02:36:22.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.271+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef41a4ac0 con 0x7fcef4103360 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fceeb7fe640 1 -- 172.21.15.116:0/3216786167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcedc046020 con 0x7fcef4103360 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fcefb7ff740 1 Processor -- start 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fcefb7ff740 1 -- start start 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fceeb7fe640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 0x7fcee00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.274+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 0x55dc600ec290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600ec7d0 0x55dc600eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dc600464b0 con 0x55dc600e32d0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dc5ffeed70 con 0x55dc600e3bf0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc60042550 con 0x55dc600ec7d0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef2d9d640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 0x7fcee00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fceeb7fe640 1 -- 172.21.15.116:0/3216786167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fcedc0f10b0 con 0x7fcef4103360 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56736/0 (socket says 172.21.15.116:56736) 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 -- 172.21.15.116:0/122075510 learned_addr learned my addr 172.21.15.116:0/122075510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fceebfff640 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 0x55dc600ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 -- 172.21.15.116:0/122075510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600ec7d0 msgr2=0x55dc600eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600ec7d0 0x55dc600eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 -- 172.21.15.116:0/122075510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 msgr2=0x55dc600ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 0x55dc600ec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 -- 172.21.15.116:0/122075510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc60011210 con 0x55dc600e32d0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fceebfff640 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 0x55dc600ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef2d9d640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 0x7fcee00a4e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fceec005fd0 tx=0x7fceec011810 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcef259c640 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009cf0 tx=0x7fcee4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7caa40472b06ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcee8ff9640 1 -- 172.21.15.116:0/122075510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee401e070 con 0x55dc600e32d0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcee8ff9640 1 -- 172.21.15.116:0/122075510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcee4009de0 con 0x55dc600e32d0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.275+0000 7fcee8ff9640 1 -- 172.21.15.116:0/122075510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4019450 con 0x55dc600e32d0 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- 172.21.15.116:0/122075510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 msgr2=0x55dc600e36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009cf0 tx=0x7fcee4011810 comp rx=0 tx=0).stop 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- 172.21.15.116:0/122075510 shutdown_connections 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600ec7d0 0x55dc600eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e3bf0 0x55dc600ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/122075510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600e36b0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- 172.21.15.116:0/122075510 >> 172.21.15.116:0/122075510 conn(0x55dc600dda00 msgr2=0x55dc600ddde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- 172.21.15.116:0/122075510 shutdown_connections 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- 172.21.15.116:0/122075510 wait complete. 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.276+0000 7fcefb7ff740 1 -- start start 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 Processor -- start 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600ea290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dc600ef5c0 con 0x55dc600e3bf0 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dc60002310 con 0x55dc600ec7d0 2024-03-19T02:36:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc600ef740 con 0x55dc600e32d0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcefb7ff740 10 client.0 _ll_register_callbacks cb 0x55dc5fff62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcef259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcef2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56746/0 (socket says 172.21.15.116:56746) 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcef259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57798/0 (socket says 172.21.15.116:57798) 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 -- 172.21.15.116:0/1142549495 learned_addr learned my addr 172.21.15.116:0/1142549495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 -- 172.21.15.116:0/1142549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 msgr2=0x55dc600e9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 -- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 msgr2=0x55dc600ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600ea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 -- 172.21.15.116:0/1142549495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc5fd5a3f0 con 0x55dc600e3bf0 2024-03-19T02:36:22.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcef259c640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fcef2d9d640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fceebfff640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0b9e10 tx=0x7fcedc0bcda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67984fc0e2d8834c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.277+0000 7fced9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc03cc90 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fcedc003b10 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcee4009de0 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcedc0bd440 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 10 client.18126 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc5fe247d0 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcee4009de0 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcedc03c8c0 con 0x55dc600e3bf0 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 10 client.18126 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 10 client.18126 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 20 client.18126 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc600fca80 0x55dc600feed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 4 client.18126 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc600a43e0 con 0x55dc600fca80 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.278+0000 7fcef259c640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc600fca80 0x55dc600feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.279+0000 7fced9ffb640 10 client.18126 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.279+0000 7fcef259c640 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc600fca80 0x55dc600feed0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f43b7c5909d80e6 server_cookie=bb1123b2583b762c in_seq=0 out_seq=0 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.279+0000 7fced9ffb640 1 -- 172.21.15.116:0/1142549495 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc600a43e0 con 0x55dc600fca80 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.279+0000 7fced9ffb640 10 client.18126 handle_command_reply: tid=0 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.279+0000 7fcefb7ff740 1 client.18126 shutdown 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 msgr2=0x55dc600e9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0b9e10 tx=0x7fcedc0bcda0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 shutdown_connections 2024-03-19T02:36:22.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc600fca80 0x55dc600feed0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e3bf0 0x55dc600e9d50 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/1142549495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e32d0 0x55dc600e9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fced9ffb640 0 client.18126 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 >> 172.21.15.116:0/1142549495 conn(0x55dc600dda00 msgr2=0x55dc600df750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 shutdown_connections 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 1 -- 172.21.15.116:0/1142549495 wait complete. 2024-03-19T02:36:22.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.280+0000 7fcefb7ff740 20 client.18126 trim_cache size 0 max 16384 2024-03-19T02:36:22.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 Processor -- start 2024-03-19T02:36:22.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 -- start start 2024-03-19T02:36:22.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e32d0 0x55dc600ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600f8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 0x55dc601047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41aec20 con 0x55dc600ec7d0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41b0760 con 0x55dc600e32d0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41af2f0 con 0x55dc600e3bf0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcef2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 0x55dc601047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcef2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 0x55dc601047a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56756/0 (socket says 172.21.15.116:56756) 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fceebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcef2d9d640 1 -- 172.21.15.116:0/2204244954 learned_addr learned my addr 172.21.15.116:0/2204244954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fcef259c640 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e32d0 0x55dc600ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fceebfff640 1 -- 172.21.15.116:0/2204244954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e32d0 msgr2=0x55dc600ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fceebfff640 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e32d0 0x55dc600ebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fceebfff640 1 -- 172.21.15.116:0/2204244954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 msgr2=0x55dc601047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.295+0000 7fceebfff640 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 0x55dc601047a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fceebfff640 1 -- 172.21.15.116:0/2204244954 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc600d7fb0 con 0x55dc600e3bf0 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fceebfff640 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600f8390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0bd3d0 tx=0x7fcedc047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=712237a2b65df8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fcedbfff640 1 -- 172.21.15.116:0/2204244954 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0c5be0 con 0x55dc600e3bf0 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fcedbfff640 1 -- 172.21.15.116:0/2204244954 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcedc0c1590 con 0x55dc600e3bf0 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fcedbfff640 1 -- 172.21.15.116:0/2204244954 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc0b9c00 con 0x55dc600e3bf0 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fcefb7ff740 1 -- 172.21.15.116:0/2204244954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 msgr2=0x55dc600f8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.296+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600f8390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0bd3d0 tx=0x7fcedc047000 comp rx=0 tx=0).stop 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- 172.21.15.116:0/2204244954 shutdown_connections 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600ec7d0 0x55dc601047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600f8390 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/2204244954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600e32d0 0x55dc600ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- 172.21.15.116:0/2204244954 >> 172.21.15.116:0/2204244954 conn(0x55dc600dda00 msgr2=0x55dc600e3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- 172.21.15.116:0/2204244954 shutdown_connections 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- 172.21.15.116:0/2204244954 wait complete. 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- start start 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 Processor -- start 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600efd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41b0090 con 0x55dc600e32d0 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41af9c0 con 0x55dc600ec7d0 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41acc20 con 0x55dc600e3bf0 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcefb7ff740 10 client.0 _ll_register_callbacks cb 0x7fcedc0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcef259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcef259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56766/0 (socket says 172.21.15.116:56766) 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcef259c640 1 -- 172.21.15.116:0/847443546 learned_addr learned my addr 172.21.15.116:0/847443546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fceebfff640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.297+0000 7fcef2d9d640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 -- 172.21.15.116:0/847443546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 msgr2=0x55dc600efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 -- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 msgr2=0x55dc600f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600f02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 -- 172.21.15.116:0/847443546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc600f0810 con 0x55dc600e32d0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcef259c640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fcee4000c00 tx=0x7fcee4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279199e071d50848 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee401e040 con 0x55dc600e32d0 2024-03-19T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fcee4009de0 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcedc0c1590 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcee4038050 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcefb7ff740 10 client.18132 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc60162b00 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcedc0c1590 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.298+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcee4019450 con 0x55dc600e32d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 10 client.18132 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 10 client.18132 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 20 client.18132 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc601630d0 0x55dc601654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 4 client.18132 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc600539f0 con 0x55dc601630d0 2024-03-19T02:36:22.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fceebfff640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc601630d0 0x55dc601654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fced9ffb640 10 client.18132 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fceebfff640 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc601630d0 0x55dc601654c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8c186086f89e2a9 server_cookie=5a54ad96f14fd5e7 in_seq=0 out_seq=0 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fced9ffb640 1 -- 172.21.15.116:0/847443546 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc600539f0 con 0x55dc601630d0 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fced9ffb640 10 client.18132 handle_command_reply: tid=0 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 client.18132 shutdown 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 msgr2=0x55dc600f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fcee4000c00 tx=0x7fcee4011da0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 shutdown_connections 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dc601630d0 0x55dc601654c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc600ec7d0 0x55dc600f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc600e3bf0 0x55dc600efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fced9ffb640 0 client.18132 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 --2- 172.21.15.116:0/847443546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc600e32d0 0x55dc600f4990 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 >> 172.21.15.116:0/847443546 conn(0x55dc600dda00 msgr2=0x55dc600dee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 shutdown_connections 2024-03-19T02:36:22.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.299+0000 7fcefb7ff740 1 -- 172.21.15.116:0/847443546 wait complete. 2024-03-19T02:36:22.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcefb7ff740 20 client.18132 trim_cache size 0 max 16384 2024-03-19T02:36:22.307 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 msgr2=0x7fcee00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 0x7fcee00a4e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fceec005fd0 tx=0x7fceec011810 comp rx=0 tx=0).stop 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 msgr2=0x7fcef4198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fcedc005940 tx=0x7fcedc02c690 comp rx=0 tx=0).stop 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 shutdown_connections 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee00a2970 0x7fcee00a4e30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef410bd80 0x7fcef419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef4103360 0x7fcef4198490 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 --2- 172.21.15.116:0/3216786167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef4102a40 0x7fcef4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 >> 172.21.15.116:0/3216786167 conn(0x7fcef40778d0 msgr2=0x7fcef40fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 shutdown_connections 2024-03-19T02:36:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.300+0000 7fcef8b3b640 1 -- 172.21.15.116:0/3216786167 wait complete. 2024-03-19T02:36:22.313 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/1562765133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 msgr2=0x7f4f34101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/1562765133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34101c10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009900 tx=0x7f4f2402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/1562765133 shutdown_connections 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/1562765133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f3410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/1562765133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34101c10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/1562765133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 0x7f4f34107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/1562765133 >> 172.21.15.116:0/1562765133 conn(0x7f4f340fd2c0 msgr2=0x7f4f340ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/1562765133 shutdown_connections 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.425+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/1562765133 wait complete. 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 Processor -- start 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 -- start start 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 0x7f4f3419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34111a60 con 0x7f4f34102150 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341118e0 con 0x7f4f34101790 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3b4ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34111be0 con 0x7f4f341077a0 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f38a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f38a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56778/0 (socket says 172.21.15.116:56778) 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f38a49640 1 -- 172.21.15.116:0/600384326 learned_addr learned my addr 172.21.15.116:0/600384326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f3924a640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.426+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 0x7f4f3419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 -- 172.21.15.116:0/600384326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 msgr2=0x7f4f3419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 0x7f4f3419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 -- 172.21.15.116:0/600384326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 msgr2=0x7f4f34197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 -- 172.21.15.116:0/600384326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3419cfc0 con 0x7f4f34102150 2024-03-19T02:36:22.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f38a49640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006010 tx=0x7f4f24005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f2a7fc640 1 -- 172.21.15.116:0/600384326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24043020 con 0x7f4f34102150 2024-03-19T02:36:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3419d250 con 0x7f4f34102150 2024-03-19T02:36:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f2a7fc640 1 -- 172.21.15.116:0/600384326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f24004350 con 0x7f4f34102150 2024-03-19T02:36:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f2a7fc640 1 -- 172.21.15.116:0/600384326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24003a10 con 0x7f4f34102150 2024-03-19T02:36:22.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.427+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f341a4b40 con 0x7f4f34102150 2024-03-19T02:36:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.429+0000 7f4f2a7fc640 1 -- 172.21.15.116:0/600384326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4f24046020 con 0x7f4f34102150 2024-03-19T02:36:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 Processor -- start 2024-03-19T02:36:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 -- start start 2024-03-19T02:36:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f2a7fc640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 0x7f4f200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56530ec674b0 con 0x56530ed0d7d0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56530ec0fd70 con 0x56530ed042d0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3e170740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56530ec63550 con 0x56530ed04bf0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f38a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57822/0 (socket says 172.21.15.116:57822) 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f3924a640 1 -- 172.21.15.116:0/3660104694 learned_addr learned my addr 172.21.15.116:0/3660104694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38998/0 (socket says 172.21.15.116:38998) 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 0x7f4f200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.430+0000 7f4f2a7fc640 1 -- 172.21.15.116:0/600384326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f4f240f0be0 con 0x7f4f34102150 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 -- 172.21.15.116:0/3660104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 msgr2=0x56530ed046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 -- 172.21.15.116:0/3660104694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 msgr2=0x56530ed0fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 -- 172.21.15.116:0/3660104694 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56530ec32210 con 0x56530ed04bf0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 0x7f4f200a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c005fd0 tx=0x7f4f2c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009ca0 tx=0x7f4f30011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce25e68d7bf27159 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f1f7fe640 1 -- 172.21.15.116:0/3660104694 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3001e070 con 0x56530ed04bf0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f1f7fe640 1 -- 172.21.15.116:0/3660104694 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f30019ae0 con 0x56530ed04bf0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.431+0000 7f4f1f7fe640 1 -- 172.21.15.116:0/3660104694 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000e9f0 con 0x56530ed04bf0 2024-03-19T02:36:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- 172.21.15.116:0/3660104694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 msgr2=0x56530ed0d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009ca0 tx=0x7f4f30011810 comp rx=0 tx=0).stop 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- 172.21.15.116:0/3660104694 shutdown_connections 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed04bf0 0x56530ed0d290 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3660104694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed042d0 0x56530ed046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- 172.21.15.116:0/3660104694 >> 172.21.15.116:0/3660104694 conn(0x56530ecfea00 msgr2=0x56530ecfede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- 172.21.15.116:0/3660104694 shutdown_connections 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- 172.21.15.116:0/3660104694 wait complete. 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 -- start start 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 Processor -- start 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.432+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed0acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed0d7d0 0x56530ed0b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56530ed105c0 con 0x56530ed042d0 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56530ec23310 con 0x56530ed04bf0 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56530ed10740 con 0x56530ed0d7d0 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 10 client.0 _ll_register_callbacks cb 0x56530ec172d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56792/0 (socket says 172.21.15.116:56792) 2024-03-19T02:36:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 -- 172.21.15.116:0/3720133855 learned_addr learned my addr 172.21.15.116:0/3720133855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed0d7d0 0x56530ed0b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f38a49640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed0acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 -- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed0d7d0 msgr2=0x56530ed0b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed0d7d0 0x56530ed0b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 -- 172.21.15.116:0/3720133855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 msgr2=0x56530ed0acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed0acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 -- 172.21.15.116:0/3720133855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56530e97b3f0 con 0x56530ed042d0 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f38a49640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed0acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3924a640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f4f30005890 tx=0x7f4f30019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e873804a4c131d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30027020 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f4f30005250 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f30019ae0 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4f3001e070 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 10 client.18144 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.433+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56530ea457d0 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f30019ae0 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4f30004810 con 0x56530ed042d0 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 10 client.18144 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 10 client.18144 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 20 client.18144 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed1db20 0x56530ed1ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 4 client.18144 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56530ecc53e0 con 0x56530ed1db20 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f38a49640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed1db20 0x56530ed1ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f1cff9640 10 client.18144 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.434+0000 7f4f38a49640 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed1db20 0x56530ed1ff10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b7619d43f5d4de0 server_cookie=a2f062fd9c270a1a in_seq=0 out_seq=0 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.435+0000 7f4f1cff9640 1 -- 172.21.15.116:0/3720133855 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56530ecc53e0 con 0x56530ed1db20 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.435+0000 7f4f1cff9640 10 client.18144 handle_command_reply: tid=0 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 client.18144 shutdown 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 msgr2=0x56530ed0a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f4f30005890 tx=0x7f4f30019660 comp rx=0 tx=0).stop 2024-03-19T02:36:22.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 shutdown_connections 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed1db20 0x56530ed1ff10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed0d7d0 0x56530ed0b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f1cff9640 0 client.18144 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed0acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 --2- 172.21.15.116:0/3720133855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed042d0 0x56530ed0a7b0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 >> 172.21.15.116:0/3720133855 conn(0x56530ecfea00 msgr2=0x56530ed00750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 shutdown_connections 2024-03-19T02:36:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 1 -- 172.21.15.116:0/3720133855 wait complete. 2024-03-19T02:36:22.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.436+0000 7f4f3e170740 20 client.18144 trim_cache size 0 max 16384 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 Processor -- start 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 -- start start 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed19390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ecf9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341aeba0 con 0x56530ed0d7d0 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b06e0 con 0x56530ed04bf0 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3e170740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341af270 con 0x56530ed042d0 2024-03-19T02:36:22.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed19390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56800/0 (socket says 172.21.15.116:56800) 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 -- 172.21.15.116:0/104501188 learned_addr learned my addr 172.21.15.116:0/104501188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f38a49640 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ecf9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 -- 172.21.15.116:0/104501188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 msgr2=0x56530ed19390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed19390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 -- 172.21.15.116:0/104501188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 msgr2=0x56530ecf9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ecf9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.450+0000 7f4f39a4b640 1 -- 172.21.15.116:0/104501188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56530ed09760 con 0x56530ed0d7d0 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009fb0 tx=0x7f4f2c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd5642065f00fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f1effd640 1 -- 172.21.15.116:0/104501188 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c004b20 con 0x56530ed0d7d0 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f1effd640 1 -- 172.21.15.116:0/104501188 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f2c01fe40 con 0x56530ed0d7d0 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f1effd640 1 -- 172.21.15.116:0/104501188 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0054f0 con 0x56530ed0d7d0 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f3e170740 1 -- 172.21.15.116:0/104501188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 msgr2=0x56530ed0b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.451+0000 7f4f3e170740 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009fb0 tx=0x7f4f2c018000 comp rx=0 tx=0).stop 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 -- 172.21.15.116:0/104501188 shutdown_connections 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed0b050 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ecf9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 --2- 172.21.15.116:0/104501188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed19390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 -- 172.21.15.116:0/104501188 >> 172.21.15.116:0/104501188 conn(0x56530ecfea00 msgr2=0x56530ecfdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 -- 172.21.15.116:0/104501188 shutdown_connections 2024-03-19T02:36:22.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 -- 172.21.15.116:0/104501188 wait complete. 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 -- start start 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.452+0000 7f4f3e170740 1 Processor -- start 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed15c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed10f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b0010 con 0x56530ed0d7d0 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341af940 con 0x56530ed04bf0 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341acba0 con 0x56530ed042d0 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 10 client.0 _ll_register_callbacks cb 0x7f4f3001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56808/0 (socket says 172.21.15.116:56808) 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3924a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed15c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 -- 172.21.15.116:0/2780716045 learned_addr learned my addr 172.21.15.116:0/2780716045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f38a49640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed10f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 -- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 msgr2=0x56530ed15c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed15c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 -- 172.21.15.116:0/2780716045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 msgr2=0x56530ed10f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed10f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 -- 172.21.15.116:0/2780716045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56530ed11c00 con 0x56530ed0d7d0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f39a4b640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009c00 tx=0x7f4f2c0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971ef8c108411158 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c037020 con 0x56530ed0d7d0 2024-03-19T02:36:22.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f4f2c004c40 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.453+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56530ec749f0 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c042050 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 10 client.18150 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56530ed83bb0 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56530ec749f0 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4f2c01d070 con 0x56530ed0d7d0 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 10 client.18150 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 10 client.18150 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 20 client.18150 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed842f0 0x56530ed86710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 4 client.18150 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:22.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56530ec6ed40 con 0x56530ed842f0 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.454+0000 7f4f3924a640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed842f0 0x56530ed86710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f1cff9640 10 client.18150 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3924a640 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed842f0 0x56530ed86710 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26806b75d81cdb41 server_cookie=3179883afdd8481a in_seq=0 out_seq=0 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f1cff9640 1 -- 172.21.15.116:0/2780716045 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56530ec6ed40 con 0x56530ed842f0 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f1cff9640 10 client.18150 handle_command_reply: tid=0 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 client.18150 shutdown 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 msgr2=0x56530ed114c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009c00 tx=0x7f4f2c0190d0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 shutdown_connections 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56530ed842f0 0x56530ed86710 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530ed0d7d0 0x56530ed114c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530ed04bf0 0x56530ed10f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f1cff9640 0 client.18150 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 --2- 172.21.15.116:0/2780716045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530ed042d0 0x56530ed15c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 >> 172.21.15.116:0/2780716045 conn(0x56530ecfea00 msgr2=0x56530ecffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 shutdown_connections 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 1 -- 172.21.15.116:0/2780716045 wait complete. 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.455+0000 7f4f3e170740 20 client.18150 trim_cache size 0 max 16384 2024-03-19T02:36:22.467 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 msgr2=0x7f4f200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 0x7f4f200a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c005fd0 tx=0x7f4f2c009500 comp rx=0 tx=0).stop 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 msgr2=0x7f4f34198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006010 tx=0x7f4f24005d90 comp rx=0 tx=0).stop 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 shutdown_connections 2024-03-19T02:36:22.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4f200a2970 0x7f4f200a4e30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f341077a0 0x7f4f3419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f34102150 0x7f4f34198510 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 --2- 172.21.15.116:0/600384326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f34101790 0x7f4f34197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 >> 172.21.15.116:0/600384326 conn(0x7f4f340fd2c0 msgr2=0x7f4f340fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 shutdown_connections 2024-03-19T02:36:22.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.456+0000 7f4f3b4ae640 1 -- 172.21.15.116:0/600384326 wait complete. 2024-03-19T02:36:22.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 -- 172.21.15.116:0/884751623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 msgr2=0x7fab9810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 --2- 172.21.15.116:0/884751623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9810e220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fab90009ca0 tx=0x7fab9002c180 comp rx=0 tx=0).stop 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 -- 172.21.15.116:0/884751623 shutdown_connections 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 --2- 172.21.15.116:0/884751623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9810e220 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 --2- 172.21.15.116:0/884751623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98103360 0x7fab9810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 --2- 172.21.15.116:0/884751623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98102a40 0x7fab98102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 -- 172.21.15.116:0/884751623 >> 172.21.15.116:0/884751623 conn(0x7fab980778d0 msgr2=0x7fab98077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 -- 172.21.15.116:0/884751623 shutdown_connections 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 -- 172.21.15.116:0/884751623 wait complete. 2024-03-19T02:36:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.582+0000 7fab9e559640 1 Processor -- start 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 -- start start 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98103360 0x7fab98198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab98111a30 con 0x7fab9810bd80 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab981118b0 con 0x7fab98103360 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9e559640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab98111bb0 con 0x7fab98102a40 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab977fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98103360 0x7fab98198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab9caf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57850/0 (socket says 172.21.15.116:57850) 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 -- 172.21.15.116:0/1772562506 learned_addr learned my addr 172.21.15.116:0/1772562506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 -- 172.21.15.116:0/1772562506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98103360 msgr2=0x7fab98198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98103360 0x7fab98198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 -- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 msgr2=0x7fab9819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.583+0000 7fab97fff640 1 -- 172.21.15.116:0/1772562506 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9819cee0 con 0x7fab98102a40 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab97fff640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d950 tx=0x7fab8800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab957fa640 1 -- 172.21.15.116:0/1772562506 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88017030 con 0x7fab98102a40 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9819d1d0 con 0x7fab98102a40 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab957fa640 1 -- 172.21.15.116:0/1772562506 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab88004020 con 0x7fab98102a40 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab957fa640 1 -- 172.21.15.116:0/1772562506 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880047a0 con 0x7fab98102a40 2024-03-19T02:36:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.584+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981a4a60 con 0x7fab98102a40 2024-03-19T02:36:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.586+0000 7fab957fa640 1 -- 172.21.15.116:0/1772562506 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fab880041d0 con 0x7fab98102a40 2024-03-19T02:36:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.586+0000 7faba1211740 1 Processor -- start 2024-03-19T02:36:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.586+0000 7faba1211740 1 -- start start 2024-03-19T02:36:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab957fa640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 0x7fab800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b6953d4b0 con 0x561b695e37d0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b694e5d70 con 0x561b695da2d0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7faba1211740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b69539550 con 0x561b695dabf0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab9caf6640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 0x7fab800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab957fa640 1 -- 172.21.15.116:0/1772562506 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fab880cec20 con 0x7fab98102a40 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39064/0 (socket says 172.21.15.116:39064) 2024-03-19T02:36:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab9caf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57860/0 (socket says 172.21.15.116:57860) 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 -- 172.21.15.116:0/66261747 learned_addr learned my addr 172.21.15.116:0/66261747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 -- 172.21.15.116:0/66261747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 msgr2=0x561b695e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 -- 172.21.15.116:0/66261747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 msgr2=0x561b695e5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695e5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 -- 172.21.15.116:0/66261747 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b69508210 con 0x561b695da2d0 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab9caf6640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 0x7fab800a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fab90004af0 tx=0x7fab90032340 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab97fff640 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fab88097870 tx=0x7fab8809ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=558c29b6f3de123f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.587+0000 7fab7e7fc640 1 -- 172.21.15.116:0/66261747 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880786a0 con 0x561b695da2d0 2024-03-19T02:36:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7fab7e7fc640 1 -- 172.21.15.116:0/66261747 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8809b5a0 con 0x561b695da2d0 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7fab7e7fc640 1 -- 172.21.15.116:0/66261747 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88078880 con 0x561b695da2d0 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 -- 172.21.15.116:0/66261747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 msgr2=0x561b695da6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fab88097870 tx=0x7fab8809ab10 comp rx=0 tx=0).stop 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 -- 172.21.15.116:0/66261747 shutdown_connections 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695e5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 --2- 172.21.15.116:0/66261747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695da6b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 -- 172.21.15.116:0/66261747 >> 172.21.15.116:0/66261747 conn(0x561b695d4a00 msgr2=0x561b695d4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 -- 172.21.15.116:0/66261747 shutdown_connections 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.588+0000 7faba1211740 1 -- 172.21.15.116:0/66261747 wait complete. 2024-03-19T02:36:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 -- start start 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 Processor -- start 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b695e65c0 con 0x561b695da2d0 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b694f9310 con 0x561b695e37d0 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b695e6740 con 0x561b695dabf0 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7faba1211740 10 client.0 _ll_register_callbacks cb 0x561b694ed2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56842/0 (socket says 172.21.15.116:56842) 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 -- 172.21.15.116:0/2437581971 learned_addr learned my addr 172.21.15.116:0/2437581971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab977fe640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab9caf6640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 -- 172.21.15.116:0/2437581971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 msgr2=0x561b695e0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 -- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 msgr2=0x561b695e12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.589+0000 7fab97fff640 1 -- 172.21.15.116:0/2437581971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b692513f0 con 0x561b695da2d0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab97fff640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fab880cc040 tx=0x7fab880a36b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa19606b47b0b440 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880782e0 con 0x561b695da2d0 2024-03-19T02:36:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fab88078070 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab8809b5a0 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fab880784c0 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 10 client.18162 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561b6931b7d0 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8809b5a0 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fab8809a380 con 0x561b695da2d0 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 10 client.18162 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 20 client.18162 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b695f3a40 0x561b695f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 4 client.18162 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.590+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6959b3e0 con 0x561b695f3a40 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.591+0000 7fab977fe640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b695f3a40 0x561b695f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.591+0000 7fab977fe640 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b695f3a40 0x561b695f5e30 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30e2f58c457a3b5 server_cookie=5568d0aad31530a3 in_seq=0 out_seq=0 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.591+0000 7fab777fe640 10 client.18162 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7fab777fe640 1 -- 172.21.15.116:0/2437581971 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b6959b3e0 con 0x561b695f3a40 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7fab777fe640 10 client.18162 handle_command_reply: tid=0 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 client.18162 shutdown 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 msgr2=0x561b695e0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fab880cc040 tx=0x7fab880a36b0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 shutdown_connections 2024-03-19T02:36:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b695f3a40 0x561b695f5e30 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7fab777fe640 0 client.18162 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695e0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 --2- 172.21.15.116:0/2437581971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695e0870 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 >> 172.21.15.116:0/2437581971 conn(0x561b695d4a00 msgr2=0x561b695d6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 shutdown_connections 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 1 -- 172.21.15.116:0/2437581971 wait complete. 2024-03-19T02:36:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.592+0000 7faba1211740 20 client.18162 trim_cache size 0 max 16384 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 Processor -- start 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 -- start start 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695ef360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab981aec80 con 0x561b695da2d0 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b07c0 con 0x561b695e37d0 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.608+0000 7faba1211740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab981af350 con 0x561b695dabf0 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56848/0 (socket says 172.21.15.116:56848) 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 -- 172.21.15.116:0/2186580573 learned_addr learned my addr 172.21.15.116:0/2186580573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab9caf6640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab977fe640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695ef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 -- 172.21.15.116:0/2186580573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 msgr2=0x561b695ef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695ef360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 -- 172.21.15.116:0/2186580573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 msgr2=0x561b695e1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 -- 172.21.15.116:0/2186580573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b695d02c0 con 0x561b695da2d0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab9caf6640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab97fff640 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fab88099150 tx=0x7fab880a3640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955c8407045023c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab7dffb640 1 -- 172.21.15.116:0/2186580573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8809a9e0 con 0x561b695da2d0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab7dffb640 1 -- 172.21.15.116:0/2186580573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8809b5a0 con 0x561b695da2d0 2024-03-19T02:36:22.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.609+0000 7fab7dffb640 1 -- 172.21.15.116:0/2186580573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88078880 con 0x561b695da2d0 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- 172.21.15.116:0/2186580573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 msgr2=0x561b695dfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fab88099150 tx=0x7fab880a3640 comp rx=0 tx=0).stop 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- 172.21.15.116:0/2186580573 shutdown_connections 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695e37d0 0x561b695e1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695ef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 --2- 172.21.15.116:0/2186580573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695da2d0 0x561b695dfb20 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- 172.21.15.116:0/2186580573 >> 172.21.15.116:0/2186580573 conn(0x561b695d4a00 msgr2=0x561b695d3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- 172.21.15.116:0/2186580573 shutdown_connections 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- 172.21.15.116:0/2186580573 wait complete. 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.610+0000 7faba1211740 1 -- start start 2024-03-19T02:36:22.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 Processor -- start 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695dda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab981b00f0 con 0x561b695e37d0 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab981afa20 con 0x561b695da2d0 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab981acc80 con 0x561b695dabf0 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7faba1211740 10 client.0 _ll_register_callbacks cb 0x7fab8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab9caf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab97fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab9caf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56850/0 (socket says 172.21.15.116:56850) 2024-03-19T02:36:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab9caf6640 1 -- 172.21.15.116:0/1693485988 learned_addr learned my addr 172.21.15.116:0/1693485988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 -- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 msgr2=0x561b695dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 -- 172.21.15.116:0/1693485988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 msgr2=0x561b695ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab977fe640 1 -- 172.21.15.116:0/1693485988 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b69658b40 con 0x561b695dabf0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab9caf6640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.611+0000 7fab97fff640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab977fe640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695dda60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x561b695f6400 tx=0x7fab8c004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9db984f1876fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c0035f0 con 0x561b695dabf0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab8809b5a0 con 0x561b695dabf0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fab8c003790 con 0x561b695dabf0 2024-03-19T02:36:22.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fab8c031050 con 0x561b695dabf0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 10 client.27910 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561b69659e80 con 0x561b695dabf0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8809b5a0 con 0x561b695dabf0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fab8c010ad0 con 0x561b695dabf0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 10 client.27910 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 10 client.27910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 20 client.27910 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b6965a680 0x561b6965caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 4 client.27910 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.612+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b6954a9f0 con 0x561b6965a680 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab97fff640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b6965a680 0x561b6965caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab97fff640 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b6965a680 0x561b6965caa0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ab55b30463c5dc server_cookie=de414a5027ea213d in_seq=0 out_seq=0 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab777fe640 10 client.27910 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab777fe640 1 -- 172.21.15.116:0/1693485988 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b6954a9f0 con 0x561b6965a680 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab777fe640 10 client.27910 handle_command_reply: tid=0 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 client.27910 shutdown 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 msgr2=0x561b695dda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695dda60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x561b695f6400 tx=0x7fab8c004410 comp rx=0 tx=0).stop 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 shutdown_connections 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b6965a680 0x561b6965caa0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b695e37d0 0x561b695ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b695dabf0 0x561b695dda60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7fab777fe640 0 client.27910 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 --2- 172.21.15.116:0/1693485988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b695da2d0 0x561b695dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.613+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 >> 172.21.15.116:0/1693485988 conn(0x561b695d4a00 msgr2=0x561b695d5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 shutdown_connections 2024-03-19T02:36:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7faba1211740 1 -- 172.21.15.116:0/1693485988 wait complete. 2024-03-19T02:36:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7faba1211740 20 client.27910 trim_cache size 0 max 16384 2024-03-19T02:36:22.625 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 msgr2=0x7fab800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 0x7fab800a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fab90004af0 tx=0x7fab90032340 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 msgr2=0x7fab98197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d950 tx=0x7fab8800de10 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 shutdown_connections 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fab800a2970 0x7fab800a4e30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab9810bd80 0x7fab9819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab98103360 0x7fab98198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 --2- 172.21.15.116:0/1772562506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab98102a40 0x7fab98197ef0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 >> 172.21.15.116:0/1772562506 conn(0x7fab980778d0 msgr2=0x7fab980fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 shutdown_connections 2024-03-19T02:36:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:22.614+0000 7fab9e559640 1 -- 172.21.15.116:0/1772562506 wait complete. 2024-03-19T02:36:22.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 276435 3.17 MB/sec execute 345 sec latency 15.062 ms 2024-03-19T02:36:23.628 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:36:23.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:36:23.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:36:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 -- 172.21.15.116:0/1525431228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448068a20 msgr2=0x7f0448068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 --2- 172.21.15.116:0/1525431228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448068a20 0x7f0448068ea0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f043c009930 tx=0x7f043c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 -- 172.21.15.116:0/1525431228 shutdown_connections 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 --2- 172.21.15.116:0/1525431228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 0x7f044810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 --2- 172.21.15.116:0/1525431228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448068a20 0x7f0448068ea0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 --2- 172.21.15.116:0/1525431228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448107870 0x7f0448107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 -- 172.21.15.116:0/1525431228 >> 172.21.15.116:0/1525431228 conn(0x7f0448077820 msgr2=0x7f0448077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 -- 172.21.15.116:0/1525431228 shutdown_connections 2024-03-19T02:36:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.741+0000 7f044ccc0640 1 -- 172.21.15.116:0/1525431228 wait complete. 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 Processor -- start 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 -- start start 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448068a20 0x7f0448197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 0x7f0448198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0448111a70 con 0x7f0448107870 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04481118f0 con 0x7f04480693e0 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044ccc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0448111bf0 con 0x7f0448068a20 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56874/0 (socket says 172.21.15.116:56874) 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 0x7f0448198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 -- 172.21.15.116:0/17382920 learned_addr learned my addr 172.21.15.116:0/17382920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f044659c640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448068a20 0x7f0448197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 -- 172.21.15.116:0/17382920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448068a20 msgr2=0x7f0448197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448068a20 0x7f0448197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 -- 172.21.15.116:0/17382920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 msgr2=0x7f0448198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 0x7f0448198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.742+0000 7f0446d9d640 1 -- 172.21.15.116:0/17382920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044819cfc0 con 0x7f0448107870 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f0446d9d640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f044000eb10 tx=0x7f044000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f04377fe640 1 -- 172.21.15.116:0/17382920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440004b20 con 0x7f0448107870 2024-03-19T02:36:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044819d2b0 con 0x7f0448107870 2024-03-19T02:36:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f04377fe640 1 -- 172.21.15.116:0/17382920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0440004cc0 con 0x7f0448107870 2024-03-19T02:36:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f04377fe640 1 -- 172.21.15.116:0/17382920 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440010690 con 0x7f0448107870 2024-03-19T02:36:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.743+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04481a4b40 con 0x7f0448107870 2024-03-19T02:36:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.745+0000 7f04377fe640 1 -- 172.21.15.116:0/17382920 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0440005020 con 0x7f0448107870 2024-03-19T02:36:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.745+0000 7f044f984740 1 Processor -- start 2024-03-19T02:36:23.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 -- start start 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f04377fe640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 0x7f04300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x559687488760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x559687493c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5596873ec160 con 0x559687488ca0 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559687394e10 con 0x559687488380 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044f984740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5596873e8200 con 0x559687491880 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 0x7f04300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f04377fe640 1 -- 172.21.15.116:0/17382920 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f04400cf7c0 con 0x7f0448107870 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x559687488760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56876/0 (socket says 172.21.15.116:56876) 2024-03-19T02:36:23.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 -- 172.21.15.116:0/3021686395 learned_addr learned my addr 172.21.15.116:0/3021686395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0446d9d640 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x559687493c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 -- 172.21.15.116:0/3021686395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 msgr2=0x559687493c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x559687493c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 -- 172.21.15.116:0/3021686395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 msgr2=0x559687488760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x559687488760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f0445d9b640 1 -- 172.21.15.116:0/3021686395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596871c87e0 con 0x559687488ca0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.746+0000 7f044659c640 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x559687488760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f0445d9b640 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c680 tx=0x7f043c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c38a9bd273d67e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f0445d9b640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 0x7f04300a4e30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f043c036040 tx=0x7f043c039000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f0434ff9640 1 -- 172.21.15.116:0/3021686395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c005d50 con 0x559687488ca0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f0434ff9640 1 -- 172.21.15.116:0/3021686395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f043c051070 con 0x559687488ca0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f0434ff9640 1 -- 172.21.15.116:0/3021686395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c004d70 con 0x559687488ca0 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f044f984740 1 -- 172.21.15.116:0/3021686395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 msgr2=0x559687491340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.747+0000 7f044f984740 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c680 tx=0x7f043c0040c0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- 172.21.15.116:0/3021686395 shutdown_connections 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x559687493c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x559687491340 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- 172.21.15.116:0/3021686395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x559687488760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- 172.21.15.116:0/3021686395 >> 172.21.15.116:0/3021686395 conn(0x559687482ab0 msgr2=0x559687482e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- 172.21.15.116:0/3021686395 shutdown_connections 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- 172.21.15.116:0/3021686395 wait complete. 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- start start 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 Processor -- start 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488380 0x55968748e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559687494610 con 0x559687488ca0 2024-03-19T02:36:23.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5596873a81a0 con 0x559687491880 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559687494790 con 0x559687488380 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044f984740 10 client.0 _ll_register_callbacks cb 0x5596870ffc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56890/0 (socket says 172.21.15.116:56890) 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f0445d9b640 1 -- 172.21.15.116:0/1668779874 learned_addr learned my addr 172.21.15.116:0/1668779874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f044659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488380 0x55968748e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.748+0000 7f0446d9d640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 -- 172.21.15.116:0/1668779874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488380 msgr2=0x55968748e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488380 0x55968748e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 -- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 msgr2=0x55968748f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 -- 172.21.15.116:0/1668779874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559687228f70 con 0x559687488ca0 2024-03-19T02:36:23.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c170 tx=0x7f043c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957250713237ba26 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f042dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c05b070 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f043c051070 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f043c0035f0 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f043c005d50 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f044f984740 10 client.18180 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5596870c0170 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.749+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f043c051070 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f043c04b030 con 0x559687488ca0 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 10 client.18180 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 10 client.18180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 20 client.18180 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5596874a19d0 0x5596874a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 4 client.18180 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596873f96a0 con 0x5596874a19d0 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044659c640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5596874a19d0 0x5596874a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f042dffb640 10 client.18180 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.750+0000 7f044659c640 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5596874a19d0 0x5596874a3dc0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0ff681d5c1efa5e server_cookie=9e57d9aeb9cbe78d in_seq=0 out_seq=0 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f042dffb640 1 -- 172.21.15.116:0/1668779874 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5596873f96a0 con 0x5596874a19d0 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f042dffb640 10 client.18180 handle_command_reply: tid=0 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 client.18180 shutdown 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 msgr2=0x55968748ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c170 tx=0x7f043c005c90 comp rx=0 tx=0).stop 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 shutdown_connections 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5596874a19d0 0x5596874a3dc0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748ed80 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f042dffb640 0 client.18180 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 --2- 172.21.15.116:0/1668779874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488380 0x55968748e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 >> 172.21.15.116:0/1668779874 conn(0x559687482ab0 msgr2=0x559687484800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 shutdown_connections 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 1 -- 172.21.15.116:0/1668779874 wait complete. 2024-03-19T02:36:23.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.751+0000 7f044f984740 20 client.18180 trim_cache size 0 max 16384 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 Processor -- start 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 -- start start 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x55968749d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x5596874a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04481aec40 con 0x559687488ca0 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04481b0780 con 0x559687488380 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044f984740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04481af310 con 0x559687491880 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56900/0 (socket says 172.21.15.116:56900) 2024-03-19T02:36:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0446d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x5596874a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 -- 172.21.15.116:0/1836412722 learned_addr learned my addr 172.21.15.116:0/1836412722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f044659c640 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x55968749d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 -- 172.21.15.116:0/1836412722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 msgr2=0x5596874a93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x5596874a93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 -- 172.21.15.116:0/1836412722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 msgr2=0x55968749d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x55968749d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0445d9b640 1 -- 172.21.15.116:0/1836412722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559687484da0 con 0x559687488ca0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.771+0000 7f0446d9d640 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x5596874a93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f0445d9b640 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f043c000c00 tx=0x7f043c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785d29507713e67a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f042ffff640 1 -- 172.21.15.116:0/1836412722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c05b030 con 0x559687488ca0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f042ffff640 1 -- 172.21.15.116:0/1836412722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f043c051040 con 0x559687488ca0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f042ffff640 1 -- 172.21.15.116:0/1836412722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c004e80 con 0x559687488ca0 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 -- 172.21.15.116:0/1836412722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 msgr2=0x55968748f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f043c000c00 tx=0x7f043c0040c0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 -- 172.21.15.116:0/1836412722 shutdown_connections 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687491880 0x5596874a93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488ca0 0x55968748f0e0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 --2- 172.21.15.116:0/1836412722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687488380 0x55968749d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.772+0000 7f044f984740 1 -- 172.21.15.116:0/1836412722 >> 172.21.15.116:0/1836412722 conn(0x559687482ab0 msgr2=0x559687481b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- 172.21.15.116:0/1836412722 shutdown_connections 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- 172.21.15.116:0/1836412722 wait complete. 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- start start 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 Processor -- start 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488ca0 0x55968748bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04481b00b0 con 0x559687488380 2024-03-19T02:36:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04481af9e0 con 0x559687491880 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04481acc40 con 0x559687488ca0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044f984740 10 client.0 _ll_register_callbacks cb 0x7f0438009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56902/0 (socket says 172.21.15.116:56902) 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 -- 172.21.15.116:0/2005351945 learned_addr learned my addr 172.21.15.116:0/2005351945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f0445d9b640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488ca0 0x55968748bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f0446d9d640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 -- 172.21.15.116:0/2005351945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488ca0 msgr2=0x55968748bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488ca0 0x55968748bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.773+0000 7f044659c640 1 -- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 msgr2=0x55968748bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044659c640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044659c640 1 -- 172.21.15.116:0/2005351945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559687506bb0 con 0x559687488380 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044659c640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x5596874948e0 tx=0x7f04380108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e525f7d6ada975 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438004050 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f04380041f0 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f043c051040 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f043802e050 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 10 client.18186 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559687507f10 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f043c051040 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0438027470 con 0x559687488380 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 10 client.18186 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 10 client.18186 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 20 client.18186 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559687508630 0x55968750aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 4 client.18186 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:36:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.774+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5596873f39f0 con 0x559687508630 2024-03-19T02:36:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.775+0000 7f0445d9b640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559687508630 0x55968750aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.775+0000 7f042dffb640 10 client.18186 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.775+0000 7f0445d9b640 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559687508630 0x55968750aa40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dd7e32e99a918da server_cookie=c4fa10fae6fc7272 in_seq=0 out_seq=0 2024-03-19T02:36:23.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.776+0000 7f042dffb640 1 -- 172.21.15.116:0/2005351945 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5596873f39f0 con 0x559687508630 2024-03-19T02:36:23.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.776+0000 7f042dffb640 10 client.18186 handle_command_reply: tid=0 2024-03-19T02:36:23.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.776+0000 7f044f984740 1 client.18186 shutdown 2024-03-19T02:36:23.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.776+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 msgr2=0x55968748b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.776+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x5596874948e0 tx=0x7f04380108d0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 shutdown_connections 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559687508630 0x55968750aa40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559687491880 0x55968748bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559687488ca0 0x55968748bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f042dffb640 0 client.18186 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 --2- 172.21.15.116:0/2005351945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559687488380 0x55968748b570 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 >> 172.21.15.116:0/2005351945 conn(0x559687482ab0 msgr2=0x559687483d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 shutdown_connections 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 1 -- 172.21.15.116:0/2005351945 wait complete. 2024-03-19T02:36:23.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044f984740 20 client.18186 trim_cache size 0 max 16384 2024-03-19T02:36:23.784 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:23.784 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:36:23.784 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "ebd4edf6-f2be-45ea-8fae-393b57898b88", 2024-03-19T02:36:23.784 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:36:23.784 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 msgr2=0x7f04300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 0x7f04300a4e30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f043c036040 tx=0x7f043c039000 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 msgr2=0x7f044819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f044000eb10 tx=0x7f044000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 shutdown_connections 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04300a2970 0x7f04300a4e30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0448107870 0x7f044819c890 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04480693e0 0x7f0448198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 --2- 172.21.15.116:0/17382920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0448068a20 0x7f0448197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 >> 172.21.15.116:0/17382920 conn(0x7f0448077820 msgr2=0x7f04480fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 shutdown_connections 2024-03-19T02:36:23.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.777+0000 7f044ccc0640 1 -- 172.21.15.116:0/17382920 wait complete. 2024-03-19T02:36:23.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:36:23.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:36:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 276812 3.17 MB/sec execute 346 sec latency 17.490 ms 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 -- 172.21.15.116:0/1508914632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781017a0 msgr2=0x7f7e78101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 --2- 172.21.15.116:0/1508914632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781017a0 0x7f7e78101c20 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009930 tx=0x7f7e6c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 -- 172.21.15.116:0/1508914632 shutdown_connections 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 --2- 172.21.15.116:0/1508914632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e78102160 0x7f7e7810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 --2- 172.21.15.116:0/1508914632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781017a0 0x7f7e78101c20 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 --2- 172.21.15.116:0/1508914632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e781077b0 0x7f7e78107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.904+0000 7f7e7e227640 1 -- 172.21.15.116:0/1508914632 >> 172.21.15.116:0/1508914632 conn(0x7f7e780fd2b0 msgr2=0x7f7e780ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.905+0000 7f7e7e227640 1 -- 172.21.15.116:0/1508914632 shutdown_connections 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.905+0000 7f7e7e227640 1 -- 172.21.15.116:0/1508914632 wait complete. 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.905+0000 7f7e7e227640 1 Processor -- start 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.905+0000 7f7e7e227640 1 -- start start 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 0x7f7e78074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781077b0 0x7f7e780745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e78111a30 con 0x7f7e781077b0 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781118b0 con 0x7f7e78102160 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e7e227640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e78111bb0 con 0x7f7e781017a0 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 0x7f7e78074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:57948/0 (socket says 172.21.15.116:57948) 2024-03-19T02:36:23.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781077b0 0x7f7e780745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 -- 172.21.15.116:0/682048896 learned_addr learned my addr 172.21.15.116:0/682048896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 -- 172.21.15.116:0/682048896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 msgr2=0x7f7e78074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 0x7f7e78074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 -- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781077b0 msgr2=0x7f7e780745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781077b0 0x7f7e780745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 -- 172.21.15.116:0/682048896 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e781a6550 con 0x7f7e781017a0 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 0x7f7e78074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.906+0000 7f7e777fe640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e9f0 tx=0x7f7e6800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.907+0000 7f7e74ff9640 1 -- 172.21.15.116:0/682048896 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800cd80 con 0x7f7e781017a0 2024-03-19T02:36:23.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.907+0000 7f7e74ff9640 1 -- 172.21.15.116:0/682048896 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e680108e0 con 0x7f7e781017a0 2024-03-19T02:36:23.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.907+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e781a6840 con 0x7f7e781017a0 2024-03-19T02:36:23.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.907+0000 7f7e74ff9640 1 -- 172.21.15.116:0/682048896 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68018920 con 0x7f7e781017a0 2024-03-19T02:36:23.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.907+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e781a6d60 con 0x7f7e781017a0 2024-03-19T02:36:23.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e74ff9640 1 -- 172.21.15.116:0/682048896 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7e68010430 con 0x7f7e781017a0 2024-03-19T02:36:23.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e74ff9640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 0x7f7e600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e80e0e740 1 Processor -- start 2024-03-19T02:36:23.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 0x7f7e600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e80e0e740 1 -- start start 2024-03-19T02:36:23.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.909+0000 7f7e74ff9640 1 -- 172.21.15.116:0/682048896 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f7e680cf4f0 con 0x7f7e781017a0 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 0x561c62494200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62496b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561c623e6500 con 0x561c62482410 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561c6238ebe0 con 0x561c62482dc0 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c623e25a0 con 0x561c62494740 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e77fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62496b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56944/0 (socket says 172.21.15.116:56944) 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 -- 172.21.15.116:0/4153532826 learned_addr learned my addr 172.21.15.116:0/4153532826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 0x7f7e600a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005ab0 tx=0x7f7e6c005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 0x561c62494200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 -- 172.21.15.116:0/4153532826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 msgr2=0x561c62496b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62496b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 -- 172.21.15.116:0/4153532826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 msgr2=0x561c62494200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 0x561c62494200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e777fe640 1 -- 172.21.15.116:0/4153532826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c623b0000 con 0x561c62482410 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 0x561c62494200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.910+0000 7f7e77fff640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62496b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7e680981f0 tx=0x7f7e6809b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbfe3b14afe132a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e5dffb640 1 -- 172.21.15.116:0/4153532826 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6809f8c0 con 0x561c62482410 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e5dffb640 1 -- 172.21.15.116:0/4153532826 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e680a3f10 con 0x561c62482410 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e5dffb640 1 -- 172.21.15.116:0/4153532826 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68086070 con 0x561c62482410 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4153532826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 msgr2=0x561c624827f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7e680981f0 tx=0x7f7e6809b720 comp rx=0 tx=0).stop 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4153532826 shutdown_connections 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62496b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.911+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482dc0 0x561c62494200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4153532826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482410 0x561c624827f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4153532826 >> 172.21.15.116:0/4153532826 conn(0x561c6247d160 msgr2=0x561c6247d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4153532826 shutdown_connections 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4153532826 wait complete. 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- start start 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 Processor -- start 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62491710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62492190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561c624974a0 con 0x561c62482dc0 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561c623a2160 con 0x561c62482410 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c62497620 con 0x561c62494740 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e80e0e740 10 client.0 _ll_register_callbacks cb 0x561c623dea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62491710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56956/0 (socket says 172.21.15.116:56956) 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e77fff640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62492190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.912+0000 7f7e76ffd640 1 -- 172.21.15.116:0/4135448649 learned_addr learned my addr 172.21.15.116:0/4135448649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 -- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 msgr2=0x561c62491710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62491710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 -- 172.21.15.116:0/4135448649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 msgr2=0x561c62491c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 -- 172.21.15.116:0/4135448649 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c621c47d0 con 0x561c62494740 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62491710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e77fff640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62492190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7e70005fd0 tx=0x7f7e7000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0646b551ff6ebd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:23.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000fe60 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e680a3f10 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f7e70003970 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e70003b70 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e80e0e740 10 client.27940 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.913+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c620fa3f0 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e680a3f10 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7e7001a410 con 0x561c62494740 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 10 client.27940 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 10 client.27940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 20 client.27940 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:23.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c624a4a70 0x561c624a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 4 client.27940 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c623f3a40 con 0x561c624a4a70 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c624a4a70 0x561c624a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e777fe640 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c624a4a70 0x561c624a6e90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d657d310e66f507 server_cookie=86dec6d1fc254497 in_seq=0 out_seq=0 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.914+0000 7f7e56ffd640 10 client.27940 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.915+0000 7f7e56ffd640 1 -- 172.21.15.116:0/4135448649 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c623f3a40 con 0x561c624a4a70 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.915+0000 7f7e56ffd640 10 client.27940 handle_command_reply: tid=0 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.915+0000 7f7e80e0e740 1 client.27940 shutdown 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 msgr2=0x561c62492190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62492190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7e70005fd0 tx=0x7f7e7000f790 comp rx=0 tx=0).stop 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 shutdown_connections 2024-03-19T02:36:23.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c624a4a70 0x561c624a6e90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c62492190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c62491c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/4135448649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62491710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e56ffd640 0 client.27940 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 >> 172.21.15.116:0/4135448649 conn(0x561c6247d160 msgr2=0x561c6247eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 shutdown_connections 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 1 -- 172.21.15.116:0/4135448649 wait complete. 2024-03-19T02:36:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.916+0000 7f7e80e0e740 20 client.27940 trim_cache size 0 max 16384 2024-03-19T02:36:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 Processor -- start 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 -- start start 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62490650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c624a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781acb30 con 0x561c62482dc0 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781ae670 con 0x561c62482410 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781ad200 con 0x561c62494740 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56972/0 (socket says 172.21.15.116:56972) 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e76ffd640 1 -- 172.21.15.116:0/3348834588 learned_addr learned my addr 172.21.15.116:0/3348834588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e777fe640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62490650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.938+0000 7f7e77fff640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c624a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 -- 172.21.15.116:0/3348834588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 msgr2=0x561c624a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c624a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 -- 172.21.15.116:0/3348834588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 msgr2=0x561c62490650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62490650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 -- 172.21.15.116:0/3348834588 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c620ba0f0 con 0x561c62482dc0 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e77fff640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c624a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e777fe640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62490650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c000c00 tx=0x7f7e6c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ca22ddfe475bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e5d7fa640 1 -- 172.21.15.116:0/3348834588 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c052030 con 0x561c62482dc0 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e5d7fa640 1 -- 172.21.15.116:0/3348834588 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e6c046070 con 0x561c62482dc0 2024-03-19T02:36:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.939+0000 7f7e5d7fa640 1 -- 172.21.15.116:0/3348834588 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c04e480 con 0x561c62482dc0 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3348834588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 msgr2=0x561c6248ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c000c00 tx=0x7f7e6c03f000 comp rx=0 tx=0).stop 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3348834588 shutdown_connections 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62494740 0x561c624a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62482dc0 0x561c6248ca20 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3348834588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c62490650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3348834588 >> 172.21.15.116:0/3348834588 conn(0x561c6247d160 msgr2=0x561c6247c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3348834588 shutdown_connections 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3348834588 wait complete. 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 -- start start 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.940+0000 7f7e80e0e740 1 Processor -- start 2024-03-19T02:36:23.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c6249c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62482dc0 0x561c6249cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781adfa0 con 0x561c62494740 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781ad8d0 con 0x561c62482410 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781aab30 con 0x561c62482dc0 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 10 client.0 _ll_register_callbacks cb 0x7f7e70009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e777fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c6249c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56984/0 (socket says 172.21.15.116:56984) 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 -- 172.21.15.116:0/3766527842 learned_addr learned my addr 172.21.15.116:0/3766527842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e76ffd640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62482dc0 0x561c6249cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 -- 172.21.15.116:0/3766527842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62482dc0 msgr2=0x561c6249cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62482dc0 0x561c6249cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 -- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 msgr2=0x561c6249c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c6249c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 -- 172.21.15.116:0/3766527842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c624984d0 con 0x561c62494740 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e77fff640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f7e70002230 tx=0x7f7e7001a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f046835389bee9f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7002d070 con 0x561c62494740 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f7e70003970 con 0x561c62494740 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.941+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e6c046070 con 0x561c62494740 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e70029630 con 0x561c62494740 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 10 client.18204 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561c6250a940 con 0x561c62494740 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e6c046070 con 0x561c62494740 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7e700327b0 con 0x561c62494740 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 10 client.18204 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 10 client.18204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 20 client.18204 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c6250b0c0 0x561c6250d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 4 client.18204 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561c623edd90 con 0x561c6250b0c0 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.942+0000 7f7e777fe640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c6250b0c0 0x561c6250d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e56ffd640 10 client.18204 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e777fe640 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c6250b0c0 0x561c6250d4e0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d861a220826573 server_cookie=58f51ab7a3874309 in_seq=0 out_seq=0 2024-03-19T02:36:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e56ffd640 1 -- 172.21.15.116:0/3766527842 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561c623edd90 con 0x561c6250b0c0 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e56ffd640 10 client.18204 handle_command_reply: tid=0 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 client.18204 shutdown 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 msgr2=0x561c62497ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f7e70002230 tx=0x7f7e7001a590 comp rx=0 tx=0).stop 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 shutdown_connections 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c6250b0c0 0x561c6250d4e0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c62494740 0x561c62497ca0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c62482dc0 0x561c6249cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e56ffd640 0 client.18204 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 --2- 172.21.15.116:0/3766527842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c62482410 0x561c6249c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 >> 172.21.15.116:0/3766527842 conn(0x561c6247d160 msgr2=0x561c62492dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 shutdown_connections 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.943+0000 7f7e80e0e740 1 -- 172.21.15.116:0/3766527842 wait complete. 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e80e0e740 20 client.18204 trim_cache size 0 max 16384 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout: "ebd4edf6-f2be-45ea-8fae-393b57898b88": { 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:36:23.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "ebd4edf6-f2be-45ea-8fae-393b57898b88", 2024-03-19T02:36:23.953 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:36:23.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:36:23.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:36:23.953 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 msgr2=0x7f7e600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 0x7f7e600a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005ab0 tx=0x7f7e6c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 msgr2=0x7f7e78073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e9f0 tx=0x7f7e6800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 shutdown_connections 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7e600a2970 0x7f7e600a4e30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e781077b0 0x7f7e780745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e78102160 0x7f7e78074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 --2- 172.21.15.116:0/682048896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e781017a0 0x7f7e78073b20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 >> 172.21.15.116:0/682048896 conn(0x7f7e780fd2b0 msgr2=0x7f7e780fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 shutdown_connections 2024-03-19T02:36:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:23.944+0000 7f7e7e227640 1 -- 172.21.15.116:0/682048896 wait complete. 2024-03-19T02:36:23.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:23 smithi116 ceph-mon[32263]: pgmap v266: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-03-19T02:36:23.956 INFO:tasks.cephfs.filesystem:scrub status for tag:ebd4edf6-f2be-45ea-8fae-393b57898b88 - {'path': '/', 'tag': 'ebd4edf6-f2be-45ea-8fae-393b57898b88', 'options': 'recursive,force'} 2024-03-19T02:36:24.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:23 smithi162 ceph-mon[35761]: pgmap v266: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-03-19T02:36:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:23 smithi154 ceph-mon[35693]: pgmap v266: 129 pgs: 129 active+clean; 270 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-03-19T02:36:24.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 277208 3.16 MB/sec execute 347 sec latency 33.954 ms 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:25.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 277753 3.16 MB/sec execute 348 sec latency 28.689 ms 2024-03-19T02:36:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:25 smithi162 ceph-mon[35761]: pgmap v267: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:36:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:25 smithi154 ceph-mon[35693]: pgmap v267: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:36:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:25 smithi116 ceph-mon[32263]: pgmap v267: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-19T02:36:26.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 278311 3.16 MB/sec execute 349 sec latency 64.256 ms 2024-03-19T02:36:27.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 278873 3.15 MB/sec execute 350 sec latency 68.281 ms 2024-03-19T02:36:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:27 smithi162 ceph-mon[35761]: pgmap v268: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:27 smithi154 ceph-mon[35693]: pgmap v268: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:27 smithi116 ceph-mon[32263]: pgmap v268: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:28.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 279592 3.16 MB/sec execute 351 sec latency 17.068 ms 2024-03-19T02:36:29.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 280065 3.16 MB/sec execute 352 sec latency 42.095 ms 2024-03-19T02:36:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:29 smithi162 ceph-mon[35761]: pgmap v269: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:30.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:29 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:36:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:29 smithi154 ceph-mon[35693]: pgmap v269: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:29 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:36:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:29 smithi116 ceph-mon[32263]: pgmap v269: 129 pgs: 129 active+clean; 273 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:29 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:36:30.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 280610 3.16 MB/sec execute 353 sec latency 22.400 ms 2024-03-19T02:36:31.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:36:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:36:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:36:31.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 281067 3.16 MB/sec execute 354 sec latency 33.370 ms 2024-03-19T02:36:32.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:31 smithi162 ceph-mon[35761]: pgmap v270: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-03-19T02:36:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:31 smithi154 ceph-mon[35693]: pgmap v270: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-03-19T02:36:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:31 smithi116 ceph-mon[32263]: pgmap v270: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-03-19T02:36:32.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 281714 3.16 MB/sec execute 355 sec latency 22.477 ms 2024-03-19T02:36:33.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 282245 3.15 MB/sec execute 356 sec latency 23.963 ms 2024-03-19T02:36:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:33 smithi162 ceph-mon[35761]: pgmap v271: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-03-19T02:36:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:33 smithi154 ceph-mon[35693]: pgmap v271: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-03-19T02:36:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:33 smithi116 ceph-mon[32263]: pgmap v271: 129 pgs: 129 active+clean; 276 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-03-19T02:36:34.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 282812 3.15 MB/sec execute 357 sec latency 22.413 ms 2024-03-19T02:36:35.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 283560 3.16 MB/sec execute 358 sec latency 16.277 ms 2024-03-19T02:36:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:35 smithi162 ceph-mon[35761]: pgmap v272: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:36:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:35 smithi154 ceph-mon[35693]: pgmap v272: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:36:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:35 smithi116 ceph-mon[32263]: pgmap v272: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.3 MiB/s wr, 121 op/s 2024-03-19T02:36:36.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 284017 3.16 MB/sec execute 359 sec latency 54.626 ms 2024-03-19T02:36:37.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 284441 3.16 MB/sec execute 360 sec latency 39.169 ms 2024-03-19T02:36:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:37 smithi162 ceph-mon[35761]: pgmap v273: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:37 smithi154 ceph-mon[35693]: pgmap v273: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:37 smithi116 ceph-mon[32263]: pgmap v273: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:38.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 285027 3.15 MB/sec execute 361 sec latency 16.055 ms 2024-03-19T02:36:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 285610 3.15 MB/sec execute 362 sec latency 30.920 ms 2024-03-19T02:36:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:39 smithi162 ceph-mon[35761]: pgmap v274: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:39 smithi154 ceph-mon[35693]: pgmap v274: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:39 smithi116 ceph-mon[32263]: pgmap v274: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:36:40.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 286181 3.15 MB/sec execute 363 sec latency 20.576 ms 2024-03-19T02:36:41.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 287062 3.16 MB/sec execute 364 sec latency 17.287 ms 2024-03-19T02:36:42.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:41 smithi162 ceph-mon[35761]: pgmap v275: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:36:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:41 smithi154 ceph-mon[35693]: pgmap v275: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:36:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:41 smithi116 ceph-mon[32263]: pgmap v275: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-19T02:36:42.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 287416 3.16 MB/sec execute 365 sec latency 18.669 ms 2024-03-19T02:36:43.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 287838 3.16 MB/sec execute 366 sec latency 16.874 ms 2024-03-19T02:36:44.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:43 smithi162 ceph-mon[35761]: pgmap v276: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:36:44.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:43 smithi154 ceph-mon[35693]: pgmap v276: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:36:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:43 smithi116 ceph-mon[32263]: pgmap v276: 129 pgs: 129 active+clean; 282 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:36:44.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 288375 3.15 MB/sec execute 367 sec latency 31.523 ms 2024-03-19T02:36:45.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 288927 3.15 MB/sec execute 368 sec latency 33.457 ms 2024-03-19T02:36:46.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:45 smithi162 ceph-mon[35761]: pgmap v277: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:36:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:45 smithi154 ceph-mon[35693]: pgmap v277: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:36:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:45 smithi116 ceph-mon[32263]: pgmap v277: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:36:46.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 289475 3.15 MB/sec execute 369 sec latency 22.614 ms 2024-03-19T02:36:47.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 290111 3.15 MB/sec execute 370 sec latency 17.866 ms 2024-03-19T02:36:48.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:47 smithi162 ceph-mon[35761]: pgmap v278: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:47 smithi154 ceph-mon[35693]: pgmap v278: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:47 smithi116 ceph-mon[32263]: pgmap v278: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:48.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 290717 3.15 MB/sec execute 371 sec latency 16.511 ms 2024-03-19T02:36:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 291246 3.16 MB/sec execute 372 sec latency 18.639 ms 2024-03-19T02:36:50.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:49 smithi162 ceph-mon[35761]: pgmap v279: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:50.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:49 smithi154 ceph-mon[35693]: pgmap v279: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:49 smithi116 ceph-mon[32263]: pgmap v279: 129 pgs: 129 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:36:50.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 291703 3.15 MB/sec execute 373 sec latency 37.981 ms 2024-03-19T02:36:51.791 INFO:tasks.workunit.client.0.smithi116.stdout: 1 292348 3.15 MB/sec execute 374 sec latency 27.325 ms 2024-03-19T02:36:52.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:51 smithi154 ceph-mon[35693]: pgmap v280: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:51 smithi116 ceph-mon[32263]: pgmap v280: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:51 smithi162 ceph-mon[35761]: pgmap v280: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:52.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 292879 3.15 MB/sec execute 375 sec latency 26.089 ms 2024-03-19T02:36:53.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 293470 3.15 MB/sec execute 376 sec latency 17.710 ms 2024-03-19T02:36:53.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:36:53.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:36:54.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 -- 172.21.15.116:0/2807211958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804106750 msgr2=0x7f1804106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 --2- 172.21.15.116:0/2807211958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804106750 0x7f1804106b30 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009930 tx=0x7f17f402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 -- 172.21.15.116:0/2807211958 shutdown_connections 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 --2- 172.21.15.116:0/2807211958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804101100 0x7f180410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 --2- 172.21.15.116:0/2807211958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 --2- 172.21.15.116:0/2807211958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804106750 0x7f1804106b30 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 -- 172.21.15.116:0/2807211958 >> 172.21.15.116:0/2807211958 conn(0x7f1804077880 msgr2=0x7f1804077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 -- 172.21.15.116:0/2807211958 shutdown_connections 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.070+0000 7f180a4ea640 1 -- 172.21.15.116:0/2807211958 wait complete. 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 Processor -- start 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 -- start start 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804106750 0x7f180419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1804111b10 con 0x7f1804101100 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1804111990 con 0x7f1804100740 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f180a4ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1804111c90 con 0x7f1804106750 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54324/0 (socket says 172.21.15.116:54324) 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 -- 172.21.15.116:0/3184996223 learned_addr learned my addr 172.21.15.116:0/3184996223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f1803fff640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f1808a87640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804106750 0x7f180419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 -- 172.21.15.116:0/3184996223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804106750 msgr2=0x7f180419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804106750 0x7f180419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 -- 172.21.15.116:0/3184996223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 msgr2=0x7f1804198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f18037fe640 1 -- 172.21.15.116:0/3184996223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180419d020 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.071+0000 7f1803fff640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f18037fe640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f17f800e9f0 tx=0x7f17f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f18017fa640 1 -- 172.21.15.116:0/3184996223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f800cd80 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180419d310 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f18017fa640 1 -- 172.21.15.116:0/3184996223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17f80108e0 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f18017fa640 1 -- 172.21.15.116:0/3184996223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8018920 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.072+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18041a4ba0 con 0x7f1804101100 2024-03-19T02:36:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.074+0000 7f18017fa640 1 -- 172.21.15.116:0/3184996223 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f17f8010a90 con 0x7f1804101100 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 Processor -- start 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 -- start start 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18017fa640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 0x7f17ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 0x559b3e5a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 0x559b3e5b8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b3e50c500 con 0x559b3e5a9080 2024-03-19T02:36:54.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559b3e4b4be0 con 0x559b3e5a8760 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f180d1b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559b3e5085a0 con 0x559b3e5b6340 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f1808a87640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 0x7f17ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f1803fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 0x559b3e5a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54326/0 (socket says 172.21.15.116:54326) 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 -- 172.21.15.116:0/2654522769 learned_addr learned my addr 172.21.15.116:0/2654522769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f1808a87640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 0x559b3e5b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18017fa640 1 -- 172.21.15.116:0/3184996223 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f17f80cf4a0 con 0x7f1804101100 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 -- 172.21.15.116:0/2654522769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 msgr2=0x559b3e5b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 0x559b3e5b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 -- 172.21.15.116:0/2654522769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 msgr2=0x559b3e5a8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 0x559b3e5a8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 -- 172.21.15.116:0/2654522769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b3e4d6000 con 0x559b3e5a9080 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f1808a87640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 0x559b3e5b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f1803fff640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 0x559b3e5a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.075+0000 7f18037fe640 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c6a0 tx=0x7f17f809b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2c391f7c060c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f1808a87640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 0x7f17ec0a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x559b3e5a42f0 tx=0x7f17fc009a90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f17ea7fc640 1 -- 172.21.15.116:0/2654522769 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80b4880 con 0x559b3e5a9080 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f17ea7fc640 1 -- 172.21.15.116:0/2654522769 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17f8010de0 con 0x559b3e5a9080 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f17ea7fc640 1 -- 172.21.15.116:0/2654522769 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8099c40 con 0x559b3e5a9080 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 -- 172.21.15.116:0/2654522769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 msgr2=0x559b3e5b5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c6a0 tx=0x7f17f809b3b0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 -- 172.21.15.116:0/2654522769 shutdown_connections 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5b6340 0x559b3e5b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5b5e00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 --2- 172.21.15.116:0/2654522769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a8760 0x559b3e5a8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 -- 172.21.15.116:0/2654522769 >> 172.21.15.116:0/2654522769 conn(0x559b3e5a3160 msgr2=0x559b3e5a3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 -- 172.21.15.116:0/2654522769 shutdown_connections 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.076+0000 7f180d1b3740 1 -- 172.21.15.116:0/2654522769 wait complete. 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 -- start start 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 Processor -- start 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 0x559b3e5b3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 0x559b3e5b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b3e5b90a0 con 0x559b3e5b6340 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559b3e4c8160 con 0x559b3e5a9080 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559b3e5b9220 con 0x559b3e5a8760 2024-03-19T02:36:54.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f180d1b3740 10 client.0 _ll_register_callbacks cb 0x559b3e504a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f1803fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f1808a87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 0x559b3e5b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f1803fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43318/0 (socket says 172.21.15.116:43318) 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f18037fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 0x559b3e5b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.077+0000 7f1803fff640 1 -- 172.21.15.116:0/213576241 learned_addr learned my addr 172.21.15.116:0/213576241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 -- 172.21.15.116:0/213576241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 msgr2=0x559b3e5b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 -- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 msgr2=0x559b3e5b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 0x559b3e5b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 -- 172.21.15.116:0/213576241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b3e2ea7d0 con 0x559b3e5b6340 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f18037fe640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 0x559b3e5b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f1808a87640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 0x559b3e5b3e90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f17fc005fd0 tx=0x7f17fc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc0c5194a957467 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f17e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc025070 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f17fc0045d0 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b3e519a40 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17fc031050 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f180d1b3740 10 client.18216 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b3e2203f0 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.078+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559b3e519a40 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17fc021410 con 0x559b3e5b6340 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 10 client.18216 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 10 client.18216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 20 client.18216 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5c6700 0x559b3e5c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 4 client.18216 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b3e513d90 con 0x559b3e5c6700 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f1803fff640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5c6700 0x559b3e5c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f1803fff640 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5c6700 0x559b3e5c8af0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d59e3c1bb89e4e7 server_cookie=33e61a40bff39253 in_seq=0 out_seq=0 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.079+0000 7f17e37fe640 10 client.18216 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.081+0000 7f17e37fe640 1 -- 172.21.15.116:0/213576241 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559b3e513d90 con 0x559b3e5c6700 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.081+0000 7f17e37fe640 10 client.18216 handle_command_reply: tid=0 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.081+0000 7f180d1b3740 1 client.18216 shutdown 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 msgr2=0x559b3e5b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 0x559b3e5b3e90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f17fc005fd0 tx=0x7f17fc009500 comp rx=0 tx=0).stop 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 shutdown_connections 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5c6700 0x559b3e5c8af0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5b6340 0x559b3e5b3e90 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f17e37fe640 0 client.18216 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a9080 0x559b3e5b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 --2- 172.21.15.116:0/213576241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 >> 172.21.15.116:0/213576241 conn(0x559b3e5a3160 msgr2=0x559b3e5a4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 shutdown_connections 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 1 -- 172.21.15.116:0/213576241 wait complete. 2024-03-19T02:36:54.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.082+0000 7f180d1b3740 20 client.18216 trim_cache size 0 max 16384 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 Processor -- start 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 -- start start 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18041acae0 con 0x559b3e5a9080 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18041ae620 con 0x559b3e5a3160 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f180d1b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18041ad1b0 con 0x559b3e5a8760 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54338/0 (socket says 172.21.15.116:54338) 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f18037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 -- 172.21.15.116:0/1799496473 learned_addr learned my addr 172.21.15.116:0/1799496473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1803fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 -- 172.21.15.116:0/1799496473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 msgr2=0x559b3e5c1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5c1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 -- 172.21.15.116:0/1799496473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 msgr2=0x559b3e5a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.103+0000 7f1808a87640 1 -- 172.21.15.116:0/1799496473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b3e5c8e60 con 0x559b3e5a9080 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f1808a87640 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f17fc004020 tx=0x7f17fc0137c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a274881a976dea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f17e9ffb640 1 -- 172.21.15.116:0/1799496473 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc03a070 con 0x559b3e5a9080 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f17e9ffb640 1 -- 172.21.15.116:0/1799496473 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17fc021a70 con 0x559b3e5a9080 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f17e9ffb640 1 -- 172.21.15.116:0/1799496473 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc02e070 con 0x559b3e5a9080 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 -- 172.21.15.116:0/1799496473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 msgr2=0x559b3e5a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f17fc004020 tx=0x7f17fc0137c0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 -- 172.21.15.116:0/1799496473 shutdown_connections 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a9080 0x559b3e5a4ca0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a8760 0x559b3e5c1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 --2- 172.21.15.116:0/1799496473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.104+0000 7f180d1b3740 1 -- 172.21.15.116:0/1799496473 >> 172.21.15.116:0/1799496473 conn(0x559b3e5aede0 msgr2=0x559b3e5a4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- 172.21.15.116:0/1799496473 shutdown_connections 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- 172.21.15.116:0/1799496473 wait complete. 2024-03-19T02:36:54.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- start start 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 Processor -- start 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5ba420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a9080 0x559b3e62b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18041adf50 con 0x559b3e5a8760 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18041ad880 con 0x559b3e5a3160 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18041aaae0 con 0x559b3e5a9080 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.105+0000 7f180d1b3740 10 client.0 _ll_register_callbacks cb 0x7f17fc013540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f1808a87640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a9080 0x559b3e62b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54342/0 (socket says 172.21.15.116:54342) 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 -- 172.21.15.116:0/3776951864 learned_addr learned my addr 172.21.15.116:0/3776951864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f1803fff640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5ba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 -- 172.21.15.116:0/3776951864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a9080 msgr2=0x559b3e62b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a9080 0x559b3e62b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 -- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 msgr2=0x559b3e5ba420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5ba420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 -- 172.21.15.116:0/3776951864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b3e5a8eb0 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f18037fe640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f17f809b800 tx=0x7f17f8097510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48177c425b61019f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f17e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80a3f60 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f17f80b45f0 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17fc021a70 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f17f809f910 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f180d1b3740 10 client.18222 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.106+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b3e1e00f0 con 0x559b3e5a8760 2024-03-19T02:36:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17fc021a70 con 0x559b3e5a8760 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17f80a3ad0 con 0x559b3e5a8760 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 10 client.18222 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 10 client.18222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 20 client.18222 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5b6340 0x559b3e62f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 4 client.18222 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b3e491540 con 0x559b3e5b6340 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f1803fff640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5b6340 0x559b3e62f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f1803fff640 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5b6340 0x559b3e62f060 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b87687edfd545c server_cookie=4415a6b0428f8b4f in_seq=0 out_seq=0 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.107+0000 7f17e37fe640 10 client.18222 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f17e37fe640 1 -- 172.21.15.116:0/3776951864 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559b3e491540 con 0x559b3e5b6340 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f17e37fe640 10 client.18222 handle_command_reply: tid=0 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 client.18222 shutdown 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 msgr2=0x559b3e5ba960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f17f809b800 tx=0x7f17f8097510 comp rx=0 tx=0).stop 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 shutdown_connections 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b3e5b6340 0x559b3e62f060 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b3e5a9080 0x559b3e62b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b3e5a8760 0x559b3e5ba960 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 --2- 172.21.15.116:0/3776951864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b3e5a3160 0x559b3e5ba420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f17e37fe640 0 client.18222 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 >> 172.21.15.116:0/3776951864 conn(0x559b3e5aede0 msgr2=0x559b3e5a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 shutdown_connections 2024-03-19T02:36:54.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 1 -- 172.21.15.116:0/3776951864 wait complete. 2024-03-19T02:36:54.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.108+0000 7f180d1b3740 20 client.18222 trim_cache size 0 max 16384 2024-03-19T02:36:54.116 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:36:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:36:54.117 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:54.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 msgr2=0x7f17ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 0x7f17ec0a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x559b3e5a42f0 tx=0x7f17fc009a90 comp rx=0 tx=0).stop 2024-03-19T02:36:54.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 msgr2=0x7f1804198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f17f800e9f0 tx=0x7f17f800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 shutdown_connections 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17ec0a2970 0x7f17ec0a4e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1804106750 0x7f180419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1804101100 0x7f1804198570 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 --2- 172.21.15.116:0/3184996223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1804100740 0x7f1804198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 >> 172.21.15.116:0/3184996223 conn(0x7f1804077880 msgr2=0x7f18041033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 shutdown_connections 2024-03-19T02:36:54.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.109+0000 7f180a4ea640 1 -- 172.21.15.116:0/3184996223 wait complete. 2024-03-19T02:36:54.120 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:36:54.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 -- 172.21.15.116:0/548208663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b81010e0 msgr2=0x7fe9b810d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 --2- 172.21.15.116:0/548208663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b81010e0 0x7fe9b810d5f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009cd0 tx=0x7fe9b002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 -- 172.21.15.116:0/548208663 shutdown_connections 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 --2- 172.21.15.116:0/548208663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b81010e0 0x7fe9b810d5f0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 --2- 172.21.15.116:0/548208663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8100720 0x7fe9b8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 --2- 172.21.15.116:0/548208663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b8106730 0x7fe9b8106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 -- 172.21.15.116:0/548208663 >> 172.21.15.116:0/548208663 conn(0x7fe9b8077820 msgr2=0x7fe9b8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 -- 172.21.15.116:0/548208663 shutdown_connections 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.229+0000 7fe9be268640 1 -- 172.21.15.116:0/548208663 wait complete. 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 Processor -- start 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 -- start start 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 0x7fe9b8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 0x7fe9b819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b81109f0 con 0x7fe9b8100720 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b8110870 con 0x7fe9b81010e0 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.230+0000 7fe9be268640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b8110b70 con 0x7fe9b8106730 2024-03-19T02:36:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54360/0 (socket says 172.21.15.116:54360) 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b77fe640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 0x7fe9b8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 -- 172.21.15.116:0/1600086097 learned_addr learned my addr 172.21.15.116:0/1600086097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9bc805640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 0x7fe9b819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 -- 172.21.15.116:0/1600086097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 msgr2=0x7fe9b819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 0x7fe9b819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 -- 172.21.15.116:0/1600086097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 msgr2=0x7fe9b8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 0x7fe9b8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 -- 172.21.15.116:0/1600086097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b819cfd0 con 0x7fe9b8100720 2024-03-19T02:36:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b77fe640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 0x7fe9b8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9bc805640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 0x7fe9b819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b7fff640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800e9f0 tx=0x7fe9a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a800cd80 con 0x7fe9b8100720 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b819d2c0 con 0x7fe9b8100720 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9a80108e0 con 0x7fe9b8100720 2024-03-19T02:36:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8018920 con 0x7fe9b8100720 2024-03-19T02:36:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.231+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b81a4b50 con 0x7fe9b8100720 2024-03-19T02:36:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.232+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b8068610 con 0x7fe9b8100720 2024-03-19T02:36:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.233+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe9a8010a90 con 0x7fe9b8100720 2024-03-19T02:36:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.234+0000 7fe9b57fa640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 0x7fe9a00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.234+0000 7fe9b77fe640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 0x7fe9a00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.234+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fe9a80cf4f0 con 0x7fe9b8100720 2024-03-19T02:36:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.234+0000 7fe9b77fe640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 0x7fe9a00a4e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8100300 tx=0x7fe9ac005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.239+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9a809b820 con 0x7fe9b8100720 2024-03-19T02:36:54.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:53 smithi154 ceph-mon[35693]: pgmap v281: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-03-19T02:36:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:53 smithi116 ceph-mon[32263]: pgmap v281: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-03-19T02:36:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:53 smithi162 ceph-mon[35761]: pgmap v281: 129 pgs: 129 active+clean; 292 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-03-19T02:36:54.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.496+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9b8109cc0 con 0x7fe9b8100720 2024-03-19T02:36:54.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.496+0000 7fe9b57fa640 1 -- 172.21.15.116:0/1600086097 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fe9a809f8c0 con 0x7fe9b8100720 2024-03-19T02:36:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:36:54.498 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:36:54.498 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:36:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 msgr2=0x7fe9a00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 0x7fe9a00a4e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8100300 tx=0x7fe9ac005f90 comp rx=0 tx=0).stop 2024-03-19T02:36:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 msgr2=0x7fe9b8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800e9f0 tx=0x7fe9a800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 shutdown_connections 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9a00a2970 0x7fe9a00a4e30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b8106730 0x7fe9b819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9b81010e0 0x7fe9b8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 --2- 172.21.15.116:0/1600086097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9b8100720 0x7fe9b8197fe0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 >> 172.21.15.116:0/1600086097 conn(0x7fe9b8077820 msgr2=0x7fe9b8068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 shutdown_connections 2024-03-19T02:36:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.499+0000 7fe9be268640 1 -- 172.21.15.116:0/1600086097 wait complete. 2024-03-19T02:36:54.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:36:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- 172.21.15.116:0/1240592065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 msgr2=0x7f60c8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 --2- 172.21.15.116:0/1240592065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8100b20 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009930 tx=0x7f60b402bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- 172.21.15.116:0/1240592065 shutdown_connections 2024-03-19T02:36:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 --2- 172.21.15.116:0/1240592065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 0x7f60c810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 --2- 172.21.15.116:0/1240592065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8100b20 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 --2- 172.21.15.116:0/1240592065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- 172.21.15.116:0/1240592065 >> 172.21.15.116:0/1240592065 conn(0x7f60c80778d0 msgr2=0x7f60c8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- 172.21.15.116:0/1240592065 shutdown_connections 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- 172.21.15.116:0/1240592065 wait complete. 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 Processor -- start 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.622+0000 7f60cf13b640 1 -- start start 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 0x7f60c8197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8110970 con 0x7f60c81006a0 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81107f0 con 0x7f60c8101060 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8110af0 con 0x7f60c81066b0 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54378/0 (socket says 172.21.15.116:54378) 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 -- 172.21.15.116:0/973442908 learned_addr learned my addr 172.21.15.116:0/973442908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60c7fff640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 0x7f60c8197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 -- 172.21.15.116:0/973442908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 msgr2=0x7f60c819b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c819b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 -- 172.21.15.116:0/973442908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 msgr2=0x7f60c8197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 0x7f60c8197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 -- 172.21.15.116:0/973442908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c819bee0 con 0x7f60c81006a0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c819b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cced7640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00e9f0 tx=0x7f60bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60c5ffb640 1 -- 172.21.15.116:0/973442908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00cd80 con 0x7f60c81006a0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c819c1d0 con 0x7f60c81006a0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60c5ffb640 1 -- 172.21.15.116:0/973442908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60bc0054d0 con 0x7f60c81006a0 2024-03-19T02:36:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.623+0000 7f60c5ffb640 1 -- 172.21.15.116:0/973442908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc010490 con 0x7f60c81006a0 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.624+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81a3a60 con 0x7f60c81006a0 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60c5ffb640 1 -- 172.21.15.116:0/973442908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f60bc005020 con 0x7f60c81006a0 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60c5ffb640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 0x7f60b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 Processor -- start 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 -- start start 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60c7fff640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 0x7f60b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60c5ffb640 1 -- 172.21.15.116:0/973442908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f60bc0cf530 con 0x7f60c81006a0 2024-03-19T02:36:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abada4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621aba324b0 con 0x5621abada9e0 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5621ab9dad70 con 0x5621abad14e0 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.626+0000 7f60d1e00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5621aba2e550 con 0x5621abad1e00 2024-03-19T02:36:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54386/0 (socket says 172.21.15.116:54386) 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cced7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 -- 172.21.15.116:0/514869858 learned_addr learned my addr 172.21.15.116:0/514869858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60c7fff640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 0x7f60b80a4e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60b4005ab0 tx=0x7f60b4005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60c7fff640 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abada4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 -- 172.21.15.116:0/514869858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 msgr2=0x5621abada4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abada4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 -- 172.21.15.116:0/514869858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 msgr2=0x5621abad18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 -- 172.21.15.116:0/514869858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621ab9fd210 con 0x5621abada9e0 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60c7fff640 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abada4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f60c00117e0 tx=0x7f60c0011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d46566774b34ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60b2ffd640 1 -- 172.21.15.116:0/514869858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0011310 con 0x5621abada9e0 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60b2ffd640 1 -- 172.21.15.116:0/514869858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c0003e20 con 0x5621abada9e0 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.627+0000 7f60b2ffd640 1 -- 172.21.15.116:0/514869858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c00044c0 con 0x5621abada9e0 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- 172.21.15.116:0/514869858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 msgr2=0x5621abadcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f60c00117e0 tx=0x7f60c0011ca0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- 172.21.15.116:0/514869858 shutdown_connections 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abadcdd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abada4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 --2- 172.21.15.116:0/514869858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- 172.21.15.116:0/514869858 >> 172.21.15.116:0/514869858 conn(0x5621abacbc10 msgr2=0x5621abacbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- 172.21.15.116:0/514869858 shutdown_connections 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- 172.21.15.116:0/514869858 wait complete. 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 -- start start 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.628+0000 7f60d1e00740 1 Processor -- start 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abad7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621abadd7d0 con 0x5621abada9e0 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5621ab9ee310 con 0x5621abad14e0 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5621abadd950 con 0x5621abad1e00 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60d1e00740 10 client.0 _ll_register_callbacks cb 0x5621ab9e22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cced7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54396/0 (socket says 172.21.15.116:54396) 2024-03-19T02:36:54.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 -- 172.21.15.116:0/2977588699 learned_addr learned my addr 172.21.15.116:0/2977588699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60c7fff640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abad7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 -- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 msgr2=0x5621abad7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abad7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 -- 172.21.15.116:0/2977588699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 msgr2=0x5621abad7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 -- 172.21.15.116:0/2977588699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621ab7463f0 con 0x5621abada9e0 2024-03-19T02:36:54.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cced7640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f60c0011370 tx=0x7f60c00113a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fb9feb2593d402 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.629+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0004e50 con 0x5621abada9e0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f60c000f9a0 con 0x5621abada9e0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c0003e20 con 0x5621abada9e0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60c000fba0 con 0x5621abada9e0 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 10 client.18240 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5621ab8107d0 con 0x5621abada9e0 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c0003e20 con 0x5621abada9e0 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f60c0018020 con 0x5621abada9e0 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 10 client.18240 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 10 client.18240 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 20 client.18240 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abaeab90 0x5621abaecf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 4 client.18240 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621aba903e0 con 0x5621abaeab90 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.630+0000 7f60cced7640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abaeab90 0x5621abaecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.631+0000 7f60cced7640 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abaeab90 0x5621abaecf80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a060980849babc02 server_cookie=f68728536cb831a6 in_seq=0 out_seq=0 2024-03-19T02:36:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.631+0000 7f60a7fff640 10 client.18240 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.631+0000 7f60a7fff640 1 -- 172.21.15.116:0/2977588699 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621aba903e0 con 0x5621abaeab90 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.631+0000 7f60a7fff640 10 client.18240 handle_command_reply: tid=0 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.631+0000 7f60d1e00740 1 client.18240 shutdown 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 msgr2=0x5621abad8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f60c0011370 tx=0x7f60c00113a0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 shutdown_connections 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abaeab90 0x5621abaecf80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abad8490 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad1e00 0x5621abad7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 --2- 172.21.15.116:0/2977588699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abad7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60a7fff640 0 client.18240 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 >> 172.21.15.116:0/2977588699 conn(0x5621abacbc10 msgr2=0x5621abac8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 shutdown_connections 2024-03-19T02:36:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 1 -- 172.21.15.116:0/2977588699 wait complete. 2024-03-19T02:36:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.632+0000 7f60d1e00740 20 client.18240 trim_cache size 0 max 16384 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 Processor -- start 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 -- start start 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abada0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abada9e0 0x5621abaf2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81acae0 con 0x5621abad1e00 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81ae620 con 0x5621abad14e0 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60d1e00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81ad1b0 con 0x5621abada9e0 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54398/0 (socket says 172.21.15.116:54398) 2024-03-19T02:36:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 -- 172.21.15.116:0/4258347456 learned_addr learned my addr 172.21.15.116:0/4258347456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60cced7640 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abada0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abada9e0 0x5621abaf2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 -- 172.21.15.116:0/4258347456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abada9e0 msgr2=0x5621abaf2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abada9e0 0x5621abaf2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 -- 172.21.15.116:0/4258347456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 msgr2=0x5621abada0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abada0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 -- 172.21.15.116:0/4258347456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621aba9f880 con 0x5621abad1e00 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.647+0000 7f60c7fff640 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009930 tx=0x7f60b403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6410271030aa2982 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60b27fc640 1 -- 172.21.15.116:0/4258347456 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4051070 con 0x5621abad1e00 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60b27fc640 1 -- 172.21.15.116:0/4258347456 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60b4045040 con 0x5621abad1e00 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60b27fc640 1 -- 172.21.15.116:0/4258347456 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b4004bb0 con 0x5621abad1e00 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 -- 172.21.15.116:0/4258347456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 msgr2=0x5621abae65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009930 tx=0x7f60b403e000 comp rx=0 tx=0).stop 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 -- 172.21.15.116:0/4258347456 shutdown_connections 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abada9e0 0x5621abaf2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abad1e00 0x5621abae65a0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 --2- 172.21.15.116:0/4258347456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad14e0 0x5621abada0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 -- 172.21.15.116:0/4258347456 >> 172.21.15.116:0/4258347456 conn(0x5621abacbc10 msgr2=0x5621abac8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 -- 172.21.15.116:0/4258347456 shutdown_connections 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.648+0000 7f60d1e00740 1 -- 172.21.15.116:0/4258347456 wait complete. 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 -- start start 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 Processor -- start 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad1e00 0x5621abb49670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81adf50 con 0x5621abada9e0 2024-03-19T02:36:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81ad880 con 0x5621abad1e00 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81aaae0 con 0x5621abad14e0 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60d1e00740 10 client.0 _ll_register_callbacks cb 0x7f60bc004aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60cced7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad1e00 0x5621abb49670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60cd6d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54402/0 (socket says 172.21.15.116:54402) 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60cd6d8640 1 -- 172.21.15.116:0/472538379 learned_addr learned my addr 172.21.15.116:0/472538379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.649+0000 7f60cced7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43414/0 (socket says 172.21.15.116:43414) 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 -- 172.21.15.116:0/472538379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad1e00 msgr2=0x5621abb49670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad1e00 0x5621abb49670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 -- 172.21.15.116:0/472538379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 msgr2=0x5621abaddc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 -- 172.21.15.116:0/472538379 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621abadd6b0 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cd6d8640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60cced7640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0cc8a0 tx=0x7f60bc086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e2ab11d30fac2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc0b8020 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60b4045040 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f60bc0860b0 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f60bc09b7b0 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60d1e00740 10 client.27985 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5621abb506e0 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.650+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60b4045040 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f60bc097e20 con 0x5621abad14e0 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 10 client.27985 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 10 client.27985 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:36:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 20 client.27985 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abb50e00 0x5621abb531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 4 client.27985 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621aba3f9f0 con 0x5621abb50e00 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60c7fff640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abb50e00 0x5621abb531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60c7fff640 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abb50e00 0x5621abb531f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87e0d1cd525ae32d server_cookie=b9931fd2edd2f789 in_seq=0 out_seq=0 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60a7fff640 10 client.27985 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60a7fff640 1 -- 172.21.15.116:0/472538379 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621aba3f9f0 con 0x5621abb50e00 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60a7fff640 10 client.27985 handle_command_reply: tid=0 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 client.27985 shutdown 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 msgr2=0x5621abb49130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0cc8a0 tx=0x7f60bc086040 comp rx=0 tx=0).stop 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 shutdown_connections 2024-03-19T02:36:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5621abb50e00 0x5621abb531f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621abada9e0 0x5621abaddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621abad1e00 0x5621abb49670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 --2- 172.21.15.116:0/472538379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5621abad14e0 0x5621abb49130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60a7fff640 0 client.27985 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.651+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 >> 172.21.15.116:0/472538379 conn(0x5621abacbc10 msgr2=0x5621abad79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 shutdown_connections 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60d1e00740 1 -- 172.21.15.116:0/472538379 wait complete. 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60d1e00740 20 client.27985 trim_cache size 0 max 16384 2024-03-19T02:36:54.659 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 msgr2=0x7f60b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 0x7f60b80a4e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60b4005ab0 tx=0x7f60b4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 msgr2=0x7f60c8196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00e9f0 tx=0x7f60bc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 shutdown_connections 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60b80a2970 0x7f60b80a4e30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f60c81066b0 0x7f60c819b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60c8101060 0x7f60c8197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 --2- 172.21.15.116:0/973442908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60c81006a0 0x7f60c8196ef0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 >> 172.21.15.116:0/973442908 conn(0x7f60c80778d0 msgr2=0x7f60c8102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 shutdown_connections 2024-03-19T02:36:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.652+0000 7f60cf13b640 1 -- 172.21.15.116:0/973442908 wait complete. 2024-03-19T02:36:54.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:36:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- 172.21.15.116:0/2999821908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81077b0 msgr2=0x7ff7c8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2999821908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81077b0 0x7ff7c8107b90 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009930 tx=0x7ff7bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- 172.21.15.116:0/2999821908 shutdown_connections 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2999821908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c8102160 0x7ff7c810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2999821908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c81017a0 0x7ff7c8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2999821908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81077b0 0x7ff7c8107b90 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- 172.21.15.116:0/2999821908 >> 172.21.15.116:0/2999821908 conn(0x7ff7c80fd2d0 msgr2=0x7ff7c80ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- 172.21.15.116:0/2999821908 shutdown_connections 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- 172.21.15.116:0/2999821908 wait complete. 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 Processor -- start 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.776+0000 7ff7cf236640 1 -- start start 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 0x7ff7c8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c81077b0 0x7ff7c819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8111a70 con 0x7ff7c81017a0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81118f0 con 0x7ff7c8102160 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8111bf0 con 0x7ff7c81077b0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54414/0 (socket says 172.21.15.116:54414) 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 -- 172.21.15.116:0/2217968516 learned_addr learned my addr 172.21.15.116:0/2217968516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 0x7ff7c8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c81077b0 0x7ff7c819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 -- 172.21.15.116:0/2217968516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c81077b0 msgr2=0x7ff7c819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c81077b0 0x7ff7c819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 -- 172.21.15.116:0/2217968516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 msgr2=0x7ff7c8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 0x7ff7c8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 -- 172.21.15.116:0/2217968516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c819cfc0 con 0x7ff7c81017a0 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 0x7ff7c8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7ccfd2640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009230 tx=0x7ff7bc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7c5ffb640 1 -- 172.21.15.116:0/2217968516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc043020 con 0x7ff7c81017a0 2024-03-19T02:36:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c819d250 con 0x7ff7c81017a0 2024-03-19T02:36:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7c5ffb640 1 -- 172.21.15.116:0/2217968516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7bc02ca70 con 0x7ff7c81017a0 2024-03-19T02:36:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.777+0000 7ff7c5ffb640 1 -- 172.21.15.116:0/2217968516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc003a50 con 0x7ff7c81017a0 2024-03-19T02:36:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.778+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c81a4b40 con 0x7ff7c81017a0 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 Processor -- start 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 -- start start 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7c5ffb640 1 -- 172.21.15.116:0/2217968516 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff7bc046020 con 0x7ff7c81017a0 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 0x563939a20290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 0x563939a22bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56393997a4b0 con 0x563939a17bf0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563939922d70 con 0x563939a172d0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.780+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563939976550 con 0x563939a207d0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7ccfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 0x563939a20290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:42254/0 (socket says 172.21.15.116:42254) 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1616657037 learned_addr learned my addr 172.21.15.116:0/1616657037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7ccfd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 0x563939a20290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54418/0 (socket says 172.21.15.116:54418) 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 0x563939a22bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1616657037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 msgr2=0x563939a22bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 0x563939a22bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1616657037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 msgr2=0x563939a20290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 0x563939a20290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1616657037 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563939945210 con 0x563939a172d0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 0x563939a22bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7c5ffb640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 0x7ff7b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7ccfd2640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 0x7ff7b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009f50 tx=0x7ff7c00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e44fd342d837fca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7c5ffb640 1 -- 172.21.15.116:0/2217968516 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff7bc0f1120 con 0x7ff7c81017a0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7b2ffd640 1 -- 172.21.15.116:0/1616657037 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0004d20 con 0x563939a172d0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7b2ffd640 1 -- 172.21.15.116:0/1616657037 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7c0004050 con 0x563939a172d0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.781+0000 7ff7b2ffd640 1 -- 172.21.15.116:0/1616657037 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0005280 con 0x563939a172d0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1616657037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 msgr2=0x563939a176b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7ccfd2640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 0x7ff7b80a4e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0ee650 tx=0x7ff7bc0f4c70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009f50 tx=0x7ff7c00118d0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1616657037 shutdown_connections 2024-03-19T02:36:54.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a207d0 0x563939a22bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a17bf0 0x563939a20290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1616657037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a176b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1616657037 >> 172.21.15.116:0/1616657037 conn(0x563939a11a00 msgr2=0x563939a11de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1616657037 shutdown_connections 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.782+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1616657037 wait complete. 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 -- start start 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 Processor -- start 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a1dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563939a235c0 con 0x563939a207d0 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563939936310 con 0x563939a17bf0 2024-03-19T02:36:54.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563939a23740 con 0x563939a172d0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7d1e25740 10 client.0 _ll_register_callbacks cb 0x56393992a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54422/0 (socket says 172.21.15.116:54422) 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/3790737134 learned_addr learned my addr 172.21.15.116:0/3790737134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a1dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 msgr2=0x563939a1d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/3790737134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 msgr2=0x563939a1dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a1dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.783+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/3790737134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56393968e3f0 con 0x563939a207d0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00044d0 tx=0x7ff7c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289a6369bd8065f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0004530 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff7c0004c50 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c0004050 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7c0028450 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 10 client.18258 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5639397587d0 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7c0004050 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff7c000f610 con 0x563939a207d0 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 10 client.18258 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 10 client.18258 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 20 client.18258 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a309f0 0x563939a32de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 4 client.18258 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:54.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.784+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639399d83e0 con 0x563939a309f0 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.785+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a309f0 0x563939a32de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.785+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a309f0 0x563939a32de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b984474a50ba8685 server_cookie=b113300c0124ca4b in_seq=0 out_seq=0 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.785+0000 7ff7a7fff640 10 client.18258 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7a7fff640 1 -- 172.21.15.116:0/3790737134 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639399d83e0 con 0x563939a309f0 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7a7fff640 10 client.18258 handle_command_reply: tid=0 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 client.18258 shutdown 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 msgr2=0x563939a1e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00044d0 tx=0x7ff7c00047b0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 shutdown_connections 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a309f0 0x563939a32de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a1e240 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a1dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/3790737134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7a7fff640 0 client.18258 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 >> 172.21.15.116:0/3790737134 conn(0x563939a11a00 msgr2=0x563939a13750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 shutdown_connections 2024-03-19T02:36:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 1 -- 172.21.15.116:0/3790737134 wait complete. 2024-03-19T02:36:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.786+0000 7ff7d1e25740 20 client.18258 trim_cache size 0 max 16384 2024-03-19T02:36:54.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 294194 3.15 MB/sec execute 377 sec latency 16.599 ms 2024-03-19T02:36:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 Processor -- start 2024-03-19T02:36:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 -- start start 2024-03-19T02:36:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a2c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81aed00 con 0x563939a207d0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81b0840 con 0x563939a17bf0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81af3d0 con 0x563939a172d0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7c7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a2c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54428/0 (socket says 172.21.15.116:54428) 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/2584517164 learned_addr learned my addr 172.21.15.116:0/2584517164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/2584517164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 msgr2=0x563939a1fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/2584517164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 msgr2=0x563939a2c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a2c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/2584517164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563939a11830 con 0x563939a207d0 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.801+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a2c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005d70 tx=0x7ff7c00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d9bb9b6ff386d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7b27fc640 1 -- 172.21.15.116:0/2584517164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037070 con 0x563939a207d0 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7b27fc640 1 -- 172.21.15.116:0/2584517164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7c0038040 con 0x563939a207d0 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7b27fc640 1 -- 172.21.15.116:0/2584517164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c000fe20 con 0x563939a207d0 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 -- 172.21.15.116:0/2584517164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 msgr2=0x563939a38550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005d70 tx=0x7ff7c00118d0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 -- 172.21.15.116:0/2584517164 shutdown_connections 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a38550 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a17bf0 0x563939a2c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/2584517164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a172d0 0x563939a1fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.802+0000 7ff7d1e25740 1 -- 172.21.15.116:0/2584517164 >> 172.21.15.116:0/2584517164 conn(0x563939a11a00 msgr2=0x563939a10ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- 172.21.15.116:0/2584517164 shutdown_connections 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- 172.21.15.116:0/2584517164 wait complete. 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- start start 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 Processor -- start 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a17bf0 0x563939a24970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81b0170 con 0x563939a207d0 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81afaa0 con 0x563939a172d0 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81acd00 con 0x563939a17bf0 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7d1e25740 10 client.0 _ll_register_callbacks cb 0x7ff7bc0f4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7cd7d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54436/0 (socket says 172.21.15.116:54436) 2024-03-19T02:36:54.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1957652303 learned_addr learned my addr 172.21.15.116:0/1957652303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a17bf0 0x563939a24970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.803+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a17bf0 msgr2=0x563939a24970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a17bf0 0x563939a24970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1957652303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 msgr2=0x563939a24430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a24430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 -- 172.21.15.116:0/1957652303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563939a95ce0 con 0x563939a207d0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7cd7d3640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000fb20 tx=0x7ff7c0003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5842160ad463614f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037070 con 0x563939a207d0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff7c0009dc0 con 0x563939a207d0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c0038040 con 0x563939a207d0 2024-03-19T02:36:54.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7c0003af0 con 0x563939a207d0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7d1e25740 10 client.18264 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.804+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563939a96fc0 con 0x563939a207d0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7c0038040 con 0x563939a207d0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff7c0027440 con 0x563939a207d0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 10 client.18264 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 10 client.18264 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 20 client.18264 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a97630 0x563939a99a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 4 client.18264 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639399879f0 con 0x563939a97630 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a97630 0x563939a99a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7c7fff640 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a97630 0x563939a99a20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=40773f8fa8036c3 server_cookie=5ecafa26c4afac1b in_seq=0 out_seq=0 2024-03-19T02:36:54.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.805+0000 7ff7a7fff640 10 client.18264 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7a7fff640 1 -- 172.21.15.116:0/1957652303 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639399879f0 con 0x563939a97630 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7a7fff640 10 client.18264 handle_command_reply: tid=0 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 client.18264 shutdown 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 msgr2=0x563939a24eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000fb20 tx=0x7ff7c0003fd0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 shutdown_connections 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x563939a97630 0x563939a99a20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563939a207d0 0x563939a24eb0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7a7fff640 0 client.18264 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563939a17bf0 0x563939a24970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 --2- 172.21.15.116:0/1957652303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563939a172d0 0x563939a24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 >> 172.21.15.116:0/1957652303 conn(0x563939a11a00 msgr2=0x563939a1d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 shutdown_connections 2024-03-19T02:36:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 1 -- 172.21.15.116:0/1957652303 wait complete. 2024-03-19T02:36:54.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7d1e25740 20 client.18264 trim_cache size 0 max 16384 2024-03-19T02:36:54.814 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.806+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 msgr2=0x7ff7b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 0x7ff7b80a4e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0ee650 tx=0x7ff7bc0f4c70 comp rx=0 tx=0).stop 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 msgr2=0x7ff7c8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009230 tx=0x7ff7bc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 shutdown_connections 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff7b80a2970 0x7ff7b80a4e30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7c81077b0 0x7ff7c819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7c8102160 0x7ff7c8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 --2- 172.21.15.116:0/2217968516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7c81017a0 0x7ff7c8197fd0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 >> 172.21.15.116:0/2217968516 conn(0x7ff7c80fd2d0 msgr2=0x7ff7c80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 shutdown_connections 2024-03-19T02:36:54.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.807+0000 7ff7cf236640 1 -- 172.21.15.116:0/2217968516 wait complete. 2024-03-19T02:36:54.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:36:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.927+0000 7f46fc3c4640 1 -- 172.21.15.116:0/4018497056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 msgr2=0x7f46f4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.927+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/4018497056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f4106a90 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009930 tx=0x7f46f002bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- 172.21.15.116:0/4018497056 shutdown_connections 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/4018497056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 0x7f46f410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/4018497056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/4018497056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f4106a90 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- 172.21.15.116:0/4018497056 >> 172.21.15.116:0/4018497056 conn(0x7f46f40778d0 msgr2=0x7f46f4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- 172.21.15.116:0/4018497056 shutdown_connections 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- 172.21.15.116:0/4018497056 wait complete. 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 Processor -- start 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- start start 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 0x7f46f4199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4110970 con 0x7f46f41066b0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41107f0 con 0x7f46f4101060 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.928+0000 7f46fc3c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4110af0 con 0x7f46f41006a0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54450/0 (socket says 172.21.15.116:54450) 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa160640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 -- 172.21.15.116:0/1393689672 learned_addr learned my addr 172.21.15.116:0/1393689672 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46f995f640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 0x7f46f4199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 -- 172.21.15.116:0/1393689672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 msgr2=0x7f46f4198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 -- 172.21.15.116:0/1393689672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 msgr2=0x7f46f4199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 0x7f46f4199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 -- 172.21.15.116:0/1393689672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f41a4410 con 0x7f46f41066b0 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa160640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fa961640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00eb10 tx=0x7f46ec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46eb7fe640 1 -- 172.21.15.116:0/1393689672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec004b20 con 0x7f46f41066b0 2024-03-19T02:36:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46eb7fe640 1 -- 172.21.15.116:0/1393689672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46ec004cc0 con 0x7f46f41066b0 2024-03-19T02:36:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a45e0 con 0x7f46f41066b0 2024-03-19T02:36:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46eb7fe640 1 -- 172.21.15.116:0/1393689672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec010690 con 0x7f46f41066b0 2024-03-19T02:36:54.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.929+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a4a50 con 0x7f46f41066b0 2024-03-19T02:36:54.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.931+0000 7f46eb7fe640 1 -- 172.21.15.116:0/1393689672 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f46ec005020 con 0x7f46f41066b0 2024-03-19T02:36:54.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 Processor -- start 2024-03-19T02:36:54.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46eb7fe640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46f995f640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 -- start start 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46eb7fe640 1 -- 172.21.15.116:0/1393689672 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f46ec0cf790 con 0x7f46f41066b0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add8bf0 0x558b9ade1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ade3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b9ad3b4b0 con 0x558b9add82d0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b9ace3d70 con 0x558b9add8bf0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.932+0000 7f46fefaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b9ad37550 con 0x558b9ade17d0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa961640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ade3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54464/0 (socket says 172.21.15.116:54464) 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 -- 172.21.15.116:0/1110021633 learned_addr learned my addr 172.21.15.116:0/1110021633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46f995f640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46f4069480 tx=0x7f46e4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46f995f640 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add8bf0 0x558b9ade1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 -- 172.21.15.116:0/1110021633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 msgr2=0x558b9ade3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ade3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 -- 172.21.15.116:0/1110021633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add8bf0 msgr2=0x558b9ade1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add8bf0 0x558b9ade1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 -- 172.21.15.116:0/1110021633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b9ad06210 con 0x558b9add82d0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa961640 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ade3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46fa160640 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f46f0036040 tx=0x7f46f00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c226ebd00a08135 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46e8bfe640 1 -- 172.21.15.116:0/1110021633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0045070 con 0x558b9add82d0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46e8bfe640 1 -- 172.21.15.116:0/1110021633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.933+0000 7f46e8bfe640 1 -- 172.21.15.116:0/1110021633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0003a70 con 0x558b9add82d0 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 -- 172.21.15.116:0/1110021633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 msgr2=0x558b9add86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f46f0036040 tx=0x7f46f00095f0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 -- 172.21.15.116:0/1110021633 shutdown_connections 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ade3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add8bf0 0x558b9ade1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 --2- 172.21.15.116:0/1110021633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9add86b0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 -- 172.21.15.116:0/1110021633 >> 172.21.15.116:0/1110021633 conn(0x558b9add2a00 msgr2=0x558b9add2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 -- 172.21.15.116:0/1110021633 shutdown_connections 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.934+0000 7f46fefaa740 1 -- 172.21.15.116:0/1110021633 wait complete. 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 -- start start 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 Processor -- start 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9addedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9addf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b9ade4560 con 0x558b9add82d0 2024-03-19T02:36:54.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b9acf7310 con 0x558b9ade17d0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b9ade46e0 con 0x558b9add8bf0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fefaa740 10 client.0 _ll_register_callbacks cb 0x558b9aceb2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54476/0 (socket says 172.21.15.116:54476) 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 -- 172.21.15.116:0/242214670 learned_addr learned my addr 172.21.15.116:0/242214670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46f995f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9addedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa961640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9addf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 -- 172.21.15.116:0/242214670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 msgr2=0x558b9addedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9addedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 -- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 msgr2=0x558b9addf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.935+0000 7f46fa160640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9addf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46fa160640 1 -- 172.21.15.116:0/242214670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b9aa4f3f0 con 0x558b9add82d0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46fa160640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006010 tx=0x7f46f003f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa15f34465dbfa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0045040 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f46f003fc70 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f46f0059050 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46fefaa740 10 client.18276 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558b9ab197d0 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.936+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f46f0034440 con 0x558b9add82d0 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 10 client.18276 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 10 client.18276 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 20 client.18276 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9adf1a00 0x558b9adf3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 4 client.18276 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:54.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b9ad993e0 con 0x558b9adf1a00 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46f995f640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9adf1a00 0x558b9adf3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46f995f640 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9adf1a00 0x558b9adf3df0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a1742d0d9ea5a134 server_cookie=f30a13e7e91d447f in_seq=0 out_seq=0 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.937+0000 7f46ddffb640 10 client.18276 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46ddffb640 1 -- 172.21.15.116:0/242214670 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b9ad993e0 con 0x558b9adf1a00 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46ddffb640 10 client.18276 handle_command_reply: tid=0 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 client.18276 shutdown 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 msgr2=0x558b9adde890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f46f0006010 tx=0x7f46f003f670 comp rx=0 tx=0).stop 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 shutdown_connections 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9adf1a00 0x558b9adf3df0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9addf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9addedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 --2- 172.21.15.116:0/242214670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9adde890 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 >> 172.21.15.116:0/242214670 conn(0x558b9add2a00 msgr2=0x558b9add4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.938+0000 7f46ddffb640 0 client.18276 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.939+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 shutdown_connections 2024-03-19T02:36:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.939+0000 7f46fefaa740 1 -- 172.21.15.116:0/242214670 wait complete. 2024-03-19T02:36:54.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.939+0000 7f46fefaa740 20 client.18276 trim_cache size 0 max 16384 2024-03-19T02:36:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.954+0000 7f46fefaa740 1 Processor -- start 2024-03-19T02:36:54.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.954+0000 7f46fefaa740 1 -- start start 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.954+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9adf9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9ade9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fefaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41aec30 con 0x558b9add82d0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fefaa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41b0770 con 0x558b9ade17d0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fefaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41af300 con 0x558b9add8bf0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54492/0 (socket says 172.21.15.116:54492) 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 -- 172.21.15.116:0/2956046857 learned_addr learned my addr 172.21.15.116:0/2956046857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46f995f640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9adf9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa961640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9ade9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 -- 172.21.15.116:0/2956046857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 msgr2=0x558b9adf9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9adf9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 -- 172.21.15.116:0/2956046857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 msgr2=0x558b9ade9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9ade9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 -- 172.21.15.116:0/2956046857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b9ade4690 con 0x558b9add82d0 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46f995f640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9adf9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46fa160640 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f46f002c170 tx=0x7f46f00095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f26653246fbabb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46dffff640 1 -- 172.21.15.116:0/2956046857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0045040 con 0x558b9add82d0 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46dffff640 1 -- 172.21.15.116:0/2956046857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.955+0000 7f46dffff640 1 -- 172.21.15.116:0/2956046857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f003fdf0 con 0x558b9add82d0 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- 172.21.15.116:0/2956046857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 msgr2=0x558b9aded300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f46f002c170 tx=0x7f46f00095f0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- 172.21.15.116:0/2956046857 shutdown_connections 2024-03-19T02:36:54.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9ade17d0 0x558b9ade9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9add8bf0 0x558b9adf9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 --2- 172.21.15.116:0/2956046857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add82d0 0x558b9aded300 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- 172.21.15.116:0/2956046857 >> 172.21.15.116:0/2956046857 conn(0x558b9add2a00 msgr2=0x558b9add1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- 172.21.15.116:0/2956046857 shutdown_connections 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- 172.21.15.116:0/2956046857 wait complete. 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 -- start start 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.956+0000 7f46fefaa740 1 Processor -- start 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 0x558b9ade6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 0x558b9ade7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ae4ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41b00a0 con 0x558b9add8bf0 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41af9d0 con 0x558b9add82d0 2024-03-19T02:36:54.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41acc30 con 0x558b9ade17d0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fefaa740 10 client.0 _ll_register_callbacks cb 0x7f46ec084f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46f995f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 0x558b9ade7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 0x558b9ade6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46f995f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 0x558b9ade7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54508/0 (socket says 172.21.15.116:54508) 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46f995f640 1 -- 172.21.15.116:0/587268886 learned_addr learned my addr 172.21.15.116:0/587268886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa961640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ae4ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 -- 172.21.15.116:0/587268886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 msgr2=0x558b9ae4ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ae4ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 -- 172.21.15.116:0/587268886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 msgr2=0x558b9ade7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 0x558b9ade7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 -- 172.21.15.116:0/587268886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b9ae4f0b0 con 0x558b9add82d0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46fa160640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 0x558b9ade6f20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46f0000c00 tx=0x7f46f0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3014aab47988fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:36:54.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.957+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0045040 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f46f0034ad0 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f46f006f050 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 10 client.28076 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558b9ae57650 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46f002c7e0 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f46f004a9e0 con 0x558b9add82d0 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 10 client.28076 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 10 client.28076 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 20 client.28076 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9ae57c30 0x558b9ae5a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 4 client.28076 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b9ad489f0 con 0x558b9ae57c30 2024-03-19T02:36:54.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.958+0000 7f46f995f640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9ae57c30 0x558b9ae5a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46ddffb640 10 client.28076 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46f995f640 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9ae57c30 0x558b9ae5a050 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=abf74e82c2ac36ce server_cookie=c74e25f69a0779ae in_seq=0 out_seq=0 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46ddffb640 1 -- 172.21.15.116:0/587268886 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b9ad489f0 con 0x558b9ae57c30 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46ddffb640 10 client.28076 handle_command_reply: tid=0 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 client.28076 shutdown 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 msgr2=0x558b9ade6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 0x558b9ade6f20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46f0000c00 tx=0x7f46f0009780 comp rx=0 tx=0).stop 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 shutdown_connections 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558b9ae57c30 0x558b9ae5a050 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b9ade17d0 0x558b9ae4ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46ddffb640 0 client.28076 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b9add8bf0 0x558b9ade7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 --2- 172.21.15.116:0/587268886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b9add82d0 0x558b9ade6f20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 >> 172.21.15.116:0/587268886 conn(0x558b9add2a00 msgr2=0x558b9add3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 shutdown_connections 2024-03-19T02:36:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 1 -- 172.21.15.116:0/587268886 wait complete. 2024-03-19T02:36:54.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.959+0000 7f46fefaa740 20 client.28076 trim_cache size 0 max 16384 2024-03-19T02:36:54.967 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 msgr2=0x7f46e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46f4069480 tx=0x7f46e4005f90 comp rx=0 tx=0).stop 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 msgr2=0x7f46f41a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00eb10 tx=0x7f46ec00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 shutdown_connections 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f41a3ed0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f4101060 0x7f46f4199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 --2- 172.21.15.116:0/1393689672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41006a0 0x7f46f4198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 >> 172.21.15.116:0/1393689672 conn(0x7f46f40778d0 msgr2=0x7f46f4102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 shutdown_connections 2024-03-19T02:36:54.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:54.960+0000 7f46fc3c4640 1 -- 172.21.15.116:0/1393689672 wait complete. 2024-03-19T02:36:54.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 -- 172.21.15.116:0/3174280702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181006a0 msgr2=0x7f6318100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 --2- 172.21.15.116:0/3174280702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181006a0 0x7f6318100b20 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6300009930 tx=0x7f630002bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 -- 172.21.15.116:0/3174280702 shutdown_connections 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 --2- 172.21.15.116:0/3174280702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6318101060 0x7f631810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 --2- 172.21.15.116:0/3174280702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181006a0 0x7f6318100b20 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 --2- 172.21.15.116:0/3174280702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63181066b0 0x7f6318106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 -- 172.21.15.116:0/3174280702 >> 172.21.15.116:0/3174280702 conn(0x7f63180778d0 msgr2=0x7f6318077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 -- 172.21.15.116:0/3174280702 shutdown_connections 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.083+0000 7f631ca7e640 1 -- 172.21.15.116:0/3174280702 wait complete. 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 Processor -- start 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 -- start start 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6318101060 0x7f63181984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6318110970 con 0x7f63181066b0 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63181107f0 con 0x7f63181006a0 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631ca7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6318110af0 con 0x7f6318101060 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54512/0 (socket says 172.21.15.116:54512) 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f6316d9d640 1 -- 172.21.15.116:0/2658813294 learned_addr learned my addr 172.21.15.116:0/2658813294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f630ffff640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6318101060 0x7f63181984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.084+0000 7f631659c640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 -- 172.21.15.116:0/2658813294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6318101060 msgr2=0x7f63181984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6318101060 0x7f63181984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 -- 172.21.15.116:0/2658813294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 msgr2=0x7f6318197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 -- 172.21.15.116:0/2658813294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631819cf90 con 0x7f63181066b0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f6316d9d640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f631000eb10 tx=0x7f631000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f630f7fe640 1 -- 172.21.15.116:0/2658813294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310004b20 con 0x7f63181066b0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631819d280 con 0x7f63181066b0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f630f7fe640 1 -- 172.21.15.116:0/2658813294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6310004cc0 con 0x7f63181066b0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f630f7fe640 1 -- 172.21.15.116:0/2658813294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310010690 con 0x7f63181066b0 2024-03-19T02:36:55.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.085+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63181a4b10 con 0x7f63181066b0 2024-03-19T02:36:55.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f630f7fe640 1 -- 172.21.15.116:0/2658813294 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6310005020 con 0x7f63181066b0 2024-03-19T02:36:55.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 Processor -- start 2024-03-19T02:36:55.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 -- start start 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f630f7fe640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 0x7f63040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560f69ec74b0 con 0x560f69f642d0 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f630f7fe640 1 -- 172.21.15.116:0/2658813294 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f63100cf7c0 con 0x7f63181066b0 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f69e6fd70 con 0x560f69f64bf0 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f631f661740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f69ec3550 con 0x560f69f6d7d0 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.088+0000 7f630ffff640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 0x7f63040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43556/0 (socket says 172.21.15.116:43556) 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 -- 172.21.15.116:0/4035031000 learned_addr learned my addr 172.21.15.116:0/4035031000 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f630ffff640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f631659c640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 -- 172.21.15.116:0/4035031000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 msgr2=0x560f69f6d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 -- 172.21.15.116:0/4035031000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 msgr2=0x560f69f646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 -- 172.21.15.116:0/4035031000 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f69e92210 con 0x560f69f6d7d0 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f631659c640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f630ffff640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 0x7f63040a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6300005ab0 tx=0x7f630002c6a0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f6316d9d640 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6310098410 tx=0x7f631009b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db35a8d1d2e9b40c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f630cff9640 1 -- 172.21.15.116:0/4035031000 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631009ba40 con 0x560f69f6d7d0 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f630cff9640 1 -- 172.21.15.116:0/4035031000 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f631009fb90 con 0x560f69f6d7d0 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.089+0000 7f630cff9640 1 -- 172.21.15.116:0/4035031000 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63100a3f70 con 0x560f69f6d7d0 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- 172.21.15.116:0/4035031000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 msgr2=0x560f69f6fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6310098410 tx=0x7f631009b580 comp rx=0 tx=0).stop 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- 172.21.15.116:0/4035031000 shutdown_connections 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6fbc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 --2- 172.21.15.116:0/4035031000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- 172.21.15.116:0/4035031000 >> 172.21.15.116:0/4035031000 conn(0x560f69f5ea00 msgr2=0x560f69f5ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- 172.21.15.116:0/4035031000 shutdown_connections 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- 172.21.15.116:0/4035031000 wait complete. 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.090+0000 7f631f661740 1 -- start start 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 Processor -- start 2024-03-19T02:36:55.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560f69f705c0 con 0x560f69f642d0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f69e83310 con 0x560f69f64bf0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f69f70740 con 0x560f69f6d7d0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631f661740 10 client.0 _ll_register_callbacks cb 0x560f69e772d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54526/0 (socket says 172.21.15.116:54526) 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 -- 172.21.15.116:0/3816302954 learned_addr learned my addr 172.21.15.116:0/3816302954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f630ffff640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f6316d9d640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 -- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 msgr2=0x560f69f6b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 -- 172.21.15.116:0/3816302954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 msgr2=0x560f69f6add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 -- 172.21.15.116:0/3816302954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f69bdb3f0 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f630ffff640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f631659c640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6308009a80 tx=0x7f6308009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a181611e641b1f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f62fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.091+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308023060 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f631009fb90 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f6308004050 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f630802a050 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 10 client.18294 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560f69ca57d0 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f631009fb90 con 0x560f69f642d0 2024-03-19T02:36:55.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f630801e400 con 0x560f69f642d0 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 10 client.18294 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 10 client.18294 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 20 client.18294 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69f7dc30 0x560f69f80060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 4 client.18294 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f69f253e0 con 0x560f69f7dc30 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.092+0000 7f630ffff640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69f7dc30 0x560f69f80060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.093+0000 7f62fdffb640 10 client.18294 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.093+0000 7f630ffff640 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69f7dc30 0x560f69f80060 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1875cd3cda13e560 server_cookie=1fd2ef982394ade1 in_seq=0 out_seq=0 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.093+0000 7f62fdffb640 1 -- 172.21.15.116:0/3816302954 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f69f253e0 con 0x560f69f7dc30 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.093+0000 7f62fdffb640 10 client.18294 handle_command_reply: tid=0 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.093+0000 7f631f661740 1 client.18294 shutdown 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 msgr2=0x560f69f6a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6308009a80 tx=0x7f6308009f40 comp rx=0 tx=0).stop 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 shutdown_connections 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69f7dc30 0x560f69f80060 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f6b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f62fdffb640 0 client.18294 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f6add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 --2- 172.21.15.116:0/3816302954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f6a890 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 >> 172.21.15.116:0/3816302954 conn(0x560f69f5ea00 msgr2=0x560f69f60750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 shutdown_connections 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 1 -- 172.21.15.116:0/3816302954 wait complete. 2024-03-19T02:36:55.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.094+0000 7f631f661740 20 client.18294 trim_cache size 0 max 16384 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 Processor -- start 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 -- start start 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f75980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f5da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ac9e0 con 0x560f69f642d0 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ae520 con 0x560f69f64bf0 2024-03-19T02:36:55.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631f661740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ad0b0 con 0x560f69f6d7d0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54530/0 (socket says 172.21.15.116:54530) 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f631659c640 1 -- 172.21.15.116:0/1754380845 learned_addr learned my addr 172.21.15.116:0/1754380845 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f630ffff640 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f75980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.108+0000 7f6316d9d640 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f5da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 -- 172.21.15.116:0/1754380845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 msgr2=0x560f69f5da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f5da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 -- 172.21.15.116:0/1754380845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 msgr2=0x560f69f75980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f75980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 -- 172.21.15.116:0/1754380845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f69f5e530 con 0x560f69f642d0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f631659c640 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f6308005bc0 tx=0x7f6308005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d67d128600e88a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f62fffff640 1 -- 172.21.15.116:0/1754380845 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630800ed20 con 0x560f69f642d0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f62fffff640 1 -- 172.21.15.116:0/1754380845 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f630801e400 con 0x560f69f642d0 2024-03-19T02:36:55.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.109+0000 7f62fffff640 1 -- 172.21.15.116:0/1754380845 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63080038e0 con 0x560f69f642d0 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- 172.21.15.116:0/1754380845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 msgr2=0x560f69f85930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f6308005bc0 tx=0x7f6308005d30 comp rx=0 tx=0).stop 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- 172.21.15.116:0/1754380845 shutdown_connections 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f6d7d0 0x560f69f5da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f75980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- 172.21.15.116:0/1754380845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f642d0 0x560f69f85930 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- 172.21.15.116:0/1754380845 >> 172.21.15.116:0/1754380845 conn(0x560f69f5ea00 msgr2=0x560f69f5d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- 172.21.15.116:0/1754380845 shutdown_connections 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- 172.21.15.116:0/1754380845 wait complete. 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- start start 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 Processor -- start 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f71970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 0x560f69f71eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ade50 con 0x560f69f6d7d0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.110+0000 7f631f661740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63181ad780 con 0x560f69f64bf0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631f661740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63181aa9e0 con 0x560f69f642d0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631f661740 10 client.0 _ll_register_callbacks cb 0x7f63080137d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 0x560f69f71eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f630ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f71970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43572/0 (socket says 172.21.15.116:43572) 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 -- 172.21.15.116:0/654853272 learned_addr learned my addr 172.21.15.116:0/654853272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 -- 172.21.15.116:0/654853272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 msgr2=0x560f69f71970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f71970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 -- 172.21.15.116:0/654853272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 msgr2=0x560f69f71eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 0x560f69f71eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 -- 172.21.15.116:0/654853272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f69fe3130 con 0x560f69f642d0 2024-03-19T02:36:55.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f630ffff640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f71970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f6316d9d640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 0x560f69f71eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631659c640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6308005890 tx=0x7f630800ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f75e0f06a49a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f62fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308023050 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f630801e400 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.111+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f630801eba0 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6308046050 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 10 client.28039 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560f69fe43f0 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f630801e400 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6308003af0 con 0x560f69f642d0 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 10 client.28039 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:55.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 10 client.28039 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 20 client.28039 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69fe4a50 0x560f69fe6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 4 client.28039 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f69ed49f0 con 0x560f69fe4a50 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f630ffff640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69fe4a50 0x560f69fe6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f62fdffb640 10 client.28039 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.112+0000 7f630ffff640 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69fe4a50 0x560f69fe6e40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9f8ca787188ebdf server_cookie=b132c2eab26d1a3c in_seq=0 out_seq=0 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f62fdffb640 1 -- 172.21.15.116:0/654853272 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f69ed49f0 con 0x560f69fe4a50 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f62fdffb640 10 client.28039 handle_command_reply: tid=0 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 client.28039 shutdown 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 msgr2=0x560f69f71430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6308005890 tx=0x7f630800ef50 comp rx=0 tx=0).stop 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 shutdown_connections 2024-03-19T02:36:55.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560f69fe4a50 0x560f69fe6e40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f69f6d7d0 0x560f69f71eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f62fdffb640 0 client.28039 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f69f64bf0 0x560f69f71970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 --2- 172.21.15.116:0/654853272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f69f642d0 0x560f69f71430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 >> 172.21.15.116:0/654853272 conn(0x560f69f5ea00 msgr2=0x560f69f5dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 shutdown_connections 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 1 -- 172.21.15.116:0/654853272 wait complete. 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.113+0000 7f631f661740 20 client.28039 trim_cache size 0 max 16384 2024-03-19T02:36:55.120 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 msgr2=0x7f63040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 0x7f63040a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6300005ab0 tx=0x7f630002c6a0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 msgr2=0x7f631819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f631000eb10 tx=0x7f631000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 shutdown_connections 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f63040a2970 0x7f63040a4e30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63181066b0 0x7f631819c860 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6318101060 0x7f63181984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 --2- 172.21.15.116:0/2658813294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63181006a0 0x7f6318197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 >> 172.21.15.116:0/2658813294 conn(0x7f63180778d0 msgr2=0x7f6318068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 shutdown_connections 2024-03-19T02:36:55.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.114+0000 7f631ca7e640 1 -- 172.21.15.116:0/2658813294 wait complete. 2024-03-19T02:36:55.125 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:54 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1600086097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:55.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:36:55.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2258576562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481006a0 msgr2=0x7ff248100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2258576562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481006a0 0x7ff248100b20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009930 tx=0x7ff23c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2258576562 shutdown_connections 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2258576562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff248101060 0x7ff24810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2258576562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481006a0 0x7ff248100b20 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2258576562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2481066b0 0x7ff248106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2258576562 >> 172.21.15.116:0/2258576562 conn(0x7ff2480778d0 msgr2=0x7ff248077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2258576562 shutdown_connections 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2258576562 wait complete. 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 Processor -- start 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.237+0000 7ff24d7a9640 1 -- start start 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2481006a0 0x7ff248196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff248101060 0x7ff248197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff248110970 con 0x7ff2481066b0 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481107f0 con 0x7ff248101060 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff248110af0 con 0x7ff2481006a0 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2481006a0 0x7ff248196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54556/0 (socket says 172.21.15.116:54556) 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 -- 172.21.15.116:0/2740419394 learned_addr learned my addr 172.21.15.116:0/2740419394 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2467fc640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff248101060 0x7ff248197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 -- 172.21.15.116:0/2740419394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2481006a0 msgr2=0x7ff248196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2481006a0 0x7ff248196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 -- 172.21.15.116:0/2740419394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff248101060 msgr2=0x7ff248197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff248101060 0x7ff248197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 -- 172.21.15.116:0/2740419394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2481a6490 con 0x7ff2481066b0 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff2477fe640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb10 tx=0x7ff24000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff22ffff640 1 -- 172.21.15.116:0/2740419394 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240004b20 con 0x7ff2481066b0 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2481a6780 con 0x7ff2481066b0 2024-03-19T02:36:55.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff22ffff640 1 -- 172.21.15.116:0/2740419394 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff240004cc0 con 0x7ff2481066b0 2024-03-19T02:36:55.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff22ffff640 1 -- 172.21.15.116:0/2740419394 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240010690 con 0x7ff2481066b0 2024-03-19T02:36:55.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.238+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2481a6cf0 con 0x7ff2481066b0 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff22ffff640 1 -- 172.21.15.116:0/2740419394 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff240005020 con 0x7ff2481066b0 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 Processor -- start 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 -- start start 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393592d0 0x5634393596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff22ffff640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 0x7ff2300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439364bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5634392bc4b0 con 0x563439359bf0 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563439264d70 con 0x5634393592d0 2024-03-19T02:36:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.241+0000 7ff25038e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634392b8550 con 0x5634393627d0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff246ffd640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 0x7ff2300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54558/0 (socket says 172.21.15.116:54558) 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 -- 172.21.15.116:0/1214273689 learned_addr learned my addr 172.21.15.116:0/1214273689 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393592d0 0x5634393596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2477fe640 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439364bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff22ffff640 1 -- 172.21.15.116:0/2740419394 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff2400cf940 con 0x7ff2481066b0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 -- 172.21.15.116:0/1214273689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 msgr2=0x563439364bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439364bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 -- 172.21.15.116:0/1214273689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393592d0 msgr2=0x5634393596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393592d0 0x5634393596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 -- 172.21.15.116:0/1214273689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563439287210 con 0x563439359bf0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff246ffd640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 0x7ff2300a4e30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff238009b50 tx=0x7ff238005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff2467fc640 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff23c036040 tx=0x7ff23c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8d0ba0adb980f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff22d7fa640 1 -- 172.21.15.116:0/1214273689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c04a070 con 0x563439359bf0 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff22d7fa640 1 -- 172.21.15.116:0/1214273689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff23c02cc90 con 0x563439359bf0 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.242+0000 7ff22d7fa640 1 -- 172.21.15.116:0/1214273689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c004920 con 0x563439359bf0 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 -- 172.21.15.116:0/1214273689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 msgr2=0x563439362290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff23c036040 tx=0x7ff23c005d90 comp rx=0 tx=0).stop 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 -- 172.21.15.116:0/1214273689 shutdown_connections 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439364bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x563439362290 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 --2- 172.21.15.116:0/1214273689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393592d0 0x5634393596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 -- 172.21.15.116:0/1214273689 >> 172.21.15.116:0/1214273689 conn(0x563439353a00 msgr2=0x563439353de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 -- 172.21.15.116:0/1214273689 shutdown_connections 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.243+0000 7ff25038e740 1 -- 172.21.15.116:0/1214273689 wait complete. 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 -- start start 2024-03-19T02:36:55.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 Processor -- start 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563439359bf0 0x56343935fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439360350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563439365560 con 0x5634393592d0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563439278310 con 0x5634393627d0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634393656e0 con 0x563439359bf0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff25038e740 10 client.0 _ll_register_callbacks cb 0x56343926c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54574/0 (socket says 172.21.15.116:54574) 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 -- 172.21.15.116:0/449572861 learned_addr learned my addr 172.21.15.116:0/449572861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff2467fc640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563439359bf0 0x56343935fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff2477fe640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439360350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 -- 172.21.15.116:0/449572861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563439359bf0 msgr2=0x56343935fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563439359bf0 0x56343935fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 -- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 msgr2=0x563439360350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439360350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.244+0000 7ff246ffd640 1 -- 172.21.15.116:0/449572861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563438fd03f0 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff246ffd640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff238003970 tx=0x7ff238004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eeddfda8a50dc38 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23800ecc0 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff23801e420 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff23c02cc90 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff23802f050 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 10 client.18312 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56343909a7d0 con 0x5634393592d0 2024-03-19T02:36:55.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff23c02cc90 con 0x5634393592d0 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff2380265c0 con 0x5634393592d0 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 10 client.18312 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 20 client.18312 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563439372c30 0x563439375080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 4 client.18312 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.245+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56343931a3e0 con 0x563439372c30 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.246+0000 7ff2467fc640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563439372c30 0x563439375080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.246+0000 7ff2467fc640 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563439372c30 0x563439375080 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=865f5999a26bc83f server_cookie=1f09d4cc00bc4875 in_seq=0 out_seq=0 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.246+0000 7ff22aff5640 10 client.18312 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff22aff5640 1 -- 172.21.15.116:0/449572861 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56343931a3e0 con 0x563439372c30 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff22aff5640 10 client.18312 handle_command_reply: tid=0 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 client.18312 shutdown 2024-03-19T02:36:55.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 msgr2=0x56343935f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff238003970 tx=0x7ff238004730 comp rx=0 tx=0).stop 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 shutdown_connections 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563439372c30 0x563439375080 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439360350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563439359bf0 0x56343935fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff22aff5640 0 client.18312 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 --2- 172.21.15.116:0/449572861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343935f8d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 >> 172.21.15.116:0/449572861 conn(0x563439353a00 msgr2=0x563439355750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 shutdown_connections 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 1 -- 172.21.15.116:0/449572861 wait complete. 2024-03-19T02:36:55.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.247+0000 7ff25038e740 20 client.18312 trim_cache size 0 max 16384 2024-03-19T02:36:55.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:54 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1600086097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.263+0000 7ff25038e740 1 Processor -- start 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.263+0000 7ff25038e740 1 -- start start 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563439359bf0 0x563439360140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439352a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481b0f80 con 0x5634393592d0 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481b2ac0 con 0x563439359bf0 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff25038e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481b1650 con 0x5634393627d0 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54584/0 (socket says 172.21.15.116:54584) 2024-03-19T02:36:55.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 -- 172.21.15.116:0/2091330671 learned_addr learned my addr 172.21.15.116:0/2091330671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff2467fc640 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563439359bf0 0x563439360140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff2477fe640 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439352a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 -- 172.21.15.116:0/2091330671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 msgr2=0x563439352a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439352a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 -- 172.21.15.116:0/2091330671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563439359bf0 msgr2=0x563439360140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563439359bf0 0x563439360140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 -- 172.21.15.116:0/2091330671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563439353530 con 0x5634393592d0 2024-03-19T02:36:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff246ffd640 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff2380045f0 tx=0x7ff2380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b66b67dfb3050d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff22cff9640 1 -- 172.21.15.116:0/2091330671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238005b90 con 0x5634393592d0 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff22cff9640 1 -- 172.21.15.116:0/2091330671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff238037070 con 0x5634393592d0 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.264+0000 7ff22cff9640 1 -- 172.21.15.116:0/2091330671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238026760 con 0x5634393592d0 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 -- 172.21.15.116:0/2091330671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 msgr2=0x56343936e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff2380045f0 tx=0x7ff2380046c0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 -- 172.21.15.116:0/2091330671 shutdown_connections 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393627d0 0x563439352a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563439359bf0 0x563439360140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 --2- 172.21.15.116:0/2091330671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634393592d0 0x56343936e300 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 -- 172.21.15.116:0/2091330671 >> 172.21.15.116:0/2091330671 conn(0x563439353a00 msgr2=0x563439352310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 -- 172.21.15.116:0/2091330671 shutdown_connections 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.265+0000 7ff25038e740 1 -- 172.21.15.116:0/2091330671 wait complete. 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 -- start start 2024-03-19T02:36:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 Processor -- start 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393592d0 0x563439366370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439366df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481b23f0 con 0x563439359bf0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481b1d20 con 0x5634393627d0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481aef80 con 0x5634393592d0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff25038e740 10 client.0 _ll_register_callbacks cb 0x7ff238004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff246ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393592d0 0x563439366370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54594/0 (socket says 172.21.15.116:54594) 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 -- 172.21.15.116:0/4125867585 learned_addr learned my addr 172.21.15.116:0/4125867585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2477fe640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439366df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 -- 172.21.15.116:0/4125867585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393592d0 msgr2=0x563439366370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393592d0 0x563439366370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 -- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 msgr2=0x563439366df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439366df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.266+0000 7ff2467fc640 1 -- 172.21.15.116:0/4125867585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634393d7fb0 con 0x563439359bf0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff2467fc640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009690 tx=0x7ff23c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b6a1e7bba7f0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c04a030 con 0x563439359bf0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7ff23c02cbf0 con 0x563439359bf0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff238037070 con 0x563439359bf0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff23c005030 con 0x563439359bf0 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 10 client.18318 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5634393d9290 con 0x563439359bf0 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff238037070 con 0x563439359bf0 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff23c053570 con 0x563439359bf0 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 10 client.18318 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 10 client.18318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 20 client.18318 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5634393d9580 0x5634393db970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 4 client.18318 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:36:55.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634392c99f0 con 0x5634393d9580 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.267+0000 7ff246ffd640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5634393d9580 0x5634393db970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.268+0000 7ff246ffd640 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5634393d9580 0x5634393db970 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1241d172753476fa server_cookie=8ff230e58b09f209 in_seq=0 out_seq=0 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.268+0000 7ff22aff5640 10 client.18318 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff22aff5640 1 -- 172.21.15.116:0/4125867585 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634392c99f0 con 0x5634393d9580 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff22aff5640 10 client.18318 handle_command_reply: tid=0 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 client.18318 shutdown 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 msgr2=0x5634393668b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009690 tx=0x7ff23c009620 comp rx=0 tx=0).stop 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 shutdown_connections 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5634393d9580 0x5634393db970 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634393627d0 0x563439366df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563439359bf0 0x5634393668b0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff22aff5640 0 client.18318 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 --2- 172.21.15.116:0/4125867585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634393592d0 0x563439366370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 >> 172.21.15.116:0/4125867585 conn(0x563439353a00 msgr2=0x563439354e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 shutdown_connections 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 1 -- 172.21.15.116:0/4125867585 wait complete. 2024-03-19T02:36:55.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.269+0000 7ff25038e740 20 client.18318 trim_cache size 0 max 16384 2024-03-19T02:36:55.280 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 msgr2=0x7ff2300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 0x7ff2300a4e30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff238009b50 tx=0x7ff238005f90 comp rx=0 tx=0).stop 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 msgr2=0x7ff2481976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff24000eb10 tx=0x7ff24000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 shutdown_connections 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2300a2970 0x7ff2300a4e30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2481066b0 0x7ff2481976a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff248101060 0x7ff248197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 --2- 172.21.15.116:0/2740419394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2481006a0 0x7ff248196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 >> 172.21.15.116:0/2740419394 conn(0x7ff2480778d0 msgr2=0x7ff248102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 shutdown_connections 2024-03-19T02:36:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:55.270+0000 7ff24d7a9640 1 -- 172.21.15.116:0/2740419394 wait complete. 2024-03-19T02:36:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:54 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1600086097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:36:55.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 294670 3.16 MB/sec execute 378 sec latency 16.642 ms 2024-03-19T02:36:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:55 smithi154 ceph-mon[35693]: pgmap v282: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:55 smithi116 ceph-mon[32263]: pgmap v282: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:56.283 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:36:56.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:36:56.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 -- 172.21.15.116:0/871419213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80107870 msgr2=0x7fea80107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 --2- 172.21.15.116:0/871419213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80107870 0x7fea80107c50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fea70009930 tx=0x7fea7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 -- 172.21.15.116:0/871419213 shutdown_connections 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 --2- 172.21.15.116:0/871419213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea800693e0 0x7fea8010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 --2- 172.21.15.116:0/871419213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 --2- 172.21.15.116:0/871419213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80107870 0x7fea80107c50 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 -- 172.21.15.116:0/871419213 >> 172.21.15.116:0/871419213 conn(0x7fea80077820 msgr2=0x7fea80077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 -- 172.21.15.116:0/871419213 shutdown_connections 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.400+0000 7fea854bf640 1 -- 172.21.15.116:0/871419213 wait complete. 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 Processor -- start 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 -- start start 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 0x7fea80198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea80107870 0x7fea8019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea80111a70 con 0x7fea800693e0 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea801118f0 con 0x7fea80107870 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea854bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fea80111bf0 con 0x7fea80068a20 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.401+0000 7fea767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 0x7fea80198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 0x7fea80198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54608/0 (socket says 172.21.15.116:54608) 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea80107870 0x7fea8019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea767fc640 1 -- 172.21.15.116:0/880949945 learned_addr learned my addr 172.21.15.116:0/880949945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 -- 172.21.15.116:0/880949945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea80107870 msgr2=0x7fea8019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea80107870 0x7fea8019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 -- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 msgr2=0x7fea80198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 0x7fea80198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 -- 172.21.15.116:0/880949945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8019d000 con 0x7fea80068a20 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7effd640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80198010 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea70009230 tx=0x7fea7002cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7cff9640 1 -- 172.21.15.116:0/880949945 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70043020 con 0x7fea80068a20 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8019d290 con 0x7fea80068a20 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7cff9640 1 -- 172.21.15.116:0/880949945 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea700043d0 con 0x7fea80068a20 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea7cff9640 1 -- 172.21.15.116:0/880949945 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70003af0 con 0x7fea80068a20 2024-03-19T02:36:56.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.402+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea801a4b80 con 0x7fea80068a20 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.405+0000 7fea7cff9640 1 -- 172.21.15.116:0/880949945 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fea70046020 con 0x7fea80068a20 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.405+0000 7fea88188740 1 Processor -- start 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.405+0000 7fea88188740 1 -- start start 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d973760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d97c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c9d8d7160 con 0x563c9d97c880 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563c9d87fe10 con 0x563c9d973ca0 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea88188740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c9d8d3200 con 0x563c9d973380 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7cff9640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 0x7fea6c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d97c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54610/0 (socket says 172.21.15.116:54610) 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d97c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:42438/0 (socket says 172.21.15.116:42438) 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 -- 172.21.15.116:0/2359272076 learned_addr learned my addr 172.21.15.116:0/2359272076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d973760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 0x7fea6c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea7cff9640 1 -- 172.21.15.116:0/880949945 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fea700f0a10 con 0x7fea80068a20 2024-03-19T02:36:56.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 -- 172.21.15.116:0/2359272076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 msgr2=0x563c9d973760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d973760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 -- 172.21.15.116:0/2359272076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 msgr2=0x563c9d97c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d97c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 -- 172.21.15.116:0/2359272076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9d6b37e0 con 0x563c9d97c880 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.406+0000 7fea767fc640 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fea68009db0 tx=0x7fea68011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634de76191ed74ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea757fa640 1 -- 172.21.15.116:0/2359272076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68005220 con 0x563c9d97c880 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 0x7fea6c0a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea78009fd0 tx=0x7fea780096a0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea757fa640 1 -- 172.21.15.116:0/2359272076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea68004c50 con 0x563c9d97c880 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea757fa640 1 -- 172.21.15.116:0/2359272076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68019860 con 0x563c9d97c880 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 -- 172.21.15.116:0/2359272076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 msgr2=0x563c9d97ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fea68009db0 tx=0x7fea68011810 comp rx=0 tx=0).stop 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 -- 172.21.15.116:0/2359272076 shutdown_connections 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97ec70 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d97c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 --2- 172.21.15.116:0/2359272076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d973760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.407+0000 7fea88188740 1 -- 172.21.15.116:0/2359272076 >> 172.21.15.116:0/2359272076 conn(0x563c9d96dab0 msgr2=0x563c9d96de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- 172.21.15.116:0/2359272076 shutdown_connections 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- 172.21.15.116:0/2359272076 wait complete. 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- start start 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 Processor -- start 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 0x563c9d9797a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973ca0 0x563c9d979ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c9d97f670 con 0x563c9d97c880 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563c9d8931a0 con 0x563c9d973380 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563c9d97f7f0 con 0x563c9d973ca0 2024-03-19T02:36:56.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea88188740 10 client.0 _ll_register_callbacks cb 0x563c9d5eac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea7effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 0x563c9d9797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54622/0 (socket says 172.21.15.116:54622) 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea7f7fe640 1 -- 172.21.15.116:0/1712929785 learned_addr learned my addr 172.21.15.116:0/1712929785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.408+0000 7fea767fc640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973ca0 0x563c9d979ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 -- 172.21.15.116:0/1712929785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973ca0 msgr2=0x563c9d979ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973ca0 0x563c9d979ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 -- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 msgr2=0x563c9d9797a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 0x563c9d9797a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 -- 172.21.15.116:0/1712929785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9d713f70 con 0x563c9d97c880 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7effd640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 0x563c9d9797a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fea78003dc0 tx=0x7fea78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce44c662d16965b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea667fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:56.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78025070 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fea780045e0 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea68004c50 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea78030050 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea88188740 10 client.18330 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c9d5ab170 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.409+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea68004c50 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fea7801e050 con 0x563c9d97c880 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 10 client.18330 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 10 client.18330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 20 client.18330 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d98caf0 0x563c9d98ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 4 client.18330 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c9d8e46a0 con 0x563c9d98caf0 2024-03-19T02:36:56.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea7effd640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d98caf0 0x563c9d98ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea667fc640 10 client.18330 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.410+0000 7fea7effd640 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d98caf0 0x563c9d98ef40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb549045ec495a2c server_cookie=c07b34cdafa5c1d4 in_seq=0 out_seq=0 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea667fc640 1 -- 172.21.15.116:0/1712929785 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c9d8e46a0 con 0x563c9d98caf0 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea667fc640 10 client.18330 handle_command_reply: tid=0 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 client.18330 shutdown 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 msgr2=0x563c9d97a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fea78003dc0 tx=0x7fea78004a40 comp rx=0 tx=0).stop 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 shutdown_connections 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d98caf0 0x563c9d98ef40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d97a220 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973ca0 0x563c9d979ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea667fc640 0 client.18330 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 --2- 172.21.15.116:0/1712929785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973380 0x563c9d9797a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 >> 172.21.15.116:0/1712929785 conn(0x563c9d96dab0 msgr2=0x563c9d96f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 shutdown_connections 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.411+0000 7fea88188740 1 -- 172.21.15.116:0/1712929785 wait complete. 2024-03-19T02:36:56.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.412+0000 7fea88188740 20 client.18330 trim_cache size 0 max 16384 2024-03-19T02:36:56.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:55 smithi162 ceph-mon[35761]: pgmap v282: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:36:56.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 Processor -- start 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 -- start start 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 0x563c9d988410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d97c880 0x563c9d994810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea801aec20 con 0x563c9d973ca0 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea801b0760 con 0x563c9d97c880 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea88188740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fea801af2f0 con 0x563c9d973380 2024-03-19T02:36:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea7effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 0x563c9d988410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea7effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43658/0 (socket says 172.21.15.116:43658) 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.431+0000 7fea7effd640 1 -- 172.21.15.116:0/1724765971 learned_addr learned my addr 172.21.15.116:0/1724765971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d97c880 0x563c9d994810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 -- 172.21.15.116:0/1724765971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d97c880 msgr2=0x563c9d994810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d97c880 0x563c9d994810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 -- 172.21.15.116:0/1724765971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 msgr2=0x563c9d988410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 0x563c9d988410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 -- 172.21.15.116:0/1724765971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9d96d8e0 con 0x563c9d973380 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea767fc640 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 0x563c9d988410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea7effd640 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea700bcaa0 tx=0x7fea7002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b221dc405181700e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea74ff9640 1 -- 172.21.15.116:0/1724765971 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea700d5e60 con 0x563c9d973380 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea74ff9640 1 -- 172.21.15.116:0/1724765971 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea700d2050 con 0x563c9d973380 2024-03-19T02:36:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.432+0000 7fea74ff9640 1 -- 172.21.15.116:0/1724765971 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea700ee000 con 0x563c9d973380 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- 172.21.15.116:0/1724765971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 msgr2=0x563c9d97bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea700bcaa0 tx=0x7fea7002bc30 comp rx=0 tx=0).stop 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- 172.21.15.116:0/1724765971 shutdown_connections 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d97c880 0x563c9d994810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d973ca0 0x563c9d988410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 --2- 172.21.15.116:0/1724765971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d97bed0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- 172.21.15.116:0/1724765971 >> 172.21.15.116:0/1724765971 conn(0x563c9d96dab0 msgr2=0x563c9d96cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- 172.21.15.116:0/1724765971 shutdown_connections 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- 172.21.15.116:0/1724765971 wait complete. 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 -- start start 2024-03-19T02:36:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.433+0000 7fea88188740 1 Processor -- start 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d976c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d9f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea801b0090 con 0x563c9d97c880 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea801af9c0 con 0x563c9d973ca0 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fea801acc20 con 0x563c9d973380 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea88188740 10 client.0 _ll_register_callbacks cb 0x7fea78003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d9f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea767fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d976c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43674/0 (socket says 172.21.15.116:43674) 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 -- 172.21.15.116:0/919201256 learned_addr learned my addr 172.21.15.116:0/919201256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 -- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 msgr2=0x563c9d976c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d976c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 -- 172.21.15.116:0/919201256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 msgr2=0x563c9d9f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d9f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 -- 172.21.15.116:0/919201256 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9d9774b0 con 0x563c9d973380 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7f7fe640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d9f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea7effd640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea700b91b0 tx=0x7fea700053b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea25cbcd065e37c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea667fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.434+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea700d9020 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea700d2050 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7fea700b9410 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fea700a4050 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 10 client.28075 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c9d9f3490 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea700d2050 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fea70005440 con 0x563c9d973380 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 10 client.28075 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 10 client.28075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 20 client.28075 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d9f3c10 0x563c9d9f6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 4 client.28075 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563c9d8de9f0 con 0x563c9d9f3c10 2024-03-19T02:36:56.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.435+0000 7fea767fc640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d9f3c10 0x563c9d9f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.436+0000 7fea767fc640 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d9f3c10 0x563c9d9f6030 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1dd1c5f8f94f7a server_cookie=33d7501daa3cf980 in_seq=0 out_seq=0 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.436+0000 7fea667fc640 10 client.28075 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea667fc640 1 -- 172.21.15.116:0/919201256 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563c9d8de9f0 con 0x563c9d9f3c10 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea667fc640 10 client.28075 handle_command_reply: tid=0 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 client.28075 shutdown 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 msgr2=0x563c9d9766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea700b91b0 tx=0x7fea700053b0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 shutdown_connections 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563c9d9f3c10 0x563c9d9f6030 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c9d97c880 0x563c9d9f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563c9d973ca0 0x563c9d976c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea667fc640 0 client.28075 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 --2- 172.21.15.116:0/919201256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563c9d973380 0x563c9d9766e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 >> 172.21.15.116:0/919201256 conn(0x563c9d96dab0 msgr2=0x563c9d96c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 shutdown_connections 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 1 -- 172.21.15.116:0/919201256 wait complete. 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.437+0000 7fea88188740 20 client.28075 trim_cache size 0 max 16384 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3d28e81b-4848-43b5-a7a4-c6fcc82b696f", 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:36:56.444 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 msgr2=0x7fea6c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 0x7fea6c0a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea78009fd0 tx=0x7fea780096a0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 msgr2=0x7fea80198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80198010 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea70009230 tx=0x7fea7002cd20 comp rx=0 tx=0).stop 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 shutdown_connections 2024-03-19T02:36:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fea6c0a2970 0x7fea6c0a4e30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea80107870 0x7fea8019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea800693e0 0x7fea80198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 --2- 172.21.15.116:0/880949945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea80068a20 0x7fea80198010 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 >> 172.21.15.116:0/880949945 conn(0x7fea80077820 msgr2=0x7fea800fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 shutdown_connections 2024-03-19T02:36:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.438+0000 7fea854bf640 1 -- 172.21.15.116:0/880949945 wait complete. 2024-03-19T02:36:56.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:36:56.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:36:56.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2889648171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100680 msgr2=0x7f06d410e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2889648171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100680 0x7f06d410e640 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009ca0 tx=0x7f06cc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2889648171 shutdown_connections 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2889648171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d4100680 0x7f06d410e640 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2889648171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d40ffcc0 0x7f06d4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2889648171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d41077c0 0x7f06d4107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2889648171 >> 172.21.15.116:0/2889648171 conn(0x7f06d4076c40 msgr2=0x7f06d40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2889648171 shutdown_connections 2024-03-19T02:36:56.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2889648171 wait complete. 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 Processor -- start 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.567+0000 7f06d9fe6640 1 -- start start 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06d4111a60 con 0x7f06d41077c0 2024-03-19T02:36:56.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41118e0 con 0x7f06d40ffcc0 2024-03-19T02:36:56.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d9fe6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06d4111be0 con 0x7f06d4100680 2024-03-19T02:36:56.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54660/0 (socket says 172.21.15.116:54660) 2024-03-19T02:36:56.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43690/0 (socket says 172.21.15.116:43690) 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:42484/0 (socket says 172.21.15.116:42484) 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 -- 172.21.15.116:0/2202735427 learned_addr learned my addr 172.21.15.116:0/2202735427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 -- 172.21.15.116:0/2202735427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 msgr2=0x7f06d4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 -- 172.21.15.116:0/2202735427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 msgr2=0x7f06d4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 -- 172.21.15.116:0/2202735427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d419cfd0 con 0x7f06d41077c0 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d37fe640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d3fff640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02c700 tx=0x7f06cc035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.568+0000 7f06d0ff9640 1 -- 172.21.15.116:0/2202735427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc032900 con 0x7f06d41077c0 2024-03-19T02:36:56.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.569+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d419d260 con 0x7f06d41077c0 2024-03-19T02:36:56.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.569+0000 7f06d0ff9640 1 -- 172.21.15.116:0/2202735427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06cc003e20 con 0x7f06d41077c0 2024-03-19T02:36:56.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.569+0000 7f06d0ff9640 1 -- 172.21.15.116:0/2202735427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc004800 con 0x7f06d41077c0 2024-03-19T02:36:56.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.569+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d4100dd0 con 0x7f06d41077c0 2024-03-19T02:36:56.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.571+0000 7f06d0ff9640 1 -- 172.21.15.116:0/2202735427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f06cc04d060 con 0x7f06d41077c0 2024-03-19T02:36:56.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.571+0000 7f06dcbd5740 1 Processor -- start 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 -- start start 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d0ff9640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 0x7f06bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835704e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835710320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561835668500 con 0x56183570df30 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561835610be0 con 0x561835705350 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5618356645a0 con 0x561835704a30 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d0ff9640 1 -- 172.21.15.116:0/2202735427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f06cc0f4f30 con 0x7f06d41077c0 2024-03-19T02:36:56.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 0x7f06bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835704e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835704e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43702/0 (socket says 172.21.15.116:43702) 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835710320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d37fe640 1 -- 172.21.15.116:0/1877010151 learned_addr learned my addr 172.21.15.116:0/1877010151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 -- 172.21.15.116:0/1877010151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 msgr2=0x561835704e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835704e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 -- 172.21.15.116:0/1877010151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 msgr2=0x56183570d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 -- 172.21.15.116:0/1877010151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561835632000 con 0x56183570df30 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 0x7f06bc0a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009fd0 tx=0x7f06c8009ee0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.572+0000 7f06d3fff640 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835710320 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0f23b0 tx=0x7f06cc0c0ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc781030eb019645 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06b9ffb640 1 -- 172.21.15.116:0/1877010151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0c5410 con 0x56183570df30 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06b9ffb640 1 -- 172.21.15.116:0/1877010151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06cc0c9a60 con 0x56183570df30 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06b9ffb640 1 -- 172.21.15.116:0/1877010151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0bdad0 con 0x56183570df30 2024-03-19T02:36:56.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 -- 172.21.15.116:0/1877010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 msgr2=0x561835710320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835710320 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0f23b0 tx=0x7f06cc0c0ec0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 -- 172.21.15.116:0/1877010151 shutdown_connections 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835710320 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/1877010151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835704e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 -- 172.21.15.116:0/1877010151 >> 172.21.15.116:0/1877010151 conn(0x5618356ff160 msgr2=0x5618356ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 -- 172.21.15.116:0/1877010151 shutdown_connections 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.573+0000 7f06dcbd5740 1 -- 172.21.15.116:0/1877010151 wait complete. 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 -- start start 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 Processor -- start 2024-03-19T02:36:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x56183570adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561835668500 con 0x56183570df30 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561835624160 con 0x561835705350 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561835710ae0 con 0x561835704a30 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06dcbd5740 10 client.0 _ll_register_callbacks cb 0x561835660a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54678/0 (socket says 172.21.15.116:54678) 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d3fff640 1 -- 172.21.15.116:0/3431774285 learned_addr learned my addr 172.21.15.116:0/3431774285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x56183570adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.574+0000 7f06d3fff640 1 -- 172.21.15.116:0/3431774285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 msgr2=0x56183570adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d3fff640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x56183570adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d3fff640 1 -- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 msgr2=0x56183570b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d3fff640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d3fff640 1 -- 172.21.15.116:0/3431774285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618354467d0 con 0x56183570df30 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x56183570adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06d3fff640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0f2460 tx=0x7f06cc034790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d6edfa2de86fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0c5170 con 0x56183570df30 2024-03-19T02:36:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f06cc0e10c0 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06cc0c9a60 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06cc0c1520 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06dcbd5740 10 client.18348 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56183537c3f0 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06cc0c9a60 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f06cc004b40 con 0x56183570df30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06dcbd5740 10 client.18348 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.575+0000 7f06dcbd5740 10 client.18348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06dcbd5740 20 client.18348 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:56.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56183571db50 0x56183571ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06dcbd5740 4 client.18348 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561835675a40 con 0x56183571db50 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56183571db50 0x56183571ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06b2ffd640 10 client.18348 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.576+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56183571db50 0x56183571ff40 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61797a9356d853b8 server_cookie=b6255cbb454388ba in_seq=0 out_seq=0 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.578+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3431774285 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561835675a40 con 0x56183571db50 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.578+0000 7f06b2ffd640 10 client.18348 handle_command_reply: tid=0 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.578+0000 7f06dcbd5740 1 client.18348 shutdown 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.578+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 msgr2=0x56183570b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.578+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0f2460 tx=0x7f06cc034790 comp rx=0 tx=0).stop 2024-03-19T02:36:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 shutdown_connections 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56183571db50 0x56183571ff40 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x56183570b840 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3431774285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x56183570adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 >> 172.21.15.116:0/3431774285 conn(0x5618356ff160 msgr2=0x561835700eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06b2ffd640 0 client.18348 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 shutdown_connections 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3431774285 wait complete. 2024-03-19T02:36:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.579+0000 7f06dcbd5740 20 client.18348 trim_cache size 0 max 16384 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 Processor -- start 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 -- start start 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x5618357256b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41acad0 con 0x56183570df30 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41ae610 con 0x561835705350 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.596+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41ad1a0 con 0x561835704a30 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x5618357256b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:42526/0 (socket says 172.21.15.116:42526) 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43716/0 (socket says 172.21.15.116:43716) 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d2ffd640 1 -- 172.21.15.116:0/2113070126 learned_addr learned my addr 172.21.15.116:0/2113070126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 -- 172.21.15.116:0/2113070126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 msgr2=0x56183570b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 -- 172.21.15.116:0/2113070126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 msgr2=0x5618357256b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x5618357256b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 -- 172.21.15.116:0/2113070126 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56183533c0f0 con 0x561835704a30 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06d37fe640 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x561835720620 tx=0x7f06c4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ebe6b2253de00a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06b97fa640 1 -- 172.21.15.116:0/2113070126 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4012e60 con 0x561835704a30 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06b97fa640 1 -- 172.21.15.116:0/2113070126 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c4024030 con 0x561835704a30 2024-03-19T02:36:56.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.597+0000 7f06b97fa640 1 -- 172.21.15.116:0/2113070126 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4004b20 con 0x561835704a30 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- 172.21.15.116:0/2113070126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 msgr2=0x561835719670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x561835720620 tx=0x7f06c4005f90 comp rx=0 tx=0).stop 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- 172.21.15.116:0/2113070126 shutdown_connections 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x5618357256b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x56183570b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/2113070126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835719670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- 172.21.15.116:0/2113070126 >> 172.21.15.116:0/2113070126 conn(0x5618356ff160 msgr2=0x5618356fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- 172.21.15.116:0/2113070126 shutdown_connections 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- 172.21.15.116:0/2113070126 wait complete. 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 -- start start 2024-03-19T02:36:56.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.598+0000 7f06dcbd5740 1 Processor -- start 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x561835711320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41adf40 con 0x56183570df30 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41ad870 con 0x561835705350 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06d41aaad0 con 0x561835704a30 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 10 client.0 _ll_register_callbacks cb 0x7f06cc0bd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43720/0 (socket says 172.21.15.116:43720) 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54700/0 (socket says 172.21.15.116:54700) 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 -- 172.21.15.116:0/3910301956 learned_addr learned my addr 172.21.15.116:0/3910301956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x561835711320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 -- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 msgr2=0x561835711320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x561835711320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 -- 172.21.15.116:0/3910301956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 msgr2=0x561835781cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 -- 172.21.15.116:0/3910301956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618357822e0 con 0x561835704a30 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d3fff640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:36:56.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06d37fe640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f06c4013e70 tx=0x7f06c4013ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=498cce3ed03eeb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c40120d0 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c4024030 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.599+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9510+0+0 (secure 0 0 0) 0x7f06c4005030 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06c4039050 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 10 client.28093 fetch_fsmap learned FSMap version 30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618357835f0 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c4024030 con 0x561835704a30 2024-03-19T02:36:56.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f06c4004c30 con 0x561835704a30 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 10 client.28093 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 10 client.28093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 20 client.28093 populate_metadata read hostname 'smithi116' 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561835783d80 0x5618357861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 4 client.28093 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56183566fd90 con 0x561835783d80 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.600+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561835783d80 0x5618357861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06d2ffd640 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561835783d80 0x5618357861a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ef2149cd286adc7 server_cookie=26822d44becbce19 in_seq=0 out_seq=0 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06b2ffd640 10 client.28093 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06b2ffd640 1 -- 172.21.15.116:0/3910301956 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56183566fd90 con 0x561835783d80 2024-03-19T02:36:56.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06b2ffd640 10 client.28093 handle_command_reply: tid=0 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 client.28093 shutdown 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 msgr2=0x561835710de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f06c4013e70 tx=0x7f06c4013ea0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 shutdown_connections 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561835783d80 0x5618357861a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56183570df30 0x561835781cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561835705350 0x561835711320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 --2- 172.21.15.116:0/3910301956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561835704a30 0x561835710de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06b2ffd640 0 client.28093 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 >> 172.21.15.116:0/3910301956 conn(0x5618356ff160 msgr2=0x561835700330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 shutdown_connections 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 1 -- 172.21.15.116:0/3910301956 wait complete. 2024-03-19T02:36:56.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.601+0000 7f06dcbd5740 20 client.28093 trim_cache size 0 max 16384 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "3d28e81b-4848-43b5-a7a4-c6fcc82b696f": { 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "3d28e81b-4848-43b5-a7a4-c6fcc82b696f", 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:36:56.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:36:56.613 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 msgr2=0x7f06bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 0x7f06bc0a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009fd0 tx=0x7f06c8009ee0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 msgr2=0x7f06d419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02c700 tx=0x7f06cc035d60 comp rx=0 tx=0).stop 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 shutdown_connections 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06bc0a2970 0x7f06bc0a4e30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d41077c0 0x7f06d419c8a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06d4100680 0x7f06d4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 --2- 172.21.15.116:0/2202735427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06d40ffcc0 0x7f06d4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:36:56.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 >> 172.21.15.116:0/2202735427 conn(0x7f06d4076c40 msgr2=0x7f06d40fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:36:56.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 shutdown_connections 2024-03-19T02:36:56.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:36:56.602+0000 7f06d9fe6640 1 -- 172.21.15.116:0/2202735427 wait complete. 2024-03-19T02:36:56.615 INFO:tasks.cephfs.filesystem:scrub status for tag:3d28e81b-4848-43b5-a7a4-c6fcc82b696f - {'path': '/', 'tag': '3d28e81b-4848-43b5-a7a4-c6fcc82b696f', 'options': 'recursive,force'} 2024-03-19T02:36:56.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 295050 3.15 MB/sec execute 379 sec latency 29.507 ms 2024-03-19T02:36:57.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:56 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:36:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:36:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:56 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:36:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:36:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:56 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:36:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:36:57.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 295664 3.15 MB/sec execute 380 sec latency 15.416 ms 2024-03-19T02:36:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:57 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:36:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:57 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:57 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:57 smithi154 ceph-mon[35693]: pgmap v283: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:57 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:36:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:57 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:57 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:57 smithi116 ceph-mon[32263]: pgmap v283: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:57 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:36:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:57 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:57 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:36:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:57 smithi162 ceph-mon[35761]: pgmap v283: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:58.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 296251 3.15 MB/sec execute 381 sec latency 33.585 ms 2024-03-19T02:36:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:36:58 smithi154 ceph-mon[35693]: pgmap v284: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:36:58 smithi116 ceph-mon[32263]: pgmap v284: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:36:58 smithi162 ceph-mon[35761]: pgmap v284: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-03-19T02:36:59.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 296834 3.14 MB/sec execute 382 sec latency 23.753 ms 2024-03-19T02:37:00.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 297705 3.15 MB/sec execute 383 sec latency 45.945 ms 2024-03-19T02:37:01.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 298007 3.15 MB/sec execute 384 sec latency 31.652 ms 2024-03-19T02:37:02.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:01 smithi116 ceph-mon[32263]: pgmap v285: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:37:02.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:01 smithi162 ceph-mon[35761]: pgmap v285: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:37:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:01 smithi154 ceph-mon[35693]: pgmap v285: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 122 op/s 2024-03-19T02:37:02.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 298418 3.15 MB/sec execute 385 sec latency 18.349 ms 2024-03-19T02:37:03.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:02 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:37:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:02 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:37:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:02 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:37:03.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 298886 3.14 MB/sec execute 386 sec latency 54.983 ms 2024-03-19T02:37:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:03 smithi162 ceph-mon[35761]: pgmap v286: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:37:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:03 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:37:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:03 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:37:04.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:03 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:37:04.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:03 smithi154 ceph-mon[35693]: pgmap v286: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:37:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:03 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:37:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:03 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:37:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:03 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:37:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:03 smithi116 ceph-mon[32263]: pgmap v286: 129 pgs: 129 active+clean; 297 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-03-19T02:37:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:03 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:37:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:03 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:37:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:03 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:37:04.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 299492 3.15 MB/sec execute 387 sec latency 20.479 ms 2024-03-19T02:37:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:04 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:37:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:04 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:37:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:04 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:37:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 300038 3.14 MB/sec execute 388 sec latency 23.762 ms 2024-03-19T02:37:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:37:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:05 smithi162 ceph-mon[35761]: pgmap v287: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:37:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:05 smithi162 ceph-mon[35761]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T02:37:06.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:37:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:05 smithi154 ceph-mon[35693]: pgmap v287: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:37:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:05 smithi154 ceph-mon[35693]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T02:37:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:37:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:05 smithi116 ceph-mon[32263]: pgmap v287: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:37:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:05 smithi116 ceph-mon[32263]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T02:37:06.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 300595 3.14 MB/sec execute 389 sec latency 18.303 ms 2024-03-19T02:37:07.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 301303 3.15 MB/sec execute 390 sec latency 19.159 ms 2024-03-19T02:37:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:07 smithi162 ceph-mon[35761]: pgmap v289: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:08.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:07 smithi154 ceph-mon[35693]: pgmap v289: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:07 smithi116 ceph-mon[32263]: pgmap v289: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:08.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 301831 3.15 MB/sec execute 391 sec latency 20.162 ms 2024-03-19T02:37:09.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 302246 3.14 MB/sec execute 392 sec latency 23.858 ms 2024-03-19T02:37:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:09 smithi162 ceph-mon[35761]: pgmap v290: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:10.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:09 smithi154 ceph-mon[35693]: pgmap v290: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:09 smithi116 ceph-mon[32263]: pgmap v290: 129 pgs: 129 active+clean; 306 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:37:10.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 302905 3.15 MB/sec execute 393 sec latency 16.993 ms 2024-03-19T02:37:11.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 303392 3.14 MB/sec execute 394 sec latency 36.191 ms 2024-03-19T02:37:12.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:11 smithi154 ceph-mon[35693]: pgmap v291: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-03-19T02:37:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:11 smithi154 ceph-mon[35693]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T02:37:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:11 smithi116 ceph-mon[32263]: pgmap v291: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-03-19T02:37:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:11 smithi116 ceph-mon[32263]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T02:37:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:11 smithi162 ceph-mon[35761]: pgmap v291: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-03-19T02:37:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:11 smithi162 ceph-mon[35761]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T02:37:12.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 303964 3.14 MB/sec execute 395 sec latency 19.142 ms 2024-03-19T02:37:13.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 304818 3.15 MB/sec execute 396 sec latency 16.312 ms 2024-03-19T02:37:14.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:13 smithi154 ceph-mon[35693]: pgmap v293: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-03-19T02:37:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:13 smithi116 ceph-mon[32263]: pgmap v293: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-03-19T02:37:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:13 smithi162 ceph-mon[35761]: pgmap v293: 129 pgs: 129 active+clean; 311 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.3 MiB/s wr, 59 op/s 2024-03-19T02:37:14.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 305204 3.15 MB/sec execute 397 sec latency 21.166 ms 2024-03-19T02:37:15.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 305623 3.14 MB/sec execute 398 sec latency 16.052 ms 2024-03-19T02:37:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:15 smithi154 ceph-mon[35693]: pgmap v294: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-03-19T02:37:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:15 smithi116 ceph-mon[32263]: pgmap v294: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-03-19T02:37:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:15 smithi162 ceph-mon[35761]: pgmap v294: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-03-19T02:37:16.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 306174 3.14 MB/sec execute 399 sec latency 48.088 ms 2024-03-19T02:37:17.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 306733 3.14 MB/sec execute 400 sec latency 28.840 ms 2024-03-19T02:37:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:17 smithi154 ceph-mon[35693]: pgmap v295: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:17 smithi116 ceph-mon[32263]: pgmap v295: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:17 smithi162 ceph-mon[35761]: pgmap v295: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:18.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 307373 3.14 MB/sec execute 401 sec latency 20.180 ms 2024-03-19T02:37:19.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 308087 3.14 MB/sec execute 402 sec latency 15.600 ms 2024-03-19T02:37:20.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:19 smithi162 ceph-mon[35761]: pgmap v296: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:19 smithi154 ceph-mon[35693]: pgmap v296: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:19 smithi116 ceph-mon[32263]: pgmap v296: 129 pgs: 129 active+clean; 314 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-03-19T02:37:20.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 308599 3.15 MB/sec execute 403 sec latency 19.462 ms 2024-03-19T02:37:21.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 309040 3.14 MB/sec execute 404 sec latency 17.648 ms 2024-03-19T02:37:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:21 smithi154 ceph-mon[35693]: pgmap v297: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-03-19T02:37:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:21 smithi116 ceph-mon[32263]: pgmap v297: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-03-19T02:37:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:21 smithi162 ceph-mon[35761]: pgmap v297: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-03-19T02:37:22.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 309554 3.14 MB/sec execute 405 sec latency 34.603 ms 2024-03-19T02:37:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 310172 3.14 MB/sec execute 406 sec latency 19.558 ms 2024-03-19T02:37:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:23 smithi154 ceph-mon[35693]: pgmap v298: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-03-19T02:37:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:23 smithi116 ceph-mon[32263]: pgmap v298: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-03-19T02:37:24.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:23 smithi162 ceph-mon[35761]: pgmap v298: 129 pgs: 129 active+clean; 319 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-03-19T02:37:24.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 310754 3.13 MB/sec execute 407 sec latency 30.316 ms 2024-03-19T02:37:25.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 311507 3.14 MB/sec execute 408 sec latency 19.200 ms 2024-03-19T02:37:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:25 smithi154 ceph-mon[35693]: pgmap v299: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:37:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:25 smithi116 ceph-mon[32263]: pgmap v299: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:37:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:25 smithi162 ceph-mon[35761]: pgmap v299: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 123 op/s 2024-03-19T02:37:26.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:37:26.616 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 -- 172.21.15.116:0/1552670302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f4106750 msgr2=0x7f70f4106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/1552670302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f4106750 0x7f70f4106b30 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009900 tx=0x7f70e402bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 -- 172.21.15.116:0/1552670302 shutdown_connections 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/1552670302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f41013c0 0x7f70f410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/1552670302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 0x7f70f4100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/1552670302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f4106750 0x7f70f4106b30 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 -- 172.21.15.116:0/1552670302 >> 172.21.15.116:0/1552670302 conn(0x7f70f4077700 msgr2=0x7f70f4077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 -- 172.21.15.116:0/1552670302 shutdown_connections 2024-03-19T02:37:26.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.731+0000 7f70f9bb9640 1 -- 172.21.15.116:0/1552670302 wait complete. 2024-03-19T02:37:26.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 Processor -- start 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 -- start start 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 0x7f70f4198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f4106750 0x7f70f41a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70f4110950 con 0x7f70f41013c0 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41107d0 con 0x7f70f4106750 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f9bb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70f4110ad0 con 0x7f70f4100a00 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.732+0000 7f70f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55076/0 (socket says 172.21.15.116:55076) 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 0x7f70f4198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 -- 172.21.15.116:0/631701437 learned_addr learned my addr 172.21.15.116:0/631701437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f3fff640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f4106750 0x7f70f41a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 -- 172.21.15.116:0/631701437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 msgr2=0x7f70f4198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 0x7f70f4198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 -- 172.21.15.116:0/631701437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f4106750 msgr2=0x7f70f41a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f4106750 0x7f70f41a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 -- 172.21.15.116:0/631701437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f41a4350 con 0x7f70f41013c0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f70e800e9f0 tx=0x7f70e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f0ff9640 1 -- 172.21.15.116:0/631701437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e800cd80 con 0x7f70f41013c0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f41a4520 con 0x7f70f41013c0 2024-03-19T02:37:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f0ff9640 1 -- 172.21.15.116:0/631701437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70e80054d0 con 0x7f70f41013c0 2024-03-19T02:37:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f0ff9640 1 -- 172.21.15.116:0/631701437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8010490 con 0x7f70f41013c0 2024-03-19T02:37:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.733+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f41a49f0 con 0x7f70f41013c0 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f0ff9640 1 -- 172.21.15.116:0/631701437 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f70e8005020 con 0x7f70f41013c0 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 Processor -- start 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 -- start start 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bbce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d1b13500 con 0x5633d1baf410 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5633d1abbbe0 con 0x5633d1bafdc0 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633d1b0f5a0 con 0x5633d1bbd340 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55080/0 (socket says 172.21.15.116:55080) 2024-03-19T02:37:26.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f3fff640 1 -- 172.21.15.116:0/4142045455 learned_addr learned my addr 172.21.15.116:0/4142045455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f37fe640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bbce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.736+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f0ff9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 0x7f70dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 -- 172.21.15.116:0/4142045455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 msgr2=0x5633d1bbf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 -- 172.21.15.116:0/4142045455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 msgr2=0x5633d1bbce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bbce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 -- 172.21.15.116:0/4142045455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d1add000 con 0x5633d1baf410 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f37fe640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bbce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f0ff9640 1 -- 172.21.15.116:0/631701437 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f70e80cf5b0 con 0x7f70f41013c0 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 0x7f70dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70f3fff640 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011590 tx=0x7f70ec011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d90df128e094f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70d9ffb640 1 -- 172.21.15.116:0/4142045455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec01f020 con 0x5633d1baf410 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70d9ffb640 1 -- 172.21.15.116:0/4142045455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ec00fd40 con 0x5633d1baf410 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.737+0000 7f70d9ffb640 1 -- 172.21.15.116:0/4142045455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec017050 con 0x5633d1baf410 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- 172.21.15.116:0/4142045455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 msgr2=0x5633d1baf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70f3fff640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 0x7f70dc0a4e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f70ec020000 tx=0x7f70ec023d40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011590 tx=0x7f70ec011a50 comp rx=0 tx=0).stop 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- 172.21.15.116:0/4142045455 shutdown_connections 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bbce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/4142045455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1baf7f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- 172.21.15.116:0/4142045455 >> 172.21.15.116:0/4142045455 conn(0x5633d1baa160 msgr2=0x5633d1baa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- 172.21.15.116:0/4142045455 shutdown_connections 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- 172.21.15.116:0/4142045455 wait complete. 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.738+0000 7f70fc7a3740 1 -- start start 2024-03-19T02:37:26.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 Processor -- start 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 0x5633d1bba2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d1bc0100 con 0x5633d1bafdc0 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5633d1acf160 con 0x5633d1baf410 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633d1bc0280 con 0x5633d1bbd340 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70fc7a3740 10 client.0 _ll_register_callbacks cb 0x5633d1b0ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 0x5633d1bba2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55090/0 (socket says 172.21.15.116:55090) 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f2ffd640 1 -- 172.21.15.116:0/2191974587 learned_addr learned my addr 172.21.15.116:0/2191974587 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f3fff640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 -- 172.21.15.116:0/2191974587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 msgr2=0x5633d1bbad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 -- 172.21.15.116:0/2191974587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 msgr2=0x5633d1bba830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f37fe640 1 -- 172.21.15.116:0/2191974587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633d18f17d0 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.739+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70f37fe640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 0x5633d1bba2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e402c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88a09f34afe83a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e402c610 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70ec00fd40 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f70e4005030 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f70e4005230 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 10 client.28160 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5633d18273f0 con 0x5633d1baf410 2024-03-19T02:37:26.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ec00fd40 con 0x5633d1baf410 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f70e40035f0 con 0x5633d1baf410 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 10 client.28160 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 10 client.28160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 20 client.28160 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1bcd580 0x5633d1bcf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 4 client.28160 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.740+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633d1b20a40 con 0x5633d1bcd580 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.741+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1bcd580 0x5633d1bcf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.741+0000 7f70d2ffd640 10 client.28160 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.741+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1bcd580 0x5633d1bcf9b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c88a929fe119a072 server_cookie=b81e05cd15a85e4e in_seq=0 out_seq=0 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.741+0000 7f70d2ffd640 1 -- 172.21.15.116:0/2191974587 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633d1b20a40 con 0x5633d1bcd580 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.741+0000 7f70d2ffd640 10 client.28160 handle_command_reply: tid=0 2024-03-19T02:37:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 client.28160 shutdown 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 msgr2=0x5633d1bba2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 0x5633d1bba2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e402c9f0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 shutdown_connections 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1bcd580 0x5633d1bcf9b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bbad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bafdc0 0x5633d1bba830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70d2ffd640 0 client.28160 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/2191974587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1baf410 0x5633d1bba2f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 >> 172.21.15.116:0/2191974587 conn(0x5633d1baa160 msgr2=0x5633d1babeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 shutdown_connections 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 1 -- 172.21.15.116:0/2191974587 wait complete. 2024-03-19T02:37:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.742+0000 7f70fc7a3740 20 client.28160 trim_cache size 0 max 16384 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 Processor -- start 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 -- start start 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1baf410 0x5633d1bbca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41aca40 con 0x5633d1bbd340 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41ae580 con 0x5633d1bafdc0 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41ad110 con 0x5633d1baf410 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55100/0 (socket says 172.21.15.116:55100) 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70f3fff640 1 -- 172.21.15.116:0/1068172694 learned_addr learned my addr 172.21.15.116:0/1068172694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.762+0000 7f70f37fe640 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1baf410 0x5633d1bbca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 -- 172.21.15.116:0/1068172694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1baf410 msgr2=0x5633d1bbca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1baf410 0x5633d1bbca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 -- 172.21.15.116:0/1068172694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 msgr2=0x5633d1bb9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 -- 172.21.15.116:0/1068172694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d17e70f0 con 0x5633d1bbd340 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011a90 tx=0x7f70ec026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abebb7ecd1442f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70d97fa640 1 -- 172.21.15.116:0/1068172694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec01f030 con 0x5633d1bbd340 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70d97fa640 1 -- 172.21.15.116:0/1068172694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ec023e70 con 0x5633d1bbd340 2024-03-19T02:37:26.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.763+0000 7f70d97fa640 1 -- 172.21.15.116:0/1068172694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec004c20 con 0x5633d1bbd340 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- 172.21.15.116:0/1068172694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 msgr2=0x5633d1bc8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f70ec011a90 tx=0x7f70ec026000 comp rx=0 tx=0).stop 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- 172.21.15.116:0/1068172694 shutdown_connections 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1bbd340 0x5633d1bc8ee0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/1068172694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1baf410 0x5633d1bbca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- 172.21.15.116:0/1068172694 >> 172.21.15.116:0/1068172694 conn(0x5633d1baa160 msgr2=0x5633d1ba91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- 172.21.15.116:0/1068172694 shutdown_connections 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- 172.21.15.116:0/1068172694 wait complete. 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.764+0000 7f70fc7a3740 1 -- start start 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 Processor -- start 2024-03-19T02:37:26.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bb79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41adeb0 con 0x5633d1baf410 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41ad7e0 con 0x5633d1bafdc0 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70f41aaa40 con 0x5633d1bbd340 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70fc7a3740 10 client.0 _ll_register_callbacks cb 0x7f70e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55110/0 (socket says 172.21.15.116:55110) 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 -- 172.21.15.116:0/3933717650 learned_addr learned my addr 172.21.15.116:0/3933717650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f3fff640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bb79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 -- 172.21.15.116:0/3933717650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 msgr2=0x5633d1bb79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bb79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 -- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 msgr2=0x5633d1bb7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.765+0000 7f70f37fe640 1 -- 172.21.15.116:0/3933717650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633d1c32900 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70f37fe640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c170 tx=0x7f70e40051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3c72008bd08c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e402c610 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f70e40053d0 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70ec023e70 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f70e4062050 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 10 client.18372 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5633d1c33c10 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ec023e70 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f70e4004c00 con 0x5633d1baf410 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 10 client.18372 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 10 client.18372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 20 client.18372 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1c343b0 0x5633d1c367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 4 client.18372 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.766+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5633d1b1ad90 con 0x5633d1c343b0 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1c343b0 0x5633d1c367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70d2ffd640 10 client.18372 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1c343b0 0x5633d1c367d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956f7cfabf30f662 server_cookie=d0e1120d0d4f4581 in_seq=0 out_seq=0 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70d2ffd640 1 -- 172.21.15.116:0/3933717650 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5633d1b1ad90 con 0x5633d1c343b0 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70d2ffd640 10 client.18372 handle_command_reply: tid=0 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 client.18372 shutdown 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 msgr2=0x5633d1bb6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c170 tx=0x7f70e40051a0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 shutdown_connections 2024-03-19T02:37:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5633d1c343b0 0x5633d1c367d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633d1bbd340 0x5633d1bb79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633d1bafdc0 0x5633d1bb7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 --2- 172.21.15.116:0/3933717650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d1baf410 0x5633d1bb6f30 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70d2ffd640 0 client.18372 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.767+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 >> 172.21.15.116:0/3933717650 conn(0x5633d1baa160 msgr2=0x5633d1bab210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 shutdown_connections 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70fc7a3740 1 -- 172.21.15.116:0/3933717650 wait complete. 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70fc7a3740 20 client.18372 trim_cache size 0 max 16384 2024-03-19T02:37:26.774 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:37:26.775 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:37:26.775 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:37:26.775 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 msgr2=0x7f70dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 0x7f70dc0a4e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f70ec020000 tx=0x7f70ec023d40 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 msgr2=0x7f70f41992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f70e800e9f0 tx=0x7f70e800edf0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 shutdown_connections 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f70dc0a2970 0x7f70dc0a4e30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f4106750 0x7f70f41a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70f41013c0 0x7f70f41992d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 --2- 172.21.15.116:0/631701437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70f4100a00 0x7f70f4198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 >> 172.21.15.116:0/631701437 conn(0x7f70f4077700 msgr2=0x7f70f40ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 shutdown_connections 2024-03-19T02:37:26.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.768+0000 7f70f9bb9640 1 -- 172.21.15.116:0/631701437 wait complete. 2024-03-19T02:37:26.780 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:37:26.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:37:26.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 312004 3.14 MB/sec execute 409 sec latency 16.610 ms 2024-03-19T02:37:26.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- 172.21.15.116:0/3878862312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4106730 msgr2=0x7f88d4106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 --2- 172.21.15.116:0/3878862312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4106730 0x7f88d4106b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009930 tx=0x7f88c402bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- 172.21.15.116:0/3878862312 shutdown_connections 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 --2- 172.21.15.116:0/3878862312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d41010e0 0x7f88d410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 --2- 172.21.15.116:0/3878862312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d4100720 0x7f88d4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 --2- 172.21.15.116:0/3878862312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4106730 0x7f88d4106b10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- 172.21.15.116:0/3878862312 >> 172.21.15.116:0/3878862312 conn(0x7f88d4077820 msgr2=0x7f88d4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- 172.21.15.116:0/3878862312 shutdown_connections 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- 172.21.15.116:0/3878862312 wait complete. 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 Processor -- start 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.890+0000 7f88da44b640 1 -- start start 2024-03-19T02:37:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d41010e0 0x7f88d4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 0x7f88d419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88d41109b0 con 0x7f88d4106730 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88d4110830 con 0x7f88d41010e0 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88da44b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f88d4110b30 con 0x7f88d4100720 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d89e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 0x7f88d419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d41010e0 0x7f88d4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:49060/0 (socket says 172.21.15.116:49060) 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 -- 172.21.15.116:0/4217848850 learned_addr learned my addr 172.21.15.116:0/4217848850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 -- 172.21.15.116:0/4217848850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d41010e0 msgr2=0x7f88d4198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d41010e0 0x7f88d4198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 -- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 msgr2=0x7f88d419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:26.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 0x7f88d419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 -- 172.21.15.116:0/4217848850 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d419d060 con 0x7f88d4100720 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d89e8640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 0x7f88d419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.891+0000 7f88d3fff640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009230 tx=0x7f88c402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.892+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4043020 con 0x7f88d4100720 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.892+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88c402ca70 con 0x7f88d4100720 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.892+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d419d2f0 con 0x7f88d4100720 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.892+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4003a50 con 0x7f88d4100720 2024-03-19T02:37:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.892+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d41a4be0 con 0x7f88d4100720 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.893+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d4068610 con 0x7f88d4100720 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.894+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f88c4046020 con 0x7f88d4100720 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.895+0000 7f88d17fa640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 0x7f88bc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.895+0000 7f88d37fe640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 0x7f88bc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.895+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f88c40f1430 con 0x7f88d4100720 2024-03-19T02:37:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.895+0000 7f88d37fe640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 0x7f88bc0a4eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88d4069a60 tx=0x7f88c8005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:26.898+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f88c404d020 con 0x7f88d4100720 2024-03-19T02:37:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.153+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d41993c0 con 0x7f88d4100720 2024-03-19T02:37:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.154+0000 7f88d17fa640 1 -- 172.21.15.116:0/4217848850 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f88d41993c0 con 0x7f88d4100720 2024-03-19T02:37:27.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:37:27.154 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:37:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:37:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 msgr2=0x7f88bc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 0x7f88bc0a4eb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88d4069a60 tx=0x7f88c8005f90 comp rx=0 tx=0).stop 2024-03-19T02:37:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 msgr2=0x7f88d41980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009230 tx=0x7f88c402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 shutdown_connections 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f88bc0a29f0 0x7f88bc0a4eb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88d4106730 0x7f88d419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88d41010e0 0x7f88d4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 --2- 172.21.15.116:0/4217848850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88d4100720 0x7f88d41980c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 >> 172.21.15.116:0/4217848850 conn(0x7f88d4077820 msgr2=0x7f88d40ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 shutdown_connections 2024-03-19T02:37:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.156+0000 7f88da44b640 1 -- 172.21.15.116:0/4217848850 wait complete. 2024-03-19T02:37:27.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:37:27.233 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:26 smithi116 ceph-mon[32263]: pgmap v300: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:26 smithi154 ceph-mon[35693]: pgmap v300: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:27.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 -- 172.21.15.116:0/2643736038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5541077a0 msgr2=0x7ff554107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 --2- 172.21.15.116:0/2643736038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5541077a0 0x7ff554107b80 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff544009930 tx=0x7ff54402bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 -- 172.21.15.116:0/2643736038 shutdown_connections 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 --2- 172.21.15.116:0/2643736038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff554102150 0x7ff55410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 --2- 172.21.15.116:0/2643736038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 --2- 172.21.15.116:0/2643736038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5541077a0 0x7ff554107b80 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 -- 172.21.15.116:0/2643736038 >> 172.21.15.116:0/2643736038 conn(0x7ff5540fd2c0 msgr2=0x7ff5540ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 -- 172.21.15.116:0/2643736038 shutdown_connections 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 -- 172.21.15.116:0/2643736038 wait complete. 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.321+0000 7ff559722640 1 Processor -- start 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 -- start start 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5541077a0 0x7ff5540745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff554111a20 con 0x7ff554102150 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541118a0 con 0x7ff5541077a0 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff559722640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff554111ba0 con 0x7ff554101790 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55146/0 (socket says 172.21.15.116:55146) 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:49084/0 (socket says 172.21.15.116:49084) 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff5527fc640 1 -- 172.21.15.116:0/3226322212 learned_addr learned my addr 172.21.15.116:0/3226322212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff5537fe640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5541077a0 0x7ff5540745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 -- 172.21.15.116:0/3226322212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5541077a0 msgr2=0x7ff5540745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5541077a0 0x7ff5540745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 -- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 msgr2=0x7ff554074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff552ffd640 1 -- 172.21.15.116:0/3226322212 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5541a64f0 con 0x7ff554101790 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.322+0000 7ff5527fc640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff552ffd640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff544009230 tx=0x7ff5440045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff53bfff640 1 -- 172.21.15.116:0/3226322212 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544043020 con 0x7ff554101790 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5541a6780 con 0x7ff554101790 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5541a6cf0 con 0x7ff554101790 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff53bfff640 1 -- 172.21.15.116:0/3226322212 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff54402ca70 con 0x7ff554101790 2024-03-19T02:37:27.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.323+0000 7ff53bfff640 1 -- 172.21.15.116:0/3226322212 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544003a10 con 0x7ff554101790 2024-03-19T02:37:27.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 Processor -- start 2024-03-19T02:37:27.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 -- start start 2024-03-19T02:37:27.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1fbf0 0x55c21ea28290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea2abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c21e9824b0 con 0x55c21ea1f2d0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c21e92ad70 con 0x55c21ea1fbf0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c21e97e550 con 0x55c21ea287d0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff53bfff640 1 -- 172.21.15.116:0/3226322212 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff544046020 con 0x7ff554101790 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.325+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea2abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55152/0 (socket says 172.21.15.116:55152) 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 -- 172.21.15.116:0/1569939914 learned_addr learned my addr 172.21.15.116:0/1569939914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff552ffd640 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1fbf0 0x55c21ea28290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 -- 172.21.15.116:0/1569939914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 msgr2=0x55c21ea2abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea2abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 -- 172.21.15.116:0/1569939914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1fbf0 msgr2=0x55c21ea28290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1fbf0 0x55c21ea28290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 -- 172.21.15.116:0/1569939914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c21e94d210 con 0x55c21ea1f2d0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff53bfff640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 0x7ff53c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5537fe640 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009f50 tx=0x7ff54c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adc37e6b445240b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff552ffd640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 0x7ff53c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5397fa640 1 -- 172.21.15.116:0/1569939914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c004d20 con 0x55c21ea1f2d0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5397fa640 1 -- 172.21.15.116:0/1569939914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff54c004050 con 0x55c21ea1f2d0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff5397fa640 1 -- 172.21.15.116:0/1569939914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c005280 con 0x55c21ea1f2d0 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.326+0000 7ff53bfff640 1 -- 172.21.15.116:0/3226322212 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7ff5440f0a00 con 0x7ff554101790 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- 172.21.15.116:0/1569939914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 msgr2=0x55c21ea1f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff552ffd640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 0x7ff53c0a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5440edf30 tx=0x7ff5440f4550 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009f50 tx=0x7ff54c0118d0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- 172.21.15.116:0/1569939914 shutdown_connections 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea2abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1fbf0 0x55c21ea28290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/1569939914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea1f6b0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- 172.21.15.116:0/1569939914 >> 172.21.15.116:0/1569939914 conn(0x55c21ea19a00 msgr2=0x55c21ea19de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- 172.21.15.116:0/1569939914 shutdown_connections 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- 172.21.15.116:0/1569939914 wait complete. 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 -- start start 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.327+0000 7ff55c3ed740 1 Processor -- start 2024-03-19T02:37:27.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1fbf0 0x55c21ea25d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea26270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c21ea2b5c0 con 0x55c21ea1f2d0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c21e93e310 con 0x55c21ea287d0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c21ea2b740 con 0x55c21ea1fbf0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 10 client.0 _ll_register_callbacks cb 0x55c21e9322d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55168/0 (socket says 172.21.15.116:55168) 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 -- 172.21.15.116:0/2075195581 learned_addr learned my addr 172.21.15.116:0/2075195581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1fbf0 0x55c21ea25d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea26270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 -- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1fbf0 msgr2=0x55c21ea25d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1fbf0 0x55c21ea25d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 -- 172.21.15.116:0/2075195581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 msgr2=0x55c21ea26270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea26270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 -- 172.21.15.116:0/2075195581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c21e6963f0 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009250 tx=0x7ff54c004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbdfc4aa708d202 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c02b020 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7ff54c005040 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.328+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff54c004050 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff54c004e50 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 10 client.18390 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c21e7607d0 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff54c004050 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff54c027680 con 0x55c21ea1f2d0 2024-03-19T02:37:27.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 10 client.18390 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 10 client.18390 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 20 client.18390 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea38b30 0x55c21ea3af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 4 client.18390 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c21e9e03e0 con 0x55c21ea38b30 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.329+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea38b30 0x55c21ea3af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.330+0000 7ff536ff5640 10 client.18390 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.330+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea38b30 0x55c21ea3af20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7689a72334533c99 server_cookie=d604140a7372968d in_seq=0 out_seq=0 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff536ff5640 1 -- 172.21.15.116:0/2075195581 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c21e9e03e0 con 0x55c21ea38b30 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff536ff5640 10 client.18390 handle_command_reply: tid=0 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 client.18390 shutdown 2024-03-19T02:37:27.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 msgr2=0x55c21ea257f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009250 tx=0x7ff54c004930 comp rx=0 tx=0).stop 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 shutdown_connections 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea38b30 0x55c21ea3af20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea26270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1fbf0 0x55c21ea25d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff536ff5640 0 client.18390 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2075195581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1f2d0 0x55c21ea257f0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 >> 172.21.15.116:0/2075195581 conn(0x55c21ea19a00 msgr2=0x55c21ea1b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 shutdown_connections 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2075195581 wait complete. 2024-03-19T02:37:27.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.331+0000 7ff55c3ed740 20 client.18390 trim_cache size 0 max 16384 2024-03-19T02:37:27.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 Processor -- start 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 -- start start 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1f2d0 0x55c21ea34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea40550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541b0ef0 con 0x55c21ea1fbf0 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541b2a30 con 0x55c21ea1f2d0 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541b15c0 con 0x55c21ea287d0 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55178/0 (socket says 172.21.15.116:55178) 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff5527fc640 1 -- 172.21.15.116:0/2336469796 learned_addr learned my addr 172.21.15.116:0/2336469796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1f2d0 0x55c21ea34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.345+0000 7ff5537fe640 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea40550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 -- 172.21.15.116:0/2336469796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 msgr2=0x55c21ea40550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea40550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 -- 172.21.15.116:0/2336469796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1f2d0 msgr2=0x55c21ea34360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1f2d0 0x55c21ea34360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 -- 172.21.15.116:0/2336469796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c21ea2abc0 con 0x55c21ea1fbf0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff5527fc640 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff548012700 tx=0x7ff548005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b967499575cdec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff538ff9640 1 -- 172.21.15.116:0/2336469796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548011a70 con 0x55c21ea1fbf0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff538ff9640 1 -- 172.21.15.116:0/2336469796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff548024030 con 0x55c21ea1fbf0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.346+0000 7ff538ff9640 1 -- 172.21.15.116:0/2336469796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548004b20 con 0x55c21ea1fbf0 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2336469796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 msgr2=0x55c21ea26090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff548012700 tx=0x7ff548005f90 comp rx=0 tx=0).stop 2024-03-19T02:37:27.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2336469796 shutdown_connections 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea287d0 0x55c21ea40550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea26090 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/2336469796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea1f2d0 0x55c21ea34360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2336469796 >> 172.21.15.116:0/2336469796 conn(0x55c21ea19a00 msgr2=0x55c21ea18ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2336469796 shutdown_connections 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- 172.21.15.116:0/2336469796 wait complete. 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- start start 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 Processor -- start 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1f2d0 0x55c21ea30bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea225a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541b2360 con 0x55c21ea1fbf0 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541b1c90 con 0x55c21ea287d0 2024-03-19T02:37:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.347+0000 7ff55c3ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541aeef0 con 0x55c21ea1f2d0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff55c3ed740 10 client.0 _ll_register_callbacks cb 0x7ff5440f4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55188/0 (socket says 172.21.15.116:55188) 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 -- 172.21.15.116:0/3112756796 learned_addr learned my addr 172.21.15.116:0/3112756796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5537fe640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1f2d0 0x55c21ea30bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5537fe640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea225a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 -- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1f2d0 msgr2=0x55c21ea30bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1f2d0 0x55c21ea30bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 -- 172.21.15.116:0/3112756796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 msgr2=0x55c21ea225a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea225a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 -- 172.21.15.116:0/3112756796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c21ea22da0 con 0x55c21ea1fbf0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff5527fc640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff548014010 tx=0x7ff5480051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedb6f00353eccbb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480055c0 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7ff5480200d0 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff548024030 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff54803a050 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff55c3ed740 10 client.18396 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.348+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c21ea9eb50 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff548024030 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff548011870 con 0x55c21ea1fbf0 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 10 client.18396 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 10 client.18396 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 20 client.18396 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea9f1c0 0x55c21eaa15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 4 client.18396 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c21e98f9f0 con 0x55c21ea9f1c0 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff5537fe640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea9f1c0 0x55c21eaa15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff5537fe640 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea9f1c0 0x55c21eaa15e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b00dd8cc79f41264 server_cookie=bf660f650d64355c in_seq=0 out_seq=0 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff536ff5640 10 client.18396 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff536ff5640 1 -- 172.21.15.116:0/3112756796 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c21e98f9f0 con 0x55c21ea9f1c0 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff536ff5640 10 client.18396 handle_command_reply: tid=0 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.349+0000 7ff55c3ed740 1 client.18396 shutdown 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 msgr2=0x55c21ea22060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff548014010 tx=0x7ff5480051d0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 shutdown_connections 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c21ea9f1c0 0x55c21eaa15e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c21ea287d0 0x55c21ea225a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff536ff5640 0 client.18396 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c21ea1fbf0 0x55c21ea22060 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 --2- 172.21.15.116:0/3112756796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c21ea1f2d0 0x55c21ea30bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 >> 172.21.15.116:0/3112756796 conn(0x55c21ea19a00 msgr2=0x55c21ea1abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 shutdown_connections 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 1 -- 172.21.15.116:0/3112756796 wait complete. 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff55c3ed740 20 client.18396 trim_cache size 0 max 16384 2024-03-19T02:37:27.358 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:37:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 msgr2=0x7ff53c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 0x7ff53c0a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5440edf30 tx=0x7ff5440f4550 comp rx=0 tx=0).stop 2024-03-19T02:37:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 msgr2=0x7ff554073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff544009230 tx=0x7ff5440045e0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 shutdown_connections 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff53c0a2970 0x7ff53c0a4e30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5541077a0 0x7ff5540745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff554102150 0x7ff554074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 --2- 172.21.15.116:0/3226322212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff554101790 0x7ff554073b20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.350+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 >> 172.21.15.116:0/3226322212 conn(0x7ff5540fd2c0 msgr2=0x7ff5540fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.351+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 shutdown_connections 2024-03-19T02:37:27.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.351+0000 7ff559722640 1 -- 172.21.15.116:0/3226322212 wait complete. 2024-03-19T02:37:27.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:37:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:26 smithi162 ceph-mon[35761]: pgmap v300: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- 172.21.15.116:0/1894171341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8081066b0 msgr2=0x7fc808106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/1894171341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8081066b0 0x7fc808106a90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fc804009930 tx=0x7fc80402bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- 172.21.15.116:0/1894171341 shutdown_connections 2024-03-19T02:37:27.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/1894171341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc808101060 0x7fc80810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/1894171341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081006a0 0x7fc808100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/1894171341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8081066b0 0x7fc808106a90 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- 172.21.15.116:0/1894171341 >> 172.21.15.116:0/1894171341 conn(0x7fc8080778d0 msgr2=0x7fc808077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- 172.21.15.116:0/1894171341 shutdown_connections 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- 172.21.15.116:0/1894171341 wait complete. 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 Processor -- start 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.523+0000 7fc80fc3d640 1 -- start start 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8081006a0 0x7fc808102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 0x7fc808103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc808110970 con 0x7fc808101060 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081107f0 con 0x7fc8081066b0 2024-03-19T02:37:27.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80fc3d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc808110af0 con 0x7fc8081006a0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55212/0 (socket says 172.21.15.116:55212) 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80d1d8640 1 -- 172.21.15.116:0/2404464205 learned_addr learned my addr 172.21.15.116:0/2404464205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 0x7fc808103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8081006a0 0x7fc808102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 -- 172.21.15.116:0/2404464205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8081006a0 msgr2=0x7fc808102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8081006a0 0x7fc808102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 -- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 msgr2=0x7fc808103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 -- 172.21.15.116:0/2404464205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8081a64c0 con 0x7fc8081066b0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.524+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 0x7fc808103790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc800009fd0 tx=0x7fc80000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.525+0000 7fc7feffd640 1 -- 172.21.15.116:0/2404464205 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80000ce20 con 0x7fc8081066b0 2024-03-19T02:37:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.525+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8081a67b0 con 0x7fc8081066b0 2024-03-19T02:37:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.525+0000 7fc7feffd640 1 -- 172.21.15.116:0/2404464205 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc800005020 con 0x7fc8081066b0 2024-03-19T02:37:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.525+0000 7fc7feffd640 1 -- 172.21.15.116:0/2404464205 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800010690 con 0x7fc8081066b0 2024-03-19T02:37:27.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.525+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8081a6d50 con 0x7fc8081066b0 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.527+0000 7fc812821740 1 Processor -- start 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.527+0000 7fc7feffd640 1 -- 172.21.15.116:0/2404464205 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc8000051d0 con 0x7fc8081066b0 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.527+0000 7fc812821740 1 -- start start 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb82290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb827d0 0x5654bcb84bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654bcadc4b0 con 0x5654bcb827d0 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5654bca84d70 con 0x5654bcb79bf0 2024-03-19T02:37:27.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc812821740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5654bcad8550 con 0x5654bcb792d0 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb827d0 0x5654bcb84bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d9d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb82290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:49130/0 (socket says 172.21.15.116:49130) 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d9d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb82290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:44918/0 (socket says 172.21.15.116:44918) 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 -- 172.21.15.116:0/808607574 learned_addr learned my addr 172.21.15.116:0/808607574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc7feffd640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 0x7fc7f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 -- 172.21.15.116:0/808607574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 msgr2=0x5654bcb82290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb82290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 -- 172.21.15.116:0/808607574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb827d0 msgr2=0x5654bcb84bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb827d0 0x5654bcb84bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 -- 172.21.15.116:0/808607574 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bcaa7210 con 0x5654bcb792d0 2024-03-19T02:37:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 0x7fc7f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8010de0 tx=0x7fc7f8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f850bc171799f47e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.528+0000 7fc7feffd640 1 -- 172.21.15.116:0/2404464205 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc8000cf890 con 0x7fc8081066b0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc7f3fff640 1 -- 172.21.15.116:0/808607574 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8016070 con 0x5654bcb792d0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc7f3fff640 1 -- 172.21.15.116:0/808607574 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f800fe50 con 0x5654bcb792d0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc7f3fff640 1 -- 172.21.15.116:0/808607574 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8019040 con 0x5654bcb792d0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 -- 172.21.15.116:0/808607574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 msgr2=0x5654bcb796b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc80e1da640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 0x7fc7f40a4e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc800097f70 tx=0x7fc80006e7f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8010de0 tx=0x7fc7f8005f90 comp rx=0 tx=0).stop 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 -- 172.21.15.116:0/808607574 shutdown_connections 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb827d0 0x5654bcb84bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb82290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 --2- 172.21.15.116:0/808607574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb792d0 0x5654bcb796b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.529+0000 7fc812821740 1 -- 172.21.15.116:0/808607574 >> 172.21.15.116:0/808607574 conn(0x5654bcb73a00 msgr2=0x5654bcb73de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- 172.21.15.116:0/808607574 shutdown_connections 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- 172.21.15.116:0/808607574 wait complete. 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- start start 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 Processor -- start 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb79bf0 0x5654bcb7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb827d0 0x5654bcb80260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb85560 con 0x5654bcb792d0 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5654bca98310 con 0x5654bcb827d0 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.530+0000 7fc812821740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb856e0 con 0x5654bcb79bf0 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc812821740 10 client.0 _ll_register_callbacks cb 0x5654bca8c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55230/0 (socket says 172.21.15.116:55230) 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1462793445 learned_addr learned my addr 172.21.15.116:0/1462793445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb79bf0 0x5654bcb7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb827d0 0x5654bcb80260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb79bf0 msgr2=0x5654bcb7fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb79bf0 0x5654bcb7fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1462793445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb827d0 msgr2=0x5654bcb80260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb827d0 0x5654bcb80260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1462793445 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bc7f03f0 con 0x5654bcb792d0 2024-03-19T02:37:27.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800a010 tx=0x7fc7f8010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c8b6f6d5a918b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc7f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8016040 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc7f800e7d0 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f800fe50 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc7f800e9d0 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc812821740 10 client.18408 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5654bc8ba7d0 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.531+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f800fe50 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc7f801eb20 con 0x5654bcb792d0 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 10 client.18408 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 10 client.18408 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 20 client.18408 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcb929b0 0x5654bcb94da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 4 client.18408 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654bcb3a3e0 con 0x5654bcb929b0 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcb929b0 0x5654bcb94da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc7f17fa640 10 client.18408 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.532+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcb929b0 0x5654bcb94da0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29da950873731c30 server_cookie=b43ffe4cd9a3af57 in_seq=0 out_seq=0 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc7f17fa640 1 -- 172.21.15.116:0/1462793445 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5654bcb3a3e0 con 0x5654bcb929b0 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc7f17fa640 10 client.18408 handle_command_reply: tid=0 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 client.18408 shutdown 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 msgr2=0x5654bcb7f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800a010 tx=0x7fc7f8010f90 comp rx=0 tx=0).stop 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 shutdown_connections 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb827d0 0x5654bcb80260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb79bf0 0x5654bcb7fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcb929b0 0x5654bcb94da0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 --2- 172.21.15.116:0/1462793445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f7e0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc7f17fa640 0 client.18408 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 >> 172.21.15.116:0/1462793445 conn(0x5654bcb73a00 msgr2=0x5654bcb75750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 shutdown_connections 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 1 -- 172.21.15.116:0/1462793445 wait complete. 2024-03-19T02:37:27.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.533+0000 7fc812821740 20 client.18408 trim_cache size 0 max 16384 2024-03-19T02:37:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.548+0000 7fc812821740 1 Processor -- start 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.548+0000 7fc812821740 1 -- start start 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb8e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb72310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081b0ef0 con 0x5654bcb792d0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081b2a30 con 0x5654bcb79bf0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc812821740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081b15c0 con 0x5654bcb827d0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55240/0 (socket says 172.21.15.116:55240) 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1441618260 learned_addr learned my addr 172.21.15.116:0/1441618260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb8e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb72310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1441618260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 msgr2=0x5654bcb72310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb72310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1441618260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 msgr2=0x5654bcb8e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb8e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 -- 172.21.15.116:0/1441618260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bcb79a20 con 0x5654bcb792d0 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb8e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8006010 tx=0x7fc7f802b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dece273265a9a121 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.549+0000 7fc7f37fe640 1 -- 172.21.15.116:0/1441618260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8016070 con 0x5654bcb792d0 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc7f37fe640 1 -- 172.21.15.116:0/1441618260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f8009c10 con 0x5654bcb792d0 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc7f37fe640 1 -- 172.21.15.116:0/1441618260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8023040 con 0x5654bcb792d0 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 -- 172.21.15.116:0/1441618260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 msgr2=0x5654bcb7f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8006010 tx=0x7fc7f802b420 comp rx=0 tx=0).stop 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 -- 172.21.15.116:0/1441618260 shutdown_connections 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb72310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb8e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 --2- 172.21.15.116:0/1441618260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb7f0c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 -- 172.21.15.116:0/1441618260 >> 172.21.15.116:0/1441618260 conn(0x5654bcb73a00 msgr2=0x5654bcb80030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 -- 172.21.15.116:0/1441618260 shutdown_connections 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.550+0000 7fc812821740 1 -- 172.21.15.116:0/1441618260 wait complete. 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 -- start start 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 Processor -- start 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb7c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb7c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081b2360 con 0x5654bcb792d0 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081b1c90 con 0x5654bcb79bf0 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081aeef0 con 0x5654bcb827d0 2024-03-19T02:37:27.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc812821740 10 client.0 _ll_register_callbacks cb 0x7fc7f8010720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc80d1d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55246/0 (socket says 172.21.15.116:55246) 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc80d1d8640 1 -- 172.21.15.116:0/2258114646 learned_addr learned my addr 172.21.15.116:0/2258114646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb7c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.551+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb7c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 -- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 msgr2=0x5654bcb7c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb7c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 -- 172.21.15.116:0/2258114646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 msgr2=0x5654bcb7c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb7c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 -- 172.21.15.116:0/2258114646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bcb7cba0 con 0x5654bcb792d0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc80d1d8640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8010950 tx=0x7fc7f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8edd1c1a717b2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc7f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8016040 con 0x5654bcb792d0 2024-03-19T02:37:27.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc7f800e5d0 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f8009c10 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc7f800e7d0 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc812821740 10 client.18414 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5654bcbf8920 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.552+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f8009c10 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc7f80035f0 con 0x5654bcb792d0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 10 client.18414 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 10 client.18414 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 20 client.18414 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcbf8f80 0x5654bcbfb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 4 client.18414 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654bcae99f0 con 0x5654bcbf8f80 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcbf8f80 0x5654bcbfb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc80d9d9640 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcbf8f80 0x5654bcbfb370 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a71194c9ca3ad74e server_cookie=fd36a44427de5f2c in_seq=0 out_seq=0 2024-03-19T02:37:27.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc7f17fa640 10 client.18414 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc7f17fa640 1 -- 172.21.15.116:0/2258114646 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654bcae99f0 con 0x5654bcbf8f80 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc7f17fa640 10 client.18414 handle_command_reply: tid=0 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 1 client.18414 shutdown 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 msgr2=0x5654bcb8ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.553+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8010950 tx=0x7fc7f8009780 comp rx=0 tx=0).stop 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 shutdown_connections 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcb827d0 0x5654bcb7c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5654bcb79bf0 0x5654bcb7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5654bcbf8f80 0x5654bcbfb370 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 --2- 172.21.15.116:0/2258114646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654bcb792d0 0x5654bcb8ab20 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc7f17fa640 0 client.18414 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 >> 172.21.15.116:0/2258114646 conn(0x5654bcb73a00 msgr2=0x5654bcb74c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 shutdown_connections 2024-03-19T02:37:27.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 1 -- 172.21.15.116:0/2258114646 wait complete. 2024-03-19T02:37:27.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc812821740 20 client.18414 trim_cache size 0 max 16384 2024-03-19T02:37:27.562 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 msgr2=0x7fc7f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 0x7fc7f40a4e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc800097f70 tx=0x7fc80006e7f0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 msgr2=0x7fc808103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 0x7fc808103790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc800009fd0 tx=0x7fc80000ef10 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 shutdown_connections 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7f40a2970 0x7fc7f40a4e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8081066b0 0x7fc808103790 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc808101060 0x7fc808103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 --2- 172.21.15.116:0/2404464205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8081006a0 0x7fc808102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 >> 172.21.15.116:0/2404464205 conn(0x7fc8080778d0 msgr2=0x7fc808068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 shutdown_connections 2024-03-19T02:37:27.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.554+0000 7fc80fc3d640 1 -- 172.21.15.116:0/2404464205 wait complete. 2024-03-19T02:37:27.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 -- 172.21.15.116:0/657004989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770105690 msgr2=0x7f1770105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 --2- 172.21.15.116:0/657004989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770105690 0x7f1770105a70 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f176c009900 tx=0x7f176c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 -- 172.21.15.116:0/657004989 shutdown_connections 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 --2- 172.21.15.116:0/657004989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770101330 0x7f177010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 --2- 172.21.15.116:0/657004989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 0x7f1770100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 --2- 172.21.15.116:0/657004989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770105690 0x7f1770105a70 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 -- 172.21.15.116:0/657004989 >> 172.21.15.116:0/657004989 conn(0x7f17700774b0 msgr2=0x7f17700778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 -- 172.21.15.116:0/657004989 shutdown_connections 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.680+0000 7f17774f8640 1 -- 172.21.15.116:0/657004989 wait complete. 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 Processor -- start 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 -- start start 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 0x7f1770196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770105690 0x7f1770197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1770110a20 con 0x7f1770101330 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17701108a0 con 0x7f1770100970 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f17774f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1770110ba0 con 0x7f1770105690 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55450/0 (socket says 172.21.15.116:55450) 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f1774a93640 1 -- 172.21.15.116:0/1854260502 learned_addr learned my addr 172.21.15.116:0/1854260502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f1775294640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 0x7f1770196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.681+0000 7f1775a95640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770105690 0x7f1770197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 -- 172.21.15.116:0/1854260502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770105690 msgr2=0x7f1770197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770105690 0x7f1770197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 -- 172.21.15.116:0/1854260502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 msgr2=0x7f1770196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 0x7f1770196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 -- 172.21.15.116:0/1854260502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17701a64b0 con 0x7f1770101330 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f1774a93640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f176000ea20 tx=0x7f176000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f17667fc640 1 -- 172.21.15.116:0/1854260502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176000cd80 con 0x7f1770101330 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f17667fc640 1 -- 172.21.15.116:0/1854260502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17600108e0 con 0x7f1770101330 2024-03-19T02:37:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17701a67a0 con 0x7f1770101330 2024-03-19T02:37:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f17667fc640 1 -- 172.21.15.116:0/1854260502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760018840 con 0x7f1770101330 2024-03-19T02:37:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.682+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17701a6cf0 con 0x7f1770101330 2024-03-19T02:37:27.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.684+0000 7f17667fc640 1 -- 172.21.15.116:0/1854260502 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1760010a90 con 0x7f1770101330 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 Processor -- start 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 -- start start 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f17667fc640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 0x7f175c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 0x561bfe503290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe505bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561bfe45d4b0 con 0x561bfe5037d0 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561bfe405d70 con 0x561bfe4fabf0 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f177a1c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bfe459550 con 0x561bfe4fa2d0 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1775a95640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 0x7f175c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1775294640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f17667fc640 1 -- 172.21.15.116:0/1854260502 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f17600cf4d0 con 0x7f1770101330 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1775294640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48188/0 (socket says 172.21.15.116:48188) 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1775294640 1 -- 172.21.15.116:0/3488162153 learned_addr learned my addr 172.21.15.116:0/3488162153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1774a93640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 0x561bfe503290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.685+0000 7f1775a95640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe505bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 -- 172.21.15.116:0/3488162153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 msgr2=0x561bfe503290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 0x561bfe503290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 -- 172.21.15.116:0/3488162153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 msgr2=0x561bfe505bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe505bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 -- 172.21.15.116:0/3488162153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bfe428210 con 0x561bfe4fa2d0 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1774a93640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 0x561bfe503290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775a95640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 0x7f175c0a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1768005fd0 tx=0x7f1768009f10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f1775294640 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f176c009900 tx=0x7f176c005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c719ebb7d3044d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f175b7fe640 1 -- 172.21.15.116:0/3488162153 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c02c800 con 0x561bfe4fa2d0 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f175b7fe640 1 -- 172.21.15.116:0/3488162153 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f176c02c9a0 con 0x561bfe4fa2d0 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.686+0000 7f175b7fe640 1 -- 172.21.15.116:0/3488162153 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c0046c0 con 0x561bfe4fa2d0 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- 172.21.15.116:0/3488162153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 msgr2=0x561bfe4fa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f176c009900 tx=0x7f176c005e40 comp rx=0 tx=0).stop 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- 172.21.15.116:0/3488162153 shutdown_connections 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe505bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fabf0 0x561bfe503290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 --2- 172.21.15.116:0/3488162153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fa6b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- 172.21.15.116:0/3488162153 >> 172.21.15.116:0/3488162153 conn(0x561bfe4f4a00 msgr2=0x561bfe4f4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- 172.21.15.116:0/3488162153 shutdown_connections 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- 172.21.15.116:0/3488162153 wait complete. 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 -- start start 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.687+0000 7f177a1c1740 1 Processor -- start 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe5007c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe501240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561bfe506560 con 0x561bfe4fabf0 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561bfe419310 con 0x561bfe4fa2d0 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bfe5066e0 con 0x561bfe5037d0 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f177a1c1740 10 client.0 _ll_register_callbacks cb 0x561bfe40d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1775294640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe5007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55464/0 (socket says 172.21.15.116:55464) 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 -- 172.21.15.116:0/2473913964 learned_addr learned my addr 172.21.15.116:0/2473913964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1775a95640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe501240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 -- 172.21.15.116:0/2473913964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 msgr2=0x561bfe501240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe501240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 -- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 msgr2=0x561bfe5007c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe5007c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 -- 172.21.15.116:0/2473913964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bfe1713f0 con 0x561bfe4fabf0 2024-03-19T02:37:27.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1775294640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe5007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1774a93640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1760009830 tx=0x7f1760010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5904b99d26ed93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1758ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17600825e0 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.688+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f17600a3b00 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f176c02c9a0 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1760082780 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 10 client.18426 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561bfe23b7d0 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f176c02c9a0 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17600bb040 con 0x561bfe4fabf0 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 10 client.18426 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 10 client.18426 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 20 client.18426 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe515da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 4 client.18426 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bfe4bb3e0 con 0x561bfe513930 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.689+0000 7f1775294640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe515da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.690+0000 7f1775294640 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe515da0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b6fec0874235439 server_cookie=c02447841b188dbc in_seq=0 out_seq=0 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.690+0000 7f1758ff9640 10 client.18426 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f1758ff9640 1 -- 172.21.15.116:0/2473913964 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561bfe4bb3e0 con 0x561bfe513930 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f1758ff9640 10 client.18426 handle_command_reply: tid=0 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 client.18426 shutdown 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 msgr2=0x561bfe500d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f1760009830 tx=0x7f1760010f20 comp rx=0 tx=0).stop 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 shutdown_connections 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe515da0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe501240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe500d00 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f1758ff9640 0 client.18426 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2473913964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe5007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 >> 172.21.15.116:0/2473913964 conn(0x561bfe4f4a00 msgr2=0x561bfe4f6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 shutdown_connections 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 1 -- 172.21.15.116:0/2473913964 wait complete. 2024-03-19T02:37:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.691+0000 7f177a1c1740 20 client.18426 trim_cache size 0 max 16384 2024-03-19T02:37:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 Processor -- start 2024-03-19T02:37:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 -- start start 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe502ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe4f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b0d50 con 0x561bfe5037d0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b2890 con 0x561bfe4fa2d0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f177a1c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b1420 con 0x561bfe4fabf0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1775a95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe4f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1774a93640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1775294640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe502ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1774a93640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48196/0 (socket says 172.21.15.116:48196) 2024-03-19T02:37:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1775a95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe4f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55470/0 (socket says 172.21.15.116:55470) 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.705+0000 7f1774a93640 1 -- 172.21.15.116:0/1110103021 learned_addr learned my addr 172.21.15.116:0/1110103021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 -- 172.21.15.116:0/1110103021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 msgr2=0x561bfe502ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe502ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 -- 172.21.15.116:0/1110103021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 msgr2=0x561bfe4f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe4f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 -- 172.21.15.116:0/1110103021 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bfe4ff770 con 0x561bfe4fabf0 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f1774a93640 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f176009b5a0 tx=0x7f1760000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54cea97ac4784c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f175affd640 1 -- 172.21.15.116:0/1110103021 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760082140 con 0x561bfe4fabf0 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f175affd640 1 -- 172.21.15.116:0/1110103021 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1760097f10 con 0x561bfe4fabf0 2024-03-19T02:37:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.706+0000 7f175affd640 1 -- 172.21.15.116:0/1110103021 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17600a3d80 con 0x561bfe4fabf0 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- 172.21.15.116:0/1110103021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 msgr2=0x561bfe50f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f176009b5a0 tx=0x7f1760000bc0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- 172.21.15.116:0/1110103021 shutdown_connections 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe5037d0 0x561bfe4f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe4fabf0 0x561bfe50f300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 --2- 172.21.15.116:0/1110103021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe502ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- 172.21.15.116:0/1110103021 >> 172.21.15.116:0/1110103021 conn(0x561bfe4f4a00 msgr2=0x561bfe4f4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- 172.21.15.116:0/1110103021 shutdown_connections 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- 172.21.15.116:0/1110103021 wait complete. 2024-03-19T02:37:27.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 -- start start 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.707+0000 7f177a1c1740 1 Processor -- start 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe4fde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b21c0 con 0x561bfe4fabf0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17701b1af0 con 0x561bfe4fa2d0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17701aed50 con 0x561bfe5037d0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f177a1c1740 10 client.0 _ll_register_callbacks cb 0x7f176009b700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1775294640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55486/0 (socket says 172.21.15.116:55486) 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1775a95640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe4fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 -- 172.21.15.116:0/2832000095 learned_addr learned my addr 172.21.15.116:0/2832000095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 -- 172.21.15.116:0/2832000095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 msgr2=0x561bfe4fde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe4fde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 -- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 msgr2=0x561bfe4fd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 -- 172.21.15.116:0/2832000095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bfe578cd0 con 0x561bfe4fabf0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1774a93640 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1760098270 tx=0x7f17600a4110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5489b5a8e7efecd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.708+0000 7f1758ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760010de0 con 0x561bfe4fabf0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f1760010430 con 0x561bfe4fabf0 2024-03-19T02:37:27.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1760097f10 con 0x561bfe4fabf0 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1760010630 con 0x561bfe4fabf0 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 10 client.18432 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561bfe579fb0 con 0x561bfe4fabf0 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1760097f10 con 0x561bfe4fabf0 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f17600bb050 con 0x561bfe4fabf0 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 10 client.18432 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 10 client.18432 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 20 client.18432 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe57c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 4 client.18432 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561bfe46a9f0 con 0x561bfe513930 2024-03-19T02:37:27.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.709+0000 7f1775294640 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe57c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f1758ff9640 10 client.18432 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f1775294640 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe57c3c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bfa52792e9007b43 server_cookie=52ee9c1b12dc4c78 in_seq=0 out_seq=0 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f1758ff9640 1 -- 172.21.15.116:0/2832000095 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561bfe46a9f0 con 0x561bfe513930 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f1758ff9640 10 client.18432 handle_command_reply: tid=0 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 client.18432 shutdown 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 msgr2=0x561bfe4fd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f1760098270 tx=0x7f17600a4110 comp rx=0 tx=0).stop 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 shutdown_connections 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561bfe513930 0x561bfe57c3c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bfe5037d0 0x561bfe4fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f1758ff9640 0 client.18432 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:37:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bfe4fabf0 0x561bfe4fd8c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 --2- 172.21.15.116:0/2832000095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561bfe4fa2d0 0x561bfe4fd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 >> 172.21.15.116:0/2832000095 conn(0x561bfe4f4a00 msgr2=0x561bfe501010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 shutdown_connections 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 1 -- 172.21.15.116:0/2832000095 wait complete. 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.710+0000 7f177a1c1740 20 client.18432 trim_cache size 0 max 16384 2024-03-19T02:37:27.718 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:37:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 msgr2=0x7f175c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 0x7f175c0a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1768005fd0 tx=0x7f1768009f10 comp rx=0 tx=0).stop 2024-03-19T02:37:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 msgr2=0x7f1770197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f176000ea20 tx=0x7f176000eee0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 shutdown_connections 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f175c0a2970 0x7f175c0a4e30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1770105690 0x7f1770197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1770101330 0x7f1770197210 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 --2- 172.21.15.116:0/1854260502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1770100970 0x7f1770196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 >> 172.21.15.116:0/1854260502 conn(0x7f17700774b0 msgr2=0x7f17700fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 shutdown_connections 2024-03-19T02:37:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.711+0000 7f17774f8640 1 -- 172.21.15.116:0/1854260502 wait complete. 2024-03-19T02:37:27.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:37:27.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 312543 3.14 MB/sec execute 410 sec latency 17.873 ms 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 -- 172.21.15.116:0/1196454027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18104380 msgr2=0x7f5e18110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/1196454027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18104380 0x7f5e18110890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009c10 tx=0x7f5e1002bef0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 -- 172.21.15.116:0/1196454027 shutdown_connections 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/1196454027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18104380 0x7f5e18110890 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/1196454027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e18103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/1196454027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e181099d0 0x7f5e18109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 -- 172.21.15.116:0/1196454027 >> 172.21.15.116:0/1196454027 conn(0x7f5e180ff4d0 msgr2=0x7f5e181018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 -- 172.21.15.116:0/1196454027 shutdown_connections 2024-03-19T02:37:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.835+0000 7f5e1dd45640 1 -- 172.21.15.116:0/1196454027 wait complete. 2024-03-19T02:37:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 Processor -- start 2024-03-19T02:37:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 -- start start 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18104380 0x7f5e1819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 0x7f5e1819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18113c90 con 0x7f5e181039c0 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18113b10 con 0x7f5e18104380 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e1dd45640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18113e10 con 0x7f5e181099d0 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 0x7f5e1819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55506/0 (socket says 172.21.15.116:55506) 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18104380 0x7f5e1819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 -- 172.21.15.116:0/110959063 learned_addr learned my addr 172.21.15.116:0/110959063 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 -- 172.21.15.116:0/110959063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 msgr2=0x7f5e1819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 0x7f5e1819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 -- 172.21.15.116:0/110959063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18104380 msgr2=0x7f5e1819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.836+0000 7f5e177fe640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18104380 0x7f5e1819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e177fe640 1 -- 172.21.15.116:0/110959063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1819f140 con 0x7f5e181039c0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e17fff640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 0x7f5e1819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e177fe640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800e9f0 tx=0x7f5e0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e14ff9640 1 -- 172.21.15.116:0/110959063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800ccc0 con 0x7f5e181039c0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1819f430 con 0x7f5e181039c0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e14ff9640 1 -- 172.21.15.116:0/110959063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080108e0 con 0x7f5e181039c0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e14ff9640 1 -- 172.21.15.116:0/110959063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e080187c0 con 0x7f5e181039c0 2024-03-19T02:37:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.837+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e181a6cc0 con 0x7f5e181039c0 2024-03-19T02:37:27.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.839+0000 7f5e14ff9640 1 -- 172.21.15.116:0/110959063 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5e08010a90 con 0x7f5e181039c0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 Processor -- start 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 -- start start 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e14ff9640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 0x7f5e000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af34bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e3ae8c4b0 con 0x563e3af29bf0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563e3ae34d70 con 0x563e3af327d0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563e3ae88550 con 0x563e3af292d0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e17fff640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 0x7f5e000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e14ff9640 1 -- 172.21.15.116:0/110959063 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5e080cf4e0 con 0x7f5e181039c0 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55514/0 (socket says 172.21.15.116:55514) 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 -- 172.21.15.116:0/1563053151 learned_addr learned my addr 172.21.15.116:0/1563053151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 -- 172.21.15.116:0/1563053151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 msgr2=0x563e3af296b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af296b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e17fff640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af34bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 -- 172.21.15.116:0/1563053151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 msgr2=0x563e3af34bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af34bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e16ffd640 1 -- 172.21.15.116:0/1563053151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e3ae57210 con 0x563e3af29bf0 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e177fe640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.840+0000 7f5e17fff640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af34bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.841+0000 7f5e17fff640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 0x7f5e000a4e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5e10004af0 tx=0x7f5e10033790 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.841+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009ca0 tx=0x7f5e0c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c192343dc515250 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.841+0000 7f5dfdffb640 1 -- 172.21.15.116:0/1563053151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c004230 con 0x563e3af29bf0 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.841+0000 7f5dfdffb640 1 -- 172.21.15.116:0/1563053151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e0c0043d0 con 0x563e3af29bf0 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.841+0000 7f5dfdffb640 1 -- 172.21.15.116:0/1563053151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0196d0 con 0x563e3af29bf0 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- 172.21.15.116:0/1563053151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 msgr2=0x563e3af32290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009ca0 tx=0x7f5e0c011810 comp rx=0 tx=0).stop 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- 172.21.15.116:0/1563053151 shutdown_connections 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af34bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af32290 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/1563053151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- 172.21.15.116:0/1563053151 >> 172.21.15.116:0/1563053151 conn(0x563e3af23a00 msgr2=0x563e3af23de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- 172.21.15.116:0/1563053151 shutdown_connections 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- 172.21.15.116:0/1563053151 wait complete. 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- start start 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 Processor -- start 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af2f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af30210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.842+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e3af355c0 con 0x563e3af29bf0 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563e3ae48310 con 0x563e3af327d0 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563e3af35740 con 0x563e3af292d0 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e20a0d740 10 client.0 _ll_register_callbacks cb 0x563e3ae3c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55524/0 (socket says 172.21.15.116:55524) 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af2f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e16ffd640 1 -- 172.21.15.116:0/982813894 learned_addr learned my addr 172.21.15.116:0/982813894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e17fff640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af30210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 -- 172.21.15.116:0/982813894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 msgr2=0x563e3af30210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af30210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 -- 172.21.15.116:0/982813894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 msgr2=0x563e3af2fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 -- 172.21.15.116:0/982813894 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e3aba03f0 con 0x563e3af292d0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e177fe640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af2f790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e080cc8f0 tx=0x7f5e080175f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d48f9bf0ee6f89d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5df6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0809bf00 con 0x563e3af292d0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5e08099b00 con 0x563e3af292d0 2024-03-19T02:37:27.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e0c0043d0 con 0x563e3af292d0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.843+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e0809b800 con 0x563e3af292d0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 10 client.28183 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563e3ac6a7d0 con 0x563e3af292d0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e0c0043d0 con 0x563e3af292d0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5e08097e70 con 0x563e3af292d0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 10 client.28183 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 10 client.28183 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 20 client.28183 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3af42b30 0x563e3af44ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 4 client.28183 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e3aeea3e0 con 0x563e3af42b30 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.844+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3af42b30 0x563e3af44ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.845+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3af42b30 0x563e3af44ff0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f20bf09f9398f705 server_cookie=1d615238cb354f44 in_seq=0 out_seq=0 2024-03-19T02:37:27.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.845+0000 7f5df6ffd640 10 client.28183 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.845+0000 7f5df6ffd640 1 -- 172.21.15.116:0/982813894 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e3aeea3e0 con 0x563e3af42b30 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.845+0000 7f5df6ffd640 10 client.28183 handle_command_reply: tid=0 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 client.28183 shutdown 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 msgr2=0x563e3af2f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af2f790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e080cc8f0 tx=0x7f5e080175f0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 shutdown_connections 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3af42b30 0x563e3af44ff0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af30210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5df6ffd640 0 client.28183 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af2fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/982813894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af2f790 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 >> 172.21.15.116:0/982813894 conn(0x563e3af23a00 msgr2=0x563e3af25750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 shutdown_connections 2024-03-19T02:37:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 1 -- 172.21.15.116:0/982813894 wait complete. 2024-03-19T02:37:27.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.846+0000 7f5e20a0d740 20 client.28183 trim_cache size 0 max 16384 2024-03-19T02:37:27.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 Processor -- start 2024-03-19T02:37:27.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 -- start start 2024-03-19T02:37:27.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af30030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af4a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b0e30 con 0x563e3af29bf0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b2970 con 0x563e3af327d0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b1500 con 0x563e3af292d0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af30030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af4a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48254/0 (socket says 172.21.15.116:48254) 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af30030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55532/0 (socket says 172.21.15.116:55532) 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 -- 172.21.15.116:0/3697895312 learned_addr learned my addr 172.21.15.116:0/3697895312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 -- 172.21.15.116:0/3697895312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 msgr2=0x563e3af4a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af4a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 -- 172.21.15.116:0/3697895312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 msgr2=0x563e3af30030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af30030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.861+0000 7f5e177fe640 1 -- 172.21.15.116:0/3697895312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e3af2b360 con 0x563e3af292d0 2024-03-19T02:37:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.862+0000 7f5e177fe640 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5e080a3a70 tx=0x7f5e08097bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3acffbcf6e26920 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.862+0000 7f5dfd7fa640 1 -- 172.21.15.116:0/3697895312 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0809bf00 con 0x563e3af292d0 2024-03-19T02:37:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.862+0000 7f5dfd7fa640 1 -- 172.21.15.116:0/3697895312 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080a3fa0 con 0x563e3af292d0 2024-03-19T02:37:27.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.862+0000 7f5dfd7fa640 1 -- 172.21.15.116:0/3697895312 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08099bd0 con 0x563e3af292d0 2024-03-19T02:37:27.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- 172.21.15.116:0/3697895312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 msgr2=0x563e3af3e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5e080a3a70 tx=0x7f5e08097bd0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- 172.21.15.116:0/3697895312 shutdown_connections 2024-03-19T02:37:27.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af327d0 0x563e3af4a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af30030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/3697895312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af292d0 0x563e3af3e390 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- 172.21.15.116:0/3697895312 >> 172.21.15.116:0/3697895312 conn(0x563e3af23a00 msgr2=0x563e3af256a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- 172.21.15.116:0/3697895312 shutdown_connections 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- 172.21.15.116:0/3697895312 wait complete. 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 -- start start 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 Processor -- start 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.863+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 0x563e3af3aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af327d0 0x563e3af36420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b22a0 con 0x563e3af29bf0 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b1bd0 con 0x563e3af292d0 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181aee30 con 0x563e3af327d0 2024-03-19T02:37:27.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 10 client.0 _ll_register_callbacks cb 0x7f5e0800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 0x563e3af3aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55536/0 (socket says 172.21.15.116:55536) 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 -- 172.21.15.116:0/286970374 learned_addr learned my addr 172.21.15.116:0/286970374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 0x563e3af3aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47042/0 (socket says 172.21.15.116:47042) 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e17fff640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af327d0 0x563e3af36420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 -- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af327d0 msgr2=0x563e3af36420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af327d0 0x563e3af36420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 -- 172.21.15.116:0/286970374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 msgr2=0x563e3af3aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 0x563e3af3aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 -- 172.21.15.116:0/286970374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e3af36a60 con 0x563e3af29bf0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e16ffd640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009690 tx=0x7f5e0c009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95927e2d331cdfe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5df6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:27.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c01f040 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5e0c005270 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e080a3fa0 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e0c03b050 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 10 client.18450 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.864+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563e3afa8cb0 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080a3fa0 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5e0c018070 con 0x563e3af29bf0 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 10 client.18450 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 10 client.18450 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 20 client.18450 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3afa9290 0x563e3afab680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 4 client.18450 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e3ae999f0 con 0x563e3afa9290 2024-03-19T02:37:27.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e177fe640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3afa9290 0x563e3afab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5e177fe640 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3afa9290 0x563e3afab680 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32cf9ff06edd0470 server_cookie=780afb3551bac158 in_seq=0 out_seq=0 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.865+0000 7f5df6ffd640 10 client.18450 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5df6ffd640 1 -- 172.21.15.116:0/286970374 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e3ae999f0 con 0x563e3afa9290 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5df6ffd640 10 client.18450 handle_command_reply: tid=0 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 client.18450 shutdown 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 msgr2=0x563e3af35ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009690 tx=0x7f5e0c009ee0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 shutdown_connections 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x563e3afa9290 0x563e3afab680 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e3af327d0 0x563e3af36420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5df6ffd640 0 client.18450 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e3af29bf0 0x563e3af35ee0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 --2- 172.21.15.116:0/286970374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e3af292d0 0x563e3af3aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 >> 172.21.15.116:0/286970374 conn(0x563e3af23a00 msgr2=0x563e3af24d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 shutdown_connections 2024-03-19T02:37:27.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 1 -- 172.21.15.116:0/286970374 wait complete. 2024-03-19T02:37:27.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.866+0000 7f5e20a0d740 20 client.18450 trim_cache size 0 max 16384 2024-03-19T02:37:27.873 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 msgr2=0x7f5e000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 0x7f5e000a4e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5e10004af0 tx=0x7f5e10033790 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 msgr2=0x7f5e1819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800e9f0 tx=0x7f5e0800edf0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 shutdown_connections 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e000a2970 0x7f5e000a4e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e181099d0 0x7f5e1819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18104380 0x7f5e1819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 --2- 172.21.15.116:0/110959063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181039c0 0x7f5e1819a150 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 >> 172.21.15.116:0/110959063 conn(0x7f5e180ff4d0 msgr2=0x7f5e180ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 shutdown_connections 2024-03-19T02:37:27.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.867+0000 7f5e1dd45640 1 -- 172.21.15.116:0/110959063 wait complete. 2024-03-19T02:37:27.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- 172.21.15.116:0/41691475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa690101060 msgr2=0x7fa69010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 --2- 172.21.15.116:0/41691475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa690101060 0x7fa69010d570 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa688009cd0 tx=0x7fa68802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- 172.21.15.116:0/41691475 shutdown_connections 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 --2- 172.21.15.116:0/41691475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa690101060 0x7fa69010d570 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 --2- 172.21.15.116:0/41691475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6901006a0 0x7fa690100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 --2- 172.21.15.116:0/41691475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 0x7fa690106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- 172.21.15.116:0/41691475 >> 172.21.15.116:0/41691475 conn(0x7fa6900778d0 msgr2=0x7fa690077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:27.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- 172.21.15.116:0/41691475 shutdown_connections 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- 172.21.15.116:0/41691475 wait complete. 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 Processor -- start 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.989+0000 7fa694a26640 1 -- start start 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa690101060 0x7fa690198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 0x7fa69019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa690110930 con 0x7fa6901006a0 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901107b0 con 0x7fa690101060 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa694a26640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa690110ab0 con 0x7fa6901066b0 2024-03-19T02:37:27.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55558/0 (socket says 172.21.15.116:55558) 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 -- 172.21.15.116:0/1764558673 learned_addr learned my addr 172.21.15.116:0/1764558673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa687fff640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa690101060 0x7fa690198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 0x7fa69019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 -- 172.21.15.116:0/1764558673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 msgr2=0x7fa69019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 0x7fa69019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 -- 172.21.15.116:0/1764558673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa690101060 msgr2=0x7fa690198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa690101060 0x7fa690198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 -- 172.21.15.116:0/1764558673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69019cf20 con 0x7fa6901006a0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa68e59c640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fa68000ea60 tx=0x7fa68000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.990+0000 7fa6877fe640 1 -- 172.21.15.116:0/1764558673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68000cde0 con 0x7fa6901006a0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.991+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69019d210 con 0x7fa6901006a0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.991+0000 7fa6877fe640 1 -- 172.21.15.116:0/1764558673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6800035f0 con 0x7fa6901006a0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.991+0000 7fa6877fe640 1 -- 172.21.15.116:0/1764558673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680010690 con 0x7fa6901006a0 2024-03-19T02:37:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.991+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6901a4aa0 con 0x7fa6901006a0 2024-03-19T02:37:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.993+0000 7fa6877fe640 1 -- 172.21.15.116:0/1764558673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa6800037a0 con 0x7fa6901006a0 2024-03-19T02:37:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.993+0000 7fa69760b740 1 Processor -- start 2024-03-19T02:37:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.993+0000 7fa69760b740 1 -- start start 2024-03-19T02:37:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117350290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 0x55e117352bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1172aa4b0 con 0x55e1173472d0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e117252d70 con 0x55e117347bf0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa69760b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1172a6550 con 0x55e1173507d0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa6877fe640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 0x7fa67c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55570/0 (socket says 172.21.15.116:55570) 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68ed9d640 1 -- 172.21.15.116:0/2714428171 learned_addr learned my addr 172.21.15.116:0/2714428171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117350290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa687fff640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 0x55e117352bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 0x7fa67c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa6877fe640 1 -- 172.21.15.116:0/1764558673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fa6800cf430 con 0x7fa6901006a0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 -- 172.21.15.116:0/2714428171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 msgr2=0x55e117352bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 0x55e117352bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 -- 172.21.15.116:0/2714428171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 msgr2=0x55e1173476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 -- 172.21.15.116:0/2714428171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e117275210 con 0x55e117347bf0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa687fff640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 0x55e117352bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.994+0000 7fa68e59c640 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117350290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6800cc7a0 tx=0x7fa68009b530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0e7116e00bc3ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 0x7fa67c0a4e30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa688009a60 tx=0x7fa68802c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa684ff9640 1 -- 172.21.15.116:0/2714428171 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680003b30 con 0x55e117347bf0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa684ff9640 1 -- 172.21.15.116:0/2714428171 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa68009bdb0 con 0x55e117347bf0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa684ff9640 1 -- 172.21.15.116:0/2714428171 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6800cca20 con 0x55e117347bf0 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 -- 172.21.15.116:0/2714428171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 msgr2=0x55e117350290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117350290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6800cc7a0 tx=0x7fa68009b530 comp rx=0 tx=0).stop 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 -- 172.21.15.116:0/2714428171 shutdown_connections 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173507d0 0x55e117352bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117350290 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 --2- 172.21.15.116:0/2714428171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e1173476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 -- 172.21.15.116:0/2714428171 >> 172.21.15.116:0/2714428171 conn(0x55e117341a00 msgr2=0x55e117341de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 -- 172.21.15.116:0/2714428171 shutdown_connections 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.995+0000 7fa69760b740 1 -- 172.21.15.116:0/2714428171 wait complete. 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 -- start start 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 Processor -- start 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11734e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e117353560 con 0x55e1173472d0 2024-03-19T02:37:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e117266310 con 0x55e1173507d0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1173536e0 con 0x55e117347bf0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa69760b740 10 client.0 _ll_register_callbacks cb 0x55e11725a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55572/0 (socket says 172.21.15.116:55572) 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 -- 172.21.15.116:0/3128698524 learned_addr learned my addr 172.21.15.116:0/3128698524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa687fff640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11734e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 -- 172.21.15.116:0/3128698524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 msgr2=0x55e11734ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 -- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 msgr2=0x55e11734e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11734e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa68e59c640 1 -- 172.21.15.116:0/3128698524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e116fbe3f0 con 0x55e1173472d0 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.996+0000 7fa687fff640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fa680098080 tx=0x7fa680097950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c72b5c98c5a2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680003b30 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fa6800999b0 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa68009bdb0 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa68009f800 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 10 client.18462 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1170887d0 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa68009bdb0 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa680097400 con 0x55e1173472d0 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 10 client.18462 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 10 client.18462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 20 client.18462 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173609e0 0x55e117362dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 4 client.18462 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.997+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1173083e0 con 0x55e1173609e0 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.998+0000 7fa687fff640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173609e0 0x55e117362dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.998+0000 7fa687fff640 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173609e0 0x55e117362dd0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=609b79dbfc4e3749 server_cookie=22f0be5c4b00c1bb in_seq=0 out_seq=0 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.998+0000 7fa675ffb640 10 client.18462 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa675ffb640 1 -- 172.21.15.116:0/3128698524 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e1173083e0 con 0x55e1173609e0 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa675ffb640 10 client.18462 handle_command_reply: tid=0 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 client.18462 shutdown 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 msgr2=0x55e11734d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fa680098080 tx=0x7fa680097950 comp rx=0 tx=0).stop 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 shutdown_connections 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173609e0 0x55e117362dd0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11734e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa675ffb640 0 client.18462 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 --2- 172.21.15.116:0/3128698524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734d870 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 >> 172.21.15.116:0/3128698524 conn(0x55e117341a00 msgr2=0x55e117343750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 shutdown_connections 2024-03-19T02:37:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 1 -- 172.21.15.116:0/3128698524 wait complete. 2024-03-19T02:37:28.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:27.999+0000 7fa69760b740 20 client.18462 trim_cache size 0 max 16384 2024-03-19T02:37:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 Processor -- start 2024-03-19T02:37:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 -- start start 2024-03-19T02:37:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11735c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901aca90 con 0x55e1173472d0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901ae5d0 con 0x55e1173507d0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa69760b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901ad160 con 0x55e117347bf0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55578/0 (socket says 172.21.15.116:55578) 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa68e59c640 1 -- 172.21.15.116:0/3712375831 learned_addr learned my addr 172.21.15.116:0/3712375831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa687fff640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.013+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11735c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 -- 172.21.15.116:0/3712375831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 msgr2=0x55e11734e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 -- 172.21.15.116:0/3712375831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 msgr2=0x55e11735c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11735c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 -- 172.21.15.116:0/3712375831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e117317880 con 0x55e1173472d0 2024-03-19T02:37:28.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa687fff640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c760 tx=0x7fa680004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef01cd8582b48063 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa677fff640 1 -- 172.21.15.116:0/3712375831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680003b30 con 0x55e1173472d0 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa677fff640 1 -- 172.21.15.116:0/3712375831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa68009bdb0 con 0x55e1173472d0 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.014+0000 7fa677fff640 1 -- 172.21.15.116:0/3712375831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6800cca20 con 0x55e1173472d0 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- 172.21.15.116:0/3712375831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 msgr2=0x55e11734ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c760 tx=0x7fa680004000 comp rx=0 tx=0).stop 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- 172.21.15.116:0/3712375831 shutdown_connections 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1173507d0 0x55e11735c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e117347bf0 0x55e11734e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- 172.21.15.116:0/3712375831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173472d0 0x55e11734ffa0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- 172.21.15.116:0/3712375831 >> 172.21.15.116:0/3712375831 conn(0x55e117341a00 msgr2=0x55e11735b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- 172.21.15.116:0/3712375831 shutdown_connections 2024-03-19T02:37:28.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- 172.21.15.116:0/3712375831 wait complete. 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- start start 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 Processor -- start 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 0x55e1173638c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117363e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901adf00 con 0x55e1173507d0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901ad830 con 0x55e117347bf0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6901aaa90 con 0x55e1173472d0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa69760b740 10 client.0 _ll_register_callbacks cb 0x7fa678005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.015+0000 7fa68ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 0x55e1173638c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55580/0 (socket says 172.21.15.116:55580) 2024-03-19T02:37:28.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa687fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117363e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 -- 172.21.15.116:0/3489646443 learned_addr learned my addr 172.21.15.116:0/3489646443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 -- 172.21.15.116:0/3489646443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 msgr2=0x55e1173638c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 0x55e1173638c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 -- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 msgr2=0x55e117363e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117363e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 -- 172.21.15.116:0/3489646443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1173c5b50 con 0x55e1173507d0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 0x55e1173638c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa687fff640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117363e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa68ed9d640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa688009cd0 tx=0x7fa6880095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fd804e75f10063 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa675ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688039040 con 0x55e1173507d0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fa688004510 con 0x55e1173507d0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa68009bdb0 con 0x55e1173507d0 2024-03-19T02:37:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa68805f050 con 0x55e1173507d0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa69760b740 10 client.18468 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1173c6e10 con 0x55e1173507d0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.016+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa68009bdb0 con 0x55e1173507d0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa68803f070 con 0x55e1173507d0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 10 client.18468 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 10 client.18468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 20 client.18468 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173c75e0 0x55e1173c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 4 client.18468 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1172b79f0 con 0x55e1173c75e0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173c75e0 0x55e1173c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa68e59c640 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173c75e0 0x55e1173c9a00 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc08242184db6ad7 server_cookie=acd98af4c90006aa in_seq=0 out_seq=0 2024-03-19T02:37:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa675ffb640 10 client.18468 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:28.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa675ffb640 1 -- 172.21.15.116:0/3489646443 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1172b79f0 con 0x55e1173c75e0 2024-03-19T02:37:28.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa675ffb640 10 client.18468 handle_command_reply: tid=0 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 client.18468 shutdown 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 msgr2=0x55e117364340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa688009cd0 tx=0x7fa6880095c0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 shutdown_connections 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1173c75e0 0x55e1173c9a00 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa675ffb640 0 client.18468 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.017+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1173507d0 0x55e117364340 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e117347bf0 0x55e117363e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 1 --2- 172.21.15.116:0/3489646443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1173472d0 0x55e1173638c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 >> 172.21.15.116:0/3489646443 conn(0x55e117341a00 msgr2=0x55e11735c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 shutdown_connections 2024-03-19T02:37:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 1 -- 172.21.15.116:0/3489646443 wait complete. 2024-03-19T02:37:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa69760b740 20 client.18468 trim_cache size 0 max 16384 2024-03-19T02:37:28.031 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:37:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 msgr2=0x7fa67c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 0x7fa67c0a4e30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa688009a60 tx=0x7fa68802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 msgr2=0x7fa690197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fa68000ea60 tx=0x7fa68000ef20 comp rx=0 tx=0).stop 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 shutdown_connections 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa67c0a2970 0x7fa67c0a4e30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6901066b0 0x7fa69019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa690101060 0x7fa690198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 --2- 172.21.15.116:0/1764558673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6901006a0 0x7fa690197f30 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 >> 172.21.15.116:0/1764558673 conn(0x7fa6900778d0 msgr2=0x7fa690068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 shutdown_connections 2024-03-19T02:37:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:28.018+0000 7fa694a26640 1 -- 172.21.15.116:0/1764558673 wait complete. 2024-03-19T02:37:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:27 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4217848850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:37:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:27 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4217848850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:37:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:27 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4217848850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:37:28.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 312980 3.14 MB/sec execute 411 sec latency 41.278 ms 2024-03-19T02:37:29.034 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:37:29.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:37:29.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:37:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 -- 172.21.15.116:0/86786307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c069320 msgr2=0x7f698c10dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 --2- 172.21.15.116:0/86786307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c069320 0x7f698c10dc80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6980009900 tx=0x7f698002bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 -- 172.21.15.116:0/86786307 shutdown_connections 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 --2- 172.21.15.116:0/86786307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c10e1c0 0x7f698c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 --2- 172.21.15.116:0/86786307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c069320 0x7f698c10dc80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 --2- 172.21.15.116:0/86786307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c068a00 0x7f698c068de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 -- 172.21.15.116:0/86786307 >> 172.21.15.116:0/86786307 conn(0x7f698c077820 msgr2=0x7f698c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 -- 172.21.15.116:0/86786307 shutdown_connections 2024-03-19T02:37:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.160+0000 7f6991177640 1 -- 172.21.15.116:0/86786307 wait complete. 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.161+0000 7f6991177640 1 Processor -- start 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.161+0000 7f6991177640 1 -- start start 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 0x7f698c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c10e1c0 0x7f698c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f698c113c80 con 0x7f698c069320 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f698c113b00 con 0x7f698c10e1c0 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f698c113e00 con 0x7f698c068a00 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 0x7f698c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c10e1c0 0x7f698c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55606/0 (socket says 172.21.15.116:55606) 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 0x7f698c19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48328/0 (socket says 172.21.15.116:48328) 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 -- 172.21.15.116:0/2043634397 learned_addr learned my addr 172.21.15.116:0/2043634397 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 -- 172.21.15.116:0/2043634397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 msgr2=0x7f698c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 0x7f698c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 -- 172.21.15.116:0/2043634397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c10e1c0 msgr2=0x7f698c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c10e1c0 0x7f698c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 -- 172.21.15.116:0/2043634397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c19f210 con 0x7f698c069320 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f698a59c640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6980000c00 tx=0x7f698002c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6973fff640 1 -- 172.21.15.116:0/2043634397 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980043020 con 0x7f698c069320 2024-03-19T02:37:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6973fff640 1 -- 172.21.15.116:0/2043634397 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698002cb30 con 0x7f698c069320 2024-03-19T02:37:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c19f4a0 con 0x7f698c069320 2024-03-19T02:37:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6973fff640 1 -- 172.21.15.116:0/2043634397 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980003a50 con 0x7f698c069320 2024-03-19T02:37:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.162+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c1a6d90 con 0x7f698c069320 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f6973fff640 1 -- 172.21.15.116:0/2043634397 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6980046020 con 0x7f698c069320 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 Processor -- start 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f6973fff640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 0x7f69740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 -- start start 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f698b59e640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 0x7f69740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f6973fff640 1 -- 172.21.15.116:0/2043634397 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f69800f11b0 con 0x7f698c069320 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e2bd41160 con 0x563e2bde6880 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563e2bce9e10 con 0x563e2bddd380 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.165+0000 7f69939fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563e2bd3d200 con 0x563e2bdddca0 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48336/0 (socket says 172.21.15.116:48336) 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55612/0 (socket says 172.21.15.116:55612) 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698b59e640 1 -- 172.21.15.116:0/3701338443 learned_addr learned my addr 172.21.15.116:0/3701338443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47112/0 (socket says 172.21.15.116:47112) 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698b59e640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 0x7f69740a4e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6984013770 tx=0x7f6984013b70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 -- 172.21.15.116:0/3701338443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 msgr2=0x563e2bddd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 -- 172.21.15.116:0/3701338443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 msgr2=0x563e2bde8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 -- 172.21.15.116:0/3701338443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2bb1d7e0 con 0x563e2bdddca0 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698ad9d640 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698b59e640 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f698a59c640 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f69800ee630 tx=0x7f69800b99a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eba2fa2b03241545 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f69717fa640 1 -- 172.21.15.116:0/3701338443 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800bd5d0 con 0x563e2bdddca0 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f69717fa640 1 -- 172.21.15.116:0/3701338443 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69800c1720 con 0x563e2bdddca0 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.166+0000 7f69717fa640 1 -- 172.21.15.116:0/3701338443 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980005420 con 0x563e2bdddca0 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- 172.21.15.116:0/3701338443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 msgr2=0x563e2bde6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f69800ee630 tx=0x7f69800b99a0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- 172.21.15.116:0/3701338443 shutdown_connections 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bde6880 0x563e2bde8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde6340 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 --2- 172.21.15.116:0/3701338443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bddd380 0x563e2bddd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- 172.21.15.116:0/3701338443 >> 172.21.15.116:0/3701338443 conn(0x563e2bdd7ab0 msgr2=0x563e2bdd7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- 172.21.15.116:0/3701338443 shutdown_connections 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- 172.21.15.116:0/3701338443 wait complete. 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 -- start start 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.167+0000 7f69939fd740 1 Processor -- start 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bde4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e2bde9430 con 0x563e2bddd380 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563e2bcfd1a0 con 0x563e2bde6880 2024-03-19T02:37:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563e2bde95b0 con 0x563e2bdddca0 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f69939fd740 10 client.0 _ll_register_callbacks cb 0x563e2ba54c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48352/0 (socket says 172.21.15.116:48352) 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bde4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698a59c640 1 -- 172.21.15.116:0/3621070767 learned_addr learned my addr 172.21.15.116:0/3621070767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55620/0 (socket says 172.21.15.116:55620) 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 -- 172.21.15.116:0/3621070767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 msgr2=0x563e2bde3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 -- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 msgr2=0x563e2bde4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bde4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 -- 172.21.15.116:0/3621070767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e2bb7df70 con 0x563e2bddd380 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698a59c640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f698ad9d640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f697c012710 tx=0x7f697c012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16896a44d51a5fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.168+0000 7f696eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c004050 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f697c0041f0 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69800c1720 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f697c025050 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 10 client.18480 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563e2ba15170 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69800c1720 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f697c003a40 con 0x563e2bddd380 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 10 client.18480 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 10 client.18480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 20 client.18480 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:29.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2bdf69b0 0x563e2bdf8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 4 client.18480 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e2bd4e6a0 con 0x563e2bdf69b0 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.169+0000 7f698a59c640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2bdf69b0 0x563e2bdf8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.170+0000 7f696eff5640 10 client.18480 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.170+0000 7f698a59c640 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2bdf69b0 0x563e2bdf8dc0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1c07fc96349df3 server_cookie=3804734c784a365f in_seq=0 out_seq=0 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f696eff5640 1 -- 172.21.15.116:0/3621070767 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e2bd4e6a0 con 0x563e2bdf69b0 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f696eff5640 10 client.18480 handle_command_reply: tid=0 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 client.18480 shutdown 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 msgr2=0x563e2bde38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f697c012710 tx=0x7f697c012bd0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 shutdown_connections 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2bdf69b0 0x563e2bdf8dc0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bde4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f696eff5640 0 client.18480 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bde3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 --2- 172.21.15.116:0/3621070767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde38d0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 >> 172.21.15.116:0/3621070767 conn(0x563e2bdd7ab0 msgr2=0x563e2bdd9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 shutdown_connections 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 1 -- 172.21.15.116:0/3621070767 wait complete. 2024-03-19T02:37:29.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.171+0000 7f69939fd740 20 client.18480 trim_cache size 0 max 16384 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.189+0000 7f69939fd740 1 Processor -- start 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.189+0000 7f69939fd740 1 -- start start 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bdf2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bdfe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b0e30 con 0x563e2bddd380 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b2970 con 0x563e2bde6880 2024-03-19T02:37:29.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f69939fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b1500 con 0x563e2bdddca0 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55634/0 (socket says 172.21.15.116:55634) 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 -- 172.21.15.116:0/2793476449 learned_addr learned my addr 172.21.15.116:0/2793476449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698b59e640 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bdfe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698a59c640 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bdf2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 -- 172.21.15.116:0/2793476449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 msgr2=0x563e2bdf2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bdf2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 -- 172.21.15.116:0/2793476449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 msgr2=0x563e2bdfe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bdfe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 -- 172.21.15.116:0/2793476449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2bdddad0 con 0x563e2bddd380 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f697c012420 tx=0x7f697c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6a797ff2f46045 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f6970ff9640 1 -- 172.21.15.116:0/2793476449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c010430 con 0x563e2bddd380 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f6970ff9640 1 -- 172.21.15.116:0/2793476449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f697c02e070 con 0x563e2bddd380 2024-03-19T02:37:29.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.190+0000 7f6970ff9640 1 -- 172.21.15.116:0/2793476449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c00fd00 con 0x563e2bddd380 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- 172.21.15.116:0/2793476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 msgr2=0x563e2bde6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f697c012420 tx=0x7f697c009620 comp rx=0 tx=0).stop 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- 172.21.15.116:0/2793476449 shutdown_connections 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2bdfe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bdddca0 0x563e2bdf2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 --2- 172.21.15.116:0/2793476449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bddd380 0x563e2bde6000 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- 172.21.15.116:0/2793476449 >> 172.21.15.116:0/2793476449 conn(0x563e2bdd7ab0 msgr2=0x563e2bde4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- 172.21.15.116:0/2793476449 shutdown_connections 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- 172.21.15.116:0/2793476449 wait complete. 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 -- start start 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.191+0000 7f69939fd740 1 Processor -- start 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bddd380 0x563e2bde0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2be5bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b22a0 con 0x563e2bdddca0 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b1bd0 con 0x563e2bde6880 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1aee30 con 0x563e2bddd380 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 10 client.0 _ll_register_callbacks cb 0x7f697c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55650/0 (socket says 172.21.15.116:55650) 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 -- 172.21.15.116:0/2675849941 learned_addr learned my addr 172.21.15.116:0/2675849941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bddd380 0x563e2bde0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698b59e640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2be5bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 -- 172.21.15.116:0/2675849941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bddd380 msgr2=0x563e2bde0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bddd380 0x563e2bde0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 -- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 msgr2=0x563e2be5bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2be5bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 -- 172.21.15.116:0/2675849941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e2be5c200 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f698a59c640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6980005d00 tx=0x7f6980005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f30179fa0296c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f696eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800bd110 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f6980005420 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697c02e070 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.192+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f69800c1720 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 10 client.18486 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563e2be5d570 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f697c02e070 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f69800c5850 con 0x563e2bdddca0 2024-03-19T02:37:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 10 client.18486 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 10 client.18486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 20 client.18486 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2be5dbb0 0x563e2be5ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 4 client.18486 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563e2bd489f0 con 0x563e2be5dbb0 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2be5dbb0 0x563e2be5ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f696eff5640 10 client.18486 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.193+0000 7f698ad9d640 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2be5dbb0 0x563e2be5ffd0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0e74d96df9429d5 server_cookie=1ff8072ca95e7c74 in_seq=0 out_seq=0 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f696eff5640 1 -- 172.21.15.116:0/2675849941 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563e2bd489f0 con 0x563e2be5dbb0 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f696eff5640 10 client.18486 handle_command_reply: tid=0 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 client.18486 shutdown 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 msgr2=0x563e2bde1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f6980005d00 tx=0x7f6980005fd0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 shutdown_connections 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563e2be5dbb0 0x563e2be5ffd0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563e2bde6880 0x563e2be5bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e2bdddca0 0x563e2bde1100 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.195+0000 7f69939fd740 1 --2- 172.21.15.116:0/2675849941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563e2bddd380 0x563e2bde0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 >> 172.21.15.116:0/2675849941 conn(0x563e2bdd7ab0 msgr2=0x563e2bdd62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f696eff5640 0 client.18486 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 shutdown_connections 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f69939fd740 1 -- 172.21.15.116:0/2675849941 wait complete. 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f69939fd740 20 client.18486 trim_cache size 0 max 16384 2024-03-19T02:37:29.204 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:37:29.205 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:37:29.205 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "33e489c3-c8e4-4db4-966d-f258a2d06c23", 2024-03-19T02:37:29.205 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:37:29.205 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 msgr2=0x7f69740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 0x7f69740a4e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6984013770 tx=0x7f6984013b70 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 msgr2=0x7f698c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6980000c00 tx=0x7f698002c630 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 shutdown_connections 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69740a2970 0x7f69740a4e30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f698c10e1c0 0x7f698c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f698c069320 0x7f698c19a760 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 --2- 172.21.15.116:0/2043634397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f698c068a00 0x7f698c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 >> 172.21.15.116:0/2043634397 conn(0x7f698c077820 msgr2=0x7f698c10f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.196+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 shutdown_connections 2024-03-19T02:37:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.197+0000 7f6991177640 1 -- 172.21.15.116:0/2043634397 wait complete. 2024-03-19T02:37:29.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:37:29.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:37:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:28 smithi154 ceph-mon[35693]: pgmap v301: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:28 smithi116 ceph-mon[32263]: pgmap v301: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- 172.21.15.116:0/343551202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 msgr2=0x7f915410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- 172.21.15.116:0/343551202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f915410d560 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f914c009cd0 tx=0x7f914c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- 172.21.15.116:0/343551202 shutdown_connections 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- 172.21.15.116:0/343551202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f915410d560 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- 172.21.15.116:0/343551202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- 172.21.15.116:0/343551202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 0x7f9154106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- 172.21.15.116:0/343551202 >> 172.21.15.116:0/343551202 conn(0x7f9154077820 msgr2=0x7f9154077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- 172.21.15.116:0/343551202 shutdown_connections 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- 172.21.15.116:0/343551202 wait complete. 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 Processor -- start 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 -- start start 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.324+0000 7f915b5b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 0x7f915419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9154110960 con 0x7f9154101050 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91541107e0 con 0x7f9154100690 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9154110ae0 con 0x7f91541066a0 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9159355640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55676/0 (socket says 172.21.15.116:55676) 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9159b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 0x7f915419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 -- 172.21.15.116:0/1294002103 learned_addr learned my addr 172.21.15.116:0/1294002103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 -- 172.21.15.116:0/1294002103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 msgr2=0x7f915419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 0x7f915419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 -- 172.21.15.116:0/1294002103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 msgr2=0x7f9154197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 -- 172.21.15.116:0/1294002103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915419cf60 con 0x7f9154101050 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9159355640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f9158b54640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f914400e9f0 tx=0x7f914400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f914a7fc640 1 -- 172.21.15.116:0/1294002103 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914400cd80 con 0x7f9154101050 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915419d250 con 0x7f9154101050 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f914a7fc640 1 -- 172.21.15.116:0/1294002103 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91440108e0 con 0x7f9154101050 2024-03-19T02:37:29.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f914a7fc640 1 -- 172.21.15.116:0/1294002103 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144018920 con 0x7f9154101050 2024-03-19T02:37:29.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.325+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541a4ae0 con 0x7f9154101050 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f914a7fc640 1 -- 172.21.15.116:0/1294002103 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9144010a90 con 0x7f9154101050 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 Processor -- start 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 -- start start 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1ace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1b8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b01c110500 con 0x55b01c1aca30 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b01c0b8be0 con 0x55b01c1ad350 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.328+0000 7f915e1a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b01c10c5a0 con 0x55b01c1b5f30 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:47164/0 (socket says 172.21.15.116:47164) 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 -- 172.21.15.116:0/3215715050 learned_addr learned my addr 172.21.15.116:0/3215715050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55688/0 (socket says 172.21.15.116:55688) 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9158b54640 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f914a7fc640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 0x7f91400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159b56640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 0x7f91400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 -- 172.21.15.116:0/3215715050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 msgr2=0x55b01c1b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1b8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 -- 172.21.15.116:0/3215715050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 msgr2=0x55b01c1ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 -- 172.21.15.116:0/3215715050 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01c0da000 con 0x55b01c1ad350 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f914a7fc640 1 -- 172.21.15.116:0/1294002103 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f91440cf4c0 con 0x7f9154101050 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159355640 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f91500114f0 tx=0x7f91500119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec44796a55f05cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f9159b56640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 0x7f91400a4e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f914c005cc0 tx=0x7f914c02c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f913f7fe640 1 -- 172.21.15.116:0/3215715050 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915001e070 con 0x55b01c1ad350 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f913f7fe640 1 -- 172.21.15.116:0/3215715050 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f915000fa70 con 0x55b01c1ad350 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.329+0000 7f913f7fe640 1 -- 172.21.15.116:0/3215715050 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150019a80 con 0x55b01c1ad350 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- 172.21.15.116:0/3215715050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 msgr2=0x55b01c1b59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f91500114f0 tx=0x7f91500119b0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- 172.21.15.116:0/3215715050 shutdown_connections 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1b59f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 --2- 172.21.15.116:0/3215715050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- 172.21.15.116:0/3215715050 >> 172.21.15.116:0/3215715050 conn(0x55b01c1a7160 msgr2=0x55b01c1a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- 172.21.15.116:0/3215715050 shutdown_connections 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- 172.21.15.116:0/3215715050 wait complete. 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.330+0000 7f915e1a2740 1 -- start start 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 Processor -- start 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 0x55b01c1b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1b5f30 0x55b01c1b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b01c1b8d20 con 0x55b01c1aca30 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b01c0cc160 con 0x55b01c1b5f30 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b01c1b8ea0 con 0x55b01c1ad350 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f915e1a2740 10 client.0 _ll_register_callbacks cb 0x55b01c108a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:29.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9159355640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 0x55b01c1b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9159b56640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1b5f30 0x55b01c1b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9159355640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55696/0 (socket says 172.21.15.116:55696) 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9159355640 1 -- 172.21.15.116:0/854322272 learned_addr learned my addr 172.21.15.116:0/854322272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 -- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1b5f30 msgr2=0x55b01c1b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1b5f30 0x55b01c1b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 -- 172.21.15.116:0/854322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 msgr2=0x55b01c1b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9158b54640 1 -- 172.21.15.116:0/854322272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01beee7d0 con 0x55b01c1ad350 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.331+0000 7f9159355640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f9158b54640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 0x55b01c1b3480 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f91440cc8a0 tx=0x7f9144009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd66ad904a82f355 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:37:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914409f930 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f915000fa70 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f914409f580 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f914409b890 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 10 client.28237 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b01be243f0 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f915000fa70 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f914409bb80 con 0x55b01c1ad350 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 10 client.28237 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 10 client.28237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 20 client.28237 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c1c6260 0x55b01c1c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 4 client.28237 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.332+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b01c11da40 con 0x55b01c1c6260 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.333+0000 7f9159355640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c1c6260 0x55b01c1c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.333+0000 7f913cff9640 10 client.28237 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.333+0000 7f9159355640 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c1c6260 0x55b01c1c8650 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f143e114f9d098af server_cookie=87bf0cb500bca21b in_seq=0 out_seq=0 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f913cff9640 1 -- 172.21.15.116:0/854322272 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b01c11da40 con 0x55b01c1c6260 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f913cff9640 10 client.28237 handle_command_reply: tid=0 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 client.28237 shutdown 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 msgr2=0x55b01c1b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 0x55b01c1b3480 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f91440cc8a0 tx=0x7f9144009310 comp rx=0 tx=0).stop 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 shutdown_connections 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c1c6260 0x55b01c1c8650 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1b5f30 0x55b01c1b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1ad350 0x55b01c1b3480 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 --2- 172.21.15.116:0/854322272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f913cff9640 0 client.28237 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 >> 172.21.15.116:0/854322272 conn(0x55b01c1a7160 msgr2=0x55b01c1a8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 shutdown_connections 2024-03-19T02:37:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 1 -- 172.21.15.116:0/854322272 wait complete. 2024-03-19T02:37:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.334+0000 7f915e1a2740 20 client.28237 trim_cache size 0 max 16384 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.355+0000 7f915e1a2740 1 Processor -- start 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.355+0000 7f915e1a2740 1 -- start start 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1aca30 0x55b01c1c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91541aed40 con 0x55b01c1ad350 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91541b0880 con 0x55b01c1aca30 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f915e1a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91541af410 con 0x55b01c1b5f30 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55708/0 (socket says 172.21.15.116:55708) 2024-03-19T02:37:29.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 -- 172.21.15.116:0/2274874278 learned_addr learned my addr 172.21.15.116:0/2274874278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9159b56640 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9159355640 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1aca30 0x55b01c1c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 -- 172.21.15.116:0/2274874278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 msgr2=0x55b01c1cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 -- 172.21.15.116:0/2274874278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1aca30 msgr2=0x55b01c1c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1aca30 0x55b01c1c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 -- 172.21.15.116:0/2274874278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01c1c89c0 con 0x55b01c1ad350 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f9158b54640 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f9144009380 tx=0x7f91440cf8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9096111896572ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.356+0000 7f913effd640 1 -- 172.21.15.116:0/2274874278 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91440a3f80 con 0x55b01c1ad350 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f913effd640 1 -- 172.21.15.116:0/2274874278 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91440afc10 con 0x55b01c1ad350 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f913effd640 1 -- 172.21.15.116:0/2274874278 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144097040 con 0x55b01c1ad350 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 -- 172.21.15.116:0/2274874278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 msgr2=0x55b01c1a8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f9144009380 tx=0x7f91440cf8c0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 -- 172.21.15.116:0/2274874278 shutdown_connections 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1ad350 0x55b01c1a8e20 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 --2- 172.21.15.116:0/2274874278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1aca30 0x55b01c1c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 -- 172.21.15.116:0/2274874278 >> 172.21.15.116:0/2274874278 conn(0x55b01c1a7160 msgr2=0x55b01c1b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 -- 172.21.15.116:0/2274874278 shutdown_connections 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.357+0000 7f915e1a2740 1 -- 172.21.15.116:0/2274874278 wait complete. 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 -- start start 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 Processor -- start 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1bb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91541b01b0 con 0x55b01c1aca30 2024-03-19T02:37:29.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91541afae0 con 0x55b01c1ad350 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91541acd40 con 0x55b01c1b5f30 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f915e1a2740 10 client.0 _ll_register_callbacks cb 0x7f914400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f9159355640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f9159355640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55716/0 (socket says 172.21.15.116:55716) 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f9159355640 1 -- 172.21.15.116:0/483893289 learned_addr learned my addr 172.21.15.116:0/483893289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f9158b54640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.358+0000 7f9159b56640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 -- 172.21.15.116:0/483893289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 msgr2=0x55b01c1bb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1bb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 -- 172.21.15.116:0/483893289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 msgr2=0x55b01c1be120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 -- 172.21.15.116:0/483893289 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01bde40f0 con 0x55b01c1ad350 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9159355640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:29.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f9158b54640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1bb0f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55b01c1c8940 tx=0x7f914400c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f80daf31349ec963 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f913cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914409bcf0 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f9144097040 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91440afc10 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f91440a3f80 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f915e1a2740 10 client.28295 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b01c22ce60 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.359+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91440afc10 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f914409cb10 con 0x55b01c1ad350 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 10 client.28295 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 10 client.28295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 20 client.28295 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c22d1a0 0x55b01c22f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 4 client.28295 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b01c117d90 con 0x55b01c22d1a0 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f9159355640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c22d1a0 0x55b01c22f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f913cff9640 10 client.28295 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f9159355640 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c22d1a0 0x55b01c22f5c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1542f731e9da61ea server_cookie=5bf3525f977e54ab in_seq=0 out_seq=0 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f913cff9640 1 -- 172.21.15.116:0/483893289 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b01c117d90 con 0x55b01c22d1a0 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f913cff9640 10 client.28295 handle_command_reply: tid=0 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 1 client.28295 shutdown 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 msgr2=0x55b01c1bb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.360+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1bb0f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55b01c1c8940 tx=0x7f914400c7d0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 shutdown_connections 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b01c22d1a0 0x55b01c22f5c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b01c1b5f30 0x55b01c1bb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b01c1ad350 0x55b01c1bb0f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f913cff9640 0 client.28295 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 --2- 172.21.15.116:0/483893289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b01c1aca30 0x55b01c1be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 >> 172.21.15.116:0/483893289 conn(0x55b01c1a7160 msgr2=0x55b01c1a61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 shutdown_connections 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 1 -- 172.21.15.116:0/483893289 wait complete. 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915e1a2740 20 client.28295 trim_cache size 0 max 16384 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stdout: "33e489c3-c8e4-4db4-966d-f258a2d06c23": { 2024-03-19T02:37:29.369 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:37:29.370 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "33e489c3-c8e4-4db4-966d-f258a2d06c23", 2024-03-19T02:37:29.370 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:37:29.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:37:29.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:37:29.370 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 msgr2=0x7f91400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 0x7f91400a4e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f914c005cc0 tx=0x7f914c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 msgr2=0x7f91541984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f914400e9f0 tx=0x7f914400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 shutdown_connections 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91400a2970 0x7f91400a4e30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91541066a0 0x7f915419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9154101050 0x7f91541984b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 --2- 172.21.15.116:0/1294002103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9154100690 0x7f9154197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 >> 172.21.15.116:0/1294002103 conn(0x7f9154077820 msgr2=0x7f9154068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 shutdown_connections 2024-03-19T02:37:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:29.361+0000 7f915b5b9640 1 -- 172.21.15.116:0/1294002103 wait complete. 2024-03-19T02:37:29.373 INFO:tasks.cephfs.filesystem:scrub status for tag:33e489c3-c8e4-4db4-966d-f258a2d06c23 - {'path': '/', 'tag': '33e489c3-c8e4-4db4-966d-f258a2d06c23', 'options': 'recursive,force'} 2024-03-19T02:37:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:28 smithi162 ceph-mon[35761]: pgmap v301: 129 pgs: 129 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:37:29.792 INFO:tasks.workunit.client.0.smithi116.stdout: 1 313590 3.14 MB/sec execute 412 sec latency 25.299 ms 2024-03-19T02:37:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:29 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:37:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:37:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:29 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:37:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:37:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:29 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:37:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:37:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:30.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 314073 3.14 MB/sec execute 413 sec latency 35.169 ms 2024-03-19T02:37:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:30 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:37:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:30 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:30 smithi154 ceph-mon[35693]: pgmap v302: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:37:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:30 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:37:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:30 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:30 smithi116 ceph-mon[32263]: pgmap v302: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:37:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:30 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:37:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:30 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:37:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:30 smithi162 ceph-mon[35761]: pgmap v302: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:37:31.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 314649 3.13 MB/sec execute 414 sec latency 29.248 ms 2024-03-19T02:37:32.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 315456 3.14 MB/sec execute 415 sec latency 21.405 ms 2024-03-19T02:37:33.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 315859 3.14 MB/sec execute 416 sec latency 18.892 ms 2024-03-19T02:37:34.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:33 smithi162 ceph-mon[35761]: pgmap v303: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:33 smithi154 ceph-mon[35693]: pgmap v303: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:33 smithi116 ceph-mon[32263]: pgmap v303: 129 pgs: 129 active+clean; 329 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:34.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 316293 3.14 MB/sec execute 417 sec latency 15.746 ms 2024-03-19T02:37:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:34 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:37:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:34 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:37:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:34 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:37:35.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 316851 3.13 MB/sec execute 418 sec latency 33.589 ms 2024-03-19T02:37:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:35 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:37:36.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:35 smithi162 ceph-mon[35761]: pgmap v304: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:37:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:35 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:37:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:35 smithi154 ceph-mon[35693]: pgmap v304: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:37:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:35 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:37:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:35 smithi116 ceph-mon[32263]: pgmap v304: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T02:37:36.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 317401 3.14 MB/sec execute 419 sec latency 32.905 ms 2024-03-19T02:37:37.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 318040 3.13 MB/sec execute 420 sec latency 20.552 ms 2024-03-19T02:37:38.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:37 smithi162 ceph-mon[35761]: pgmap v305: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:37 smithi154 ceph-mon[35693]: pgmap v305: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:37 smithi116 ceph-mon[32263]: pgmap v305: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:38.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 318732 3.14 MB/sec execute 421 sec latency 16.437 ms 2024-03-19T02:37:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 319248 3.14 MB/sec execute 422 sec latency 27.501 ms 2024-03-19T02:37:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:39 smithi154 ceph-mon[35693]: pgmap v306: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:39 smithi116 ceph-mon[32263]: pgmap v306: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:40.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:39 smithi162 ceph-mon[35761]: pgmap v306: 129 pgs: 129 active+clean; 332 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:37:40.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 319691 3.14 MB/sec execute 423 sec latency 19.336 ms 2024-03-19T02:37:41.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 320185 3.13 MB/sec execute 424 sec latency 34.973 ms 2024-03-19T02:37:42.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:41 smithi154 ceph-mon[35693]: pgmap v307: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:37:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:41 smithi116 ceph-mon[32263]: pgmap v307: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:37:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:41 smithi162 ceph-mon[35761]: pgmap v307: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:37:42.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 320776 3.14 MB/sec execute 425 sec latency 28.983 ms 2024-03-19T02:37:43.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 321337 3.13 MB/sec execute 426 sec latency 23.189 ms 2024-03-19T02:37:44.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:43 smithi154 ceph-mon[35693]: pgmap v308: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:43 smithi116 ceph-mon[32263]: pgmap v308: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:44.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:43 smithi162 ceph-mon[35761]: pgmap v308: 129 pgs: 129 active+clean; 336 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:44.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 322005 3.13 MB/sec execute 427 sec latency 17.609 ms 2024-03-19T02:37:45.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 322580 3.14 MB/sec execute 428 sec latency 23.272 ms 2024-03-19T02:37:46.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:45 smithi154 ceph-mon[35693]: pgmap v309: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:45 smithi116 ceph-mon[32263]: pgmap v309: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:46.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:45 smithi162 ceph-mon[35761]: pgmap v309: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:46.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 323109 3.14 MB/sec execute 429 sec latency 20.380 ms 2024-03-19T02:37:47.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 323552 3.13 MB/sec execute 430 sec latency 31.685 ms 2024-03-19T02:37:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:47 smithi154 ceph-mon[35693]: pgmap v310: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:47 smithi116 ceph-mon[32263]: pgmap v310: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:47 smithi162 ceph-mon[35761]: pgmap v310: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:48.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 324183 3.14 MB/sec execute 431 sec latency 17.302 ms 2024-03-19T02:37:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 324655 3.13 MB/sec execute 432 sec latency 36.247 ms 2024-03-19T02:37:50.218 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:49 smithi162 ceph-mon[35761]: pgmap v311: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:49 smithi154 ceph-mon[35693]: pgmap v311: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:49 smithi116 ceph-mon[32263]: pgmap v311: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:50.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 325227 3.13 MB/sec execute 433 sec latency 21.824 ms 2024-03-19T02:37:51.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 326073 3.14 MB/sec execute 434 sec latency 35.881 ms 2024-03-19T02:37:52.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:51 smithi154 ceph-mon[35693]: pgmap v312: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:37:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:51 smithi116 ceph-mon[32263]: pgmap v312: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:37:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:51 smithi162 ceph-mon[35761]: pgmap v312: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 124 op/s 2024-03-19T02:37:52.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 326407 3.14 MB/sec execute 435 sec latency 18.022 ms 2024-03-19T02:37:53.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 326812 3.13 MB/sec execute 436 sec latency 17.395 ms 2024-03-19T02:37:54.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:53 smithi154 ceph-mon[35693]: pgmap v313: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:53 smithi116 ceph-mon[32263]: pgmap v313: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:53 smithi162 ceph-mon[35761]: pgmap v313: 129 pgs: 129 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-03-19T02:37:54.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 327341 3.13 MB/sec execute 437 sec latency 28.847 ms 2024-03-19T02:37:55.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 327949 3.13 MB/sec execute 438 sec latency 31.877 ms 2024-03-19T02:37:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:55 smithi154 ceph-mon[35693]: pgmap v314: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:55 smithi116 ceph-mon[32263]: pgmap v314: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:55 smithi162 ceph-mon[35761]: pgmap v314: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:37:56.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 328516 3.12 MB/sec execute 439 sec latency 27.690 ms 2024-03-19T02:37:57.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 329253 3.13 MB/sec execute 440 sec latency 19.813 ms 2024-03-19T02:37:58.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:57 smithi154 ceph-mon[35693]: pgmap v315: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:57 smithi116 ceph-mon[32263]: pgmap v315: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:57 smithi162 ceph-mon[35761]: pgmap v315: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:37:58.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 329722 3.13 MB/sec execute 441 sec latency 19.896 ms 2024-03-19T02:37:59.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:37:59.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:37:59.846 INFO:tasks.workunit.client.0.smithi116.stdout: 1 330254 3.13 MB/sec execute 442 sec latency 17.099 ms 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 -- 172.21.15.116:0/2694583979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 msgr2=0x7f6000106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 --2- 172.21.15.116:0/2694583979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f6000106a80 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009930 tx=0x7f5ffc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 -- 172.21.15.116:0/2694583979 shutdown_connections 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 --2- 172.21.15.116:0/2694583979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000101050 0x7f600010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 --2- 172.21.15.116:0/2694583979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000100690 0x7f6000100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 --2- 172.21.15.116:0/2694583979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f6000106a80 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 -- 172.21.15.116:0/2694583979 >> 172.21.15.116:0/2694583979 conn(0x7f6000077820 msgr2=0x7f6000077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 -- 172.21.15.116:0/2694583979 shutdown_connections 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.959+0000 7f600773b640 1 -- 172.21.15.116:0/2694583979 wait complete. 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 Processor -- start 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 -- start start 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 0x7f6000197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6000110920 con 0x7f60001066a0 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60001107a0 con 0x7f6000101050 2024-03-19T02:37:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f600773b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6000110aa0 con 0x7f6000100690 2024-03-19T02:37:59.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6004cd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f60054d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 0x7f6000197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6004cd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39746/0 (socket says 172.21.15.116:39746) 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f60054d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 0x7f6000197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44476/0 (socket says 172.21.15.116:44476) 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56722/0 (socket says 172.21.15.116:56722) 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6004cd6640 1 -- 172.21.15.116:0/2791277386 learned_addr learned my addr 172.21.15.116:0/2791277386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 -- 172.21.15.116:0/2791277386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 msgr2=0x7f6000197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 0x7f6000197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 -- 172.21.15.116:0/2791277386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 msgr2=0x7f60001984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 -- 172.21.15.116:0/2791277386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600019cfa0 con 0x7f60001066a0 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6004cd6640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:59.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.960+0000 7f6005cd8640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009fd0 tx=0x7f5ff800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.961+0000 7f5ff67fc640 1 -- 172.21.15.116:0/2791277386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff800ce20 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.961+0000 7f5ff67fc640 1 -- 172.21.15.116:0/2791277386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ff8005020 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.961+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600019d290 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.961+0000 7f5ff67fc640 1 -- 172.21.15.116:0/2791277386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8010690 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.961+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60001a4b20 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.963+0000 7f5ff67fc640 1 -- 172.21.15.116:0/2791277386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5ff80051d0 con 0x7f60001066a0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 Processor -- start 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 -- start start 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f5ff67fc640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 0x7f5fec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c189f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f6004cd6640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 0x7f5fec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d204b73500 con 0x55d204c10350 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d204b1bbe0 con 0x55d204c18f30 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f600a3fe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d204b6f5a0 con 0x55d204c0fa30 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f5ff67fc640 1 -- 172.21.15.116:0/2791277386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5ff80cf830 con 0x7f60001066a0 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44486/0 (socket says 172.21.15.116:44486) 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f6004cd6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 -- 172.21.15.116:0/3597754737 learned_addr learned my addr 172.21.15.116:0/3597754737 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f6005cd8640 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 -- 172.21.15.116:0/3597754737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 msgr2=0x55d204c1b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 -- 172.21.15.116:0/3597754737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 msgr2=0x55d204c189f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c189f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f60054d7640 1 -- 172.21.15.116:0/3597754737 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d204b3d000 con 0x55d204c0fa30 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.964+0000 7f6004cd6640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 0x7f5fec0a4eb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009fd0 tx=0x7f5ff0013460 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f60054d7640 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc036040 tx=0x7f5ffc009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caa701928a17cc35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f5feb7fe640 1 -- 172.21.15.116:0/3597754737 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc045020 con 0x55d204c0fa30 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f5feb7fe640 1 -- 172.21.15.116:0/3597754737 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ffc02c7e0 con 0x55d204c0fa30 2024-03-19T02:37:59.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f5feb7fe640 1 -- 172.21.15.116:0/3597754737 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0053b0 con 0x55d204c0fa30 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 -- 172.21.15.116:0/3597754737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 msgr2=0x55d204c0fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc036040 tx=0x7f5ffc009560 comp rx=0 tx=0).stop 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 -- 172.21.15.116:0/3597754737 shutdown_connections 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c189f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3597754737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c0fe10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.965+0000 7f600a3fe740 1 -- 172.21.15.116:0/3597754737 >> 172.21.15.116:0/3597754737 conn(0x55d204c0a160 msgr2=0x55d204c0a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- 172.21.15.116:0/3597754737 shutdown_connections 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- 172.21.15.116:0/3597754737 wait complete. 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- start start 2024-03-19T02:37:59.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 Processor -- start 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c15e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 0x55d204c16390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d204c1bcc0 con 0x55d204c18f30 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d204b2f160 con 0x55d204c10350 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d204c1be40 con 0x55d204c0fa30 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f600a3fe740 10 client.0 _ll_register_callbacks cb 0x55d204b6ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f6005cd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f60054d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c15e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f6005cd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56744/0 (socket says 172.21.15.116:56744) 2024-03-19T02:37:59.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f6005cd8640 1 -- 172.21.15.116:0/1923097803 learned_addr learned my addr 172.21.15.116:0/1923097803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.966+0000 7f6004cd6640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 0x55d204c16390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 -- 172.21.15.116:0/1923097803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 msgr2=0x55d204c15e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c15e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 -- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 msgr2=0x55d204c16390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 0x55d204c16390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 -- 172.21.15.116:0/1923097803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2049517d0 con 0x55d204c18f30 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6004cd6640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 0x55d204c16390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f6005cd8640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f5ff809ba40 tx=0x7f5ff809bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2e75c4e9b28da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f5fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff80b9060 con 0x55d204c18f30 2024-03-19T02:37:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5ff8098120 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ffc02c7e0 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ff80b3030 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f600a3fe740 10 client.18516 fetch_fsmap learned FSMap version 30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d2048873f0 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.967+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ffc02c7e0 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5ff80ce040 con 0x55d204c18f30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 10 client.18516 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 10 client.18516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 20 client.18516 populate_metadata read hostname 'smithi116' 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c29060 0x55d204c2b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 4 client.18516 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:37:59.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d204b80a40 con 0x55d204c29060 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f60054d7640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c29060 0x55d204c2b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f5fe8ff9640 10 client.18516 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.968+0000 7f60054d7640 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c29060 0x55d204c2b470 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11c0a3ea35907380 server_cookie=9f32de7c0c907455 in_seq=0 out_seq=0 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1923097803 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d204b80a40 con 0x55d204c29060 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f5fe8ff9640 10 client.18516 handle_command_reply: tid=0 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 client.18516 shutdown 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 msgr2=0x55d204c168d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f5ff809ba40 tx=0x7f5ff809bb70 comp rx=0 tx=0).stop 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 shutdown_connections 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c29060 0x55d204c2b470 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c18f30 0x55d204c168d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c10350 0x55d204c16390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f5fe8ff9640 0 client.18516 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1923097803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c0fa30 0x55d204c15e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 >> 172.21.15.116:0/1923097803 conn(0x55d204c0a160 msgr2=0x55d204c0beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 shutdown_connections 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 1 -- 172.21.15.116:0/1923097803 wait complete. 2024-03-19T02:37:59.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.969+0000 7f600a3fe740 20 client.18516 trim_cache size 0 max 16384 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 Processor -- start 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 -- start start 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 0x55d204c18580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c24a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c18f30 0x55d204c30d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60001aeb80 con 0x55d204c10350 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60001b06c0 con 0x55d204c0fa30 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f600a3fe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60001af250 con 0x55d204c18f30 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f6004cd6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c24a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f60054d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 0x55d204c18580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f6005cd8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c18f30 0x55d204c30d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f6004cd6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c24a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56754/0 (socket says 172.21.15.116:56754) 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.988+0000 7f6004cd6640 1 -- 172.21.15.116:0/3676894791 learned_addr learned my addr 172.21.15.116:0/3676894791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 -- 172.21.15.116:0/3676894791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c18f30 msgr2=0x55d204c30d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c18f30 0x55d204c30d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 -- 172.21.15.116:0/3676894791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 msgr2=0x55d204c24a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c24a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 -- 172.21.15.116:0/3676894791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2048470f0 con 0x55d204c0fa30 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f60054d7640 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 0x55d204c18580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55d204c2bce0 tx=0x7f5ffc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb77d76763e1fe16 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f5feaffd640 1 -- 172.21.15.116:0/3676894791 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc045040 con 0x55d204c0fa30 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f5feaffd640 1 -- 172.21.15.116:0/3676894791 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ffc02cb70 con 0x55d204c0fa30 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.989+0000 7f5feaffd640 1 -- 172.21.15.116:0/3676894791 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0052c0 con 0x55d204c0fa30 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- 172.21.15.116:0/3676894791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 msgr2=0x55d204c18580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:37:59.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 0x55d204c18580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55d204c2bce0 tx=0x7f5ffc000bc0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- 172.21.15.116:0/3676894791 shutdown_connections 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c18f30 0x55d204c30d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c10350 0x55d204c24a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 --2- 172.21.15.116:0/3676894791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c0fa30 0x55d204c18580 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- 172.21.15.116:0/3676894791 >> 172.21.15.116:0/3676894791 conn(0x55d204c0a160 msgr2=0x55d204c15e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- 172.21.15.116:0/3676894791 shutdown_connections 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- 172.21.15.116:0/3676894791 wait complete. 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 -- start start 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.990+0000 7f600a3fe740 1 Processor -- start 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c10350 0x55d204c1c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60001afff0 con 0x55d204c0fa30 2024-03-19T02:38:00.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60001af920 con 0x55d204c18f30 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60001acb80 con 0x55d204c10350 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f600a3fe740 10 client.0 _ll_register_callbacks cb 0x7f5ff809b610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56766/0 (socket says 172.21.15.116:56766) 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 -- 172.21.15.116:0/1705375679 learned_addr learned my addr 172.21.15.116:0/1705375679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f6004cd6640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c10350 0x55d204c1c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f6005cd8640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 -- 172.21.15.116:0/1705375679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c10350 msgr2=0x55d204c1c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c10350 0x55d204c1c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 -- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 msgr2=0x55d204c1cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 -- 172.21.15.116:0/1705375679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d204c1d290 con 0x55d204c0fa30 2024-03-19T02:38:00.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f60054d7640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0095a0 tx=0x7f5ffc02cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2ff2572ce259a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f5fe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc04e070 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.991+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f5ffc054ab0 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ffc02cb70 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ffc063050 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 10 client.18522 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d204c8f2a0 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ffc02cb70 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5ffc004270 con 0x55d204c0fa30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 10 client.18522 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 10 client.18522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 20 client.18522 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c8f5a0 0x55d204c919c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 4 client.18522 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d204b7ad90 con 0x55d204c8f5a0 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.992+0000 7f6004cd6640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c8f5a0 0x55d204c919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f5fe8ff9640 10 client.18522 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f6004cd6640 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c8f5a0 0x55d204c919c0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac7905d8c8fc382f server_cookie=a03358e5155792b6 in_seq=0 out_seq=0 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f5fe8ff9640 1 -- 172.21.15.116:0/1705375679 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d204b7ad90 con 0x55d204c8f5a0 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f5fe8ff9640 10 client.18522 handle_command_reply: tid=0 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 client.18522 shutdown 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 msgr2=0x55d204c21310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0095a0 tx=0x7f5ffc02cda0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 shutdown_connections 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d204c8f5a0 0x55d204c919c0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d204c18f30 0x55d204c1cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d204c10350 0x55d204c1c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 --2- 172.21.15.116:0/1705375679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d204c0fa30 0x55d204c21310 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 >> 172.21.15.116:0/1705375679 conn(0x55d204c0a160 msgr2=0x55d204c166a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f5fe8ff9640 0 client.18522 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 shutdown_connections 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 1 -- 172.21.15.116:0/1705375679 wait complete. 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.993+0000 7f600a3fe740 20 client.18522 trim_cache size 0 max 16384 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:38:00.004 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:38:00.005 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 msgr2=0x7f5fec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 0x7f5fec0a4eb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009fd0 tx=0x7f5ff0013460 comp rx=0 tx=0).stop 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 msgr2=0x7f600019c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009fd0 tx=0x7f5ff800ef10 comp rx=0 tx=0).stop 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 shutdown_connections 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5fec0a29f0 0x7f5fec0a4eb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60001066a0 0x7f600019c870 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6000101050 0x7f60001984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 --2- 172.21.15.116:0/2791277386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6000100690 0x7f6000197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 >> 172.21.15.116:0/2791277386 conn(0x7f6000077820 msgr2=0x7f60000ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 shutdown_connections 2024-03-19T02:38:00.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:37:59.994+0000 7f600773b640 1 -- 172.21.15.116:0/2791277386 wait complete. 2024-03-19T02:38:00.007 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:38:00.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.113+0000 7f8026b10640 1 -- 172.21.15.116:0/2630725256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020106730 msgr2=0x7f8020106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.113+0000 7f8026b10640 1 --2- 172.21.15.116:0/2630725256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020106730 0x7f8020106b10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8014009930 tx=0x7f801402bc50 comp rx=0 tx=0).stop 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.113+0000 7f8026b10640 1 -- 172.21.15.116:0/2630725256 shutdown_connections 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.113+0000 7f8026b10640 1 --2- 172.21.15.116:0/2630725256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80201010e0 0x7f802010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.113+0000 7f8026b10640 1 --2- 172.21.15.116:0/2630725256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8020100720 0x7f8020100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 --2- 172.21.15.116:0/2630725256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020106730 0x7f8020106b10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- 172.21.15.116:0/2630725256 >> 172.21.15.116:0/2630725256 conn(0x7f8020077820 msgr2=0x7f8020077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- 172.21.15.116:0/2630725256 shutdown_connections 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- 172.21.15.116:0/2630725256 wait complete. 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 Processor -- start 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- start start 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80201010e0 0x7f8020198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 0x7f802019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8020110a30 con 0x7f8020100720 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80201108b0 con 0x7f80201010e0 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f8026b10640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8020110bb0 con 0x7f8020106730 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.114+0000 7f80248ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56788/0 (socket says 172.21.15.116:56788) 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 -- 172.21.15.116:0/2432796908 learned_addr learned my addr 172.21.15.116:0/2432796908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f801ffff640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80201010e0 0x7f8020198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80250ad640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 0x7f802019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 -- 172.21.15.116:0/2432796908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 msgr2=0x7f802019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 0x7f802019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 -- 172.21.15.116:0/2432796908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80201010e0 msgr2=0x7f8020198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80201010e0 0x7f8020198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 -- 172.21.15.116:0/2432796908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802019d0a0 con 0x7f8020100720 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80250ad640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 0x7f802019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f80248ac640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8014009230 tx=0x7f801402c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8014043020 con 0x7f8020100720 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f801402cb10 con 0x7f8020100720 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802019d330 con 0x7f8020100720 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8014003b50 con 0x7f8020100720 2024-03-19T02:38:00.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.115+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80201a4c20 con 0x7f8020100720 2024-03-19T02:38:00.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.117+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8020068610 con 0x7f8020100720 2024-03-19T02:38:00.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.117+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8014046020 con 0x7f8020100720 2024-03-19T02:38:00.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.118+0000 7f801dffb640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 0x7f80100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.118+0000 7f801ffff640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 0x7f80100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.118+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f80140f1320 con 0x7f8020100720 2024-03-19T02:38:00.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.119+0000 7f801ffff640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 0x7f80100a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f800c006fd0 tx=0x7f800c008040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.124+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f801404d020 con 0x7f8020100720 2024-03-19T02:38:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:37:59 smithi154 ceph-mon[35693]: pgmap v316: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:38:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:37:59 smithi116 ceph-mon[32263]: pgmap v316: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:38:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.370+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8020109ca0 con 0x7f8020100720 2024-03-19T02:38:00.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.371+0000 7f801dffb640 1 -- 172.21.15.116:0/2432796908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f80140dd030 con 0x7f8020100720 2024-03-19T02:38:00.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:38:00.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:38:00.372 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 msgr2=0x7f80100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 0x7f80100a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f800c006fd0 tx=0x7f800c008040 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 msgr2=0x7f8020198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8014009230 tx=0x7f801402c730 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 shutdown_connections 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80100a2970 0x7f80100a4e30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8020106730 0x7f802019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80201010e0 0x7f8020198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 --2- 172.21.15.116:0/2432796908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8020100720 0x7f8020198100 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 >> 172.21.15.116:0/2432796908 conn(0x7f8020077820 msgr2=0x7f8020068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 shutdown_connections 2024-03-19T02:38:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.373+0000 7f8026b10640 1 -- 172.21.15.116:0/2432796908 wait complete. 2024-03-19T02:38:00.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:38:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:37:59 smithi162 ceph-mon[35761]: pgmap v316: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- 172.21.15.116:0/485588189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 msgr2=0x7f184c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 --2- 172.21.15.116:0/485588189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c105a70 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1848009900 tx=0x7f184802bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- 172.21.15.116:0/485588189 shutdown_connections 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 --2- 172.21.15.116:0/485588189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c101330 0x7f184c10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 --2- 172.21.15.116:0/485588189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c100970 0x7f184c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 --2- 172.21.15.116:0/485588189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c105a70 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- 172.21.15.116:0/485588189 >> 172.21.15.116:0/485588189 conn(0x7f184c077580 msgr2=0x7f184c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- 172.21.15.116:0/485588189 shutdown_connections 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- 172.21.15.116:0/485588189 wait complete. 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 Processor -- start 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.500+0000 7f1854030640 1 -- start start 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 0x7f184c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c101330 0x7f184c19bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10f870 con 0x7f184c105690 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10f6f0 con 0x7f184c101330 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1854030640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10f9f0 con 0x7f184c100970 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f18525cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 0x7f184c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f18525cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56810/0 (socket says 172.21.15.116:56810) 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f18525cd640 1 -- 172.21.15.116:0/435938743 learned_addr learned my addr 172.21.15.116:0/435938743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f18515cb640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c101330 0x7f184c19bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 -- 172.21.15.116:0/435938743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c101330 msgr2=0x7f184c19bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c101330 0x7f184c19bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 -- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 msgr2=0x7f184c195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 -- 172.21.15.116:0/435938743 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c1963d0 con 0x7f184c100970 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f18525cd640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1851dcc640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 0x7f184c19ba50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1848036040 tx=0x7f184802cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.501+0000 7f1842ffd640 1 -- 172.21.15.116:0/435938743 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848043020 con 0x7f184c100970 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.502+0000 7f1842ffd640 1 -- 172.21.15.116:0/435938743 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1848004490 con 0x7f184c100970 2024-03-19T02:38:00.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.502+0000 7f1842ffd640 1 -- 172.21.15.116:0/435938743 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848003b80 con 0x7f184c100970 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.502+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c196660 con 0x7f184c100970 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.502+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c1a29d0 con 0x7f184c100970 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1842ffd640 1 -- 172.21.15.116:0/435938743 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1848046020 con 0x7f184c100970 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 Processor -- start 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 -- start start 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1842ffd640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 0x7f18380a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae3cef44b0 con 0x55ae3cf91bf0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ae3ce9cd70 con 0x55ae3cf9a7d0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.504+0000 7f1856c17740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ae3cef0550 con 0x55ae3cf912d0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44574/0 (socket says 172.21.15.116:44574) 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 -- 172.21.15.116:0/599488146 learned_addr learned my addr 172.21.15.116:0/599488146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18515cb640 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 0x7f18380a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f1842ffd640 1 -- 172.21.15.116:0/435938743 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f18480f0ac0 con 0x7f184c100970 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 -- 172.21.15.116:0/599488146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf9a7d0 msgr2=0x55ae3cf9cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 -- 172.21.15.116:0/599488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 msgr2=0x55ae3cf9a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 -- 172.21.15.116:0/599488146 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae3cebf210 con 0x55ae3cf912d0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 0x7f18380a4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f184400a7b0 tx=0x7f1844020000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f18525cd640 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f184401c800 tx=0x7f184401ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a419d5a127cbd90e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f1837fff640 1 -- 172.21.15.116:0/599488146 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18440105f0 con 0x55ae3cf912d0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f1837fff640 1 -- 172.21.15.116:0/599488146 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f184402a070 con 0x55ae3cf912d0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.505+0000 7f1837fff640 1 -- 172.21.15.116:0/599488146 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844025640 con 0x55ae3cf912d0 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 -- 172.21.15.116:0/599488146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 msgr2=0x55ae3cf916b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f184401c800 tx=0x7f184401ccc0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 -- 172.21.15.116:0/599488146 shutdown_connections 2024-03-19T02:38:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 --2- 172.21.15.116:0/599488146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf912d0 0x55ae3cf916b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 -- 172.21.15.116:0/599488146 >> 172.21.15.116:0/599488146 conn(0x55ae3cf8ba00 msgr2=0x55ae3cf8bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 -- 172.21.15.116:0/599488146 shutdown_connections 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.506+0000 7f1856c17740 1 -- 172.21.15.116:0/599488146 wait complete. 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 -- start start 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 Processor -- start 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cf977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf98270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae3cf9d560 con 0x55ae3cf91bf0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ae3ceb0310 con 0x55ae3cf912d0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ae3cf9d6e0 con 0x55ae3cf9a7d0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1856c17740 10 client.0 _ll_register_callbacks cb 0x55ae3cea42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cf977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56836/0 (socket says 172.21.15.116:56836) 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 -- 172.21.15.116:0/3100144232 learned_addr learned my addr 172.21.15.116:0/3100144232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18525cd640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf98270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 -- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 msgr2=0x55ae3cf98270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf98270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 -- 172.21.15.116:0/3100144232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 msgr2=0x55ae3cf977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cf977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.507+0000 7f18515cb640 1 -- 172.21.15.116:0/3100144232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae3cc083f0 con 0x55ae3cf91bf0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18515cb640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f183c00bad0 tx=0x7f183c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791157645ebd74fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c003e20 con 0x55ae3cf91bf0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f183c011440 con 0x55ae3cf91bf0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f184402a070 con 0x55ae3cf91bf0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f183c005020 con 0x55ae3cf91bf0 2024-03-19T02:38:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 10 client.18540 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ae3ccd27d0 con 0x55ae3cf91bf0 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f184402a070 con 0x55ae3cf91bf0 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f183c011640 con 0x55ae3cf91bf0 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 10 client.18540 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 10 client.18540 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 20 client.18540 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3cfaab80 0x55ae3cfad000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 4 client.18540 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.508+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae3cf523e0 con 0x55ae3cfaab80 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.509+0000 7f1851dcc640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3cfaab80 0x55ae3cfad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.509+0000 7f18357fa640 10 client.18540 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.509+0000 7f1851dcc640 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3cfaab80 0x55ae3cfad000 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=503fe47b6464971d server_cookie=b4e7a457d4115031 in_seq=0 out_seq=0 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f18357fa640 1 -- 172.21.15.116:0/3100144232 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae3cf523e0 con 0x55ae3cfaab80 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f18357fa640 10 client.18540 handle_command_reply: tid=0 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 client.18540 shutdown 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 msgr2=0x55ae3cf97d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f183c00bad0 tx=0x7f183c00bf90 comp rx=0 tx=0).stop 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 shutdown_connections 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3cfaab80 0x55ae3cfad000 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf98270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf97d30 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 --2- 172.21.15.116:0/3100144232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cf977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 >> 172.21.15.116:0/3100144232 conn(0x55ae3cf8ba00 msgr2=0x55ae3cf8d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f18357fa640 0 client.18540 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 shutdown_connections 2024-03-19T02:38:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 1 -- 172.21.15.116:0/3100144232 wait complete. 2024-03-19T02:38:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.510+0000 7f1856c17740 20 client.18540 trim_cache size 0 max 16384 2024-03-19T02:38:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 Processor -- start 2024-03-19T02:38:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 -- start start 2024-03-19T02:38:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 0x55ae3cfa6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cfb2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1ac920 con 0x55ae3cf912d0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1ae460 con 0x55ae3cf91bf0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1856c17740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1acff0 con 0x55ae3cf9a7d0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56842/0 (socket says 172.21.15.116:56842) 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 -- 172.21.15.116:0/2157400453 learned_addr learned my addr 172.21.15.116:0/2157400453 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f18515cb640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 0x55ae3cfa6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f18525cd640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cfb2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 -- 172.21.15.116:0/2157400453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 msgr2=0x55ae3cfb2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cfb2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 -- 172.21.15.116:0/2157400453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 msgr2=0x55ae3cfa6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 0x55ae3cfa6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f1851dcc640 1 -- 172.21.15.116:0/2157400453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae3cf61880 con 0x55ae3cf912d0 2024-03-19T02:38:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.526+0000 7f18515cb640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 0x55ae3cfa6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1851dcc640 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f184802c720 tx=0x7f184808fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c100a72a1fbe269a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f18377fe640 1 -- 172.21.15.116:0/2157400453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184808a020 con 0x55ae3cf912d0 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f18377fe640 1 -- 172.21.15.116:0/2157400453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18480bf070 con 0x55ae3cf912d0 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f18377fe640 1 -- 172.21.15.116:0/2157400453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18480bb150 con 0x55ae3cf912d0 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1856c17740 1 -- 172.21.15.116:0/2157400453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 msgr2=0x55ae3cf99f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1856c17740 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f184802c720 tx=0x7f184808fb30 comp rx=0 tx=0).stop 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1856c17740 1 -- 172.21.15.116:0/2157400453 shutdown_connections 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1856c17740 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cfb2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.527+0000 7f1856c17740 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf91bf0 0x55ae3cfa6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 --2- 172.21.15.116:0/2157400453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf912d0 0x55ae3cf99f20 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- 172.21.15.116:0/2157400453 >> 172.21.15.116:0/2157400453 conn(0x55ae3cf8ba00 msgr2=0x55ae3cf8d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- 172.21.15.116:0/2157400453 shutdown_connections 2024-03-19T02:38:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- 172.21.15.116:0/2157400453 wait complete. 2024-03-19T02:38:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- start start 2024-03-19T02:38:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 Processor -- start 2024-03-19T02:38:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1add90 con 0x55ae3cf9a7d0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1ad6c0 con 0x55ae3cf912d0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f184c1aa920 con 0x55ae3cf91bf0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1856c17740 10 client.0 _ll_register_callbacks cb 0x7f183c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f18525cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f1851dcc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39866/0 (socket says 172.21.15.116:39866) 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 -- 172.21.15.116:0/1893146424 learned_addr learned my addr 172.21.15.116:0/1893146424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.528+0000 7f18525cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56850/0 (socket says 172.21.15.116:56850) 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18515cb640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 -- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 msgr2=0x55ae3cf9f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 -- 172.21.15.116:0/1893146424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 msgr2=0x55ae3cf9fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 -- 172.21.15.116:0/1893146424 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae3cf90a70 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18515cb640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1851dcc640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f184802c170 tx=0x7f184802c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903d1d30779ca104 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18357fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184808a070 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18480bf070 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f1848003d20 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18480bd550 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1856c17740 10 client.28334 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.529+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ae3d011b30 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18480bf070 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f18480bc2a0 con 0x55ae3cf912d0 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 10 client.28334 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 10 client.28334 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 20 client.28334 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3d0121b0 0x55ae3d0145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 4 client.28334 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae3cf019f0 con 0x55ae3d0121b0 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f18515cb640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3d0121b0 0x55ae3d0145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f18357fa640 10 client.28334 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.530+0000 7f18515cb640 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3d0121b0 0x55ae3d0145a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b8a66ed50ad5d32a server_cookie=13d74b8ce7e69c1 in_seq=0 out_seq=0 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f18357fa640 1 -- 172.21.15.116:0/1893146424 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae3cf019f0 con 0x55ae3d0121b0 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f18357fa640 10 client.28334 handle_command_reply: tid=0 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 client.28334 shutdown 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 msgr2=0x55ae3cfa2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f184802c170 tx=0x7f184802c680 comp rx=0 tx=0).stop 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 shutdown_connections 2024-03-19T02:38:00.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ae3d0121b0 0x55ae3d0145a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae3cf9a7d0 0x55ae3cf9fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f18357fa640 0 client.28334 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ae3cf91bf0 0x55ae3cf9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 --2- 172.21.15.116:0/1893146424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae3cf912d0 0x55ae3cfa2b20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 >> 172.21.15.116:0/1893146424 conn(0x55ae3cf8ba00 msgr2=0x55ae3cf8ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 shutdown_connections 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 1 -- 172.21.15.116:0/1893146424 wait complete. 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1856c17740 20 client.28334 trim_cache size 0 max 16384 2024-03-19T02:38:00.538 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 msgr2=0x7f18380a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 0x7f18380a4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f184400a7b0 tx=0x7f1844020000 comp rx=0 tx=0).stop 2024-03-19T02:38:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 msgr2=0x7f184c19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 0x7f184c19ba50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1848036040 tx=0x7f184802cd20 comp rx=0 tx=0).stop 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 shutdown_connections 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18380a29f0 0x7f18380a4eb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f184c105690 0x7f184c195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f184c101330 0x7f184c19bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 --2- 172.21.15.116:0/435938743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f184c100970 0x7f184c19ba50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.531+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 >> 172.21.15.116:0/435938743 conn(0x7f184c077580 msgr2=0x7f184c0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.532+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 shutdown_connections 2024-03-19T02:38:00.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.532+0000 7f1854030640 1 -- 172.21.15.116:0/435938743 wait complete. 2024-03-19T02:38:00.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:38:00.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- 172.21.15.116:0/950052498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a0101060 msgr2=0x7f75a010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 --2- 172.21.15.116:0/950052498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a0101060 0x7f75a010d570 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f759800dfa0 tx=0x7f759802fe50 comp rx=0 tx=0).stop 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- 172.21.15.116:0/950052498 shutdown_connections 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 --2- 172.21.15.116:0/950052498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a0101060 0x7f75a010d570 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 --2- 172.21.15.116:0/950052498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a01006a0 0x7f75a0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 --2- 172.21.15.116:0/950052498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 0x7f75a0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- 172.21.15.116:0/950052498 >> 172.21.15.116:0/950052498 conn(0x7f75a00778d0 msgr2=0x7f75a0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- 172.21.15.116:0/950052498 shutdown_connections 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- 172.21.15.116:0/950052498 wait complete. 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 Processor -- start 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.666+0000 7f75a6259640 1 -- start start 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 0x7f75a019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 0x7f75a0195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010f800 con 0x7f75a0101060 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010f680 con 0x7f75a01066b0 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010f980 con 0x7f75a01006a0 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56858/0 (socket says 172.21.15.116:56858) 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 -- 172.21.15.116:0/3262900245 learned_addr learned my addr 172.21.15.116:0/3262900245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759f7fe640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 0x7f75a019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759ffff640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 0x7f75a0195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 -- 172.21.15.116:0/3262900245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 msgr2=0x7f75a019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 0x7f75a019b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 -- 172.21.15.116:0/3262900245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 msgr2=0x7f75a0195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 0x7f75a0195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 -- 172.21.15.116:0/3262900245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a01962a0 con 0x7f75a0101060 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759f7fe640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 0x7f75a019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759effd640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f759cff9640 1 -- 172.21.15.116:0/3262900245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759400ccc0 con 0x7f75a0101060 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.667+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a0196590 con 0x7f75a0101060 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.668+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a01a27e0 con 0x7f75a0101060 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.668+0000 7f759cff9640 1 -- 172.21.15.116:0/3262900245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75940108e0 con 0x7f75a0101060 2024-03-19T02:38:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.668+0000 7f759cff9640 1 -- 172.21.15.116:0/3262900245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75940187c0 con 0x7f75a0101060 2024-03-19T02:38:00.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.670+0000 7f75a8e42740 1 Processor -- start 2024-03-19T02:38:00.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 -- start start 2024-03-19T02:38:00.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759cff9640 1 -- 172.21.15.116:0/3262900245 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7594018960 con 0x7f75a0101060 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e5763526b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e5762b54b0 con 0x55e5763522d0 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e57625dd70 con 0x55e576352bf0 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f75a8e42740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e5762b1550 con 0x55e57635b7d0 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e5763526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39876/0 (socket says 172.21.15.116:39876) 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e5763526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56868/0 (socket says 172.21.15.116:56868) 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 -- 172.21.15.116:0/2721300360 learned_addr learned my addr 172.21.15.116:0/2721300360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759effd640 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 -- 172.21.15.116:0/2721300360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 msgr2=0x55e57635dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 -- 172.21.15.116:0/2721300360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 msgr2=0x55e5763526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e5763526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 -- 172.21.15.116:0/2721300360 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e576280210 con 0x55e576352bf0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759f7fe640 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f75900114f0 tx=0x7f75900119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2005acbc2908c67d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.671+0000 7f759cff9640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 0x7f75880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f759ffff640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 0x7f75880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f7585ffb640 1 -- 172.21.15.116:0/2721300360 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590004020 con 0x55e576352bf0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f759cff9640 1 -- 172.21.15.116:0/3262900245 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f75940cf620 con 0x7f75a0101060 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f7585ffb640 1 -- 172.21.15.116:0/2721300360 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75900041c0 con 0x55e576352bf0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f7585ffb640 1 -- 172.21.15.116:0/2721300360 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759000f8f0 con 0x55e576352bf0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f759ffff640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 0x7f75880a4e30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f759803a040 tx=0x7f7598032040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 -- 172.21.15.116:0/2721300360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 msgr2=0x55e57635b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f75900114f0 tx=0x7f75900119b0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 -- 172.21.15.116:0/2721300360 shutdown_connections 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635b290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 --2- 172.21.15.116:0/2721300360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e5763526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.672+0000 7f75a8e42740 1 -- 172.21.15.116:0/2721300360 >> 172.21.15.116:0/2721300360 conn(0x55e57634ca00 msgr2=0x55e57634cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- 172.21.15.116:0/2721300360 shutdown_connections 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- 172.21.15.116:0/2721300360 wait complete. 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- start start 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 Processor -- start 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576358d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e5763592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e57635e5c0 con 0x55e5763522d0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e576271310 con 0x55e576352bf0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e57635e740 con 0x55e57635b7d0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f75a8e42740 10 client.0 _ll_register_callbacks cb 0x55e5762652d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.673+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56876/0 (socket says 172.21.15.116:56876) 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576358d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 -- 172.21.15.116:0/1613204095 learned_addr learned my addr 172.21.15.116:0/1613204095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759ffff640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e5763592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 -- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 msgr2=0x55e5763592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e5763592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 -- 172.21.15.116:0/1613204095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 msgr2=0x55e576358d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576358d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 -- 172.21.15.116:0/1613204095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e575fc93f0 con 0x55e5763522d0 2024-03-19T02:38:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759effd640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576358d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f759f7fe640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7590009230 tx=0x7f7590005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f8da6a8fb71155 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f757effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590028020 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f7590004370 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75900041c0 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7590033050 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f75a8e42740 10 client.18558 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.674+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e5760937d0 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75900041c0 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7590004a90 con 0x55e5763522d0 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 10 client.18558 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 10 client.18558 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 20 client.18558 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e57636bb80 0x55e57636df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 4 client.18558 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5763133e0 con 0x55e57636bb80 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f759effd640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e57636bb80 0x55e57636df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f759effd640 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e57636bb80 0x55e57636df70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3f2205eba8de3e47 server_cookie=49ed4ffe4e896bd9 in_seq=0 out_seq=0 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.675+0000 7f757effd640 10 client.18558 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.676+0000 7f757effd640 1 -- 172.21.15.116:0/1613204095 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e5763133e0 con 0x55e57636bb80 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.676+0000 7f757effd640 10 client.18558 handle_command_reply: tid=0 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.676+0000 7f75a8e42740 1 client.18558 shutdown 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 msgr2=0x55e576358830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7590009230 tx=0x7f7590005e30 comp rx=0 tx=0).stop 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 shutdown_connections 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e5763592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576358d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e57636bb80 0x55e57636df70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 --2- 172.21.15.116:0/1613204095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e576358830 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f757effd640 0 client.18558 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 >> 172.21.15.116:0/1613204095 conn(0x55e57634ca00 msgr2=0x55e57634e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 shutdown_connections 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 1 -- 172.21.15.116:0/1613204095 wait complete. 2024-03-19T02:38:00.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.677+0000 7f75a8e42740 20 client.18558 trim_cache size 0 max 16384 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.691+0000 7f75a8e42740 1 Processor -- start 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.691+0000 7f75a8e42740 1 -- start start 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 0x55e576357990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576367360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01aca80 con 0x55e57635b7d0 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01ae5c0 con 0x55e576352bf0 2024-03-19T02:38:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75a8e42740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01ad150 con 0x55e5763522d0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56888/0 (socket says 172.21.15.116:56888) 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 -- 172.21.15.116:0/838264843 learned_addr learned my addr 172.21.15.116:0/838264843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759f7fe640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 0x55e576357990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759effd640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576367360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 -- 172.21.15.116:0/838264843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 msgr2=0x55e576357990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 0x55e576357990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 -- 172.21.15.116:0/838264843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 msgr2=0x55e576367360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576367360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 -- 172.21.15.116:0/838264843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e576352a20 con 0x55e57635b7d0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759effd640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576367360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759f7fe640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 0x55e576357990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f759ffff640 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f759800dfd0 tx=0x7f7598043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461a76be4e6adb6d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75857fa640 1 -- 172.21.15.116:0/838264843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598047070 con 0x55e57635b7d0 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.692+0000 7f75857fa640 1 -- 172.21.15.116:0/838264843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7598030e50 con 0x55e57635b7d0 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75857fa640 1 -- 172.21.15.116:0/838264843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759800b940 con 0x55e57635b7d0 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 -- 172.21.15.116:0/838264843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 msgr2=0x55e5763590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f759800dfd0 tx=0x7f7598043000 comp rx=0 tx=0).stop 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 -- 172.21.15.116:0/838264843 shutdown_connections 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e57635b7d0 0x55e5763590d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e576367360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 --2- 172.21.15.116:0/838264843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5763522d0 0x55e576357990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 -- 172.21.15.116:0/838264843 >> 172.21.15.116:0/838264843 conn(0x55e57634ca00 msgr2=0x55e57634b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 -- 172.21.15.116:0/838264843 shutdown_connections 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.693+0000 7f75a8e42740 1 -- 172.21.15.116:0/838264843 wait complete. 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 -- start start 2024-03-19T02:38:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 Processor -- start 2024-03-19T02:38:00.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01adef0 con 0x55e5763522d0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01ad820 con 0x55e576352bf0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75a01aaa80 con 0x55e57635b7d0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f75a8e42740 10 client.0 _ll_register_callbacks cb 0x7f7590005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56902/0 (socket says 172.21.15.116:56902) 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 -- 172.21.15.116:0/4202442450 learned_addr learned my addr 172.21.15.116:0/4202442450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759ffff640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759effd640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 -- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 msgr2=0x55e57635f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 -- 172.21.15.116:0/4202442450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 msgr2=0x55e57635ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.694+0000 7f759f7fe640 1 -- 172.21.15.116:0/4202442450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5763d0060 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f759f7fe640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f7590009de0 tx=0x7f7590009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a140a519b12e7877 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590028040 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f759000fe00 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7598030e50 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f759000f970 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 10 client.18564 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e5763d1320 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7598030e50 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7590024600 con 0x55e5763522d0 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 10 client.18564 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 10 client.18564 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:38:00.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 20 client.18564 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e5763d1ac0 0x55e5763d3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 4 client.18564 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.695+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5762c29f0 con 0x55e5763d1ac0 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f759effd640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e5763d1ac0 0x55e5763d3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f757effd640 10 client.18564 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f759effd640 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e5763d1ac0 0x55e5763d3ee0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=edd7dde236f48f6e server_cookie=3c4e11cd61aed41 in_seq=0 out_seq=0 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f757effd640 1 -- 172.21.15.116:0/4202442450 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5762c29f0 con 0x55e5763d1ac0 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f757effd640 10 client.18564 handle_command_reply: tid=0 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 client.18564 shutdown 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 msgr2=0x55e57635e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f7590009de0 tx=0x7f7590009ea0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 shutdown_connections 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e57635b7d0 0x55e57635f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e576352bf0 0x55e57635ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e5763d1ac0 0x55e5763d3ee0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 --2- 172.21.15.116:0/4202442450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5763522d0 0x55e57635e950 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f757effd640 0 client.18564 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.696+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 >> 172.21.15.116:0/4202442450 conn(0x55e57634ca00 msgr2=0x55e57634dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 shutdown_connections 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a8e42740 1 -- 172.21.15.116:0/4202442450 wait complete. 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a8e42740 20 client.18564 trim_cache size 0 max 16384 2024-03-19T02:38:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 msgr2=0x7f75880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 0x7f75880a4e30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f759803a040 tx=0x7f7598032040 comp rx=0 tx=0).stop 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 msgr2=0x7f75a019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f759400e9f0 tx=0x7f759400edf0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 shutdown_connections 2024-03-19T02:38:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f75880a2970 0x7f75880a4e30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75a01066b0 0x7f75a0195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75a0101060 0x7f75a019be60 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 --2- 172.21.15.116:0/3262900245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75a01006a0 0x7f75a019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.697+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 >> 172.21.15.116:0/3262900245 conn(0x7f75a00778d0 msgr2=0x7f75a0102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.698+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 shutdown_connections 2024-03-19T02:38:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.698+0000 7f75a6259640 1 -- 172.21.15.116:0/3262900245 wait complete. 2024-03-19T02:38:00.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:38:00.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 330712 3.13 MB/sec execute 443 sec latency 31.415 ms 2024-03-19T02:38:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.828+0000 7f9dee905640 1 -- 172.21.15.116:0/1446676211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de81021b0 msgr2=0x7f9de810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.828+0000 7f9dee905640 1 --2- 172.21.15.116:0/1446676211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de81021b0 0x7f9de810e6c0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009cd0 tx=0x7f9de002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 -- 172.21.15.116:0/1446676211 shutdown_connections 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 --2- 172.21.15.116:0/1446676211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de81021b0 0x7f9de810e6c0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 --2- 172.21.15.116:0/1446676211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81017f0 0x7f9de8101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 --2- 172.21.15.116:0/1446676211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de8107800 0x7f9de8107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 -- 172.21.15.116:0/1446676211 >> 172.21.15.116:0/1446676211 conn(0x7f9de80fd2c0 msgr2=0x7f9de80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 -- 172.21.15.116:0/1446676211 shutdown_connections 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 -- 172.21.15.116:0/1446676211 wait complete. 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 Processor -- start 2024-03-19T02:38:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.829+0000 7f9dee905640 1 -- start start 2024-03-19T02:38:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de81017f0 0x7f9de8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 0x7f9de81984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8111a80 con 0x7f9de8107800 2024-03-19T02:38:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8111900 con 0x7f9de81021b0 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8111c00 con 0x7f9de81017f0 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56914/0 (socket says 172.21.15.116:56914) 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de81017f0 0x7f9de8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 -- 172.21.15.116:0/1506837070 learned_addr learned my addr 172.21.15.116:0/1506837070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de77fe640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 0x7f9de81984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 -- 172.21.15.116:0/1506837070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de81017f0 msgr2=0x7f9de8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de81017f0 0x7f9de8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 -- 172.21.15.116:0/1506837070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 msgr2=0x7f9de81984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 0x7f9de81984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 -- 172.21.15.116:0/1506837070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de819cf70 con 0x7f9de8107800 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de77fe640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 0x7f9de81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9decea2640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9de0004af0 tx=0x7f9de0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de57fa640 1 -- 172.21.15.116:0/1506837070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0039040 con 0x7f9de8107800 2024-03-19T02:38:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de57fa640 1 -- 172.21.15.116:0/1506837070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9de0044070 con 0x7f9de8107800 2024-03-19T02:38:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de819d200 con 0x7f9de8107800 2024-03-19T02:38:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.830+0000 7f9de57fa640 1 -- 172.21.15.116:0/1506837070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de00042a0 con 0x7f9de8107800 2024-03-19T02:38:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.831+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de81a4af0 con 0x7f9de8107800 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.833+0000 7f9df15c5740 1 Processor -- start 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.833+0000 7f9de57fa640 1 -- 172.21.15.116:0/1506837070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9de0003970 con 0x7f9de8107800 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 -- start start 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a00f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a01cbe0 0x55b44a01efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b449f724b0 con 0x55b44a00f8a0 2024-03-19T02:38:00.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b449f1ad70 con 0x55b44a01cbe0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9df15c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b449f6e550 con 0x55b44a00ef80 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a00f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56920/0 (socket says 172.21.15.116:56920) 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 -- 172.21.15.116:0/3143877413 learned_addr learned my addr 172.21.15.116:0/3143877413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a01cbe0 0x55b44a01efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de57fa640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 0x7f9dd00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 -- 172.21.15.116:0/3143877413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 msgr2=0x55b44a00f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a00f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 -- 172.21.15.116:0/3143877413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a01cbe0 msgr2=0x55b44a01efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a01cbe0 0x55b44a01efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 -- 172.21.15.116:0/3143877413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b449f3d210 con 0x55b44a00f8a0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9decea2640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 0x7f9dd00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de57fa640 1 -- 172.21.15.116:0/1506837070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f9de00f12e0 con 0x7f9de8107800 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.834+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009c70 tx=0x7f9dd8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350a350a0e182c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9dce7fc640 1 -- 172.21.15.116:0/3143877413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd801e070 con 0x55b44a00f8a0 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9dce7fc640 1 -- 172.21.15.116:0/3143877413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dd8009e20 con 0x55b44a00f8a0 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9dce7fc640 1 -- 172.21.15.116:0/3143877413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8019690 con 0x55b44a00f8a0 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 -- 172.21.15.116:0/3143877413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 msgr2=0x55b44a01c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8009c70 tx=0x7f9dd8011810 comp rx=0 tx=0).stop 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9decea2640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 0x7f9dd00a4e30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9de00ee810 tx=0x7f9de00b2b00 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 -- 172.21.15.116:0/3143877413 shutdown_connections 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a01cbe0 0x55b44a01efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00f8a0 0x55b44a01c6a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3143877413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a00f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 -- 172.21.15.116:0/3143877413 >> 172.21.15.116:0/3143877413 conn(0x55b44a009a00 msgr2=0x55b44a009de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 -- 172.21.15.116:0/3143877413 shutdown_connections 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.835+0000 7f9df15c5740 1 -- 172.21.15.116:0/3143877413 wait complete. 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 -- start start 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 Processor -- start 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00ef80 0x55b44a019b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00f8a0 0x55b44a01a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b44a01fa20 con 0x55b44a01cbe0 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b449f2e310 con 0x55b44a00ef80 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b44a01fba0 con 0x55b44a00f8a0 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9df15c5740 10 client.0 _ll_register_callbacks cb 0x55b449f222d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56934/0 (socket says 172.21.15.116:56934) 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9de77fe640 1 -- 172.21.15.116:0/3321534369 learned_addr learned my addr 172.21.15.116:0/3321534369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00f8a0 0x55b44a01a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.836+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00ef80 0x55b44a019b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 -- 172.21.15.116:0/3321534369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00f8a0 msgr2=0x55b44a01a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00f8a0 0x55b44a01a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 -- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00ef80 msgr2=0x55b44a019b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00ef80 0x55b44a019b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 -- 172.21.15.116:0/3321534369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b449c863f0 con 0x55b44a01cbe0 2024-03-19T02:38:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00f4f0 tx=0x7f9ddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e947eb0b3cb417 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00fe70 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f9ddc011280 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dd8009e20 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ddc004050 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9df15c5740 10 client.18576 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b449d507d0 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dd8009e20 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9ddc022430 con 0x55b44a01cbe0 2024-03-19T02:38:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9df15c5740 10 client.18576 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.837+0000 7f9df15c5740 10 client.18576 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9df15c5740 20 client.18576 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a02d0e0 0x55b44a02f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9df15c5740 4 client.18576 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b449fd03e0 con 0x55b44a02d0e0 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a02d0e0 0x55b44a02f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9dc77fe640 10 client.18576 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.838+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a02d0e0 0x55b44a02f540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df03e14e736dc170 server_cookie=b07ffae18262abeb in_seq=0 out_seq=0 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3321534369 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b449fd03e0 con 0x55b44a02d0e0 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9dc77fe640 10 client.18576 handle_command_reply: tid=0 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 client.18576 shutdown 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 msgr2=0x55b44a01a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00f4f0 tx=0x7f9ddc0046c0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 shutdown_connections 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a02d0e0 0x55b44a02f540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a01a600 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00f8a0 0x55b44a01a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3321534369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00ef80 0x55b44a019b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9dc77fe640 0 client.18576 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 >> 172.21.15.116:0/3321534369 conn(0x55b44a009a00 msgr2=0x55b44a00b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 shutdown_connections 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.839+0000 7f9df15c5740 1 -- 172.21.15.116:0/3321534369 wait complete. 2024-03-19T02:38:00.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.840+0000 7f9df15c5740 20 client.18576 trim_cache size 0 max 16384 2024-03-19T02:38:00.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 Processor -- start 2024-03-19T02:38:00.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 -- start start 2024-03-19T02:38:00.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a028890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a01cbe0 0x55b44a008a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81aca80 con 0x55b44a00ef80 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81ae5c0 con 0x55b44a00f8a0 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9df15c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81ad150 con 0x55b44a01cbe0 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a028890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56940/0 (socket says 172.21.15.116:56940) 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.854+0000 7f9de77fe640 1 -- 172.21.15.116:0/3251436633 learned_addr learned my addr 172.21.15.116:0/3251436633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a01cbe0 0x55b44a008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 -- 172.21.15.116:0/3251436633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a01cbe0 msgr2=0x55b44a008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a01cbe0 0x55b44a008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 -- 172.21.15.116:0/3251436633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 msgr2=0x55b44a028890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a028890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 -- 172.21.15.116:0/3251436633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b44a009830 con 0x55b44a00ef80 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc011e80 tx=0x7f9ddc00f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fa7ef607634fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9dcdffb640 1 -- 172.21.15.116:0/3251436633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00fe40 con 0x55b44a00ef80 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9dcdffb640 1 -- 172.21.15.116:0/3251436633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc011280 con 0x55b44a00ef80 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.855+0000 7f9dcdffb640 1 -- 172.21.15.116:0/3251436633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc003850 con 0x55b44a00ef80 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- 172.21.15.116:0/3251436633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 msgr2=0x55b44a01c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc011e80 tx=0x7f9ddc00f400 comp rx=0 tx=0).stop 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- 172.21.15.116:0/3251436633 shutdown_connections 2024-03-19T02:38:00.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a01cbe0 0x55b44a008a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a028890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3251436633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a00ef80 0x55b44a01c2b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- 172.21.15.116:0/3251436633 >> 172.21.15.116:0/3251436633 conn(0x55b44a009a00 msgr2=0x55b44a008310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- 172.21.15.116:0/3251436633 shutdown_connections 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- 172.21.15.116:0/3251436633 wait complete. 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 -- start start 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 Processor -- start 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.856+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a0209a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a020ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81adef0 con 0x55b44a01cbe0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81ad820 con 0x55b44a00f8a0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81aaa80 con 0x55b44a00ef80 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 10 client.0 _ll_register_callbacks cb 0x7f9ddc0118f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a0209a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56950/0 (socket says 172.21.15.116:56950) 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 -- 172.21.15.116:0/3451940679 learned_addr learned my addr 172.21.15.116:0/3451940679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a020ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 -- 172.21.15.116:0/3451940679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 msgr2=0x55b44a0209a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a0209a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 -- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 msgr2=0x55b44a020ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a020ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 -- 172.21.15.116:0/3451940679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b44a01efd0 con 0x55b44a01cbe0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a020ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9de77fe640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00f4f0 tx=0x7f9ddc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4698d8569f1a376 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9dc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc00fe40 con 0x55b44a01cbe0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f9ddc005470 con 0x55b44a01cbe0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ddc011280 con 0x55b44a01cbe0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9ddc003820 con 0x55b44a01cbe0 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 10 client.18582 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.857+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b449fdf880 con 0x55b44a01cbe0 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc011280 con 0x55b44a01cbe0 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9ddc03f070 con 0x55b44a01cbe0 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 10 client.18582 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 10 client.18582 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 20 client.18582 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a093b10 0x55b44a095f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 4 client.18582 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b449f7f9f0 con 0x55b44a093b10 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a093b10 0x55b44a095f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9dc77fe640 10 client.18582 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.858+0000 7f9de7fff640 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a093b10 0x55b44a095f30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b231c70cfb67657 server_cookie=4879a004300b9e18 in_seq=0 out_seq=0 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dc77fe640 1 -- 172.21.15.116:0/3451940679 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b449f7f9f0 con 0x55b44a093b10 2024-03-19T02:38:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dc77fe640 10 client.18582 handle_command_reply: tid=0 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 client.18582 shutdown 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 msgr2=0x55b44a021420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00f4f0 tx=0x7f9ddc000bc0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 shutdown_connections 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b44a093b10 0x55b44a095f30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b44a01cbe0 0x55b44a021420 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b44a00f8a0 0x55b44a020ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dc77fe640 0 client.18582 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 --2- 172.21.15.116:0/3451940679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b44a00ef80 0x55b44a0209a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 >> 172.21.15.116:0/3451940679 conn(0x55b44a009a00 msgr2=0x55b44a008ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 shutdown_connections 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 1 -- 172.21.15.116:0/3451940679 wait complete. 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9df15c5740 20 client.18582 trim_cache size 0 max 16384 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 msgr2=0x7f9dd00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 0x7f9dd00a4e30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9de00ee810 tx=0x7f9de00b2b00 comp rx=0 tx=0).stop 2024-03-19T02:38:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 msgr2=0x7f9de819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.859+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9de0004af0 tx=0x7f9de0035ce0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 shutdown_connections 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9dd00a2970 0x7f9dd00a4e30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9de8107800 0x7f9de819c840 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9de81021b0 0x7f9de81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 --2- 172.21.15.116:0/1506837070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9de81017f0 0x7f9de8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 >> 172.21.15.116:0/1506837070 conn(0x7f9de80fd2c0 msgr2=0x7f9de80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 shutdown_connections 2024-03-19T02:38:00.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.860+0000 7f9dee905640 1 -- 172.21.15.116:0/1506837070 wait complete. 2024-03-19T02:38:00.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 -- 172.21.15.116:0/2471977675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c1077a0 msgr2=0x7f768c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 --2- 172.21.15.116:0/2471977675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c1077a0 0x7f768c107b80 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f767c009930 tx=0x7f767c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 -- 172.21.15.116:0/2471977675 shutdown_connections 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 --2- 172.21.15.116:0/2471977675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c100680 0x7f768c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 --2- 172.21.15.116:0/2471977675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c0ffcc0 0x7f768c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 --2- 172.21.15.116:0/2471977675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c1077a0 0x7f768c107b80 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 -- 172.21.15.116:0/2471977675 >> 172.21.15.116:0/2471977675 conn(0x7f768c076c50 msgr2=0x7f768c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 -- 172.21.15.116:0/2471977675 shutdown_connections 2024-03-19T02:38:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.996+0000 7f7691aea640 1 -- 172.21.15.116:0/2471977675 wait complete. 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 Processor -- start 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 -- start start 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 0x7f768c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c19caa0 0x7f768c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f768c111a60 con 0x7f768c19caa0 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1118e0 con 0x7f768c0ffcc0 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f7691aea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f768c111be0 con 0x7f768c100680 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39986/0 (socket says 172.21.15.116:39986) 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f768b7fe640 1 -- 172.21.15.116:0/2289418345 learned_addr learned my addr 172.21.15.116:0/2289418345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f768affd640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 0x7f768c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.997+0000 7f768bfff640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c19caa0 0x7f768c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 -- 172.21.15.116:0/2289418345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 msgr2=0x7f768c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 0x7f768c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 -- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c19caa0 msgr2=0x7f768c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c19caa0 0x7f768c19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 -- 172.21.15.116:0/2289418345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768c19d590 con 0x7f768c0ffcc0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768affd640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 0x7f768c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f767c009230 tx=0x7f767c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f7688ff9640 1 -- 172.21.15.116:0/2289418345 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c043020 con 0x7f768c0ffcc0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768c19d820 con 0x7f768c0ffcc0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f7688ff9640 1 -- 172.21.15.116:0/2289418345 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f767c02ca70 con 0x7f768c0ffcc0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f768c1010c0 con 0x7f768c0ffcc0 2024-03-19T02:38:01.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:00.998+0000 7f7688ff9640 1 -- 172.21.15.116:0/2289418345 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c0038e0 con 0x7f768c0ffcc0 2024-03-19T02:38:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f7688ff9640 1 -- 172.21.15.116:0/2289418345 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f767c046020 con 0x7f768c0ffcc0 2024-03-19T02:38:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 Processor -- start 2024-03-19T02:38:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 -- start start 2024-03-19T02:38:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7a2d0 0x55b49cb7a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7abf0 0x55b49cb83290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.001+0000 7f76947b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b49cadd4b0 con 0x55b49cb837d0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f76947b5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b49ca85d70 con 0x55b49cb7abf0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f76947b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b49cad9550 con 0x55b49cb7a2d0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56964/0 (socket says 172.21.15.116:56964) 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7abf0 0x55b49cb83290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7a2d0 0x55b49cb7a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 -- 172.21.15.116:0/180020590 learned_addr learned my addr 172.21.15.116:0/180020590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f7688ff9640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 0x7f76740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 -- 172.21.15.116:0/180020590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7a2d0 msgr2=0x55b49cb7a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7a2d0 0x55b49cb7a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 -- 172.21.15.116:0/180020590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7abf0 msgr2=0x55b49cb83290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7abf0 0x55b49cb83290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 -- 172.21.15.116:0/180020590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b49caa8210 con 0x55b49cb837d0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768bfff640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 0x7f76740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f7688ff9640 1 -- 172.21.15.116:0/2289418345 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f767c0f1070 con 0x7f768c0ffcc0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f768affd640 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f7680010f30 tx=0x7f7680005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3130c6e82e3ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.002+0000 7f7671ffb640 1 -- 172.21.15.116:0/180020590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768000e670 con 0x55b49cb837d0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f7671ffb640 1 -- 172.21.15.116:0/180020590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f768000ecc0 con 0x55b49cb837d0 2024-03-19T02:38:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f7671ffb640 1 -- 172.21.15.116:0/180020590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768000e670 con 0x55b49cb837d0 2024-03-19T02:38:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f76947b5740 1 -- 172.21.15.116:0/180020590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 msgr2=0x55b49cb85bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f768bfff640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 0x7f76740a4eb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7684005c40 tx=0x7f7684004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f76947b5740 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f7680010f30 tx=0x7f7680005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f76947b5740 1 -- 172.21.15.116:0/180020590 shutdown_connections 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f76947b5740 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb837d0 0x55b49cb85bc0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.003+0000 7f76947b5740 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7abf0 0x55b49cb83290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.004+0000 7f76947b5740 1 --2- 172.21.15.116:0/180020590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7a2d0 0x55b49cb7a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.004+0000 7f76947b5740 1 -- 172.21.15.116:0/180020590 >> 172.21.15.116:0/180020590 conn(0x55b49cb74a00 msgr2=0x55b49cb74de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.004+0000 7f76947b5740 1 -- 172.21.15.116:0/180020590 shutdown_connections 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.004+0000 7f76947b5740 1 -- 172.21.15.116:0/180020590 wait complete. 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.004+0000 7f76947b5740 1 -- start start 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 Processor -- start 2024-03-19T02:38:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb80d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b49cb865c0 con 0x55b49cb7abf0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b49ca99310 con 0x55b49cb7a2d0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b49cb86740 con 0x55b49cb81250 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f76947b5740 10 client.0 _ll_register_callbacks cb 0x55b49ca8d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb80d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40000/0 (socket says 172.21.15.116:40000) 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 -- 172.21.15.116:0/2952914678 learned_addr learned my addr 172.21.15.116:0/2952914678 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 -- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 msgr2=0x55b49cb8fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 -- 172.21.15.116:0/2952914678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 msgr2=0x55b49cb80d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.005+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb80d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f768b7fe640 1 -- 172.21.15.116:0/2952914678 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b49c7f13f0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f768b7fe640 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f767c0b9800 tx=0x7f767c009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a41165a057b68e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f766affd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c0bcd00 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f767c0c56b0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f768000ecc0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f767c0bdaa0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f76947b5740 10 client.28382 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.006+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b49c8bb7d0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f768000ecc0 con 0x55b49cb7a2d0 2024-03-19T02:38:01.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f767c0c1100 con 0x55b49cb7a2d0 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 10 client.28382 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 10 client.28382 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 20 client.28382 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cb964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 4 client.28382 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b49cb3b3e0 con 0x55b49cb93fe0 2024-03-19T02:38:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.007+0000 7f768affd640 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cb964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.012 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:00 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2432796908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:01.012 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:00 smithi116 ceph-mon[32263]: pgmap v317: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 120 op/s 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.008+0000 7f766affd640 10 client.28382 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.008+0000 7f768affd640 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cb964a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db185af04b96cc3f server_cookie=51d510adaf20f530 in_seq=0 out_seq=0 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.009+0000 7f766affd640 1 -- 172.21.15.116:0/2952914678 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b49cb3b3e0 con 0x55b49cb93fe0 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.009+0000 7f766affd640 10 client.28382 handle_command_reply: tid=0 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.009+0000 7f76947b5740 1 client.28382 shutdown 2024-03-19T02:38:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.009+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 msgr2=0x55b49cb807d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.009+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f767c0b9800 tx=0x7f767c009680 comp rx=0 tx=0).stop 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 shutdown_connections 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cb964a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f766affd640 0 client.28382 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb80d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 --2- 172.21.15.116:0/2952914678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb807d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 >> 172.21.15.116:0/2952914678 conn(0x55b49cb74a00 msgr2=0x55b49cb76750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 shutdown_connections 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 1 -- 172.21.15.116:0/2952914678 wait complete. 2024-03-19T02:38:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.010+0000 7f76947b5740 20 client.28382 trim_cache size 0 max 16384 2024-03-19T02:38:01.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 Processor -- start 2024-03-19T02:38:01.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 -- start start 2024-03-19T02:38:01.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb766c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1aced0 con 0x55b49cb7abf0 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1aea10 con 0x55b49cb7a2d0 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f76947b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1ad5a0 con 0x55b49cb81250 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb766c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56978/0 (socket says 172.21.15.116:56978) 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 -- 172.21.15.116:0/3425123978 learned_addr learned my addr 172.21.15.116:0/3425123978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 -- 172.21.15.116:0/3425123978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 msgr2=0x55b49cb8b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 -- 172.21.15.116:0/3425123978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 msgr2=0x55b49cb766c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb766c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768affd640 1 -- 172.21.15.116:0/3425123978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b49cb4a880 con 0x55b49cb7abf0 2024-03-19T02:38:01.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.030+0000 7f768b7fe640 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb766c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.031+0000 7f768affd640 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7680012010 tx=0x7f7680005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b4c8d9470cda8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.031+0000 7f76717fa640 1 -- 172.21.15.116:0/3425123978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768002f020 con 0x55b49cb7abf0 2024-03-19T02:38:01.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.031+0000 7f76717fa640 1 -- 172.21.15.116:0/3425123978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f768002b7f0 con 0x55b49cb7abf0 2024-03-19T02:38:01.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.031+0000 7f76717fa640 1 -- 172.21.15.116:0/3425123978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680018c40 con 0x55b49cb7abf0 2024-03-19T02:38:01.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- 172.21.15.116:0/3425123978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 msgr2=0x55b49cb9bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7680012010 tx=0x7f7680005850 comp rx=0 tx=0).stop 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- 172.21.15.116:0/3425123978 shutdown_connections 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb81250 0x55b49cb8b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb7abf0 0x55b49cb9bd70 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- 172.21.15.116:0/3425123978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb766c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- 172.21.15.116:0/3425123978 >> 172.21.15.116:0/3425123978 conn(0x55b49cb74a00 msgr2=0x55b49cb73310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- 172.21.15.116:0/3425123978 shutdown_connections 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- 172.21.15.116:0/3425123978 wait complete. 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- start start 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 Processor -- start 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb87ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7abf0 0x55b49cb88000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.032+0000 7f76947b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1ae340 con 0x55b49cb81250 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1adc70 con 0x55b49cb7a2d0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1aaed0 con 0x55b49cb7abf0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 10 client.0 _ll_register_callbacks cb 0x7f767c009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56992/0 (socket says 172.21.15.116:56992) 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 -- 172.21.15.116:0/3624539941 learned_addr learned my addr 172.21.15.116:0/3624539941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768b7fe640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb87ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768affd640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7abf0 0x55b49cb88000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 -- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7abf0 msgr2=0x55b49cb88000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7abf0 0x55b49cb88000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 -- 172.21.15.116:0/3624539941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 msgr2=0x55b49cb87ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb87ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 -- 172.21.15.116:0/3624539941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b49cb79cd0 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768b7fe640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb87ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f768bfff640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7684020850 tx=0x7f7684016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d277d4e22ab5bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f766affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684033020 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f7684009040 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f768002b7f0 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f768403e050 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 10 client.18600 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.033+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b49cb864a0 con 0x55b49cb81250 2024-03-19T02:38:01.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f768002b7f0 con 0x55b49cb81250 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f768402c050 con 0x55b49cb81250 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 10 client.18600 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 10 client.18600 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 20 client.18600 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cbfd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 4 client.18600 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b49caea9f0 con 0x55b49cb93fe0 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f768b7fe640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cbfd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f768b7fe640 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cbfd140 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee676553d2c10cf6 server_cookie=1ac8663c4a613e in_seq=0 out_seq=0 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.034+0000 7f766affd640 10 client.18600 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f766affd640 1 -- 172.21.15.116:0/3624539941 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b49caea9f0 con 0x55b49cb93fe0 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f766affd640 10 client.18600 handle_command_reply: tid=0 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 client.18600 shutdown 2024-03-19T02:38:01.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 msgr2=0x55b49cbf93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7684020850 tx=0x7f7684016040 comp rx=0 tx=0).stop 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 shutdown_connections 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55b49cb93fe0 0x55b49cbfd140 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b49cb81250 0x55b49cbf93a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b49cb7abf0 0x55b49cb88000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f766affd640 0 client.18600 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 --2- 172.21.15.116:0/3624539941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b49cb7a2d0 0x55b49cb87ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 >> 172.21.15.116:0/3624539941 conn(0x55b49cb74a00 msgr2=0x55b49cb76540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 shutdown_connections 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 1 -- 172.21.15.116:0/3624539941 wait complete. 2024-03-19T02:38:01.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.035+0000 7f76947b5740 20 client.18600 trim_cache size 0 max 16384 2024-03-19T02:38:01.043 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:01.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 msgr2=0x7f76740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 0x7f76740a4eb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7684005c40 tx=0x7f7684004a40 comp rx=0 tx=0).stop 2024-03-19T02:38:01.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 msgr2=0x7f768c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f767c009230 tx=0x7f767c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 shutdown_connections 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f76740a29f0 0x7f76740a4eb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768c19caa0 0x7f768c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f768c100680 0x7f768c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 --2- 172.21.15.116:0/2289418345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f768c0ffcc0 0x7f768c1981e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 >> 172.21.15.116:0/2289418345 conn(0x7f768c076c50 msgr2=0x7f768c0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 shutdown_connections 2024-03-19T02:38:01.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.036+0000 7f7691aea640 1 -- 172.21.15.116:0/2289418345 wait complete. 2024-03-19T02:38:01.047 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:38:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 -- 172.21.15.116:0/1230717425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41066b0 msgr2=0x7f80e4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 --2- 172.21.15.116:0/1230717425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41066b0 0x7f80e4106a90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009930 tx=0x7f80d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 -- 172.21.15.116:0/1230717425 shutdown_connections 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 --2- 172.21.15.116:0/1230717425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 0x7f80e410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 --2- 172.21.15.116:0/1230717425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41006a0 0x7f80e4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 --2- 172.21.15.116:0/1230717425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41066b0 0x7f80e4106a90 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 -- 172.21.15.116:0/1230717425 >> 172.21.15.116:0/1230717425 conn(0x7f80e40778d0 msgr2=0x7f80e4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 -- 172.21.15.116:0/1230717425 shutdown_connections 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.166+0000 7f80eafb9640 1 -- 172.21.15.116:0/1230717425 wait complete. 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 Processor -- start 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 -- start start 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 0x7f80e4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41066b0 0x7f80e419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41109c0 con 0x7f80e41006a0 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4110840 con 0x7f80e4101060 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80eafb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4110b40 con 0x7f80e41066b0 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57010/0 (socket says 172.21.15.116:57010) 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 -- 172.21.15.116:0/2392483204 learned_addr learned my addr 172.21.15.116:0/2392483204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e9556640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41066b0 0x7f80e419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e3fff640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 0x7f80e4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 -- 172.21.15.116:0/2392483204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41066b0 msgr2=0x7f80e419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41066b0 0x7f80e419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 -- 172.21.15.116:0/2392483204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 msgr2=0x7f80e4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 0x7f80e4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.167+0000 7f80e8d55640 1 -- 172.21.15.116:0/2392483204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e419cfe0 con 0x7f80e41006a0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80e8d55640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f80d8000c00 tx=0x7f80d802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80e1ffb640 1 -- 172.21.15.116:0/2392483204 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8043020 con 0x7f80e41006a0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80e1ffb640 1 -- 172.21.15.116:0/2392483204 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d802caf0 con 0x7f80e41006a0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e419d270 con 0x7f80e41006a0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80e1ffb640 1 -- 172.21.15.116:0/2392483204 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8003a80 con 0x7f80e41006a0 2024-03-19T02:38:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.168+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e4102c50 con 0x7f80e41006a0 2024-03-19T02:38:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e1ffb640 1 -- 172.21.15.116:0/2392483204 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f80d8046020 con 0x7f80e41006a0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 Processor -- start 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 -- start start 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f97d0 0x55a27c0fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a27c0534b0 con 0x55a27c0f0bf0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a27bffbd70 con 0x55a27c0f97d0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80edba3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a27c04f550 con 0x55a27c0f02d0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e9556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e9556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44788/0 (socket says 172.21.15.116:44788) 2024-03-19T02:38:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e9556640 1 -- 172.21.15.116:0/3228538219 learned_addr learned my addr 172.21.15.116:0/3228538219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57020/0 (socket says 172.21.15.116:57020) 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.171+0000 7f80e3fff640 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f97d0 0x55a27c0fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e1ffb640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 0x7f80d40a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 -- 172.21.15.116:0/3228538219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f97d0 msgr2=0x55a27c0fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f97d0 0x55a27c0fbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 -- 172.21.15.116:0/3228538219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 msgr2=0x55a27c0f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 -- 172.21.15.116:0/3228538219 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27c01e210 con 0x55a27c0f02d0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e1ffb640 1 -- 172.21.15.116:0/2392483204 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f80d80f10f0 con 0x7f80e41006a0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 0x7f80d40a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80dc011590 tx=0x7f80dc011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188e79de018c2f03 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80ceffd640 1 -- 172.21.15.116:0/3228538219 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc017050 con 0x55a27c0f02d0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80e9556640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 0x7f80d40a9240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80dc022000 tx=0x7f80dc023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80ceffd640 1 -- 172.21.15.116:0/3228538219 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80dc00f7a0 con 0x55a27c0f02d0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.172+0000 7f80ceffd640 1 -- 172.21.15.116:0/3228538219 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00ee00 con 0x55a27c0f02d0 2024-03-19T02:38:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- 172.21.15.116:0/3228538219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 msgr2=0x55a27c0f06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80dc011590 tx=0x7f80dc011a50 comp rx=0 tx=0).stop 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- 172.21.15.116:0/3228538219 shutdown_connections 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f97d0 0x55a27c0fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 --2- 172.21.15.116:0/3228538219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f02d0 0x55a27c0f06b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- 172.21.15.116:0/3228538219 >> 172.21.15.116:0/3228538219 conn(0x55a27c0eaa00 msgr2=0x55a27c0eade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- 172.21.15.116:0/3228538219 shutdown_connections 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- 172.21.15.116:0/3228538219 wait complete. 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 -- start start 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.173+0000 7f80edba3740 1 Processor -- start 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a27c0fc5c0 con 0x55a27c0f02d0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a27c00f310 con 0x55a27c0f0bf0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a27c0fc740 con 0x55a27c0f97d0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 10 client.0 _ll_register_callbacks cb 0x55a27c0032d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57022/0 (socket says 172.21.15.116:57022) 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e9556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 -- 172.21.15.116:0/1926800701 learned_addr learned my addr 172.21.15.116:0/1926800701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 -- 172.21.15.116:0/1926800701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 msgr2=0x55a27c0f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 -- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 msgr2=0x55a27c0f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 -- 172.21.15.116:0/1926800701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27bd673f0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e9556640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f80d80ee5d0 tx=0x7f80d802c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac530510cb99a638 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d80b9720 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f80d80bd6e0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.174+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a27c0b13e0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d80bcde0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 10 client.18612 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a27be317d0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a27c0b13e0 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f80d80c5790 con 0x55a27c0f02d0 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 10 client.18612 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 10 client.18612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 20 client.18612 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c109a80 0x55a27c10beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 4 client.18612 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a27c0609f0 con 0x55a27c109a80 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.175+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c109a80 0x55a27c10beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.176+0000 7f80c3fff640 10 client.18612 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.176+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c109a80 0x55a27c10beb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40f9fe433561119d server_cookie=f2de4de6f6238508 in_seq=0 out_seq=0 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.176+0000 7f80c3fff640 1 -- 172.21.15.116:0/1926800701 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a27c0609f0 con 0x55a27c109a80 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.176+0000 7f80c3fff640 10 client.18612 handle_command_reply: tid=0 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.176+0000 7f80edba3740 1 client.18612 shutdown 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 msgr2=0x55a27c0f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f80d80ee5d0 tx=0x7f80d802c2b0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 shutdown_connections 2024-03-19T02:38:01.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c109a80 0x55a27c10beb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80c3fff640 0 client.18612 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 --2- 172.21.15.116:0/1926800701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f6810 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 >> 172.21.15.116:0/1926800701 conn(0x55a27c0eaa00 msgr2=0x55a27c0ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 shutdown_connections 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 1 -- 172.21.15.116:0/1926800701 wait complete. 2024-03-19T02:38:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.177+0000 7f80edba3740 20 client.18612 trim_cache size 0 max 16384 2024-03-19T02:38:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 Processor -- start 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 -- start start 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41aca20 con 0x55a27c0f02d0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ae560 con 0x55a27c0f0bf0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.190+0000 7f80edba3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ad0f0 con 0x55a27c0f97d0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44816/0 (socket says 172.21.15.116:44816) 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57038/0 (socket says 172.21.15.116:57038) 2024-03-19T02:38:01.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40078/0 (socket says 172.21.15.116:40078) 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 -- 172.21.15.116:0/1994298626 learned_addr learned my addr 172.21.15.116:0/1994298626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 -- 172.21.15.116:0/1994298626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 msgr2=0x55a27c0f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 -- 172.21.15.116:0/1994298626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 msgr2=0x55a27c0f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 -- 172.21.15.116:0/1994298626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27c0f15a0 con 0x55a27c0f97d0 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80e9556640 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80dc005fd0 tx=0x7f80dc00efa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=108ea79b49015816 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80ce7fc640 1 -- 172.21.15.116:0/1994298626 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc034070 con 0x55a27c0f97d0 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80ce7fc640 1 -- 172.21.15.116:0/1994298626 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80dc004a00 con 0x55a27c0f97d0 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.191+0000 7f80ce7fc640 1 -- 172.21.15.116:0/1994298626 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00e8d0 con 0x55a27c0f97d0 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- 172.21.15.116:0/1994298626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 msgr2=0x55a27c105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80dc005fd0 tx=0x7f80dc00efa0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- 172.21.15.116:0/1994298626 shutdown_connections 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c105360 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 --2- 172.21.15.116:0/1994298626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f02d0 0x55a27c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- 172.21.15.116:0/1994298626 >> 172.21.15.116:0/1994298626 conn(0x55a27c0eaa00 msgr2=0x55a27c0e9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- 172.21.15.116:0/1994298626 shutdown_connections 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- 172.21.15.116:0/1994298626 wait complete. 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 -- start start 2024-03-19T02:38:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.192+0000 7f80edba3740 1 Processor -- start 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 0x55a27c101990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ade90 con 0x55a27c0f0bf0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ad7c0 con 0x55a27c0f02d0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41aaa20 con 0x55a27c0f97d0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80edba3740 10 client.0 _ll_register_callbacks cb 0x7f80d8009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e8d55640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 0x55a27c101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57040/0 (socket says 172.21.15.116:57040) 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 -- 172.21.15.116:0/1111885945 learned_addr learned my addr 172.21.15.116:0/1111885945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e9556640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 -- 172.21.15.116:0/1111885945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 msgr2=0x55a27c0f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 -- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 msgr2=0x55a27c101990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 0x55a27c101990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 -- 172.21.15.116:0/1111885945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27c102050 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 0x55a27c101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80e3fff640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f80d0012580 tx=0x7f80d0012ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb411cb55286aec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.193+0000 7f80c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0004230 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80dc004a00 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f80d00043d0 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d0033050 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 10 client.18618 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a27c16f800 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80dc004a00 con 0x55a27c0f0bf0 2024-03-19T02:38:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f80d0022500 con 0x55a27c0f0bf0 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 10 client.18618 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 10 client.18618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 20 client.18618 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c16ff80 0x55a27c1723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 4 client.18618 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a27c05ad40 con 0x55a27c16ff80 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.194+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c16ff80 0x55a27c1723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.195+0000 7f80e8d55640 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c16ff80 0x55a27c1723a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22dc4840cca9e0b server_cookie=165dafb467fe8045 in_seq=0 out_seq=0 2024-03-19T02:38:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.195+0000 7f80c3fff640 10 client.18618 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80c3fff640 1 -- 172.21.15.116:0/1111885945 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a27c05ad40 con 0x55a27c16ff80 2024-03-19T02:38:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80c3fff640 10 client.18618 handle_command_reply: tid=0 2024-03-19T02:38:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 client.18618 shutdown 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 msgr2=0x55a27c0f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f80d0012580 tx=0x7f80d0012ca0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 shutdown_connections 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a27c16ff80 0x55a27c1723a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a27c0f97d0 0x55a27c0f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a27c0f0bf0 0x55a27c0f2ac0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80c3fff640 0 client.18618 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 --2- 172.21.15.116:0/1111885945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27c0f02d0 0x55a27c101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 >> 172.21.15.116:0/1111885945 conn(0x55a27c0eaa00 msgr2=0x55a27c0e9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 shutdown_connections 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 1 -- 172.21.15.116:0/1111885945 wait complete. 2024-03-19T02:38:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.196+0000 7f80edba3740 20 client.18618 trim_cache size 0 max 16384 2024-03-19T02:38:01.204 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 msgr2=0x7f80d40a9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 0x7f80d40a9240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80dc022000 tx=0x7f80dc023000 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 msgr2=0x7f80e4197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f80d8000c00 tx=0x7f80d802c660 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 shutdown_connections 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80d40a6d80 0x7f80d40a9240 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e41066b0 0x7f80e419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e4101060 0x7f80e4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 --2- 172.21.15.116:0/2392483204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80e41006a0 0x7f80e4197ff0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 >> 172.21.15.116:0/2392483204 conn(0x7f80e40778d0 msgr2=0x7f80e4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 shutdown_connections 2024-03-19T02:38:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:01.197+0000 7f80eafb9640 1 -- 172.21.15.116:0/2392483204 wait complete. 2024-03-19T02:38:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:00 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2432796908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:00 smithi154 ceph-mon[35693]: pgmap v317: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 120 op/s 2024-03-19T02:38:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:00 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2432796908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:00 smithi162 ceph-mon[35761]: pgmap v317: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 120 op/s 2024-03-19T02:38:01.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 331321 3.13 MB/sec execute 444 sec latency 25.991 ms 2024-03-19T02:38:02.210 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:38:02.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:38:02.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 -- 172.21.15.116:0/3707665172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 msgr2=0x7f18a810dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 --2- 172.21.15.116:0/3707665172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a810dc80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1898009900 tx=0x7f189802bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 -- 172.21.15.116:0/3707665172 shutdown_connections 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 --2- 172.21.15.116:0/3707665172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 0x7f18a81105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 --2- 172.21.15.116:0/3707665172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a810dc80 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 --2- 172.21.15.116:0/3707665172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a8068de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 -- 172.21.15.116:0/3707665172 >> 172.21.15.116:0/3707665172 conn(0x7f18a8077820 msgr2=0x7f18a8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 -- 172.21.15.116:0/3707665172 shutdown_connections 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.324+0000 7f18b0137640 1 -- 172.21.15.116:0/3707665172 wait complete. 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 Processor -- start 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 -- start start 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 0x7f18a819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18a8113c30 con 0x7f18a8069320 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18a8113ab0 con 0x7f18a8068a00 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18b0137640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18a8113db0 con 0x7f18a810e1c0 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57066/0 (socket says 172.21.15.116:57066) 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18aded3640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18ad6d2640 1 -- 172.21.15.116:0/3941750719 learned_addr learned my addr 172.21.15.116:0/3941750719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.325+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 0x7f18a819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 -- 172.21.15.116:0/3941750719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 msgr2=0x7f18a819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 0x7f18a819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 -- 172.21.15.116:0/3941750719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 msgr2=0x7f18a819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 -- 172.21.15.116:0/3941750719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a819f240 con 0x7f18a8069320 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18aded3640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1898005940 tx=0x7f18980045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f189effd640 1 -- 172.21.15.116:0/3941750719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898043020 con 0x7f18a8069320 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a819f4d0 con 0x7f18a8069320 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f189effd640 1 -- 172.21.15.116:0/3941750719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f189802caf0 con 0x7f18a8069320 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f189effd640 1 -- 172.21.15.116:0/3941750719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898003a50 con 0x7f18a8069320 2024-03-19T02:38:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.326+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a81a6dc0 con 0x7f18a8069320 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.328+0000 7f189effd640 1 -- 172.21.15.116:0/3941750719 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1898046020 con 0x7f18a8069320 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 Processor -- start 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 -- start start 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f189effd640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 0x7f18940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd207340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd207880 0x55dcbd209c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd162160 con 0x55dcbd1fe380 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd10ae10 con 0x55dcbd1feca0 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd15e200 con 0x55dcbd207880 2024-03-19T02:38:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 0x7f18940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18aded3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f189effd640 1 -- 172.21.15.116:0/3941750719 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f18980f1220 con 0x7f18a8069320 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18aded3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57082/0 (socket says 172.21.15.116:57082) 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18aded3640 1 -- 172.21.15.116:0/2634532328 learned_addr learned my addr 172.21.15.116:0/2634532328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd207880 0x55dcbd209c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.329+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd207340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 -- 172.21.15.116:0/2634532328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd207880 msgr2=0x55dcbd209c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd207880 0x55dcbd209c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 -- 172.21.15.116:0/2634532328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 msgr2=0x55dcbd207340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd207340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 -- 172.21.15.116:0/2634532328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcbcf3e7e0 con 0x55dcbd1fe380 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18aded3640 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f18a40115f0 tx=0x7f18a4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8f5443b990aacc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 0x7f18940a4e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009990 tx=0x7f18a0009de0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f1893fff640 1 -- 172.21.15.116:0/2634532328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a401e070 con 0x55dcbd1fe380 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f1893fff640 1 -- 172.21.15.116:0/2634532328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a400faf0 con 0x55dcbd1fe380 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.330+0000 7f1893fff640 1 -- 172.21.15.116:0/2634532328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4019b40 con 0x55dcbd1fe380 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- 172.21.15.116:0/2634532328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 msgr2=0x55dcbd1fe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f18a40115f0 tx=0x7f18a4011ab0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- 172.21.15.116:0/2634532328 shutdown_connections 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd207880 0x55dcbd209c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd207340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/2634532328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1fe380 0x55dcbd1fe760 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- 172.21.15.116:0/2634532328 >> 172.21.15.116:0/2634532328 conn(0x55dcbd1f8ab0 msgr2=0x55dcbd1f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- 172.21.15.116:0/2634532328 shutdown_connections 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- 172.21.15.116:0/2634532328 wait complete. 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 -- start start 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.331+0000 7f18b2dfc740 1 Processor -- start 2024-03-19T02:38:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd204860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd2052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd20a670 con 0x55dcbd1feca0 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd11e1a0 con 0x55dcbd207880 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dcbd20a7f0 con 0x55dcbd1fe380 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18b2dfc740 10 client.0 _ll_register_callbacks cb 0x55dcbce75c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57098/0 (socket says 172.21.15.116:57098) 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 -- 172.21.15.116:0/4242606796 learned_addr learned my addr 172.21.15.116:0/4242606796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18aded3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd204860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd2052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 -- 172.21.15.116:0/4242606796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 msgr2=0x55dcbd204860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd204860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 -- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 msgr2=0x55dcbd2052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd2052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 -- 172.21.15.116:0/4242606796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcbcf9ef70 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f18980ee750 tx=0x7f18980bcf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ef1105aad37690 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18917fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189802cca0 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.332+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f18980a1470 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a400faf0 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18980bdcb0 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 10 client.18630 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dcbce36170 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a400faf0 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f18980eece0 con 0x55dcbd1feca0 2024-03-19T02:38:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 10 client.18630 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 10 client.18630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 20 client.18630 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd217bf0 0x55dcbd21a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 4 client.18630 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcbd16f6a0 con 0x55dcbd217bf0 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.333+0000 7f18aded3640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd217bf0 0x55dcbd21a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.334+0000 7f18aded3640 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd217bf0 0x55dcbd21a030 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8742088163731038 server_cookie=55bd122d88f5e832 in_seq=0 out_seq=0 2024-03-19T02:38:02.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.334+0000 7f18917fa640 10 client.18630 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18917fa640 1 -- 172.21.15.116:0/4242606796 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcbd16f6a0 con 0x55dcbd217bf0 2024-03-19T02:38:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18917fa640 10 client.18630 handle_command_reply: tid=0 2024-03-19T02:38:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 client.18630 shutdown 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 msgr2=0x55dcbd204da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f18980ee750 tx=0x7f18980bcf90 comp rx=0 tx=0).stop 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 shutdown_connections 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd217bf0 0x55dcbd21a030 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd2052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd204da0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18917fa640 0 client.18630 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/4242606796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd204860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 >> 172.21.15.116:0/4242606796 conn(0x55dcbd1f8ab0 msgr2=0x55dcbd1fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 shutdown_connections 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 1 -- 172.21.15.116:0/4242606796 wait complete. 2024-03-19T02:38:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.335+0000 7f18b2dfc740 20 client.18630 trim_cache size 0 max 16384 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 Processor -- start 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 -- start start 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd206f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd213410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81b0ec0 con 0x55dcbd207880 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81b2a00 con 0x55dcbd1feca0 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81b1590 con 0x55dcbd1fe380 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18ae6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18ae6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57106/0 (socket says 172.21.15.116:57106) 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18ae6d4640 1 -- 172.21.15.116:0/3660037502 learned_addr learned my addr 172.21.15.116:0/3660037502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18aded3640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd206f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.353+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd213410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 -- 172.21.15.116:0/3660037502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 msgr2=0x55dcbd206f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd206f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 -- 172.21.15.116:0/3660037502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 msgr2=0x55dcbd213410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd213410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 -- 172.21.15.116:0/3660037502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcbd1f88e0 con 0x55dcbd207880 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18aded3640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd206f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f18a0010b00 tx=0x7f18a0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbb8af30452e238 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18937fe640 1 -- 172.21.15.116:0/3660037502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0020a10 con 0x55dcbd207880 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18937fe640 1 -- 172.21.15.116:0/3660037502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a0020bb0 con 0x55dcbd207880 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.354+0000 7f18937fe640 1 -- 172.21.15.116:0/3660037502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00113e0 con 0x55dcbd207880 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- 172.21.15.116:0/3660037502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 msgr2=0x55dcbd21f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f18a0010b00 tx=0x7f18a0020000 comp rx=0 tx=0).stop 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- 172.21.15.116:0/3660037502 shutdown_connections 2024-03-19T02:38:02.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd207880 0x55dcbd21f940 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd1feca0 0x55dcbd213410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/3660037502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd206f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- 172.21.15.116:0/3660037502 >> 172.21.15.116:0/3660037502 conn(0x55dcbd1f8ab0 msgr2=0x55dcbd1fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- 172.21.15.116:0/3660037502 shutdown_connections 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- 172.21.15.116:0/3660037502 wait complete. 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 -- start start 2024-03-19T02:38:02.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.355+0000 7f18b2dfc740 1 Processor -- start 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd20fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd20cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81b2330 con 0x55dcbd1feca0 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81b1c60 con 0x55dcbd207880 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81aeec0 con 0x55dcbd1fe380 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18b2dfc740 10 client.0 _ll_register_callbacks cb 0x7f18980bd3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57110/0 (socket says 172.21.15.116:57110) 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 -- 172.21.15.116:0/1455561418 learned_addr learned my addr 172.21.15.116:0/1455561418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18aded3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd20fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ae6d4640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd20cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 -- 172.21.15.116:0/1455561418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 msgr2=0x55dcbd20fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd20fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 -- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 msgr2=0x55dcbd20cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd20cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 -- 172.21.15.116:0/1455561418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcbd20d490 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18ad6d2640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f189809f110 tx=0x7f18980a80b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7847a55370e0be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18917fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.356+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18980bd0f0 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7f18980c58c0 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a0020bb0 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f18980bd5b0 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 10 client.18636 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dcbd27ec00 con 0x55dcbd1feca0 2024-03-19T02:38:02.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a0020bb0 con 0x55dcbd1feca0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f18980eece0 con 0x55dcbd1feca0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 10 client.18636 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 10 client.18636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 20 client.18636 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd27f1c0 0x55dcbd2815e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 4 client.18636 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dcbd1699f0 con 0x55dcbd27f1c0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.357+0000 7f18aded3640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd27f1c0 0x55dcbd2815e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.358+0000 7f18917fa640 10 client.18636 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.358+0000 7f18aded3640 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd27f1c0 0x55dcbd2815e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f8fccf635abedd server_cookie=639a9fecea8a71a in_seq=0 out_seq=0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18917fa640 1 -- 172.21.15.116:0/1455561418 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dcbd1699f0 con 0x55dcbd27f1c0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18917fa640 10 client.18636 handle_command_reply: tid=0 2024-03-19T02:38:02.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 client.18636 shutdown 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 msgr2=0x55dcbd20ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f189809f110 tx=0x7f18980a80b0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 shutdown_connections 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dcbd27f1c0 0x55dcbd2815e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dcbd207880 0x55dcbd20cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcbd1feca0 0x55dcbd20ca10 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18917fa640 0 client.18636 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 --2- 172.21.15.116:0/1455561418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dcbd1fe380 0x55dcbd20fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 >> 172.21.15.116:0/1455561418 conn(0x55dcbd1f8ab0 msgr2=0x55dcbd1f9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 shutdown_connections 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 1 -- 172.21.15.116:0/1455561418 wait complete. 2024-03-19T02:38:02.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.360+0000 7f18b2dfc740 20 client.18636 trim_cache size 0 max 16384 2024-03-19T02:38:02.366 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:38:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "12425e27-11e9-4799-8088-a9eaa75e7773", 2024-03-19T02:38:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:38:02.366 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 msgr2=0x7f18940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 0x7f18940a4e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009990 tx=0x7f18a0009de0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 msgr2=0x7f18a819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1898005940 tx=0x7f18980045e0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 shutdown_connections 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f18940a2970 0x7f18940a4e30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18a810e1c0 0x7f18a819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18a8069320 0x7f18a819a790 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 --2- 172.21.15.116:0/3941750719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a8068a00 0x7f18a819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 >> 172.21.15.116:0/3941750719 conn(0x7f18a8077820 msgr2=0x7f18a810f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 shutdown_connections 2024-03-19T02:38:02.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.361+0000 7f18b0137640 1 -- 172.21.15.116:0/3941750719 wait complete. 2024-03-19T02:38:02.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:38:02.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 -- 172.21.15.116:0/3711107136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc558108820 msgr2=0x7fc558108ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 --2- 172.21.15.116:0/3711107136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc558108820 0x7fc558108ca0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009900 tx=0x7fc54c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 -- 172.21.15.116:0/3711107136 shutdown_connections 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 --2- 172.21.15.116:0/3711107136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 0x7fc5581150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 --2- 172.21.15.116:0/3711107136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc558108820 0x7fc558108ca0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 --2- 172.21.15.116:0/3711107136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc55810d220 0x7fc55810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 -- 172.21.15.116:0/3711107136 >> 172.21.15.116:0/3711107136 conn(0x7fc55806c900 msgr2=0x7fc55806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 -- 172.21.15.116:0/3711107136 shutdown_connections 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.497+0000 7fc55c80b640 1 -- 172.21.15.116:0/3711107136 wait complete. 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 Processor -- start 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 -- start start 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc558108820 0x7fc5581a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 0x7fc5581a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc558117fe0 con 0x7fc55810d220 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc558117e60 con 0x7fc558108820 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55c80b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc558118160 con 0x7fc5581091e0 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc556d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc556d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57130/0 (socket says 172.21.15.116:57130) 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 0x7fc5581a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc556d9d640 1 -- 172.21.15.116:0/4146256263 learned_addr learned my addr 172.21.15.116:0/4146256263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc55659c640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc558108820 0x7fc5581a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.498+0000 7fc556d9d640 1 -- 172.21.15.116:0/4146256263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 msgr2=0x7fc5581a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc556d9d640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 0x7fc5581a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc556d9d640 1 -- 172.21.15.116:0/4146256263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc558108820 msgr2=0x7fc5581a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc556d9d640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc558108820 0x7fc5581a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc556d9d640 1 -- 172.21.15.116:0/4146256263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5581a2fa0 con 0x7fc55810d220 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc556d9d640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc55000a780 tx=0x7fc55000e130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc5477fe640 1 -- 172.21.15.116:0/4146256263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000ed70 con 0x7fc55810d220 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc5477fe640 1 -- 172.21.15.116:0/4146256263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc55001b070 con 0x7fc55810d220 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5581a3290 con 0x7fc55810d220 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc5477fe640 1 -- 172.21.15.116:0/4146256263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550015750 con 0x7fc55810d220 2024-03-19T02:38:02.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.499+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5581af460 con 0x7fc55810d220 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.501+0000 7fc5477fe640 1 -- 172.21.15.116:0/4146256263 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc550014050 con 0x7fc55810d220 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 Processor -- start 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 -- start start 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x5555616f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55556164b500 con 0x5555616e7a30 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555615f3be0 con 0x5555616e8350 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5555616475a0 con 0x5555616f0f30 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc5477fe640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 0x7fc5400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x5555616f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616e7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57144/0 (socket says 172.21.15.116:57144) 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc555d9b640 1 -- 172.21.15.116:0/3749803798 learned_addr learned my addr 172.21.15.116:0/3749803798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc556d9d640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 0x7fc5400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc5477fe640 1 -- 172.21.15.116:0/4146256263 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc5500cee90 con 0x7fc55810d220 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55659c640 1 -- 172.21.15.116:0/3749803798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 msgr2=0x5555616f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55659c640 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55659c640 1 -- 172.21.15.116:0/3749803798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 msgr2=0x5555616e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.502+0000 7fc55659c640 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616e7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc55659c640 1 -- 172.21.15.116:0/3749803798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555561615000 con 0x5555616e8350 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc55659c640 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x5555616f09f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc54800bc40 tx=0x7fc548005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b10da7142a744a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc556d9d640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 0x7fc5400a4e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cc2b0 tx=0x7fc550008820 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc544ff9640 1 -- 172.21.15.116:0/3749803798 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54800be50 con 0x5555616e8350 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc544ff9640 1 -- 172.21.15.116:0/3749803798 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5480035f0 con 0x5555616e8350 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.503+0000 7fc544ff9640 1 -- 172.21.15.116:0/3749803798 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5480044e0 con 0x5555616e8350 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3749803798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 msgr2=0x5555616f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x5555616f09f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc54800bc40 tx=0x7fc548005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3749803798 shutdown_connections 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x5555616f09f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3749803798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3749803798 >> 172.21.15.116:0/3749803798 conn(0x5555616e2160 msgr2=0x5555616e2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3749803798 shutdown_connections 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3749803798 wait complete. 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 -- start start 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.504+0000 7fc55f3f4740 1 Processor -- start 2024-03-19T02:38:02.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616edda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616ee820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5555616f3d70 con 0x5555616e8350 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555561607160 con 0x5555616e7a30 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5555616f3ef0 con 0x5555616f0f30 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55f3f4740 10 client.0 _ll_register_callbacks cb 0x555561643a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57154/0 (socket says 172.21.15.116:57154) 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 -- 172.21.15.116:0/1479902553 learned_addr learned my addr 172.21.15.116:0/1479902553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc555d9b640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc555d9b640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 -- 172.21.15.116:0/1479902553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 msgr2=0x5555616ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616ee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 -- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 msgr2=0x5555616edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 -- 172.21.15.116:0/1479902553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555614297d0 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc55659c640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc548000c00 tx=0x7fc5480039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a449728c12db762d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.505+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc548026070 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc548004200 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5480035f0 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc548032050 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 10 client.18648 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55556135f3f0 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5480035f0 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc548016430 con 0x5555616e8350 2024-03-19T02:38:02.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 10 client.18648 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 10 client.18648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 20 client.18648 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561701090 0x555561703480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 4 client.18648 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.506+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555561658a40 con 0x555561701090 2024-03-19T02:38:02.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.507+0000 7fc555d9b640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561701090 0x555561703480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.507+0000 7fc555d9b640 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561701090 0x555561703480 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49df7a63b63b9da2 server_cookie=4938da50c2279e79 in_seq=0 out_seq=0 2024-03-19T02:38:02.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.507+0000 7fc53dffb640 10 client.18648 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc53dffb640 1 -- 172.21.15.116:0/1479902553 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555561658a40 con 0x555561701090 2024-03-19T02:38:02.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc53dffb640 10 client.18648 handle_command_reply: tid=0 2024-03-19T02:38:02.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 client.18648 shutdown 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 msgr2=0x5555616ee2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc548000c00 tx=0x7fc5480039d0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 shutdown_connections 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561701090 0x555561703480 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e8350 0x5555616ee2e0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/1479902553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc53dffb640 0 client.18648 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 >> 172.21.15.116:0/1479902553 conn(0x5555616e2160 msgr2=0x5555616e3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 shutdown_connections 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 1 -- 172.21.15.116:0/1479902553 wait complete. 2024-03-19T02:38:02.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.508+0000 7fc55f3f4740 20 client.18648 trim_cache size 0 max 16384 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 Processor -- start 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 -- start start 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x555561708b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581b9670 con 0x5555616e7a30 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581bb1b0 con 0x5555616e8350 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581b9d40 con 0x5555616f0f30 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57166/0 (socket says 172.21.15.116:57166) 2024-03-19T02:38:02.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc55659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x555561708b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc555d9b640 1 -- 172.21.15.116:0/3958210302 learned_addr learned my addr 172.21.15.116:0/3958210302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.529+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 -- 172.21.15.116:0/3958210302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 msgr2=0x5555616f9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 -- 172.21.15.116:0/3958210302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 msgr2=0x555561708b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x555561708b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 -- 172.21.15.116:0/3958210302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55556131f0f0 con 0x5555616e7a30 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc55659c640 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x555561708b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009230 tx=0x7fc54c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5fe5cd7051ebde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc53ffff640 1 -- 172.21.15.116:0/3958210302 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c04ecf0 con 0x5555616e7a30 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc53ffff640 1 -- 172.21.15.116:0/3958210302 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc54c04e340 con 0x5555616e7a30 2024-03-19T02:38:02.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.530+0000 7fc53ffff640 1 -- 172.21.15.116:0/3958210302 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c0035f0 con 0x5555616e7a30 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3958210302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 msgr2=0x5555616fcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009230 tx=0x7fc54c02c6c0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3958210302 shutdown_connections 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616f0f30 0x5555616f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e8350 0x555561708b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3958210302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616e7a30 0x5555616fcb60 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3958210302 >> 172.21.15.116:0/3958210302 conn(0x5555616e2160 msgr2=0x5555616e0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3958210302 shutdown_connections 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.531+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3958210302 wait complete. 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 -- start start 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 Processor -- start 2024-03-19T02:38:02.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616e8350 0x5555616eb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581baae0 con 0x5555616f0f30 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581ba410 con 0x5555616e7a30 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc5581b7670 con 0x5555616e8350 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55f3f4740 10 client.0 _ll_register_callbacks cb 0x5555616f0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57174/0 (socket says 172.21.15.116:57174) 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 -- 172.21.15.116:0/3298510168 learned_addr learned my addr 172.21.15.116:0/3298510168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc55659c640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616e8350 0x5555616eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 -- 172.21.15.116:0/3298510168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616e8350 msgr2=0x5555616eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616e8350 0x5555616eb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 -- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 msgr2=0x5555616eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 -- 172.21.15.116:0/3298510168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555561766670 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc555d9b640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc54c005ec0 tx=0x7fc54c02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d02cd1531ea1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.532+0000 7fc53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c004040 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9510+0+0 (secure 0 0 0) 0x7fc54c0041e0 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc54c04e340 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc54c0043e0 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 10 client.18654 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555561767880 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc54c04e340 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc54c043650 con 0x5555616f0f30 2024-03-19T02:38:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 10 client.18654 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 10 client.18654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 20 client.18654 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561767e00 0x55556176a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 4 client.18654 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555616e8be0 con 0x555561767e00 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.533+0000 7fc55659c640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561767e00 0x55556176a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55659c640 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561767e00 0x55556176a220 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adde2f64f47d21bb server_cookie=3c2db3d3c83c384f in_seq=0 out_seq=0 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc53dffb640 10 client.18654 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc53dffb640 1 -- 172.21.15.116:0/3298510168 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5555616e8be0 con 0x555561767e00 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc53dffb640 10 client.18654 handle_command_reply: tid=0 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 client.18654 shutdown 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 msgr2=0x555561766130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc54c005ec0 tx=0x7fc54c02c330 comp rx=0 tx=0).stop 2024-03-19T02:38:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 shutdown_connections 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555561767e00 0x55556176a220 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555616f0f30 0x555561766130 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc53dffb640 0 client.18654 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555616e8350 0x5555616eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 --2- 172.21.15.116:0/3298510168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555616e7a30 0x5555616eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.534+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 >> 172.21.15.116:0/3298510168 conn(0x5555616e2160 msgr2=0x5555616e11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 shutdown_connections 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55f3f4740 1 -- 172.21.15.116:0/3298510168 wait complete. 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55f3f4740 20 client.18654 trim_cache size 0 max 16384 2024-03-19T02:38:02.541 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "12425e27-11e9-4799-8088-a9eaa75e7773": { 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "12425e27-11e9-4799-8088-a9eaa75e7773", 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:38:02.542 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 msgr2=0x7fc5400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 0x7fc5400a4e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cc2b0 tx=0x7fc550008820 comp rx=0 tx=0).stop 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 msgr2=0x7fc5581a27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fc55000a780 tx=0x7fc55000e130 comp rx=0 tx=0).stop 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 shutdown_connections 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5400a2970 0x7fc5400a4e30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc55810d220 0x7fc5581a27d0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5581091e0 0x7fc5581a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 --2- 172.21.15.116:0/4146256263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc558108820 0x7fc5581a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 >> 172.21.15.116:0/4146256263 conn(0x7fc55806c900 msgr2=0x7fc5580711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 shutdown_connections 2024-03-19T02:38:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:02.535+0000 7fc55c80b640 1 -- 172.21.15.116:0/4146256263 wait complete. 2024-03-19T02:38:02.550 INFO:tasks.cephfs.filesystem:scrub status for tag:12425e27-11e9-4799-8088-a9eaa75e7773 - {'path': '/', 'tag': '12425e27-11e9-4799-8088-a9eaa75e7773', 'options': 'recursive,force'} 2024-03-19T02:38:02.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 331801 3.12 MB/sec execute 445 sec latency 43.002 ms 2024-03-19T02:38:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:02 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:38:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:38:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:02 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:38:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:38:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:02 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:38:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:38:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.793 INFO:tasks.workunit.client.0.smithi116.stdout: 1 332336 3.12 MB/sec execute 446 sec latency 19.302 ms 2024-03-19T02:38:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:03 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:38:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:03 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:03 smithi162 ceph-mon[35761]: pgmap v318: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:03 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:38:04.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:03 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:38:04.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:03 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:04.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:03 smithi154 ceph-mon[35693]: pgmap v318: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:04.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:03 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:38:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:03 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:38:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:03 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:03 smithi116 ceph-mon[32263]: pgmap v318: 129 pgs: 129 active+clean; 354 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:03 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:38:04.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 333129 3.13 MB/sec execute 447 sec latency 94.876 ms 2024-03-19T02:38:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 333396 3.13 MB/sec execute 448 sec latency 107.197 ms 2024-03-19T02:38:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T02:38:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: pgmap v320: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:38:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:05 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: pgmap v320: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:05 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: pgmap v320: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 2.1 MiB/s wr, 116 op/s 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:05 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:06.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 333819 3.13 MB/sec execute 449 sec latency 18.688 ms 2024-03-19T02:38:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:38:06.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:06 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:38:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:38:07.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:06 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:38:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:38:07.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:06 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:38:07.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 334255 3.12 MB/sec execute 450 sec latency 45.574 ms 2024-03-19T02:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: pgmap v322: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:38:08.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: pgmap v322: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:38:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:38:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: pgmap v322: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:38:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:38:08.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 334854 3.12 MB/sec execute 451 sec latency 28.951 ms 2024-03-19T02:38:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:08 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:38:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:08 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:38:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:08 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:38:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:08 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:38:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:08 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:38:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:08 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:38:09.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 335347 3.12 MB/sec execute 452 sec latency 32.284 ms 2024-03-19T02:38:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:09 smithi162 ceph-mon[35761]: pgmap v323: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:38:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:09 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:38:10.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:09 smithi154 ceph-mon[35693]: pgmap v323: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:38:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:09 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:38:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:09 smithi116 ceph-mon[32263]: pgmap v323: 129 pgs: 129 active+clean; 356 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.3 MiB/s wr, 86 op/s 2024-03-19T02:38:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:09 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:38:10.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 335933 3.12 MB/sec execute 453 sec latency 20.126 ms 2024-03-19T02:38:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:38:11.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:38:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:38:11.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 336735 3.12 MB/sec execute 454 sec latency 17.694 ms 2024-03-19T02:38:12.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:11 smithi162 ceph-mon[35761]: pgmap v324: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-03-19T02:38:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:11 smithi154 ceph-mon[35693]: pgmap v324: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-03-19T02:38:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:11 smithi116 ceph-mon[32263]: pgmap v324: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 144 op/s 2024-03-19T02:38:12.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 337191 3.13 MB/sec execute 455 sec latency 19.208 ms 2024-03-19T02:38:13.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 337579 3.12 MB/sec execute 456 sec latency 17.083 ms 2024-03-19T02:38:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:13 smithi162 ceph-mon[35761]: pgmap v325: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-03-19T02:38:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:13 smithi154 ceph-mon[35693]: pgmap v325: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-03-19T02:38:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:13 smithi116 ceph-mon[32263]: pgmap v325: 129 pgs: 129 active+clean; 358 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 138 op/s 2024-03-19T02:38:14.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 338152 3.12 MB/sec execute 457 sec latency 28.501 ms 2024-03-19T02:38:15.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 338682 3.12 MB/sec execute 458 sec latency 22.291 ms 2024-03-19T02:38:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:15 smithi154 ceph-mon[35693]: pgmap v326: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:15 smithi116 ceph-mon[32263]: pgmap v326: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:15 smithi162 ceph-mon[35761]: pgmap v326: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:16.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 339314 3.12 MB/sec execute 459 sec latency 20.195 ms 2024-03-19T02:38:17.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 340151 3.12 MB/sec execute 460 sec latency 16.947 ms 2024-03-19T02:38:18.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:17 smithi154 ceph-mon[35693]: pgmap v327: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:17 smithi116 ceph-mon[32263]: pgmap v327: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:17 smithi162 ceph-mon[35761]: pgmap v327: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-03-19T02:38:18.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 340592 3.13 MB/sec execute 461 sec latency 18.005 ms 2024-03-19T02:38:19.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 340994 3.12 MB/sec execute 462 sec latency 18.593 ms 2024-03-19T02:38:20.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:19 smithi162 ceph-mon[35761]: pgmap v328: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:19 smithi154 ceph-mon[35693]: pgmap v328: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:19 smithi116 ceph-mon[32263]: pgmap v328: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-03-19T02:38:20.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 341497 3.12 MB/sec execute 463 sec latency 35.840 ms 2024-03-19T02:38:21.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 342112 3.12 MB/sec execute 464 sec latency 26.242 ms 2024-03-19T02:38:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:21 smithi154 ceph-mon[35693]: pgmap v329: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 118 op/s 2024-03-19T02:38:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:21 smithi116 ceph-mon[32263]: pgmap v329: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 118 op/s 2024-03-19T02:38:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:21 smithi162 ceph-mon[35761]: pgmap v329: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 118 op/s 2024-03-19T02:38:22.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 342696 3.11 MB/sec execute 465 sec latency 22.049 ms 2024-03-19T02:38:23.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 343278 3.12 MB/sec execute 466 sec latency 57.165 ms 2024-03-19T02:38:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:23 smithi154 ceph-mon[35693]: pgmap v330: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-19T02:38:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:23 smithi116 ceph-mon[32263]: pgmap v330: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-19T02:38:24.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:23 smithi162 ceph-mon[35761]: pgmap v330: 129 pgs: 129 active+clean; 370 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-03-19T02:38:24.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 343851 3.12 MB/sec execute 467 sec latency 19.487 ms 2024-03-19T02:38:25.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 344368 3.12 MB/sec execute 468 sec latency 18.023 ms 2024-03-19T02:38:26.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:25 smithi154 ceph-mon[35693]: pgmap v331: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-03-19T02:38:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:25 smithi116 ceph-mon[32263]: pgmap v331: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-03-19T02:38:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:25 smithi162 ceph-mon[35761]: pgmap v331: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-03-19T02:38:26.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 344744 3.12 MB/sec execute 469 sec latency 40.896 ms 2024-03-19T02:38:27.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 345352 3.11 MB/sec execute 470 sec latency 15.532 ms 2024-03-19T02:38:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:27 smithi154 ceph-mon[35693]: pgmap v332: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:27 smithi116 ceph-mon[32263]: pgmap v332: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:27 smithi162 ceph-mon[35761]: pgmap v332: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:28.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 345909 3.11 MB/sec execute 471 sec latency 34.694 ms 2024-03-19T02:38:29.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 346501 3.11 MB/sec execute 472 sec latency 22.884 ms 2024-03-19T02:38:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:29 smithi154 ceph-mon[35693]: pgmap v333: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:29 smithi116 ceph-mon[32263]: pgmap v333: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:29 smithi162 ceph-mon[35761]: pgmap v333: 129 pgs: 129 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:38:30.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 347353 3.12 MB/sec execute 473 sec latency 16.483 ms 2024-03-19T02:38:31.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 347743 3.12 MB/sec execute 474 sec latency 22.687 ms 2024-03-19T02:38:32.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:31 smithi154 ceph-mon[35693]: pgmap v334: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-03-19T02:38:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:31 smithi116 ceph-mon[32263]: pgmap v334: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-03-19T02:38:32.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:31 smithi162 ceph-mon[35761]: pgmap v334: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 122 op/s 2024-03-19T02:38:32.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:38:32.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 -- 172.21.15.116:0/716320524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c8101050 msgr2=0x7f57c810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 --2- 172.21.15.116:0/716320524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c8101050 0x7f57c810d560 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009cd0 tx=0x7f57c002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 -- 172.21.15.116:0/716320524 shutdown_connections 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 --2- 172.21.15.116:0/716320524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c8101050 0x7f57c810d560 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 --2- 172.21.15.116:0/716320524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 --2- 172.21.15.116:0/716320524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c81066a0 0x7f57c8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 -- 172.21.15.116:0/716320524 >> 172.21.15.116:0/716320524 conn(0x7f57c8077820 msgr2=0x7f57c8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 -- 172.21.15.116:0/716320524 shutdown_connections 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.669+0000 7f57cfda9640 1 -- 172.21.15.116:0/716320524 wait complete. 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 Processor -- start 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 -- start start 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c8101050 0x7f57c81993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57c8110960 con 0x7f57c81066a0 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81107e0 con 0x7f57c8100690 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cfda9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57c8110ae0 con 0x7f57c8101050 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57ce346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57ce346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34052/0 (socket says 172.21.15.116:34052) 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56290/0 (socket says 172.21.15.116:56290) 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57ce346640 1 -- 172.21.15.116:0/1762492214 learned_addr learned my addr 172.21.15.116:0/1762492214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.670+0000 7f57cd344640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c8101050 0x7f57c81993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 -- 172.21.15.116:0/1762492214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c8101050 msgr2=0x7f57c81993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c8101050 0x7f57c81993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 -- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 msgr2=0x7f57c81a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 -- 172.21.15.116:0/1762492214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c81a4450 con 0x7f57c8100690 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57ce346640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cdb45640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57c400e9f0 tx=0x7f57c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57beffd640 1 -- 172.21.15.116:0/1762492214 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c400ccc0 con 0x7f57c8100690 2024-03-19T02:38:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c81a4620 con 0x7f57c8100690 2024-03-19T02:38:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57beffd640 1 -- 172.21.15.116:0/1762492214 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57c4005020 con 0x7f57c8100690 2024-03-19T02:38:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57beffd640 1 -- 172.21.15.116:0/1762492214 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c4004a00 con 0x7f57c8100690 2024-03-19T02:38:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.671+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c81a4b40 con 0x7f57c8100690 2024-03-19T02:38:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.674+0000 7f57beffd640 1 -- 172.21.15.116:0/1762492214 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f57c4004ba0 con 0x7f57c8100690 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.674+0000 7f57d2994740 1 Processor -- start 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.674+0000 7f57d2994740 1 -- start start 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe33089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 0x55fbe330b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fbe3263500 con 0x55fbe3300350 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fbe320bbe0 con 0x55fbe32ffa30 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57d2994740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbe325f5a0 con 0x55fbe3308f30 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe33089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56300/0 (socket says 172.21.15.116:56300) 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 -- 172.21.15.116:0/2156467906 learned_addr learned my addr 172.21.15.116:0/2156467906 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57cd344640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 0x55fbe330b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57beffd640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 0x7f57b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 -- 172.21.15.116:0/2156467906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 msgr2=0x55fbe330b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 0x55fbe330b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 -- 172.21.15.116:0/2156467906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 msgr2=0x55fbe33089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe33089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 -- 172.21.15.116:0/2156467906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe322d000 con 0x55fbe32ffa30 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57cd344640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 0x55fbe330b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57cdb45640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe33089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57beffd640 1 -- 172.21.15.116:0/1762492214 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f57c40cf160 con 0x7f57c8100690 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.675+0000 7f57ce346640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 0x7f57b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.676+0000 7f57ce346640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 0x7f57b40a4e30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f57c0043000 tx=0x7f57c0045000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.676+0000 7f57ce346640 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009dc0 tx=0x7f57c0009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e4d1cf35833436 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.676+0000 7f57b3fff640 1 -- 172.21.15.116:0/2156467906 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0039040 con 0x55fbe32ffa30 2024-03-19T02:38:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.676+0000 7f57b3fff640 1 -- 172.21.15.116:0/2156467906 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57c004e070 con 0x55fbe32ffa30 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.676+0000 7f57b3fff640 1 -- 172.21.15.116:0/2156467906 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00347a0 con 0x55fbe32ffa30 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 -- 172.21.15.116:0/2156467906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 msgr2=0x55fbe32ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009dc0 tx=0x7f57c0009860 comp rx=0 tx=0).stop 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 -- 172.21.15.116:0/2156467906 shutdown_connections 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3308f30 0x55fbe330b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe33089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 --2- 172.21.15.116:0/2156467906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe32ffe10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 -- 172.21.15.116:0/2156467906 >> 172.21.15.116:0/2156467906 conn(0x55fbe32fa160 msgr2=0x55fbe32fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 -- 172.21.15.116:0/2156467906 shutdown_connections 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.677+0000 7f57d2994740 1 -- 172.21.15.116:0/2156467906 wait complete. 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 -- start start 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 Processor -- start 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe3305f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe33064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fbe330bcc0 con 0x55fbe3308f30 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fbe321f160 con 0x55fbe32ffa30 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbe330be40 con 0x55fbe3300350 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57d2994740 10 client.0 _ll_register_callbacks cb 0x55fbe325ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34072/0 (socket says 172.21.15.116:34072) 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe3305f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 -- 172.21.15.116:0/3494259491 learned_addr learned my addr 172.21.15.116:0/3494259491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57cd344640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe33064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 -- 172.21.15.116:0/3494259491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 msgr2=0x55fbe33064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe33064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 -- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 msgr2=0x55fbe3305f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.678+0000 7f57ce346640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe3305f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57ce346640 1 -- 172.21.15.116:0/3494259491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe30417d0 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57cdb45640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe3305f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57ce346640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f57c0034990 tx=0x7f57c0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56007c9f1c25bcf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0057070 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f57c0049470 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57c004e070 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57c0049670 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57d2994740 10 client.18666 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fbe2f773f0 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57c0049cc0 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.679+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f57c003f040 con 0x55fbe3308f30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 10 client.18666 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 10 client.18666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:32.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 20 client.18666 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe331f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 4 client.18666 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f57c004e070 con 0x55fbe331d4f0 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57cdb45640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe331f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57b17fa640 10 client.18666 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.680+0000 7f57cdb45640 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe331f8e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc1660d7121bec2 server_cookie=d24c2d00c50fed7e in_seq=0 out_seq=0 2024-03-19T02:38:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.681+0000 7f57b17fa640 1 -- 172.21.15.116:0/3494259491 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f57c004e070 con 0x55fbe331d4f0 2024-03-19T02:38:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.681+0000 7f57b17fa640 10 client.18666 handle_command_reply: tid=0 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.681+0000 7f57d2994740 1 client.18666 shutdown 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.681+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 msgr2=0x55fbe33069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.681+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f57c0034990 tx=0x7f57c0009360 comp rx=0 tx=0).stop 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 shutdown_connections 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe331f8e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57b17fa640 0 client.18666 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3308f30 0x55fbe33069e0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe33064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 --2- 172.21.15.116:0/3494259491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe32ffa30 0x55fbe3305f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 >> 172.21.15.116:0/3494259491 conn(0x55fbe32fa160 msgr2=0x55fbe32fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 shutdown_connections 2024-03-19T02:38:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 1 -- 172.21.15.116:0/3494259491 wait complete. 2024-03-19T02:38:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.682+0000 7f57d2994740 20 client.18666 trim_cache size 0 max 16384 2024-03-19T02:38:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 Processor -- start 2024-03-19T02:38:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 -- start start 2024-03-19T02:38:32.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe32ffa30 0x55fbe3308690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe32f9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81acaa0 con 0x55fbe3300350 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81ae5e0 con 0x55fbe3308f30 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57d2994740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81ad170 con 0x55fbe32ffa30 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57cd344640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57cd344640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34084/0 (socket says 172.21.15.116:34084) 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe32ffa30 0x55fbe3308690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57cd344640 1 -- 172.21.15.116:0/3237732153 learned_addr learned my addr 172.21.15.116:0/3237732153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.700+0000 7f57ce346640 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe32f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 -- 172.21.15.116:0/3237732153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe32ffa30 msgr2=0x55fbe3308690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe32ffa30 0x55fbe3308690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 -- 172.21.15.116:0/3237732153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 msgr2=0x55fbe32f9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe32f9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 -- 172.21.15.116:0/3237732153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe3300180 con 0x55fbe3300350 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57cd344640 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009c80 tx=0x7f57b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffd3daeb5c785c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57b37fe640 1 -- 172.21.15.116:0/3237732153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8009e30 con 0x55fbe3300350 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57b37fe640 1 -- 172.21.15.116:0/3237732153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b80054d0 con 0x55fbe3300350 2024-03-19T02:38:32.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.701+0000 7f57b37fe640 1 -- 172.21.15.116:0/3237732153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8004c80 con 0x55fbe3300350 2024-03-19T02:38:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- 172.21.15.116:0/3237732153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 msgr2=0x55fbe3314a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009c80 tx=0x7f57b8005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- 172.21.15.116:0/3237732153 shutdown_connections 2024-03-19T02:38:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe32f9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe3300350 0x55fbe3314a60 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 --2- 172.21.15.116:0/3237732153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe32ffa30 0x55fbe3308690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- 172.21.15.116:0/3237732153 >> 172.21.15.116:0/3237732153 conn(0x55fbe32fa160 msgr2=0x55fbe32f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- 172.21.15.116:0/3237732153 shutdown_connections 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- 172.21.15.116:0/3237732153 wait complete. 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 -- start start 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 Processor -- start 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.702+0000 7f57d2994740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe3303160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe33036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81adf10 con 0x55fbe32ffa30 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81ad840 con 0x55fbe3308f30 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81aaaa0 con 0x55fbe3300350 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cd344640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe3303160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34090/0 (socket says 172.21.15.116:34090) 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 -- 172.21.15.116:0/199137019 learned_addr learned my addr 172.21.15.116:0/199137019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 10 client.0 _ll_register_callbacks cb 0x7f57c40d2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57ce346640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe33036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 -- 172.21.15.116:0/199137019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 msgr2=0x55fbe3303160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe3303160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 -- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 msgr2=0x55fbe33036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe33036a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 -- 172.21.15.116:0/199137019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe2f370f0 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57cdb45640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f57c4097db0 tx=0x7f57c4097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e2b8f77833dbfc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c401c040 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f57c40996e0 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.703+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b80054d0 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57c4097680 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 10 client.18672 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fbe3383790 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b80054d0 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f57c4097a50 con 0x55fbe32ffa30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 10 client.18672 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 10 client.18672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 20 client.18672 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe3385e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 4 client.18672 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbe3270a40 con 0x55fbe331d4f0 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.704+0000 7f57cd344640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe3385e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57cd344640 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe3385e80 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66229817b9d5ae4d server_cookie=bf0318b94c8ee33e in_seq=0 out_seq=0 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57b17fa640 10 client.18672 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57b17fa640 1 -- 172.21.15.116:0/199137019 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fbe3270a40 con 0x55fbe331d4f0 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57b17fa640 10 client.18672 handle_command_reply: tid=0 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57d2994740 1 client.18672 shutdown 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 msgr2=0x55fbe3302c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.705+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f57c4097db0 tx=0x7f57c4097040 comp rx=0 tx=0).stop 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 shutdown_connections 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fbe331d4f0 0x55fbe3385e80 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57b17fa640 0 client.18672 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fbe3308f30 0x55fbe33036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbe3300350 0x55fbe3303160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 --2- 172.21.15.116:0/199137019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbe32ffa30 0x55fbe3302c20 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 >> 172.21.15.116:0/199137019 conn(0x55fbe32fa160 msgr2=0x55fbe3306790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 shutdown_connections 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 1 -- 172.21.15.116:0/199137019 wait complete. 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57d2994740 20 client.18672 trim_cache size 0 max 16384 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:38:32.712 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:38:32.713 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:32.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 msgr2=0x7f57b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 0x7f57b40a4e30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f57c0043000 tx=0x7f57c0045000 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 msgr2=0x7f57c8198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57c400e9f0 tx=0x7f57c400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 shutdown_connections 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57b40a2970 0x7f57b40a4e30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c81066a0 0x7f57c81a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57c8101050 0x7f57c81993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 --2- 172.21.15.116:0/1762492214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c8100690 0x7f57c8198e90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.706+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 >> 172.21.15.116:0/1762492214 conn(0x7f57c8077820 msgr2=0x7f57c8102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.707+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 shutdown_connections 2024-03-19T02:38:32.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.707+0000 7f57cfda9640 1 -- 172.21.15.116:0/1762492214 wait complete. 2024-03-19T02:38:32.718 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:38:32.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:38:32.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 348114 3.12 MB/sec execute 475 sec latency 47.286 ms 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 -- 172.21.15.116:0/354676236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330107d40 msgr2=0x7fa33010e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 --2- 172.21.15.116:0/354676236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330107d40 0x7fa33010e6c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ca0 tx=0x7fa32802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 -- 172.21.15.116:0/354676236 shutdown_connections 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 --2- 172.21.15.116:0/354676236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330107d40 0x7fa33010e6c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 --2- 172.21.15.116:0/354676236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330100f20 0x7fa330107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 --2- 172.21.15.116:0/354676236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa330100570 0x7fa330100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 -- 172.21.15.116:0/354676236 >> 172.21.15.116:0/354676236 conn(0x7fa330075a40 msgr2=0x7fa330075e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:32.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 -- 172.21.15.116:0/354676236 shutdown_connections 2024-03-19T02:38:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.828+0000 7fa335710640 1 -- 172.21.15.116:0/354676236 wait complete. 2024-03-19T02:38:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 Processor -- start 2024-03-19T02:38:32.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 -- start start 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330100570 0x7fa330197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330107d40 0x7fa3301984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa330111a50 con 0x7fa330100570 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3301118d0 con 0x7fa33019c850 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa335710640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa330111bd0 con 0x7fa330107d40 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330100570 0x7fa330197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56348/0 (socket says 172.21.15.116:56348) 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 -- 172.21.15.116:0/3912922063 learned_addr learned my addr 172.21.15.116:0/3912922063 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32e7fc640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330107d40 0x7fa3301984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 -- 172.21.15.116:0/3912922063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330107d40 msgr2=0x7fa3301984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330107d40 0x7fa3301984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 -- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330100570 msgr2=0x7fa330197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330100570 0x7fa330197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.829+0000 7fa32f7fe640 1 -- 172.21.15.116:0/3912922063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33019d340 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa32802c700 tx=0x7fa328035d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328039040 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33019d5d0 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa328044070 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280042a0 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.830+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3301a4df0 con 0x7fa33019c850 2024-03-19T02:38:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.831+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa330067d80 con 0x7fa33019c850 2024-03-19T02:38:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.832+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa328071bc0 con 0x7fa33019c850 2024-03-19T02:38:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.833+0000 7fa317fff640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.833+0000 7fa32effd640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.833+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fa3280f14c0 con 0x7fa33019c850 2024-03-19T02:38:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.833+0000 7fa32effd640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 0x7fa3180a4e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa320009bd0 tx=0x7fa320005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:32.836+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa3280bdf50 con 0x7fa33019c850 2024-03-19T02:38:33.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.090+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa330100f20 con 0x7fa33019c850 2024-03-19T02:38:33.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.091+0000 7fa317fff640 1 -- 172.21.15.116:0/3912922063 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fa3280bd850 con 0x7fa33019c850 2024-03-19T02:38:33.092 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:38:33.092 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:38:33.092 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 msgr2=0x7fa3180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 0x7fa3180a4e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa320009bd0 tx=0x7fa320005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 msgr2=0x7fa33019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa32802c700 tx=0x7fa328035d60 comp rx=0 tx=0).stop 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 shutdown_connections 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3180a2970 0x7fa3180a4e30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa33019c850 0x7fa33019cd00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa330107d40 0x7fa3301984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 --2- 172.21.15.116:0/3912922063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330100570 0x7fa330197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.094+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 >> 172.21.15.116:0/3912922063 conn(0x7fa330075a40 msgr2=0x7fa3300ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.095+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 shutdown_connections 2024-03-19T02:38:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.095+0000 7fa335710640 1 -- 172.21.15.116:0/3912922063 wait complete. 2024-03-19T02:38:33.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 -- 172.21.15.116:0/1368842214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241066b0 msgr2=0x7fad24106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 --2- 172.21.15.116:0/1368842214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241066b0 0x7fad24106a90 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fad20009930 tx=0x7fad2002bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 -- 172.21.15.116:0/1368842214 shutdown_connections 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 --2- 172.21.15.116:0/1368842214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad24101060 0x7fad2410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 --2- 172.21.15.116:0/1368842214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad24100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 --2- 172.21.15.116:0/1368842214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241066b0 0x7fad24106a90 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 -- 172.21.15.116:0/1368842214 >> 172.21.15.116:0/1368842214 conn(0x7fad240778d0 msgr2=0x7fad24077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 -- 172.21.15.116:0/1368842214 shutdown_connections 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.228+0000 7fad2c17e640 1 -- 172.21.15.116:0/1368842214 wait complete. 2024-03-19T02:38:33.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 Processor -- start 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 -- start start 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad241066b0 0x7fad2410ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111ac0 con 0x7fad24101060 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111940 con 0x7fad241066b0 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad2c17e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad24111c40 con 0x7fad241006a0 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad29719640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47196/0 (socket says 172.21.15.116:47196) 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad29719640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34116/0 (socket says 172.21.15.116:34116) 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.229+0000 7fad29f1a640 1 -- 172.21.15.116:0/2884467237 learned_addr learned my addr 172.21.15.116:0/2884467237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad2a71b640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad241066b0 0x7fad2410ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 -- 172.21.15.116:0/2884467237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad241066b0 msgr2=0x7fad2410ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad241066b0 0x7fad2410ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 -- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 msgr2=0x7fad2410e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 -- 172.21.15.116:0/2884467237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad241a64a0 con 0x7fad241006a0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29719640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fad20000c00 tx=0x7fad2002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad1affd640 1 -- 172.21.15.116:0/2884467237 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20043020 con 0x7fad241006a0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad241a6730 con 0x7fad241006a0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad1affd640 1 -- 172.21.15.116:0/2884467237 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad2002caf0 con 0x7fad241006a0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad1affd640 1 -- 172.21.15.116:0/2884467237 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20003a80 con 0x7fad241006a0 2024-03-19T02:38:33.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.230+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241a6cd0 con 0x7fad241006a0 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.232+0000 7fad1affd640 1 -- 172.21.15.116:0/2884467237 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fad20046020 con 0x7fad241006a0 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 Processor -- start 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 -- start start 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 0x55e995c1f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c1fbf0 0x55e995c28290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e995b824b0 con 0x55e995c1fbf0 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e995b2ad70 con 0x55e995c287d0 2024-03-19T02:38:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e995b7e550 con 0x55e995c1f2d0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad1affd640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 0x7fad100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c1fbf0 0x55e995c28290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2a71b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 0x55e995c1f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad29719640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad29719640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56374/0 (socket says 172.21.15.116:56374) 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2a71b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 0x55e995c1f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47204/0 (socket says 172.21.15.116:47204) 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad29719640 1 -- 172.21.15.116:0/4028686476 learned_addr learned my addr 172.21.15.116:0/4028686476 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.233+0000 7fad2a71b640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 0x7fad100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad1affd640 1 -- 172.21.15.116:0/2884467237 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fad200f1160 con 0x7fad241006a0 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 -- 172.21.15.116:0/4028686476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 msgr2=0x55e995c1f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 0x55e995c1f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 -- 172.21.15.116:0/4028686476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c1fbf0 msgr2=0x55e995c28290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c1fbf0 0x55e995c28290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 -- 172.21.15.116:0/4028686476 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e995b4d210 con 0x55e995c287d0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad2a71b640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 0x7fad100a4e30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005fd0 tx=0x7fad1c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad29719640 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fad140114f0 tx=0x7fad140119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5314f9ed3ff622f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad0ffff640 1 -- 172.21.15.116:0/4028686476 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004020 con 0x55e995c287d0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad0ffff640 1 -- 172.21.15.116:0/4028686476 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad140041c0 con 0x55e995c287d0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.234+0000 7fad0ffff640 1 -- 172.21.15.116:0/4028686476 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1400f860 con 0x55e995c287d0 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4028686476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 msgr2=0x55e995c2abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fad140114f0 tx=0x7fad140119b0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4028686476 shutdown_connections 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c287d0 0x55e995c2abc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c1fbf0 0x55e995c28290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4028686476 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1f2d0 0x55e995c1f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4028686476 >> 172.21.15.116:0/4028686476 conn(0x55e995c19a00 msgr2=0x55e995c19de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4028686476 shutdown_connections 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4028686476 wait complete. 2024-03-19T02:38:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.235+0000 7fad2ed6e740 1 -- start start 2024-03-19T02:38:33.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 Processor -- start 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c25dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c26310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e995c2b5c0 con 0x55e995c287d0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e995b3e310 con 0x55e995c1f2d0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e995c2b740 con 0x55e995c1fbf0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2ed6e740 10 client.0 _ll_register_callbacks cb 0x55e995b322d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56380/0 (socket says 172.21.15.116:56380) 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 -- 172.21.15.116:0/3919330599 learned_addr learned my addr 172.21.15.116:0/3919330599 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad2a71b640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c26310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29719640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c25dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 -- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 msgr2=0x55e995c25dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c25dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 -- 172.21.15.116:0/3919330599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 msgr2=0x55e995c26310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c26310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29f1a640 1 -- 172.21.15.116:0/3919330599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9958963f0 con 0x55e995c1f2d0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.236+0000 7fad29719640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c25dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad29f1a640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fad200b9ec0 tx=0x7fad2008c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4249a954dc5f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad200bce50 con 0x55e995c1f2d0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fad200bd750 con 0x55e995c1f2d0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad140041c0 con 0x55e995c1f2d0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad200cd420 con 0x55e995c1f2d0 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad2ed6e740 10 client.28478 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e9959607d0 con 0x55e995c1f2d0 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad140041c0 con 0x55e995c1f2d0 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.237+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fad200c5800 con 0x55e995c1f2d0 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 10 client.28478 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 10 client.28478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 20 client.28478 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c38b80 0x55e995c3b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 4 client.28478 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e995be03e0 con 0x55e995c38b80 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad29719640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c38b80 0x55e995c3b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad0d7fa640 10 client.28478 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.238+0000 7fad29719640 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c38b80 0x55e995c3b010 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdb0399e827099ac server_cookie=bb7a6415d29e11b in_seq=0 out_seq=0 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad0d7fa640 1 -- 172.21.15.116:0/3919330599 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e995be03e0 con 0x55e995c38b80 2024-03-19T02:38:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad0d7fa640 10 client.28478 handle_command_reply: tid=0 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 client.28478 shutdown 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 msgr2=0x55e995c25890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fad200b9ec0 tx=0x7fad2008c300 comp rx=0 tx=0).stop 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 shutdown_connections 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c38b80 0x55e995c3b010 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c26310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c25dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/3919330599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c25890 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 >> 172.21.15.116:0/3919330599 conn(0x55e995c19a00 msgr2=0x55e995c1b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.239+0000 7fad0d7fa640 0 client.28478 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.240+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 shutdown_connections 2024-03-19T02:38:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.240+0000 7fad2ed6e740 1 -- 172.21.15.116:0/3919330599 wait complete. 2024-03-19T02:38:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.240+0000 7fad2ed6e740 20 client.28478 trim_cache size 0 max 16384 2024-03-19T02:38:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 Processor -- start 2024-03-19T02:38:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 -- start start 2024-03-19T02:38:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c18a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b0c70 con 0x55e995c287d0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.255+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b27b0 con 0x55e995c1f2d0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b1340 con 0x55e995c1fbf0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad2a71b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c18a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47222/0 (socket says 172.21.15.116:47222) 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 -- 172.21.15.116:0/4023896440 learned_addr learned my addr 172.21.15.116:0/4023896440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 -- 172.21.15.116:0/4023896440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 msgr2=0x55e995c34360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c34360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 -- 172.21.15.116:0/4023896440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 msgr2=0x55e995c18a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c18a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 -- 172.21.15.116:0/4023896440 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e995c1fa20 con 0x55e995c1fbf0 2024-03-19T02:38:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29f1a640 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c34360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad29719640 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e995c1ab90 tx=0x7fad14005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c426d29510fe77c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad0f7fe640 1 -- 172.21.15.116:0/4023896440 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004090 con 0x55e995c1fbf0 2024-03-19T02:38:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.256+0000 7fad0f7fe640 1 -- 172.21.15.116:0/4023896440 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad14004230 con 0x55e995c1fbf0 2024-03-19T02:38:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad0f7fe640 1 -- 172.21.15.116:0/4023896440 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad140035f0 con 0x55e995c1fbf0 2024-03-19T02:38:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4023896440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 msgr2=0x55e995c26100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e995c1ab90 tx=0x7fad14005850 comp rx=0 tx=0).stop 2024-03-19T02:38:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4023896440 shutdown_connections 2024-03-19T02:38:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c18a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c26100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/4023896440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c34360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4023896440 >> 172.21.15.116:0/4023896440 conn(0x55e995c19a00 msgr2=0x55e995c18310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4023896440 shutdown_connections 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.257+0000 7fad2ed6e740 1 -- 172.21.15.116:0/4023896440 wait complete. 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 -- start start 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 Processor -- start 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c2bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c2c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b20e0 con 0x55e995c287d0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b1a10 con 0x55e995c1f2d0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad241aec70 con 0x55e995c1fbf0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2ed6e740 10 client.0 _ll_register_callbacks cb 0x7fad20009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad29719640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c2bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad29f1a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56404/0 (socket says 172.21.15.116:56404) 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad29719640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c2bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47228/0 (socket says 172.21.15.116:47228) 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad2a71b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c2c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.258+0000 7fad29f1a640 1 -- 172.21.15.116:0/2009963171 learned_addr learned my addr 172.21.15.116:0/2009963171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 -- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 msgr2=0x55e995c2bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c2bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 -- 172.21.15.116:0/2009963171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 msgr2=0x55e995c2c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c2c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 -- 172.21.15.116:0/2009963171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e995c2ca00 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad2a71b640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c2c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad29f1a640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad200ee690 tx=0x7fad200c5b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71bbefe60c17c764 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad200bce50 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad14004230 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fad200bc940 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad200b9790 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad2ed6e740 10 client.28484 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.259+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e995c9eca0 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad14004230 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fad200bc3b0 con 0x55e995c1f2d0 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 10 client.28484 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 10 client.28484 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:38:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 20 client.28484 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c9f250 0x55e995ca1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 4 client.28484 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e995b8f9f0 con 0x55e995c9f250 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad29719640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c9f250 0x55e995ca1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad0d7fa640 10 client.28484 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.260+0000 7fad29719640 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c9f250 0x55e995ca1670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=20d771103d22839c server_cookie=3468d3044a4905dc in_seq=0 out_seq=0 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad0d7fa640 1 -- 172.21.15.116:0/2009963171 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e995b8f9f0 con 0x55e995c9f250 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad0d7fa640 10 client.28484 handle_command_reply: tid=0 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 client.28484 shutdown 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 msgr2=0x55e995c30990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad200ee690 tx=0x7fad200c5b10 comp rx=0 tx=0).stop 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 shutdown_connections 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e995c9f250 0x55e995ca1670 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e995c287d0 0x55e995c2c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad0d7fa640 0 client.28484 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e995c1fbf0 0x55e995c2bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 --2- 172.21.15.116:0/2009963171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e995c1f2d0 0x55e995c30990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 >> 172.21.15.116:0/2009963171 conn(0x55e995c19a00 msgr2=0x55e995c1abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 shutdown_connections 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 1 -- 172.21.15.116:0/2009963171 wait complete. 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2ed6e740 20 client.28484 trim_cache size 0 max 16384 2024-03-19T02:38:33.269 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 msgr2=0x7fad100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.261+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 0x7fad100a4e30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005fd0 tx=0x7fad1c009500 comp rx=0 tx=0).stop 2024-03-19T02:38:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 msgr2=0x7fad2410e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fad20000c00 tx=0x7fad2002c660 comp rx=0 tx=0).stop 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 shutdown_connections 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad100a2970 0x7fad100a4e30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad241066b0 0x7fad2410ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad24101060 0x7fad2410e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 --2- 172.21.15.116:0/2884467237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad241006a0 0x7fad2410e000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 >> 172.21.15.116:0/2884467237 conn(0x7fad240778d0 msgr2=0x7fad24102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 shutdown_connections 2024-03-19T02:38:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.262+0000 7fad2c17e640 1 -- 172.21.15.116:0/2884467237 wait complete. 2024-03-19T02:38:33.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:38:33.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.394+0000 7f5838eea640 1 -- 172.21.15.116:0/1320767589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 msgr2=0x7f5834100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.394+0000 7f5838eea640 1 --2- 172.21.15.116:0/1320767589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834100b20 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f5828009930 tx=0x7f582802bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:33.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 -- 172.21.15.116:0/1320767589 shutdown_connections 2024-03-19T02:38:33.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 --2- 172.21.15.116:0/1320767589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5834101060 0x7f583410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 --2- 172.21.15.116:0/1320767589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834100b20 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 --2- 172.21.15.116:0/1320767589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58341066b0 0x7f5834106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 -- 172.21.15.116:0/1320767589 >> 172.21.15.116:0/1320767589 conn(0x7f58340778d0 msgr2=0x7f5834077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 -- 172.21.15.116:0/1320767589 shutdown_connections 2024-03-19T02:38:33.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 -- 172.21.15.116:0/1320767589 wait complete. 2024-03-19T02:38:33.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.395+0000 7f5838eea640 1 Processor -- start 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 -- start start 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 0x7f5834199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58341066b0 0x7f58341a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5834110970 con 0x7f58341006a0 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58341107f0 con 0x7f58341066b0 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5838eea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5834110af0 con 0x7f5834101060 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 0x7f5834199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34178/0 (socket says 172.21.15.116:34178) 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 -- 172.21.15.116:0/161166514 learned_addr learned my addr 172.21.15.116:0/161166514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5832d9d640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58341066b0 0x7f58341a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 -- 172.21.15.116:0/161166514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 msgr2=0x7f5834199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 0x7f5834199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 -- 172.21.15.116:0/161166514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58341066b0 msgr2=0x7f58341a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58341066b0 0x7f58341a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f583259c640 1 -- 172.21.15.116:0/161166514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58341a43b0 con 0x7f58341006a0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.396+0000 7f5831d9b640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 0x7f5834199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f583259c640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f582400e9f0 tx=0x7f582400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f58237fe640 1 -- 172.21.15.116:0/161166514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582400cd80 con 0x7f58341006a0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f58237fe640 1 -- 172.21.15.116:0/161166514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58240108e0 con 0x7f58341006a0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58341a4580 con 0x7f58341006a0 2024-03-19T02:38:33.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f58237fe640 1 -- 172.21.15.116:0/161166514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5824018920 con 0x7f58341006a0 2024-03-19T02:38:33.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.397+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58341a4aa0 con 0x7f58341006a0 2024-03-19T02:38:33.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.399+0000 7f58237fe640 1 -- 172.21.15.116:0/161166514 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5824010a90 con 0x7f58341006a0 2024-03-19T02:38:33.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.400+0000 7f58237fe640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 0x7f581c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.400+0000 7f58237fe640 1 -- 172.21.15.116:0/161166514 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f58240cf450 con 0x7f58341006a0 2024-03-19T02:38:33.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.400+0000 7f5831d9b640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 0x7f581c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 Processor -- start 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 -- start start 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f5831d9b640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 0x7f581c0a4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5828005ab0 tx=0x7f5828005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e79504e7d0 0x55e795050bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e794fa84b0 con 0x55e7950452d0 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e794f50d70 con 0x55e79504e7d0 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583bbb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e794fa4550 con 0x55e795045bf0 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47250/0 (socket says 172.21.15.116:47250) 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f583259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34186/0 (socket says 172.21.15.116:34186) 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f5831d9b640 1 -- 172.21.15.116:0/3107022767 learned_addr learned my addr 172.21.15.116:0/3107022767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.401+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e79504e7d0 0x55e795050bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 -- 172.21.15.116:0/3107022767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e79504e7d0 msgr2=0x55e795050bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e79504e7d0 0x55e795050bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 -- 172.21.15.116:0/3107022767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 msgr2=0x55e7950456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 -- 172.21.15.116:0/3107022767 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e794f73210 con 0x55e795045bf0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f583259c640 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5828009930 tx=0x7f582803f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=980712c010a612fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5820ff9640 1 -- 172.21.15.116:0/3107022767 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58280044a0 con 0x55e795045bf0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5820ff9640 1 -- 172.21.15.116:0/3107022767 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5828051070 con 0x55e795045bf0 2024-03-19T02:38:33.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.402+0000 7f5820ff9640 1 -- 172.21.15.116:0/3107022767 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582803f440 con 0x55e795045bf0 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 -- 172.21.15.116:0/3107022767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 msgr2=0x55e79504e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5828009930 tx=0x7f582803f000 comp rx=0 tx=0).stop 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 -- 172.21.15.116:0/3107022767 shutdown_connections 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e79504e7d0 0x55e795050bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504e290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3107022767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7950452d0 0x55e7950456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 -- 172.21.15.116:0/3107022767 >> 172.21.15.116:0/3107022767 conn(0x55e79503fa00 msgr2=0x55e79503fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 -- 172.21.15.116:0/3107022767 shutdown_connections 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.403+0000 7f583bbb2740 1 -- 172.21.15.116:0/3107022767 wait complete. 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 -- start start 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 Processor -- start 2024-03-19T02:38:33.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e79504c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e795051560 con 0x55e79504e7d0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e794f64310 con 0x55e7950452d0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e7950516e0 con 0x55e795045bf0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583bbb2740 10 client.0 _ll_register_callbacks cb 0x55e794f582d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f583259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47262/0 (socket says 172.21.15.116:47262) 2024-03-19T02:38:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f5831d9b640 1 -- 172.21.15.116:0/2772110031 learned_addr learned my addr 172.21.15.116:0/2772110031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.404+0000 7f5832d9d640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e79504c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 -- 172.21.15.116:0/2772110031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 msgr2=0x55e79504b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 -- 172.21.15.116:0/2772110031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 msgr2=0x55e79504c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e79504c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 -- 172.21.15.116:0/2772110031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e794cbc3f0 con 0x55e795045bf0 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5831d9b640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5828000c00 tx=0x7f5828003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c3c693a538ad50 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:38:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828059070 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5828051070 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.405+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f582802cbf0 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5828065050 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 10 client.28447 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e794d867d0 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5828051070 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f58280344d0 con 0x55e795045bf0 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 10 client.28447 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 10 client.28447 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 20 client.28447 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e79505ed00 0x55e795061120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 4 client.28447 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7950063e0 con 0x55e79505ed00 2024-03-19T02:38:33.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.406+0000 7f583259c640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e79505ed00 0x55e795061120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.407+0000 7f5819ffb640 10 client.28447 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.407+0000 7f583259c640 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e79505ed00 0x55e795061120 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=99e021d6ac1d3688 server_cookie=87b65ef70d522f26 in_seq=0 out_seq=0 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.407+0000 7f5819ffb640 1 -- 172.21.15.116:0/2772110031 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7950063e0 con 0x55e79505ed00 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.407+0000 7f5819ffb640 10 client.28447 handle_command_reply: tid=0 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 client.28447 shutdown 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 msgr2=0x55e79504bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5828000c00 tx=0x7f5828003820 comp rx=0 tx=0).stop 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 shutdown_connections 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e79505ed00 0x55e795061120 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e79504c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f5819ffb640 0 client.28447 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504bd50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 --2- 172.21.15.116:0/2772110031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 >> 172.21.15.116:0/2772110031 conn(0x55e79503fa00 msgr2=0x55e795041750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 shutdown_connections 2024-03-19T02:38:33.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 1 -- 172.21.15.116:0/2772110031 wait complete. 2024-03-19T02:38:33.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.408+0000 7f583bbb2740 20 client.28447 trim_cache size 0 max 16384 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.427+0000 7f583bbb2740 1 Processor -- start 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.427+0000 7f583bbb2740 1 -- start start 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58341ac990 con 0x55e79504e7d0 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58341ae4d0 con 0x55e7950452d0 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583bbb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58341ad060 con 0x55e795045bf0 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34196/0 (socket says 172.21.15.116:34196) 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f583259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56432/0 (socket says 172.21.15.116:56432) 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 -- 172.21.15.116:0/3340118841 learned_addr learned my addr 172.21.15.116:0/3340118841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 -- 172.21.15.116:0/3340118841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 msgr2=0x55e79504c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 -- 172.21.15.116:0/3340118841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 msgr2=0x55e79504df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 -- 172.21.15.116:0/3340118841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e795046030 con 0x55e79504e7d0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.428+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f582c003cc0 tx=0x7f582c015c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c4aa7b3c1a01a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f581bfff640 1 -- 172.21.15.116:0/3340118841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c003db0 con 0x55e79504e7d0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f581bfff640 1 -- 172.21.15.116:0/3340118841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f582c015d50 con 0x55e79504e7d0 2024-03-19T02:38:33.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f581bfff640 1 -- 172.21.15.116:0/3340118841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c020df0 con 0x55e79504e7d0 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 -- 172.21.15.116:0/3340118841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 msgr2=0x55e7950669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f582c003cc0 tx=0x7f582c015c20 comp rx=0 tx=0).stop 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 -- 172.21.15.116:0/3340118841 shutdown_connections 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e79504e7d0 0x55e7950669d0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e795045bf0 0x55e79504c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3340118841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e79504df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 -- 172.21.15.116:0/3340118841 >> 172.21.15.116:0/3340118841 conn(0x55e79503fa00 msgr2=0x55e795040d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 -- 172.21.15.116:0/3340118841 shutdown_connections 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.429+0000 7f583bbb2740 1 -- 172.21.15.116:0/3340118841 wait complete. 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 -- start start 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 Processor -- start 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e795056c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 0x55e7950524b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58341ade00 con 0x55e795045bf0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58341ad730 con 0x55e7950452d0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58341aa990 con 0x55e79504e7d0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583bbb2740 10 client.0 _ll_register_callbacks cb 0x7f5824097300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f5831d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34208/0 (socket says 172.21.15.116:34208) 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f5831d9b640 1 -- 172.21.15.116:0/3724313692 learned_addr learned my addr 172.21.15.116:0/3724313692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f583259c640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e795056c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.430+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 0x55e7950524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 -- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 msgr2=0x55e7950524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 0x55e7950524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 -- 172.21.15.116:0/3724313692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 msgr2=0x55e795056c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e795056c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 -- 172.21.15.116:0/3724313692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e795052ce0 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f583259c640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e795056c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5832d9d640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 0x55e7950524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5831d9b640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f5828034920 tx=0x7f5828004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc84ba2109a012d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828059040 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f582802c9f0 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f582c015d50 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5828003b30 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f583bbb2740 10 client.18714 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e7950c5060 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f582c015d50 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.431+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f582805eaf0 con 0x55e795045bf0 2024-03-19T02:38:33.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 10 client.18714 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 10 client.18714 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 20 client.18714 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e7950c55c0 0x55e7950c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 4 client.18714 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e794fb59f0 con 0x55e7950c55c0 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583259c640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e7950c55c0 0x55e7950c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f5819ffb640 10 client.18714 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583259c640 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e7950c55c0 0x55e7950c79e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9dce68bb3d44a61d server_cookie=6ad6075f67a5b5f0 in_seq=0 out_seq=0 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f5819ffb640 1 -- 172.21.15.116:0/3724313692 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e794fb59f0 con 0x55e7950c55c0 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f5819ffb640 10 client.18714 handle_command_reply: tid=0 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 client.18714 shutdown 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 msgr2=0x55e795051f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f5828034920 tx=0x7f5828004370 comp rx=0 tx=0).stop 2024-03-19T02:38:33.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 shutdown_connections 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e79504e7d0 0x55e7950524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55e7950c55c0 0x55e7950c79e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f5819ffb640 0 client.18714 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e795045bf0 0x55e795051f70 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 --2- 172.21.15.116:0/3724313692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7950452d0 0x55e795056c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.432+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 >> 172.21.15.116:0/3724313692 conn(0x55e79503fa00 msgr2=0x55e795040d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 shutdown_connections 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f583bbb2740 1 -- 172.21.15.116:0/3724313692 wait complete. 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f583bbb2740 20 client.18714 trim_cache size 0 max 16384 2024-03-19T02:38:33.440 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 msgr2=0x7f581c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 0x7f581c0a4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5828005ab0 tx=0x7f5828005fd0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 msgr2=0x7f5834198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f582400e9f0 tx=0x7f582400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 shutdown_connections 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f581c0a29f0 0x7f581c0a4eb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58341066b0 0x7f58341a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5834101060 0x7f5834199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 --2- 172.21.15.116:0/161166514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58341006a0 0x7f5834198df0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 >> 172.21.15.116:0/161166514 conn(0x7f58340778d0 msgr2=0x7f5834103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 shutdown_connections 2024-03-19T02:38:33.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.433+0000 7f5838eea640 1 -- 172.21.15.116:0/161166514 wait complete. 2024-03-19T02:38:33.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:38:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 -- 172.21.15.116:0/399129734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e010bd80 msgr2=0x7f35e010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 --2- 172.21.15.116:0/399129734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e010bd80 0x7f35e010e220 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009ca0 tx=0x7f35d802c190 comp rx=0 tx=0).stop 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 -- 172.21.15.116:0/399129734 shutdown_connections 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 --2- 172.21.15.116:0/399129734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e010bd80 0x7f35e010e220 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 --2- 172.21.15.116:0/399129734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e0103360 0x7f35e010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 --2- 172.21.15.116:0/399129734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.570+0000 7f35e57e6640 1 -- 172.21.15.116:0/399129734 >> 172.21.15.116:0/399129734 conn(0x7f35e00778d0 msgr2=0x7f35e0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.571+0000 7f35e57e6640 1 -- 172.21.15.116:0/399129734 shutdown_connections 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.571+0000 7f35e57e6640 1 -- 172.21.15.116:0/399129734 wait complete. 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.571+0000 7f35e57e6640 1 Processor -- start 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.571+0000 7f35e57e6640 1 -- start start 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e010b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 0x7f35e0109b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010f800 con 0x7f35e0103360 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010f680 con 0x7f35e010bd80 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35e57e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010f980 con 0x7f35e0102a40 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34232/0 (socket says 172.21.15.116:34232) 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35deffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e010b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 0x7f35e0109b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 -- 172.21.15.116:0/1600900235 learned_addr learned my addr 172.21.15.116:0/1600900235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 -- 172.21.15.116:0/1600900235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 msgr2=0x7f35e010b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e010b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 -- 172.21.15.116:0/1600900235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 msgr2=0x7f35e0109b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 0x7f35e0109b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 -- 172.21.15.116:0/1600900235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e010a300 con 0x7f35e0103360 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35df7fe640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 0x7f35e0109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35deffd640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e010b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.572+0000 7f35de7fc640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f35d400e9f0 tx=0x7f35d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.573+0000 7f35c7fff640 1 -- 172.21.15.116:0/1600900235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d400cd00 con 0x7f35e0103360 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.573+0000 7f35c7fff640 1 -- 172.21.15.116:0/1600900235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d40108e0 con 0x7f35e0103360 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.573+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e01aab90 con 0x7f35e0103360 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.573+0000 7f35c7fff640 1 -- 172.21.15.116:0/1600900235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4018850 con 0x7f35e0103360 2024-03-19T02:38:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.573+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e01ab0b0 con 0x7f35e0103360 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 Processor -- start 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 -- start start 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35c7fff640 1 -- 172.21.15.116:0/1600900235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f35d40189f0 con 0x7f35e0103360 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddba0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddaf84b0 con 0x55b0ddb952d0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddaa0d70 con 0x55b0ddb95bf0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.576+0000 7f35e83ce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddaf4550 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34242/0 (socket says 172.21.15.116:34242) 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 -- 172.21.15.116:0/1989695524 learned_addr learned my addr 172.21.15.116:0/1989695524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35deffd640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35de7fc640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddba0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 -- 172.21.15.116:0/1989695524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 msgr2=0x55b0ddba0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddba0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 -- 172.21.15.116:0/1989695524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 msgr2=0x55b0ddb9e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 -- 172.21.15.116:0/1989695524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ddac3210 con 0x55b0ddb952d0 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35deffd640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35c7fff640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 0x7f35c80a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35de7fc640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddba0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35df7fe640 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009ca0 tx=0x7f35d8032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141c9326ef8fd48e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35c7fff640 1 -- 172.21.15.116:0/1600900235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f35d40cf530 con 0x7f35e0103360 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.577+0000 7f35deffd640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 0x7f35c80a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35c57fa640 1 -- 172.21.15.116:0/1989695524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8043070 con 0x55b0ddb952d0 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35c57fa640 1 -- 172.21.15.116:0/1989695524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d8003e20 con 0x55b0ddb952d0 2024-03-19T02:38:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35c57fa640 1 -- 172.21.15.116:0/1989695524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8039040 con 0x55b0ddb952d0 2024-03-19T02:38:33.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 -- 172.21.15.116:0/1989695524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 msgr2=0x55b0ddb956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009ca0 tx=0x7f35d8032440 comp rx=0 tx=0).stop 2024-03-19T02:38:33.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35deffd640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 0x7f35c80a4ef0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b0ddb94da0 tx=0x7f35d0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 -- 172.21.15.116:0/1989695524 shutdown_connections 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddba0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 --2- 172.21.15.116:0/1989695524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb952d0 0x55b0ddb956b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.578+0000 7f35e83ce740 1 -- 172.21.15.116:0/1989695524 >> 172.21.15.116:0/1989695524 conn(0x55b0ddb8fa00 msgr2=0x55b0ddb8fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- 172.21.15.116:0/1989695524 shutdown_connections 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- 172.21.15.116:0/1989695524 wait complete. 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- start start 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 Processor -- start 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddba1560 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddab4310 con 0x55b0ddb952d0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0ddba16e0 con 0x55b0ddb95bf0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35e83ce740 10 client.0 _ll_register_callbacks cb 0x55b0ddaa82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35deffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34250/0 (socket says 172.21.15.116:34250) 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35df7fe640 1 -- 172.21.15.116:0/4184164743 learned_addr learned my addr 172.21.15.116:0/4184164743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.579+0000 7f35de7fc640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 -- 172.21.15.116:0/4184164743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 msgr2=0x55b0ddb9bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 -- 172.21.15.116:0/4184164743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 msgr2=0x55b0ddb9c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 -- 172.21.15.116:0/4184164743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0dd80c3f0 con 0x55b0ddb952d0 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35df7fe640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35deffd640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9b820 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35d0012e50 tx=0x7f35d0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f43f600d0ac298 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d002d060 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f35d0018030 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35d8003e20 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35d0034050 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35e83ce740 10 client.28514 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.580+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b0dd8d67d0 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d8003e20 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35d00287a0 con 0x55b0ddb952d0 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 10 client.28514 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 10 client.28514 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 20 client.28514 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddbaeba0 0x55b0ddbb0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 4 client.28514 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0ddb563e0 con 0x55b0ddbaeba0 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.581+0000 7f35de7fc640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddbaeba0 0x55b0ddbb0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.582+0000 7f35de7fc640 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddbaeba0 0x55b0ddbb0fd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db1f4c985b42e72 server_cookie=e4e9322a8017fcba in_seq=0 out_seq=0 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.582+0000 7f35c2ff5640 10 client.28514 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.582+0000 7f35c2ff5640 1 -- 172.21.15.116:0/4184164743 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0ddb563e0 con 0x55b0ddbaeba0 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.582+0000 7f35c2ff5640 10 client.28514 handle_command_reply: tid=0 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 client.28514 shutdown 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 msgr2=0x55b0ddb9b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9b820 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35d0012e50 tx=0x7f35d0004730 comp rx=0 tx=0).stop 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 shutdown_connections 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddbaeba0 0x55b0ddbb0fd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb9bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 --2- 172.21.15.116:0/4184164743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9b820 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35c2ff5640 0 client.28514 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 >> 172.21.15.116:0/4184164743 conn(0x55b0ddb8fa00 msgr2=0x55b0ddb91750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 shutdown_connections 2024-03-19T02:38:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 1 -- 172.21.15.116:0/4184164743 wait complete. 2024-03-19T02:38:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.583+0000 7f35e83ce740 20 client.28514 trim_cache size 0 max 16384 2024-03-19T02:38:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 Processor -- start 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 -- start start 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddbaa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b52b0 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b6df0 con 0x55b0ddb95bf0 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35e83ce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b5980 con 0x55b0ddb952d0 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34258/0 (socket says 172.21.15.116:34258) 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35deffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35df7fe640 1 -- 172.21.15.116:0/3568963737 learned_addr learned my addr 172.21.15.116:0/3568963737 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddbaa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 -- 172.21.15.116:0/3568963737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 msgr2=0x55b0ddb9a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 -- 172.21.15.116:0/3568963737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 msgr2=0x55b0ddb9c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35de7fc640 1 -- 172.21.15.116:0/3568963737 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ddb97440 con 0x55b0ddb95bf0 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35deffd640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.598+0000 7f35df7fe640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.599+0000 7f35de7fc640 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddbaa330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f35d40cc840 tx=0x7f35d40d1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8ec6cbdd0089ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.599+0000 7f35c4ff9640 1 -- 172.21.15.116:0/3568963737 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d40d2e30 con 0x55b0ddb95bf0 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.599+0000 7f35c4ff9640 1 -- 172.21.15.116:0/3568963737 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d409b800 con 0x55b0ddb95bf0 2024-03-19T02:38:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.599+0000 7f35c4ff9640 1 -- 172.21.15.116:0/3568963737 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4010a90 con 0x55b0ddb95bf0 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- 172.21.15.116:0/3568963737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 msgr2=0x55b0ddbaa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddbaa330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f35d40cc840 tx=0x7f35d40d1040 comp rx=0 tx=0).stop 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- 172.21.15.116:0/3568963737 shutdown_connections 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb9c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb95bf0 0x55b0ddbaa330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- 172.21.15.116:0/3568963737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb952d0 0x55b0ddb9a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- 172.21.15.116:0/3568963737 >> 172.21.15.116:0/3568963737 conn(0x55b0ddb8fa00 msgr2=0x55b0ddb90e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- 172.21.15.116:0/3568963737 shutdown_connections 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- 172.21.15.116:0/3568963737 wait complete. 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- start start 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 Processor -- start 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddba6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb980d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b6720 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b6050 con 0x55b0ddb952d0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35e01b32b0 con 0x55b0ddb95bf0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35e83ce740 10 client.0 _ll_register_callbacks cb 0x7f35d0009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.600+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35de7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34266/0 (socket says 172.21.15.116:34266) 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 -- 172.21.15.116:0/2333490718 learned_addr learned my addr 172.21.15.116:0/2333490718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35deffd640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddba6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 -- 172.21.15.116:0/2333490718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 msgr2=0x55b0ddb980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 -- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 msgr2=0x55b0ddba6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddba6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 -- 172.21.15.116:0/2333490718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ddb98bd0 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35de7fc640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35deffd640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddba6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35df7fe640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f35d802c7a0 tx=0x7f35d802c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08e5f2243d897b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8039040 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35d409b800 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.601+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f35d80327f0 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f35d8043040 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 10 client.18732 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b0ddc14d10 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d409b800 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f35d80489e0 con 0x55b0ddb9e7d0 2024-03-19T02:38:33.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 10 client.18732 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 10 client.18732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 20 client.18732 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddc15050 0x55b0ddc17440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 4 client.18732 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0ddb059f0 con 0x55b0ddc15050 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.602+0000 7f35deffd640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddc15050 0x55b0ddc17440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35c2ff5640 10 client.18732 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35deffd640 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddc15050 0x55b0ddc17440 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2429ec32e7fcc67d server_cookie=c4ab2ad1f06730ed in_seq=0 out_seq=0 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35c2ff5640 1 -- 172.21.15.116:0/2333490718 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0ddb059f0 con 0x55b0ddc15050 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35c2ff5640 10 client.18732 handle_command_reply: tid=0 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 client.18732 shutdown 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 msgr2=0x55b0ddb98610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f35d802c7a0 tx=0x7f35d802c190 comp rx=0 tx=0).stop 2024-03-19T02:38:33.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 shutdown_connections 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b0ddc15050 0x55b0ddc17440 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0ddb9e7d0 0x55b0ddb98610 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35c2ff5640 0 client.18732 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0ddb95bf0 0x55b0ddb980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 --2- 172.21.15.116:0/2333490718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0ddb952d0 0x55b0ddba6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 >> 172.21.15.116:0/2333490718 conn(0x55b0ddb8fa00 msgr2=0x55b0ddb9a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 shutdown_connections 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 1 -- 172.21.15.116:0/2333490718 wait complete. 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.603+0000 7f35e83ce740 20 client.18732 trim_cache size 0 max 16384 2024-03-19T02:38:33.610 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:33.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 msgr2=0x7f35c80a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 0x7f35c80a4ef0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b0ddb94da0 tx=0x7f35d0005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:33.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 msgr2=0x7f35e0109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f35d400e9f0 tx=0x7f35d400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 shutdown_connections 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f35c80a2a30 0x7f35c80a4ef0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e010bd80 0x7f35e0109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35e0103360 0x7f35e0109620 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 --2- 172.21.15.116:0/1600900235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35e0102a40 0x7f35e010b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 >> 172.21.15.116:0/1600900235 conn(0x7f35e00778d0 msgr2=0x7f35e00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 shutdown_connections 2024-03-19T02:38:33.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.604+0000 7f35e57e6640 1 -- 172.21.15.116:0/1600900235 wait complete. 2024-03-19T02:38:33.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:38:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 -- 172.21.15.116:0/73274285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 msgr2=0x7fa674106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 --2- 172.21.15.116:0/73274285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa674106a90 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa670009930 tx=0x7fa67002bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 -- 172.21.15.116:0/73274285 shutdown_connections 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 --2- 172.21.15.116:0/73274285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa67410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 --2- 172.21.15.116:0/73274285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 --2- 172.21.15.116:0/73274285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa674106a90 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.737+0000 7fa67bab2640 1 -- 172.21.15.116:0/73274285 >> 172.21.15.116:0/73274285 conn(0x7fa6740778d0 msgr2=0x7fa674077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.738+0000 7fa67bab2640 1 -- 172.21.15.116:0/73274285 shutdown_connections 2024-03-19T02:38:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.738+0000 7fa67bab2640 1 -- 172.21.15.116:0/73274285 wait complete. 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.738+0000 7fa67bab2640 1 Processor -- start 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.738+0000 7fa67bab2640 1 -- start start 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa674199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa674110970 con 0x7fa6741066b0 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741107f0 con 0x7fa6741006a0 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67bab2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa674110af0 con 0x7fa674101060 2024-03-19T02:38:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67904d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa674199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67904d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa674199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47356/0 (socket says 172.21.15.116:47356) 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34286/0 (socket says 172.21.15.116:34286) 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67904d640 1 -- 172.21.15.116:0/3994394140 learned_addr learned my addr 172.21.15.116:0/3994394140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 -- 172.21.15.116:0/3994394140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 msgr2=0x7fa674199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa674199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 -- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 msgr2=0x7fa6741a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 -- 172.21.15.116:0/3994394140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6741a4390 con 0x7fa6741006a0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67a04f640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.739+0000 7fa67984e640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674198dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa670009230 tx=0x7fa6700045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.740+0000 7fa66affd640 1 -- 172.21.15.116:0/3994394140 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670043020 con 0x7fa6741006a0 2024-03-19T02:38:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.740+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa674196bc0 con 0x7fa6741006a0 2024-03-19T02:38:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.740+0000 7fa66affd640 1 -- 172.21.15.116:0/3994394140 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa67002ca70 con 0x7fa6741006a0 2024-03-19T02:38:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.740+0000 7fa66affd640 1 -- 172.21.15.116:0/3994394140 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700039b0 con 0x7fa6741006a0 2024-03-19T02:38:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.740+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6741970e0 con 0x7fa6741006a0 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.742+0000 7fa66affd640 1 -- 172.21.15.116:0/3994394140 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa670046020 con 0x7fa6741006a0 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa66affd640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 0x7fa6600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa66affd640 1 -- 172.21.15.116:0/3994394140 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fa6700f1040 con 0x7fa6741006a0 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 Processor -- start 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67904d640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 0x7fa6600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 -- start start 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 0x55ec8ae71aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec8adc34b0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec8ad6bd70 con 0x55ec8ae69400 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67e779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ec8adbf550 con 0x55ec8ae71fe0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67904d640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 0x7fa6600a4e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa6640095f0 tx=0x7fa664005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34288/0 (socket says 172.21.15.116:34288) 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67984e640 1 -- 172.21.15.116:0/3645177262 learned_addr learned my addr 172.21.15.116:0/3645177262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67904d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 0x55ec8ae71aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.743+0000 7fa67a04f640 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 -- 172.21.15.116:0/3645177262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae71fe0 msgr2=0x55ec8ae743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 -- 172.21.15.116:0/3645177262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 msgr2=0x55ec8ae71aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 0x55ec8ae71aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 -- 172.21.15.116:0/3645177262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec8ad8e210 con 0x55ec8ae68ae0 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67904d640 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 0x55ec8ae71aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa67984e640 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa6700bd0d0 tx=0x7fa67002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4e1ece7c595666 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa65ffff640 1 -- 172.21.15.116:0/3645177262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700c1520 con 0x55ec8ae68ae0 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa65ffff640 1 -- 172.21.15.116:0/3645177262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6700c5b70 con 0x55ec8ae68ae0 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.744+0000 7fa65ffff640 1 -- 172.21.15.116:0/3645177262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700c5920 con 0x55ec8ae68ae0 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- 172.21.15.116:0/3645177262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 msgr2=0x55ec8ae68ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa6700bd0d0 tx=0x7fa67002bc30 comp rx=0 tx=0).stop 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- 172.21.15.116:0/3645177262 shutdown_connections 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae69400 0x55ec8ae71aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 --2- 172.21.15.116:0/3645177262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae68ec0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- 172.21.15.116:0/3645177262 >> 172.21.15.116:0/3645177262 conn(0x55ec8ae63210 msgr2=0x55ec8ae635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- 172.21.15.116:0/3645177262 shutdown_connections 2024-03-19T02:38:33.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- 172.21.15.116:0/3645177262 wait complete. 2024-03-19T02:38:33.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.745+0000 7fa67e779740 1 -- start start 2024-03-19T02:38:33.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 Processor -- start 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec8ae74b90 con 0x55ec8ae69400 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec8ad7f310 con 0x55ec8ae71fe0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ec8ae74d10 con 0x55ec8ae68ae0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67e779740 10 client.0 _ll_register_callbacks cb 0x55ec8ad732d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67984e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56480/0 (socket says 172.21.15.116:56480) 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67a04f640 1 -- 172.21.15.116:0/2624830172 learned_addr learned my addr 172.21.15.116:0/2624830172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34296/0 (socket says 172.21.15.116:34296) 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67984e640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47376/0 (socket says 172.21.15.116:47376) 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 -- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 msgr2=0x55ec8ae6eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 -- 172.21.15.116:0/2624830172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 msgr2=0x55ec8ae6fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67904d640 1 -- 172.21.15.116:0/2624830172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec8aad73f0 con 0x55ec8ae69400 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.746+0000 7fa67a04f640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa67904d640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fa6640098e0 tx=0x7fa664004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a971274de1d8156 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa65d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66402b060 con 0x55ec8ae69400 2024-03-19T02:38:33.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6700c5b70 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fa66401cae0 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa66401cce0 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa67e779740 10 client.18744 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.747+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ec8aba17d0 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6700c5b70 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa66401b390 con 0x55ec8ae69400 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 10 client.18744 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 10 client.18744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 20 client.18744 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8ae82030 0x55ec8ae84460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 4 client.18744 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec8ae213e0 con 0x55ec8ae82030 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67984e640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8ae82030 0x55ec8ae84460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa65d7fa640 10 client.18744 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.748+0000 7fa67984e640 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8ae82030 0x55ec8ae84460 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=abc631fc12e1243b server_cookie=46b2799bc3d30890 in_seq=0 out_seq=0 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.749+0000 7fa65d7fa640 1 -- 172.21.15.116:0/2624830172 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec8ae213e0 con 0x55ec8ae82030 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.749+0000 7fa65d7fa640 10 client.18744 handle_command_reply: tid=0 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.749+0000 7fa67e779740 1 client.18744 shutdown 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 msgr2=0x55ec8ae6f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fa6640098e0 tx=0x7fa664004730 comp rx=0 tx=0).stop 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 shutdown_connections 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8ae82030 0x55ec8ae84460 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae6fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae69400 0x55ec8ae6f530 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa65d7fa640 0 client.18744 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 --2- 172.21.15.116:0/2624830172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 >> 172.21.15.116:0/2624830172 conn(0x55ec8ae63210 msgr2=0x55ec8ae59ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 shutdown_connections 2024-03-19T02:38:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 1 -- 172.21.15.116:0/2624830172 wait complete. 2024-03-19T02:38:33.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.750+0000 7fa67e779740 20 client.18744 trim_cache size 0 max 16384 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.765+0000 7fa67e779740 1 Processor -- start 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.765+0000 7fa67e779740 1 -- start start 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae89d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741aca40 con 0x55ec8ae68ae0 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741ae580 con 0x55ec8ae71fe0 2024-03-19T02:38:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67e779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741ad110 con 0x55ec8ae69400 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae89d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67a04f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae89d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56492/0 (socket says 172.21.15.116:56492) 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47384/0 (socket says 172.21.15.116:47384) 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34300/0 (socket says 172.21.15.116:34300) 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67a04f640 1 -- 172.21.15.116:0/3814922900 learned_addr learned my addr 172.21.15.116:0/3814922900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 -- 172.21.15.116:0/3814922900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 msgr2=0x55ec8ae89d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae89d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 -- 172.21.15.116:0/3814922900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 msgr2=0x55ec8ae71720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 -- 172.21.15.116:0/3814922900 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec8ae7dd70 con 0x55ec8ae69400 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67984e640 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.766+0000 7fa67904d640 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa66401b800 tx=0x7fa66401c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54d66593bbf57667 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.767+0000 7fa65f7fe640 1 -- 172.21.15.116:0/3814922900 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66401ce80 con 0x55ec8ae69400 2024-03-19T02:38:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.767+0000 7fa65f7fe640 1 -- 172.21.15.116:0/3814922900 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa66401cae0 con 0x55ec8ae69400 2024-03-19T02:38:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.767+0000 7fa65f7fe640 1 -- 172.21.15.116:0/3814922900 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66401bb70 con 0x55ec8ae69400 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- 172.21.15.116:0/3814922900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 msgr2=0x55ec8ae7d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa66401b800 tx=0x7fa66401c7d0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- 172.21.15.116:0/3814922900 shutdown_connections 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8ae89d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae7d830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 --2- 172.21.15.116:0/3814922900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- 172.21.15.116:0/3814922900 >> 172.21.15.116:0/3814922900 conn(0x55ec8ae63210 msgr2=0x55ec8ae635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- 172.21.15.116:0/3814922900 shutdown_connections 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- 172.21.15.116:0/3814922900 wait complete. 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 -- start start 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.768+0000 7fa67e779740 1 Processor -- start 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae6c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8aee73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741adeb0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741ad7e0 con 0x55ec8ae71fe0 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6741aaa40 con 0x55ec8ae69400 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67e779740 10 client.0 _ll_register_callbacks cb 0x7fa664009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67904d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae6c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34306/0 (socket says 172.21.15.116:34306) 2024-03-19T02:38:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 -- 172.21.15.116:0/4206636886 learned_addr learned my addr 172.21.15.116:0/4206636886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67a04f640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8aee73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 -- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 msgr2=0x55ec8ae6c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae6c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 -- 172.21.15.116:0/4206636886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 msgr2=0x55ec8aee73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8aee73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 -- 172.21.15.116:0/4206636886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec8aee78e0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67904d640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae6c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa67984e640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fa6700f1360 tx=0x7fa6700bd2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93de1f4b687819e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.769+0000 7fa65d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700c1520 con 0x55ec8ae68ae0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fa6700b9a40 con 0x55ec8ae68ae0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa66401cae0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa6700d6280 con 0x55ec8ae68ae0 2024-03-19T02:38:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 10 client.18750 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ec8aee8bb0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa66401cae0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa6700a2be0 con 0x55ec8ae68ae0 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 10 client.18750 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 10 client.18750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 20 client.18750 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8aee90f0 0x55ec8aeeb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 4 client.18750 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec8add09f0 con 0x55ec8aee90f0 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.770+0000 7fa67904d640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8aee90f0 0x55ec8aeeb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa65d7fa640 10 client.18750 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67904d640 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8aee90f0 0x55ec8aeeb4e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76188a73b1f322ea server_cookie=c00b658d5848eb1 in_seq=0 out_seq=0 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa65d7fa640 1 -- 172.21.15.116:0/4206636886 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec8add09f0 con 0x55ec8aee90f0 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa65d7fa640 10 client.18750 handle_command_reply: tid=0 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 client.18750 shutdown 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 msgr2=0x55ec8ae6c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fa6700f1360 tx=0x7fa6700bd2b0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 shutdown_connections 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55ec8aee90f0 0x55ec8aeeb4e0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec8ae71fe0 0x55ec8aee73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec8ae69400 0x55ec8ae6c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 --2- 172.21.15.116:0/4206636886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec8ae68ae0 0x55ec8ae6c2d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 >> 172.21.15.116:0/4206636886 conn(0x55ec8ae63210 msgr2=0x55ec8ae59310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa65d7fa640 0 client.18750 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:38:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 shutdown_connections 2024-03-19T02:38:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.771+0000 7fa67e779740 1 -- 172.21.15.116:0/4206636886 wait complete. 2024-03-19T02:38:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67e779740 20 client.18750 trim_cache size 0 max 16384 2024-03-19T02:38:33.778 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 msgr2=0x7fa6600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 0x7fa6600a4e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa6640095f0 tx=0x7fa664005f90 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 msgr2=0x7fa674198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674198dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa670009230 tx=0x7fa6700045e0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 shutdown_connections 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6600a2970 0x7fa6600a4e30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6741066b0 0x7fa6741a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa674101060 0x7fa674199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 --2- 172.21.15.116:0/3994394140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6741006a0 0x7fa674198dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 >> 172.21.15.116:0/3994394140 conn(0x7fa6740778d0 msgr2=0x7fa674103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 shutdown_connections 2024-03-19T02:38:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.772+0000 7fa67bab2640 1 -- 172.21.15.116:0/3994394140 wait complete. 2024-03-19T02:38:33.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:38:33.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 348613 3.11 MB/sec execute 476 sec latency 36.187 ms 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.899+0000 7fef1f977640 1 -- 172.21.15.116:0/1794205429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18102a40 msgr2=0x7fef18102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.899+0000 7fef1f977640 1 --2- 172.21.15.116:0/1794205429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18102a40 0x7fef18102e20 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fef14009900 tx=0x7fef1402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 -- 172.21.15.116:0/1794205429 shutdown_connections 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 --2- 172.21.15.116:0/1794205429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1810bd80 0x7fef1810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 --2- 172.21.15.116:0/1794205429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef18103360 0x7fef1810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 --2- 172.21.15.116:0/1794205429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18102a40 0x7fef18102e20 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 -- 172.21.15.116:0/1794205429 >> 172.21.15.116:0/1794205429 conn(0x7fef180778d0 msgr2=0x7fef18077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 -- 172.21.15.116:0/1794205429 shutdown_connections 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 -- 172.21.15.116:0/1794205429 wait complete. 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 Processor -- start 2024-03-19T02:38:33.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 -- start start 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 0x7fef18198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.900+0000 7fef1f977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 0x7fef1819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef18111a30 con 0x7fef18103360 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef181118b0 con 0x7fef1810bd80 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef18111bb0 con 0x7fef18102a40 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34332/0 (socket says 172.21.15.116:34332) 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 -- 172.21.15.116:0/4191811362 learned_addr learned my addr 172.21.15.116:0/4191811362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1d713640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 0x7fef18198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1df14640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 0x7fef1819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 -- 172.21.15.116:0/4191811362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 msgr2=0x7fef18198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 0x7fef18198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 -- 172.21.15.116:0/4191811362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 msgr2=0x7fef1819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 0x7fef1819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 -- 172.21.15.116:0/4191811362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1819d000 con 0x7fef18103360 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1d713640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 0x7fef18198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1df14640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 0x7fef1819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fef0800ea60 tx=0x7fef0800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef0e7fc640 1 -- 172.21.15.116:0/4191811362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0800cde0 con 0x7fef18103360 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef0e7fc640 1 -- 172.21.15.116:0/4191811362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef080035f0 con 0x7fef18103360 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1819d2f0 con 0x7fef18103360 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef0e7fc640 1 -- 172.21.15.116:0/4191811362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08010690 con 0x7fef18103360 2024-03-19T02:38:33.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.901+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef181a4b80 con 0x7fef18103360 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.904+0000 7fef0e7fc640 1 -- 172.21.15.116:0/4191811362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fef080037a0 con 0x7fef18103360 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.904+0000 7fef221fd740 1 Processor -- start 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.904+0000 7fef0e7fc640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.904+0000 7fef221fd740 1 -- start start 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef0e7fc640 1 -- 172.21.15.116:0/4191811362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fef080cf530 con 0x7fef18103360 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1df14640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffaff9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb0066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 0x55dffb008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf5c4b0 con 0x55dffb006be0 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf04d70 con 0x55dffaff98a0 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef221fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf58550 con 0x55dffaff8f80 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1df14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 0x55dffb008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1df14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 0x55dffb008fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34338/0 (socket says 172.21.15.116:34338) 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1d713640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffaff9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb0066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1df14640 1 -- 172.21.15.116:0/720288338 learned_addr learned my addr 172.21.15.116:0/720288338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 -- 172.21.15.116:0/720288338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 msgr2=0x55dffaff9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffaff9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 -- 172.21.15.116:0/720288338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 msgr2=0x55dffb008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 0x55dffb008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 -- 172.21.15.116:0/720288338 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffaf27210 con 0x55dffaff98a0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1df14640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef100137b0 tx=0x7fef10013bb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.905+0000 7fef1cf12640 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb0066a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef080cc8a0 tx=0x7fef0809b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b583562ead26b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef037fe640 1 -- 172.21.15.116:0/720288338 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0809beb0 con 0x55dffaff98a0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef037fe640 1 -- 172.21.15.116:0/720288338 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0809b7b0 con 0x55dffaff98a0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef037fe640 1 -- 172.21.15.116:0/720288338 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08097e20 con 0x55dffaff98a0 2024-03-19T02:38:33.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 -- 172.21.15.116:0/720288338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 msgr2=0x55dffb0066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb0066a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef080cc8a0 tx=0x7fef0809b600 comp rx=0 tx=0).stop 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 -- 172.21.15.116:0/720288338 shutdown_connections 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffb006be0 0x55dffb008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb0066a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 --2- 172.21.15.116:0/720288338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffaff9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 -- 172.21.15.116:0/720288338 >> 172.21.15.116:0/720288338 conn(0x55dffaff3a00 msgr2=0x55dffaff3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 -- 172.21.15.116:0/720288338 shutdown_connections 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.906+0000 7fef221fd740 1 -- 172.21.15.116:0/720288338 wait complete. 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 -- start start 2024-03-19T02:38:33.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 Processor -- start 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 0x55dffb0045d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dffb009a20 con 0x55dffaff98a0 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf18310 con 0x55dffb006be0 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dffb009ba0 con 0x55dffaff8f80 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef221fd740 10 client.0 _ll_register_callbacks cb 0x55dffaf0c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1cf12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1d713640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1df14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 0x55dffb0045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1d713640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47424/0 (socket says 172.21.15.116:47424) 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1cf12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34354/0 (socket says 172.21.15.116:34354) 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1df14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 0x55dffb0045d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:56548/0 (socket says 172.21.15.116:56548) 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.907+0000 7fef1d713640 1 -- 172.21.15.116:0/2950073811 learned_addr learned my addr 172.21.15.116:0/2950073811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 -- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 msgr2=0x55dffb0045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 0x55dffb0045d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 -- 172.21.15.116:0/2950073811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 msgr2=0x55dffb004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 -- 172.21.15.116:0/2950073811 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dffac703f0 con 0x55dffaff8f80 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1cf12640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef1d713640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fef14009900 tx=0x7fef1402cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1852c3ee6230090b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14044070 con 0x55dffaff8f80 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef0809b7b0 con 0x55dffaff8f80 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fef140035f0 con 0x55dffaff8f80 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef1402c690 con 0x55dffaff8f80 2024-03-19T02:38:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef221fd740 10 client.28501 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.908+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dffad3a7d0 con 0x55dffaff8f80 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0809b7b0 con 0x55dffaff8f80 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fef140348f0 con 0x55dffaff8f80 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 10 client.28501 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 10 client.28501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 20 client.28501 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb016ec0 0x55dffb019310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 4 client.28501 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dffafba3e0 con 0x55dffb016ec0 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef1cf12640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb016ec0 0x55dffb019310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef00ff9640 10 client.28501 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.909+0000 7fef1cf12640 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb016ec0 0x55dffb019310 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=173f0d7ffc22c07f server_cookie=8e0a3589485b337d in_seq=0 out_seq=0 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef00ff9640 1 -- 172.21.15.116:0/2950073811 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dffafba3e0 con 0x55dffb016ec0 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef00ff9640 10 client.28501 handle_command_reply: tid=0 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 client.28501 shutdown 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 msgr2=0x55dffb003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fef14009900 tx=0x7fef1402cab0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 shutdown_connections 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb016ec0 0x55dffb019310 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb006be0 0x55dffb0045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef00ff9640 0 client.28501 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff98a0 0x55dffb004090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 --2- 172.21.15.116:0/2950073811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffaff8f80 0x55dffb003b50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 >> 172.21.15.116:0/2950073811 conn(0x55dffaff3a00 msgr2=0x55dffaff5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 shutdown_connections 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 1 -- 172.21.15.116:0/2950073811 wait complete. 2024-03-19T02:38:33.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.911+0000 7fef221fd740 20 client.28501 trim_cache size 0 max 16384 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 Processor -- start 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 -- start start 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef1d713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef1d713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34362/0 (socket says 172.21.15.116:34362) 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef1d713640 1 -- 172.21.15.116:0/269948854 learned_addr learned my addr 172.21.15.116:0/269948854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffaff2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb01ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef181acae0 con 0x55dffaff8f80 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef181ae620 con 0x55dffaff98a0 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef181ad1b0 con 0x55dffb006be0 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef1df14640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb01ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.938+0000 7fef1cf12640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffaff2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 -- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 msgr2=0x55dffb01ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb01ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 -- 172.21.15.116:0/269948854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 msgr2=0x55dffaff2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffaff2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 -- 172.21.15.116:0/269948854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffafc9880 con 0x55dffaff8f80 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1df14640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb01ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef1d713640 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fef14034eb0 tx=0x7fef1402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c700104c8e7a255 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef02ffd640 1 -- 172.21.15.116:0/269948854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14044070 con 0x55dffaff8f80 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef02ffd640 1 -- 172.21.15.116:0/269948854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef140043e0 con 0x55dffaff8f80 2024-03-19T02:38:33.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.939+0000 7fef02ffd640 1 -- 172.21.15.116:0/269948854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14040bd0 con 0x55dffaff8f80 2024-03-19T02:38:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 msgr2=0x55dffb012800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fef14034eb0 tx=0x7fef1402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 shutdown_connections 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb01ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffaff2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 --2- 172.21.15.116:0/269948854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb012800 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 >> 172.21.15.116:0/269948854 conn(0x55dffaff3a00 msgr2=0x55dffaff2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 shutdown_connections 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- 172.21.15.116:0/269948854 wait complete. 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.940+0000 7fef221fd740 1 -- start start 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 Processor -- start 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb00be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb00c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef181adf50 con 0x55dffaff8f80 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef181ad880 con 0x55dffaff98a0 2024-03-19T02:38:33.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef181aaae0 con 0x55dffb006be0 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34378/0 (socket says 172.21.15.116:34378) 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 -- 172.21.15.116:0/4251976388 learned_addr learned my addr 172.21.15.116:0/4251976388 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb00be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1df14640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb00c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef221fd740 10 client.0 _ll_register_callbacks cb 0x7fef14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 -- 172.21.15.116:0/4251976388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 msgr2=0x55dffb00c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb00c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 -- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 msgr2=0x55dffb00be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb00be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.941+0000 7fef1d713640 1 -- 172.21.15.116:0/4251976388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dffb00c880 con 0x55dffaff8f80 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb00be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef140043e0 con 0x55dffaff8f80 2024-03-19T02:38:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef1d713640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fef140045f0 tx=0x7fef14005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c59b3b838e6d292 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef00ff9640 10 client.18768 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef00ff9640 1 -- 172.21.15.116:0/4251976388 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14064050 con 0x55dffaff8f80 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef00ff9640 1 -- 172.21.15.116:0/4251976388 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7fef14034640 con 0x55dffaff8f80 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef00ff9640 1 -- 172.21.15.116:0/4251976388 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fef14044070 con 0x55dffaff8f80 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 10 client.18768 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55dffb07dc60 con 0x55dffaff8f80 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef00ff9640 1 -- 172.21.15.116:0/4251976388 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fef14034880 con 0x55dffaff8f80 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 10 client.18768 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 10 client.18768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.942+0000 7fef221fd740 20 client.18768 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb07e130 0x55dffb080550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef221fd740 4 client.18768 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dffaf699f0 con 0x55dffb07e130 2024-03-19T02:38:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb07e130 0x55dffb080550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef00ff9640 10 client.18768 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef1cf12640 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb07e130 0x55dffb080550 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366aaca95131609d server_cookie=d5c09b0fd9096e5f in_seq=0 out_seq=0 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef00ff9640 1 -- 172.21.15.116:0/4251976388 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dffaf699f0 con 0x55dffb07e130 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef00ff9640 10 client.18768 handle_command_reply: tid=0 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.943+0000 7fef221fd740 1 client.18768 shutdown 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 msgr2=0x55dffb00ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fef140045f0 tx=0x7fef14005d90 comp rx=0 tx=0).stop 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 shutdown_connections 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dffb07e130 0x55dffb080550 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dffb006be0 0x55dffb00c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffaff98a0 0x55dffb00be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 --2- 172.21.15.116:0/4251976388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dffaff8f80 0x55dffb00ee30 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 >> 172.21.15.116:0/4251976388 conn(0x55dffaff3a00 msgr2=0x55dffaff2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef00ff9640 0 client.18768 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 shutdown_connections 2024-03-19T02:38:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 1 -- 172.21.15.116:0/4251976388 wait complete. 2024-03-19T02:38:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.944+0000 7fef221fd740 20 client.18768 trim_cache size 0 max 16384 2024-03-19T02:38:33.951 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:38:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 msgr2=0x7fef040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fef100137b0 tx=0x7fef10013bb0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 msgr2=0x7fef18198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fef0800ea60 tx=0x7fef0800ef20 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 shutdown_connections 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1810bd80 0x7fef1819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef18103360 0x7fef18198550 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 --2- 172.21.15.116:0/4191811362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef18102a40 0x7fef18198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 >> 172.21.15.116:0/4191811362 conn(0x7fef180778d0 msgr2=0x7fef180feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 shutdown_connections 2024-03-19T02:38:33.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:33.945+0000 7fef1f977640 1 -- 172.21.15.116:0/4191811362 wait complete. 2024-03-19T02:38:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:33 smithi154 ceph-mon[35693]: pgmap v335: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:38:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:33 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3912922063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:33 smithi116 ceph-mon[32263]: pgmap v335: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:38:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:33 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3912922063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:34.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:33 smithi162 ceph-mon[35761]: pgmap v335: 129 pgs: 129 active+clean; 377 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-03-19T02:38:34.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:33 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3912922063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:38:34.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 349171 3.11 MB/sec execute 477 sec latency 29.236 ms 2024-03-19T02:38:34.959 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:38:34.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:38:34.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:38:35.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 -- 172.21.15.116:0/3134643540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e74102180 msgr2=0x7f8e7410e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3134643540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e74102180 0x7f8e7410e690 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009cd0 tx=0x7f8e6c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 -- 172.21.15.116:0/3134643540 shutdown_connections 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3134643540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e74102180 0x7f8e7410e690 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3134643540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 0x7f8e74101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3134643540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e741077d0 0x7f8e74107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 -- 172.21.15.116:0/3134643540 >> 172.21.15.116:0/3134643540 conn(0x7f8e740fd250 msgr2=0x7f8e740ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 -- 172.21.15.116:0/3134643540 shutdown_connections 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 -- 172.21.15.116:0/3134643540 wait complete. 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.075+0000 7f8e79b10640 1 Processor -- start 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 -- start start 2024-03-19T02:38:35.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 0x7f8e74073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 0x7f8e74074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74111a90 con 0x7f8e741077d0 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74111910 con 0x7f8e74102180 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e79b10640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74111c10 con 0x7f8e741017c0 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e737fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 0x7f8e74073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34392/0 (socket says 172.21.15.116:34392) 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 -- 172.21.15.116:0/3739764523 learned_addr learned my addr 172.21.15.116:0/3739764523 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 0x7f8e74074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 -- 172.21.15.116:0/3739764523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 msgr2=0x7f8e74073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 0x7f8e74073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 -- 172.21.15.116:0/3739764523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 msgr2=0x7f8e74074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 0x7f8e74074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 -- 172.21.15.116:0/3739764523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e741a6510 con 0x7f8e741077d0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 0x7f8e74074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.076+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c004af0 tx=0x7f8e6c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.077+0000 7f8e70ff9640 1 -- 172.21.15.116:0/3739764523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c039040 con 0x7f8e741077d0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.077+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e741a67a0 con 0x7f8e741077d0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.077+0000 7f8e70ff9640 1 -- 172.21.15.116:0/3739764523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e6c044070 con 0x7f8e741077d0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.077+0000 7f8e70ff9640 1 -- 172.21.15.116:0/3739764523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0042a0 con 0x7f8e741077d0 2024-03-19T02:38:35.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.077+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e741a6ce0 con 0x7f8e741077d0 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.079+0000 7f8e70ff9640 1 -- 172.21.15.116:0/3739764523 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8e6c003970 con 0x7f8e741077d0 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.079+0000 7f8e7c7dd740 1 Processor -- start 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.079+0000 7f8e7c7dd740 1 -- start start 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e70ff9640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 0x7f8e5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5c5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f4b529160 con 0x564f4b5ce880 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f4b4d1e10 con 0x564f4b5c5380 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564f4b525200 con 0x564f4b5c5ca0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 0x7f8e5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e70ff9640 1 -- 172.21.15.116:0/3739764523 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f8e6c0f15f0 con 0x7f8e741077d0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34398/0 (socket says 172.21.15.116:34398) 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e737fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 -- 172.21.15.116:0/2200274389 learned_addr learned my addr 172.21.15.116:0/2200274389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 -- 172.21.15.116:0/2200274389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 msgr2=0x564f4b5ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 -- 172.21.15.116:0/2200274389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 msgr2=0x564f4b5c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e73fff640 1 -- 172.21.15.116:0/2200274389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4b3057e0 con 0x564f4b5ce880 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.080+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.081+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 0x7f8e5c0a4e30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e740755b0 tx=0x7f8e680117b0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.081+0000 7f8e73fff640 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0c5bf0 tx=0x7f8e6c0b9c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e14a5d3c2d3cbca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.081+0000 7f8e59ffb640 1 -- 172.21.15.116:0/2200274389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0bd2e0 con 0x564f4b5ce880 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.081+0000 7f8e59ffb640 1 -- 172.21.15.116:0/2200274389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e6c034700 con 0x564f4b5ce880 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.081+0000 7f8e59ffb640 1 -- 172.21.15.116:0/2200274389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0eebe0 con 0x564f4b5ce880 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2200274389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 msgr2=0x564f4b5d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0c5bf0 tx=0x7f8e6c0b9c20 comp rx=0 tx=0).stop 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2200274389 shutdown_connections 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d0c70 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2200274389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2200274389 >> 172.21.15.116:0/2200274389 conn(0x564f4b5bfab0 msgr2=0x564f4b5bfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2200274389 shutdown_connections 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2200274389 wait complete. 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.082+0000 7f8e7c7dd740 1 -- start start 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 Processor -- start 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5380 0x564f4b5cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 0x564f4b5cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f4b5d1430 con 0x564f4b5c5ca0 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f4b4e51a0 con 0x564f4b5ce880 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564f4b5d15b0 con 0x564f4b5c5380 2024-03-19T02:38:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e737fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5380 0x564f4b5cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34400/0 (socket says 172.21.15.116:34400) 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 -- 172.21.15.116:0/2280137272 learned_addr learned my addr 172.21.15.116:0/2280137272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e7c7dd740 10 client.0 _ll_register_callbacks cb 0x564f4b23cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e73fff640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 0x564f4b5cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 -- 172.21.15.116:0/2280137272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5380 msgr2=0x564f4b5cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5380 0x564f4b5cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 -- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 msgr2=0x564f4b5cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 0x564f4b5cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e72ffd640 1 -- 172.21.15.116:0/2280137272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4b365f70 con 0x564f4b5c5ca0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.083+0000 7f8e73fff640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 0x564f4b5cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8e68012c20 tx=0x7f8e68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5396eea067025b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68012e50 con 0x564f4b5c5ca0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f8e68021420 con 0x564f4b5c5ca0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f4b5366a0 con 0x564f4b5c5ca0 2024-03-19T02:38:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8e68030050 con 0x564f4b5c5ca0 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 10 client.18780 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f4b1fd170 con 0x564f4b5c5ca0 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564f4b5366a0 con 0x564f4b5c5ca0 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8e68029990 con 0x564f4b5c5ca0 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 10 client.18780 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 10 client.18780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 20 client.18780 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b5de920 0x564f4b5e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 4 client.18780 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.084+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f4b5309f0 con 0x564f4b5de920 2024-03-19T02:38:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.085+0000 7f8e737fe640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b5de920 0x564f4b5e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.085+0000 7f8e737fe640 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b5de920 0x564f4b5e0d80 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20795f9f7f08c00 server_cookie=f5fbbf196d931853 in_seq=0 out_seq=0 2024-03-19T02:38:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.085+0000 7f8e52ffd640 10 client.18780 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e52ffd640 1 -- 172.21.15.116:0/2280137272 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f4b5309f0 con 0x564f4b5de920 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e52ffd640 10 client.18780 handle_command_reply: tid=0 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 client.18780 shutdown 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 msgr2=0x564f4b5cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8e68012c20 tx=0x7f8e68004730 comp rx=0 tx=0).stop 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 shutdown_connections 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b5de920 0x564f4b5e0d80 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5ce880 0x564f4b5cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5c5ca0 0x564f4b5cbd90 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/2280137272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5380 0x564f4b5cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 >> 172.21.15.116:0/2280137272 conn(0x564f4b5bfab0 msgr2=0x564f4b5c1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e52ffd640 0 client.18780 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 shutdown_connections 2024-03-19T02:38:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/2280137272 wait complete. 2024-03-19T02:38:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.086+0000 7f8e7c7dd740 20 client.18780 trim_cache size 0 max 16384 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 Processor -- start 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 -- start start 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5da010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741b0e90 con 0x564f4b5ce880 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741b29d0 con 0x564f4b5c5380 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741b1560 con 0x564f4b5c5ca0 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34408/0 (socket says 172.21.15.116:34408) 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e73fff640 1 -- 172.21.15.116:0/3703087057 learned_addr learned my addr 172.21.15.116:0/3703087057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.104+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5da010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e737fe640 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 -- 172.21.15.116:0/3703087057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 msgr2=0x564f4b5da010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5da010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 -- 172.21.15.116:0/3703087057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 msgr2=0x564f4b5cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 -- 172.21.15.116:0/3703087057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4b5c5ad0 con 0x564f4b5ce880 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e73fff640 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0eef60 tx=0x7f8e6c02c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce9a3aaaa4a9236 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e597fa640 1 -- 172.21.15.116:0/3703087057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0098d0 con 0x564f4b5ce880 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e597fa640 1 -- 172.21.15.116:0/3703087057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e6c0bd980 con 0x564f4b5ce880 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.105+0000 7f8e597fa640 1 -- 172.21.15.116:0/3703087057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0b9ff0 con 0x564f4b5ce880 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/3703087057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 msgr2=0x564f4b5e6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0eef60 tx=0x7f8e6c02c910 comp rx=0 tx=0).stop 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/3703087057 shutdown_connections 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5e6650 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5da010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/3703087057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/3703087057 >> 172.21.15.116:0/3703087057 conn(0x564f4b5bfab0 msgr2=0x564f4b5cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/3703087057 shutdown_connections 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/3703087057 wait complete. 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.106+0000 7f8e7c7dd740 1 -- start start 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 Processor -- start 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741b2300 con 0x564f4b5ce880 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741b1c30 con 0x564f4b5c5380 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e741aee90 con 0x564f4b5c5ca0 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e7c7dd740 10 client.0 _ll_register_callbacks cb 0x7f8e68009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34414/0 (socket says 172.21.15.116:34414) 2024-03-19T02:38:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 -- 172.21.15.116:0/1093249205 learned_addr learned my addr 172.21.15.116:0/1093249205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e737fe640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 -- 172.21.15.116:0/1093249205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 msgr2=0x564f4b5d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5d2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 -- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 msgr2=0x564f4b5d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 -- 172.21.15.116:0/1093249205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4b643fa0 con 0x564f4b5ce880 2024-03-19T02:38:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e72ffd640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e73fff640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c02c9a0 tx=0x7f8e6c0b9d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f46b53af92f5eda server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.107+0000 7f8e52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0c1ad0 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7f8e6c0bdbe0 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f4b4aebe0 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c0c6120 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 10 client.18786 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f4b645280 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564f4b4aebe0 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8e6c03c030 con 0x564f4b5ce880 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 10 client.18786 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 10 client.18786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 20 client.18786 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b6454f0 0x564f4b6478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 4 client.18786 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564f4b4faa90 con 0x564f4b6454f0 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.108+0000 7f8e737fe640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b6454f0 0x564f4b6478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.109+0000 7f8e52ffd640 10 client.18786 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.109+0000 7f8e737fe640 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b6454f0 0x564f4b6478e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db61e3f901f7c27f server_cookie=dfd49eaf80ad2350 in_seq=0 out_seq=0 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.111+0000 7f8e52ffd640 1 -- 172.21.15.116:0/1093249205 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564f4b4faa90 con 0x564f4b6454f0 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.111+0000 7f8e52ffd640 10 client.18786 handle_command_reply: tid=0 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.111+0000 7f8e7c7dd740 1 client.18786 shutdown 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.111+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 msgr2=0x564f4b5d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.111+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c02c9a0 tx=0x7f8e6c0b9d50 comp rx=0 tx=0).stop 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 shutdown_connections 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f4b6454f0 0x564f4b6478e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f4b5ce880 0x564f4b5d2ce0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f4b5c5ca0 0x564f4b5d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e52ffd640 0 client.18786 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 --2- 172.21.15.116:0/1093249205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f4b5c5380 0x564f4b5d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 >> 172.21.15.116:0/1093249205 conn(0x564f4b5bfab0 msgr2=0x564f4b5beb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 shutdown_connections 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 1 -- 172.21.15.116:0/1093249205 wait complete. 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e7c7dd740 20 client.18786 trim_cache size 0 max 16384 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:38:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "5c5ecd60-d7d1-47e2-b292-1ac58924d9a7", 2024-03-19T02:38:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:38:35.122 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 msgr2=0x7f8e5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 0x7f8e5c0a4e30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e740755b0 tx=0x7f8e680117b0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 msgr2=0x7f8e740745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c004af0 tx=0x7f8e6c035ce0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 shutdown_connections 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e5c0a2970 0x7f8e5c0a4e30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e741077d0 0x7f8e740745a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e74102180 0x7f8e74074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 --2- 172.21.15.116:0/3739764523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e741017c0 0x7f8e74073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.112+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 >> 172.21.15.116:0/3739764523 conn(0x7f8e740fd250 msgr2=0x7f8e740fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.113+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 shutdown_connections 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.113+0000 7f8e79b10640 1 -- 172.21.15.116:0/3739764523 wait complete. 2024-03-19T02:38:35.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:38:35.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 -- 172.21.15.116:0/3586378712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc1059f0 msgr2=0x7facbc10d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 --2- 172.21.15.116:0/3586378712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc1059f0 0x7facbc10d600 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7facb4009cd0 tx=0x7facb402c140 comp rx=0 tx=0).stop 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 -- 172.21.15.116:0/3586378712 shutdown_connections 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 --2- 172.21.15.116:0/3586378712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc1059f0 0x7facbc10d600 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 --2- 172.21.15.116:0/3586378712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc1019f0 0x7facbc1052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 --2- 172.21.15.116:0/3586378712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc101040 0x7facbc101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 -- 172.21.15.116:0/3586378712 >> 172.21.15.116:0/3586378712 conn(0x7facbc076c40 msgr2=0x7facbc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 -- 172.21.15.116:0/3586378712 shutdown_connections 2024-03-19T02:38:35.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 -- 172.21.15.116:0/3586378712 wait complete. 2024-03-19T02:38:35.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.242+0000 7facc13f3640 1 Processor -- start 2024-03-19T02:38:35.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 -- start start 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc1059f0 0x7facbc1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc19b740 0x7facbc19bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7facbc110960 con 0x7facbc101040 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1107e0 con 0x7facbc19b740 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facc13f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7facbc110ae0 con 0x7facbc1059f0 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34444/0 (socket says 172.21.15.116:34444) 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 -- 172.21.15.116:0/4090808223 learned_addr learned my addr 172.21.15.116:0/4090808223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc19b740 0x7facbc19bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facba7fc640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc1059f0 0x7facbc1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 -- 172.21.15.116:0/4090808223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc1059f0 msgr2=0x7facbc1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc1059f0 0x7facbc1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 -- 172.21.15.116:0/4090808223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc19b740 msgr2=0x7facbc19bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc19b740 0x7facbc19bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.243+0000 7facbaffd640 1 -- 172.21.15.116:0/4090808223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facbc19c230 con 0x7facbc101040 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7facbaffd640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9f0 tx=0x7facac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7faca3fff640 1 -- 172.21.15.116:0/4090808223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac00cd80 con 0x7facbc101040 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facbc19c520 con 0x7facbc101040 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facbc1a3d60 con 0x7facbc101040 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7faca3fff640 1 -- 172.21.15.116:0/4090808223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facac0108e0 con 0x7facbc101040 2024-03-19T02:38:35.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.244+0000 7faca3fff640 1 -- 172.21.15.116:0/4090808223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac018920 con 0x7facbc101040 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.246+0000 7faca3fff640 1 -- 172.21.15.116:0/4090808223 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7facac010a90 con 0x7facbc101040 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 Processor -- start 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 -- start start 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b4010 0x564acc4bec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 0x564acc4c1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564acc417500 con 0x564acc4b4010 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564acc3bfbe0 con 0x564acc4bf140 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facc3fdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564acc4135a0 con 0x564acc4b36f0 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7faca3fff640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 0x7faca40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47538/0 (socket says 172.21.15.116:47538) 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbb7fe640 1 -- 172.21.15.116:0/2861783349 learned_addr learned my addr 172.21.15.116:0/2861783349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 0x564acc4c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbaffd640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b4010 0x564acc4bec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 0x7faca40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 -- 172.21.15.116:0/2861783349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 msgr2=0x564acc4b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 -- 172.21.15.116:0/2861783349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b4010 msgr2=0x564acc4bec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b4010 0x564acc4bec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7faca3fff640 1 -- 172.21.15.116:0/4090808223 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7facac0cf4c0 con 0x7facbc101040 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facba7fc640 1 -- 172.21.15.116:0/2861783349 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acc3e1000 con 0x564acc4bf140 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.247+0000 7facbb7fe640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.248+0000 7facba7fc640 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 0x564acc4c1530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7facbc068950 tx=0x7facb0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f483286fcb9ecd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.248+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 0x7faca40a4e30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7facb4009a60 tx=0x7facb402c140 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.248+0000 7faca17fa640 1 -- 172.21.15.116:0/2861783349 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0004050 con 0x564acc4bf140 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.248+0000 7faca17fa640 1 -- 172.21.15.116:0/2861783349 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facb00054d0 con 0x564acc4bf140 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.248+0000 7faca17fa640 1 -- 172.21.15.116:0/2861783349 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0004c30 con 0x564acc4bf140 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- 172.21.15.116:0/2861783349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 msgr2=0x564acc4c1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 0x564acc4c1530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7facbc068950 tx=0x7facb0011810 comp rx=0 tx=0).stop 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- 172.21.15.116:0/2861783349 shutdown_connections 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4bf140 0x564acc4c1530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b4010 0x564acc4bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2861783349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- 172.21.15.116:0/2861783349 >> 172.21.15.116:0/2861783349 conn(0x564acc4ae160 msgr2=0x564acc4ae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- 172.21.15.116:0/2861783349 shutdown_connections 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- 172.21.15.116:0/2861783349 wait complete. 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- start start 2024-03-19T02:38:35.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 Processor -- start 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4bcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564acc4c1f30 con 0x564acc4b36f0 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564acc3d3160 con 0x564acc4b4010 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564acc4c20b0 con 0x564acc4bf140 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facc3fdc740 10 client.0 _ll_register_callbacks cb 0x564acc40fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34450/0 (socket says 172.21.15.116:34450) 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facbaffd640 1 -- 172.21.15.116:0/2591390539 learned_addr learned my addr 172.21.15.116:0/2591390539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facba7fc640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.249+0000 7facbb7fe640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4bcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 -- 172.21.15.116:0/2591390539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 msgr2=0x564acc4bcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4bcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 -- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 msgr2=0x564acc4bc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 -- 172.21.15.116:0/2591390539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acc1f57d0 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facbaffd640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9c0 tx=0x7facac010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820d58d26c52bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac026040 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7facac0cf170 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb00054d0 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7facac0860b0 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 10 client.18798 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564acc12b3f0 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facb00054d0 con 0x564acc4b36f0 2024-03-19T02:38:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7facac09ae70 con 0x564acc4b36f0 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 10 client.18798 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 10 client.18798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 20 client.18798 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc4b6400 0x564acc4d3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 4 client.18798 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.250+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564acc424a40 con 0x564acc4b6400 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.251+0000 7facba7fc640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc4b6400 0x564acc4d3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.251+0000 7fac9eff5640 10 client.18798 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.251+0000 7facba7fc640 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc4b6400 0x564acc4d3e50 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b93673b3e6f55934 server_cookie=f13803bc641e538e in_seq=0 out_seq=0 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7fac9eff5640 1 -- 172.21.15.116:0/2591390539 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564acc424a40 con 0x564acc4b6400 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7fac9eff5640 10 client.18798 handle_command_reply: tid=0 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 client.18798 shutdown 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 msgr2=0x564acc4bc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9c0 tx=0x7facac010f20 comp rx=0 tx=0).stop 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 shutdown_connections 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc4b6400 0x564acc4d3e50 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4bcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7fac9eff5640 0 client.18798 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 --2- 172.21.15.116:0/2591390539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4bc110 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 >> 172.21.15.116:0/2591390539 conn(0x564acc4ae160 msgr2=0x564acc4afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 shutdown_connections 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 1 -- 172.21.15.116:0/2591390539 wait complete. 2024-03-19T02:38:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.252+0000 7facc3fdc740 20 client.18798 trim_cache size 0 max 16384 2024-03-19T02:38:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 Processor -- start 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 -- start start 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1acab0 con 0x564acc4b36f0 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ae5f0 con 0x564acc4b4010 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facc3fdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ad180 con 0x564acc4bf140 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34454/0 (socket says 172.21.15.116:34454) 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbaffd640 1 -- 172.21.15.116:0/4124801671 learned_addr learned my addr 172.21.15.116:0/4124801671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facba7fc640 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbb7fe640 1 -- 172.21.15.116:0/4124801671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 msgr2=0x564acc4bb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.271+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7facbb7fe640 1 -- 172.21.15.116:0/4124801671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 msgr2=0x564acc4be840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7facbb7fe640 1 -- 172.21.15.116:0/4124801671 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acc0eb0f0 con 0x564acc4bf140 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7facbaffd640 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7facbb7fe640 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4d6e70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7facb4009430 tx=0x7facb4009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=425823c447c8ed4e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7faca0ff9640 1 -- 172.21.15.116:0/4124801671 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4039040 con 0x564acc4bf140 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7faca0ff9640 1 -- 172.21.15.116:0/4124801671 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facb4053070 con 0x564acc4bf140 2024-03-19T02:38:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.272+0000 7faca0ff9640 1 -- 172.21.15.116:0/4124801671 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb404e730 con 0x564acc4bf140 2024-03-19T02:38:35.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- 172.21.15.116:0/4124801671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 msgr2=0x564acc4d6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4d6e70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7facb4009430 tx=0x7facb4009360 comp rx=0 tx=0).stop 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- 172.21.15.116:0/4124801671 shutdown_connections 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4bf140 0x564acc4d6e70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4bb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- 172.21.15.116:0/4124801671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4b36f0 0x564acc4be840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- 172.21.15.116:0/4124801671 >> 172.21.15.116:0/4124801671 conn(0x564acc4ae160 msgr2=0x564acc4ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- 172.21.15.116:0/4124801671 shutdown_connections 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- 172.21.15.116:0/4124801671 wait complete. 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- start start 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 Processor -- start 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4d4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1adf20 con 0x564acc4bf140 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1ad850 con 0x564acc4b4010 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7facbc1aaab0 con 0x564acc4b36f0 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.273+0000 7facc3fdc740 10 client.0 _ll_register_callbacks cb 0x7facac00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34460/0 (socket says 172.21.15.116:34460) 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 -- 172.21.15.116:0/3452066050 learned_addr learned my addr 172.21.15.116:0/3452066050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbaffd640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facba7fc640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4d4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 -- 172.21.15.116:0/3452066050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 msgr2=0x564acc4d4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4d4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 -- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 msgr2=0x564acc4d4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:38:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4d4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 -- 172.21.15.116:0/3452066050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acc536ca0 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facbb7fe640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7facb4033000 tx=0x7facb40099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7eccef2785d4729 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7fac9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb405b070 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9510+0+0 (secure 0 0 0) 0x7facb40044d0 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facb4053070 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7facb4069050 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facc3fdc740 10 client.18804 fetch_fsmap learned FSMap version 30 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564acc537f90 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.274+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facb4053070 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7facb403f070 con 0x564acc4bf140 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 10 client.18804 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 10 client.18804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:38:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 20 client.18804 populate_metadata read hostname 'smithi116' 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc538680 0x564acc53aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 4 client.18804 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564acc41ed90 con 0x564acc538680 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facbaffd640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc538680 0x564acc53aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7fac9eff5640 10 client.18804 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.275+0000 7facbaffd640 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc538680 0x564acc53aaa0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4511d11715268318 server_cookie=c0f3ae107885d708 in_seq=0 out_seq=0 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7fac9eff5640 1 -- 172.21.15.116:0/3452066050 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564acc41ed90 con 0x564acc538680 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7fac9eff5640 10 client.18804 handle_command_reply: tid=0 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 client.18804 shutdown 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 msgr2=0x564acc4d54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7facb4033000 tx=0x7facb40099c0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 shutdown_connections 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564acc538680 0x564acc53aaa0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564acc4bf140 0x564acc4d54f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564acc4b4010 0x564acc4d4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7fac9eff5640 0 client.18804 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 --2- 172.21.15.116:0/3452066050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564acc4b36f0 0x564acc4d4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 >> 172.21.15.116:0/3452066050 conn(0x564acc4ae160 msgr2=0x564acc4bc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 shutdown_connections 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 1 -- 172.21.15.116:0/3452066050 wait complete. 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.277+0000 7facc3fdc740 20 client.18804 trim_cache size 0 max 16384 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:38:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "5c5ecd60-d7d1-47e2-b292-1ac58924d9a7": { 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "5c5ecd60-d7d1-47e2-b292-1ac58924d9a7", 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:38:35.283 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 msgr2=0x7faca40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 0x7faca40a4e30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7facb4009a60 tx=0x7facb402c140 comp rx=0 tx=0).stop 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 msgr2=0x7facbc196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7facac00e9f0 tx=0x7facac00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 shutdown_connections 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faca40a2970 0x7faca40a4e30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facbc19b740 0x7facbc19bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facbc1059f0 0x7facbc1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 --2- 172.21.15.116:0/4090808223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facbc101040 0x7facbc196e80 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:38:35.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 >> 172.21.15.116:0/4090808223 conn(0x7facbc076c40 msgr2=0x7facbc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:38:35.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 shutdown_connections 2024-03-19T02:38:35.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:38:35.278+0000 7facc13f3640 1 -- 172.21.15.116:0/4090808223 wait complete. 2024-03-19T02:38:35.288 INFO:tasks.cephfs.filesystem:scrub status for tag:5c5ecd60-d7d1-47e2-b292-1ac58924d9a7 - {'path': '/', 'tag': '5c5ecd60-d7d1-47e2-b292-1ac58924d9a7', 'options': 'recursive,force'} 2024-03-19T02:38:35.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 349745 3.11 MB/sec execute 478 sec latency 35.302 ms 2024-03-19T02:38:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:35 smithi154 ceph-mon[35693]: pgmap v336: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:38:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:35 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:38:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:38:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:35 smithi116 ceph-mon[32263]: pgmap v336: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:38:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:35 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:38:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:38:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:35 smithi162 ceph-mon[35761]: pgmap v336: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-03-19T02:38:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:35 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:38:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:38:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:36.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 350368 3.11 MB/sec execute 479 sec latency 19.387 ms 2024-03-19T02:38:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:36 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:38:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:36 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:36 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:38:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:36 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:36 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:38:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:36 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:38:37.794 INFO:tasks.workunit.client.0.smithi116.stdout: 1 350950 3.11 MB/sec execute 480 sec latency 18.374 ms 2024-03-19T02:38:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:37 smithi154 ceph-mon[35693]: pgmap v337: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:37 smithi116 ceph-mon[32263]: pgmap v337: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:38.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:37 smithi162 ceph-mon[35761]: pgmap v337: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:38.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 351483 3.12 MB/sec execute 481 sec latency 17.488 ms 2024-03-19T02:38:39.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:38 smithi154 ceph-mon[35693]: pgmap v338: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:38 smithi116 ceph-mon[32263]: pgmap v338: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:38 smithi162 ceph-mon[35761]: pgmap v338: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 351948 3.11 MB/sec execute 482 sec latency 41.411 ms 2024-03-19T02:38:40.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 352568 3.12 MB/sec execute 483 sec latency 19.099 ms 2024-03-19T02:38:41.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 353035 3.11 MB/sec execute 484 sec latency 30.535 ms 2024-03-19T02:38:42.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:41 smithi154 ceph-mon[35693]: pgmap v339: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 123 op/s 2024-03-19T02:38:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:41 smithi116 ceph-mon[32263]: pgmap v339: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 123 op/s 2024-03-19T02:38:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:41 smithi162 ceph-mon[35761]: pgmap v339: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 123 op/s 2024-03-19T02:38:42.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 353627 3.11 MB/sec execute 485 sec latency 21.053 ms 2024-03-19T02:38:43.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 354453 3.12 MB/sec execute 486 sec latency 21.834 ms 2024-03-19T02:38:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:43 smithi154 ceph-mon[35693]: pgmap v340: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-03-19T02:38:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:43 smithi116 ceph-mon[32263]: pgmap v340: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-03-19T02:38:44.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:43 smithi162 ceph-mon[35761]: pgmap v340: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-03-19T02:38:44.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 354819 3.12 MB/sec execute 487 sec latency 23.216 ms 2024-03-19T02:38:45.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:44 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:38:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:44 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:38:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:44 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:38:45.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 355212 3.11 MB/sec execute 488 sec latency 17.848 ms 2024-03-19T02:38:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:38:46.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:45 smithi154 ceph-mon[35693]: pgmap v341: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 120 op/s 2024-03-19T02:38:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:38:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:45 smithi116 ceph-mon[32263]: pgmap v341: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 120 op/s 2024-03-19T02:38:46.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:38:46.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:45 smithi162 ceph-mon[35761]: pgmap v341: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 120 op/s 2024-03-19T02:38:46.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 355768 3.11 MB/sec execute 489 sec latency 36.915 ms 2024-03-19T02:38:47.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 356349 3.11 MB/sec execute 490 sec latency 35.095 ms 2024-03-19T02:38:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:47 smithi154 ceph-mon[35693]: pgmap v342: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:47 smithi116 ceph-mon[32263]: pgmap v342: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:47 smithi162 ceph-mon[35761]: pgmap v342: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:48.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 356994 3.11 MB/sec execute 491 sec latency 21.029 ms 2024-03-19T02:38:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 357675 3.11 MB/sec execute 492 sec latency 20.592 ms 2024-03-19T02:38:50.218 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:49 smithi162 ceph-mon[35761]: pgmap v343: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:50.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:49 smithi154 ceph-mon[35693]: pgmap v343: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:49 smithi116 ceph-mon[32263]: pgmap v343: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-03-19T02:38:50.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 358071 3.11 MB/sec execute 493 sec latency 25.034 ms 2024-03-19T02:38:51.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 358620 3.11 MB/sec execute 494 sec latency 17.993 ms 2024-03-19T02:38:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:51 smithi154 ceph-mon[35693]: pgmap v344: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:38:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:51 smithi116 ceph-mon[32263]: pgmap v344: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:38:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:51 smithi162 ceph-mon[35761]: pgmap v344: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-03-19T02:38:52.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 359078 3.11 MB/sec execute 495 sec latency 47.124 ms 2024-03-19T02:38:53.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 359702 3.11 MB/sec execute 496 sec latency 26.871 ms 2024-03-19T02:38:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:53 smithi154 ceph-mon[35693]: pgmap v345: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:38:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:53 smithi116 ceph-mon[32263]: pgmap v345: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:38:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:53 smithi162 ceph-mon[35761]: pgmap v345: 129 pgs: 129 active+clean; 398 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:38:54.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 360229 3.11 MB/sec execute 497 sec latency 23.674 ms 2024-03-19T02:38:55.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 360808 3.10 MB/sec execute 498 sec latency 19.613 ms 2024-03-19T02:38:56.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:55 smithi154 ceph-mon[35693]: pgmap v346: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:38:56.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:55 smithi116 ceph-mon[32263]: pgmap v346: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:38:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:55 smithi162 ceph-mon[35761]: pgmap v346: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 121 op/s 2024-03-19T02:38:56.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 361558 3.11 MB/sec execute 499 sec latency 17.922 ms 2024-03-19T02:38:57.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 362025 3.11 MB/sec execute 500 sec latency 21.680 ms 2024-03-19T02:38:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:57 smithi154 ceph-mon[35693]: pgmap v347: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:57 smithi116 ceph-mon[32263]: pgmap v347: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:57 smithi162 ceph-mon[35761]: pgmap v347: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:58.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 362402 3.11 MB/sec execute 501 sec latency 36.524 ms 2024-03-19T02:38:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:38:58 smithi154 ceph-mon[35693]: pgmap v348: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:38:58 smithi116 ceph-mon[32263]: pgmap v348: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:38:58 smithi162 ceph-mon[35761]: pgmap v348: 129 pgs: 129 active+clean; 401 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:38:59.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 363005 3.11 MB/sec execute 502 sec latency 16.011 ms 2024-03-19T02:39:00.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 363569 3.11 MB/sec execute 503 sec latency 33.047 ms 2024-03-19T02:39:01.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 364154 3.10 MB/sec execute 504 sec latency 41.270 ms 2024-03-19T02:39:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:01 smithi154 ceph-mon[35693]: pgmap v349: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:01 smithi116 ceph-mon[32263]: pgmap v349: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:02.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:01 smithi162 ceph-mon[35761]: pgmap v349: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:02.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 364956 3.11 MB/sec execute 505 sec latency 44.826 ms 2024-03-19T02:39:03.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 365349 3.11 MB/sec execute 506 sec latency 17.665 ms 2024-03-19T02:39:04.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:03 smithi154 ceph-mon[35693]: pgmap v350: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:03 smithi116 ceph-mon[32263]: pgmap v350: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:04.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:03 smithi162 ceph-mon[35761]: pgmap v350: 129 pgs: 129 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:04.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 365808 3.11 MB/sec execute 507 sec latency 17.187 ms 2024-03-19T02:39:05.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:39:05.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 -- 172.21.15.116:0/2270669223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404100690 msgr2=0x7f5404100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 --2- 172.21.15.116:0/2270669223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404100690 0x7f5404100b10 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009930 tx=0x7f53f802bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 -- 172.21.15.116:0/2270669223 shutdown_connections 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 --2- 172.21.15.116:0/2270669223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404101050 0x7f540410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 --2- 172.21.15.116:0/2270669223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404100690 0x7f5404100b10 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 --2- 172.21.15.116:0/2270669223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 0x7f5404106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 -- 172.21.15.116:0/2270669223 >> 172.21.15.116:0/2270669223 conn(0x7f5404077820 msgr2=0x7f5404077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 -- 172.21.15.116:0/2270669223 shutdown_connections 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.404+0000 7f5409df4640 1 -- 172.21.15.116:0/2270669223 wait complete. 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.405+0000 7f5409df4640 1 Processor -- start 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.405+0000 7f5409df4640 1 -- start start 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.405+0000 7f5409df4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.405+0000 7f5409df4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 0x7f5404199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5409df4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 0x7f54041a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5409df4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5404110960 con 0x7f5404101050 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5409df4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54041107e0 con 0x7f54041066a0 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5409df4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5404110ae0 con 0x7f5404100690 2024-03-19T02:39:05.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50710/0 (socket says 172.21.15.116:50710) 2024-03-19T02:39:05.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5402ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 0x7f5404199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 -- 172.21.15.116:0/2037988484 learned_addr learned my addr 172.21.15.116:0/2037988484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5403fff640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 0x7f54041a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 -- 172.21.15.116:0/2037988484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 msgr2=0x7f54041a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 0x7f54041a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 -- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 msgr2=0x7f5404199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 0x7f5404199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 -- 172.21.15.116:0/2037988484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54041a4400 con 0x7f5404100690 2024-03-19T02:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5402ffd640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 0x7f5404199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f54037fe640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53f400e9f0 tx=0x7f53f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.406+0000 7f5400ff9640 1 -- 172.21.15.116:0/2037988484 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f400cd80 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.407+0000 7f5400ff9640 1 -- 172.21.15.116:0/2037988484 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53f40108e0 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.407+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54041a45d0 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.407+0000 7f5400ff9640 1 -- 172.21.15.116:0/2037988484 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f4018920 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.407+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54041a4af0 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f5400ff9640 1 -- 172.21.15.116:0/2037988484 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f53f4010a90 con 0x7f5404100690 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 Processor -- start 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 -- start start 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3293e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557d31f7500 con 0x5557d3294350 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5557d319fbe0 con 0x5557d329cf30 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f540c9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557d31f35a0 con 0x5557d3293a30 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f5403fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3293e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52476/0 (socket says 172.21.15.116:52476) 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f54037fe640 1 -- 172.21.15.116:0/1439472321 learned_addr learned my addr 172.21.15.116:0/1439472321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f5400ff9640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 0x7f53ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.409+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5403fff640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 0x7f53ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5400ff9640 1 -- 172.21.15.116:0/2037988484 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f53f40cf540 con 0x7f5404100690 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 -- 172.21.15.116:0/1439472321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 msgr2=0x5557d3293e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3293e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 -- 172.21.15.116:0/1439472321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 msgr2=0x5557d329c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 -- 172.21.15.116:0/1439472321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557d31c1000 con 0x5557d329cf30 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5403fff640 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3293e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329f320 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009230 tx=0x7f53f80095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61147f3d0d0aaa75 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f5403fff640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 0x7f53ec0a4e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53fc005c40 tx=0x7f53fc004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f53e9ffb640 1 -- 172.21.15.116:0/1439472321 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8045070 con 0x5557d329cf30 2024-03-19T02:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f53e9ffb640 1 -- 172.21.15.116:0/1439472321 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53f802c7e0 con 0x5557d329cf30 2024-03-19T02:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.410+0000 7f53e9ffb640 1 -- 172.21.15.116:0/1439472321 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8003a00 con 0x5557d329cf30 2024-03-19T02:39:05.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 -- 172.21.15.116:0/1439472321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 msgr2=0x5557d329f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329f320 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009230 tx=0x7f53f80095f0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 -- 172.21.15.116:0/1439472321 shutdown_connections 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329f320 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1439472321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3293e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 -- 172.21.15.116:0/1439472321 >> 172.21.15.116:0/1439472321 conn(0x5557d328e160 msgr2=0x5557d328e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 -- 172.21.15.116:0/1439472321 shutdown_connections 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.411+0000 7f540c9e9740 1 -- 172.21.15.116:0/1439472321 wait complete. 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 -- start start 2024-03-19T02:39:05.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 Processor -- start 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3299f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557d329fcc0 con 0x5557d3294350 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5557d31b3160 con 0x5557d329cf30 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557d329fe40 con 0x5557d3293a30 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f540c9e9740 10 client.0 _ll_register_callbacks cb 0x5557d31efa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f5402ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f54037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3299f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f5402ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52488/0 (socket says 172.21.15.116:52488) 2024-03-19T02:39:05.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f5402ffd640 1 -- 172.21.15.116:0/1614234971 learned_addr learned my addr 172.21.15.116:0/1614234971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.412+0000 7f5403fff640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 -- 172.21.15.116:0/1614234971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 msgr2=0x5557d3299f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3299f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 -- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 msgr2=0x5557d329a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 -- 172.21.15.116:0/1614234971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557d2fd57d0 con 0x5557d3294350 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f54037fe640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3299f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:05.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009900 tx=0x7f53f8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1e2b51cee5da0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f53e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8051070 con 0x5557d3294350 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f53f803fc80 con 0x5557d3294350 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53f802c7e0 con 0x5557d3294350 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f53f805d050 con 0x5557d3294350 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f540c9e9740 10 client.18816 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:05.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5557d2f0b3f0 con 0x5557d3294350 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.413+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53f802c7e0 con 0x5557d3294350 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f53f8034440 con 0x5557d3294350 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 10 client.18816 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 10 client.18816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 20 client.18816 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d32ad0c0 0x5557d32af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 4 client.18816 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557d3204a40 con 0x5557d32ad0c0 2024-03-19T02:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f54037fe640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d32ad0c0 0x5557d32af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f53e2ffd640 10 client.18816 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.414+0000 7f54037fe640 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d32ad0c0 0x5557d32af4b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d09a3cffb9cfc6d1 server_cookie=2a534f4dddaa8543 in_seq=0 out_seq=0 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f53e2ffd640 1 -- 172.21.15.116:0/1614234971 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557d3204a40 con 0x5557d32ad0c0 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f53e2ffd640 10 client.18816 handle_command_reply: tid=0 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 client.18816 shutdown 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 msgr2=0x5557d329a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009900 tx=0x7f53f8005b50 comp rx=0 tx=0).stop 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 shutdown_connections 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d32ad0c0 0x5557d32af4b0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d329a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3294350 0x5557d329a480 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f53e2ffd640 0 client.18816 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1614234971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3293a30 0x5557d3299f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 >> 172.21.15.116:0/1614234971 conn(0x5557d328e160 msgr2=0x5557d328feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 shutdown_connections 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 1 -- 172.21.15.116:0/1614234971 wait complete. 2024-03-19T02:39:05.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.415+0000 7f540c9e9740 20 client.18816 trim_cache size 0 max 16384 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 Processor -- start 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 -- start start 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d32a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d32b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54041aece0 con 0x5557d3293a30 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54041b0820 con 0x5557d329cf30 2024-03-19T02:39:05.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f540c9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54041af3b0 con 0x5557d3294350 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52496/0 (socket says 172.21.15.116:52496) 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f54037fe640 1 -- 172.21.15.116:0/1155509265 learned_addr learned my addr 172.21.15.116:0/1155509265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f5402ffd640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d32a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.433+0000 7f5403fff640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d32b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 -- 172.21.15.116:0/1155509265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 msgr2=0x5557d32a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d32a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 -- 172.21.15.116:0/1155509265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 msgr2=0x5557d32b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d32b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 -- 172.21.15.116:0/1155509265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557d329f690 con 0x5557d3293a30 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f5403fff640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d32b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f54037fe640 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f53f409b520 tx=0x7f53f40d1a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b51db1d28f13a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f53e97fa640 1 -- 172.21.15.116:0/1155509265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f4026040 con 0x5557d3293a30 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f53e97fa640 1 -- 172.21.15.116:0/1155509265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53f4010de0 con 0x5557d3293a30 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.434+0000 7f53e97fa640 1 -- 172.21.15.116:0/1155509265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f4099b20 con 0x5557d3293a30 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- 172.21.15.116:0/1155509265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 msgr2=0x5557d329a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f53f409b520 tx=0x7f53f40d1a50 comp rx=0 tx=0).stop 2024-03-19T02:39:05.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- 172.21.15.116:0/1155509265 shutdown_connections 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d32b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d32a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 --2- 172.21.15.116:0/1155509265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d329a790 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- 172.21.15.116:0/1155509265 >> 172.21.15.116:0/1155509265 conn(0x5557d328e160 msgr2=0x5557d328c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- 172.21.15.116:0/1155509265 shutdown_connections 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- 172.21.15.116:0/1155509265 wait complete. 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 -- start start 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.435+0000 7f540c9e9740 1 Processor -- start 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d3297360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d33120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54041b0150 con 0x5557d3293a30 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54041afa80 con 0x5557d329cf30 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54041acce0 con 0x5557d3294350 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 10 client.0 _ll_register_callbacks cb 0x7f53f40978f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52508/0 (socket says 172.21.15.116:52508) 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 -- 172.21.15.116:0/3697135207 learned_addr learned my addr 172.21.15.116:0/3697135207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f5403fff640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d33120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f5402ffd640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d3297360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 -- 172.21.15.116:0/3697135207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 msgr2=0x5557d3297360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d3297360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 -- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 msgr2=0x5557d33120b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d33120b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 -- 172.21.15.116:0/3697135207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557d2ecb0f0 con 0x5557d3293a30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f54037fe640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f53f40ccde0 tx=0x7f53f4009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c54b77c0a9a474 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f53e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f409b180 con 0x5557d3293a30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f53f4097570 con 0x5557d3293a30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5557d31fed90 con 0x5557d3293a30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.436+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f53f409b820 con 0x5557d3293a30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 10 client.18822 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:05.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5557d33134f0 con 0x5557d3293a30 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5557d31fed90 con 0x5557d3293a30 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f53f4010430 con 0x5557d3293a30 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 10 client.18822 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 10 client.18822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 20 client.18822 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d3313bb0 0x5557d3315fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 4 client.18822 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5557d317c540 con 0x5557d3313bb0 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f5402ffd640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d3313bb0 0x5557d3315fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f53e2ffd640 10 client.18822 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.437+0000 7f5402ffd640 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d3313bb0 0x5557d3315fa0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31604f270e588c38 server_cookie=8cab71cde4cdab62 in_seq=0 out_seq=0 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f53e2ffd640 1 -- 172.21.15.116:0/3697135207 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5557d317c540 con 0x5557d3313bb0 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f53e2ffd640 10 client.18822 handle_command_reply: tid=0 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 client.18822 shutdown 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 msgr2=0x5557d3296e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f53f40ccde0 tx=0x7f53f4009770 comp rx=0 tx=0).stop 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 shutdown_connections 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5557d3313bb0 0x5557d3315fa0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557d329cf30 0x5557d33120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557d3294350 0x5557d3297360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 --2- 172.21.15.116:0/3697135207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557d3293a30 0x5557d3296e20 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f53e2ffd640 0 client.18822 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 >> 172.21.15.116:0/3697135207 conn(0x5557d328e160 msgr2=0x5557d328d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 shutdown_connections 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 1 -- 172.21.15.116:0/3697135207 wait complete. 2024-03-19T02:39:05.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f540c9e9740 20 client.18822 trim_cache size 0 max 16384 2024-03-19T02:39:05.451 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:39:05.452 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:39:05.452 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:05.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 msgr2=0x7f53ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 0x7f53ec0a4e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53fc005c40 tx=0x7f53fc004ab0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 msgr2=0x7f5404198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.439+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53f400e9f0 tx=0x7f53f400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 shutdown_connections 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53ec0a2970 0x7f53ec0a4e30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54041066a0 0x7f54041a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101050 0x7f5404199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 --2- 172.21.15.116:0/2037988484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5404100690 0x7f5404198e40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 >> 172.21.15.116:0/2037988484 conn(0x7f5404077820 msgr2=0x7f5404103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 shutdown_connections 2024-03-19T02:39:05.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.440+0000 7f5409df4640 1 -- 172.21.15.116:0/2037988484 wait complete. 2024-03-19T02:39:05.455 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:39:05.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 -- 172.21.15.116:0/4049243680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 msgr2=0x7f011c10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 --2- 172.21.15.116:0/4049243680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c10e6c0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0114009ca0 tx=0x7f011402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 -- 172.21.15.116:0/4049243680 shutdown_connections 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 --2- 172.21.15.116:0/4049243680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c10e6c0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 --2- 172.21.15.116:0/4049243680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100f20 0x7f011c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 --2- 172.21.15.116:0/4049243680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c100570 0x7f011c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 -- 172.21.15.116:0/4049243680 >> 172.21.15.116:0/4049243680 conn(0x7f011c075a40 msgr2=0x7f011c075e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 -- 172.21.15.116:0/4049243680 shutdown_connections 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.572+0000 7f0124685640 1 -- 172.21.15.116:0/4049243680 wait complete. 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 Processor -- start 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 -- start start 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100570 0x7f011c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c19c890 0x7f011c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f011c111a50 con 0x7f011c107d40 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1118d0 con 0x7f011c100570 2024-03-19T02:39:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0124685640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f011c111bd0 con 0x7f011c19c890 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52526/0 (socket says 172.21.15.116:52526) 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 -- 172.21.15.116:0/441922200 learned_addr learned my addr 172.21.15.116:0/441922200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0122421640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100570 0x7f011c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0122c22640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c19c890 0x7f011c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 -- 172.21.15.116:0/441922200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c19c890 msgr2=0x7f011c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c19c890 0x7f011c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 -- 172.21.15.116:0/441922200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100570 msgr2=0x7f011c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100570 0x7f011c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.573+0000 7f0121c20640 1 -- 172.21.15.116:0/441922200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c19d380 con 0x7f011c107d40 2024-03-19T02:39:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f0121c20640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f010c005fd0 tx=0x7f010c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c017070 con 0x7f011c107d40 2024-03-19T02:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011c19d670 con 0x7f011c107d40 2024-03-19T02:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f010c009c30 con 0x7f011c107d40 2024-03-19T02:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c003d40 con 0x7f011c107d40 2024-03-19T02:39:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.574+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011c1a4eb0 con 0x7f011c107d40 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.575+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f011c067d80 con 0x7f011c107d40 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.576+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f010c003ee0 con 0x7f011c107d40 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.577+0000 7f01137fe640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 0x7f01080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.577+0000 7f0122421640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 0x7f01080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.577+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f010c0cee20 con 0x7f011c107d40 2024-03-19T02:39:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.577+0000 7f0122421640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 0x7f01080a4e30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0118009bd0 tx=0x7f0118005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.582+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f010c020030 con 0x7f011c107d40 2024-03-19T02:39:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 366286 3.10 MB/sec execute 508 sec latency 37.180 ms 2024-03-19T02:39:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.840+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f011c109db0 con 0x7f011c107d40 2024-03-19T02:39:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.841+0000 7f01137fe640 1 -- 172.21.15.116:0/441922200 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7f010c09b750 con 0x7f011c107d40 2024-03-19T02:39:05.842 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:39:05.842 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:39:05.842 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:39:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 msgr2=0x7f01080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 0x7f01080a4e30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0118009bd0 tx=0x7f0118005f90 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 msgr2=0x7f011c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f010c005fd0 tx=0x7f010c00dde0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 shutdown_connections 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f01080a2970 0x7f01080a4e30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f011c19c890 0x7f011c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f011c107d40 0x7f011c198510 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 --2- 172.21.15.116:0/441922200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f011c100570 0x7f011c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 >> 172.21.15.116:0/441922200 conn(0x7f011c075a40 msgr2=0x7f011c0ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 shutdown_connections 2024-03-19T02:39:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.843+0000 7f0124685640 1 -- 172.21.15.116:0/441922200 wait complete. 2024-03-19T02:39:05.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/763088674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78102a40 msgr2=0x7f1a78102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/763088674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78102a40 0x7f1a78102e20 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a6802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/763088674 shutdown_connections 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/763088674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 0x7f1a7810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/763088674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78103360 0x7f1a7810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/763088674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78102a40 0x7f1a78102e20 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/763088674 >> 172.21.15.116:0/763088674 conn(0x7f1a780778d0 msgr2=0x7f1a78077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/763088674 shutdown_connections 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.969+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/763088674 wait complete. 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 Processor -- start 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 -- start start 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102a40 0x7f1a78198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 0x7f1a7819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78111a30 con 0x7f1a78103360 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781118b0 con 0x7f1a7810bd80 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7cfb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78111bb0 con 0x7f1a78102a40 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a75d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a75d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52544/0 (socket says 172.21.15.116:52544) 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3967516297 learned_addr learned my addr 172.21.15.116:0/3967516297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a7659c640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102a40 0x7f1a78198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.970+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 0x7f1a7819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3967516297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102a40 msgr2=0x7f1a78198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102a40 0x7f1a78198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3967516297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 msgr2=0x7f1a7819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 0x7f1a7819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3967516297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7819d020 con 0x7f1a78103360 2024-03-19T02:39:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00e9f0 tx=0x7f1a6c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a677fe640 1 -- 172.21.15.116:0/3967516297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c00cd80 con 0x7f1a78103360 2024-03-19T02:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a677fe640 1 -- 172.21.15.116:0/3967516297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a6c0054d0 con 0x7f1a78103360 2024-03-19T02:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7819d310 con 0x7f1a78103360 2024-03-19T02:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a677fe640 1 -- 172.21.15.116:0/3967516297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c010490 con 0x7f1a78103360 2024-03-19T02:39:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.971+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a781a4ba0 con 0x7f1a78103360 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.973+0000 7f1a677fe640 1 -- 172.21.15.116:0/3967516297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1a6c005020 con 0x7f1a78103360 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 Processor -- start 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 -- start start 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a677fe640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 0x7f1a600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f65290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f67bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a677fe640 1 -- 172.21.15.116:0/3967516297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f1a6c0cf600 con 0x7f1a78103360 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561de3ebf4b0 con 0x561de3f657d0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561de3e67d70 con 0x561de3f5c2d0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 0x7f1a600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37932/0 (socket says 172.21.15.116:37932) 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7659c640 1 -- 172.21.15.116:0/171487461 learned_addr learned my addr 172.21.15.116:0/171487461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f65290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f67bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.974+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561de3ebb550 con 0x561de3f5cbf0 2024-03-19T02:39:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 -- 172.21.15.116:0/171487461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 msgr2=0x561de3f65290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f65290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 -- 172.21.15.116:0/171487461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 msgr2=0x561de3f67bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f67bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 -- 172.21.15.116:0/171487461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561de3e8a210 con 0x561de3f5c2d0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a7659c640 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a680095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94cc3d29868c1bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 0x7f1a600a4e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a70005fd0 tx=0x7f1a70009e90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a64ff9640 1 -- 172.21.15.116:0/171487461 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68004040 con 0x561de3f5c2d0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a64ff9640 1 -- 172.21.15.116:0/171487461 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a680054f0 con 0x561de3f5c2d0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.975+0000 7f1a64ff9640 1 -- 172.21.15.116:0/171487461 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802cdb0 con 0x561de3f5c2d0 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- 172.21.15.116:0/171487461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 msgr2=0x561de3f5c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a680095f0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- 172.21.15.116:0/171487461 shutdown_connections 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f67bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f65290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/171487461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f5c6b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- 172.21.15.116:0/171487461 >> 172.21.15.116:0/171487461 conn(0x561de3f56a00 msgr2=0x561de3f56de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- 172.21.15.116:0/171487461 shutdown_connections 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- 172.21.15.116:0/171487461 wait complete. 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.976+0000 7f1a7fba4740 1 -- start start 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 Processor -- start 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f63250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561de3f68560 con 0x561de3f5cbf0 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561de3e7b310 con 0x561de3f5c2d0 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561de3f686e0 con 0x561de3f657d0 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7fba4740 10 client.0 _ll_register_callbacks cb 0x561de3e6f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:05.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a75d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a75d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52566/0 (socket says 172.21.15.116:52566) 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3624267917 learned_addr learned my addr 172.21.15.116:0/3624267917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a7659c640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.977+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f63250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 msgr2=0x561de3f63250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f63250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3624267917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 msgr2=0x561de3f627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 -- 172.21.15.116:0/3624267917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561de3bd33f0 con 0x561de3f5cbf0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c097c90 tx=0x7f1a6c099d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dc14dd7c4d0f34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0b3070 con 0x561de3f5cbf0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f1a6c097ea0 con 0x561de3f5cbf0 2024-03-19T02:39:05.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a680054f0 con 0x561de3f5cbf0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a6c09b830 con 0x561de3f5cbf0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a7fba4740 10 client.18840 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.978+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561de3c9d7d0 con 0x561de3f5cbf0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a680054f0 con 0x561de3f5cbf0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1a6c0a7030 con 0x561de3f5cbf0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 10 client.18840 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 10 client.18840 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 20 client.18840 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3f75ab0 0x561de3f77ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 4 client.18840 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561de3f1d3e0 con 0x561de3f75ab0 2024-03-19T02:39:05.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.979+0000 7f1a7659c640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3f75ab0 0x561de3f77ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.980+0000 7f1a5dffb640 10 client.18840 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.980+0000 7f1a7659c640 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3f75ab0 0x561de3f77ea0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb5ff12f422ada37 server_cookie=ab2ae48bed5c713c in_seq=0 out_seq=0 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a5dffb640 1 -- 172.21.15.116:0/3624267917 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561de3f1d3e0 con 0x561de3f75ab0 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a5dffb640 10 client.18840 handle_command_reply: tid=0 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 client.18840 shutdown 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 msgr2=0x561de3f62d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c097c90 tx=0x7f1a6c099d40 comp rx=0 tx=0).stop 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 shutdown_connections 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3f75ab0 0x561de3f77ea0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f63250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5cbf0 0x561de3f62d10 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/3624267917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 >> 172.21.15.116:0/3624267917 conn(0x561de3f56a00 msgr2=0x561de3f58750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:05.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a5dffb640 0 client.18840 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 shutdown_connections 2024-03-19T02:39:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 1 -- 172.21.15.116:0/3624267917 wait complete. 2024-03-19T02:39:05.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.981+0000 7f1a7fba4740 20 client.18840 trim_cache size 0 max 16384 2024-03-19T02:39:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 Processor -- start 2024-03-19T02:39:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 -- start start 2024-03-19T02:39:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f71300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f63070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781aecd0 con 0x561de3f657d0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b0810 con 0x561de3f5c2d0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781af3a0 con 0x561de3f5cbf0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52568/0 (socket says 172.21.15.116:52568) 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 -- 172.21.15.116:0/1314850582 learned_addr learned my addr 172.21.15.116:0/1314850582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f63070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a7659c640 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f71300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 -- 172.21.15.116:0/1314850582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 msgr2=0x561de3f63070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f63070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 -- 172.21.15.116:0/1314850582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 msgr2=0x561de3f71300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f71300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 -- 172.21.15.116:0/1314850582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561de3f2c880 con 0x561de3f657d0 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.996+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1a70005ea0 tx=0x7f1a70021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6493d46cb8709c46 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a5ffff640 1 -- 172.21.15.116:0/1314850582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70004b20 con 0x561de3f657d0 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a5ffff640 1 -- 172.21.15.116:0/1314850582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a70004cc0 con 0x561de3f657d0 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a5ffff640 1 -- 172.21.15.116:0/1314850582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70005420 con 0x561de3f657d0 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 -- 172.21.15.116:0/1314850582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 msgr2=0x561de3f7d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f1a70005ea0 tx=0x7f1a70021000 comp rx=0 tx=0).stop 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 -- 172.21.15.116:0/1314850582 shutdown_connections 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f657d0 0x561de3f7d770 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f5cbf0 0x561de3f63070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/1314850582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5c2d0 0x561de3f71300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 -- 172.21.15.116:0/1314850582 >> 172.21.15.116:0/1314850582 conn(0x561de3f56a00 msgr2=0x561de3f55ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 -- 172.21.15.116:0/1314850582 shutdown_connections 2024-03-19T02:39:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.997+0000 7f1a7fba4740 1 -- 172.21.15.116:0/1314850582 wait complete. 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 -- start start 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 Processor -- start 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5cbf0 0x561de3f697d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f69d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b0140 con 0x561de3f5c2d0 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781afa70 con 0x561de3f5cbf0 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781accd0 con 0x561de3f657d0 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7fba4740 10 client.0 _ll_register_callbacks cb 0x7f1a6c09b660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52570/0 (socket says 172.21.15.116:52570) 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 -- 172.21.15.116:0/2697129489 learned_addr learned my addr 172.21.15.116:0/2697129489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a76d9d640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f69d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5cbf0 0x561de3f697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 -- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 msgr2=0x561de3f69d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f69d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 -- 172.21.15.116:0/2697129489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5cbf0 msgr2=0x561de3f697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5cbf0 0x561de3f697d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.998+0000 7f1a7659c640 1 -- 172.21.15.116:0/2697129489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561de3f67bc0 con 0x561de3f5c2d0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7659c640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a6802bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c8e83664e96977 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68004440 con 0x561de3f5c2d0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f1a68051b20 con 0x561de3f5c2d0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a70004cc0 con 0x561de3f5c2d0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1a6805c050 con 0x561de3f5c2d0 2024-03-19T02:39:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 10 client.18846 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561de3fdbf20 con 0x561de3f5c2d0 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a70004cc0 con 0x561de3f5c2d0 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f1a6c0860b0 con 0x561de3f5c2d0 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 10 client.18846 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 10 client.18846 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 20 client.18846 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3fdc280 0x561de3fde6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 4 client.18846 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:05.999+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561de3ecc9f0 con 0x561de3fdc280 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3fdc280 0x561de3fde6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a75d9b640 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3fdc280 0x561de3fde6a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=489d45daf8369363 server_cookie=d3247f30b5a746e in_seq=0 out_seq=0 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a5dffb640 10 client.18846 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:06.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a5dffb640 1 -- 172.21.15.116:0/2697129489 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561de3ecc9f0 con 0x561de3fdc280 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a5dffb640 10 client.18846 handle_command_reply: tid=0 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 client.18846 shutdown 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 msgr2=0x561de3f69290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009900 tx=0x7f1a6802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 shutdown_connections 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561de3fdc280 0x561de3fde6a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561de3f657d0 0x561de3f69d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561de3f5cbf0 0x561de3f697d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a5dffb640 0 client.18846 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 --2- 172.21.15.116:0/2697129489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561de3f5c2d0 0x561de3f69290 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 >> 172.21.15.116:0/2697129489 conn(0x561de3f56a00 msgr2=0x561de3f57b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 shutdown_connections 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 1 -- 172.21.15.116:0/2697129489 wait complete. 2024-03-19T02:39:06.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.000+0000 7f1a7fba4740 20 client.18846 trim_cache size 0 max 16384 2024-03-19T02:39:06.009 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 msgr2=0x7f1a600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 0x7f1a600a4e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a70005fd0 tx=0x7f1a70009e90 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 msgr2=0x7f1a78198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00e9f0 tx=0x7f1a6c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 shutdown_connections 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1a600a2970 0x7f1a600a4e30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a7810bd80 0x7f1a7819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a78103360 0x7f1a78198570 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 --2- 172.21.15.116:0/3967516297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1a78102a40 0x7f1a78198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 >> 172.21.15.116:0/3967516297 conn(0x7f1a780778d0 msgr2=0x7f1a780feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 shutdown_connections 2024-03-19T02:39:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.001+0000 7f1a7cfb9640 1 -- 172.21.15.116:0/3967516297 wait complete. 2024-03-19T02:39:06.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:39:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- 172.21.15.116:0/4138117816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9984101060 msgr2=0x7f998410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 --2- 172.21.15.116:0/4138117816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9984101060 0x7f998410d570 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f997c009cd0 tx=0x7f997c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- 172.21.15.116:0/4138117816 shutdown_connections 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 --2- 172.21.15.116:0/4138117816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9984101060 0x7f998410d570 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 --2- 172.21.15.116:0/4138117816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99841006a0 0x7f9984100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 --2- 172.21.15.116:0/4138117816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 0x7f9984106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- 172.21.15.116:0/4138117816 >> 172.21.15.116:0/4138117816 conn(0x7f99840778d0 msgr2=0x7f9984077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- 172.21.15.116:0/4138117816 shutdown_connections 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- 172.21.15.116:0/4138117816 wait complete. 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 Processor -- start 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.129+0000 7f9989da9640 1 -- start start 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9984101060 0x7f99841992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 0x7f99841a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9984110970 con 0x7f99841006a0 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99841107f0 con 0x7f9984101060 2024-03-19T02:39:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9984110af0 con 0x7f99841066b0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52588/0 (socket says 172.21.15.116:52588) 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 -- 172.21.15.116:0/2851919875 learned_addr learned my addr 172.21.15.116:0/2851919875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9982ffd640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9984101060 0x7f99841992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9983fff640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 0x7f99841a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 -- 172.21.15.116:0/2851919875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 msgr2=0x7f99841a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 0x7f99841a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 -- 172.21.15.116:0/2851919875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9984101060 msgr2=0x7f99841992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9984101060 0x7f99841992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 -- 172.21.15.116:0/2851919875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99841a4350 con 0x7f99841006a0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f99837fe640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f997400e9f0 tx=0x7f997400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9980ff9640 1 -- 172.21.15.116:0/2851919875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997400cd80 con 0x7f99841006a0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99841a4520 con 0x7f99841006a0 2024-03-19T02:39:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9980ff9640 1 -- 172.21.15.116:0/2851919875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99740054d0 con 0x7f99841006a0 2024-03-19T02:39:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9980ff9640 1 -- 172.21.15.116:0/2851919875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974010490 con 0x7f99841006a0 2024-03-19T02:39:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.130+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99841a4a40 con 0x7f99841006a0 2024-03-19T02:39:06.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9980ff9640 1 -- 172.21.15.116:0/2851919875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9974005020 con 0x7f99841006a0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 Processor -- start 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 -- start start 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca7bf0 0x55596dcb0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dcb2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55596dc0a4b0 con 0x55596dca72d0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55596dbb2d70 con 0x55596dca7bf0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f998c992740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55596dc06550 con 0x55596dcb07d0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9980ff9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 0x7f996c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9983fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9982ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dcb2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9983fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52604/0 (socket says 172.21.15.116:52604) 2024-03-19T02:39:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9983fff640 1 -- 172.21.15.116:0/2348751461 learned_addr learned my addr 172.21.15.116:0/2348751461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f99837fe640 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca7bf0 0x55596dcb0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.133+0000 7f9983fff640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 0x7f996c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9980ff9640 1 -- 172.21.15.116:0/2851919875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f99740cf4f0 con 0x7f99841006a0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 -- 172.21.15.116:0/2348751461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca7bf0 msgr2=0x55596dcb0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca7bf0 0x55596dcb0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 -- 172.21.15.116:0/2348751461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 msgr2=0x55596dca76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 -- 172.21.15.116:0/2348751461 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55596dbd5210 con 0x55596dcb07d0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9983fff640 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9983fff640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 0x7f996c0a4e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f997c009a60 tx=0x7f997c005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9982ffd640 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dcb2bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9978009bf0 tx=0x7f9978011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27e3348e4b928742 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9969ffb640 1 -- 172.21.15.116:0/2348751461 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9978004050 con 0x55596dcb07d0 2024-03-19T02:39:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9969ffb640 1 -- 172.21.15.116:0/2348751461 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99780054d0 con 0x55596dcb07d0 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.134+0000 7f9969ffb640 1 -- 172.21.15.116:0/2348751461 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9978004c60 con 0x55596dcb07d0 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- 172.21.15.116:0/2348751461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 msgr2=0x55596dcb2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dcb2bc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9978009bf0 tx=0x7f9978011810 comp rx=0 tx=0).stop 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- 172.21.15.116:0/2348751461 shutdown_connections 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dcb2bc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca7bf0 0x55596dcb0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- 172.21.15.116:0/2348751461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dca76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- 172.21.15.116:0/2348751461 >> 172.21.15.116:0/2348751461 conn(0x55596dca1a00 msgr2=0x55596dca1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- 172.21.15.116:0/2348751461 shutdown_connections 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- 172.21.15.116:0/2348751461 wait complete. 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- start start 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 Processor -- start 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dcadce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.135+0000 7f998c992740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55596dcb35c0 con 0x55596dcb07d0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55596dbc6310 con 0x55596dca72d0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55596dcb3740 con 0x55596dca7bf0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 10 client.0 _ll_register_callbacks cb 0x55596dbba2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9983fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f99837fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dcadce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9983fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52608/0 (socket says 172.21.15.116:52608) 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9983fff640 1 -- 172.21.15.116:0/1908912814 learned_addr learned my addr 172.21.15.116:0/1908912814 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 -- 172.21.15.116:0/1908912814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 msgr2=0x55596dcad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 -- 172.21.15.116:0/1908912814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 msgr2=0x55596dcae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 -- 172.21.15.116:0/1908912814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55596d91e3f0 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9983fff640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9982ffd640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dcadce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9978009350 tx=0x7f9978009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c58ac0a71ab68da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9978003800 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99780054d0 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f99780039a0 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9978034050 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 10 client.28594 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.136+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55596d9e87d0 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99780054d0 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9978005180 con 0x55596dca7bf0 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 10 client.28594 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 10 client.28594 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 20 client.28594 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dcc0980 0x55596dcc2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 4 client.28594 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55596dc683e0 con 0x55596dcc0980 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f99837fe640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dcc0980 0x55596dcc2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f9962ffd640 10 client.28594 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.137+0000 7f99837fe640 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dcc0980 0x55596dcc2d70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2acd02e447be311b server_cookie=f429edd808d81506 in_seq=0 out_seq=0 2024-03-19T02:39:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.138+0000 7f9962ffd640 1 -- 172.21.15.116:0/1908912814 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55596dc683e0 con 0x55596dcc0980 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.138+0000 7f9962ffd640 10 client.28594 handle_command_reply: tid=0 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 client.28594 shutdown 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 msgr2=0x55596dcadce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dcadce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9978009350 tx=0x7f9978009fd0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 shutdown_connections 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dcc0980 0x55596dcc2d70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dcb07d0 0x55596dcae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f9962ffd640 0 client.28594 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dcadce0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 --2- 172.21.15.116:0/1908912814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 >> 172.21.15.116:0/1908912814 conn(0x55596dca1a00 msgr2=0x55596dca3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 shutdown_connections 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 1 -- 172.21.15.116:0/1908912814 wait complete. 2024-03-19T02:39:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.139+0000 7f998c992740 20 client.28594 trim_cache size 0 max 16384 2024-03-19T02:39:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.153+0000 7f998c992740 1 Processor -- start 2024-03-19T02:39:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.153+0000 7f998c992740 1 -- start start 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dca9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 0x55596dcbc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99841aedd0 con 0x55596dca72d0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99841b0910 con 0x55596dcb07d0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f998c992740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99841af4a0 con 0x55596dca7bf0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52618/0 (socket says 172.21.15.116:52618) 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 -- 172.21.15.116:0/3816439896 learned_addr learned my addr 172.21.15.116:0/3816439896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f9982ffd640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dca9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f9983fff640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 0x55596dcbc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 -- 172.21.15.116:0/3816439896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 msgr2=0x55596dca9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dca9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 -- 172.21.15.116:0/3816439896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 msgr2=0x55596dcbc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 0x55596dcbc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 -- 172.21.15.116:0/3816439896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55596dcbc8a0 con 0x55596dca72d0 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f9983fff640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 0x55596dcbc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99837fe640 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f997409b4c0 tx=0x7f997406e560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a4046099ab6ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99697fa640 1 -- 172.21.15.116:0/3816439896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974005680 con 0x55596dca72d0 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99697fa640 1 -- 172.21.15.116:0/3816439896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f997409bec0 con 0x55596dca72d0 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.154+0000 7f99697fa640 1 -- 172.21.15.116:0/3816439896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974099b50 con 0x55596dca72d0 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- 172.21.15.116:0/3816439896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 msgr2=0x55596dcafed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f997409b4c0 tx=0x7f997406e560 comp rx=0 tx=0).stop 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- 172.21.15.116:0/3816439896 shutdown_connections 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dcb07d0 0x55596dcbc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dca7bf0 0x55596dca9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 --2- 172.21.15.116:0/3816439896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca72d0 0x55596dcafed0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- 172.21.15.116:0/3816439896 >> 172.21.15.116:0/3816439896 conn(0x55596dca1a00 msgr2=0x55596dcabf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- 172.21.15.116:0/3816439896 shutdown_connections 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- 172.21.15.116:0/3816439896 wait complete. 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.155+0000 7f998c992740 1 -- start start 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 Processor -- start 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcb8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dca9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99841b0240 con 0x55596dca7bf0 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99841afb70 con 0x55596dca72d0 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99841acdd0 con 0x55596dcb07d0 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f998c992740 10 client.0 _ll_register_callbacks cb 0x7f9978005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f9982ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcb8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f9982ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52624/0 (socket says 172.21.15.116:52624) 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f9982ffd640 1 -- 172.21.15.116:0/3032538436 learned_addr learned my addr 172.21.15.116:0/3032538436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f9983fff640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dca9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 -- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 msgr2=0x55596dca9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dca9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 -- 172.21.15.116:0/3032538436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 msgr2=0x55596dca9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f99837fe640 1 -- 172.21.15.116:0/3032538436 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55596dcb9000 con 0x55596dca72d0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.156+0000 7f9982ffd640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f99837fe640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcb8990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f997409c070 tx=0x7f9974005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788d43be2281d478 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740b4060 con 0x55596dca72d0 2024-03-19T02:39:06.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f997409bec0 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f99740860b0 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9974005680 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 10 client.28649 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55596dd265b0 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f997409bec0 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9974097e30 con 0x55596dca72d0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 10 client.28649 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 10 client.28649 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 20 client.28649 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dd26c30 0x55596dd29020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 4 client.28649 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55596dc179f0 con 0x55596dd26c30 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.157+0000 7f9982ffd640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dd26c30 0x55596dd29020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f9962ffd640 10 client.28649 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f9982ffd640 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dd26c30 0x55596dd29020 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=54d844b131ac0d70 server_cookie=f52fc2c9d07f7a07 in_seq=0 out_seq=0 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f9962ffd640 1 -- 172.21.15.116:0/3032538436 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55596dc179f0 con 0x55596dd26c30 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f9962ffd640 10 client.28649 handle_command_reply: tid=0 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 client.28649 shutdown 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 msgr2=0x55596dcb8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcb8990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f997409c070 tx=0x7f9974005bf0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 shutdown_connections 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55596dcb07d0 0x55596dca9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55596dd26c30 0x55596dd29020 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55596dca7bf0 0x55596dca9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f9962ffd640 0 client.28649 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 --2- 172.21.15.116:0/3032538436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55596dca72d0 0x55596dcb8990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.158+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 >> 172.21.15.116:0/3032538436 conn(0x55596dca1a00 msgr2=0x55596dcac810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 shutdown_connections 2024-03-19T02:39:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f998c992740 1 -- 172.21.15.116:0/3032538436 wait complete. 2024-03-19T02:39:06.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f998c992740 20 client.28649 trim_cache size 0 max 16384 2024-03-19T02:39:06.169 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 msgr2=0x7f996c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 0x7f996c0a4e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f997c009a60 tx=0x7f997c005c50 comp rx=0 tx=0).stop 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 msgr2=0x7f9984198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f997400e9f0 tx=0x7f997400edf0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 shutdown_connections 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f996c0a2970 0x7f996c0a4e30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99841066b0 0x7f99841a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9984101060 0x7f99841992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 --2- 172.21.15.116:0/2851919875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99841006a0 0x7f9984198d90 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 >> 172.21.15.116:0/2851919875 conn(0x7f99840778d0 msgr2=0x7f9984102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 shutdown_connections 2024-03-19T02:39:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.159+0000 7f9989da9640 1 -- 172.21.15.116:0/2851919875 wait complete. 2024-03-19T02:39:06.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:05 smithi116 ceph-mon[32263]: pgmap v351: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2024-03-19T02:39:06.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:05 smithi116 ceph-mon[32263]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T02:39:06.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:05 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/441922200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:06.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:39:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:05 smithi154 ceph-mon[35693]: pgmap v351: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2024-03-19T02:39:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:05 smithi154 ceph-mon[35693]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T02:39:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:05 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/441922200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:06.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 -- 172.21.15.116:0/473984293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc1077a0 msgr2=0x7fa3cc107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 --2- 172.21.15.116:0/473984293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc107b80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009930 tx=0x7fa3c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 -- 172.21.15.116:0/473984293 shutdown_connections 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 --2- 172.21.15.116:0/473984293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc102150 0x7fa3cc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 --2- 172.21.15.116:0/473984293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc101790 0x7fa3cc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 --2- 172.21.15.116:0/473984293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc107b80 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 -- 172.21.15.116:0/473984293 >> 172.21.15.116:0/473984293 conn(0x7fa3cc0fd2a0 msgr2=0x7fa3cc0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 -- 172.21.15.116:0/473984293 shutdown_connections 2024-03-19T02:39:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.285+0000 7fa3d4190640 1 -- 172.21.15.116:0/473984293 wait complete. 2024-03-19T02:39:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 Processor -- start 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 -- start start 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc111a60 con 0x7fa3cc102150 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1118e0 con 0x7fa3cc101790 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d4190640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc111be0 con 0x7fa3cc1077a0 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d272d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52634/0 (socket says 172.21.15.116:52634) 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d272d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50900/0 (socket says 172.21.15.116:50900) 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38030/0 (socket says 172.21.15.116:38030) 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 -- 172.21.15.116:0/2935312985 learned_addr learned my addr 172.21.15.116:0/2935312985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 -- 172.21.15.116:0/2935312985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 msgr2=0x7fa3cc19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 -- 172.21.15.116:0/2935312985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 msgr2=0x7fa3cc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d172b640 1 -- 172.21.15.116:0/2935312985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc19d020 con 0x7fa3cc102150 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.286+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3d172b640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e9f0 tx=0x7fa3bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3c2ffd640 1 -- 172.21.15.116:0/2935312985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00cd80 con 0x7fa3cc102150 2024-03-19T02:39:06.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc19d310 con 0x7fa3cc102150 2024-03-19T02:39:06.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3c2ffd640 1 -- 172.21.15.116:0/2935312985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3bc0054d0 con 0x7fa3cc102150 2024-03-19T02:39:06.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3c2ffd640 1 -- 172.21.15.116:0/2935312985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc010490 con 0x7fa3cc102150 2024-03-19T02:39:06.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.287+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc1a4ba0 con 0x7fa3cc102150 2024-03-19T02:39:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3c2ffd640 1 -- 172.21.15.116:0/2935312985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa3bc005020 con 0x7fa3cc102150 2024-03-19T02:39:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 Processor -- start 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 -- start start 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 0x561516299290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 0x56151629bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5615161f34b0 con 0x561516290bf0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56151619bd70 con 0x5615162902d0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615161ef550 con 0x5615162997d0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3c2ffd640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 0x7fa3b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 0x561516299290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d272d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 0x56151629bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d272d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38032/0 (socket says 172.21.15.116:38032) 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.290+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 0x561516299290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52640/0 (socket says 172.21.15.116:52640) 2024-03-19T02:39:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 -- 172.21.15.116:0/669517104 learned_addr learned my addr 172.21.15.116:0/669517104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 0x56151629bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50902/0 (socket says 172.21.15.116:50902) 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 0x7fa3b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3c2ffd640 1 -- 172.21.15.116:0/2935312985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fa3bc0cf600 con 0x7fa3cc102150 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 -- 172.21.15.116:0/669517104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 msgr2=0x56151629bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 0x56151629bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 -- 172.21.15.116:0/669517104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 msgr2=0x561516299290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 0x561516299290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 -- 172.21.15.116:0/669517104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615161be210 con 0x5615162902d0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4005c40 tx=0x7fa3c4013f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55b9d782e13aa16 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3d272d640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 0x7fa3b80a4e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400f850 tx=0x7fa3c401f000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3b7fff640 1 -- 172.21.15.116:0/669517104 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400fde0 con 0x5615162902d0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3b7fff640 1 -- 172.21.15.116:0/669517104 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c4022020 con 0x5615162902d0 2024-03-19T02:39:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.291+0000 7fa3b7fff640 1 -- 172.21.15.116:0/669517104 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4010690 con 0x5615162902d0 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 -- 172.21.15.116:0/669517104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 msgr2=0x5615162906b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4005c40 tx=0x7fa3c4013f50 comp rx=0 tx=0).stop 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 -- 172.21.15.116:0/669517104 shutdown_connections 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615162997d0 0x56151629bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561516290bf0 0x561516299290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/669517104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162906b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 -- 172.21.15.116:0/669517104 >> 172.21.15.116:0/669517104 conn(0x56151628aa00 msgr2=0x56151628ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 -- 172.21.15.116:0/669517104 shutdown_connections 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.292+0000 7fa3d6d77740 1 -- 172.21.15.116:0/669517104 wait complete. 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 -- start start 2024-03-19T02:39:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 Processor -- start 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x561516296d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x561516297290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56151629c560 con 0x5615162902d0 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615161af310 con 0x5615162997d0 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56151629c6e0 con 0x561516290bf0 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d6d77740 10 client.0 _ll_register_callbacks cb 0x5615161a32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52644/0 (socket says 172.21.15.116:52644) 2024-03-19T02:39:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/1179770915 learned_addr learned my addr 172.21.15.116:0/1179770915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d172b640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x561516296d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.293+0000 7fa3d272d640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x561516297290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/1179770915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 msgr2=0x561516296d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x561516296d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 msgr2=0x561516297290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x561516297290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/1179770915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561515f073f0 con 0x5615162902d0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8036040 tx=0x7fa3c802c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f74cfc91e19f7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c802c5d0 con 0x5615162902d0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fa3c80035f0 con 0x5615162902d0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c4022020 con 0x5615162902d0 2024-03-19T02:39:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa3c80037f0 con 0x5615162902d0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d6d77740 10 client.18876 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561515fd17d0 con 0x5615162902d0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.294+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c4022020 con 0x5615162902d0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa3c8034b60 con 0x5615162902d0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 10 client.18876 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 10 client.18876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 20 client.18876 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615162a9b10 0x5615162abf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 4 client.18876 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615162513e0 con 0x5615162a9b10 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d172b640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615162a9b10 0x5615162abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3d172b640 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615162a9b10 0x5615162abf60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef43dfbde520ae7c server_cookie=a2b5aab9fee4428 in_seq=0 out_seq=0 2024-03-19T02:39:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.295+0000 7fa3b57fa640 10 client.18876 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3b57fa640 1 -- 172.21.15.116:0/1179770915 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615162513e0 con 0x5615162a9b10 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3b57fa640 10 client.18876 handle_command_reply: tid=0 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 client.18876 shutdown 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 msgr2=0x561516296810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8036040 tx=0x7fa3c802c9f0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 shutdown_connections 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5615162a9b10 0x5615162abf60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x561516297290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3b57fa640 0 client.18876 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x561516296d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/1179770915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x561516296810 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 >> 172.21.15.116:0/1179770915 conn(0x56151628aa00 msgr2=0x56151628c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 shutdown_connections 2024-03-19T02:39:06.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 1 -- 172.21.15.116:0/1179770915 wait complete. 2024-03-19T02:39:06.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.296+0000 7fa3d6d77740 20 client.18876 trim_cache size 0 max 16384 2024-03-19T02:39:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 Processor -- start 2024-03-19T02:39:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 -- start start 2024-03-19T02:39:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x5615162970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x5615162b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1aedd0 con 0x5615162902d0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1b0910 con 0x5615162997d0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.311+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1af4a0 con 0x561516290bf0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x5615162970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52648/0 (socket says 172.21.15.116:52648) 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/23245068 learned_addr learned my addr 172.21.15.116:0/23245068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d272d640 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x5615162b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 -- 172.21.15.116:0/23245068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 msgr2=0x5615162b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x5615162b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 -- 172.21.15.116:0/23245068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 msgr2=0x5615162a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 -- 172.21.15.116:0/23245068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561516290a20 con 0x561516290bf0 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3d172b640 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x5615162970b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0cc920 tx=0x7fa3bc09f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8bf4d252bcccec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3b77fe640 1 -- 172.21.15.116:0/23245068 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0af070 con 0x561516290bf0 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3b77fe640 1 -- 172.21.15.116:0/23245068 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3bc09bf30 con 0x561516290bf0 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.312+0000 7fa3b77fe640 1 -- 172.21.15.116:0/23245068 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0860b0 con 0x561516290bf0 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- 172.21.15.116:0/23245068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 msgr2=0x5615162970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x5615162970b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0cc920 tx=0x7fa3bc09f040 comp rx=0 tx=0).stop 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- 172.21.15.116:0/23245068 shutdown_connections 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162997d0 0x5615162b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x5615162970b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/23245068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162902d0 0x5615162a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- 172.21.15.116:0/23245068 >> 172.21.15.116:0/23245068 conn(0x56151628aa00 msgr2=0x561516289ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- 172.21.15.116:0/23245068 shutdown_connections 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- 172.21.15.116:0/23245068 wait complete. 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.313+0000 7fa3d6d77740 1 -- start start 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 Processor -- start 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 0x56151629d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1b0240 con 0x5615162997d0 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1afb70 con 0x5615162902d0 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1acdd0 con 0x561516290bf0 2024-03-19T02:39:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d6d77740 10 client.0 _ll_register_callbacks cb 0x7fa3c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 0x56151629d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d1f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38060/0 (socket says 172.21.15.116:38060) 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d172b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50934/0 (socket says 172.21.15.116:50934) 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d1f2c640 1 -- 172.21.15.116:0/2143149668 learned_addr learned my addr 172.21.15.116:0/2143149668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 -- 172.21.15.116:0/2143149668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 msgr2=0x56151629cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 -- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 msgr2=0x5615162a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d272d640 1 -- 172.21.15.116:0/2143149668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56151629dcf0 con 0x5615162997d0 2024-03-19T02:39:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.314+0000 7fa3d172b640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d272d640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 0x56151629d4f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4012ed0 tx=0x7fa3c4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b5183585be43e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4004a00 con 0x5615162997d0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fa3c4004ba0 con 0x5615162997d0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3bc09bf30 con 0x5615162997d0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa3c403e050 con 0x5615162997d0 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 10 client.18882 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5615163142a0 con 0x5615162997d0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3bc09bf30 con 0x5615162997d0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fa3c4010b30 con 0x5615162997d0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 10 client.18882 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 10 client.18882 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 20 client.18882 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561516314900 0x561516316d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 4 client.18882 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.315+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615162009f0 con 0x561516314900 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561516314900 0x561516316d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3b57fa640 10 client.18882 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d1f2c640 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561516314900 0x561516316d20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6125b8af626a9b7a server_cookie=d84e877d360bd125 in_seq=0 out_seq=0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3b57fa640 1 -- 172.21.15.116:0/2143149668 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615162009f0 con 0x561516314900 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3b57fa640 10 client.18882 handle_command_reply: tid=0 2024-03-19T02:39:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 client.18882 shutdown 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 msgr2=0x56151629d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 0x56151629d4f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4012ed0 tx=0x7fa3c4005f10 comp rx=0 tx=0).stop 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 shutdown_connections 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561516314900 0x561516316d20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615162997d0 0x56151629d4f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561516290bf0 0x56151629cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3b57fa640 0 client.18882 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 --2- 172.21.15.116:0/2143149668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615162902d0 0x5615162a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 >> 172.21.15.116:0/2143149668 conn(0x56151628aa00 msgr2=0x56151628bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 shutdown_connections 2024-03-19T02:39:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 1 -- 172.21.15.116:0/2143149668 wait complete. 2024-03-19T02:39:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.316+0000 7fa3d6d77740 20 client.18882 trim_cache size 0 max 16384 2024-03-19T02:39:06.326 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 msgr2=0x7fa3b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 0x7fa3b80a4e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400f850 tx=0x7fa3c401f000 comp rx=0 tx=0).stop 2024-03-19T02:39:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 msgr2=0x7fa3cc198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e9f0 tx=0x7fa3bc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 shutdown_connections 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3b80a2970 0x7fa3b80a4e30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc1077a0 0x7fa3cc19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3cc102150 0x7fa3cc198570 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 --2- 172.21.15.116:0/2935312985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc101790 0x7fa3cc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 >> 172.21.15.116:0/2935312985 conn(0x7fa3cc0fd2a0 msgr2=0x7fa3cc0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 shutdown_connections 2024-03-19T02:39:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.317+0000 7fa3d4190640 1 -- 172.21.15.116:0/2935312985 wait complete. 2024-03-19T02:39:06.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:39:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:05 smithi162 ceph-mon[35761]: pgmap v351: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2024-03-19T02:39:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:05 smithi162 ceph-mon[35761]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T02:39:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:05 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/441922200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.439+0000 7fd481909640 1 -- 172.21.15.116:0/429680309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 msgr2=0x7fd47c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.439+0000 7fd481909640 1 --2- 172.21.15.116:0/429680309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c107b80 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009cd0 tx=0x7fd46c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 -- 172.21.15.116:0/429680309 shutdown_connections 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 --2- 172.21.15.116:0/429680309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c102150 0x7fd47c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 --2- 172.21.15.116:0/429680309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c101790 0x7fd47c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 --2- 172.21.15.116:0/429680309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c107b80 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 -- 172.21.15.116:0/429680309 >> 172.21.15.116:0/429680309 conn(0x7fd47c0fd2c0 msgr2=0x7fd47c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 -- 172.21.15.116:0/429680309 shutdown_connections 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 -- 172.21.15.116:0/429680309 wait complete. 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 Processor -- start 2024-03-19T02:39:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.440+0000 7fd481909640 1 -- start start 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c101790 0x7fd47c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 0x7fd47c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c111a20 con 0x7fd47c1077a0 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1118a0 con 0x7fd47c101790 2024-03-19T02:39:06.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c111ba0 con 0x7fd47c102150 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52672/0 (socket says 172.21.15.116:52672) 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 -- 172.21.15.116:0/1129914778 learned_addr learned my addr 172.21.15.116:0/1129914778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47affd640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c101790 0x7fd47c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 0x7fd47c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 -- 172.21.15.116:0/1129914778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 msgr2=0x7fd47c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 0x7fd47c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 -- 172.21.15.116:0/1129914778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c101790 msgr2=0x7fd47c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c101790 0x7fd47c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 -- 172.21.15.116:0/1129914778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47c19cfe0 con 0x7fd47c1077a0 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 0x7fd47c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fd47400e970 tx=0x7fd47400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd480907640 1 -- 172.21.15.116:0/1129914778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47400ccc0 con 0x7fd47c1077a0 2024-03-19T02:39:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd480907640 1 -- 172.21.15.116:0/1129914778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd47400ce60 con 0x7fd47c1077a0 2024-03-19T02:39:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.441+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47c19d2d0 con 0x7fd47c1077a0 2024-03-19T02:39:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.442+0000 7fd480907640 1 -- 172.21.15.116:0/1129914778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474010430 con 0x7fd47c1077a0 2024-03-19T02:39:06.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.442+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd47c1a4b60 con 0x7fd47c1077a0 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd480907640 1 -- 172.21.15.116:0/1129914778 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd474004ff0 con 0x7fd47c1077a0 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 Processor -- start 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd480907640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 0x7fd4640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 -- start start 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd47affd640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 0x7fd4640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd480907640 1 -- 172.21.15.116:0/1129914778 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd4740cf630 con 0x7fd47c1077a0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b6896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b694bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55786b5ec4b0 con 0x55786b689bf0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55786b594d70 con 0x55786b6927d0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.444+0000 7fd4845d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55786b5e8550 con 0x55786b6892d0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52678/0 (socket says 172.21.15.116:52678) 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2961388383 learned_addr learned my addr 172.21.15.116:0/2961388383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b694bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47affd640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b6896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47affd640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 0x7fd4640a4e30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd46c004af0 tx=0x7fd46c005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2961388383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 msgr2=0x55786b6896b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b6896b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2961388383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 msgr2=0x55786b694bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b694bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2961388383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786b5b7210 con 0x55786b689bf0 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47affd640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b6896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b694bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd4700114f0 tx=0x7fd4700119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddea630716831fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd461ffb640 1 -- 172.21.15.116:0/2961388383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47001e070 con 0x55786b689bf0 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd461ffb640 1 -- 172.21.15.116:0/2961388383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd47000fa70 con 0x55786b689bf0 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.445+0000 7fd461ffb640 1 -- 172.21.15.116:0/2961388383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470019b30 con 0x55786b689bf0 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- 172.21.15.116:0/2961388383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 msgr2=0x55786b692290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd4700114f0 tx=0x7fd4700119b0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- 172.21.15.116:0/2961388383 shutdown_connections 2024-03-19T02:39:06.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b694bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b692290 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2961388383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b6896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- 172.21.15.116:0/2961388383 >> 172.21.15.116:0/2961388383 conn(0x55786b683a00 msgr2=0x55786b683de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- 172.21.15.116:0/2961388383 shutdown_connections 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- 172.21.15.116:0/2961388383 wait complete. 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.446+0000 7fd4845d1740 1 -- start start 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 Processor -- start 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 0x55786b68f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 0x55786b68fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55786b6955c0 con 0x55786b6927d0 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55786b5a8310 con 0x55786b6892d0 2024-03-19T02:39:06.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55786b695740 con 0x55786b689bf0 2024-03-19T02:39:06.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd4845d1740 10 client.0 _ll_register_callbacks cb 0x55786b59c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 0x55786b68f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 0x55786b68fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52686/0 (socket says 172.21.15.116:52686) 2024-03-19T02:39:06.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 0x55786b68f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38086/0 (socket says 172.21.15.116:38086) 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47b7fe640 1 -- 172.21.15.116:0/2178681206 learned_addr learned my addr 172.21.15.116:0/2178681206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2178681206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 msgr2=0x55786b68f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 0x55786b68f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2178681206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 msgr2=0x55786b6902c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47a7fc640 1 -- 172.21.15.116:0/2178681206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786b3003f0 con 0x55786b689bf0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.447+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 0x55786b68fd80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd470000c00 tx=0x7fd470009db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bb4b8549c923ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47001e040 con 0x55786b689bf0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd470016040 con 0x55786b689bf0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd47000fa70 con 0x55786b689bf0 2024-03-19T02:39:06.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd47002f050 con 0x55786b689bf0 2024-03-19T02:39:06.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 10 client.28630 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55786b3ca7d0 con 0x55786b689bf0 2024-03-19T02:39:06.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd47000fa70 con 0x55786b689bf0 2024-03-19T02:39:06.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd470005470 con 0x55786b689bf0 2024-03-19T02:39:06.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 10 client.28630 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 10 client.28630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 20 client.28630 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b6a2b40 0x55786b6a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 4 client.28630 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.448+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55786b64a3e0 con 0x55786b6a2b40 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.449+0000 7fd47affd640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b6a2b40 0x55786b6a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.449+0000 7fd45affd640 10 client.28630 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.449+0000 7fd47affd640 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b6a2b40 0x55786b6a4f30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd78890372f83ba5 server_cookie=6f0f8e791068d424 in_seq=0 out_seq=0 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd45affd640 1 -- 172.21.15.116:0/2178681206 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55786b64a3e0 con 0x55786b6a2b40 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd45affd640 10 client.28630 handle_command_reply: tid=0 2024-03-19T02:39:06.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 client.28630 shutdown 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 msgr2=0x55786b68fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 0x55786b68fd80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd470000c00 tx=0x7fd470009db0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 shutdown_connections 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b6a2b40 0x55786b6a4f30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6902c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b689bf0 0x55786b68fd80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 --2- 172.21.15.116:0/2178681206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6892d0 0x55786b68f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd45affd640 0 client.28630 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 >> 172.21.15.116:0/2178681206 conn(0x55786b683a00 msgr2=0x55786b685750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 shutdown_connections 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 1 -- 172.21.15.116:0/2178681206 wait complete. 2024-03-19T02:39:06.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.450+0000 7fd4845d1740 20 client.28630 trim_cache size 0 max 16384 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 Processor -- start 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 -- start start 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b691f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b689bf0 0x55786b69e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1aee90 con 0x55786b6927d0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1b09d0 con 0x55786b689bf0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd4845d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1af560 con 0x55786b6892d0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52696/0 (socket says 172.21.15.116:52696) 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 -- 172.21.15.116:0/185245439 learned_addr learned my addr 172.21.15.116:0/185245439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47affd640 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b691f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b689bf0 0x55786b69e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 -- 172.21.15.116:0/185245439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 msgr2=0x55786b691f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b691f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 -- 172.21.15.116:0/185245439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b689bf0 msgr2=0x55786b69e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b689bf0 0x55786b69e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.465+0000 7fd47b7fe640 1 -- 172.21.15.116:0/185245439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786b6a54c0 con 0x55786b6927d0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd4740ccf50 tx=0x7fd474086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fa7c6d8016b37e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4617fa640 1 -- 172.21.15.116:0/185245439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4740b3070 con 0x55786b6927d0 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4617fa640 1 -- 172.21.15.116:0/185245439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd47409bf60 con 0x55786b6927d0 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4617fa640 1 -- 172.21.15.116:0/185245439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4740860b0 con 0x55786b6927d0 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 -- 172.21.15.116:0/185245439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 msgr2=0x55786b6aa6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd4740ccf50 tx=0x7fd474086040 comp rx=0 tx=0).stop 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 -- 172.21.15.116:0/185245439 shutdown_connections 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b6927d0 0x55786b6aa6f0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b689bf0 0x55786b69e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 --2- 172.21.15.116:0/185245439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b691f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.466+0000 7fd4845d1740 1 -- 172.21.15.116:0/185245439 >> 172.21.15.116:0/185245439 conn(0x55786b683a00 msgr2=0x55786b68e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- 172.21.15.116:0/185245439 shutdown_connections 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- 172.21.15.116:0/185245439 wait complete. 2024-03-19T02:39:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- start start 2024-03-19T02:39:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 Processor -- start 2024-03-19T02:39:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b695b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b706d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1b0300 con 0x55786b689bf0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1afc30 con 0x55786b6927d0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1ace90 con 0x55786b6892d0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd4845d1740 10 client.0 _ll_register_callbacks cb 0x7fd47001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd47a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd47a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52702/0 (socket says 172.21.15.116:52702) 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd47a7fc640 1 -- 172.21.15.116:0/3909080548 learned_addr learned my addr 172.21.15.116:0/3909080548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd47affd640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b695b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.467+0000 7fd47b7fe640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b706d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 -- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 msgr2=0x55786b695b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b695b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 -- 172.21.15.116:0/3909080548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 msgr2=0x55786b706d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b706d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 -- 172.21.15.116:0/3909080548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786b707250 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd47a7fc640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd4700095f0 tx=0x7fd47000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8894daa7a716cda server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd45affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470039060 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd470019d00 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd47409bf60 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd47001e030 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd4845d1740 10 client.18900 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.468+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55786b7083f0 con 0x55786b689bf0 2024-03-19T02:39:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd47409bf60 con 0x55786b689bf0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd4700035f0 con 0x55786b689bf0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 10 client.18900 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 10 client.18900 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 20 client.18900 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b708ae0 0x55786b70af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 4 client.18900 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55786b5f99f0 con 0x55786b708ae0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd47affd640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b708ae0 0x55786b70af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd45affd640 10 client.18900 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd47affd640 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b708ae0 0x55786b70af00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3d6e10d7947028c0 server_cookie=938f8a121b27821b in_seq=0 out_seq=0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd45affd640 1 -- 172.21.15.116:0/3909080548 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55786b5f99f0 con 0x55786b708ae0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.469+0000 7fd45affd640 10 client.18900 handle_command_reply: tid=0 2024-03-19T02:39:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 client.18900 shutdown 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 msgr2=0x55786b696080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd4700095f0 tx=0x7fd47000fab0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 shutdown_connections 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55786b708ae0 0x55786b70af00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55786b6927d0 0x55786b706d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55786b689bf0 0x55786b696080 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd45affd640 0 client.18900 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 --2- 172.21.15.116:0/3909080548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55786b6892d0 0x55786b695b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 >> 172.21.15.116:0/3909080548 conn(0x55786b683a00 msgr2=0x55786b68dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 shutdown_connections 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 1 -- 172.21.15.116:0/3909080548 wait complete. 2024-03-19T02:39:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd4845d1740 20 client.18900 trim_cache size 0 max 16384 2024-03-19T02:39:06.478 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 msgr2=0x7fd4640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 0x7fd4640a4e30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd46c004af0 tx=0x7fd46c005c50 comp rx=0 tx=0).stop 2024-03-19T02:39:06.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 msgr2=0x7fd47c19c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.470+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fd47400e970 tx=0x7fd47400ed70 comp rx=0 tx=0).stop 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 shutdown_connections 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd4640a2970 0x7fd4640a4e30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd47c1077a0 0x7fd47c19c8b0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd47c102150 0x7fd47c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 --2- 172.21.15.116:0/1129914778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd47c101790 0x7fd47c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 >> 172.21.15.116:0/1129914778 conn(0x7fd47c0fd2c0 msgr2=0x7fd47c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 shutdown_connections 2024-03-19T02:39:06.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.471+0000 7fd481909640 1 -- 172.21.15.116:0/1129914778 wait complete. 2024-03-19T02:39:06.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:39:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.608+0000 7f46f9590640 1 -- 172.21.15.116:0/3672304615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f4101060 msgr2=0x7f46f410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.608+0000 7f46f9590640 1 --2- 172.21.15.116:0/3672304615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f4101060 0x7f46f410d570 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009cd0 tx=0x7f46ec02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- 172.21.15.116:0/3672304615 shutdown_connections 2024-03-19T02:39:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- 172.21.15.116:0/3672304615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f4101060 0x7f46f410d570 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- 172.21.15.116:0/3672304615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 0x7f46f4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- 172.21.15.116:0/3672304615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f41066b0 0x7f46f4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- 172.21.15.116:0/3672304615 >> 172.21.15.116:0/3672304615 conn(0x7f46f40778d0 msgr2=0x7f46f4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- 172.21.15.116:0/3672304615 shutdown_connections 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- 172.21.15.116:0/3672304615 wait complete. 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 Processor -- start 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- start start 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 0x7f46f4196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4101060 0x7f46f41973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4110970 con 0x7f46f41066b0 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41107f0 con 0x7f46f41006a0 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.609+0000 7f46f9590640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f4110af0 con 0x7f46f4101060 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52716/0 (socket says 172.21.15.116:52716) 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4101060 0x7f46f41973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 0x7f46f4196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 -- 172.21.15.116:0/3229222505 learned_addr learned my addr 172.21.15.116:0/3229222505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 -- 172.21.15.116:0/3229222505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4101060 msgr2=0x7f46f41973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4101060 0x7f46f41973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 -- 172.21.15.116:0/3229222505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 msgr2=0x7f46f4196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 0x7f46f4196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 -- 172.21.15.116:0/3229222505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f419be30 con 0x7f46f41066b0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f37fe640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f46ec004af0 tx=0x7f46ec035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f0ff9640 1 -- 172.21.15.116:0/3229222505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec039040 con 0x7f46f41066b0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f419c0c0 con 0x7f46f41066b0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f0ff9640 1 -- 172.21.15.116:0/3229222505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46ec044070 con 0x7f46f41066b0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f0ff9640 1 -- 172.21.15.116:0/3229222505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0042a0 con 0x7f46f41066b0 2024-03-19T02:39:06.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.610+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a39b0 con 0x7f46f41066b0 2024-03-19T02:39:06.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.613+0000 7f46fc178740 1 Processor -- start 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.613+0000 7f46f0ff9640 1 -- 172.21.15.116:0/3229222505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f46ec003970 con 0x7f46f41066b0 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.613+0000 7f46fc178740 1 -- start start 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a2b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 0x5588d2a34290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588d298e4b0 con 0x5588d2a347d0 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5588d2936d70 con 0x5588d2a2b2d0 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46fc178740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5588d298a550 con 0x5588d2a2bbf0 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52722/0 (socket says 172.21.15.116:52722) 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46ebfff640 1 -- 172.21.15.116:0/858480215 learned_addr learned my addr 172.21.15.116:0/858480215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 0x5588d2a34290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46ebfff640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a2b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f0ff9640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 -- 172.21.15.116:0/858480215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 msgr2=0x5588d2a2b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a2b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 -- 172.21.15.116:0/858480215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 msgr2=0x5588d2a36bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f2ffd640 1 -- 172.21.15.116:0/858480215 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d2959210 con 0x5588d2a2bbf0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46ebfff640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.614+0000 7f46f37fe640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 0x5588d2a34290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009c00 tx=0x7f46e4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be812c0e2ae4f84b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46f0ff9640 1 -- 172.21.15.116:0/3229222505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f46ec0f14f0 con 0x7f46f41066b0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46e97fa640 1 -- 172.21.15.116:0/858480215 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4004050 con 0x5588d2a2bbf0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46e97fa640 1 -- 172.21.15.116:0/858480215 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46e40054d0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46e97fa640 1 -- 172.21.15.116:0/858480215 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e4004ba0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46f37fe640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0eea70 tx=0x7f46ec0a8040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46fc178740 1 -- 172.21.15.116:0/858480215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 msgr2=0x5588d2a34290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.615+0000 7f46fc178740 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 0x5588d2a34290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009c00 tx=0x7f46e4011810 comp rx=0 tx=0).stop 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- 172.21.15.116:0/858480215 shutdown_connections 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a36bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2bbf0 0x5588d2a34290 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- 172.21.15.116:0/858480215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a2b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- 172.21.15.116:0/858480215 >> 172.21.15.116:0/858480215 conn(0x5588d2a25a00 msgr2=0x5588d2a25de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- 172.21.15.116:0/858480215 shutdown_connections 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- 172.21.15.116:0/858480215 wait complete. 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- start start 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 Processor -- start 2024-03-19T02:39:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588d2a375c0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5588d294a310 con 0x5588d2a2b2d0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5588d2a37740 con 0x5588d2a347d0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46fc178740 10 client.0 _ll_register_callbacks cb 0x5588d293e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52726/0 (socket says 172.21.15.116:52726) 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46f2ffd640 1 -- 172.21.15.116:0/4073955102 learned_addr learned my addr 172.21.15.116:0/4073955102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.616+0000 7f46ebfff640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 -- 172.21.15.116:0/4073955102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 msgr2=0x5588d2a321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 -- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 msgr2=0x5588d2a31760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a31760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 -- 172.21.15.116:0/4073955102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d26a23f0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46ebfff640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a31760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f46e401b040 tx=0x7f46e40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f4be6d74337c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e402f060 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f46e4009db0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46e40054d0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.617+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f46e4003ca0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46fc178740 10 client.18912 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5588d276c7d0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46e40054d0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f46e4021bc0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46fc178740 10 client.18912 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.618+0000 7f46fc178740 10 client.18912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46fc178740 20 client.18912 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2a44970 0x5588d2a46d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46fc178740 4 client.18912 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588d29ec3e0 con 0x5588d2a44970 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46ebfff640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2a44970 0x5588d2a46d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46da7fc640 10 client.18912 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.619+0000 7f46ebfff640 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2a44970 0x5588d2a46d60 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8756fc2922303538 server_cookie=44d92d0288d09771 in_seq=0 out_seq=0 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.620+0000 7f46da7fc640 1 -- 172.21.15.116:0/4073955102 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588d29ec3e0 con 0x5588d2a44970 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.620+0000 7f46da7fc640 10 client.18912 handle_command_reply: tid=0 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.620+0000 7f46fc178740 1 client.18912 shutdown 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.620+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 msgr2=0x5588d2a31ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.620+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f46e401b040 tx=0x7f46e40038a0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 shutdown_connections 2024-03-19T02:39:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2a44970 0x5588d2a46d60 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2bbf0 0x5588d2a31ca0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46da7fc640 0 client.18912 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 --2- 172.21.15.116:0/4073955102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2b2d0 0x5588d2a31760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 >> 172.21.15.116:0/4073955102 conn(0x5588d2a25a00 msgr2=0x5588d2a27750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 shutdown_connections 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 1 -- 172.21.15.116:0/4073955102 wait complete. 2024-03-19T02:39:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.621+0000 7f46fc178740 20 client.18912 trim_cache size 0 max 16384 2024-03-19T02:39:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.634+0000 7f46fc178740 1 Processor -- start 2024-03-19T02:39:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 -- start start 2024-03-19T02:39:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a40390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a4c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41acc80 con 0x5588d2a2b2d0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ae7c0 con 0x5588d2a2bbf0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46fc178740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ad350 con 0x5588d2a347d0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52730/0 (socket says 172.21.15.116:52730) 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 -- 172.21.15.116:0/2503047245 learned_addr learned my addr 172.21.15.116:0/2503047245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a40390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a4c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 -- 172.21.15.116:0/2503047245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 msgr2=0x5588d2a4c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a4c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 -- 172.21.15.116:0/2503047245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 msgr2=0x5588d2a40390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a40390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.635+0000 7f46ebfff640 1 -- 172.21.15.116:0/2503047245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d2a472f0 con 0x5588d2a2b2d0 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.636+0000 7f46ebfff640 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005fd0 tx=0x7f46dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259a573fd94360b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.636+0000 7f46e8ff9640 1 -- 172.21.15.116:0/2503047245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc003bc0 con 0x5588d2a2b2d0 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.636+0000 7f46e8ff9640 1 -- 172.21.15.116:0/2503047245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46dc003d60 con 0x5588d2a2b2d0 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.636+0000 7f46e8ff9640 1 -- 172.21.15.116:0/2503047245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc004e10 con 0x5588d2a2b2d0 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- 172.21.15.116:0/2503047245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 msgr2=0x5588d2a33e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f46dc005fd0 tx=0x7f46dc004730 comp rx=0 tx=0).stop 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- 172.21.15.116:0/2503047245 shutdown_connections 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a347d0 0x5588d2a4c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a40390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- 172.21.15.116:0/2503047245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a2b2d0 0x5588d2a33e90 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- 172.21.15.116:0/2503047245 >> 172.21.15.116:0/2503047245 conn(0x5588d2a25a00 msgr2=0x5588d2a24310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- 172.21.15.116:0/2503047245 shutdown_connections 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- 172.21.15.116:0/2503047245 wait complete. 2024-03-19T02:39:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- start start 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 Processor -- start 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2b2d0 0x5588d2a3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a37d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ae0f0 con 0x5588d2a347d0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41ada20 con 0x5588d2a2bbf0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41aac80 con 0x5588d2a2b2d0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.637+0000 7f46fc178740 10 client.0 _ll_register_callbacks cb 0x7f46dc009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52742/0 (socket says 172.21.15.116:52742) 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 -- 172.21.15.116:0/1585654726 learned_addr learned my addr 172.21.15.116:0/1585654726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a37d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2b2d0 0x5588d2a3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 -- 172.21.15.116:0/1585654726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2b2d0 msgr2=0x5588d2a3c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2b2d0 0x5588d2a3c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 -- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 msgr2=0x5588d2a37d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a37d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 -- 172.21.15.116:0/1585654726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d2a36bc0 con 0x5588d2a347d0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a37d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46ebfff640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0136e0 tx=0x7f46dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442e50e2e8c89581 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc036060 con 0x5588d2a347d0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7f46dc026130 con 0x5588d2a347d0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46dc003d60 con 0x5588d2a347d0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f46dc003bc0 con 0x5588d2a347d0 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46fc178740 10 client.18918 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:06.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5588d2a25830 con 0x5588d2a347d0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.638+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46dc003d60 con 0x5588d2a347d0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f46dc0219c0 con 0x5588d2a347d0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 10 client.18918 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 10 client.18918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 20 client.18918 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2aaaba0 0x5588d2aacfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 4 client.18918 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588d299b9f0 con 0x5588d2aaaba0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2aaaba0 0x5588d2aacfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46da7fc640 10 client.18918 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.639+0000 7f46f2ffd640 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2aaaba0 0x5588d2aacfc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a2e9fa06d2aa8ff server_cookie=ea01b18402579e53 in_seq=0 out_seq=0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46da7fc640 1 -- 172.21.15.116:0/1585654726 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588d299b9f0 con 0x5588d2aaaba0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46da7fc640 10 client.18918 handle_command_reply: tid=0 2024-03-19T02:39:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 client.18918 shutdown 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 msgr2=0x5588d2a38270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0136e0 tx=0x7f46dc0046c0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 shutdown_connections 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5588d2aaaba0 0x5588d2aacfc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588d2a347d0 0x5588d2a38270 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46da7fc640 0 client.18918 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588d2a2bbf0 0x5588d2a37d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 --2- 172.21.15.116:0/1585654726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588d2a2b2d0 0x5588d2a3c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 >> 172.21.15.116:0/1585654726 conn(0x5588d2a25a00 msgr2=0x5588d2a31fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 shutdown_connections 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 1 -- 172.21.15.116:0/1585654726 wait complete. 2024-03-19T02:39:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46fc178740 20 client.18918 trim_cache size 0 max 16384 2024-03-19T02:39:06.648 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 msgr2=0x7f46e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0eea70 tx=0x7f46ec0a8040 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 msgr2=0x7f46f419b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f46ec004af0 tx=0x7f46ec035ce0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 shutdown_connections 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46e00a2970 0x7f46e00a4e30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f41066b0 0x7f46f419b700 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f4101060 0x7f46f41973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 --2- 172.21.15.116:0/3229222505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f41006a0 0x7f46f4196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.640+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 >> 172.21.15.116:0/3229222505 conn(0x7f46f40778d0 msgr2=0x7f46f4102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.641+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 shutdown_connections 2024-03-19T02:39:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:06.641+0000 7f46f9590640 1 -- 172.21.15.116:0/3229222505 wait complete. 2024-03-19T02:39:06.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 366863 3.11 MB/sec execute 509 sec latency 34.166 ms 2024-03-19T02:39:07.652 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:39:07.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:39:07.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:39:07.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 -- 172.21.15.116:0/200568489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe01006e0 msgr2=0x7fefe0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 --2- 172.21.15.116:0/200568489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe01006e0 0x7fefe0100b40 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009900 tx=0x7fefd002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 -- 172.21.15.116:0/200568489 shutdown_connections 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 --2- 172.21.15.116:0/200568489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe0101080 0x7fefe010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 --2- 172.21.15.116:0/200568489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe01006e0 0x7fefe0100b40 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 --2- 172.21.15.116:0/200568489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01055c0 0x7fefe01059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 -- 172.21.15.116:0/200568489 >> 172.21.15.116:0/200568489 conn(0x7fefe00777b0 msgr2=0x7fefe0077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 -- 172.21.15.116:0/200568489 shutdown_connections 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.777+0000 7fefe8620640 1 -- 172.21.15.116:0/200568489 wait complete. 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 Processor -- start 2024-03-19T02:39:07.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 -- start start 2024-03-19T02:39:07.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe01055c0 0x7fefe01a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0110920 con 0x7fefe0101080 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe01107a0 con 0x7fefe01006e0 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe8620640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0110aa0 con 0x7fefe01055c0 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60870/0 (socket says 172.21.15.116:60870) 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38188/0 (socket says 172.21.15.116:38188) 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe63bc640 1 -- 172.21.15.116:0/3717140818 learned_addr learned my addr 172.21.15.116:0/3717140818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.778+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe01055c0 0x7fefe01a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 -- 172.21.15.116:0/3717140818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe01055c0 msgr2=0x7fefe01a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe01055c0 0x7fefe01a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 -- 172.21.15.116:0/3717140818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 msgr2=0x7fefe0198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 -- 172.21.15.116:0/3717140818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe01a43c0 con 0x7fefe0101080 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe63bc640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fefd0036040 tx=0x7fefd002d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefd77fe640 1 -- 172.21.15.116:0/3717140818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0043020 con 0x7fefe0101080 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe01a4590 con 0x7fefe0101080 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe01a4aa0 con 0x7fefe0101080 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefd77fe640 1 -- 172.21.15.116:0/3717140818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefd002dc90 con 0x7fefe0101080 2024-03-19T02:39:07.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.779+0000 7fefd77fe640 1 -- 172.21.15.116:0/3717140818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0003a50 con 0x7fefe0101080 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefd77fe640 1 -- 172.21.15.116:0/3717140818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fefd0046020 con 0x7fefe0101080 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 Processor -- start 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 -- start start 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d43f23c160 con 0x55d43f2d89d0 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d43f1e4e10 con 0x55d43f2d80b0 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.782+0000 7fefeb20d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d43f238200 con 0x55d43f2e5c90 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60874/0 (socket says 172.21.15.116:60874) 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 -- 172.21.15.116:0/2593930463 learned_addr learned my addr 172.21.15.116:0/2593930463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe63bc640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefd77fe640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 0x7fefcc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 -- 172.21.15.116:0/2593930463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 msgr2=0x55d43f2e8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 -- 172.21.15.116:0/2593930463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 msgr2=0x55d43f2e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 -- 172.21.15.116:0/2593930463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d43f0187e0 con 0x55d43f2d80b0 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefd77fe640 1 -- 172.21.15.116:0/3717140818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fefd00f09d0 con 0x7fefe0101080 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe63bc640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 0x7fefcc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fefd8010510 tx=0x7fefd8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72db5c7b73bdac42 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 0x7fefcc0a4e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fefd8021000 tx=0x7fefd801e000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefd4bfe640 1 -- 172.21.15.116:0/2593930463 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd80134c0 con 0x55d43f2d80b0 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefd4bfe640 1 -- 172.21.15.116:0/2593930463 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefd801d8b0 con 0x55d43f2d80b0 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.783+0000 7fefd4bfe640 1 -- 172.21.15.116:0/2593930463 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8011900 con 0x55d43f2d80b0 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 -- 172.21.15.116:0/2593930463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 msgr2=0x55d43f2d8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fefd8010510 tx=0x7fefd8009f90 comp rx=0 tx=0).stop 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 -- 172.21.15.116:0/2593930463 shutdown_connections 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2593930463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2d8490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 -- 172.21.15.116:0/2593930463 >> 172.21.15.116:0/2593930463 conn(0x55d43f2d2ab0 msgr2=0x55d43f2d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 -- 172.21.15.116:0/2593930463 shutdown_connections 2024-03-19T02:39:07.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.784+0000 7fefeb20d740 1 -- 172.21.15.116:0/2593930463 wait complete. 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 -- start start 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 Processor -- start 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d80b0 0x55d43f2e2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 0x55d43f2e3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d43f2e8a50 con 0x55d43f2d89d0 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d43f1f81a0 con 0x55d43f2e5c90 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d43f2e8bd0 con 0x55d43f2d80b0 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefeb20d740 10 client.0 _ll_register_callbacks cb 0x55d43ef4fc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38214/0 (socket says 172.21.15.116:38214) 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d80b0 0x55d43f2e2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe5bbb640 1 -- 172.21.15.116:0/2104127566 learned_addr learned my addr 172.21.15.116:0/2104127566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 0x55d43f2e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.785+0000 7fefe5bbb640 1 -- 172.21.15.116:0/2104127566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d80b0 msgr2=0x55d43f2e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d80b0 0x55d43f2e2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe5bbb640 1 -- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 msgr2=0x55d43f2e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 0x55d43f2e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe5bbb640 1 -- 172.21.15.116:0/2104127566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d43f078f70 con 0x55d43f2d89d0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 0x55d43f2e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ede60 tx=0x7fefd0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d9c48316848d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bc850 con 0x55d43f2d89d0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd801d8b0 con 0x55d43f2d89d0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fefd00bd170 con 0x55d43f2d89d0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fefd00bc670 con 0x55d43f2d89d0 2024-03-19T02:39:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefeb20d740 10 client.18930 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d43ef10170 con 0x55d43f2d89d0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.786+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefd801d8b0 con 0x55d43f2d89d0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fefd00c5020 con 0x55d43f2d89d0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 10 client.18930 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 10 client.18930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 20 client.18930 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f2f5ec0 0x55d43f2f82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 4 client.18930 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d43f2496a0 con 0x55d43f2f5ec0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefe63bc640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f2f5ec0 0x55d43f2f82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefc9ffb640 10 client.18930 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.787+0000 7fefe63bc640 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f2f5ec0 0x55d43f2f82e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4520ec9ed49aced2 server_cookie=68d36f1d7efbeacc in_seq=0 out_seq=0 2024-03-19T02:39:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefc9ffb640 1 -- 172.21.15.116:0/2104127566 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d43f2496a0 con 0x55d43f2f5ec0 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefc9ffb640 10 client.18930 handle_command_reply: tid=0 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 client.18930 shutdown 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 msgr2=0x55d43f2e3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ede60 tx=0x7fefd0005b50 comp rx=0 tx=0).stop 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 shutdown_connections 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f2f5ec0 0x55d43f2f82e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2e5c90 0x55d43f2e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e3100 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 --2- 172.21.15.116:0/2104127566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d80b0 0x55d43f2e2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 >> 172.21.15.116:0/2104127566 conn(0x55d43f2d2ab0 msgr2=0x55d43f2d4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefc9ffb640 0 client.18930 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 shutdown_connections 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.788+0000 7fefeb20d740 1 -- 172.21.15.116:0/2104127566 wait complete. 2024-03-19T02:39:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.789+0000 7fefeb20d740 20 client.18930 trim_cache size 0 max 16384 2024-03-19T02:39:07.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 367431 3.10 MB/sec execute 510 sec latency 21.327 ms 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 Processor -- start 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 -- start start 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2f1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefc4010080 con 0x55d43f2e5c90 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefc4011bc0 con 0x55d43f2d80b0 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefeb20d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefc4010750 con 0x55d43f2d89d0 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe6bbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe6bbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38230/0 (socket says 172.21.15.116:38230) 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:35886/0 (socket says 172.21.15.116:35886) 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe6bbd640 1 -- 172.21.15.116:0/4144129195 learned_addr learned my addr 172.21.15.116:0/4144129195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.809+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2f1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60880/0 (socket says 172.21.15.116:60880) 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 -- 172.21.15.116:0/4144129195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 msgr2=0x55d43f2f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 -- 172.21.15.116:0/4144129195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 msgr2=0x55d43f2fdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 -- 172.21.15.116:0/4144129195 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d43f2ce160 con 0x55d43f2d89d0 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fefd0004650 tx=0x7fefd00f0f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea6076f31cf578b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefcbfff640 1 -- 172.21.15.116:0/4144129195 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bc850 con 0x55d43f2d89d0 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefcbfff640 1 -- 172.21.15.116:0/4144129195 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefd002de40 con 0x55d43f2d89d0 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.810+0000 7fefcbfff640 1 -- 172.21.15.116:0/4144129195 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bd1d0 con 0x55d43f2d89d0 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 -- 172.21.15.116:0/4144129195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 msgr2=0x55d43f2d45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fefd0004650 tx=0x7fefd00f0f40 comp rx=0 tx=0).stop 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 -- 172.21.15.116:0/4144129195 shutdown_connections 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2e5c90 0x55d43f2fdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2d89d0 0x55d43f2d45f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 --2- 172.21.15.116:0/4144129195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 -- 172.21.15.116:0/4144129195 >> 172.21.15.116:0/4144129195 conn(0x55d43f2d2ab0 msgr2=0x55d43f2d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 -- 172.21.15.116:0/4144129195 shutdown_connections 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.811+0000 7fefeb20d740 1 -- 172.21.15.116:0/4144129195 wait complete. 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 -- start start 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 Processor -- start 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2ee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefc40114f0 con 0x55d43f2d89d0 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefc4010e20 con 0x55d43f2d80b0 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefc400e080 con 0x55d43f2e5c90 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefeb20d740 10 client.0 _ll_register_callbacks cb 0x7fefd00057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefe63bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefe5bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38232/0 (socket says 172.21.15.116:38232) 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefe5bbb640 1 -- 172.21.15.116:0/1252836374 learned_addr learned my addr 172.21.15.116:0/1252836374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.812+0000 7fefe6bbd640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 -- 172.21.15.116:0/1252836374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 msgr2=0x55d43f2e9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 -- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 msgr2=0x55d43f2ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2ee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 -- 172.21.15.116:0/1252836374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d43f2ea3f0 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe63bc640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefe5bbb640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fefd00bcca0 tx=0x7fefd00bd1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0bcb41986da2ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bc850 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fefd00b8a60 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d43f2439f0 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fefd00c54b0 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefeb20d740 10 client.18936 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d43f2ea3f0 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.813+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d43f2439f0 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fefd00bbd70 con 0x55d43f2d89d0 2024-03-19T02:39:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 10 client.18936 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 10 client.18936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 20 client.18936 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f35c610 0x55d43f35ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 4 client.18936 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d43f1c1be0 con 0x55d43f35c610 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefe63bc640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f35c610 0x55d43f35ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefe63bc640 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f35c610 0x55d43f35ea30 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4364b8a9059d2794 server_cookie=28f7b14e1754d7de in_seq=0 out_seq=0 2024-03-19T02:39:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.814+0000 7fefc9ffb640 10 client.18936 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.815+0000 7fefc9ffb640 1 -- 172.21.15.116:0/1252836374 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d43f1c1be0 con 0x55d43f35c610 2024-03-19T02:39:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.815+0000 7fefc9ffb640 10 client.18936 handle_command_reply: tid=0 2024-03-19T02:39:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.815+0000 7fefeb20d740 1 client.18936 shutdown 2024-03-19T02:39:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 msgr2=0x55d43f2e95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fefd00bcca0 tx=0x7fefd00bd1a0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 shutdown_connections 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d43f35c610 0x55d43f35ea30 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d43f2e5c90 0x55d43f2e9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d43f2d89d0 0x55d43f2e95e0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefc9ffb640 0 client.18936 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 --2- 172.21.15.116:0/1252836374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d43f2d80b0 0x55d43f2ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 >> 172.21.15.116:0/1252836374 conn(0x55d43f2d2ab0 msgr2=0x55d43f2d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 shutdown_connections 2024-03-19T02:39:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 1 -- 172.21.15.116:0/1252836374 wait complete. 2024-03-19T02:39:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.816+0000 7fefeb20d740 20 client.18936 trim_cache size 0 max 16384 2024-03-19T02:39:07.830 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:07.830 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:39:07.830 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "13b341cd-a5de-4367-a41e-ac58962d0f2c", 2024-03-19T02:39:07.830 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:39:07.830 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:07.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 msgr2=0x7fefcc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 0x7fefcc0a4e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fefd8021000 tx=0x7fefd801e000 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 msgr2=0x7fefe0199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fefd0036040 tx=0x7fefd002d890 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 shutdown_connections 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefcc0a2970 0x7fefcc0a4e30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefe01055c0 0x7fefe01a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe0101080 0x7fefe0199340 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 --2- 172.21.15.116:0/3717140818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe01006e0 0x7fefe0198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 >> 172.21.15.116:0/3717140818 conn(0x7fefe00777b0 msgr2=0x7fefe00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 shutdown_connections 2024-03-19T02:39:07.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.817+0000 7fefe8620640 1 -- 172.21.15.116:0/3717140818 wait complete. 2024-03-19T02:39:07.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:39:07.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 -- 172.21.15.116:0/4178061544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8100690 msgr2=0x7fd9b8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/4178061544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8100690 0x7fd9b8100b10 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009930 tx=0x7fd9a402bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 -- 172.21.15.116:0/4178061544 shutdown_connections 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/4178061544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/4178061544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8100690 0x7fd9b8100b10 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/4178061544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b81066a0 0x7fd9b8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 -- 172.21.15.116:0/4178061544 >> 172.21.15.116:0/4178061544 conn(0x7fd9b8077820 msgr2=0x7fd9b8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.959+0000 7fd9bf08a640 1 -- 172.21.15.116:0/4178061544 shutdown_connections 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 -- 172.21.15.116:0/4178061544 wait complete. 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 Processor -- start 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 -- start start 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110960 con 0x7fd9b81066a0 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81107e0 con 0x7fd9b8101050 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bf08a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110ae0 con 0x7fd9b8100690 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.960+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:35920/0 (socket says 172.21.15.116:35920) 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bce26640 1 -- 172.21.15.116:0/3898887005 learned_addr learned my addr 172.21.15.116:0/3898887005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 -- 172.21.15.116:0/3898887005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 msgr2=0x7fd9b8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 -- 172.21.15.116:0/3898887005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 msgr2=0x7fd9b8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 -- 172.21.15.116:0/3898887005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b819cfe0 con 0x7fd9b81066a0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bce26640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bd627640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c8b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009fd0 tx=0x7fd9b000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9b5ffb640 1 -- 172.21.15.116:0/3898887005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000ce20 con 0x7fd9b81066a0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b819d2d0 con 0x7fd9b81066a0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9b5ffb640 1 -- 172.21.15.116:0/3898887005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9b0005020 con 0x7fd9b81066a0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9b5ffb640 1 -- 172.21.15.116:0/3898887005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0010690 con 0x7fd9b81066a0 2024-03-19T02:39:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.961+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b81a4b60 con 0x7fd9b81066a0 2024-03-19T02:39:07.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.964+0000 7fd9b5ffb640 1 -- 172.21.15.116:0/3898887005 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd9b00051d0 con 0x7fd9b81066a0 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 Processor -- start 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 -- start start 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b5ffb640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 0x7fd9a80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce683350 0x5577ce68b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 0x5577ce68e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577ce5e6500 con 0x5577ce682a30 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5577ce58ebe0 con 0x5577ce68bf30 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5577ce5e25a0 con 0x5577ce683350 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38244/0 (socket says 172.21.15.116:38244) 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b7fff640 1 -- 172.21.15.116:0/2018615049 learned_addr learned my addr 172.21.15.116:0/2018615049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b5ffb640 1 -- 172.21.15.116:0/3898887005 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd9b00cf990 con 0x7fd9b81066a0 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9bd627640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 0x5577ce68e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce683350 0x5577ce68b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.965+0000 7fd9bce26640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 0x7fd9a80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 -- 172.21.15.116:0/2018615049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce683350 msgr2=0x5577ce68b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce683350 0x5577ce68b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 -- 172.21.15.116:0/2018615049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 msgr2=0x5577ce68e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 0x5577ce68e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 -- 172.21.15.116:0/2018615049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577ce5b0000 con 0x5577ce682a30 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9bd627640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 0x5577ce68e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x5577ce682500 tx=0x7fd9a4005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9906644001282302 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9bce26640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 0x7fd9a80a4e30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac011440 tx=0x7fd9ac005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9a2ffd640 1 -- 172.21.15.116:0/2018615049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a402c7e0 con 0x5577ce682a30 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9a2ffd640 1 -- 172.21.15.116:0/2018615049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a40035f0 con 0x5577ce682a30 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.966+0000 7fd9a2ffd640 1 -- 172.21.15.116:0/2018615049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40426d0 con 0x5577ce682a30 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2018615049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 msgr2=0x5577ce682e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x5577ce682500 tx=0x7fd9a4005e50 comp rx=0 tx=0).stop 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2018615049 shutdown_connections 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce68bf30 0x5577ce68e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce683350 0x5577ce68b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2018615049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce682e10 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2018615049 >> 172.21.15.116:0/2018615049 conn(0x5577ce67d160 msgr2=0x5577ce67d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2018615049 shutdown_connections 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2018615049 wait complete. 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 -- start start 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.967+0000 7fd9c1c71740 1 Processor -- start 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 0x5577ce689510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce689a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577ce6930d0 con 0x5577ce683350 2024-03-19T02:39:07.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5577ce5a2160 con 0x5577ce682a30 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5577ce693250 con 0x5577ce68bf30 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9c1c71740 10 client.0 _ll_register_callbacks cb 0x5577ce5dea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 0x5577ce689510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60934/0 (socket says 172.21.15.116:60934) 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 -- 172.21.15.116:0/2476709979 learned_addr learned my addr 172.21.15.116:0/2476709979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bd627640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce689a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 -- 172.21.15.116:0/2476709979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 msgr2=0x5577ce689a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce689a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 -- 172.21.15.116:0/2476709979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 msgr2=0x5577ce689510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 0x5577ce689510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bce26640 1 -- 172.21.15.116:0/2476709979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577ce3c47d0 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9bd627640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce689a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.968+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 0x5577ce689510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9bce26640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00fb90 tx=0x7fd9ac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7635d49e375e52d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac02b060 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a40035f0 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd9ac01cc90 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd9ac032050 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 10 client.28733 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5577ce2fa3f0 con 0x5577ce682a30 2024-03-19T02:39:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a40035f0 con 0x5577ce682a30 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd9ac01a390 con 0x5577ce682a30 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 10 client.28733 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 10 client.28733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 20 client.28733 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce6a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 4 client.28733 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.969+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577ce5f3a40 con 0x5577ce6a06a0 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.970+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce6a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.970+0000 7fd997fff640 10 client.28733 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.970+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce6a2af0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5ccee67ab3f8ddc server_cookie=a383ee5f47203960 in_seq=0 out_seq=0 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd997fff640 1 -- 172.21.15.116:0/2476709979 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577ce5f3a40 con 0x5577ce6a06a0 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd997fff640 10 client.28733 handle_command_reply: tid=0 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 client.28733 shutdown 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 msgr2=0x5577ce688fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00fb90 tx=0x7fd9ac004730 comp rx=0 tx=0).stop 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 shutdown_connections 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce6a2af0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce689a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd997fff640 0 client.28733 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce683350 0x5577ce689510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/2476709979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce682a30 0x5577ce688fd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 >> 172.21.15.116:0/2476709979 conn(0x5577ce67d160 msgr2=0x5577ce67eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 shutdown_connections 2024-03-19T02:39:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 1 -- 172.21.15.116:0/2476709979 wait complete. 2024-03-19T02:39:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.971+0000 7fd9c1c71740 20 client.28733 trim_cache size 0 max 16384 2024-03-19T02:39:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 Processor -- start 2024-03-19T02:39:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 -- start start 2024-03-19T02:39:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce69be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce6a8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81acb80 con 0x5577ce682a30 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ae6c0 con 0x5577ce683350 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.990+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ad250 con 0x5577ce68bf30 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce69be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38262/0 (socket says 172.21.15.116:38262) 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bce26640 1 -- 172.21.15.116:0/760832914 learned_addr learned my addr 172.21.15.116:0/760832914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce6a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 -- 172.21.15.116:0/760832914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 msgr2=0x5577ce69be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce69be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 -- 172.21.15.116:0/760832914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 msgr2=0x5577ce68b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 -- 172.21.15.116:0/760832914 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577ce683180 con 0x5577ce68bf30 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bce26640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce69be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9bd627640 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce6a8400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00cccb0 tx=0x7fd9b009b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc8db36b83d47a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9a27fc640 1 -- 172.21.15.116:0/760832914 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00a4360 con 0x5577ce68bf30 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9a27fc640 1 -- 172.21.15.116:0/760832914 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9b0005540 con 0x5577ce68bf30 2024-03-19T02:39:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.991+0000 7fd9a27fc640 1 -- 172.21.15.116:0/760832914 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00a4130 con 0x5577ce68bf30 2024-03-19T02:39:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- 172.21.15.116:0/760832914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 msgr2=0x5577ce6a8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce6a8400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9b00cccb0 tx=0x7fd9b009b830 comp rx=0 tx=0).stop 2024-03-19T02:39:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- 172.21.15.116:0/760832914 shutdown_connections 2024-03-19T02:39:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce6a8400 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce69be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/760832914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce68b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- 172.21.15.116:0/760832914 >> 172.21.15.116:0/760832914 conn(0x5577ce67d160 msgr2=0x5577ce683730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- 172.21.15.116:0/760832914 shutdown_connections 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- 172.21.15.116:0/760832914 wait complete. 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.992+0000 7fd9c1c71740 1 -- start start 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 Processor -- start 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce693c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce694340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81adff0 con 0x5577ce682a30 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ad920 con 0x5577ce683350 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81aab80 con 0x5577ce68bf30 2024-03-19T02:39:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9c1c71740 10 client.0 _ll_register_callbacks cb 0x7fd9ac009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38276/0 (socket says 172.21.15.116:38276) 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 -- 172.21.15.116:0/781578153 learned_addr learned my addr 172.21.15.116:0/781578153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce693c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bd627640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce694340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 -- 172.21.15.116:0/781578153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 msgr2=0x5577ce694340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce694340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 -- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 msgr2=0x5577ce693c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce693c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9bce26640 1 -- 172.21.15.116:0/781578153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577ce2ba0f0 con 0x5577ce682a30 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.993+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce693c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9bce26640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac01a760 tx=0x7fd9ac01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6151258caa614c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac011d50 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9598+0+0 (secure 0 0 0) 0x7fd9ac01a9d0 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9b0005540 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd9ac044050 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 10 client.18954 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5577ce7069b0 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9b0005540 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd9ac036810 con 0x5577ce682a30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 10 client.18954 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 10 client.18954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 20 client.18954 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce709130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 4 client.18954 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:39:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.994+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5577ce5edd90 con 0x5577ce6a06a0 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce709130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9b7fff640 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce709130 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=151b4bb691a29d8e server_cookie=83522c1f0256f035 in_seq=0 out_seq=0 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd997fff640 10 client.18954 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd997fff640 1 -- 172.21.15.116:0/781578153 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5577ce5edd90 con 0x5577ce6a06a0 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd997fff640 10 client.18954 handle_command_reply: tid=0 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 client.18954 shutdown 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 msgr2=0x5577ce698890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac01a760 tx=0x7fd9ac01afd0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 shutdown_connections 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5577ce6a06a0 0x5577ce709130 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577ce68bf30 0x5577ce694340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd997fff640 0 client.18954 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577ce683350 0x5577ce693c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 --2- 172.21.15.116:0/781578153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577ce682a30 0x5577ce698890 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.995+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 >> 172.21.15.116:0/781578153 conn(0x5577ce67d160 msgr2=0x5577ce67b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 shutdown_connections 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9c1c71740 1 -- 172.21.15.116:0/781578153 wait complete. 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9c1c71740 20 client.18954 trim_cache size 0 max 16384 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "13b341cd-a5de-4367-a41e-ac58962d0f2c": { 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "13b341cd-a5de-4367-a41e-ac58962d0f2c", 2024-03-19T02:39:08.005 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:39:08.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:39:08.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:39:08.006 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 msgr2=0x7fd9a80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 0x7fd9a80a4e30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac011440 tx=0x7fd9ac005f90 comp rx=0 tx=0).stop 2024-03-19T02:39:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 msgr2=0x7fd9b819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c8b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009fd0 tx=0x7fd9b000ef10 comp rx=0 tx=0).stop 2024-03-19T02:39:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 shutdown_connections 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd9a80a2970 0x7fd9a80a4e30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c8b0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b8101050 0x7fd9b8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 --2- 172.21.15.116:0/3898887005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b8100690 0x7fd9b8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 >> 172.21.15.116:0/3898887005 conn(0x7fd9b8077820 msgr2=0x7fd9b8068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 shutdown_connections 2024-03-19T02:39:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:07.996+0000 7fd9bf08a640 1 -- 172.21.15.116:0/3898887005 wait complete. 2024-03-19T02:39:08.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:07 smithi116 ceph-mon[32263]: pgmap v353: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:08.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:07 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:39:08.012 INFO:tasks.cephfs.filesystem:scrub status for tag:13b341cd-a5de-4367-a41e-ac58962d0f2c - {'path': '/', 'tag': '13b341cd-a5de-4367-a41e-ac58962d0f2c', 'options': 'recursive,force'} 2024-03-19T02:39:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:07 smithi162 ceph-mon[35761]: pgmap v353: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:07 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:39:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:07 smithi154 ceph-mon[35693]: pgmap v353: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:07 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:39:08.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 367930 3.10 MB/sec execute 511 sec latency 30.676 ms 2024-03-19T02:39:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:39:09.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:08 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:08 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:39:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:39:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:08 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:09.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 368655 3.11 MB/sec execute 512 sec latency 21.715 ms 2024-03-19T02:39:10.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:09 smithi162 ceph-mon[35761]: pgmap v354: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:10.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:09 smithi154 ceph-mon[35693]: pgmap v354: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:09 smithi116 ceph-mon[32263]: pgmap v354: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-03-19T02:39:10.795 INFO:tasks.workunit.client.0.smithi116.stdout: 1 369118 3.11 MB/sec execute 513 sec latency 18.392 ms 2024-03-19T02:39:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:10 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:10 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:10 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:39:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:10 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:39:11.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:10 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:10 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:10 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:10 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:39:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:10 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:39:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:10 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:10 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:10 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:10 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:39:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:10 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:39:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:10 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:39:11.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 369495 3.10 MB/sec execute 514 sec latency 29.681 ms 2024-03-19T02:39:12.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:11 smithi162 ceph-mon[35761]: pgmap v355: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:11 smithi154 ceph-mon[35693]: pgmap v355: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:11 smithi116 ceph-mon[32263]: pgmap v355: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:12.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 370067 3.10 MB/sec execute 515 sec latency 20.816 ms 2024-03-19T02:39:13.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 370590 3.10 MB/sec execute 516 sec latency 30.948 ms 2024-03-19T02:39:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:13 smithi154 ceph-mon[35693]: pgmap v356: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:13 smithi116 ceph-mon[32263]: pgmap v356: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:13 smithi162 ceph-mon[35761]: pgmap v356: 129 pgs: 129 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-03-19T02:39:14.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 371223 3.10 MB/sec execute 517 sec latency 26.877 ms 2024-03-19T02:39:15.799 INFO:tasks.workunit.client.0.smithi116.stdout: 1 371912 3.10 MB/sec execute 518 sec latency 15.728 ms 2024-03-19T02:39:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:39:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:39:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:15 smithi154 ceph-mon[35693]: pgmap v357: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:39:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:39:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:15 smithi116 ceph-mon[32263]: pgmap v357: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:39:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:39:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:15 smithi162 ceph-mon[35761]: pgmap v357: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:16.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 372427 3.11 MB/sec execute 519 sec latency 26.105 ms 2024-03-19T02:39:17.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 372862 3.10 MB/sec execute 520 sec latency 19.539 ms 2024-03-19T02:39:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:17 smithi162 ceph-mon[35761]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T02:39:18.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:17 smithi162 ceph-mon[35761]: pgmap v359: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:17 smithi154 ceph-mon[35693]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T02:39:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:17 smithi154 ceph-mon[35693]: pgmap v359: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:17 smithi116 ceph-mon[32263]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T02:39:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:17 smithi116 ceph-mon[32263]: pgmap v359: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:18.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 373322 3.10 MB/sec execute 521 sec latency 36.885 ms 2024-03-19T02:39:19.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 373924 3.10 MB/sec execute 522 sec latency 26.546 ms 2024-03-19T02:39:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:19 smithi154 ceph-mon[35693]: pgmap v360: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:19 smithi116 ceph-mon[32263]: pgmap v360: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:19 smithi162 ceph-mon[35761]: pgmap v360: 129 pgs: 129 active+clean; 418 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-03-19T02:39:20.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 374477 3.10 MB/sec execute 523 sec latency 22.797 ms 2024-03-19T02:39:21.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 375035 3.10 MB/sec execute 524 sec latency 19.955 ms 2024-03-19T02:39:22.218 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:21 smithi162 ceph-mon[35761]: pgmap v361: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:21 smithi154 ceph-mon[35693]: pgmap v361: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:21 smithi116 ceph-mon[32263]: pgmap v361: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 375755 3.10 MB/sec execute 525 sec latency 16.567 ms 2024-03-19T02:39:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 376254 3.10 MB/sec execute 526 sec latency 17.964 ms 2024-03-19T02:39:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:23 smithi154 ceph-mon[35693]: pgmap v362: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:23 smithi116 ceph-mon[32263]: pgmap v362: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:24.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:23 smithi162 ceph-mon[35761]: pgmap v362: 129 pgs: 129 active+clean; 426 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 98 op/s 2024-03-19T02:39:24.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 376653 3.10 MB/sec execute 527 sec latency 39.888 ms 2024-03-19T02:39:25.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 377259 3.10 MB/sec execute 528 sec latency 17.634 ms 2024-03-19T02:39:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:25 smithi154 ceph-mon[35693]: pgmap v363: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-19T02:39:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:25 smithi116 ceph-mon[32263]: pgmap v363: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-19T02:39:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:25 smithi162 ceph-mon[35761]: pgmap v363: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-19T02:39:26.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 377803 3.10 MB/sec execute 529 sec latency 33.175 ms 2024-03-19T02:39:27.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 378415 3.10 MB/sec execute 530 sec latency 22.466 ms 2024-03-19T02:39:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:27 smithi154 ceph-mon[35693]: pgmap v364: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 93 op/s 2024-03-19T02:39:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:27 smithi116 ceph-mon[32263]: pgmap v364: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 93 op/s 2024-03-19T02:39:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:27 smithi162 ceph-mon[35761]: pgmap v364: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 93 op/s 2024-03-19T02:39:28.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 379298 3.10 MB/sec execute 531 sec latency 17.558 ms 2024-03-19T02:39:29.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 379688 3.10 MB/sec execute 532 sec latency 19.920 ms 2024-03-19T02:39:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:29 smithi154 ceph-mon[35693]: pgmap v365: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:29 smithi116 ceph-mon[32263]: pgmap v365: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:29 smithi162 ceph-mon[35761]: pgmap v365: 129 pgs: 129 active+clean; 431 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:30.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 380097 3.10 MB/sec execute 533 sec latency 17.493 ms 2024-03-19T02:39:31.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 380619 3.10 MB/sec execute 534 sec latency 36.109 ms 2024-03-19T02:39:32.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:31 smithi154 ceph-mon[35693]: pgmap v366: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:31 smithi116 ceph-mon[32263]: pgmap v366: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:32.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:31 smithi162 ceph-mon[35761]: pgmap v366: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:32.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 381186 3.10 MB/sec execute 535 sec latency 34.198 ms 2024-03-19T02:39:33.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 381787 3.09 MB/sec execute 536 sec latency 21.752 ms 2024-03-19T02:39:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:33 smithi154 ceph-mon[35693]: pgmap v367: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:33 smithi116 ceph-mon[32263]: pgmap v367: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:34.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:33 smithi162 ceph-mon[35761]: pgmap v367: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-03-19T02:39:34.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 382477 3.10 MB/sec execute 537 sec latency 17.079 ms 2024-03-19T02:39:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:34 smithi154 ceph-mon[35693]: pgmap v368: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:34 smithi116 ceph-mon[32263]: pgmap v368: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:34 smithi162 ceph-mon[35761]: pgmap v368: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-03-19T02:39:35.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 382911 3.10 MB/sec execute 538 sec latency 17.013 ms 2024-03-19T02:39:36.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 383432 3.10 MB/sec execute 539 sec latency 18.324 ms 2024-03-19T02:39:37.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 383862 3.10 MB/sec execute 540 sec latency 48.638 ms 2024-03-19T02:39:38.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:39:38.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 -- 172.21.15.116:0/2606397792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc674106660 msgr2=0x7fc674106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 --2- 172.21.15.116:0/2606397792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc674106660 0x7fc674106a40 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc664009900 tx=0x7fc66402bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 -- 172.21.15.116:0/2606397792 shutdown_connections 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 --2- 172.21.15.116:0/2606397792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674100f60 0x7fc67410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 --2- 172.21.15.116:0/2606397792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6741005a0 0x7fc674100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 --2- 172.21.15.116:0/2606397792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc674106660 0x7fc674106a40 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 -- 172.21.15.116:0/2606397792 >> 172.21.15.116:0/2606397792 conn(0x7fc6740fc070 msgr2=0x7fc6740fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 -- 172.21.15.116:0/2606397792 shutdown_connections 2024-03-19T02:39:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.134+0000 7fc67359e640 1 -- 172.21.15.116:0/2606397792 wait complete. 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 Processor -- start 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 -- start start 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 0x7fc674198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674106660 0x7fc67419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc674110850 con 0x7fc6741005a0 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741106d0 con 0x7fc674100f60 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.135+0000 7fc67359e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741109d0 con 0x7fc674106660 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53122/0 (socket says 172.21.15.116:53122) 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 -- 172.21.15.116:0/535816691 learned_addr learned my addr 172.21.15.116:0/535816691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc671d9b640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 0x7fc674198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc672d9d640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674106660 0x7fc67419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 -- 172.21.15.116:0/535816691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674106660 msgr2=0x7fc67419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674106660 0x7fc67419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 -- 172.21.15.116:0/535816691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 msgr2=0x7fc674198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 0x7fc674198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 -- 172.21.15.116:0/535816691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67419ce00 con 0x7fc6741005a0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc671d9b640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 0x7fc674198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67259c640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc664006010 tx=0x7fc664005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc6637fe640 1 -- 172.21.15.116:0/535816691 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664043020 con 0x7fc6741005a0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67419d090 con 0x7fc6741005a0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc6637fe640 1 -- 172.21.15.116:0/535816691 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc664004350 con 0x7fc6741005a0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc6637fe640 1 -- 172.21.15.116:0/535816691 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664003a10 con 0x7fc6741005a0 2024-03-19T02:39:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.136+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6741a4980 con 0x7fc6741005a0 2024-03-19T02:39:38.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.138+0000 7fc6637fe640 1 -- 172.21.15.116:0/535816691 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc664046020 con 0x7fc6741005a0 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc6637fe640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 0x7fc65c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 Processor -- start 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc671d9b640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 0x7fc65c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc6637fe640 1 -- 172.21.15.116:0/535816691 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7fc6640f0be0 con 0x7fc6741005a0 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 -- start start 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d573fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556e5d4ca500 con 0x556e5d574500 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556e5d472be0 con 0x556e5d567240 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.139+0000 7fc67b596740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556e5d4c65a0 con 0x556e5d566920 2024-03-19T02:39:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc671d9b640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 0x7fc65c0a4e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc674105c30 tx=0x7fc668005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37386/0 (socket says 172.21.15.116:37386) 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 -- 172.21.15.116:0/2882217153 learned_addr learned my addr 172.21.15.116:0/2882217153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc672d9d640 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc671d9b640 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d573fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 -- 172.21.15.116:0/2882217153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 msgr2=0x556e5d573fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d573fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 -- 172.21.15.116:0/2882217153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 msgr2=0x556e5d5768f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5768f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 -- 172.21.15.116:0/2882217153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e5d494000 con 0x556e5d566920 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc67259c640 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc6640b93d0 tx=0x7fc6640bc990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9508917c772bacf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc660ff9640 1 -- 172.21.15.116:0/2882217153 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640c10c0 con 0x556e5d566920 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc660ff9640 1 -- 172.21.15.116:0/2882217153 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6640c5710 con 0x556e5d566920 2024-03-19T02:39:38.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.140+0000 7fc660ff9640 1 -- 172.21.15.116:0/2882217153 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640c5280 con 0x556e5d566920 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- 172.21.15.116:0/2882217153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 msgr2=0x556e5d566d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc6640b93d0 tx=0x7fc6640bc990 comp rx=0 tx=0).stop 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- 172.21.15.116:0/2882217153 shutdown_connections 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5768f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d573fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 --2- 172.21.15.116:0/2882217153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d566d00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- 172.21.15.116:0/2882217153 >> 172.21.15.116:0/2882217153 conn(0x556e5d561320 msgr2=0x556e5d561700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- 172.21.15.116:0/2882217153 shutdown_connections 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- 172.21.15.116:0/2882217153 wait complete. 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.141+0000 7fc67b596740 1 -- start start 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 Processor -- start 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d5719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d571f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556e5d4c65a0 con 0x556e5d566920 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556e5d486160 con 0x556e5d567240 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556e5d577080 con 0x556e5d574500 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 10 client.0 _ll_register_callbacks cb 0x556e5d4c2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53144/0 (socket says 172.21.15.116:53144) 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 -- 172.21.15.116:0/3138230630 learned_addr learned my addr 172.21.15.116:0/3138230630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc672d9d640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d571f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc671d9b640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d5719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 -- 172.21.15.116:0/3138230630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 msgr2=0x556e5d571f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d571f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 -- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 msgr2=0x556e5d5719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d5719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 -- 172.21.15.116:0/3138230630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e5d2a87d0 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67259c640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ee550 tx=0x7fc6640bd0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f635f3d8c77793 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640c5280 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7fc6640c5420 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.142+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6640c5710 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc6640a3050 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 10 client.18966 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556e5d1de3f0 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6640c5710 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc6640b8cc0 con 0x556e5d566920 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 10 client.18966 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 10 client.18966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 20 client.18966 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5842d0 0x556e5d586770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 4 client.18966 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e5d4d7a40 con 0x556e5d5842d0 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.143+0000 7fc671d9b640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5842d0 0x556e5d586770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.144+0000 7fc671d9b640 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5842d0 0x556e5d586770 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51c6609426c50142 server_cookie=a632292435900812 in_seq=0 out_seq=0 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.144+0000 7fc659ffb640 10 client.18966 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc659ffb640 1 -- 172.21.15.116:0/3138230630 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e5d4d7a40 con 0x556e5d5842d0 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc659ffb640 10 client.18966 handle_command_reply: tid=0 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 client.18966 shutdown 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 msgr2=0x556e5d5714b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ee550 tx=0x7fc6640bd0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 shutdown_connections 2024-03-19T02:39:38.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5842d0 0x556e5d586770 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d571f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d5719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc659ffb640 0 client.18966 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 --2- 172.21.15.116:0/3138230630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d5714b0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 >> 172.21.15.116:0/3138230630 conn(0x556e5d561320 msgr2=0x556e5d563070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 shutdown_connections 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 1 -- 172.21.15.116:0/3138230630 wait complete. 2024-03-19T02:39:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.145+0000 7fc67b596740 20 client.18966 trim_cache size 0 max 16384 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 Processor -- start 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 -- start start 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d58c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d57c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ac780 con 0x556e5d566920 2024-03-19T02:39:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ae2c0 con 0x556e5d567240 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67b596740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ace50 con 0x556e5d574500 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53160/0 (socket says 172.21.15.116:53160) 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc67259c640 1 -- 172.21.15.116:0/30569125 learned_addr learned my addr 172.21.15.116:0/30569125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc671d9b640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d58c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.165+0000 7fc672d9d640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d57c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 -- 172.21.15.116:0/30569125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 msgr2=0x556e5d57c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d57c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 -- 172.21.15.116:0/30569125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 msgr2=0x556e5d58c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d58c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 -- 172.21.15.116:0/30569125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e5d19e0f0 con 0x556e5d566920 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc671d9b640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d58c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc67259c640 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc6640bcc70 tx=0x7fc6640a6390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea9f300747d4697 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc65bfff640 1 -- 172.21.15.116:0/30569125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66409f020 con 0x556e5d566920 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc65bfff640 1 -- 172.21.15.116:0/30569125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6640da070 con 0x556e5d566920 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.166+0000 7fc65bfff640 1 -- 172.21.15.116:0/30569125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640c10c0 con 0x556e5d566920 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- 172.21.15.116:0/30569125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 msgr2=0x556e5d57fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc6640bcc70 tx=0x7fc6640a6390 comp rx=0 tx=0).stop 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- 172.21.15.116:0/30569125 shutdown_connections 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d574500 0x556e5d57c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d58c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- 172.21.15.116:0/30569125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d566920 0x556e5d57fca0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- 172.21.15.116:0/30569125 >> 172.21.15.116:0/30569125 conn(0x556e5d561320 msgr2=0x556e5d55fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- 172.21.15.116:0/30569125 shutdown_connections 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- 172.21.15.116:0/30569125 wait complete. 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- start start 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 Processor -- start 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d56e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5e96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741adbf0 con 0x556e5d574500 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ad520 con 0x556e5d567240 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741aa780 con 0x556e5d566920 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.167+0000 7fc67b596740 10 client.0 _ll_register_callbacks cb 0x7fc66c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc67259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d56e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55476/0 (socket says 172.21.15.116:55476) 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 -- 172.21.15.116:0/1550431785 learned_addr learned my addr 172.21.15.116:0/1550431785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc672d9d640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5e96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 -- 172.21.15.116:0/1550431785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 msgr2=0x556e5d56e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d56e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 -- 172.21.15.116:0/1550431785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 msgr2=0x556e5d5e96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5e96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 -- 172.21.15.116:0/1550431785 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e5d5e9c00 con 0x556e5d567240 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc671d9b640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc668009e80 tx=0x7fc668004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c9576ef3f135be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66802c220 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6640da070 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7fc66802c3c0 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc66803c050 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc67b596740 10 client.28757 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.168+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556e5d5eae40 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6640da070 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fc66801a040 con 0x556e5d567240 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 10 client.28757 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 10 client.28757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 20 client.28757 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5eb5a0 0x556e5d5ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 4 client.28757 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556e5d4d1d90 con 0x556e5d5eb5a0 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67259c640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5eb5a0 0x556e5d5ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc67259c640 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5eb5a0 0x556e5d5ed9c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c1eedbdb499b3e4 server_cookie=7d57dde731f5643c in_seq=0 out_seq=0 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.169+0000 7fc659ffb640 10 client.28757 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.171+0000 7fc659ffb640 1 -- 172.21.15.116:0/1550431785 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556e5d4d1d90 con 0x556e5d5eb5a0 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.171+0000 7fc659ffb640 10 client.28757 handle_command_reply: tid=0 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.171+0000 7fc67b596740 1 client.28757 shutdown 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 msgr2=0x556e5d56eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc668009e80 tx=0x7fc668004730 comp rx=0 tx=0).stop 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 shutdown_connections 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556e5d5eb5a0 0x556e5d5ed9c0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556e5d574500 0x556e5d5e96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc659ffb640 0 client.28757 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556e5d567240 0x556e5d56eb60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 --2- 172.21.15.116:0/1550431785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556e5d566920 0x556e5d56e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 >> 172.21.15.116:0/1550431785 conn(0x556e5d561320 msgr2=0x556e5d5604a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 shutdown_connections 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 1 -- 172.21.15.116:0/1550431785 wait complete. 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67b596740 20 client.28757 trim_cache size 0 max 16384 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:38.181 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:39:38.182 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:39:38.182 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 msgr2=0x7fc65c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 0x7fc65c0a4e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc674105c30 tx=0x7fc668005f90 comp rx=0 tx=0).stop 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 msgr2=0x7fc674197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc664006010 tx=0x7fc664005d90 comp rx=0 tx=0).stop 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 shutdown_connections 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc65c0a2970 0x7fc65c0a4e30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc674106660 0x7fc67419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc674100f60 0x7fc674198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 --2- 172.21.15.116:0/535816691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6741005a0 0x7fc674197e10 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.172+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 >> 172.21.15.116:0/535816691 conn(0x7fc6740fc070 msgr2=0x7fc6740fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.173+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 shutdown_connections 2024-03-19T02:39:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.173+0000 7fc67359e640 1 -- 172.21.15.116:0/535816691 wait complete. 2024-03-19T02:39:38.184 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:39:38.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:39:38.210 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:37 smithi116 ceph-mon[32263]: pgmap v369: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:37 smithi154 ceph-mon[35693]: pgmap v369: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:38.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- 172.21.15.116:0/279018541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae601010e0 msgr2=0x7fae6010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 --2- 172.21.15.116:0/279018541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae601010e0 0x7fae6010d5f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fae58009cd0 tx=0x7fae5802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- 172.21.15.116:0/279018541 shutdown_connections 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 --2- 172.21.15.116:0/279018541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae601010e0 0x7fae6010d5f0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 --2- 172.21.15.116:0/279018541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60100720 0x7fae60100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 --2- 172.21.15.116:0/279018541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae60106730 0x7fae60106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- 172.21.15.116:0/279018541 >> 172.21.15.116:0/279018541 conn(0x7fae60077820 msgr2=0x7fae60077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- 172.21.15.116:0/279018541 shutdown_connections 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- 172.21.15.116:0/279018541 wait complete. 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 Processor -- start 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 -- start start 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.300+0000 7fae6737e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae601010e0 0x7fae60198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60106730 0x7fae6019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae601109f0 con 0x7fae60100720 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae60110870 con 0x7fae60106730 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae60110b70 con 0x7fae601010e0 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53186/0 (socket says 172.21.15.116:53186) 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 -- 172.21.15.116:0/2178922531 learned_addr learned my addr 172.21.15.116:0/2178922531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6591b640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60106730 0x7fae6019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae64919640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae601010e0 0x7fae60198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 -- 172.21.15.116:0/2178922531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae601010e0 msgr2=0x7fae60198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae601010e0 0x7fae60198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 -- 172.21.15.116:0/2178922531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60106730 msgr2=0x7fae6019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60106730 0x7fae6019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 -- 172.21.15.116:0/2178922531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6019cff0 con 0x7fae60100720 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6511a640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9f0 tx=0x7fae5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c00cd80 con 0x7fae60100720 2024-03-19T02:39:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6019d2e0 con 0x7fae60100720 2024-03-19T02:39:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae5c0054d0 con 0x7fae60100720 2024-03-19T02:39:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c010490 con 0x7fae60100720 2024-03-19T02:39:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.301+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae601a4b70 con 0x7fae60100720 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.302+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae60068610 con 0x7fae60100720 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.303+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fae5c005020 con 0x7fae60100720 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.304+0000 7fae567fc640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 0x7fae4c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.304+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7fae5c0cf520 con 0x7fae60100720 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.304+0000 7fae64919640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 0x7fae4c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.305+0000 7fae64919640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 0x7fae4c0a4e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae60102260 tx=0x7fae50005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.310+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae5c09bea0 con 0x7fae60100720 2024-03-19T02:39:38.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:37 smithi162 ceph-mon[35761]: pgmap v369: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.562+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae60199350 con 0x7fae60100720 2024-03-19T02:39:38.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.563+0000 7fae567fc640 1 -- 172.21.15.116:0/2178922531 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fae5c09b7a0 con 0x7fae60100720 2024-03-19T02:39:38.563 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:39:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:39:38.563 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 msgr2=0x7fae4c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 0x7fae4c0a4e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae60102260 tx=0x7fae50005f90 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 msgr2=0x7fae60198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9f0 tx=0x7fae5c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 shutdown_connections 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae4c0a2970 0x7fae4c0a4e30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60106730 0x7fae6019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae601010e0 0x7fae60198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 --2- 172.21.15.116:0/2178922531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae60100720 0x7fae60198000 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 >> 172.21.15.116:0/2178922531 conn(0x7fae60077820 msgr2=0x7fae60068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 shutdown_connections 2024-03-19T02:39:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.565+0000 7fae6737e640 1 -- 172.21.15.116:0/2178922531 wait complete. 2024-03-19T02:39:38.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:39:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/2904430622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 msgr2=0x7f9ca0107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/2904430622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca0107b80 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009930 tx=0x7f9c9002bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/2904430622 shutdown_connections 2024-03-19T02:39:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/2904430622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0102150 0x7f9ca010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/2904430622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0101790 0x7f9ca0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/2904430622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca0107b80 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/2904430622 >> 172.21.15.116:0/2904430622 conn(0x7f9ca00fd2c0 msgr2=0x7f9ca00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/2904430622 shutdown_connections 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.733+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/2904430622 wait complete. 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 Processor -- start 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 -- start start 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 0x7f9ca0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0102150 0x7f9ca0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0111a60 con 0x7f9ca01077a0 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01118e0 con 0x7f9ca0101790 2024-03-19T02:39:38.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9ca4d3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0111be0 con 0x7f9ca0102150 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 0x7f9ca0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53206/0 (socket says 172.21.15.116:53206) 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0102150 0x7f9ca0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 -- 172.21.15.116:0/1278506384 learned_addr learned my addr 172.21.15.116:0/1278506384 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 0x7f9ca0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55514/0 (socket says 172.21.15.116:55514) 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 -- 172.21.15.116:0/1278506384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0102150 msgr2=0x7f9ca0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0102150 0x7f9ca0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 -- 172.21.15.116:0/1278506384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 msgr2=0x7f9ca0198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 0x7f9ca0198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 -- 172.21.15.116:0/1278506384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca019d020 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.734+0000 7f9c9ed9d640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009fd0 tx=0x7f9c9800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.735+0000 7f9c8f7fe640 1 -- 172.21.15.116:0/1278506384 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98017070 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.735+0000 7f9c8f7fe640 1 -- 172.21.15.116:0/1278506384 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c98003e20 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.735+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca019d310 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.735+0000 7f9c8f7fe640 1 -- 172.21.15.116:0/1278506384 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98010460 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.735+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a4ba0 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.737+0000 7f9c8f7fe640 1 -- 172.21.15.116:0/1278506384 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9c98003970 con 0x7f9ca01077a0 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.737+0000 7f9ca79ff740 1 Processor -- start 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 -- start start 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557fda2a44b0 con 0x557fda3412d0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557fda24cd70 con 0x557fda341bf0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fda2a0550 con 0x557fda34a7d0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c8f7fe640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 0x7f9c880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53222/0 (socket says 172.21.15.116:53222) 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2918113414 learned_addr learned my addr 172.21.15.116:0/2918113414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9ed9d640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 0x7f9c880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c8f7fe640 1 -- 172.21.15.116:0/1278506384 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f9c980cef50 con 0x7f9ca01077a0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2918113414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 msgr2=0x557fda34cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2918113414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 msgr2=0x557fda34a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2918113414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fda26f210 con 0x557fda3412d0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.738+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9c94013630 tx=0x7f9c94009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e3b9c16fe5245f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9c9ed9d640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 0x7f9c880a4e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9c980976e0 tx=0x7f9c98019e70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9c8cff9640 1 -- 172.21.15.116:0/2918113414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94004050 con 0x557fda3412d0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9c8cff9640 1 -- 172.21.15.116:0/2918113414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c940041f0 con 0x557fda3412d0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9c8cff9640 1 -- 172.21.15.116:0/2918113414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400f7c0 con 0x557fda3412d0 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2918113414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 msgr2=0x557fda3416b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.739+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9c94013630 tx=0x7f9c94009780 comp rx=0 tx=0).stop 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2918113414 shutdown_connections 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2918113414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda3416b0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2918113414 >> 172.21.15.116:0/2918113414 conn(0x557fda33ba00 msgr2=0x557fda33bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2918113414 shutdown_connections 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2918113414 wait complete. 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- start start 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 Processor -- start 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda3412d0 0x557fda347830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 0x557fda347d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557fda34d5c0 con 0x557fda34a7d0 2024-03-19T02:39:38.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557fda260310 con 0x557fda3412d0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557fda34d740 con 0x557fda341bf0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9ca79ff740 10 client.0 _ll_register_callbacks cb 0x557fda2542d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53228/0 (socket says 172.21.15.116:53228) 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2357681773 learned_addr learned my addr 172.21.15.116:0/2357681773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda3412d0 0x557fda347830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.740+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 0x557fda347d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 msgr2=0x557fda347d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 0x557fda347d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2357681773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda3412d0 msgr2=0x557fda347830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda3412d0 0x557fda347830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2357681773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fd9fb83f0 con 0x557fda34a7d0 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 0x557fda347d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f9c94006010 tx=0x7f9c94003e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c7bb49fe95bbb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94028040 con 0x557fda34a7d0 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c940041f0 con 0x557fda34a7d0 2024-03-19T02:39:38.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f9c9400f580 con 0x557fda34a7d0 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9c94003ba0 con 0x557fda34a7d0 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9ca79ff740 10 client.18990 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557fda0827d0 con 0x557fda34a7d0 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.741+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c940041f0 con 0x557fda34a7d0 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9c940106c0 con 0x557fda34a7d0 2024-03-19T02:39:38.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 10 client.18990 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 10 client.18990 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:39:38.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 20 client.18990 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda35a9e0 0x557fda35cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 4 client.18990 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:38.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fda3023e0 con 0x557fda35a9e0 2024-03-19T02:39:38.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda35a9e0 0x557fda35cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda35a9e0 0x557fda35cdd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=185b23a22d94cf1c server_cookie=c0ffbd0ca7ad6192 in_seq=0 out_seq=0 2024-03-19T02:39:38.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.742+0000 7f9c85ffb640 10 client.18990 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:38.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2357681773 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557fda3023e0 con 0x557fda35a9e0 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9c85ffb640 10 client.18990 handle_command_reply: tid=0 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 client.18990 shutdown 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 msgr2=0x557fda3482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f9c94006010 tx=0x7f9c94003e10 comp rx=0 tx=0).stop 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 shutdown_connections 2024-03-19T02:39:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda35a9e0 0x557fda35cdd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda34a7d0 0x557fda3482b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda341bf0 0x557fda347d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9c85ffb640 0 client.18990 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2357681773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda3412d0 0x557fda347830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 >> 172.21.15.116:0/2357681773 conn(0x557fda33ba00 msgr2=0x557fda33d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 shutdown_connections 2024-03-19T02:39:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2357681773 wait complete. 2024-03-19T02:39:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.743+0000 7f9ca79ff740 20 client.18990 trim_cache size 0 max 16384 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 Processor -- start 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 -- start start 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 0x557fda347110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda34a7d0 0x557fda348f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01acac0 con 0x557fda341bf0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01ae600 con 0x557fda34a7d0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.758+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01ad190 con 0x557fda3412d0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 0x557fda347110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53240/0 (socket says 172.21.15.116:53240) 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9e59c640 1 -- 172.21.15.116:0/2598050514 learned_addr learned my addr 172.21.15.116:0/2598050514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda34a7d0 0x557fda348f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2598050514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda34a7d0 msgr2=0x557fda348f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda34a7d0 0x557fda348f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2598050514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 msgr2=0x557fda356360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2598050514 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fda341a20 con 0x557fda3412d0 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 0x557fda347110 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9c94010010 tx=0x7f9c94015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95a32bbdc0c3d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c87fff640 1 -- 172.21.15.116:0/2598050514 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94037040 con 0x557fda3412d0 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c87fff640 1 -- 172.21.15.116:0/2598050514 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c94013170 con 0x557fda3412d0 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.759+0000 7f9c87fff640 1 -- 172.21.15.116:0/2598050514 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94010850 con 0x557fda3412d0 2024-03-19T02:39:38.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2598050514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 msgr2=0x557fda347110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 0x557fda347110 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9c94010010 tx=0x7f9c94015040 comp rx=0 tx=0).stop 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2598050514 shutdown_connections 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda34a7d0 0x557fda348f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda341bf0 0x557fda356360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2598050514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda3412d0 0x557fda347110 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2598050514 >> 172.21.15.116:0/2598050514 conn(0x557fda33ba00 msgr2=0x557fda33aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2598050514 shutdown_connections 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2598050514 wait complete. 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 -- start start 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.760+0000 7f9ca79ff740 1 Processor -- start 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01adf30 con 0x557fda3412d0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01ad860 con 0x557fda341bf0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01aaac0 con 0x557fda34a7d0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 10 client.0 _ll_register_callbacks cb 0x7f9c90009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53244/0 (socket says 172.21.15.116:53244) 2024-03-19T02:39:38.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2574255308 learned_addr learned my addr 172.21.15.116:0/2574255308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 msgr2=0x557fda34e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2574255308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 msgr2=0x557fda34e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 -- 172.21.15.116:0/2574255308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fda34ee30 con 0x557fda3412d0 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c9dd9b640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f9c940039a0 tx=0x7f9c94010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964c844099cda1db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:38.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94045020 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f9c940105d0 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.761+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c94013170 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9c94037040 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 10 client.18996 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557fda3c0c50 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c94013170 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9c9400f780 con 0x557fda3412d0 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 10 client.18996 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 10 client.18996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 20 client.18996 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda3c1250 0x557fda3c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 4 client.18996 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557fda2b19f0 con 0x557fda3c1250 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.762+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda3c1250 0x557fda3c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9c85ffb640 10 client.18996 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9c9e59c640 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda3c1250 0x557fda3c3640 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=41a9c29c59a95f0b server_cookie=b52645dd72d35188 in_seq=0 out_seq=0 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9c85ffb640 1 -- 172.21.15.116:0/2574255308 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557fda2b19f0 con 0x557fda3c1250 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9c85ffb640 10 client.18996 handle_command_reply: tid=0 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 client.18996 shutdown 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 msgr2=0x557fda352c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f9c940039a0 tx=0x7f9c94010430 comp rx=0 tx=0).stop 2024-03-19T02:39:38.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 shutdown_connections 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557fda3c1250 0x557fda3c3640 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557fda34a7d0 0x557fda34e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557fda341bf0 0x557fda34e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9c85ffb640 0 client.18996 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 --2- 172.21.15.116:0/2574255308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557fda3412d0 0x557fda352c90 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 >> 172.21.15.116:0/2574255308 conn(0x557fda33ba00 msgr2=0x557fda33aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 shutdown_connections 2024-03-19T02:39:38.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 1 -- 172.21.15.116:0/2574255308 wait complete. 2024-03-19T02:39:38.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.763+0000 7f9ca79ff740 20 client.18996 trim_cache size 0 max 16384 2024-03-19T02:39:38.781 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:38.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 msgr2=0x7f9c880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 0x7f9c880a4e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9c980976e0 tx=0x7f9c98019e70 comp rx=0 tx=0).stop 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 msgr2=0x7f9ca019c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009fd0 tx=0x7f9c9800b260 comp rx=0 tx=0).stop 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 shutdown_connections 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c880a2970 0x7f9c880a4e30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ca01077a0 0x7f9ca019c8f0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ca0102150 0x7f9ca0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 --2- 172.21.15.116:0/1278506384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ca0101790 0x7f9ca0198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 >> 172.21.15.116:0/1278506384 conn(0x7f9ca00fd2c0 msgr2=0x7f9ca00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 shutdown_connections 2024-03-19T02:39:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.764+0000 7f9ca4d3b640 1 -- 172.21.15.116:0/1278506384 wait complete. 2024-03-19T02:39:38.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:39:38.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 384514 3.10 MB/sec execute 541 sec latency 15.188 ms 2024-03-19T02:39:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- 172.21.15.116:0/2931083258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b0103380 msgr2=0x7f05b010b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 --2- 172.21.15.116:0/2931083258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b0103380 0x7f05b010b860 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a402bd60 comp rx=0 tx=0).stop 2024-03-19T02:39:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- 172.21.15.116:0/2931083258 shutdown_connections 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 --2- 172.21.15.116:0/2931083258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b010bda0 0x7f05b010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 --2- 172.21.15.116:0/2931083258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b0103380 0x7f05b010b860 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 --2- 172.21.15.116:0/2931083258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0102a60 0x7f05b0102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- 172.21.15.116:0/2931083258 >> 172.21.15.116:0/2931083258 conn(0x7f05b0076aa0 msgr2=0x7f05b0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- 172.21.15.116:0/2931083258 shutdown_connections 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- 172.21.15.116:0/2931083258 wait complete. 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 Processor -- start 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.898+0000 7f05b6759640 1 -- start start 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 0x7f05b0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0103380 0x7f05b01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 0x7f05b019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05b0111a50 con 0x7f05b010bda0 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05b01118d0 con 0x7f05b0103380 2024-03-19T02:39:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b6759640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05b0111bd0 con 0x7f05b0102a60 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 0x7f05b019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 0x7f05b0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0103380 0x7f05b01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 0x7f05b0197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37500/0 (socket says 172.21.15.116:37500) 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05affff640 1 -- 172.21.15.116:0/4132157926 learned_addr learned my addr 172.21.15.116:0/4132157926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 -- 172.21.15.116:0/4132157926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 msgr2=0x7f05b0197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 0x7f05b0197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 -- 172.21.15.116:0/4132157926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0103380 msgr2=0x7f05b01984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0103380 0x7f05b01984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 -- 172.21.15.116:0/4132157926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b019cf80 con 0x7f05b010bda0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.899+0000 7f05b4cf6640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 0x7f05b019c850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009fd0 tx=0x7f05a800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.900+0000 7f05ad7fa640 1 -- 172.21.15.116:0/4132157926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a800ce20 con 0x7f05b010bda0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.900+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b019d270 con 0x7f05b010bda0 2024-03-19T02:39:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.900+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b01a4b00 con 0x7f05b010bda0 2024-03-19T02:39:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.900+0000 7f05ad7fa640 1 -- 172.21.15.116:0/4132157926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a8010430 con 0x7f05b010bda0 2024-03-19T02:39:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.900+0000 7f05ad7fa640 1 -- 172.21.15.116:0/4132157926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a800f620 con 0x7f05b010bda0 2024-03-19T02:39:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.902+0000 7f05ad7fa640 1 -- 172.21.15.116:0/4132157926 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f05a800f7c0 con 0x7f05b010bda0 2024-03-19T02:39:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 Processor -- start 2024-03-19T02:39:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 -- start start 2024-03-19T02:39:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce1bf0 0x564586cea290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564586c444b0 con 0x564586ce1bf0 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564586becd70 con 0x564586cea7d0 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b9413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564586c40550 con 0x564586ce12d0 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05ad7fa640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 0x7f05980a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce1bf0 0x564586cea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37514/0 (socket says 172.21.15.116:37514) 2024-03-19T02:39:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05af7fe640 1 -- 172.21.15.116:0/1424163215 learned_addr learned my addr 172.21.15.116:0/1424163215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05b4cf6640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 0x7f05980a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.903+0000 7f05ad7fa640 1 -- 172.21.15.116:0/4132157926 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f05a80cf3e0 con 0x7f05b010bda0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 -- 172.21.15.116:0/1424163215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 msgr2=0x564586cecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 -- 172.21.15.116:0/1424163215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce1bf0 msgr2=0x564586cea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce1bf0 0x564586cea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 -- 172.21.15.116:0/1424163215 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564586c0f210 con 0x564586ce12d0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a40056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e942a3708a4c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05b4cf6640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 0x7f05980a4eb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f05a809b300 tx=0x7f05a8066390 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05967fc640 1 -- 172.21.15.116:0/1424163215 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a404a070 con 0x564586ce12d0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05967fc640 1 -- 172.21.15.116:0/1424163215 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4003aa0 con 0x564586ce12d0 2024-03-19T02:39:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.904+0000 7f05967fc640 1 -- 172.21.15.116:0/1424163215 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a40438b0 con 0x564586ce12d0 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- 172.21.15.116:0/1424163215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 msgr2=0x564586ce16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a40056d0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- 172.21.15.116:0/1424163215 shutdown_connections 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce1bf0 0x564586cea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 --2- 172.21.15.116:0/1424163215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce16b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- 172.21.15.116:0/1424163215 >> 172.21.15.116:0/1424163215 conn(0x564586cdba00 msgr2=0x564586cdbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- 172.21.15.116:0/1424163215 shutdown_connections 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- 172.21.15.116:0/1424163215 wait complete. 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 -- start start 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.905+0000 7f05b9413740 1 Processor -- start 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586cea7d0 0x564586ce8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564586ced560 con 0x564586ce12d0 2024-03-19T02:39:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564586c00310 con 0x564586ce1bf0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564586ced6e0 con 0x564586cea7d0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 10 client.0 _ll_register_callbacks cb 0x564586bf42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53286/0 (socket says 172.21.15.116:53286) 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 -- 172.21.15.116:0/799585807 learned_addr learned my addr 172.21.15.116:0/799585807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05affff640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586cea7d0 0x564586ce8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 -- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586cea7d0 msgr2=0x564586ce8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586cea7d0 0x564586ce8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 -- 172.21.15.116:0/799585807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 msgr2=0x564586ce7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 -- 172.21.15.116:0/799585807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645869583f0 con 0x564586ce12d0 2024-03-19T02:39:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05af7fe640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036040 tx=0x7f05a40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521b63293faa4151 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f058f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a404a040 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f05a402cb10 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.906+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a4003aa0 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05a4043630 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 10 client.19008 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564586a227d0 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4003aa0 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f05a4050070 con 0x564586ce12d0 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 10 client.19008 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 10 client.19008 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 20 client.19008 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586cfa9f0 0x564586cfcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 4 client.19008 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564586ca23e0 con 0x564586cfa9f0 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.907+0000 7f05affff640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586cfa9f0 0x564586cfcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f05affff640 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586cfa9f0 0x564586cfcde0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bb2d65b4d62dd282 server_cookie=77f7b35d6e6474d4 in_seq=0 out_seq=0 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f058f7fe640 10 client.19008 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f058f7fe640 1 -- 172.21.15.116:0/799585807 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564586ca23e0 con 0x564586cfa9f0 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f058f7fe640 10 client.19008 handle_command_reply: tid=0 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f05b9413740 1 client.19008 shutdown 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 msgr2=0x564586ce77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036040 tx=0x7f05a40042e0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.908+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 shutdown_connections 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586cea7d0 0x564586ce8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586cfa9f0 0x564586cfcde0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 --2- 172.21.15.116:0/799585807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586ce77f0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f058f7fe640 0 client.19008 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 >> 172.21.15.116:0/799585807 conn(0x564586cdba00 msgr2=0x564586cdd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 shutdown_connections 2024-03-19T02:39:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 1 -- 172.21.15.116:0/799585807 wait complete. 2024-03-19T02:39:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.909+0000 7f05b9413740 20 client.19008 trim_cache size 0 max 16384 2024-03-19T02:39:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 Processor -- start 2024-03-19T02:39:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 -- start start 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f059c00ff60 con 0x564586cea7d0 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f059c011aa0 con 0x564586ce1bf0 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.923+0000 7f05b9413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f059c010630 con 0x564586ce12d0 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53302/0 (socket says 172.21.15.116:53302) 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 -- 172.21.15.116:0/1213769210 learned_addr learned my addr 172.21.15.116:0/1213769210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05affff640 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 -- 172.21.15.116:0/1213769210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 msgr2=0x564586ce9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 -- 172.21.15.116:0/1213769210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 msgr2=0x564586ce1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 -- 172.21.15.116:0/1213769210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564586ce67a0 con 0x564586cea7d0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f05a4053b00 tx=0x7f05a4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdeb05a42a11e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f0595ffb640 1 -- 172.21.15.116:0/1213769210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a404a030 con 0x564586cea7d0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f0595ffb640 1 -- 172.21.15.116:0/1213769210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4003a10 con 0x564586cea7d0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.924+0000 7f0595ffb640 1 -- 172.21.15.116:0/1213769210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a402c910 con 0x564586cea7d0 2024-03-19T02:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- 172.21.15.116:0/1213769210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 msgr2=0x564586ce8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f05a4053b00 tx=0x7f05a4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- 172.21.15.116:0/1213769210 shutdown_connections 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586cea7d0 0x564586ce8090 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586ce1bf0 0x564586ce1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 --2- 172.21.15.116:0/1213769210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce12d0 0x564586ce9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- 172.21.15.116:0/1213769210 >> 172.21.15.116:0/1213769210 conn(0x564586cdba00 msgr2=0x564586cdaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- 172.21.15.116:0/1213769210 shutdown_connections 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- 172.21.15.116:0/1213769210 wait complete. 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.925+0000 7f05b9413740 1 -- start start 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 Processor -- start 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586cfd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cfe280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f059c0113d0 con 0x564586ce12d0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f059c010d00 con 0x564586cea7d0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f059c00df60 con 0x564586ce1bf0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586cfd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05b9413740 10 client.0 _ll_register_callbacks cb 0x7f05a4048070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37532/0 (socket says 172.21.15.116:37532) 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 -- 172.21.15.116:0/1033477167 learned_addr learned my addr 172.21.15.116:0/1033477167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cfe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 -- 172.21.15.116:0/1033477167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 msgr2=0x564586cfe280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cfe280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 -- 172.21.15.116:0/1033477167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 msgr2=0x564586cfd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586cfd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 -- 172.21.15.116:0/1033477167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564586ce1a20 con 0x564586ce1bf0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586cfd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f05affff640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f05a0005d00 tx=0x7f05a0020490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c64fb4099c2e4327 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.926+0000 7f058f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0003a00 con 0x564586ce1bf0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a4003a10 con 0x564586ce1bf0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f05a0003ba0 con 0x564586ce1bf0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05a0030050 con 0x564586ce1bf0 2024-03-19T02:39:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 10 client.28750 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564586d60c00 con 0x564586ce1bf0 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4003a10 con 0x564586ce1bf0 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f05a0029410 con 0x564586ce1bf0 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 10 client.28750 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 10 client.28750 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 20 client.28750 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586d61360 0x564586d63780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 4 client.28750 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564586c519f0 con 0x564586d61360 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.927+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586d61360 0x564586d63780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05af7fe640 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586d61360 0x564586d63780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8191be82089fe807 server_cookie=71c17907e2ddd13e in_seq=0 out_seq=0 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f058f7fe640 10 client.28750 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f058f7fe640 1 -- 172.21.15.116:0/1033477167 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564586c519f0 con 0x564586d61360 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f058f7fe640 10 client.28750 handle_command_reply: tid=0 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 client.28750 shutdown 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 msgr2=0x564586cfdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f05a0005d00 tx=0x7f05a0020490 comp rx=0 tx=0).stop 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 shutdown_connections 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564586cea7d0 0x564586cfe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564586ce1bf0 0x564586cfdd40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564586d61360 0x564586d63780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 --2- 172.21.15.116:0/1033477167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564586ce12d0 0x564586cfd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f058f7fe640 0 client.28750 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 >> 172.21.15.116:0/1033477167 conn(0x564586cdba00 msgr2=0x564586cf59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 shutdown_connections 2024-03-19T02:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 1 -- 172.21.15.116:0/1033477167 wait complete. 2024-03-19T02:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.928+0000 7f05b9413740 20 client.28750 trim_cache size 0 max 16384 2024-03-19T02:39:38.938 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 msgr2=0x7f05980a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 0x7f05980a4eb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f05a809b300 tx=0x7f05a8066390 comp rx=0 tx=0).stop 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 msgr2=0x7f05b019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 0x7f05b019c850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009fd0 tx=0x7f05a800ef10 comp rx=0 tx=0).stop 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 shutdown_connections 2024-03-19T02:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f05980a29f0 0x7f05980a4eb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05b010bda0 0x7f05b019c850 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b0103380 0x7f05b01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 --2- 172.21.15.116:0/4132157926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f05b0102a60 0x7f05b0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 >> 172.21.15.116:0/4132157926 conn(0x7f05b0076aa0 msgr2=0x7f05b00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 shutdown_connections 2024-03-19T02:39:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:38.929+0000 7f05b6759640 1 -- 172.21.15.116:0/4132157926 wait complete. 2024-03-19T02:39:38.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:39:39.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.093+0000 7f0f58d22640 1 -- 172.21.15.116:0/2733363654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54102a40 msgr2=0x7f0f54102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.093+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2733363654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54102a40 0x7f0f54102e20 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009900 tx=0x7f0f4402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 -- 172.21.15.116:0/2733363654 shutdown_connections 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2733363654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 0x7f0f5410e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2733363654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54103360 0x7f0f5410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2733363654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54102a40 0x7f0f54102e20 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 -- 172.21.15.116:0/2733363654 >> 172.21.15.116:0/2733363654 conn(0x7f0f540778d0 msgr2=0x7f0f54077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 -- 172.21.15.116:0/2733363654 shutdown_connections 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 -- 172.21.15.116:0/2733363654 wait complete. 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 Processor -- start 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.094+0000 7f0f58d22640 1 -- start start 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54102a40 0x7f0f54100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 0x7f0f54101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f54111a30 con 0x7f0f54103360 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541118b0 con 0x7f0f54102a40 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f54111bb0 con 0x7f0f5410bd80 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54102a40 0x7f0f54100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53332/0 (socket says 172.21.15.116:53332) 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 -- 172.21.15.116:0/2459308541 learned_addr learned my addr 172.21.15.116:0/2459308541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 0x7f0f54101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 -- 172.21.15.116:0/2459308541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 msgr2=0x7f0f54101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 0x7f0f54101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 -- 172.21.15.116:0/2459308541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54102a40 msgr2=0x7f0f54100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54102a40 0x7f0f54100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 -- 172.21.15.116:0/2459308541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f541a6470 con 0x7f0f54103360 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800e9f0 tx=0x7f0f4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f437fe640 1 -- 172.21.15.116:0/2459308541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800cd80 con 0x7f0f54103360 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f541a6760 con 0x7f0f54103360 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f437fe640 1 -- 172.21.15.116:0/2459308541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f480108e0 con 0x7f0f54103360 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f437fe640 1 -- 172.21.15.116:0/2459308541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48018920 con 0x7f0f54103360 2024-03-19T02:39:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.095+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f541a6cb0 con 0x7f0f54103360 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.098+0000 7f0f437fe640 1 -- 172.21.15.116:0/2459308541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0f48010a90 con 0x7f0f54103360 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.098+0000 7f0f5b9ed740 1 Processor -- start 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 -- start start 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd98816290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98818bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd987704b0 con 0x55cd9880d2d0 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd98718d70 con 0x55cd988167d0 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cd9876c550 con 0x55cd9880dbf0 2024-03-19T02:39:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd98816290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53340/0 (socket says 172.21.15.116:53340) 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 -- 172.21.15.116:0/4161353978 learned_addr learned my addr 172.21.15.116:0/4161353978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f437fe640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 0x7f0f3c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98818bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 -- 172.21.15.116:0/4161353978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 msgr2=0x55cd98816290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd98816290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 -- 172.21.15.116:0/4161353978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 msgr2=0x55cd98818bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98818bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f52d9d640 1 -- 172.21.15.116:0/4161353978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd9873b210 con 0x55cd9880d2d0 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f5259c640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd98816290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.099+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98818bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 0x7f0f3c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f437fe640 1 -- 172.21.15.116:0/2459308541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f0f480cf4c0 con 0x7f0f54103360 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011590 tx=0x7f0f4c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d814b7490fdc346 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f40ff9640 1 -- 172.21.15.116:0/4161353978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c017050 con 0x55cd9880d2d0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f40ff9640 1 -- 172.21.15.116:0/4161353978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f4c01bb90 con 0x55cd9880d2d0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.100+0000 7f0f40ff9640 1 -- 172.21.15.116:0/4161353978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c025d80 con 0x55cd9880d2d0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/4161353978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 msgr2=0x55cd9880d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c011590 tx=0x7f0f4c011a50 comp rx=0 tx=0).stop 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 0x7f0f3c0a4e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c022000 tx=0x7f0f4c023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/4161353978 shutdown_connections 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98818bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd98816290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/4161353978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd9880d6b0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/4161353978 >> 172.21.15.116:0/4161353978 conn(0x55cd98807a00 msgr2=0x55cd98807de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/4161353978 shutdown_connections 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/4161353978 wait complete. 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- start start 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 Processor -- start 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 0x55cd98813d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 0x55cd988142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd988195c0 con 0x55cd9880d2d0 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd9872c310 con 0x55cd9880dbf0 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.101+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cd98819740 con 0x55cd988167d0 2024-03-19T02:39:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5b9ed740 10 client.0 _ll_register_callbacks cb 0x55cd987202d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53344/0 (socket says 172.21.15.116:53344) 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 -- 172.21.15.116:0/2629782828 learned_addr learned my addr 172.21.15.116:0/2629782828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 0x55cd98813d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 0x55cd988142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 -- 172.21.15.116:0/2629782828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 msgr2=0x55cd988142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 0x55cd988142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 -- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 msgr2=0x55cd98813d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 0x55cd98813d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 -- 172.21.15.116:0/2629782828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd984843f0 con 0x55cd9880d2d0 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 0x55cd98813d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 0x55cd988142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5259c640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009900 tx=0x7f0f44005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79114e965a397f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44043070 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f0f4402c880 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f4c01bb90 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.102+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0f4404f050 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 10 client.19026 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cd9854e7d0 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f4c01bb90 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0f44004c20 con 0x55cd9880d2d0 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 10 client.19026 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 10 client.19026 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 20 client.19026 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd98826b20 0x55cd98828f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 4 client.19026 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd987ce3e0 con 0x55cd98826b20 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.103+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd98826b20 0x55cd98828f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.104+0000 7f0f39ffb640 10 client.19026 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.104+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd98826b20 0x55cd98828f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d5c0dbf77a64dce server_cookie=c7f755e8de782c4f in_seq=0 out_seq=0 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.104+0000 7f0f39ffb640 1 -- 172.21.15.116:0/2629782828 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd987ce3e0 con 0x55cd98826b20 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.104+0000 7f0f39ffb640 10 client.19026 handle_command_reply: tid=0 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 client.19026 shutdown 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 msgr2=0x55cd98813830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009900 tx=0x7f0f44005250 comp rx=0 tx=0).stop 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 shutdown_connections 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd98826b20 0x55cd98828f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd988167d0 0x55cd988142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd9880dbf0 0x55cd98813d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f39ffb640 0 client.19026 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/2629782828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98813830 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 >> 172.21.15.116:0/2629782828 conn(0x55cd98807a00 msgr2=0x55cd98809750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 shutdown_connections 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/2629782828 wait complete. 2024-03-19T02:39:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.105+0000 7f0f5b9ed740 20 client.19026 trim_cache size 0 max 16384 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 Processor -- start 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 -- start start 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd988140d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd9882e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541b0e90 con 0x55cd9880d2d0 2024-03-19T02:39:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541b29d0 con 0x55cd988167d0 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541b1560 con 0x55cd9880dbf0 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f52d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd9882e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f51d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd988140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53354/0 (socket says 172.21.15.116:53354) 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 -- 172.21.15.116:0/1165428650 learned_addr learned my addr 172.21.15.116:0/1165428650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 -- 172.21.15.116:0/1165428650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 msgr2=0x55cd988140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd988140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 -- 172.21.15.116:0/1165428650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 msgr2=0x55cd9882e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd9882e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f5259c640 1 -- 172.21.15.116:0/1165428650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd988085d0 con 0x55cd9880d2d0 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.120+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd9882e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5259c640 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0f4402bd10 tx=0x7f0f44009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fca00358ea1abe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f3bfff640 1 -- 172.21.15.116:0/1165428650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44058070 con 0x55cd9880d2d0 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f3bfff640 1 -- 172.21.15.116:0/1165428650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f440055d0 con 0x55cd9880d2d0 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f3bfff640 1 -- 172.21.15.116:0/1165428650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44043030 con 0x55cd9880d2d0 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1165428650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 msgr2=0x55cd98815f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0f4402bd10 tx=0x7f0f44009620 comp rx=0 tx=0).stop 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1165428650 shutdown_connections 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd9882e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880dbf0 0x55cd988140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.121+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1165428650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880d2d0 0x55cd98815f60 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1165428650 >> 172.21.15.116:0/1165428650 conn(0x55cd98807a00 msgr2=0x55cd98808ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1165428650 shutdown_connections 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1165428650 wait complete. 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- start start 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 Processor -- start 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880d2d0 0x55cd988106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98811170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541b2300 con 0x55cd9880dbf0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541b1c30 con 0x55cd988167d0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541aee90 con 0x55cd9880d2d0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5b9ed740 10 client.0 _ll_register_callbacks cb 0x7f0f44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53366/0 (socket says 172.21.15.116:53366) 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 -- 172.21.15.116:0/1719159450 learned_addr learned my addr 172.21.15.116:0/1719159450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f5259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880d2d0 0x55cd988106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f52d9d640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98811170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 -- 172.21.15.116:0/1719159450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880d2d0 msgr2=0x55cd988106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880d2d0 0x55cd988106b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.122+0000 7f0f51d9b640 1 -- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 msgr2=0x55cd98811170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98811170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f51d9b640 1 -- 172.21.15.116:0/1719159450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd9888c160 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f51d9b640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f0f48098260 tx=0x7f0f480cf830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d772b457f9f23f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4809f930 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f0f48097f00 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f440055d0 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0f480cf160 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 10 client.19032 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cd9888d460 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f440055d0 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f0f480a3af0 con 0x55cd9880dbf0 2024-03-19T02:39:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 10 client.19032 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 10 client.19032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 20 client.19032 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd9888db60 0x55cd9888ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 4 client.19032 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.123+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd9877d9f0 con 0x55cd9888db60 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5259c640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd9888db60 0x55cd9888ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5259c640 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd9888db60 0x55cd9888ff50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8755d48f8a973658 server_cookie=42425a669b71dfb1 in_seq=0 out_seq=0 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f39ffb640 10 client.19032 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f39ffb640 1 -- 172.21.15.116:0/1719159450 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd9877d9f0 con 0x55cd9888db60 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f39ffb640 10 client.19032 handle_command_reply: tid=0 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 client.19032 shutdown 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 msgr2=0x55cd98810bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f0f48098260 tx=0x7f0f480cf830 comp rx=0 tx=0).stop 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 shutdown_connections 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cd9888db60 0x55cd9888ff50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd988167d0 0x55cd98811170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd9880dbf0 0x55cd98810bf0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f39ffb640 0 client.19032 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 --2- 172.21.15.116:0/1719159450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cd9880d2d0 0x55cd988106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 >> 172.21.15.116:0/1719159450 conn(0x55cd98807a00 msgr2=0x55cd98808ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 shutdown_connections 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 1 -- 172.21.15.116:0/1719159450 wait complete. 2024-03-19T02:39:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.124+0000 7f0f5b9ed740 20 client.19032 trim_cache size 0 max 16384 2024-03-19T02:39:39.135 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 msgr2=0x7f0f3c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 0x7f0f3c0a4e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c022000 tx=0x7f0f4c023000 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 msgr2=0x7f0f54100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800e9f0 tx=0x7f0f4800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 shutdown_connections 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0f3c0a2970 0x7f0f3c0a4e30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f5410bd80 0x7f0f54101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f54103360 0x7f0f54100bd0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 --2- 172.21.15.116:0/2459308541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f54102a40 0x7f0f54100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 >> 172.21.15.116:0/2459308541 conn(0x7f0f540778d0 msgr2=0x7f0f540fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 shutdown_connections 2024-03-19T02:39:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.125+0000 7f0f58d22640 1 -- 172.21.15.116:0/2459308541 wait complete. 2024-03-19T02:39:39.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 -- 172.21.15.116:0/130172218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81066b0 msgr2=0x7f78a8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 --2- 172.21.15.116:0/130172218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81066b0 0x7f78a8106a90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009890 tx=0x7f78a402bb70 comp rx=0 tx=0).stop 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 -- 172.21.15.116:0/130172218 shutdown_connections 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 --2- 172.21.15.116:0/130172218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 0x7f78a810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 --2- 172.21.15.116:0/130172218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81006a0 0x7f78a8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 --2- 172.21.15.116:0/130172218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81066b0 0x7f78a8106a90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 -- 172.21.15.116:0/130172218 >> 172.21.15.116:0/130172218 conn(0x7f78a80778d0 msgr2=0x7f78a8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 -- 172.21.15.116:0/130172218 shutdown_connections 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.253+0000 7f78b0589640 1 -- 172.21.15.116:0/130172218 wait complete. 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 Processor -- start 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 -- start start 2024-03-19T02:39:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81006a0 0x7f78a8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 0x7f78a81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78a8110930 con 0x7f78a81066b0 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81107b0 con 0x7f78a8101060 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78b0589640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a8110ab0 con 0x7f78a81006a0 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53390/0 (socket says 172.21.15.116:53390) 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78aeb26640 1 -- 172.21.15.116:0/4224372606 learned_addr learned my addr 172.21.15.116:0/4224372606 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78adb24640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 0x7f78a81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.254+0000 7f78ae325640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81006a0 0x7f78a8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 -- 172.21.15.116:0/4224372606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81006a0 msgr2=0x7f78a8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81006a0 0x7f78a8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 -- 172.21.15.116:0/4224372606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 msgr2=0x7f78a81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 0x7f78a81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 -- 172.21.15.116:0/4224372606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a819cf40 con 0x7f78a81066b0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78aeb26640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f78a000eb10 tx=0x7f78a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f789f7fe640 1 -- 172.21.15.116:0/4224372606 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0004b20 con 0x7f78a81066b0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a819d230 con 0x7f78a81066b0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f789f7fe640 1 -- 172.21.15.116:0/4224372606 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a0004cc0 con 0x7f78a81066b0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a4ac0 con 0x7f78a81066b0 2024-03-19T02:39:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.255+0000 7f789f7fe640 1 -- 172.21.15.116:0/4224372606 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0018630 con 0x7f78a81066b0 2024-03-19T02:39:39.259 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:38 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2178922531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:39.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.257+0000 7f789f7fe640 1 -- 172.21.15.116:0/4224372606 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f78a0005000 con 0x7f78a81066b0 2024-03-19T02:39:39.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 Processor -- start 2024-03-19T02:39:39.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 -- start start 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f789f7fe640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 0x7f78940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd942290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd944bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561fcd89c4b0 con 0x561fcd9427d0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561fcd844d70 con 0x561fcd9392d0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78b317b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561fcd898550 con 0x561fcd939bf0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78adb24640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 0x7f78940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd944bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78ae325640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78adb24640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd942290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78ae325640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55632/0 (socket says 172.21.15.116:55632) 2024-03-19T02:39:39.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f78ae325640 1 -- 172.21.15.116:0/315587222 learned_addr learned my addr 172.21.15.116:0/315587222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.258+0000 7f789f7fe640 1 -- 172.21.15.116:0/4224372606 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f78a00cf890 con 0x7f78a81066b0 2024-03-19T02:39:39.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 -- 172.21.15.116:0/315587222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 msgr2=0x561fcd942290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.263 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:38 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2178922531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:39.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd942290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 -- 172.21.15.116:0/315587222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 msgr2=0x561fcd944bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd944bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 -- 172.21.15.116:0/315587222 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fcd867210 con 0x561fcd9392d0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78adb24640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 0x7f78940a4e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ffc30 tx=0x7f7898009890 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f78ae325640 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009860 tx=0x7f78a4009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f15382a8fe040987 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f789cbfe640 1 -- 172.21.15.116:0/315587222 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4043070 con 0x561fcd9392d0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f789cbfe640 1 -- 172.21.15.116:0/315587222 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a4034980 con 0x561fcd9392d0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.259+0000 7f789cbfe640 1 -- 172.21.15.116:0/315587222 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4039040 con 0x561fcd9392d0 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- 172.21.15.116:0/315587222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 msgr2=0x561fcd9396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009860 tx=0x7f78a4009510 comp rx=0 tx=0).stop 2024-03-19T02:39:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- 172.21.15.116:0/315587222 shutdown_connections 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd944bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd942290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 --2- 172.21.15.116:0/315587222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd9396b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- 172.21.15.116:0/315587222 >> 172.21.15.116:0/315587222 conn(0x561fcd933a00 msgr2=0x561fcd933de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- 172.21.15.116:0/315587222 shutdown_connections 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- 172.21.15.116:0/315587222 wait complete. 2024-03-19T02:39:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 -- start start 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.260+0000 7f78b317b740 1 Processor -- start 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561fcd945560 con 0x561fcd9427d0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561fcd858310 con 0x561fcd9392d0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561fcd9456e0 con 0x561fcd939bf0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 10 client.0 _ll_register_callbacks cb 0x561fcd84c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78ae325640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53404/0 (socket says 172.21.15.116:53404) 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 -- 172.21.15.116:0/1679157984 learned_addr learned my addr 172.21.15.116:0/1679157984 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78adb24640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 -- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 msgr2=0x561fcd93fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 -- 172.21.15.116:0/1679157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 msgr2=0x561fcd93f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 -- 172.21.15.116:0/1679157984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fcd5b03f0 con 0x561fcd9427d0 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78aeb26640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f78a00984e0 tx=0x7f78a009b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dbc9c2b7c822d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f7891ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a009c210 con 0x561fcd9427d0 2024-03-19T02:39:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f78a0098180 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.261+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a4034980 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78a009bb10 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 10 client.19044 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561fcd67a7d0 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a4034980 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f78a00cf4f0 con 0x561fcd9427d0 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 10 client.19044 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 10 client.19044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 20 client.19044 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd952a40 0x561fcd954e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 4 client.19044 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fcd8fa3e0 con 0x561fcd952a40 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.262+0000 7f78ae325640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd952a40 0x561fcd954e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78ae325640 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd952a40 0x561fcd954e90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8174e24bbb985ca8 server_cookie=3a5064437e8c4520 in_seq=0 out_seq=0 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f7891ffb640 10 client.19044 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f7891ffb640 1 -- 172.21.15.116:0/1679157984 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fcd8fa3e0 con 0x561fcd952a40 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f7891ffb640 10 client.19044 handle_command_reply: tid=0 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78b317b740 1 client.19044 shutdown 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 msgr2=0x561fcd940230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f78a00984e0 tx=0x7f78a009b760 comp rx=0 tx=0).stop 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 shutdown_connections 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.263+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd952a40 0x561fcd954e90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd940230 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f7891ffb640 0 client.19044 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 --2- 172.21.15.116:0/1679157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 >> 172.21.15.116:0/1679157984 conn(0x561fcd933a00 msgr2=0x561fcd935750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 shutdown_connections 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 1 -- 172.21.15.116:0/1679157984 wait complete. 2024-03-19T02:39:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.264+0000 7f78b317b740 20 client.19044 trim_cache size 0 max 16384 2024-03-19T02:39:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 Processor -- start 2024-03-19T02:39:39.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 -- start start 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd939bf0 0x561fcd94e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 0x561fcd95a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81aebc0 con 0x561fcd9392d0 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81b0700 con 0x561fcd939bf0 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78b317b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81af290 con 0x561fcd9427d0 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53408/0 (socket says 172.21.15.116:53408) 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 -- 172.21.15.116:0/1537524710 learned_addr learned my addr 172.21.15.116:0/1537524710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78adb24640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd939bf0 0x561fcd94e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78aeb26640 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 0x561fcd95a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 -- 172.21.15.116:0/1537524710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 msgr2=0x561fcd95a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 0x561fcd95a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 -- 172.21.15.116:0/1537524710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd939bf0 msgr2=0x561fcd94e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd939bf0 0x561fcd94e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78ae325640 1 -- 172.21.15.116:0/1537524710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fcd909880 con 0x561fcd9392d0 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.277+0000 7f78aeb26640 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 0x561fcd95a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f78ae325640 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009890 tx=0x7f78a402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3a93f3cea76cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f7893fff640 1 -- 172.21.15.116:0/1537524710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4043040 con 0x561fcd9392d0 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f7893fff640 1 -- 172.21.15.116:0/1537524710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a4034980 con 0x561fcd9392d0 2024-03-19T02:39:39.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f7893fff640 1 -- 172.21.15.116:0/1537524710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4039040 con 0x561fcd9392d0 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f78b317b740 1 -- 172.21.15.116:0/1537524710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 msgr2=0x561fcd941ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.278+0000 7f78b317b740 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009890 tx=0x7f78a402e040 comp rx=0 tx=0).stop 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- 172.21.15.116:0/1537524710 shutdown_connections 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd9427d0 0x561fcd95a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd939bf0 0x561fcd94e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- 172.21.15.116:0/1537524710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9392d0 0x561fcd941ee0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- 172.21.15.116:0/1537524710 >> 172.21.15.116:0/1537524710 conn(0x561fcd933a00 msgr2=0x561fcd932ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- 172.21.15.116:0/1537524710 shutdown_connections 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- 172.21.15.116:0/1537524710 wait complete. 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- start start 2024-03-19T02:39:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 Processor -- start 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81b0030 con 0x561fcd9427d0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81af960 con 0x561fcd9392d0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78a81acbc0 con 0x561fcd939bf0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78b317b740 10 client.0 _ll_register_callbacks cb 0x7f78a009b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78adb24640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78aeb26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53420/0 (socket says 172.21.15.116:53420) 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78aeb26640 1 -- 172.21.15.116:0/748920100 learned_addr learned my addr 172.21.15.116:0/748920100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.279+0000 7f78ae325640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 -- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 msgr2=0x561fcd93ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 -- 172.21.15.116:0/748920100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 msgr2=0x561fcd93c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 -- 172.21.15.116:0/748920100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fcd944bc0 con 0x561fcd9427d0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78aeb26640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f78a00cf860 tx=0x7f78a0009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f0f016a086b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f7891ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a00b8020 con 0x561fcd9427d0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f78a00ba2c0 con 0x561fcd9427d0 2024-03-19T02:39:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a4034980 con 0x561fcd9427d0 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78a009c210 con 0x561fcd9427d0 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.280+0000 7f78b317b740 10 client.19050 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561fcd9b9240 con 0x561fcd9427d0 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a4034980 con 0x561fcd9427d0 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f78a00052f0 con 0x561fcd9427d0 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 10 client.19050 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 10 client.19050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 20 client.19050 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd9b9960 0x561fcd9bbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 4 client.19050 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fcd8a99f0 con 0x561fcd9b9960 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78ae325640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd9b9960 0x561fcd9bbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f7891ffb640 10 client.19050 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.281+0000 7f78ae325640 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd9b9960 0x561fcd9bbd80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=205b932b36c82feb server_cookie=96adbde97669c819 in_seq=0 out_seq=0 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f7891ffb640 1 -- 172.21.15.116:0/748920100 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fcd8a99f0 con 0x561fcd9b9960 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f7891ffb640 10 client.19050 handle_command_reply: tid=0 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 client.19050 shutdown 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 msgr2=0x561fcd9b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f78a00cf860 tx=0x7f78a0009a70 comp rx=0 tx=0).stop 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 shutdown_connections 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561fcd9b9960 0x561fcd9bbd80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fcd9427d0 0x561fcd9b7d90 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561fcd939bf0 0x561fcd93ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 --2- 172.21.15.116:0/748920100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561fcd9392d0 0x561fcd93c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 >> 172.21.15.116:0/748920100 conn(0x561fcd933a00 msgr2=0x561fcd93f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f7891ffb640 0 client.19050 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 shutdown_connections 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 1 -- 172.21.15.116:0/748920100 wait complete. 2024-03-19T02:39:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b317b740 20 client.19050 trim_cache size 0 max 16384 2024-03-19T02:39:39.293 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 msgr2=0x7f78940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 0x7f78940a4e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ffc30 tx=0x7f7898009890 comp rx=0 tx=0).stop 2024-03-19T02:39:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 msgr2=0x7f78a819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.282+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f78a000eb10 tx=0x7f78a000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 shutdown_connections 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78940a2970 0x7f78940a4e30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a81066b0 0x7f78a819c810 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101060 0x7f78a81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 --2- 172.21.15.116:0/4224372606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a81006a0 0x7f78a8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 >> 172.21.15.116:0/4224372606 conn(0x7f78a80778d0 msgr2=0x7f78a8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 shutdown_connections 2024-03-19T02:39:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.283+0000 7f78b0589640 1 -- 172.21.15.116:0/4224372606 wait complete. 2024-03-19T02:39:39.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:39:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:38 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2178922531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.412+0000 7f6e3b44b640 1 -- 172.21.15.116:0/967466666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e341066b0 msgr2=0x7f6e34106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.412+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/967466666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e341066b0 0x7f6e34106a90 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009930 tx=0x7f6e3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.412+0000 7f6e3b44b640 1 -- 172.21.15.116:0/967466666 shutdown_connections 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.412+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/967466666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e34101060 0x7f6e3410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/967466666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e34100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/967466666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e341066b0 0x7f6e34106a90 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 -- 172.21.15.116:0/967466666 >> 172.21.15.116:0/967466666 conn(0x7f6e340778d0 msgr2=0x7f6e34077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 -- 172.21.15.116:0/967466666 shutdown_connections 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 -- 172.21.15.116:0/967466666 wait complete. 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.413+0000 7f6e3b44b640 1 Processor -- start 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 -- start start 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e34101060 0x7f6e3419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 0x7f6e341a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410f860 con 0x7f6e34101060 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410f6e0 con 0x7f6e341006a0 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e3b44b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410f9e0 con 0x7f6e341066b0 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e34101060 0x7f6e3419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55678/0 (socket says 172.21.15.116:55678) 2024-03-19T02:39:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 0x7f6e341a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 -- 172.21.15.116:0/2405781427 learned_addr learned my addr 172.21.15.116:0/2405781427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 -- 172.21.15.116:0/2405781427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 msgr2=0x7f6e341a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 0x7f6e341a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 -- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e34101060 msgr2=0x7f6e3419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e34101060 0x7f6e3419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e391e7640 1 -- 172.21.15.116:0/2405781427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e341a3560 con 0x7f6e341006a0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.414+0000 7f6e399e8640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 0x7f6e341a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e391e7640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009230 tx=0x7f6e3002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e2a7fc640 1 -- 172.21.15.116:0/2405781427 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30043020 con 0x7f6e341006a0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e341a37f0 con 0x7f6e341006a0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e2a7fc640 1 -- 172.21.15.116:0/2405781427 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e3002ca70 con 0x7f6e341006a0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e2a7fc640 1 -- 172.21.15.116:0/2405781427 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30003a50 con 0x7f6e341006a0 2024-03-19T02:39:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.415+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e341ab0e0 con 0x7f6e341006a0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.417+0000 7f6e2a7fc640 1 -- 172.21.15.116:0/2405781427 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6e30046020 con 0x7f6e341006a0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 Processor -- start 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 -- start start 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e2a7fc640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 0x7f6e200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69b27d0 0x55fdf69b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fdf690c4b0 con 0x55fdf69a9bf0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fdf68b4d70 con 0x55fdf69b27d0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e3e036740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fdf6908550 con 0x55fdf69a92d0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e391e7640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 0x7f6e200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37670/0 (socket says 172.21.15.116:37670) 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 -- 172.21.15.116:0/3461437247 learned_addr learned my addr 172.21.15.116:0/3461437247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69b27d0 0x55fdf69b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e2a7fc640 1 -- 172.21.15.116:0/2405781427 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f6e300f1170 con 0x7f6e341006a0 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 -- 172.21.15.116:0/3461437247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69b27d0 msgr2=0x55fdf69b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69b27d0 0x55fdf69b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 -- 172.21.15.116:0/3461437247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 msgr2=0x55fdf69b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e399e8640 1 -- 172.21.15.116:0/3461437247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf68d7210 con 0x55fdf69a92d0 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.418+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.419+0000 7f6e391e7640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 0x7f6e200a4e30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e300bd1a0 tx=0x7f6e30009260 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.419+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00fe40 tx=0x7f6e2c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae4b1124cb90aae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.419+0000 7f6e1f7fe640 1 -- 172.21.15.116:0/3461437247 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00dbd0 con 0x55fdf69a92d0 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.419+0000 7f6e1f7fe640 1 -- 172.21.15.116:0/3461437247 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e2c003e20 con 0x55fdf69a92d0 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.419+0000 7f6e1f7fe640 1 -- 172.21.15.116:0/3461437247 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c01e420 con 0x55fdf69a92d0 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- 172.21.15.116:0/3461437247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 msgr2=0x55fdf69a96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00fe40 tx=0x7f6e2c009b60 comp rx=0 tx=0).stop 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- 172.21.15.116:0/3461437247 shutdown_connections 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69b27d0 0x55fdf69b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3461437247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69a92d0 0x55fdf69a96b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- 172.21.15.116:0/3461437247 >> 172.21.15.116:0/3461437247 conn(0x55fdf69a3a00 msgr2=0x55fdf69a3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- 172.21.15.116:0/3461437247 shutdown_connections 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- 172.21.15.116:0/3461437247 wait complete. 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- start start 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 Processor -- start 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fdf69b55c0 con 0x55fdf69a9bf0 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fdf68c8310 con 0x55fdf69a92d0 2024-03-19T02:39:39.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.420+0000 7f6e3e036740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fdf69b5740 con 0x55fdf69b27d0 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e3e036740 10 client.0 _ll_register_callbacks cb 0x55fdf68bc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:55700/0 (socket says 172.21.15.116:55700) 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 -- 172.21.15.116:0/451100311 learned_addr learned my addr 172.21.15.116:0/451100311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53450/0 (socket says 172.21.15.116:53450) 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 -- 172.21.15.116:0/451100311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 msgr2=0x55fdf69b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 -- 172.21.15.116:0/451100311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 msgr2=0x55fdf69afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 -- 172.21.15.116:0/451100311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf66203f0 con 0x55fdf69a92d0 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e389e6640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e399e8640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0045a0 tx=0x7f6e2c009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a046c6c28fc9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00dc60 con 0x55fdf69a92d0 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e2c003e20 con 0x55fdf69a92d0 2024-03-19T02:39:39.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f6e2c00de00 con 0x55fdf69a92d0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.421+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6e2c035050 con 0x55fdf69a92d0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 10 client.28847 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fdf66ea7d0 con 0x55fdf69a92d0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e2c003e20 con 0x55fdf69a92d0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6e2c01e520 con 0x55fdf69a92d0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 10 client.28847 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 10 client.28847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 20 client.28847 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf69c2ae0 0x55fdf69c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 4 client.28847 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdf696a3e0 con 0x55fdf69c2ae0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e389e6640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf69c2ae0 0x55fdf69c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e1cff9640 10 client.28847 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:39.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.422+0000 7f6e389e6640 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf69c2ae0 0x55fdf69c4ed0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a790bac17f90f5c server_cookie=899016a77f409b91 in_seq=0 out_seq=0 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e1cff9640 1 -- 172.21.15.116:0/451100311 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fdf696a3e0 con 0x55fdf69c2ae0 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e1cff9640 10 client.28847 handle_command_reply: tid=0 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 client.28847 shutdown 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 msgr2=0x55fdf69af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0045a0 tx=0x7f6e2c009820 comp rx=0 tx=0).stop 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 shutdown_connections 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf69c2ae0 0x55fdf69c4ed0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 --2- 172.21.15.116:0/451100311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69af880 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e1cff9640 0 client.28847 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:39.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 >> 172.21.15.116:0/451100311 conn(0x55fdf69a3a00 msgr2=0x55fdf69a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 shutdown_connections 2024-03-19T02:39:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 1 -- 172.21.15.116:0/451100311 wait complete. 2024-03-19T02:39:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.424+0000 7f6e3e036740 20 client.28847 trim_cache size 0 max 16384 2024-03-19T02:39:39.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 Processor -- start 2024-03-19T02:39:39.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 -- start start 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b5280 con 0x55fdf69a9bf0 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b6dc0 con 0x55fdf69a92d0 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e3e036740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b5950 con 0x55fdf69b27d0 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e389e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53456/0 (socket says 172.21.15.116:53456) 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e389e6640 1 -- 172.21.15.116:0/3224135017 learned_addr learned my addr 172.21.15.116:0/3224135017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.440+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 -- 172.21.15.116:0/3224135017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 msgr2=0x55fdf69b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 -- 172.21.15.116:0/3224135017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 msgr2=0x55fdf69b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 -- 172.21.15.116:0/3224135017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf69b5410 con 0x55fdf69a9bf0 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400b440 tx=0x7f6e2400d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a77c1c18ec4b54b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e1effd640 1 -- 172.21.15.116:0/3224135017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e240215f0 con 0x55fdf69a9bf0 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e1effd640 1 -- 172.21.15.116:0/3224135017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e24027070 con 0x55fdf69a9bf0 2024-03-19T02:39:39.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.441+0000 7f6e1effd640 1 -- 172.21.15.116:0/3224135017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24018a40 con 0x55fdf69a9bf0 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- 172.21.15.116:0/3224135017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 msgr2=0x55fdf69a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400b440 tx=0x7f6e2400d040 comp rx=0 tx=0).stop 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- 172.21.15.116:0/3224135017 shutdown_connections 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a9bf0 0x55fdf69a5540 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3224135017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a92d0 0x55fdf69b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- 172.21.15.116:0/3224135017 >> 172.21.15.116:0/3224135017 conn(0x55fdf69a3a00 msgr2=0x55fdf69a4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- 172.21.15.116:0/3224135017 shutdown_connections 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- 172.21.15.116:0/3224135017 wait complete. 2024-03-19T02:39:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 -- start start 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.442+0000 7f6e3e036740 1 Processor -- start 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b66f0 con 0x55fdf69a92d0 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53466/0 (socket says 172.21.15.116:53466) 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 -- 172.21.15.116:0/3789279372 learned_addr learned my addr 172.21.15.116:0/3789279372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:39.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b6020 con 0x55fdf69a9bf0 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341b3280 con 0x55fdf69b27d0 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e3e036740 10 client.0 _ll_register_callbacks cb 0x7f6e2400b6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 -- 172.21.15.116:0/3789279372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 msgr2=0x55fdf69b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 -- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 msgr2=0x55fdf69b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e399e8640 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf69b6ba0 con 0x55fdf69a92d0 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.443+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e399e8640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c01ec10 tx=0x7f6e2c009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2973bf273646e417 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c03d070 con 0x55fdf69a92d0 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f6e2c00dba0 con 0x55fdf69a92d0 2024-03-19T02:39:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e24027070 con 0x55fdf69a92d0 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6e2c046050 con 0x55fdf69a92d0 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e3e036740 10 client.19068 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fdf6a28bd0 con 0x55fdf69a92d0 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e24027070 con 0x55fdf69a92d0 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.444+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6e2c040020 con 0x55fdf69a92d0 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 10 client.19068 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 10 client.19068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 20 client.19068 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf6a29260 0x55fdf6a2b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 4 client.19068 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fdf69199f0 con 0x55fdf6a29260 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf6a29260 0x55fdf6a2b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e1cff9640 10 client.19068 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.445+0000 7f6e389e6640 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf6a29260 0x55fdf6a2b680 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d22dc0502ac32c server_cookie=a68c91b21332eb7b in_seq=0 out_seq=0 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e1cff9640 1 -- 172.21.15.116:0/3789279372 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fdf69199f0 con 0x55fdf6a29260 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e1cff9640 10 client.19068 handle_command_reply: tid=0 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 client.19068 shutdown 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 msgr2=0x55fdf69bab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c01ec10 tx=0x7f6e2c009480 comp rx=0 tx=0).stop 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 shutdown_connections 2024-03-19T02:39:39.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fdf6a29260 0x55fdf6a2b680 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf69b27d0 0x55fdf69b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e1cff9640 0 client.19068 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fdf69a9bf0 0x55fdf69b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 --2- 172.21.15.116:0/3789279372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdf69a92d0 0x55fdf69bab10 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 >> 172.21.15.116:0/3789279372 conn(0x55fdf69a3a00 msgr2=0x55fdf69a5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 shutdown_connections 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 1 -- 172.21.15.116:0/3789279372 wait complete. 2024-03-19T02:39:39.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.446+0000 7f6e3e036740 20 client.19068 trim_cache size 0 max 16384 2024-03-19T02:39:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:39:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 msgr2=0x7f6e200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 0x7f6e200a4e30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6e300bd1a0 tx=0x7f6e30009260 comp rx=0 tx=0).stop 2024-03-19T02:39:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 msgr2=0x7f6e3419e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009230 tx=0x7f6e3002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 shutdown_connections 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e200a2970 0x7f6e200a4e30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e341066b0 0x7f6e341a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e34101060 0x7f6e3419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 --2- 172.21.15.116:0/2405781427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e341006a0 0x7f6e3419e570 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 >> 172.21.15.116:0/2405781427 conn(0x7f6e340778d0 msgr2=0x7f6e34103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 shutdown_connections 2024-03-19T02:39:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:39.447+0000 7f6e3b44b640 1 -- 172.21.15.116:0/2405781427 wait complete. 2024-03-19T02:39:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 384979 3.09 MB/sec execute 542 sec latency 25.701 ms 2024-03-19T02:39:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:39 smithi154 ceph-mon[35693]: pgmap v370: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:39 smithi116 ceph-mon[32263]: pgmap v370: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:40.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:39 smithi162 ceph-mon[35761]: pgmap v370: 129 pgs: 129 active+clean; 440 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-03-19T02:39:40.459 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:39:40.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:39:40.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:39:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 -- 172.21.15.116:0/2577081276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 msgr2=0x7f939c10d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/2577081276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c10d560 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f9394009cd0 tx=0x7f939402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 -- 172.21.15.116:0/2577081276 shutdown_connections 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/2577081276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c10d560 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/2577081276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 0x7f939c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/2577081276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.578+0000 7f93a3e6c640 1 -- 172.21.15.116:0/2577081276 >> 172.21.15.116:0/2577081276 conn(0x7f939c077820 msgr2=0x7f939c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- 172.21.15.116:0/2577081276 shutdown_connections 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- 172.21.15.116:0/2577081276 wait complete. 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 Processor -- start 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- start start 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 0x7f939c196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c19b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f939c110960 con 0x7f939c101050 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1107e0 con 0x7f939c1066a0 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a3e6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f939c110ae0 con 0x7f939c100690 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a1407640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a1407640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53484/0 (socket says 172.21.15.116:53484) 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a1407640 1 -- 172.21.15.116:0/1209266957 learned_addr learned my addr 172.21.15.116:0/1209266957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.579+0000 7f93a1c08640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 0x7f939c196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a2409640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c19b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 -- 172.21.15.116:0/1209266957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 msgr2=0x7f939c196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 0x7f939c196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 -- 172.21.15.116:0/1209266957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 msgr2=0x7f939c19b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c19b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 -- 172.21.15.116:0/1209266957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c19be90 con 0x7f939c101050 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a2409640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a1407640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f938c00eb00 tx=0x7f938c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f9392ffd640 1 -- 172.21.15.116:0/1209266957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c00bc30 con 0x7f939c101050 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f9392ffd640 1 -- 172.21.15.116:0/1209266957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f938c00bdd0 con 0x7f939c101050 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f9392ffd640 1 -- 172.21.15.116:0/1209266957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c017410 con 0x7f939c101050 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c19c180 con 0x7f939c101050 2024-03-19T02:39:40.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.580+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c068980 con 0x7f939c101050 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.583+0000 7f9392ffd640 1 -- 172.21.15.116:0/1209266957 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f938c004f90 con 0x7f939c101050 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.583+0000 7f93a6a58740 1 Processor -- start 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 -- start start 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dce9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dcf4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c2dc4d160 con 0x560c2dce9ca0 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c2dbf5e10 con 0x560c2dcf2880 2024-03-19T02:39:40.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a6a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c2dc49200 con 0x560c2dce9380 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f9392ffd640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 0x7f93880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a2409640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dce9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a2409640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dce9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37742/0 (socket says 172.21.15.116:37742) 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53488/0 (socket says 172.21.15.116:53488) 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.584+0000 7f93a2409640 1 -- 172.21.15.116:0/674565221 learned_addr learned my addr 172.21.15.116:0/674565221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a2409640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 0x7f93880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f9392ffd640 1 -- 172.21.15.116:0/1209266957 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f938c0d0a30 con 0x7f939c101050 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 -- 172.21.15.116:0/674565221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 msgr2=0x560c2dce9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dce9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 -- 172.21.15.116:0/674565221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 msgr2=0x560c2dcf4c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dcf4c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 -- 172.21.15.116:0/674565221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2da297e0 con 0x560c2dce9ca0 2024-03-19T02:39:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f93a1c08640 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9398005fd0 tx=0x7f93980107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f43f352cf2830b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.585+0000 7f9387fff640 1 -- 172.21.15.116:0/674565221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939801c480 con 0x560c2dce9ca0 2024-03-19T02:39:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a2409640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 0x7f93880a4eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9394009a60 tx=0x7f939402c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f9387fff640 1 -- 172.21.15.116:0/674565221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f939801cad0 con 0x560c2dce9ca0 2024-03-19T02:39:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f9387fff640 1 -- 172.21.15.116:0/674565221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398011a80 con 0x560c2dce9ca0 2024-03-19T02:39:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 -- 172.21.15.116:0/674565221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 msgr2=0x560c2dcf2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f9398005fd0 tx=0x7f93980107a0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 -- 172.21.15.116:0/674565221 shutdown_connections 2024-03-19T02:39:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dcf4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dcf2340 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 --2- 172.21.15.116:0/674565221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dce9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 -- 172.21.15.116:0/674565221 >> 172.21.15.116:0/674565221 conn(0x560c2dce3ab0 msgr2=0x560c2dce3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.586+0000 7f93a6a58740 1 -- 172.21.15.116:0/674565221 shutdown_connections 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.587+0000 7f93a6a58740 1 -- 172.21.15.116:0/674565221 wait complete. 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.587+0000 7f93a6a58740 1 -- start start 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.587+0000 7f93a6a58740 1 Processor -- start 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcefe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dcf0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c2dcf5670 con 0x560c2dce9380 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c2dc091a0 con 0x560c2dce9ca0 2024-03-19T02:39:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c2dcf57f0 con 0x560c2dcf2880 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a6a58740 10 client.0 _ll_register_callbacks cb 0x560c2d960c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53490/0 (socket says 172.21.15.116:53490) 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.588+0000 7f93a1c08640 1 -- 172.21.15.116:0/1802803482 learned_addr learned my addr 172.21.15.116:0/1802803482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 -- 172.21.15.116:0/1802803482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 msgr2=0x560c2dcf0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dcf0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 -- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 msgr2=0x560c2dcefe10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1407640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcefe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcefe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 -- 172.21.15.116:0/1802803482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2da89f70 con 0x560c2dce9380 2024-03-19T02:39:40.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93a1c08640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f93980116e0 tx=0x7f939800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e475d5b2574b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.589+0000 7f93857fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939802a060 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f939801c680 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f939801cad0 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9398031050 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93a6a58740 10 client.19080 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c2d921170 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.590+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f939801cad0 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9398011ac0 con 0x560c2dce9380 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 10 client.19080 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 10 client.19080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 20 client.19080 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd02b60 0x560c2dd04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 4 client.19080 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c2dc5a6a0 con 0x560c2dd02b60 2024-03-19T02:39:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.591+0000 7f93a1407640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd02b60 0x560c2dd04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.592+0000 7f93a1407640 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd02b60 0x560c2dd04f50 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=807874d8c69edf67 server_cookie=c6003668d7ea8f4b in_seq=0 out_seq=0 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.592+0000 7f93857fa640 10 client.19080 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.594+0000 7f93857fa640 1 -- 172.21.15.116:0/1802803482 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c2dc5a6a0 con 0x560c2dd02b60 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.594+0000 7f93857fa640 10 client.19080 handle_command_reply: tid=0 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.594+0000 7f93a6a58740 1 client.19080 shutdown 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 msgr2=0x560c2dcef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f93980116e0 tx=0x7f939800bf50 comp rx=0 tx=0).stop 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 shutdown_connections 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd02b60 0x560c2dd04f50 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dcf0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcefe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 --2- 172.21.15.116:0/1802803482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcef8d0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.595+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 >> 172.21.15.116:0/1802803482 conn(0x560c2dce3ab0 msgr2=0x560c2dce5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.596+0000 7f93857fa640 0 client.19080 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.596+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 shutdown_connections 2024-03-19T02:39:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.597+0000 7f93a6a58740 1 -- 172.21.15.116:0/1802803482 wait complete. 2024-03-19T02:39:40.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.597+0000 7f93a6a58740 20 client.19080 trim_cache size 0 max 16384 2024-03-19T02:39:40.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 Processor -- start 2024-03-19T02:39:40.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 -- start start 2024-03-19T02:39:40.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcfe150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dd07b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1acb40 con 0x560c2dce9380 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1ae680 con 0x560c2dce9ca0 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a6a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1ad210 con 0x560c2dcf2880 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53500/0 (socket says 172.21.15.116:53500) 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 -- 172.21.15.116:0/2250710178 learned_addr learned my addr 172.21.15.116:0/2250710178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1407640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcfe150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a2409640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dd07b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 -- 172.21.15.116:0/2250710178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 msgr2=0x560c2dd07b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dd07b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 -- 172.21.15.116:0/2250710178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 msgr2=0x560c2dcfe150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcfe150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a1c08640 1 -- 172.21.15.116:0/2250710178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2dce9ad0 con 0x560c2dce9380 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.619+0000 7f93a2409640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dd07b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93a1c08640 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f939800b6c0 tx=0x7f939800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f173f4d521d18e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93877fe640 1 -- 172.21.15.116:0/2250710178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398039020 con 0x560c2dce9380 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93877fe640 1 -- 172.21.15.116:0/2250710178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9398004e10 con 0x560c2dce9380 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93877fe640 1 -- 172.21.15.116:0/2250710178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939802a040 con 0x560c2dce9380 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93a6a58740 1 -- 172.21.15.116:0/2250710178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 msgr2=0x560c2dcf2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.620+0000 7f93a6a58740 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f939800b6c0 tx=0x7f939800ba20 comp rx=0 tx=0).stop 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- 172.21.15.116:0/2250710178 shutdown_connections 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dcf2880 0x560c2dd07b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dce9ca0 0x560c2dcfe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- 172.21.15.116:0/2250710178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9380 0x560c2dcf2000 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- 172.21.15.116:0/2250710178 >> 172.21.15.116:0/2250710178 conn(0x560c2dce3ab0 msgr2=0x560c2dcfda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- 172.21.15.116:0/2250710178 shutdown_connections 2024-03-19T02:39:40.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- 172.21.15.116:0/2250710178 wait complete. 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- start start 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 Processor -- start 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dd07e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dd05790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1adfb0 con 0x560c2dce9ca0 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1ad8e0 con 0x560c2dcf2880 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1aab40 con 0x560c2dce9380 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.621+0000 7f93a6a58740 10 client.0 _ll_register_callbacks cb 0x7f938c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:40.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53512/0 (socket says 172.21.15.116:53512) 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 -- 172.21.15.116:0/3332697530 learned_addr learned my addr 172.21.15.116:0/3332697530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1c08640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dd07e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a2409640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dd05790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 -- 172.21.15.116:0/3332697530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 msgr2=0x560c2dd07e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dd07e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 -- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 msgr2=0x560c2dd05790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dd05790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 -- 172.21.15.116:0/3332697530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2dd05f90 con 0x560c2dce9ca0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a1407640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f938c09ccf0 tx=0x7f938c09c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a514cd335d072043 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93857fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c049b70 con 0x560c2dce9ca0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f938c0ce020 con 0x560c2dce9ca0 2024-03-19T02:39:40.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9398004e10 con 0x560c2dce9ca0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f938c09cd60 con 0x560c2dce9ca0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a6a58740 10 client.19086 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560c2dd68bd0 con 0x560c2dce9ca0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.622+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9398004e10 con 0x560c2dce9ca0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f938c005280 con 0x560c2dce9ca0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 10 client.19086 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 10 client.19086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 20 client.19086 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd69370 0x560c2dd6b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 4 client.19086 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560c2dc549f0 con 0x560c2dd69370 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a1c08640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd69370 0x560c2dd6b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93857fa640 10 client.19086 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.623+0000 7f93a1c08640 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd69370 0x560c2dd6b790 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f916a75f1835d019 server_cookie=88a27d4e25197a3e in_seq=0 out_seq=0 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.624+0000 7f93857fa640 1 -- 172.21.15.116:0/3332697530 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560c2dc549f0 con 0x560c2dd69370 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.624+0000 7f93857fa640 10 client.19086 handle_command_reply: tid=0 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.624+0000 7f93a6a58740 1 client.19086 shutdown 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.624+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 msgr2=0x560c2dd05250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.624+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f938c09ccf0 tx=0x7f938c09c9c0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 shutdown_connections 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560c2dd69370 0x560c2dd6b790 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c2dcf2880 0x560c2dd05790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93857fa640 0 client.19086 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c2dce9ca0 0x560c2dd05250 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 --2- 172.21.15.116:0/3332697530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c2dce9380 0x560c2dd07e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 >> 172.21.15.116:0/3332697530 conn(0x560c2dce3ab0 msgr2=0x560c2dce56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 shutdown_connections 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 1 -- 172.21.15.116:0/3332697530 wait complete. 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a6a58740 20 client.19086 trim_cache size 0 max 16384 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "aee7ef6c-dce1-424c-bca0-548567e5edab", 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:39:40.631 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 msgr2=0x7f93880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 0x7f93880a4eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9394009a60 tx=0x7f939402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 msgr2=0x7f939c1973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f938c00eb00 tx=0x7f938c00ef00 comp rx=0 tx=0).stop 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 shutdown_connections 2024-03-19T02:39:40.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f93880a29f0 0x7f93880a4eb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c1066a0 0x7f939c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f939c101050 0x7f939c1973e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 --2- 172.21.15.116:0/1209266957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f939c100690 0x7f939c196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 >> 172.21.15.116:0/1209266957 conn(0x7f939c077820 msgr2=0x7f939c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.625+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 shutdown_connections 2024-03-19T02:39:40.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.626+0000 7f93a3e6c640 1 -- 172.21.15.116:0/1209266957 wait complete. 2024-03-19T02:39:40.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:39:40.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:39:40.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 -- 172.21.15.116:0/3466644451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83901066a0 msgr2=0x7f8390106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 --2- 172.21.15.116:0/3466644451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83901066a0 0x7f8390106a80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f838c009890 tx=0x7f838c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:39:40.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 -- 172.21.15.116:0/3466644451 shutdown_connections 2024-03-19T02:39:40.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 --2- 172.21.15.116:0/3466644451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f839010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 --2- 172.21.15.116:0/3466644451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8390100690 0x7f8390100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 --2- 172.21.15.116:0/3466644451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83901066a0 0x7f8390106a80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 -- 172.21.15.116:0/3466644451 >> 172.21.15.116:0/3466644451 conn(0x7f8390077820 msgr2=0x7f8390077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 -- 172.21.15.116:0/3466644451 shutdown_connections 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.747+0000 7f83985ab640 1 -- 172.21.15.116:0/3466644451 wait complete. 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 Processor -- start 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 -- start start 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83901066a0 0x7f83901a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8390110920 con 0x7f8390101050 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83901107a0 con 0x7f83901066a0 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f83985ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8390110aa0 con 0x7f8390100690 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8395b46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37786/0 (socket says 172.21.15.116:37786) 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 -- 172.21.15.116:0/523543466 learned_addr learned my addr 172.21.15.116:0/523543466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8395b46640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53542/0 (socket says 172.21.15.116:53542) 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396b48640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83901066a0 0x7f83901a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 -- 172.21.15.116:0/523543466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83901066a0 msgr2=0x7f83901a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83901066a0 0x7f83901a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 -- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 msgr2=0x7f83901993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8396347640 1 -- 172.21.15.116:0/523543466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83901a4420 con 0x7f8390100690 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.748+0000 7f8395b46640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f8396347640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f838c000c00 tx=0x7f838c035990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f83877fe640 1 -- 172.21.15.116:0/523543466 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c041020 con 0x7f8390100690 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83901a45f0 con 0x7f8390100690 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f83877fe640 1 -- 172.21.15.116:0/523543466 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f838c02c9c0 con 0x7f8390100690 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f83877fe640 1 -- 172.21.15.116:0/523543466 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c039040 con 0x7f8390100690 2024-03-19T02:39:40.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.749+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83901a4a60 con 0x7f8390100690 2024-03-19T02:39:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.751+0000 7f83877fe640 1 -- 172.21.15.116:0/523543466 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f838c044020 con 0x7f8390100690 2024-03-19T02:39:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 Processor -- start 2024-03-19T02:39:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 -- start start 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 0x55d452db4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 0x55d452dc0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d452d18500 con 0x55d452db5350 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d452cc0be0 con 0x55d452db4a30 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f839b18a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d452d145a0 con 0x55d452dbdf30 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f83877fe640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 0x7f837c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53556/0 (socket says 172.21.15.116:53556) 2024-03-19T02:39:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8396347640 1 -- 172.21.15.116:0/761378058 learned_addr learned my addr 172.21.15.116:0/761378058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8396b48640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 0x55d452db4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8395b46640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 0x55d452dc0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.752+0000 7f8396b48640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 0x7f837c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f83877fe640 1 -- 172.21.15.116:0/523543466 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f838c0f08c0 con 0x7f8390100690 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 -- 172.21.15.116:0/761378058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 msgr2=0x55d452dc0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 0x55d452dc0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 -- 172.21.15.116:0/761378058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 msgr2=0x55d452dbd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 -- 172.21.15.116:0/761378058 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d452ce2000 con 0x55d452db4a30 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8395b46640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 0x55d452dc0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396347640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 0x7f837c0a4e30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f838800cc40 tx=0x7f8388011000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8396b48640 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 0x55d452db4e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8388005fd0 tx=0x7f838800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=721810314b3fc016 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8384bfe640 1 -- 172.21.15.116:0/761378058 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838801ce50 con 0x55d452db4a30 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8384bfe640 1 -- 172.21.15.116:0/761378058 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f838800e870 con 0x55d452db4a30 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.753+0000 7f8384bfe640 1 -- 172.21.15.116:0/761378058 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83880042b0 con 0x55d452db4a30 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- 172.21.15.116:0/761378058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 msgr2=0x55d452db4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 0x55d452db4e10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8388005fd0 tx=0x7f838800c740 comp rx=0 tx=0).stop 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- 172.21.15.116:0/761378058 shutdown_connections 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452dbdf30 0x55d452dc0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db5350 0x55d452dbd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 --2- 172.21.15.116:0/761378058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452db4a30 0x55d452db4e10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- 172.21.15.116:0/761378058 >> 172.21.15.116:0/761378058 conn(0x55d452daf160 msgr2=0x55d452daf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- 172.21.15.116:0/761378058 shutdown_connections 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- 172.21.15.116:0/761378058 wait complete. 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 -- start start 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.754+0000 7f839b18a740 1 Processor -- start 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dbb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dbb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d452dc0c60 con 0x55d452db4a30 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d452cd4160 con 0x55d452dbdf30 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d452dc0de0 con 0x55d452db5350 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53560/0 (socket says 172.21.15.116:53560) 2024-03-19T02:39:40.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 -- 172.21.15.116:0/630687521 learned_addr learned my addr 172.21.15.116:0/630687521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f839b18a740 10 client.0 _ll_register_callbacks cb 0x55d452d10a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396b48640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dbb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8395b46640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dbb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 -- 172.21.15.116:0/630687521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 msgr2=0x55d452dbb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dbb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 -- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 msgr2=0x55d452dbb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dbb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 -- 172.21.15.116:0/630687521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d452af67d0 con 0x55d452db4a30 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8395b46640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dbb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8396347640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f838c0097b0 tx=0x7f838c0b88c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e0bff06932faf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8379ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0c53f0 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.755+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f838c003680 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838800e870 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f838c0bd350 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 10 client.19098 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d452a2c3f0 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f838800e870 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f838c03c030 con 0x55d452db4a30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 10 client.19098 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 10 client.19098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 20 client.19098 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452dd06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 4 client.19098 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:39:40.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d452d25a40 con 0x55d452dce250 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.756+0000 7f8395b46640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452dd06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.757+0000 7f8395b46640 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452dd06d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11d6132fb3b32e2d server_cookie=8a6f6ae007e7de4b in_seq=0 out_seq=0 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.757+0000 7f8379ffb640 10 client.19098 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f8379ffb640 1 -- 172.21.15.116:0/630687521 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d452d25a40 con 0x55d452dce250 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f8379ffb640 10 client.19098 handle_command_reply: tid=0 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 client.19098 shutdown 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 msgr2=0x55d452dbaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f838c0097b0 tx=0x7f838c0b88c0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 shutdown_connections 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452dd06d0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dbb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f8379ffb640 0 client.19098 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dbb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 --2- 172.21.15.116:0/630687521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbaf70 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 >> 172.21.15.116:0/630687521 conn(0x55d452daf160 msgr2=0x55d452db0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 shutdown_connections 2024-03-19T02:39:40.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 1 -- 172.21.15.116:0/630687521 wait complete. 2024-03-19T02:39:40.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.758+0000 7f839b18a740 20 client.19098 trim_cache size 0 max 16384 2024-03-19T02:39:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.777+0000 7f839b18a740 1 Processor -- start 2024-03-19T02:39:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.777+0000 7f839b18a740 1 -- start start 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.777+0000 7f839b18a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.777+0000 7f839b18a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.777+0000 7f839b18a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452db62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f839b18a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83901aebe0 con 0x55d452db4a30 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f839b18a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83901b0720 con 0x55d452dbdf30 2024-03-19T02:39:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f839b18a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83901af2b0 con 0x55d452db5350 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8395b46640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53562/0 (socket says 172.21.15.116:53562) 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8395b46640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37812/0 (socket says 172.21.15.116:37812) 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 -- 172.21.15.116:0/1619173608 learned_addr learned my addr 172.21.15.116:0/1619173608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396b48640 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452db62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 -- 172.21.15.116:0/1619173608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 msgr2=0x55d452db9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 -- 172.21.15.116:0/1619173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 msgr2=0x55d452db62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452db62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 -- 172.21.15.116:0/1619173608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4529ec0f0 con 0x55d452db4a30 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8395b46640 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.778+0000 7f8396347640 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f838c0baf00 tx=0x7f838c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d412c04865d4187b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f837bfff640 1 -- 172.21.15.116:0/1619173608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0c53f0 con 0x55d452db4a30 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f837bfff640 1 -- 172.21.15.116:0/1619173608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f838c0c0da0 con 0x55d452db4a30 2024-03-19T02:39:40.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f837bfff640 1 -- 172.21.15.116:0/1619173608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c003680 con 0x55d452db4a30 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 -- 172.21.15.116:0/1619173608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 msgr2=0x55d452dbd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f838c0baf00 tx=0x7f838c0035f0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 -- 172.21.15.116:0/1619173608 shutdown_connections 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452db62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452db9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 --2- 172.21.15.116:0/1619173608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dbd630 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 -- 172.21.15.116:0/1619173608 >> 172.21.15.116:0/1619173608 conn(0x55d452daf160 msgr2=0x55d452dae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.779+0000 7f839b18a740 1 -- 172.21.15.116:0/1619173608 shutdown_connections 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 -- 172.21.15.116:0/1619173608 wait complete. 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 -- start start 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 Processor -- start 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dc15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dc1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83901b0050 con 0x55d452db4a30 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83901af980 con 0x55d452dbdf30 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83901acbe0 con 0x55d452db5350 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f839b18a740 10 client.0 _ll_register_callbacks cb 0x7f838c0bc820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f8396347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53574/0 (socket says 172.21.15.116:53574) 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f8396347640 1 -- 172.21.15.116:0/3644671130 learned_addr learned my addr 172.21.15.116:0/3644671130 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:39:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f8396b48640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dc1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.780+0000 7f8395b46640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dc15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 -- 172.21.15.116:0/3644671130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 msgr2=0x55d452dc15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dc15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 -- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 msgr2=0x55d452dc1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dc1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 -- 172.21.15.116:0/3644671130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d452dc2330 con 0x55d452db4a30 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8396347640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f838c0c5360 tx=0x7f838c0bcb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a8b0f32352e5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:39:40.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c0c53f0 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9510+0+0 (secure 0 0 0) 0x7f838c0db300 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838c0c0da0 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f838c0db500 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 10 client.19104 fetch_fsmap learned FSMap version 30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d452dc2330 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f838c0c0da0 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f838c0bbba0 con 0x55d452db4a30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 10 client.19104 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 10 client.19104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.781+0000 7f839b18a740 20 client.19104 populate_metadata read hostname 'smithi116' 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452e3f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 4 client.19104 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:39:40.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d452d1fd90 con 0x55d452dce250 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8395b46640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452e3f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8395b46640 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452e3f220 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b3a8d6fd54e968d server_cookie=1f495dee16469430 in_seq=0 out_seq=0 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8379ffb640 10 client.19104 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8379ffb640 1 -- 172.21.15.116:0/3644671130 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d452d1fd90 con 0x55d452dce250 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8379ffb640 10 client.19104 handle_command_reply: tid=0 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 client.19104 shutdown 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 msgr2=0x55d452dc6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f838c0c5360 tx=0x7f838c0bcb50 comp rx=0 tx=0).stop 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 shutdown_connections 2024-03-19T02:39:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d452dce250 0x55d452e3f220 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d452dbdf30 0x55d452dc1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d452db5350 0x55d452dc15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f8379ffb640 0 client.19104 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 --2- 172.21.15.116:0/3644671130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d452db4a30 0x55d452dc6230 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.782+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 >> 172.21.15.116:0/3644671130 conn(0x55d452daf160 msgr2=0x55d452db0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 shutdown_connections 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f839b18a740 1 -- 172.21.15.116:0/3644671130 wait complete. 2024-03-19T02:39:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f839b18a740 20 client.19104 trim_cache size 0 max 16384 2024-03-19T02:39:40.793 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:39:40.793 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:39:40.793 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:39:40.793 INFO:teuthology.orchestra.run.smithi116.stdout: "aee7ef6c-dce1-424c-bca0-548567e5edab": { 2024-03-19T02:39:40.793 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:39:40.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "aee7ef6c-dce1-424c-bca0-548567e5edab", 2024-03-19T02:39:40.794 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:39:40.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:39:40.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:39:40.794 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:39:40.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 msgr2=0x7f837c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 0x7f837c0a4e30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f838800cc40 tx=0x7f8388011000 comp rx=0 tx=0).stop 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 msgr2=0x7f8390198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f838c000c00 tx=0x7f838c035990 comp rx=0 tx=0).stop 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 shutdown_connections 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f837c0a2970 0x7f837c0a4e30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83901066a0 0x7f83901a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101050 0x7f83901993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 --2- 172.21.15.116:0/523543466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8390100690 0x7f8390198e60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:39:40.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 >> 172.21.15.116:0/523543466 conn(0x7f8390077820 msgr2=0x7f83901030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:39:40.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 shutdown_connections 2024-03-19T02:39:40.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:39:40.783+0000 7f83985ab640 1 -- 172.21.15.116:0/523543466 wait complete. 2024-03-19T02:39:40.798 INFO:tasks.cephfs.filesystem:scrub status for tag:aee7ef6c-dce1-424c-bca0-548567e5edab - {'path': '/', 'tag': 'aee7ef6c-dce1-424c-bca0-548567e5edab', 'options': 'recursive,force'} 2024-03-19T02:39:40.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 385583 3.09 MB/sec execute 543 sec latency 28.981 ms 2024-03-19T02:39:41.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 386433 3.10 MB/sec execute 544 sec latency 16.976 ms 2024-03-19T02:39:42.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:39:42.260 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:39:42.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:39:42.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.261 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:41 smithi154 ceph-mon[35693]: pgmap v371: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.262 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:41 smithi116 ceph-mon[32263]: pgmap v371: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-03-19T02:39:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:39:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:39:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:39:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:39:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:41 smithi162 ceph-mon[35761]: pgmap v371: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-03-19T02:39:42.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 386855 3.10 MB/sec execute 545 sec latency 18.911 ms 2024-03-19T02:39:43.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 387261 3.10 MB/sec execute 546 sec latency 15.581 ms 2024-03-19T02:39:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:43 smithi154 ceph-mon[35693]: pgmap v372: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:39:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:43 smithi116 ceph-mon[32263]: pgmap v372: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:39:44.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:43 smithi162 ceph-mon[35761]: pgmap v372: 129 pgs: 129 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-03-19T02:39:44.796 INFO:tasks.workunit.client.0.smithi116.stdout: 1 387797 3.09 MB/sec execute 547 sec latency 36.362 ms 2024-03-19T02:39:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:44 smithi154 ceph-mon[35693]: pgmap v373: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:44 smithi116 ceph-mon[32263]: pgmap v373: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:44 smithi162 ceph-mon[35761]: pgmap v373: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-03-19T02:39:45.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 388344 3.10 MB/sec execute 548 sec latency 25.864 ms 2024-03-19T02:39:46.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 388998 3.09 MB/sec execute 549 sec latency 23.499 ms 2024-03-19T02:39:47.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 389685 3.10 MB/sec execute 550 sec latency 16.428 ms 2024-03-19T02:39:48.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:47 smithi154 ceph-mon[35693]: pgmap v374: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:47 smithi116 ceph-mon[32263]: pgmap v374: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:47 smithi162 ceph-mon[35761]: pgmap v374: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:48.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 390200 3.10 MB/sec execute 551 sec latency 24.998 ms 2024-03-19T02:39:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 390624 3.10 MB/sec execute 552 sec latency 21.652 ms 2024-03-19T02:39:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:49 smithi154 ceph-mon[35693]: pgmap v375: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:49 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:39:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:49 smithi116 ceph-mon[32263]: pgmap v375: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:49 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:39:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:49 smithi162 ceph-mon[35761]: pgmap v375: 129 pgs: 129 active+clean; 452 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-03-19T02:39:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:49 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:39:50.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 391108 3.09 MB/sec execute 553 sec latency 42.520 ms 2024-03-19T02:39:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:39:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:39:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:39:51.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 391712 3.10 MB/sec execute 554 sec latency 30.351 ms 2024-03-19T02:39:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:51 smithi154 ceph-mon[35693]: pgmap v376: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 119 op/s 2024-03-19T02:39:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:51 smithi116 ceph-mon[32263]: pgmap v376: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 119 op/s 2024-03-19T02:39:52.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:51 smithi162 ceph-mon[35761]: pgmap v376: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 119 op/s 2024-03-19T02:39:52.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 392202 3.09 MB/sec execute 555 sec latency 49.905 ms 2024-03-19T02:39:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:52 smithi154 ceph-mon[35693]: pgmap v377: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:52 smithi116 ceph-mon[32263]: pgmap v377: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:52 smithi162 ceph-mon[35761]: pgmap v377: 129 pgs: 129 active+clean; 456 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 1.9 MiB/s wr, 78 op/s 2024-03-19T02:39:53.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 392800 3.09 MB/sec execute 556 sec latency 19.647 ms 2024-03-19T02:39:54.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 393534 3.10 MB/sec execute 557 sec latency 71.594 ms 2024-03-19T02:39:55.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 393998 3.10 MB/sec execute 558 sec latency 20.373 ms 2024-03-19T02:39:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:55 smithi154 ceph-mon[35693]: pgmap v378: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-03-19T02:39:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:55 smithi116 ceph-mon[32263]: pgmap v378: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-03-19T02:39:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:55 smithi162 ceph-mon[35761]: pgmap v378: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 140 op/s 2024-03-19T02:39:56.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 394374 3.09 MB/sec execute 559 sec latency 34.041 ms 2024-03-19T02:39:57.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 394988 3.09 MB/sec execute 560 sec latency 17.180 ms 2024-03-19T02:39:58.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:57 smithi154 ceph-mon[35693]: pgmap v379: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:39:58.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:57 smithi116 ceph-mon[32263]: pgmap v379: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:39:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:57 smithi162 ceph-mon[35761]: pgmap v379: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:39:58.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 395541 3.09 MB/sec execute 561 sec latency 38.354 ms 2024-03-19T02:39:59.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 396136 3.09 MB/sec execute 562 sec latency 18.936 ms 2024-03-19T02:40:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:39:59 smithi154 ceph-mon[35693]: pgmap v380: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:40:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:39:59 smithi116 ceph-mon[32263]: pgmap v380: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:40:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:39:59 smithi162 ceph-mon[35761]: pgmap v380: 129 pgs: 129 active+clean; 459 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 100 op/s 2024-03-19T02:40:00.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 397032 3.10 MB/sec execute 563 sec latency 30.667 ms 2024-03-19T02:40:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:00 smithi154 ceph-mon[35693]: overall HEALTH_OK 2024-03-19T02:40:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:00 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:40:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:00 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:40:01.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 397418 3.10 MB/sec execute 564 sec latency 19.210 ms 2024-03-19T02:40:02.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:01 smithi154 ceph-mon[35693]: pgmap v381: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.4 MiB/s wr, 143 op/s 2024-03-19T02:40:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:01 smithi116 ceph-mon[32263]: pgmap v381: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.4 MiB/s wr, 143 op/s 2024-03-19T02:40:02.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:01 smithi162 ceph-mon[35761]: pgmap v381: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.4 MiB/s wr, 143 op/s 2024-03-19T02:40:02.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 397816 3.09 MB/sec execute 565 sec latency 15.745 ms 2024-03-19T02:40:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:02 smithi154 ceph-mon[35693]: pgmap v382: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-03-19T02:40:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:02 smithi116 ceph-mon[32263]: pgmap v382: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-03-19T02:40:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:02 smithi162 ceph-mon[35761]: pgmap v382: 129 pgs: 129 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-03-19T02:40:03.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 398365 3.09 MB/sec execute 566 sec latency 20.431 ms 2024-03-19T02:40:04.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 398918 3.09 MB/sec execute 567 sec latency 36.962 ms 2024-03-19T02:40:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 399488 3.09 MB/sec execute 568 sec latency 24.261 ms 2024-03-19T02:40:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:05 smithi162 ceph-mon[35761]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T02:40:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:05 smithi162 ceph-mon[35761]: pgmap v384: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-03-19T02:40:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:05 smithi154 ceph-mon[35693]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T02:40:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:05 smithi154 ceph-mon[35693]: pgmap v384: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-03-19T02:40:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:05 smithi116 ceph-mon[32263]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T02:40:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:05 smithi116 ceph-mon[32263]: pgmap v384: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 2.0 MiB/s wr, 122 op/s 2024-03-19T02:40:06.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 400192 3.09 MB/sec execute 569 sec latency 19.118 ms 2024-03-19T02:40:07.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 400667 3.09 MB/sec execute 570 sec latency 19.453 ms 2024-03-19T02:40:08.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:07 smithi162 ceph-mon[35761]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T02:40:08.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:07 smithi162 ceph-mon[35761]: pgmap v386: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-03-19T02:40:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:07 smithi154 ceph-mon[35693]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T02:40:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:07 smithi154 ceph-mon[35693]: pgmap v386: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-03-19T02:40:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:07 smithi116 ceph-mon[32263]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T02:40:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:07 smithi116 ceph-mon[32263]: pgmap v386: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-03-19T02:40:08.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 401196 3.10 MB/sec execute 571 sec latency 20.436 ms 2024-03-19T02:40:09.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 401614 3.09 MB/sec execute 572 sec latency 49.268 ms 2024-03-19T02:40:10.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:09 smithi154 ceph-mon[35693]: pgmap v387: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.4 MiB/s wr, 88 op/s 2024-03-19T02:40:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:09 smithi116 ceph-mon[32263]: pgmap v387: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.4 MiB/s wr, 88 op/s 2024-03-19T02:40:10.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:09 smithi162 ceph-mon[35761]: pgmap v387: 129 pgs: 129 active+clean; 466 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.4 MiB/s wr, 88 op/s 2024-03-19T02:40:10.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 402275 3.09 MB/sec execute 573 sec latency 31.892 ms 2024-03-19T02:40:10.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:40:10.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:40:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 -- 172.21.15.116:0/1149595813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5501077a0 msgr2=0x7fe550107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 --2- 172.21.15.116:0/1149595813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5501077a0 0x7fe550107b80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe540009930 tx=0x7fe54002bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:10.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 -- 172.21.15.116:0/1149595813 shutdown_connections 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 --2- 172.21.15.116:0/1149595813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5500693c0 0x7fe55010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 --2- 172.21.15.116:0/1149595813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe550068a00 0x7fe550068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 --2- 172.21.15.116:0/1149595813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5501077a0 0x7fe550107b80 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 -- 172.21.15.116:0/1149595813 >> 172.21.15.116:0/1149595813 conn(0x7fe550077820 msgr2=0x7fe550077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 -- 172.21.15.116:0/1149595813 shutdown_connections 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.913+0000 7fe55626c640 1 -- 172.21.15.116:0/1149595813 wait complete. 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 Processor -- start 2024-03-19T02:40:10.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 -- start start 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 0x7fe550198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5501077a0 0x7fe55019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111a70 con 0x7fe5500693c0 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501118f0 con 0x7fe550068a00 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe55626c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111bf0 con 0x7fe5501077a0 2024-03-19T02:40:10.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 0x7fe550198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50794/0 (socket says 172.21.15.116:50794) 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1880075266 learned_addr learned my addr 172.21.15.116:0/1880075266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe554809640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5501077a0 0x7fe55019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1880075266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5501077a0 msgr2=0x7fe55019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5501077a0 0x7fe55019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1880075266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 msgr2=0x7fe550198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.914+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 0x7fe550198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1880075266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55019d000 con 0x7fe5500693c0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54ffff640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 0x7fe550198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe54400e9f0 tx=0x7fe54400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54d7fa640 1 -- 172.21.15.116:0/1880075266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54400cd80 con 0x7fe5500693c0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55019d2f0 con 0x7fe5500693c0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54d7fa640 1 -- 172.21.15.116:0/1880075266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5440108e0 con 0x7fe5500693c0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe54d7fa640 1 -- 172.21.15.116:0/1880075266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544018920 con 0x7fe5500693c0 2024-03-19T02:40:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.915+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5501a4b80 con 0x7fe5500693c0 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.917+0000 7fe54d7fa640 1 -- 172.21.15.116:0/1880075266 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe544010a90 con 0x7fe5500693c0 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.917+0000 7fe558e4c740 1 Processor -- start 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 -- start start 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54d7fa640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 0x7fe5380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe554809640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 0x7fe5380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54d7fa640 1 -- 172.21.15.116:0/1880075266 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fe5440cf4a0 con 0x7fe5500693c0 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f709e3500 con 0x564f70a7f760 2024-03-19T02:40:10.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f7098bbe0 con 0x564f70a80080 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe558e4c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564f709df5a0 con 0x564f70a8d340 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50808/0 (socket says 172.21.15.116:50808) 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 -- 172.21.15.116:0/3566859177 learned_addr learned my addr 172.21.15.116:0/3566859177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe554809640 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 -- 172.21.15.116:0/3566859177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 msgr2=0x564f70a8f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 -- 172.21.15.116:0/3566859177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 msgr2=0x564f70a8ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 -- 172.21.15.116:0/3566859177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f709ad000 con 0x564f70a7f760 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe554809640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 0x7fe5380a4e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe548005fd0 tx=0x7fe548013ae0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe54ffff640 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe54002c240 tx=0x7fe5400095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d456e86fda13eaac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.918+0000 7fe5367fc640 1 -- 172.21.15.116:0/3566859177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540045070 con 0x564f70a7f760 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe5367fc640 1 -- 172.21.15.116:0/3566859177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe54002c7e0 con 0x564f70a7f760 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe5367fc640 1 -- 172.21.15.116:0/3566859177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5400039e0 con 0x564f70a7f760 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 -- 172.21.15.116:0/3566859177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 msgr2=0x564f70a7fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe54002c240 tx=0x7fe5400095f0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 -- 172.21.15.116:0/3566859177 shutdown_connections 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3566859177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a7fb40 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 -- 172.21.15.116:0/3566859177 >> 172.21.15.116:0/3566859177 conn(0x564f70a7a160 msgr2=0x564f70a7a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 -- 172.21.15.116:0/3566859177 shutdown_connections 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.919+0000 7fe558e4c740 1 -- 172.21.15.116:0/3566859177 wait complete. 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 -- start start 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 Processor -- start 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f70a900a0 con 0x564f70a7f760 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f7099f160 con 0x564f70a80080 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564f70a90220 con 0x564f70a8d340 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe558e4c740 10 client.0 _ll_register_callbacks cb 0x564f709dba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50812/0 (socket says 172.21.15.116:50812) 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe54ffff640 1 -- 172.21.15.116:0/799204939 learned_addr learned my addr 172.21.15.116:0/799204939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.920+0000 7fe554809640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 -- 172.21.15.116:0/799204939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 msgr2=0x564f70a8ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 -- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 msgr2=0x564f70a8a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 -- 172.21.15.116:0/799204939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f707c17d0 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe54ffff640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe54002c170 tx=0x7fe54003f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb36194e6c99dd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54004e070 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fe54003fd70 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe54002c7e0 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe54005a050 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe558e4c740 10 client.19116 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f706f73f0 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe54002c7e0 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.921+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe540034440 con 0x564f70a7f760 2024-03-19T02:40:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 10 client.19116 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 10 client.19116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 20 client.19116 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70a9f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 4 client.19116 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f709f0a40 con 0x564f70a9d3f0 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70a9f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe52f7fe640 10 client.19116 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.922+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70a9f7e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87f6e0e0a2723967 server_cookie=6dca1f2b2a61bc4a in_seq=0 out_seq=0 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe52f7fe640 1 -- 172.21.15.116:0/799204939 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f709f0a40 con 0x564f70a9d3f0 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe52f7fe640 10 client.19116 handle_command_reply: tid=0 2024-03-19T02:40:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 client.19116 shutdown 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 msgr2=0x564f70a8a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe54002c170 tx=0x7fe54003f770 comp rx=0 tx=0).stop 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 shutdown_connections 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70a9f7e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70a8ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe52f7fe640 0 client.19116 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a8a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 --2- 172.21.15.116:0/799204939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8a260 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 >> 172.21.15.116:0/799204939 conn(0x564f70a7a160 msgr2=0x564f70a7beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 shutdown_connections 2024-03-19T02:40:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 1 -- 172.21.15.116:0/799204939 wait complete. 2024-03-19T02:40:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.923+0000 7fe558e4c740 20 client.19116 trim_cache size 0 max 16384 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 Processor -- start 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 -- start start 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 0x564f70a88a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 0x564f70a98e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501aca30 con 0x564f70a7f760 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ae570 con 0x564f70a8d340 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.943+0000 7fe558e4c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ad100 con 0x564f70a80080 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 0x564f70a88a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe554809640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 0x564f70a98e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50826/0 (socket says 172.21.15.116:50826) 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54ffff640 1 -- 172.21.15.116:0/3457636988 learned_addr learned my addr 172.21.15.116:0/3457636988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 -- 172.21.15.116:0/3457636988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 msgr2=0x564f70a98e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 0x564f70a98e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 -- 172.21.15.116:0/3457636988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 msgr2=0x564f70a8c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 -- 172.21.15.116:0/3457636988 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f706b70f0 con 0x564f70a80080 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe554809640 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 0x564f70a98e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54ffff640 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:10.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.944+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 0x564f70a88a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe5440cc8b0 tx=0x7fe5440ce040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f51c4f95d6ec90eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.945+0000 7fe535ffb640 1 -- 172.21.15.116:0/3457636988 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440b3070 con 0x564f70a80080 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.945+0000 7fe535ffb640 1 -- 172.21.15.116:0/3457636988 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5440a3f60 con 0x564f70a80080 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.945+0000 7fe535ffb640 1 -- 172.21.15.116:0/3457636988 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440a3d30 con 0x564f70a80080 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 -- 172.21.15.116:0/3457636988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 msgr2=0x564f70a88a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 0x564f70a88a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe5440cc8b0 tx=0x7fe5440ce040 comp rx=0 tx=0).stop 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 -- 172.21.15.116:0/3457636988 shutdown_connections 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a8d340 0x564f70a98e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a80080 0x564f70a88a20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 --2- 172.21.15.116:0/3457636988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a8c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 -- 172.21.15.116:0/3457636988 >> 172.21.15.116:0/3457636988 conn(0x564f70a7a160 msgr2=0x564f70a791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 -- 172.21.15.116:0/3457636988 shutdown_connections 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.946+0000 7fe558e4c740 1 -- 172.21.15.116:0/3457636988 wait complete. 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 -- start start 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 Processor -- start 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a87690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70b02470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501adea0 con 0x564f70a7f760 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501ad7d0 con 0x564f70a80080 2024-03-19T02:40:10.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501aaa30 con 0x564f70a8d340 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe558e4c740 10 client.0 _ll_register_callbacks cb 0x7fe544097b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50834/0 (socket says 172.21.15.116:50834) 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe54ffff640 1 -- 172.21.15.116:0/1736789854 learned_addr learned my addr 172.21.15.116:0/1736789854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a87690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.947+0000 7fe554809640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70b02470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1736789854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 msgr2=0x564f70b02470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70b02470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1736789854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 msgr2=0x564f70a87150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 -- 172.21.15.116:0/1736789854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f70a87f20 con 0x564f70a80080 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54ffff640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe54f7fe640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a87690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x564f70a7f230 tx=0x7fe5440056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f38ee2bc8561075 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:10.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe52f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440b3040 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5440a3f60 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fe5440afba0 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe5440afdc0 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe558e4c740 10 client.28901 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564f70b03a90 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.948+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5440a3f60 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe54409ae50 con 0x564f70a80080 2024-03-19T02:40:10.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 10 client.28901 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 10 client.28901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 20 client.28901 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70b061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 4 client.28901 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f709ead90 con 0x564f70a9d3f0 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe54ffff640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70b061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe54ffff640 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70b061b0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f1be21a82c3af31 server_cookie=89c2a229eb40b6fe in_seq=0 out_seq=0 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe52f7fe640 10 client.28901 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe52f7fe640 1 -- 172.21.15.116:0/1736789854 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f709ead90 con 0x564f70a9d3f0 2024-03-19T02:40:10.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe52f7fe640 10 client.28901 handle_command_reply: tid=0 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.949+0000 7fe558e4c740 1 client.28901 shutdown 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 msgr2=0x564f70a87690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a87690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x564f70a7f230 tx=0x7fe5440056d0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 shutdown_connections 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564f70a9d3f0 0x564f70b061b0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564f70a8d340 0x564f70b02470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f70a80080 0x564f70a87690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 --2- 172.21.15.116:0/1736789854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f70a7f760 0x564f70a87150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe52f7fe640 0 client.28901 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 >> 172.21.15.116:0/1736789854 conn(0x564f70a7a160 msgr2=0x564f70a8a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 shutdown_connections 2024-03-19T02:40:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 1 -- 172.21.15.116:0/1736789854 wait complete. 2024-03-19T02:40:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe558e4c740 20 client.28901 trim_cache size 0 max 16384 2024-03-19T02:40:10.963 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:40:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:40:10.964 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 msgr2=0x7fe5380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 0x7fe5380a4e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe548005fd0 tx=0x7fe548013ae0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 msgr2=0x7fe550198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe54400e9f0 tx=0x7fe54400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 shutdown_connections 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe5380a2970 0x7fe5380a4e30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5501077a0 0x7fe55019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5500693c0 0x7fe550198550 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 --2- 172.21.15.116:0/1880075266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe550068a00 0x7fe550198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.950+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 >> 172.21.15.116:0/1880075266 conn(0x7fe550077820 msgr2=0x7fe5500fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.951+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 shutdown_connections 2024-03-19T02:40:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:10.951+0000 7fe55626c640 1 -- 172.21.15.116:0/1880075266 wait complete. 2024-03-19T02:40:10.967 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:40:10.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- 172.21.15.116:0/2298946180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 msgr2=0x7fd7e4107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 --2- 172.21.15.116:0/2298946180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e4107ba0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009930 tx=0x7fd7e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- 172.21.15.116:0/2298946180 shutdown_connections 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 --2- 172.21.15.116:0/2298946180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 --2- 172.21.15.116:0/2298946180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 0x7fd7e4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 --2- 172.21.15.116:0/2298946180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e4107ba0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- 172.21.15.116:0/2298946180 >> 172.21.15.116:0/2298946180 conn(0x7fd7e4078c20 msgr2=0x7fd7e4079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- 172.21.15.116:0/2298946180 shutdown_connections 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- 172.21.15.116:0/2298946180 wait complete. 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 Processor -- start 2024-03-19T02:40:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.084+0000 7fd7eb137640 1 -- start start 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 0x7fd7e4073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e4074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e4111a40 con 0x7fd7e41077c0 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41118c0 con 0x7fd7e4102170 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7eb137640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e4111bc0 con 0x7fd7e41017b0 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7e96d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e4074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7e8ed3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 0x7fd7e4073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7e96d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50850/0 (socket says 172.21.15.116:50850) 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7e96d4640 1 -- 172.21.15.116:0/1759042926 learned_addr learned my addr 172.21.15.116:0/1759042926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 -- 172.21.15.116:0/1759042926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 msgr2=0x7fd7e4073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 0x7fd7e4073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 -- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 msgr2=0x7fd7e40745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 -- 172.21.15.116:0/1759042926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e41a6500 con 0x7fd7e4102170 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7e96d4640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.085+0000 7fd7dbfff640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e4074060 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400e9f0 tx=0x7fd7d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.086+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d400cd80 con 0x7fd7e4102170 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.086+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e41a67f0 con 0x7fd7e4102170 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.086+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7d40035f0 con 0x7fd7e4102170 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.086+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4010690 con 0x7fd7e4102170 2024-03-19T02:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.086+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e41a6d90 con 0x7fd7e4102170 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.087+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e4067d80 con 0x7fd7e4102170 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.088+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd7d40037a0 con 0x7fd7e4102170 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.089+0000 7fd7d9ffb640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 0x7fd7cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.089+0000 7fd7e8ed3640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 0x7fd7cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.089+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fd7d40cf510 con 0x7fd7e4102170 2024-03-19T02:40:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.089+0000 7fd7e8ed3640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 0x7fd7cc0a4e30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005ab0 tx=0x7fd7e0005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.097+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7d409be30 con 0x7fd7e4102170 2024-03-19T02:40:11.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.344+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7e4109d20 con 0x7fd7e4102170 2024-03-19T02:40:11.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.345+0000 7fd7d9ffb640 1 -- 172.21.15.116:0/1759042926 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11950 (secure 0 0 0) 0x7fd7d409b730 con 0x7fd7e4102170 2024-03-19T02:40:11.346 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:40:11.347 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:30:44.418108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:40:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-19T02:40:11.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 msgr2=0x7fd7cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 0x7fd7cc0a4e30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005ab0 tx=0x7fd7e0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 msgr2=0x7fd7e4074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e4074060 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400e9f0 tx=0x7fd7d400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 shutdown_connections 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd7cc0a2970 0x7fd7cc0a4e30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7e41077c0 0x7fd7e40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e4102170 0x7fd7e4074060 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 --2- 172.21.15.116:0/1759042926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7e41017b0 0x7fd7e4073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 >> 172.21.15.116:0/1759042926 conn(0x7fd7e4078c20 msgr2=0x7fd7e4104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 shutdown_connections 2024-03-19T02:40:11.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.348+0000 7fd7eb137640 1 -- 172.21.15.116:0/1759042926 wait complete. 2024-03-19T02:40:11.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 -- 172.21.15.116:0/2341532295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec1077a0 msgr2=0x7f02ec107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 --2- 172.21.15.116:0/2341532295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec1077a0 0x7f02ec107b80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009930 tx=0x7f02dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 -- 172.21.15.116:0/2341532295 shutdown_connections 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 --2- 172.21.15.116:0/2341532295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 0x7f02ec10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 --2- 172.21.15.116:0/2341532295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec101790 0x7f02ec101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 --2- 172.21.15.116:0/2341532295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec1077a0 0x7f02ec107b80 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 -- 172.21.15.116:0/2341532295 >> 172.21.15.116:0/2341532295 conn(0x7f02ec0fd2c0 msgr2=0x7f02ec0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 -- 172.21.15.116:0/2341532295 shutdown_connections 2024-03-19T02:40:11.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.476+0000 7f02f0859640 1 -- 172.21.15.116:0/2341532295 wait complete. 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 Processor -- start 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 -- start start 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 0x7f02ec198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec1077a0 0x7f02ec19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec111a60 con 0x7f02ec101790 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1118e0 con 0x7f02ec1077a0 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02f0859640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec111be0 con 0x7f02ec102150 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50864/0 (socket says 172.21.15.116:50864) 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02ea59c640 1 -- 172.21.15.116:0/4089481753 learned_addr learned my addr 172.21.15.116:0/4089481753 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 0x7f02ec198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.477+0000 7f02ead9d640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec1077a0 0x7f02ec19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 -- 172.21.15.116:0/4089481753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 msgr2=0x7f02ec198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 0x7f02ec198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 -- 172.21.15.116:0/4089481753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec1077a0 msgr2=0x7f02ec19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec1077a0 0x7f02ec19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 -- 172.21.15.116:0/4089481753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ec19cfc0 con 0x7f02ec101790 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02ea59c640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009230 tx=0x7f02dc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02db7fe640 1 -- 172.21.15.116:0/4089481753 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc043020 con 0x7f02ec101790 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02db7fe640 1 -- 172.21.15.116:0/4089481753 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02dc02ca70 con 0x7f02ec101790 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ec19d250 con 0x7f02ec101790 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02db7fe640 1 -- 172.21.15.116:0/4089481753 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc003a50 con 0x7f02ec101790 2024-03-19T02:40:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.478+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ec1a4b40 con 0x7f02ec101790 2024-03-19T02:40:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.480+0000 7f02db7fe640 1 -- 172.21.15.116:0/4089481753 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f02dc046020 con 0x7f02ec101790 2024-03-19T02:40:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.481+0000 7f02db7fe640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 0x7f02d40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.481+0000 7f02db7fe640 1 -- 172.21.15.116:0/4089481753 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02dc0f1090 con 0x7f02ec101790 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.481+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 0x7f02d40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.481+0000 7f02f3440740 1 Processor -- start 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 -- start start 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 0x56408d26b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d26bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d27f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408d1ce4b0 con 0x56408d27cf00 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56408d176d70 con 0x56408d26ad40 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02f3440740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56408d1ca550 con 0x56408d26b6f0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 0x7f02d40a4eb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0ffa10 tx=0x7f02e0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 0x56408d26b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d27f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d26bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d27f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50870/0 (socket says 172.21.15.116:50870) 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ead9d640 1 -- 172.21.15.116:0/1664562674 learned_addr learned my addr 172.21.15.116:0/1664562674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 -- 172.21.15.116:0/1664562674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 msgr2=0x56408d26bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d26bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 -- 172.21.15.116:0/1664562674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 msgr2=0x56408d27f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d27f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.482+0000 7f02ea59c640 1 -- 172.21.15.116:0/1664562674 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408d199210 con 0x56408d26ad40 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.483+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 0x56408d26b120 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0ee510 tx=0x7f02dc0b7f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca42e878b47985b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.483+0000 7f02d8ff9640 1 -- 172.21.15.116:0/1664562674 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0c1570 con 0x56408d26ad40 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.483+0000 7f02d8ff9640 1 -- 172.21.15.116:0/1664562674 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02dc0c5bc0 con 0x56408d26ad40 2024-03-19T02:40:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.483+0000 7f02d8ff9640 1 -- 172.21.15.116:0/1664562674 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0053d0 con 0x56408d26ad40 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 -- 172.21.15.116:0/1664562674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 msgr2=0x56408d26b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 0x56408d26b120 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0ee510 tx=0x7f02dc0b7f90 comp rx=0 tx=0).stop 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 -- 172.21.15.116:0/1664562674 shutdown_connections 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d27f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d26bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 --2- 172.21.15.116:0/1664562674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26ad40 0x56408d26b120 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 -- 172.21.15.116:0/1664562674 >> 172.21.15.116:0/1664562674 conn(0x56408d265a00 msgr2=0x56408d265de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 -- 172.21.15.116:0/1664562674 shutdown_connections 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.484+0000 7f02f3440740 1 -- 172.21.15.116:0/1664562674 wait complete. 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 -- start start 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 Processor -- start 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26b6f0 0x56408d26d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d26dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408d27fb70 con 0x56408d27cf00 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56408d18a310 con 0x56408d26b6f0 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56408d27fcf0 con 0x56408d26ad40 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02f3440740 10 client.0 _ll_register_callbacks cb 0x56408d17e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d26dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45270/0 (socket says 172.21.15.116:45270) 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26b6f0 0x56408d26d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 -- 172.21.15.116:0/2293435497 learned_addr learned my addr 172.21.15.116:0/2293435497 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 -- 172.21.15.116:0/2293435497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26b6f0 msgr2=0x56408d26d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26b6f0 0x56408d26d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 -- 172.21.15.116:0/2293435497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 msgr2=0x56408d26dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d26dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.485+0000 7f02ea59c640 1 -- 172.21.15.116:0/2293435497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408cee23f0 con 0x56408d26ad40 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02ea59c640 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327a2c4a1bfb6277 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0c1570 con 0x56408d26ad40 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02dc0c5bc0 con 0x56408d26ad40 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02dc0c1120 con 0x56408d26ad40 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f02dc0d62a0 con 0x56408d26ad40 2024-03-19T02:40:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02f3440740 10 client.28873 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56408cfac7d0 con 0x56408d26ad40 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.486+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02dc0c5bc0 con 0x56408d26ad40 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f02dc0c5730 con 0x56408d26ad40 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 10 client.28873 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 10 client.28873 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 20 client.28873 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d28f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 4 client.28873 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56408d22c3e0 con 0x56408d28ced0 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d28f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02d1ffb640 10 client.28873 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.487+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d28f2c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fec4b0664b2c83f server_cookie=5177e3e36646a46d in_seq=0 out_seq=0 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02d1ffb640 1 -- 172.21.15.116:0/2293435497 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56408d22c3e0 con 0x56408d28ced0 2024-03-19T02:40:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02d1ffb640 10 client.28873 handle_command_reply: tid=0 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 client.28873 shutdown 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 msgr2=0x56408d26d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009900 tx=0x7f02dc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 shutdown_connections 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d28f2c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d27cf00 0x56408d26dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02d1ffb640 0 client.28873 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d26b6f0 0x56408d26d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 --2- 172.21.15.116:0/2293435497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26d300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.488+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 >> 172.21.15.116:0/2293435497 conn(0x56408d265a00 msgr2=0x56408d267750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.489+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 shutdown_connections 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.489+0000 7f02f3440740 1 -- 172.21.15.116:0/2293435497 wait complete. 2024-03-19T02:40:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.489+0000 7f02f3440740 20 client.28873 trim_cache size 0 max 16384 2024-03-19T02:40:11.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.503+0000 7f02f3440740 1 Processor -- start 2024-03-19T02:40:11.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.503+0000 7f02f3440740 1 -- start start 2024-03-19T02:40:11.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.503+0000 7f02f3440740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02f3440740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02f3440740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d294950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02f3440740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1acaa0 con 0x56408d26b6f0 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02f3440740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ae5e0 con 0x56408d27cf00 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02f3440740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ad170 con 0x56408d26ad40 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50896/0 (socket says 172.21.15.116:50896) 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 -- 172.21.15.116:0/810450013 learned_addr learned my addr 172.21.15.116:0/810450013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02ea59c640 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02ead9d640 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d294950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 -- 172.21.15.116:0/810450013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 msgr2=0x56408d26fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 -- 172.21.15.116:0/810450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 msgr2=0x56408d294950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d294950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 -- 172.21.15.116:0/810450013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408d266c30 con 0x56408d26b6f0 2024-03-19T02:40:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009a90 tx=0x7f02e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aab1d97259d2c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.504+0000 7f02d3fff640 1 -- 172.21.15.116:0/810450013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0032070 con 0x56408d26b6f0 2024-03-19T02:40:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02d3fff640 1 -- 172.21.15.116:0/810450013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02e002c610 con 0x56408d26b6f0 2024-03-19T02:40:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02d3fff640 1 -- 172.21.15.116:0/810450013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e001da80 con 0x56408d26b6f0 2024-03-19T02:40:11.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 -- 172.21.15.116:0/810450013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 msgr2=0x56408d288910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009a90 tx=0x7f02e0014000 comp rx=0 tx=0).stop 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 -- 172.21.15.116:0/810450013 shutdown_connections 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d294950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26b6f0 0x56408d288910 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 --2- 172.21.15.116:0/810450013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26ad40 0x56408d26fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 -- 172.21.15.116:0/810450013 >> 172.21.15.116:0/810450013 conn(0x56408d265a00 msgr2=0x56408d2676e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 -- 172.21.15.116:0/810450013 shutdown_connections 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.505+0000 7f02f3440740 1 -- 172.21.15.116:0/810450013 wait complete. 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 -- start start 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 Processor -- start 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d27b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d27b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1adf10 con 0x56408d26ad40 2024-03-19T02:40:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1ad840 con 0x56408d27cf00 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec1aaaa0 con 0x56408d26b6f0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02f3440740 10 client.0 _ll_register_callbacks cb 0x7f02e4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50906/0 (socket says 172.21.15.116:50906) 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 -- 172.21.15.116:0/1422067840 learned_addr learned my addr 172.21.15.116:0/1422067840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d27b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ead9d640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d27b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 -- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 msgr2=0x56408d27b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d27b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 -- 172.21.15.116:0/1422067840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 msgr2=0x56408d27b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d27b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.506+0000 7f02ea59c640 1 -- 172.21.15.116:0/1422067840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408d2f20d0 con 0x56408d26ad40 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02ea59c640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0bb970 tx=0x7f02dc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e5fc6ddca85ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0ca060 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02dc0bd680 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e002c610 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f02dc0d1050 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 10 client.19146 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56408d2f20d0 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02dc0aa020 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f02dc0bc870 con 0x56408d26ad40 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 10 client.19146 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 10 client.19146 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 20 client.19146 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d2f5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 4 client.19146 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.507+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56408d1db9f0 con 0x56408d28ced0 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d2f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02e9d9b640 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d2f5720 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a2966a0bf7e7737 server_cookie=125bb1e03a87a0e1 in_seq=0 out_seq=0 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02d1ffb640 10 client.19146 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02d1ffb640 1 -- 172.21.15.116:0/1422067840 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56408d1db9f0 con 0x56408d28ced0 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02d1ffb640 10 client.19146 handle_command_reply: tid=0 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 client.19146 shutdown 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 msgr2=0x56408d27ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0bb970 tx=0x7f02dc005b50 comp rx=0 tx=0).stop 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 shutdown_connections 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56408d28ced0 0x56408d2f5720 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56408d27cf00 0x56408d27b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56408d26b6f0 0x56408d27b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 --2- 172.21.15.116:0/1422067840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408d26ad40 0x56408d27ac40 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 >> 172.21.15.116:0/1422067840 conn(0x56408d265a00 msgr2=0x56408d264310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02d1ffb640 0 client.19146 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 shutdown_connections 2024-03-19T02:40:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.508+0000 7f02f3440740 1 -- 172.21.15.116:0/1422067840 wait complete. 2024-03-19T02:40:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f3440740 20 client.19146 trim_cache size 0 max 16384 2024-03-19T02:40:11.516 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 msgr2=0x7f02d40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 0x7f02d40a4eb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0ffa10 tx=0x7f02e0005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 msgr2=0x7f02ec197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009230 tx=0x7f02dc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 shutdown_connections 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02d40a29f0 0x7f02d40a4eb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ec1077a0 0x7f02ec19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02ec102150 0x7f02ec198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 --2- 172.21.15.116:0/4089481753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02ec101790 0x7f02ec197fd0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 >> 172.21.15.116:0/4089481753 conn(0x7f02ec0fd2c0 msgr2=0x7f02ec0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 shutdown_connections 2024-03-19T02:40:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.509+0000 7f02f0859640 1 -- 172.21.15.116:0/4089481753 wait complete. 2024-03-19T02:40:11.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:40:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 -- 172.21.15.116:0/155521086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 msgr2=0x7fcdd010cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/155521086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 0x7fcdd010cd20 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009ca0 tx=0x7fcdc8030530 comp rx=0 tx=0).stop 2024-03-19T02:40:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 -- 172.21.15.116:0/155521086 shutdown_connections 2024-03-19T02:40:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/155521086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 0x7fcdd010cd20 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/155521086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd0100a30 0x7fcdd0104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/155521086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0100080 0x7fcdd0100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.640+0000 7fcdd73e8640 1 -- 172.21.15.116:0/155521086 >> 172.21.15.116:0/155521086 conn(0x7fcdd0076c50 msgr2=0x7fcdd00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.643+0000 7fcdd73e8640 1 -- 172.21.15.116:0/155521086 shutdown_connections 2024-03-19T02:40:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.643+0000 7fcdd73e8640 1 -- 172.21.15.116:0/155521086 wait complete. 2024-03-19T02:40:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 Processor -- start 2024-03-19T02:40:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 -- start start 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 0x7fcdd0195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 0x7fcdd019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010f8b0 con 0x7fcdd0105150 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010f730 con 0x7fcdd019ad30 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.644+0000 7fcdd73e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010fa30 con 0x7fcdd0196470 2024-03-19T02:40:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd4983640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 0x7fcdd019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37756/0 (socket says 172.21.15.116:37756) 2024-03-19T02:40:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd4983640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 0x7fcdd019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45302/0 (socket says 172.21.15.116:45302) 2024-03-19T02:40:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 -- 172.21.15.116:0/1587201406 learned_addr learned my addr 172.21.15.116:0/1587201406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 -- 172.21.15.116:0/1587201406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 msgr2=0x7fcdd019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 0x7fcdd019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 -- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 msgr2=0x7fcdd0195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 0x7fcdd0195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.645+0000 7fcdd5985640 1 -- 172.21.15.116:0/1587201406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd01a2aa0 con 0x7fcdd019ad30 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8035f50 tx=0x7fcdc8037630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdc67fc640 1 -- 172.21.15.116:0/1587201406 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8046070 con 0x7fcdd019ad30 2024-03-19T02:40:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdc67fc640 1 -- 172.21.15.116:0/1587201406 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdc8003e40 con 0x7fcdd019ad30 2024-03-19T02:40:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdc67fc640 1 -- 172.21.15.116:0/1587201406 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8042420 con 0x7fcdd019ad30 2024-03-19T02:40:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd01a2c70 con 0x7fcdd019ad30 2024-03-19T02:40:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.646+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd01a30e0 con 0x7fcdd019ad30 2024-03-19T02:40:11.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.650+0000 7fcdd9fcb740 1 Processor -- start 2024-03-19T02:40:11.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.651+0000 7fcdd9fcb740 1 -- start start 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.651+0000 7fcdc67fc640 1 -- 172.21.15.116:0/1587201406 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcdc803f040 con 0x7fcdd019ad30 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.651+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.651+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3b16314b0 con 0x55f3b16ce2d0 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3b15d9d70 con 0x55f3b16d77d0 2024-03-19T02:40:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f3b162d550 con 0x55f3b16cebf0 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45306/0 (socket says 172.21.15.116:45306) 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 -- 172.21.15.116:0/1557653450 learned_addr learned my addr 172.21.15.116:0/1557653450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 -- 172.21.15.116:0/1557653450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 msgr2=0x55f3b16d9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 -- 172.21.15.116:0/1557653450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 msgr2=0x55f3b16ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd5184640 1 -- 172.21.15.116:0/1557653450 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3b15fc210 con 0x55f3b16cebf0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdc67fc640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 0x7fcdbc0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.652+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdc67fc640 1 -- 172.21.15.116:0/1587201406 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fcdc80f6490 con 0x7fcdd019ad30 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009760 tx=0x7fcdcc009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50248fb1e01af842 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 0x7fcdbc0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/1557653450 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc015020 con 0x55f3b16cebf0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/1557653450 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc011420 con 0x55f3b16cebf0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.653+0000 7fcdbb7fe640 1 -- 172.21.15.116:0/1557653450 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0105e0 con 0x55f3b16cebf0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.654+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 0x7fcdbc0a5030 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f3b16c9b90 tx=0x7fcdc000b390 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.654+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1557653450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 msgr2=0x55f3b16d7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.654+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009760 tx=0x7fcdcc009c20 comp rx=0 tx=0).stop 2024-03-19T02:40:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.655+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1557653450 shutdown_connections 2024-03-19T02:40:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.655+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.655+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16d7290 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.655+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1557653450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.655+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1557653450 >> 172.21.15.116:0/1557653450 conn(0x55f3b16c8a00 msgr2=0x55f3b16c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.657+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1557653450 shutdown_connections 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.657+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1557653450 wait complete. 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.658+0000 7fcdd9fcb740 1 -- start start 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.658+0000 7fcdd9fcb740 1 Processor -- start 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16cebf0 0x55f3b16d4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3b16da5c0 con 0x55f3b16cebf0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3b15ed310 con 0x55f3b16d77d0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f3b16da740 con 0x55f3b16ce2d0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45314/0 (socket says 172.21.15.116:45314) 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd5184640 1 -- 172.21.15.116:0/3310549727 learned_addr learned my addr 172.21.15.116:0/3310549727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd4983640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16cebf0 0x55f3b16d4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd5985640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.659+0000 7fcdd9fcb740 10 client.0 _ll_register_callbacks cb 0x55f3b15e12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.660+0000 7fcdd5184640 1 -- 172.21.15.116:0/3310549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 msgr2=0x55f3b16d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.660+0000 7fcdd5184640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.660+0000 7fcdd5184640 1 -- 172.21.15.116:0/3310549727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16cebf0 msgr2=0x55f3b16d4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.660+0000 7fcdd5184640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16cebf0 0x55f3b16d4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.660+0000 7fcdd5184640 1 -- 172.21.15.116:0/3310549727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3b13453f0 con 0x55f3b16ce2d0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.661+0000 7fcdd5184640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc004880 tx=0x7fcdcc009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eb3d6bee6e137f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.661+0000 7fcdb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.661+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc015030 con 0x55f3b16ce2d0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.661+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc011420 con 0x55f3b16ce2d0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.662+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fcdcc00b8d0 con 0x55f3b16ce2d0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.662+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc020070 con 0x55f3b16ce2d0 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.662+0000 7fcdd9fcb740 10 client.28888 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.662+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f3b140f7d0 con 0x55f3b16ce2d0 2024-03-19T02:40:11.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc011420 con 0x55f3b16ce2d0 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcdcc01ac10 con 0x55f3b16ce2d0 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 10 client.28888 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 10 client.28888 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 20 client.28888 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b16f0840 0x55f3b16f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 4 client.28888 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.665+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3b168f3e0 con 0x55f3b16f0840 2024-03-19T02:40:11.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.667+0000 7fcdd4983640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b16f0840 0x55f3b16f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.667+0000 7fcdd4983640 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b16f0840 0x55f3b16f2d00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ccbeb5ad461b6d83 server_cookie=76c87ee36ee4da59 in_seq=0 out_seq=0 2024-03-19T02:40:11.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.667+0000 7fcdb8ff9640 10 client.28888 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.669+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/3310549727 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3b168f3e0 con 0x55f3b16f0840 2024-03-19T02:40:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.669+0000 7fcdb8ff9640 10 client.28888 handle_command_reply: tid=0 2024-03-19T02:40:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.669+0000 7fcdd9fcb740 1 client.28888 shutdown 2024-03-19T02:40:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 msgr2=0x55f3b16d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc004880 tx=0x7fcdcc009ee0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 shutdown_connections 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16d77d0 0x55f3b16d52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b16f0840 0x55f3b16f2d00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16cebf0 0x55f3b16d4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/3310549727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d4860 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 >> 172.21.15.116:0/3310549727 conn(0x55f3b16c8a00 msgr2=0x55f3b16ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdb8ff9640 0 client.28888 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:11.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 shutdown_connections 2024-03-19T02:40:11.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/3310549727 wait complete. 2024-03-19T02:40:11.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.671+0000 7fcdd9fcb740 20 client.28888 trim_cache size 0 max 16384 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 Processor -- start 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 -- start start 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16cebf0 0x55f3b16ebc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16d77d0 0x55f3b16f8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01acf50 con 0x55f3b16ce2d0 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01aea90 con 0x55f3b16cebf0 2024-03-19T02:40:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01ad620 con 0x55f3b16d77d0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50964/0 (socket says 172.21.15.116:50964) 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.688+0000 7fcdd5184640 1 -- 172.21.15.116:0/1816123331 learned_addr learned my addr 172.21.15.116:0/1816123331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16cebf0 0x55f3b16ebc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16d77d0 0x55f3b16f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 -- 172.21.15.116:0/1816123331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16d77d0 msgr2=0x55f3b16f8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16d77d0 0x55f3b16f8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 -- 172.21.15.116:0/1816123331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16cebf0 msgr2=0x55f3b16ebc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16cebf0 0x55f3b16ebc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 -- 172.21.15.116:0/1816123331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3b16d3810 con 0x55f3b16ce2d0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001eb0 tx=0x7fcdcc0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9eb25e5926030bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdbaffd640 1 -- 172.21.15.116:0/1816123331 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc015070 con 0x55f3b16ce2d0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdbaffd640 1 -- 172.21.15.116:0/1816123331 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc019e20 con 0x55f3b16ce2d0 2024-03-19T02:40:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.689+0000 7fcdbaffd640 1 -- 172.21.15.116:0/1816123331 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00be70 con 0x55f3b16ce2d0 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1816123331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 msgr2=0x55f3b16d6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001eb0 tx=0x7fcdcc0091e0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1816123331 shutdown_connections 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16d77d0 0x55f3b16f8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16cebf0 0x55f3b16ebc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1816123331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16ce2d0 0x55f3b16d6f90 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1816123331 >> 172.21.15.116:0/1816123331 conn(0x55f3b16c8a00 msgr2=0x55f3b16c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1816123331 shutdown_connections 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1816123331 wait complete. 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 -- start start 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.690+0000 7fcdd9fcb740 1 Processor -- start 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16e3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 0x55f3b1754c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01ae3c0 con 0x55f3b16d77d0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01adcf0 con 0x55f3b16ce2d0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01aaf50 con 0x55f3b16cebf0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd9fcb740 10 client.0 _ll_register_callbacks cb 0x7fcdcc009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5184640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37794/0 (socket says 172.21.15.116:37794) 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5184640 1 -- 172.21.15.116:0/1859128308 learned_addr learned my addr 172.21.15.116:0/1859128308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 0x55f3b1754c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 -- 172.21.15.116:0/1859128308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 msgr2=0x55f3b16e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 -- 172.21.15.116:0/1859128308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 msgr2=0x55f3b1754c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 0x55f3b1754c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd4983640 1 -- 172.21.15.116:0/1859128308 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3b1755220 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.691+0000 7fcdd5985640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 0x55f3b1754c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdd4983640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16e3970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55f3b16f35b0 tx=0x7fcdc000b5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a740a0846ad4ec8d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0004250 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fcdc00043f0 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc019e20 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcdc000f6a0 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdd9fcb740 10 client.28894 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f3b1756560 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.692+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc019e20 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcdc0030020 con 0x55f3b16cebf0 2024-03-19T02:40:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 10 client.28894 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 10 client.28894 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 20 client.28894 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b1756c90 0x55f3b17590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 4 client.28894 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3b163e9f0 con 0x55f3b1756c90 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b1756c90 0x55f3b17590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdb8ff9640 10 client.28894 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd5184640 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b1756c90 0x55f3b17590b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d42cda7d48c8cae8 server_cookie=3560eb2e065d3f59 in_seq=0 out_seq=0 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdb8ff9640 1 -- 172.21.15.116:0/1859128308 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3b163e9f0 con 0x55f3b1756c90 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdb8ff9640 10 client.28894 handle_command_reply: tid=0 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.693+0000 7fcdd9fcb740 1 client.28894 shutdown 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 msgr2=0x55f3b16e3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16e3970 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55f3b16f35b0 tx=0x7fcdc000b5b0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 shutdown_connections 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3b1756c90 0x55f3b17590b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b16d77d0 0x55f3b1754c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3b16cebf0 0x55f3b16e3970 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 --2- 172.21.15.116:0/1859128308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3b16ce2d0 0x55f3b16e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdb8ff9640 0 client.28894 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 >> 172.21.15.116:0/1859128308 conn(0x55f3b16c8a00 msgr2=0x55f3b16c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 shutdown_connections 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 1 -- 172.21.15.116:0/1859128308 wait complete. 2024-03-19T02:40:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd9fcb740 20 client.28894 trim_cache size 0 max 16384 2024-03-19T02:40:11.703 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 msgr2=0x7fcdbc0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 0x7fcdbc0a5030 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55f3b16c9b90 tx=0x7fcdc000b390 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 msgr2=0x7fcdd019b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8035f50 tx=0x7fcdc8037630 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 shutdown_connections 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdbc0a2b70 0x7fcdbc0a5030 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd019ad30 0x7fcdd019b1e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd0196470 0x7fcdd019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 --2- 172.21.15.116:0/1587201406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0105150 0x7fcdd0195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.694+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 >> 172.21.15.116:0/1587201406 conn(0x7fcdd0076c50 msgr2=0x7fcdd01034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.695+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 shutdown_connections 2024-03-19T02:40:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.695+0000 7fcdd73e8640 1 -- 172.21.15.116:0/1587201406 wait complete. 2024-03-19T02:40:11.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:40:11.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 402695 3.09 MB/sec execute 574 sec latency 100.713 ms 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 -- 172.21.15.116:0/1274134664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c069df0 msgr2=0x7f657c06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 --2- 172.21.15.116:0/1274134664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c069df0 0x7f657c06a270 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f656c009930 tx=0x7f656c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 -- 172.21.15.116:0/1274134664 shutdown_connections 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 --2- 172.21.15.116:0/1274134664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c06a7b0 0x7f657c10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 --2- 172.21.15.116:0/1274134664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c069df0 0x7f657c06a270 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 --2- 172.21.15.116:0/1274134664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c1067c0 0x7f657c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 -- 172.21.15.116:0/1274134664 >> 172.21.15.116:0/1274134664 conn(0x7f657c077700 msgr2=0x7f657c077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 -- 172.21.15.116:0/1274134664 shutdown_connections 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.819+0000 7f6583962640 1 -- 172.21.15.116:0/1274134664 wait complete. 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 Processor -- start 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 -- start start 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c069df0 0x7f657c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c06a7b0 0x7f657c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f657c111a60 con 0x7f657c1067c0 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1118e0 con 0x7f657c069df0 2024-03-19T02:40:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6583962640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f657c111be0 con 0x7f657c06a7b0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50992/0 (socket says 172.21.15.116:50992) 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 -- 172.21.15.116:0/3245132346 learned_addr learned my addr 172.21.15.116:0/3245132346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6580efd640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c06a7b0 0x7f657c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f65816fe640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c069df0 0x7f657c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 -- 172.21.15.116:0/3245132346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c06a7b0 msgr2=0x7f657c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c06a7b0 0x7f657c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 -- 172.21.15.116:0/3245132346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c069df0 msgr2=0x7f657c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.820+0000 7f6581eff640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c069df0 0x7f657c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f6581eff640 1 -- 172.21.15.116:0/3245132346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c19f190 con 0x7f657c1067c0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f6581eff640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f657400eb10 tx=0x7f657400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f65727fc640 1 -- 172.21.15.116:0/3245132346 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574004b20 con 0x7f657c1067c0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f65727fc640 1 -- 172.21.15.116:0/3245132346 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6574004cc0 con 0x7f657c1067c0 2024-03-19T02:40:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c19f480 con 0x7f657c1067c0 2024-03-19T02:40:11.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f65727fc640 1 -- 172.21.15.116:0/3245132346 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574010690 con 0x7f657c1067c0 2024-03-19T02:40:11.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.821+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c1a6d10 con 0x7f657c1067c0 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.823+0000 7f65727fc640 1 -- 172.21.15.116:0/3245132346 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6574005020 con 0x7f657c1067c0 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 Processor -- start 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 -- start start 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c23136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c231c7d0 0x55e7c231ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65727fc640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 0x7f65680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7c22764b0 con 0x55e7c23132d0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e7c221ed70 con 0x55e7c2313bf0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f658662b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e7c2272550 con 0x55e7c231c7d0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c23136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f6581eff640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 0x7f65680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c23136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50994/0 (socket says 172.21.15.116:50994) 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37804/0 (socket says 172.21.15.116:37804) 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f6580efd640 1 -- 172.21.15.116:0/2549209559 learned_addr learned my addr 172.21.15.116:0/2549209559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f6580efd640 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c231c7d0 0x55e7c231ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65727fc640 1 -- 172.21.15.116:0/3245132346 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f65740cf7c0 con 0x7f657c1067c0 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 -- 172.21.15.116:0/2549209559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c231c7d0 msgr2=0x55e7c231ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c231c7d0 0x55e7c231ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 -- 172.21.15.116:0/2549209559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 msgr2=0x55e7c23136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.824+0000 7f65816fe640 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c23136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f65816fe640 1 -- 172.21.15.116:0/2549209559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7c2241210 con 0x55e7c2313bf0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f6581eff640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 0x7f65680a4e30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f657409b740 tx=0x7f657406a850 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f65816fe640 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6578009c00 tx=0x7f6578011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21005894b76f85e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f65677fe640 1 -- 172.21.15.116:0/2549209559 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578004050 con 0x55e7c2313bf0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f65677fe640 1 -- 172.21.15.116:0/2549209559 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65780054d0 con 0x55e7c2313bf0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.825+0000 7f65677fe640 1 -- 172.21.15.116:0/2549209559 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578004cd0 con 0x55e7c2313bf0 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- 172.21.15.116:0/2549209559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 msgr2=0x55e7c231c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6578009c00 tx=0x7f6578011810 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- 172.21.15.116:0/2549209559 shutdown_connections 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c231c7d0 0x55e7c231ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231c290 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 --2- 172.21.15.116:0/2549209559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c23136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- 172.21.15.116:0/2549209559 >> 172.21.15.116:0/2549209559 conn(0x55e7c230da00 msgr2=0x55e7c230dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- 172.21.15.116:0/2549209559 shutdown_connections 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- 172.21.15.116:0/2549209559 wait complete. 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 -- start start 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.826+0000 7f658662b740 1 Processor -- start 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c2319cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c231c7d0 0x55e7c231a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7c231f5c0 con 0x55e7c23132d0 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e7c2232310 con 0x55e7c231c7d0 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e7c231f740 con 0x55e7c2313bf0 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 10 client.0 _ll_register_callbacks cb 0x55e7c22262d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51000/0 (socket says 172.21.15.116:51000) 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 -- 172.21.15.116:0/733614001 learned_addr learned my addr 172.21.15.116:0/733614001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f65816fe640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c2319cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c231c7d0 0x55e7c231a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 -- 172.21.15.116:0/733614001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 msgr2=0x55e7c2319cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c2319cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 -- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c231c7d0 msgr2=0x55e7c231a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c231c7d0 0x55e7c231a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 -- 172.21.15.116:0/733614001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7c1f8a3f0 con 0x55e7c23132d0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6580efd640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f656c005ab0 tx=0x7f656c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cdef984d230926 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6564ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c04a020 con 0x55e7c23132d0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65780054d0 con 0x55e7c23132d0 2024-03-19T02:40:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.827+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f656c0052d0 con 0x55e7c23132d0 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f656c0054d0 con 0x55e7c23132d0 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 10 client.19173 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e7c20547d0 con 0x55e7c23132d0 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65780054d0 con 0x55e7c23132d0 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f656c0036f0 con 0x55e7c23132d0 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 10 client.19173 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 10 client.19173 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 20 client.19173 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c232cb00 0x55e7c232eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 4 client.19173 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:11.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7c22d43e0 con 0x55e7c232cb00 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.828+0000 7f65816fe640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c232cb00 0x55e7c232eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.829+0000 7f6564ff9640 10 client.19173 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.829+0000 7f65816fe640 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c232cb00 0x55e7c232eef0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ce807e5b71e70cb server_cookie=41c130b7ddc74596 in_seq=0 out_seq=0 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.829+0000 7f6564ff9640 1 -- 172.21.15.116:0/733614001 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7c22d43e0 con 0x55e7c232cb00 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.829+0000 7f6564ff9640 10 client.19173 handle_command_reply: tid=0 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.829+0000 7f658662b740 1 client.19173 shutdown 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 msgr2=0x55e7c2319780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f656c005ab0 tx=0x7f656c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 shutdown_connections 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c232cb00 0x55e7c232eef0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c231c7d0 0x55e7c231a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c2319cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 --2- 172.21.15.116:0/733614001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c23132d0 0x55e7c2319780 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 >> 172.21.15.116:0/733614001 conn(0x55e7c230da00 msgr2=0x55e7c230f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f6564ff9640 0 client.19173 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:11.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 shutdown_connections 2024-03-19T02:40:11.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 1 -- 172.21.15.116:0/733614001 wait complete. 2024-03-19T02:40:11.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.830+0000 7f658662b740 20 client.19173 trim_cache size 0 max 16384 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 Processor -- start 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 -- start start 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c23132d0 0x55e7c2328390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c2334700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1b0e60 con 0x55e7c231c7d0 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1b29a0 con 0x55e7c23132d0 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f658662b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1b1530 con 0x55e7c2313bf0 2024-03-19T02:40:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c2334700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45384/0 (socket says 172.21.15.116:45384) 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 -- 172.21.15.116:0/4263796134 learned_addr learned my addr 172.21.15.116:0/4263796134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f6580efd640 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c23132d0 0x55e7c2328390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 -- 172.21.15.116:0/4263796134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c23132d0 msgr2=0x55e7c2328390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c23132d0 0x55e7c2328390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 -- 172.21.15.116:0/4263796134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 msgr2=0x55e7c2334700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c2334700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 -- 172.21.15.116:0/4263796134 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7c2313a20 con 0x55e7c2313bf0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.845+0000 7f65816fe640 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6578009990 tx=0x7f6578005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=134a54001c789d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f6566ffd640 1 -- 172.21.15.116:0/4263796134 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578032070 con 0x55e7c2313bf0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f6566ffd640 1 -- 172.21.15.116:0/4263796134 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f657802ecb0 con 0x55e7c2313bf0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f6566ffd640 1 -- 172.21.15.116:0/4263796134 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65780035f0 con 0x55e7c2313bf0 2024-03-19T02:40:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 -- 172.21.15.116:0/4263796134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 msgr2=0x55e7c231a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6578009990 tx=0x7f6578005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 -- 172.21.15.116:0/4263796134 shutdown_connections 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c2334700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c2313bf0 0x55e7c231a020 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 --2- 172.21.15.116:0/4263796134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c23132d0 0x55e7c2328390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.846+0000 7f658662b740 1 -- 172.21.15.116:0/4263796134 >> 172.21.15.116:0/4263796134 conn(0x55e7c230da00 msgr2=0x55e7c230c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- 172.21.15.116:0/4263796134 shutdown_connections 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- 172.21.15.116:0/4263796134 wait complete. 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- start start 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 Processor -- start 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c23132d0 0x55e7c2324920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1b22d0 con 0x55e7c231c7d0 2024-03-19T02:40:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1b1c00 con 0x55e7c2313bf0 2024-03-19T02:40:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1aee60 con 0x55e7c23132d0 2024-03-19T02:40:11.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f658662b740 10 client.0 _ll_register_callbacks cb 0x7f656c004290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f6580efd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51018/0 (socket says 172.21.15.116:51018) 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f6580efd640 1 -- 172.21.15.116:0/2777444807 learned_addr learned my addr 172.21.15.116:0/2777444807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f65816fe640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.847+0000 7f6580efd640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c23132d0 0x55e7c2324920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 -- 172.21.15.116:0/2777444807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c23132d0 msgr2=0x55e7c2324920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c23132d0 0x55e7c2324920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 -- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 msgr2=0x55e7c231fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 -- 172.21.15.116:0/2777444807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7c2320810 con 0x55e7c231c7d0 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f65816fe640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:11.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6580efd640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f656c005770 tx=0x7f656c009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585d95d9eebd4041 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c04a070 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f656c0046c0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f657802ecb0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f656c0048c0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f658662b740 10 client.19179 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e7c23929f0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f657802ecb0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.848+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f656c043ab0 con 0x55e7c231c7d0 2024-03-19T02:40:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 10 client.19179 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 10 client.19179 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 20 client.19179 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c2393050 0x55e7c2395470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 4 client.19179 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7c22839f0 con 0x55e7c2393050 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f65816fe640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c2393050 0x55e7c2395470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f6564ff9640 10 client.19179 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f65816fe640 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c2393050 0x55e7c2395470 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7d65831b31a92c0 server_cookie=1fc8aec6e172bcc7 in_seq=0 out_seq=0 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f6564ff9640 1 -- 172.21.15.116:0/2777444807 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7c22839f0 con 0x55e7c2393050 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f6564ff9640 10 client.19179 handle_command_reply: tid=0 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.849+0000 7f658662b740 1 client.19179 shutdown 2024-03-19T02:40:11.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 msgr2=0x55e7c23202d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f656c005770 tx=0x7f656c009660 comp rx=0 tx=0).stop 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 shutdown_connections 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55e7c2393050 0x55e7c2395470 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7c231c7d0 0x55e7c23202d0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6564ff9640 0 client.19179 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7c2313bf0 0x55e7c231fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 --2- 172.21.15.116:0/2777444807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e7c23132d0 0x55e7c2324920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 >> 172.21.15.116:0/2777444807 conn(0x55e7c230da00 msgr2=0x55e7c230ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 shutdown_connections 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 1 -- 172.21.15.116:0/2777444807 wait complete. 2024-03-19T02:40:11.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f658662b740 20 client.19179 trim_cache size 0 max 16384 2024-03-19T02:40:11.862 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 msgr2=0x7f65680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 0x7f65680a4e30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f657409b740 tx=0x7f657406a850 comp rx=0 tx=0).stop 2024-03-19T02:40:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 msgr2=0x7f657c19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f657400eb10 tx=0x7f657400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 shutdown_connections 2024-03-19T02:40:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65680a2970 0x7f65680a4e30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657c1067c0 0x7f657c19ea60 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f657c06a7b0 0x7f657c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 --2- 172.21.15.116:0/3245132346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657c069df0 0x7f657c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.850+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 >> 172.21.15.116:0/3245132346 conn(0x7f657c077700 msgr2=0x7f657c10cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.851+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 shutdown_connections 2024-03-19T02:40:11.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.851+0000 7f6583962640 1 -- 172.21.15.116:0/3245132346 wait complete. 2024-03-19T02:40:11.868 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 -- 172.21.15.116:0/3178351736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc101060 msgr2=0x7f19bc10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 --2- 172.21.15.116:0/3178351736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc101060 0x7f19bc10d570 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009cd0 tx=0x7f19b402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 -- 172.21.15.116:0/3178351736 shutdown_connections 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 --2- 172.21.15.116:0/3178351736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc101060 0x7f19bc10d570 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 --2- 172.21.15.116:0/3178351736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 0x7f19bc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 --2- 172.21.15.116:0/3178351736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc1066b0 0x7f19bc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 -- 172.21.15.116:0/3178351736 >> 172.21.15.116:0/3178351736 conn(0x7f19bc0778d0 msgr2=0x7f19bc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 -- 172.21.15.116:0/3178351736 shutdown_connections 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 -- 172.21.15.116:0/3178351736 wait complete. 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.984+0000 7f19c1b22640 1 Processor -- start 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 -- start start 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 0x7f19bc102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 0x7f19bc103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 0x7f19bc103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc110970 con 0x7f19bc1066b0 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1107f0 con 0x7f19bc1006a0 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19c1b22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc110af0 con 0x7f19bc101060 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 0x7f19bc103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 0x7f19bc103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 0x7f19bc102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 0x7f19bc103200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45406/0 (socket says 172.21.15.116:45406) 2024-03-19T02:40:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19baffd640 1 -- 172.21.15.116:0/2663279013 learned_addr learned my addr 172.21.15.116:0/2663279013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 -- 172.21.15.116:0/2663279013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 msgr2=0x7f19bc103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 0x7f19bc103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 -- 172.21.15.116:0/2663279013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 msgr2=0x7f19bc102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 0x7f19bc102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 -- 172.21.15.116:0/2663279013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc1a6450 con 0x7f19bc1066b0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.985+0000 7f19bbfff640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 0x7f19bc103740 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f19b402c630 tx=0x7f19b4035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.986+0000 7f19b8ff9640 1 -- 172.21.15.116:0/2663279013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4039040 con 0x7f19bc1066b0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.986+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc1a66e0 con 0x7f19bc1066b0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.986+0000 7f19b8ff9640 1 -- 172.21.15.116:0/2663279013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19b4044070 con 0x7f19bc1066b0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.986+0000 7f19b8ff9640 1 -- 172.21.15.116:0/2663279013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b40042a0 con 0x7f19bc1066b0 2024-03-19T02:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.986+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc1a6bf0 con 0x7f19bc1066b0 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.988+0000 7f19b8ff9640 1 -- 172.21.15.116:0/2663279013 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f19b4004440 con 0x7f19bc1066b0 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.988+0000 7f19c47ee740 1 Processor -- start 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.988+0000 7f19c47ee740 1 -- start start 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe1aa1f4b0 con 0x55fe1aabcbf0 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe1a9c7d70 con 0x55fe1aac57d0 2024-03-19T02:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19c47ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe1aa1b550 con 0x55fe1aabc2d0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19b8ff9640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 0x7f19a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45420/0 (socket says 172.21.15.116:45420) 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51044/0 (socket says 172.21.15.116:51044) 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 -- 172.21.15.116:0/3704071031 learned_addr learned my addr 172.21.15.116:0/3704071031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19bbfff640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 0x7f19a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 -- 172.21.15.116:0/3704071031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 msgr2=0x55fe1aac7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 -- 172.21.15.116:0/3704071031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 msgr2=0x55fe1aac5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 -- 172.21.15.116:0/3704071031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe1a9ea210 con 0x55fe1aabc2d0 2024-03-19T02:40:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19b8ff9640 1 -- 172.21.15.116:0/2663279013 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f19b40f1540 con 0x7f19bc1066b0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.989+0000 7f19baffd640 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b00118d0 tx=0x7f19b0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac770752c33b41f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19bbfff640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 0x7f19a40a4e30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f19b40eea70 tx=0x7f19b40a8040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19a1ffb640 1 -- 172.21.15.116:0/3704071031 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b0005020 con 0x55fe1aabc2d0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19a1ffb640 1 -- 172.21.15.116:0/3704071031 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19b00047a0 con 0x55fe1aabc2d0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19a1ffb640 1 -- 172.21.15.116:0/3704071031 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b000fa20 con 0x55fe1aabc2d0 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 -- 172.21.15.116:0/3704071031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 msgr2=0x55fe1aabc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b00118d0 tx=0x7f19b0004730 comp rx=0 tx=0).stop 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 -- 172.21.15.116:0/3704071031 shutdown_connections 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 --2- 172.21.15.116:0/3704071031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aabc6b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 -- 172.21.15.116:0/3704071031 >> 172.21.15.116:0/3704071031 conn(0x55fe1aab6a00 msgr2=0x55fe1aab6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 -- 172.21.15.116:0/3704071031 shutdown_connections 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.990+0000 7f19c47ee740 1 -- 172.21.15.116:0/3704071031 wait complete. 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 -- start start 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 Processor -- start 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aac2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aac32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe1aac85c0 con 0x55fe1aabcbf0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe1a9db310 con 0x55fe1aabc2d0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe1aac8740 con 0x55fe1aac57d0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19c47ee740 10 client.0 _ll_register_callbacks cb 0x55fe1a9cf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19baffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aac32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:11.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51048/0 (socket says 172.21.15.116:51048) 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19bb7fe640 1 -- 172.21.15.116:0/1363533484 learned_addr learned my addr 172.21.15.116:0/1363533484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.991+0000 7f19baffd640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aac2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 -- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 msgr2=0x55fe1aac32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aac32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 -- 172.21.15.116:0/1363533484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 msgr2=0x55fe1aac2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aac2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 -- 172.21.15.116:0/1363533484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1a7333f0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0126e0 tx=0x7f19ac012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24f48995d14f116 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac0120d0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f19ac00f920 con 0x55fe1aabcbf0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b00047a0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f19ac012270 con 0x55fe1aabcbf0 2024-03-19T02:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19c47ee740 10 client.19191 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe1a7fd7d0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19b00047a0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.992+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f19ac01c420 con 0x55fe1aabcbf0 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 10 client.19191 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 10 client.19191 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 20 client.19191 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1aad5c00 0x55fe1aad8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 4 client.19191 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe1aa7d3e0 con 0x55fe1aad5c00 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19baffd640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1aad5c00 0x55fe1aad8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f19baffd640 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1aad5c00 0x55fe1aad8030 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a5867f03ac034d3 server_cookie=8b26c0318d8f3d16 in_seq=0 out_seq=0 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.993+0000 7f199affd640 10 client.19191 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f199affd640 1 -- 172.21.15.116:0/1363533484 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe1aa7d3e0 con 0x55fe1aad5c00 2024-03-19T02:40:12.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f199affd640 10 client.19191 handle_command_reply: tid=0 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 client.19191 shutdown 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 msgr2=0x55fe1aac2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0126e0 tx=0x7f19ac012ba0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 shutdown_connections 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1aad5c00 0x55fe1aad8030 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aac32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f199affd640 0 client.19191 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aac2d80 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 --2- 172.21.15.116:0/1363533484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aac2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 >> 172.21.15.116:0/1363533484 conn(0x55fe1aab6a00 msgr2=0x55fe1aab8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 shutdown_connections 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 1 -- 172.21.15.116:0/1363533484 wait complete. 2024-03-19T02:40:12.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:11.994+0000 7f19c47ee740 20 client.19191 trim_cache size 0 max 16384 2024-03-19T02:40:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:11 smithi116 ceph-mon[32263]: pgmap v388: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.7 MiB/s wr, 143 op/s 2024-03-19T02:40:12.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:11 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:11 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1759042926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.009+0000 7f19c47ee740 1 Processor -- start 2024-03-19T02:40:12.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.009+0000 7f19c47ee740 1 -- start start 2024-03-19T02:40:12.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aad1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aacd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1b0df0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1b2930 con 0x55fe1aabc2d0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19c47ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1b14c0 con 0x55fe1aac57d0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19baffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aad1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51062/0 (socket says 172.21.15.116:51062) 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 -- 172.21.15.116:0/2078558553 learned_addr learned my addr 172.21.15.116:0/2078558553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19baffd640 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aacd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 -- 172.21.15.116:0/2078558553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 msgr2=0x55fe1aacd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aacd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 -- 172.21.15.116:0/2078558553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 msgr2=0x55fe1aad1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aad1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 -- 172.21.15.116:0/2078558553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe1aad86b0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19baffd640 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aad1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.010+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00f830 tx=0x7f19ac005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98b4fdf8c27d454 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19a17fa640 1 -- 172.21.15.116:0/2078558553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac02f070 con 0x55fe1aabcbf0 2024-03-19T02:40:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19a17fa640 1 -- 172.21.15.116:0/2078558553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19ac00fdd0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19a17fa640 1 -- 172.21.15.116:0/2078558553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac02a440 con 0x55fe1aabcbf0 2024-03-19T02:40:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 -- 172.21.15.116:0/2078558553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 msgr2=0x55fe1aadd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00f830 tx=0x7f19ac005690 comp rx=0 tx=0).stop 2024-03-19T02:40:12.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 -- 172.21.15.116:0/2078558553 shutdown_connections 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aac57d0 0x55fe1aacd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aadd940 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.011+0000 7f19c47ee740 1 --2- 172.21.15.116:0/2078558553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aabc2d0 0x55fe1aad1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- 172.21.15.116:0/2078558553 >> 172.21.15.116:0/2078558553 conn(0x55fe1aab6a00 msgr2=0x55fe1aab5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- 172.21.15.116:0/2078558553 shutdown_connections 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- 172.21.15.116:0/2078558553 wait complete. 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- start start 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 Processor -- start 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aacd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1b2260 con 0x55fe1aabcbf0 2024-03-19T02:40:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1b1b90 con 0x55fe1aac57d0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc1aedf0 con 0x55fe1aabc2d0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19c47ee740 10 client.0 _ll_register_callbacks cb 0x7f19ac009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51074/0 (socket says 172.21.15.116:51074) 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19bb7fe640 1 -- 172.21.15.116:0/4024365211 learned_addr learned my addr 172.21.15.116:0/4024365211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19baffd640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aacd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.012+0000 7f19baffd640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 -- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 msgr2=0x55fe1aacd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aacd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 -- 172.21.15.116:0/4024365211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 msgr2=0x55fe1aac9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 -- 172.21.15.116:0/4024365211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1aaca5c0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19bb7fe640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00f830 tx=0x7f19ac01cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16588804c3323905 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac02f070 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f19ac00f920 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19ac00fdd0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f19ac03c050 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19c47ee740 10 client.19197 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe1ab3cb90 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19ac00fdd0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f19ac0048f0 con 0x55fe1aabcbf0 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19c47ee740 10 client.19197 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.013+0000 7f19c47ee740 10 client.19197 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 20 client.19197 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1ab3d200 0x55fe1ab3f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 4 client.19197 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe1aa2c9f0 con 0x55fe1ab3d200 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19baffd640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1ab3d200 0x55fe1ab3f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f199affd640 10 client.19197 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19baffd640 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1ab3d200 0x55fe1ab3f620 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=881c41554d9bb8ba server_cookie=6c044660e941b215 in_seq=0 out_seq=0 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f199affd640 1 -- 172.21.15.116:0/4024365211 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe1aa2c9f0 con 0x55fe1ab3d200 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f199affd640 10 client.19197 handle_command_reply: tid=0 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 client.19197 shutdown 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 msgr2=0x55fe1aacdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00f830 tx=0x7f19ac01cd10 comp rx=0 tx=0).stop 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 shutdown_connections 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fe1ab3d200 0x55fe1ab3f620 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe1aac57d0 0x55fe1aac9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe1aabcbf0 0x55fe1aacdd80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f199affd640 0 client.19197 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.014+0000 7f19c47ee740 1 --2- 172.21.15.116:0/4024365211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe1aabc2d0 0x55fe1aacd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 >> 172.21.15.116:0/4024365211 conn(0x55fe1aab6a00 msgr2=0x55fe1aac3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 shutdown_connections 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c47ee740 1 -- 172.21.15.116:0/4024365211 wait complete. 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c47ee740 20 client.19197 trim_cache size 0 max 16384 2024-03-19T02:40:12.022 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 msgr2=0x7f19a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 0x7f19a40a4e30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f19b40eea70 tx=0x7f19b40a8040 comp rx=0 tx=0).stop 2024-03-19T02:40:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 msgr2=0x7f19bc103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 0x7f19bc103740 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f19b402c630 tx=0x7f19b4035d60 comp rx=0 tx=0).stop 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 shutdown_connections 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f19a40a2970 0x7f19a40a4e30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19bc1066b0 0x7f19bc103740 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19bc101060 0x7f19bc103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 --2- 172.21.15.116:0/2663279013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19bc1006a0 0x7f19bc102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 >> 172.21.15.116:0/2663279013 conn(0x7f19bc0778d0 msgr2=0x7f19bc068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 shutdown_connections 2024-03-19T02:40:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.015+0000 7f19c1b22640 1 -- 172.21.15.116:0/2663279013 wait complete. 2024-03-19T02:40:12.030 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:40:12.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 -- 172.21.15.116:0/3316174357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81066b0 msgr2=0x7f6cb8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/3316174357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81066b0 0x7f6cb8106a90 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009930 tx=0x7f6ca802bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:12.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 -- 172.21.15.116:0/3316174357 shutdown_connections 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/3316174357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 0x7f6cb810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/3316174357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81006a0 0x7f6cb8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/3316174357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81066b0 0x7f6cb8106a90 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.141+0000 7f6cbca6c640 1 -- 172.21.15.116:0/3316174357 >> 172.21.15.116:0/3316174357 conn(0x7f6cb80778d0 msgr2=0x7f6cb8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 -- 172.21.15.116:0/3316174357 shutdown_connections 2024-03-19T02:40:12.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 -- 172.21.15.116:0/3316174357 wait complete. 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 Processor -- start 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 -- start start 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 0x7f6cb810e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81066b0 0x7f6cb810ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8111a70 con 0x7f6cb81006a0 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.142+0000 7f6cbca6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81118f0 con 0x7f6cb8101060 2024-03-19T02:40:12.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cbca6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb8111bf0 con 0x7f6cb81066b0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51084/0 (socket says 172.21.15.116:51084) 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 -- 172.21.15.116:0/304539618 learned_addr learned my addr 172.21.15.116:0/304539618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81066b0 0x7f6cb810ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6caffff640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 0x7f6cb810e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 -- 172.21.15.116:0/304539618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81066b0 msgr2=0x7f6cb810ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81066b0 0x7f6cb810ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 -- 172.21.15.116:0/304539618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 msgr2=0x7f6cb810e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 0x7f6cb810e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 -- 172.21.15.116:0/304539618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb81a6470 con 0x7f6cb81006a0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cb659c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009230 tx=0x7f6ca802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6caf7fe640 1 -- 172.21.15.116:0/304539618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8043020 con 0x7f6cb81006a0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6caf7fe640 1 -- 172.21.15.116:0/304539618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ca802ca70 con 0x7f6cb81006a0 2024-03-19T02:40:12.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb81a6700 con 0x7f6cb81006a0 2024-03-19T02:40:12.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6caf7fe640 1 -- 172.21.15.116:0/304539618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8003a50 con 0x7f6cb81006a0 2024-03-19T02:40:12.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.143+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb81a6c50 con 0x7f6cb81006a0 2024-03-19T02:40:12.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6caf7fe640 1 -- 172.21.15.116:0/304539618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6ca8046020 con 0x7f6cb81006a0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6cbf658740 1 Processor -- start 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6cbf658740 1 -- start start 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6caf7fe640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 0x7f6ca40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c286f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.146+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c28787d0 0x55a3c287abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cbf658740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3c27d24b0 con 0x55a3c286fbf0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cbf658740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a3c277ad70 con 0x55a3c286f2d0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 0x7f6ca40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cbf658740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3c27ce550 con 0x55a3c28787d0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caf7fe640 1 -- 172.21.15.116:0/304539618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f6ca80f10a0 con 0x7f6cb81006a0 2024-03-19T02:40:12.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51086/0 (socket says 172.21.15.116:51086) 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 -- 172.21.15.116:0/955523154 learned_addr learned my addr 172.21.15.116:0/955523154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cb659c640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c286f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c28787d0 0x55a3c287abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 -- 172.21.15.116:0/955523154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c28787d0 msgr2=0x55a3c287abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c28787d0 0x55a3c287abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 -- 172.21.15.116:0/955523154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 msgr2=0x55a3c286f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c286f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 -- 172.21.15.116:0/955523154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c279d210 con 0x55a3c286fbf0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cb659c640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c286f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 0x7f6ca40a4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0005fd0 tx=0x7f6cb0011860 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6caffff640 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009800 tx=0x7f6ca0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c8a86e0832535 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.147+0000 7f6cacff9640 1 -- 172.21.15.116:0/955523154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0025070 con 0x55a3c286fbf0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cacff9640 1 -- 172.21.15.116:0/955523154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ca000ee40 con 0x55a3c286fbf0 2024-03-19T02:40:12.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cacff9640 1 -- 172.21.15.116:0/955523154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0020d90 con 0x55a3c286fbf0 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 -- 172.21.15.116:0/955523154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 msgr2=0x55a3c2878290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009800 tx=0x7f6ca0009c00 comp rx=0 tx=0).stop 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 -- 172.21.15.116:0/955523154 shutdown_connections 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c28787d0 0x55a3c287abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286fbf0 0x55a3c2878290 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 --2- 172.21.15.116:0/955523154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c286f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 -- 172.21.15.116:0/955523154 >> 172.21.15.116:0/955523154 conn(0x55a3c2869a00 msgr2=0x55a3c2869de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 -- 172.21.15.116:0/955523154 shutdown_connections 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.148+0000 7f6cbf658740 1 -- 172.21.15.116:0/955523154 wait complete. 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 -- start start 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 Processor -- start 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c28757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2875d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3c287b5c0 con 0x55a3c28787d0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a3c278e310 con 0x55a3c286f2d0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3c287b740 con 0x55a3c286fbf0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cbf658740 10 client.0 _ll_register_callbacks cb 0x55a3c27822d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51096/0 (socket says 172.21.15.116:51096) 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/282981889 learned_addr learned my addr 172.21.15.116:0/282981889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb659c640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c28757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6caffff640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2875d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/282981889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 msgr2=0x55a3c2875d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2875d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 msgr2=0x55a3c28757e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c28757e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/282981889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c24e63f0 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6cb659c640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c28757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.149+0000 7f6caffff640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2875d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0004280 tx=0x7f6cb0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c09ab44ed722c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb000fe10 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f6cb001e460 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a3c28303e0 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6cb001e660 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6cbf658740 10 client.19209 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3c25b07d0 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a3c28303e0 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6cb0025070 con 0x55a3c28787d0 2024-03-19T02:40:12.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.150+0000 7f6cbf658740 10 client.19209 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cbf658740 10 client.19209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cbf658740 20 client.19209 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c2888c10 0x55a3c288b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cbf658740 4 client.19209 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3c27df9f0 con 0x55a3c2888c10 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cb659c640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c2888c10 0x55a3c288b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6c9dffb640 10 client.19209 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:12.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.151+0000 7f6cb659c640 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c2888c10 0x55a3c288b040 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c021d624da654c4 server_cookie=3190327e1d85c88f in_seq=0 out_seq=0 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6c9dffb640 1 -- 172.21.15.116:0/282981889 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3c27df9f0 con 0x55a3c2888c10 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6c9dffb640 10 client.19209 handle_command_reply: tid=0 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 client.19209 shutdown 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 msgr2=0x55a3c2876260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0004280 tx=0x7f6cb0004ab0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 shutdown_connections 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c2888c10 0x55a3c288b040 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2876260 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2875d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 --2- 172.21.15.116:0/282981889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c28757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 >> 172.21.15.116:0/282981889 conn(0x55a3c2869a00 msgr2=0x55a3c286b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6c9dffb640 0 client.19209 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:12.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 shutdown_connections 2024-03-19T02:40:12.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 1 -- 172.21.15.116:0/282981889 wait complete. 2024-03-19T02:40:12.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.153+0000 7f6cbf658740 20 client.19209 trim_cache size 0 max 16384 2024-03-19T02:40:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 Processor -- start 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 -- start start 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c2877f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2884390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81aca10 con 0x55a3c28787d0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81ae550 con 0x55a3c286f2d0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cbf658740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81ad0e0 con 0x55a3c286fbf0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cb659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c2877f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51110/0 (socket says 172.21.15.116:51110) 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/4036350569 learned_addr learned my addr 172.21.15.116:0/4036350569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.168+0000 7f6caffff640 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2884390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/4036350569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 msgr2=0x55a3c2884390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2884390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/4036350569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 msgr2=0x55a3c2877f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c2877f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 -- 172.21.15.116:0/4036350569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c2874790 con 0x55a3c28787d0 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0028ff0 tx=0x7f6cb000f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c62ae5df3ede14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6c9ffff640 1 -- 172.21.15.116:0/4036350569 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb000fc10 con 0x55a3c28787d0 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6c9ffff640 1 -- 172.21.15.116:0/4036350569 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cb000fdb0 con 0x55a3c28787d0 2024-03-19T02:40:12.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.169+0000 7f6c9ffff640 1 -- 172.21.15.116:0/4036350569 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb00053e0 con 0x55a3c28787d0 2024-03-19T02:40:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- 172.21.15.116:0/4036350569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 msgr2=0x55a3c2890910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0028ff0 tx=0x7f6cb000f910 comp rx=0 tx=0).stop 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- 172.21.15.116:0/4036350569 shutdown_connections 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c28787d0 0x55a3c2890910 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2884390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- 172.21.15.116:0/4036350569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c286f2d0 0x55a3c2877f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- 172.21.15.116:0/4036350569 >> 172.21.15.116:0/4036350569 conn(0x55a3c2869a00 msgr2=0x55a3c2869de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- 172.21.15.116:0/4036350569 shutdown_connections 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- 172.21.15.116:0/4036350569 wait complete. 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- start start 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 Processor -- start 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2872070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c28787d0 0x55a3c28725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81ade80 con 0x55a3c286f2d0 2024-03-19T02:40:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81ad7b0 con 0x55a3c28787d0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb81aaa10 con 0x55a3c286fbf0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cbf658740 10 client.0 _ll_register_callbacks cb 0x7f6cb0003a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cb659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cb659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51114/0 (socket says 172.21.15.116:51114) 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.170+0000 7f6cb659c640 1 -- 172.21.15.116:0/1819500328 learned_addr learned my addr 172.21.15.116:0/1819500328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6caffff640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2872070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb6d9d640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c28787d0 0x55a3c28725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 -- 172.21.15.116:0/1819500328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 msgr2=0x55a3c2872070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2872070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 -- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c28787d0 msgr2=0x55a3c28725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c28787d0 0x55a3c28725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 -- 172.21.15.116:0/1819500328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c2872de0 con 0x55a3c286f2d0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cb659c640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006010 tx=0x7f6ca804a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd0e5e3da09e4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:12.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80c16a0 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f6ca80b9bc0 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cb000fdb0 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6ca80c5cf0 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cbf658740 10 client.19215 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3c28eef80 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cb000fdb0 con 0x55a3c286f2d0 2024-03-19T02:40:12.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.171+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f6ca80bd7b0 con 0x55a3c286f2d0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 10 client.19215 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 10 client.19215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 20 client.19215 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c28ef6a0 0x55a3c28f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 4 client.19215 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3c27d9d40 con 0x55a3c28ef6a0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6caffff640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c28ef6a0 0x55a3c28f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6caffff640 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c28ef6a0 0x55a3c28f1a90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de051530c753a764 server_cookie=364669c5b87c239 in_seq=0 out_seq=0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.172+0000 7f6c9dffb640 10 client.19215 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6c9dffb640 1 -- 172.21.15.116:0/1819500328 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3c27d9d40 con 0x55a3c28ef6a0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6c9dffb640 10 client.19215 handle_command_reply: tid=0 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 client.19215 shutdown 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 msgr2=0x55a3c2880c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006010 tx=0x7f6ca804a040 comp rx=0 tx=0).stop 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 shutdown_connections 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a3c28ef6a0 0x55a3c28f1a90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3c28787d0 0x55a3c28725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3c286fbf0 0x55a3c2872070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 --2- 172.21.15.116:0/1819500328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3c286f2d0 0x55a3c2880c90 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 >> 172.21.15.116:0/1819500328 conn(0x55a3c2869a00 msgr2=0x55a3c28757a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6c9dffb640 0 client.19215 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 shutdown_connections 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 1 -- 172.21.15.116:0/1819500328 wait complete. 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.173+0000 7f6cbf658740 20 client.19215 trim_cache size 0 max 16384 2024-03-19T02:40:12.180 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 msgr2=0x7f6ca40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 0x7f6ca40a4e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0005fd0 tx=0x7f6cb0011860 comp rx=0 tx=0).stop 2024-03-19T02:40:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 msgr2=0x7f6cb810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009230 tx=0x7f6ca802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 shutdown_connections 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ca40a29a0 0x7f6ca40a4e60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb81066b0 0x7f6cb810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cb8101060 0x7f6cb810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 --2- 172.21.15.116:0/304539618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cb81006a0 0x7f6cb810e000 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 >> 172.21.15.116:0/304539618 conn(0x7f6cb80778d0 msgr2=0x7f6cb8102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 shutdown_connections 2024-03-19T02:40:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:12.174+0000 7f6cbca6c640 1 -- 172.21.15.116:0/304539618 wait complete. 2024-03-19T02:40:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:11 smithi154 ceph-mon[35693]: pgmap v388: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.7 MiB/s wr, 143 op/s 2024-03-19T02:40:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:11 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:11 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1759042926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:11 smithi162 ceph-mon[35761]: pgmap v388: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.7 MiB/s wr, 143 op/s 2024-03-19T02:40:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:11 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:11 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1759042926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:12.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 403251 3.09 MB/sec execute 575 sec latency 22.404 ms 2024-03-19T02:40:13.186 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:40:13.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:40:13.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:40:13.211 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:12 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:40:13.211 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:12 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:40:13.211 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:12 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:40:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:12 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:40:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:12 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:40:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:12 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:40:13.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- 172.21.15.116:0/3559188272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100690 msgr2=0x7f8fe0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/3559188272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100690 0x7f8fe0100b10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009930 tx=0x7f8fd402bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- 172.21.15.116:0/3559188272 shutdown_connections 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/3559188272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0101050 0x7f8fe010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/3559188272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0100690 0x7f8fe0100b10 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/3559188272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 0x7f8fe0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- 172.21.15.116:0/3559188272 >> 172.21.15.116:0/3559188272 conn(0x7f8fe0077820 msgr2=0x7f8fe0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- 172.21.15.116:0/3559188272 shutdown_connections 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- 172.21.15.116:0/3559188272 wait complete. 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 Processor -- start 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.297+0000 7f8fe65b9640 1 -- start start 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 0x7f8fe0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 0x7f8fe019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111a60 con 0x7f8fe0101050 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01118e0 con 0x7f8fe0100690 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe65b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111be0 con 0x7f8fe01066a0 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51132/0 (socket says 172.21.15.116:51132) 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/2040280610 learned_addr learned my addr 172.21.15.116:0/2040280610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 0x7f8fe0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 0x7f8fe019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 -- 172.21.15.116:0/2040280610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 msgr2=0x7f8fe019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 0x7f8fe019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 -- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 msgr2=0x7f8fe0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 -- 172.21.15.116:0/2040280610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe019d030 con 0x7f8fe0100690 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.298+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 0x7f8fe0198040 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e9f0 tx=0x7f8fd000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.299+0000 7f8fdd7fa640 1 -- 172.21.15.116:0/2040280610 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000ccc0 con 0x7f8fe0100690 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.299+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe019d320 con 0x7f8fe0100690 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.299+0000 7f8fdd7fa640 1 -- 172.21.15.116:0/2040280610 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd0005020 con 0x7f8fe0100690 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.299+0000 7f8fdd7fa640 1 -- 172.21.15.116:0/2040280610 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0004a00 con 0x7f8fe0100690 2024-03-19T02:40:13.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.299+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01a4bb0 con 0x7f8fe0100690 2024-03-19T02:40:13.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.301+0000 7f8fdd7fa640 1 -- 172.21.15.116:0/2040280610 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8fd0004ba0 con 0x7f8fe0100690 2024-03-19T02:40:13.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.301+0000 7f8fe91a3740 1 Processor -- start 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.301+0000 7f8fe91a3740 1 -- start start 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdd7fa640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 0x7f8fc80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ed340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5ed880 0x560dbc5efc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560dbc548160 con 0x560dbc5e4380 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560dbc4f0e10 con 0x560dbc5e4ca0 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560dbc544200 con 0x560dbc5ed880 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51146/0 (socket says 172.21.15.116:51146) 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 0x7f8fc80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe4b56640 1 -- 172.21.15.116:0/397109245 learned_addr learned my addr 172.21.15.116:0/397109245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdd7fa640 1 -- 172.21.15.116:0/2040280610 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f8fd00cf160 con 0x7f8fe0100690 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/397109245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5ed880 msgr2=0x560dbc5efc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5ed880 0x560dbc5efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/397109245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 msgr2=0x560dbc5e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/397109245 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dbc3247e0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdffff640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 0x7f8fc80a4e30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00cc580 tx=0x7f8fd0066150 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ed340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009900 tx=0x7f8fd4009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b1677f053e127a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.302+0000 7f8fc67fc640 1 -- 172.21.15.116:0/397109245 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4045070 con 0x560dbc5e4ca0 2024-03-19T02:40:13.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fc67fc640 1 -- 172.21.15.116:0/397109245 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd402c7e0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fc67fc640 1 -- 172.21.15.116:0/397109245 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4003af0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- 172.21.15.116:0/397109245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 msgr2=0x560dbc5ed340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ed340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009900 tx=0x7f8fd4009670 comp rx=0 tx=0).stop 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- 172.21.15.116:0/397109245 shutdown_connections 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5ed880 0x560dbc5efc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ed340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/397109245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4380 0x560dbc5e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- 172.21.15.116:0/397109245 >> 172.21.15.116:0/397109245 conn(0x560dbc5deab0 msgr2=0x560dbc5dee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- 172.21.15.116:0/397109245 shutdown_connections 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- 172.21.15.116:0/397109245 wait complete. 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.303+0000 7f8fe91a3740 1 -- start start 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 Processor -- start 2024-03-19T02:40:13.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5ed880 0x560dbc5eb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560dbc5f0610 con 0x560dbc5ed880 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560dbc5041a0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560dbc5f0790 con 0x560dbc5e4380 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe91a3740 10 client.0 _ll_register_callbacks cb 0x560dbc25bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5ed880 0x560dbc5eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45524/0 (socket says 172.21.15.116:45524) 2024-03-19T02:40:13.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe4b56640 1 -- 172.21.15.116:0/2604637500 learned_addr learned my addr 172.21.15.116:0/2604637500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/2604637500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 msgr2=0x560dbc5ea800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/2604637500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5ed880 msgr2=0x560dbc5eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5ed880 0x560dbc5eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/2604637500 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dbc384f70 con 0x560dbc5e4ca0 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.304+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ead40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005ec0 tx=0x7f8fd4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed60b6f89b99d2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:13.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4045040 con 0x560dbc5e4ca0 2024-03-19T02:40:13.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd402c7e0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f8fd403fc00 con 0x560dbc5e4ca0 2024-03-19T02:40:13.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8fd405a050 con 0x560dbc5e4ca0 2024-03-19T02:40:13.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 10 client.29006 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:13.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560dbc21c170 con 0x560dbc5e4ca0 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd402c7e0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8fd4055440 con 0x560dbc5e4ca0 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 10 client.29006 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 10 client.29006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 20 client.29006 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc5fd970 0x560dbc5ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.305+0000 7f8fe91a3740 4 client.29006 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:13.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.306+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dbc5556a0 con 0x560dbc5fd970 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.306+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc5fd970 0x560dbc5ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.306+0000 7f8fbf7fe640 10 client.29006 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.306+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc5fd970 0x560dbc5ffd60 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f6089d0b86f9451 server_cookie=217bcbbc070abdfd in_seq=0 out_seq=0 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2604637500 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560dbc5556a0 con 0x560dbc5fd970 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fbf7fe640 10 client.29006 handle_command_reply: tid=0 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 client.29006 shutdown 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 msgr2=0x560dbc5ead40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ead40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005ec0 tx=0x7f8fd4000bc0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 shutdown_connections 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc5fd970 0x560dbc5ffd60 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5ed880 0x560dbc5eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5e4ca0 0x560dbc5ead40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fbf7fe640 0 client.29006 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2604637500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5ea800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 >> 172.21.15.116:0/2604637500 conn(0x560dbc5deab0 msgr2=0x560dbc5e0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 shutdown_connections 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2604637500 wait complete. 2024-03-19T02:40:13.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.307+0000 7f8fe91a3740 20 client.29006 trim_cache size 0 max 16384 2024-03-19T02:40:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 Processor -- start 2024-03-19T02:40:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 -- start start 2024-03-19T02:40:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5e0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01aecc0 con 0x560dbc5e4ca0 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01b0800 con 0x560dbc5ed880 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01af390 con 0x560dbc5e4380 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.327+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51162/0 (socket says 172.21.15.116:51162) 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fdf7fe640 1 -- 172.21.15.116:0/213736051 learned_addr learned my addr 172.21.15.116:0/213736051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5e0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 -- 172.21.15.116:0/213736051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 msgr2=0x560dbc5e0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5e0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 -- 172.21.15.116:0/213736051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 msgr2=0x560dbc5eb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 -- 172.21.15.116:0/213736051 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dbc5de8e0 con 0x560dbc5e4380 2024-03-19T02:40:13.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f93b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x560dbc5f06d0 tx=0x7f8fd8009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7addac38f36109c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fc5ffb640 1 -- 172.21.15.116:0/213736051 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8019040 con 0x560dbc5e4380 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fc5ffb640 1 -- 172.21.15.116:0/213736051 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd8023040 con 0x560dbc5e4380 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.328+0000 7f8fc5ffb640 1 -- 172.21.15.116:0/213736051 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8005560 con 0x560dbc5e4380 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- 172.21.15.116:0/213736051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 msgr2=0x560dbc5f93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f93b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x560dbc5f06d0 tx=0x7f8fd8009520 comp rx=0 tx=0).stop 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- 172.21.15.116:0/213736051 shutdown_connections 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5e0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/213736051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f93b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- 172.21.15.116:0/213736051 >> 172.21.15.116:0/213736051 conn(0x560dbc5deab0 msgr2=0x560dbc5dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- 172.21.15.116:0/213736051 shutdown_connections 2024-03-19T02:40:13.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- 172.21.15.116:0/213736051 wait complete. 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.329+0000 7f8fe91a3740 1 -- start start 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 Processor -- start 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01b0130 con 0x560dbc5e4ca0 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01afa60 con 0x560dbc5ed880 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01accc0 con 0x560dbc5e4380 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe91a3740 10 client.0 _ll_register_callbacks cb 0x7f8fd8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:13.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45540/0 (socket says 172.21.15.116:45540) 2024-03-19T02:40:13.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5f11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51170/0 (socket says 172.21.15.116:51170) 2024-03-19T02:40:13.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 -- 172.21.15.116:0/2641518886 learned_addr learned my addr 172.21.15.116:0/2641518886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 -- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 msgr2=0x560dbc5f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 -- 172.21.15.116:0/2641518886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 msgr2=0x560dbc5f11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5f11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 -- 172.21.15.116:0/2641518886 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dbc662960 con 0x560dbc5e4380 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fe4b56640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8010010 tx=0x7f8fd8026040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6acf299162292e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.330+0000 7f8fbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8004b00 con 0x560dbc5e4380 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd8023040 con 0x560dbc5e4380 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f8fd8004ca0 con 0x560dbc5e4380 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8fd8003970 con 0x560dbc5e4380 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 10 client.28963 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:13.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560dbc663b00 con 0x560dbc5e4380 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fd8023040 con 0x560dbc5e4380 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8fd8024ce0 con 0x560dbc5e4380 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 10 client.28963 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 10 client.28963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 20 client.28963 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc664130 0x560dbc666550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 4 client.28963 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560dbc54f9f0 con 0x560dbc664130 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.331+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc664130 0x560dbc666550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.332+0000 7f8fbf7fe640 10 client.28963 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.332+0000 7f8fdf7fe640 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc664130 0x560dbc666550 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a427eac7e0c3622 server_cookie=55baff35f8a7c625 in_seq=0 out_seq=0 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fbf7fe640 1 -- 172.21.15.116:0/2641518886 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560dbc54f9f0 con 0x560dbc664130 2024-03-19T02:40:13.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fbf7fe640 10 client.28963 handle_command_reply: tid=0 2024-03-19T02:40:13.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 client.28963 shutdown 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 msgr2=0x560dbc5f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8010010 tx=0x7f8fd8026040 comp rx=0 tx=0).stop 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 shutdown_connections 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560dbc664130 0x560dbc666550 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dbc5ed880 0x560dbc5f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fbf7fe640 0 client.28963 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560dbc5e4ca0 0x560dbc5f11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 --2- 172.21.15.116:0/2641518886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560dbc5e4380 0x560dbc5f5df0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 >> 172.21.15.116:0/2641518886 conn(0x560dbc5deab0 msgr2=0x560dbc5dd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 shutdown_connections 2024-03-19T02:40:13.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 1 -- 172.21.15.116:0/2641518886 wait complete. 2024-03-19T02:40:13.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.334+0000 7f8fe91a3740 20 client.28963 trim_cache size 0 max 16384 2024-03-19T02:40:13.343 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:40:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "557b9fd7-3a56-4338-a707-8922ed750a2d", 2024-03-19T02:40:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:40:13.344 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 msgr2=0x7f8fc80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 0x7f8fc80a4e30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00cc580 tx=0x7f8fd0066150 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 msgr2=0x7f8fe0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 0x7f8fe0198040 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e9f0 tx=0x7f8fd000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 shutdown_connections 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fc80a2970 0x7f8fc80a4e30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fe01066a0 0x7f8fe019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fe0101050 0x7f8fe0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 --2- 172.21.15.116:0/2040280610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fe0100690 0x7f8fe0198040 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 >> 172.21.15.116:0/2040280610 conn(0x7f8fe0077820 msgr2=0x7f8fe0102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 shutdown_connections 2024-03-19T02:40:13.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.335+0000 7f8fe65b9640 1 -- 172.21.15.116:0/2040280610 wait complete. 2024-03-19T02:40:13.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:40:13.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:40:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:12 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:40:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:12 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:40:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:12 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:40:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 -- 172.21.15.116:0/2065906301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 msgr2=0x7f02d0106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 --2- 172.21.15.116:0/2065906301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d0106a80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009930 tx=0x7f02c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 -- 172.21.15.116:0/2065906301 shutdown_connections 2024-03-19T02:40:13.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 --2- 172.21.15.116:0/2065906301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0101050 0x7f02d010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 --2- 172.21.15.116:0/2065906301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0100690 0x7f02d0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 --2- 172.21.15.116:0/2065906301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d0106a80 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 -- 172.21.15.116:0/2065906301 >> 172.21.15.116:0/2065906301 conn(0x7f02d0077820 msgr2=0x7f02d0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 -- 172.21.15.116:0/2065906301 shutdown_connections 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.502+0000 7f02d5b87640 1 -- 172.21.15.116:0/2065906301 wait complete. 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 Processor -- start 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 -- start start 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0100690 0x7f02d0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0101050 0x7f02d0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110960 con 0x7f02d01066a0 2024-03-19T02:40:13.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01107e0 con 0x7f02d0101050 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.503+0000 7f02d5b87640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02d0110ae0 con 0x7f02d0100690 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51196/0 (socket says 172.21.15.116:51196) 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0100690 0x7f02d0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 -- 172.21.15.116:0/14587143 learned_addr learned my addr 172.21.15.116:0/14587143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02ceffd640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0101050 0x7f02d0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 -- 172.21.15.116:0/14587143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0100690 msgr2=0x7f02d0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0100690 0x7f02d0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 -- 172.21.15.116:0/14587143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0101050 msgr2=0x7f02d0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0101050 0x7f02d0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 -- 172.21.15.116:0/14587143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d019cfe0 con 0x7f02d01066a0 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02cffff640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f02c800eb10 tx=0x7f02c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02ccff9640 1 -- 172.21.15.116:0/14587143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8004b20 con 0x7f02d01066a0 2024-03-19T02:40:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d019d2d0 con 0x7f02d01066a0 2024-03-19T02:40:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02ccff9640 1 -- 172.21.15.116:0/14587143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c8004cc0 con 0x7f02d01066a0 2024-03-19T02:40:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02ccff9640 1 -- 172.21.15.116:0/14587143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c8010690 con 0x7f02d01066a0 2024-03-19T02:40:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.504+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d01a4b60 con 0x7f02d01066a0 2024-03-19T02:40:13.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.507+0000 7f02ccff9640 1 -- 172.21.15.116:0/14587143 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f02c8005020 con 0x7f02d01066a0 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.507+0000 7f02d8775740 1 Processor -- start 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.507+0000 7f02d8775740 1 -- start start 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3ede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3f6f30 0x55e13b3f9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e13b351500 con 0x55e13b3ee350 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e13b2f9be0 con 0x55e13b3eda30 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02d8775740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e13b34d5a0 con 0x55e13b3f6f30 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51200/0 (socket says 172.21.15.116:51200) 2024-03-19T02:40:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1042836456 learned_addr learned my addr 172.21.15.116:0/1042836456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02ceffd640 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02ccff9640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 0x7f02b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02ceffd640 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3f6f30 0x55e13b3f9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cffff640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 0x7f02b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1042836456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3f6f30 msgr2=0x55e13b3f9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3f6f30 0x55e13b3f9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1042836456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 msgr2=0x55e13b3ede10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3ede10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1042836456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13b31b000 con 0x55e13b3ee350 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02ccff9640 1 -- 172.21.15.116:0/14587143 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02c80cf870 con 0x7f02d01066a0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f02c0036040 tx=0x7f02c0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8f43525ce00d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.508+0000 7f02cffff640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 0x7f02b80a4e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f02c8097f50 tx=0x7f02c8097ec0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02b5ffb640 1 -- 172.21.15.116:0/1042836456 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0045020 con 0x55e13b3ee350 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02b5ffb640 1 -- 172.21.15.116:0/1042836456 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c002c7e0 con 0x55e13b3ee350 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02b5ffb640 1 -- 172.21.15.116:0/1042836456 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0005440 con 0x55e13b3ee350 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 -- 172.21.15.116:0/1042836456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 msgr2=0x55e13b3f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f02c0036040 tx=0x7f02c0009560 comp rx=0 tx=0).stop 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 -- 172.21.15.116:0/1042836456 shutdown_connections 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3f6f30 0x55e13b3f9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f69f0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 --2- 172.21.15.116:0/1042836456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3ede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.509+0000 7f02d8775740 1 -- 172.21.15.116:0/1042836456 >> 172.21.15.116:0/1042836456 conn(0x55e13b3e8160 msgr2=0x55e13b3e8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- 172.21.15.116:0/1042836456 shutdown_connections 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- 172.21.15.116:0/1042836456 wait complete. 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- start start 2024-03-19T02:40:13.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 Processor -- start 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3eda30 0x55e13b3f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3f6f30 0x55e13b3f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e13b3f9cc0 con 0x55e13b3ee350 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e13b30d160 con 0x55e13b3f6f30 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.510+0000 7f02d8775740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e13b3f9e40 con 0x55e13b3eda30 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02d8775740 10 client.0 _ll_register_callbacks cb 0x55e13b349a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02ceffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3eda30 0x55e13b3f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51208/0 (socket says 172.21.15.116:51208) 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1619486312 learned_addr learned my addr 172.21.15.116:0/1619486312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02ceffd640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3f6f30 0x55e13b3f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1619486312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3eda30 msgr2=0x55e13b3f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3eda30 0x55e13b3f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3f6f30 msgr2=0x55e13b3f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3f6f30 0x55e13b3f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 -- 172.21.15.116:0/1619486312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13b12f7d0 con 0x55e13b3ee350 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f02c002c170 tx=0x7f02c00053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ac4f04483f7f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0050020 con 0x55e13b3ee350 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02c0004240 con 0x55e13b3ee350 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c002c7e0 con 0x55e13b3ee350 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f02c0058050 con 0x55e13b3ee350 2024-03-19T02:40:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02d8775740 10 client.19245 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.511+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e13b0653f0 con 0x55e13b3ee350 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c002c7e0 con 0x55e13b3ee350 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f02c00035f0 con 0x55e13b3ee350 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 10 client.19245 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 10 client.19245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 20 client.19245 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b407070 0x55e13b409460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 4 client.19245 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e13b35ea40 con 0x55e13b407070 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02ceffd640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b407070 0x55e13b409460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02ceffd640 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b407070 0x55e13b409460 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb007c27cdaf2892 server_cookie=9ccb467f57d80bbb in_seq=0 out_seq=0 2024-03-19T02:40:13.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.512+0000 7f02aeffd640 10 client.19245 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02aeffd640 1 -- 172.21.15.116:0/1619486312 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e13b35ea40 con 0x55e13b407070 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02aeffd640 10 client.19245 handle_command_reply: tid=0 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 client.19245 shutdown 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 msgr2=0x55e13b3f4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f02c002c170 tx=0x7f02c00053f0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 shutdown_connections 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b407070 0x55e13b409460 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3f6f30 0x55e13b3f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3ee350 0x55e13b3f4410 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 --2- 172.21.15.116:0/1619486312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3eda30 0x55e13b3f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 >> 172.21.15.116:0/1619486312 conn(0x55e13b3e8160 msgr2=0x55e13b3e9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02aeffd640 0 client.19245 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 shutdown_connections 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 1 -- 172.21.15.116:0/1619486312 wait complete. 2024-03-19T02:40:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.514+0000 7f02d8775740 20 client.19245 trim_cache size 0 max 16384 2024-03-19T02:40:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 Processor -- start 2024-03-19T02:40:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 -- start start 2024-03-19T02:40:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01ac9f0 con 0x55e13b3f6f30 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01ae530 con 0x55e13b3eda30 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02d8775740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01ad0c0 con 0x55e13b3ee350 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51214/0 (socket says 172.21.15.116:51214) 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 -- 172.21.15.116:0/3894678033 learned_addr learned my addr 172.21.15.116:0/3894678033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 -- 172.21.15.116:0/3894678033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 msgr2=0x55e13b3f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 -- 172.21.15.116:0/3894678033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 msgr2=0x55e13b3f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 -- 172.21.15.116:0/3894678033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13b0250f0 con 0x55e13b3f6f30 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02ceffd640 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009e90 tx=0x7f02c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb82ee32d3f7b91 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.532+0000 7f02b57fa640 1 -- 172.21.15.116:0/3894678033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4011230 con 0x55e13b3f6f30 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02b57fa640 1 -- 172.21.15.116:0/3894678033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c4004030 con 0x55e13b3f6f30 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02b57fa640 1 -- 172.21.15.116:0/3894678033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005290 con 0x55e13b3f6f30 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 -- 172.21.15.116:0/3894678033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 msgr2=0x55e13b3ef3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009e90 tx=0x7f02c4005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 -- 172.21.15.116:0/3894678033 shutdown_connections 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b3ef3e0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 --2- 172.21.15.116:0/3894678033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 -- 172.21.15.116:0/3894678033 >> 172.21.15.116:0/3894678033 conn(0x55e13b3e8160 msgr2=0x55e13b3f4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 -- 172.21.15.116:0/3894678033 shutdown_connections 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.533+0000 7f02d8775740 1 -- 172.21.15.116:0/3894678033 wait complete. 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 -- start start 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 Processor -- start 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b46c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01ade60 con 0x55e13b3f6f30 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01ad790 con 0x55e13b3eda30 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02d01aa9f0 con 0x55e13b3ee350 2024-03-19T02:40:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02d8775740 10 client.0 _ll_register_callbacks cb 0x7f02c4009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02ceffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02ceffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38060/0 (socket says 172.21.15.116:38060) 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02ceffd640 1 -- 172.21.15.116:0/4136140866 learned_addr learned my addr 172.21.15.116:0/4136140866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.534+0000 7f02ceffd640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b46c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 -- 172.21.15.116:0/4136140866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 msgr2=0x55e13b3f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 -- 172.21.15.116:0/4136140866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 msgr2=0x55e13b46c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b46c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 -- 172.21.15.116:0/4136140866 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13b46c5a0 con 0x55e13b3eda30 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02ceffd640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f02c401e6b0 tx=0x7f02c402f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ff2ef066ccf3d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c40035f0 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f02c4003790 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02c4004030 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f02c4042050 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02d8775740 10 client.29030 fetch_fsmap learned FSMap version 30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e13b46d820 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.535+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02c4004030 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f02c4029070 con 0x55e13b3eda30 2024-03-19T02:40:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 10 client.29030 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 10 client.29030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 20 client.29030 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b46de30 0x55e13b470250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 4 client.29030 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e13b358d90 con 0x55e13b46de30 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b46de30 0x55e13b470250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02cf7fe640 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b46de30 0x55e13b470250 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec41508b371c2da7 server_cookie=b9cd3c423b2c3e80 in_seq=0 out_seq=0 2024-03-19T02:40:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.536+0000 7f02aeffd640 10 client.29030 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.537+0000 7f02aeffd640 1 -- 172.21.15.116:0/4136140866 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e13b358d90 con 0x55e13b46de30 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.537+0000 7f02aeffd640 10 client.29030 handle_command_reply: tid=0 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.537+0000 7f02d8775740 1 client.29030 shutdown 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.537+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 msgr2=0x55e13b3f1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f02c401e6b0 tx=0x7f02c402f040 comp rx=0 tx=0).stop 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 shutdown_connections 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e13b46de30 0x55e13b470250 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e13b3f6f30 0x55e13b46c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02aeffd640 0 client.29030 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e13b3ee350 0x55e13b3f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 --2- 172.21.15.116:0/4136140866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e13b3eda30 0x55e13b3f1140 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 >> 172.21.15.116:0/4136140866 conn(0x55e13b3e8160 msgr2=0x55e13b3e6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 shutdown_connections 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 1 -- 172.21.15.116:0/4136140866 wait complete. 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d8775740 20 client.29030 trim_cache size 0 max 16384 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:13.546 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: "557b9fd7-3a56-4338-a707-8922ed750a2d": { 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "557b9fd7-3a56-4338-a707-8922ed750a2d", 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:40:13.547 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 msgr2=0x7f02b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 0x7f02b80a4e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f02c8097f50 tx=0x7f02c8097ec0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 msgr2=0x7f02d019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f02c800eb10 tx=0x7f02c800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 shutdown_connections 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02b80a2970 0x7f02b80a4e30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02d01066a0 0x7f02d019c8b0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02d0101050 0x7f02d0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 --2- 172.21.15.116:0/14587143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02d0100690 0x7f02d0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 >> 172.21.15.116:0/14587143 conn(0x7f02d0077820 msgr2=0x7f02d0068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 shutdown_connections 2024-03-19T02:40:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:13.538+0000 7f02d5b87640 1 -- 172.21.15.116:0/14587143 wait complete. 2024-03-19T02:40:13.550 INFO:tasks.cephfs.filesystem:scrub status for tag:557b9fd7-3a56-4338-a707-8922ed750a2d - {'path': '/', 'tag': '557b9fd7-3a56-4338-a707-8922ed750a2d', 'options': 'recursive,force'} 2024-03-19T02:40:13.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 404123 3.09 MB/sec execute 576 sec latency 19.199 ms 2024-03-19T02:40:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:13 smithi154 ceph-mon[35693]: pgmap v389: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 137 op/s 2024-03-19T02:40:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:13 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:40:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:13 smithi116 ceph-mon[32263]: pgmap v389: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 137 op/s 2024-03-19T02:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:13 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:13 smithi162 ceph-mon[35761]: pgmap v389: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 137 op/s 2024-03-19T02:40:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:13 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:40:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:14.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 404435 3.09 MB/sec execute 577 sec latency 22.651 ms 2024-03-19T02:40:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:14 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:14 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:14 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:14 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:14 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:14 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:15.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 404858 3.09 MB/sec execute 578 sec latency 18.792 ms 2024-03-19T02:40:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:15 smithi154 ceph-mon[35693]: pgmap v390: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-03-19T02:40:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:15 smithi116 ceph-mon[32263]: pgmap v390: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-03-19T02:40:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:15 smithi162 ceph-mon[35761]: pgmap v390: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-03-19T02:40:16.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 405342 3.09 MB/sec execute 579 sec latency 33.407 ms 2024-03-19T02:40:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:16 smithi154 ceph-mon[35693]: pgmap v391: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-03-19T02:40:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:16 smithi116 ceph-mon[32263]: pgmap v391: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-03-19T02:40:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:16 smithi162 ceph-mon[35761]: pgmap v391: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-03-19T02:40:17.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 405931 3.09 MB/sec execute 580 sec latency 21.802 ms 2024-03-19T02:40:18.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 406496 3.08 MB/sec execute 581 sec latency 24.671 ms 2024-03-19T02:40:19.797 INFO:tasks.workunit.client.0.smithi116.stdout: 1 407021 3.08 MB/sec execute 582 sec latency 19.134 ms 2024-03-19T02:40:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:19 smithi154 ceph-mon[35693]: pgmap v392: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:40:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:19 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:40:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:19 smithi116 ceph-mon[32263]: pgmap v392: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:40:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:19 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:40:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:19 smithi162 ceph-mon[35761]: pgmap v392: 129 pgs: 129 active+clean; 475 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-03-19T02:40:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:19 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:40:20.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 407714 3.09 MB/sec execute 583 sec latency 165.053 ms 2024-03-19T02:40:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:40:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:40:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:40:21.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 408178 3.09 MB/sec execute 584 sec latency 217.770 ms 2024-03-19T02:40:22.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:21 smithi154 ceph-mon[35693]: pgmap v393: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.3 MiB/s wr, 125 op/s 2024-03-19T02:40:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:21 smithi116 ceph-mon[32263]: pgmap v393: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.3 MiB/s wr, 125 op/s 2024-03-19T02:40:22.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:21 smithi162 ceph-mon[35761]: pgmap v393: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 2.3 MiB/s wr, 125 op/s 2024-03-19T02:40:22.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 408551 3.09 MB/sec execute 585 sec latency 40.527 ms 2024-03-19T02:40:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:22 smithi154 ceph-mon[35693]: pgmap v394: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.5 MiB/s wr, 88 op/s 2024-03-19T02:40:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:22 smithi116 ceph-mon[32263]: pgmap v394: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.5 MiB/s wr, 88 op/s 2024-03-19T02:40:23.329 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:22 smithi162 ceph-mon[35761]: pgmap v394: 129 pgs: 129 active+clean; 478 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.5 MiB/s wr, 88 op/s 2024-03-19T02:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 409124 3.08 MB/sec execute 586 sec latency 45.122 ms 2024-03-19T02:40:24.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 409644 3.08 MB/sec execute 587 sec latency 25.008 ms 2024-03-19T02:40:25.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 410285 3.08 MB/sec execute 588 sec latency 22.438 ms 2024-03-19T02:40:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:25 smithi154 ceph-mon[35693]: pgmap v395: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 129 op/s 2024-03-19T02:40:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:25 smithi116 ceph-mon[32263]: pgmap v395: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 129 op/s 2024-03-19T02:40:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:25 smithi162 ceph-mon[35761]: pgmap v395: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 129 op/s 2024-03-19T02:40:26.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 410964 3.09 MB/sec execute 589 sec latency 16.791 ms 2024-03-19T02:40:27.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 411449 3.09 MB/sec execute 590 sec latency 24.800 ms 2024-03-19T02:40:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:27 smithi154 ceph-mon[35693]: pgmap v396: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:27 smithi116 ceph-mon[32263]: pgmap v396: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:27 smithi162 ceph-mon[35761]: pgmap v396: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:28.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 411905 3.09 MB/sec execute 591 sec latency 19.791 ms 2024-03-19T02:40:29.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 412382 3.08 MB/sec execute 592 sec latency 29.672 ms 2024-03-19T02:40:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:29 smithi154 ceph-mon[35693]: pgmap v397: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:29 smithi116 ceph-mon[32263]: pgmap v397: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:29 smithi162 ceph-mon[35761]: pgmap v397: 129 pgs: 129 active+clean; 481 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-03-19T02:40:30.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 412983 3.09 MB/sec execute 593 sec latency 26.235 ms 2024-03-19T02:40:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:31 smithi154 ceph-mon[35693]: pgmap v398: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:40:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:31 smithi116 ceph-mon[32263]: pgmap v398: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:40:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:31 smithi162 ceph-mon[35761]: pgmap v398: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 120 op/s 2024-03-19T02:40:31.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 413538 3.08 MB/sec execute 594 sec latency 29.528 ms 2024-03-19T02:40:32.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 414076 3.08 MB/sec execute 595 sec latency 17.629 ms 2024-03-19T02:40:33.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 414814 3.09 MB/sec execute 596 sec latency 16.004 ms 2024-03-19T02:40:34.203 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:33 smithi162 ceph-mon[35761]: pgmap v399: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:40:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:33 smithi154 ceph-mon[35693]: pgmap v399: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:40:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:33 smithi116 ceph-mon[32263]: pgmap v399: 129 pgs: 129 active+clean; 488 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-03-19T02:40:34.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 415292 3.09 MB/sec execute 597 sec latency 18.075 ms 2024-03-19T02:40:35.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 415651 3.08 MB/sec execute 598 sec latency 41.623 ms 2024-03-19T02:40:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:35 smithi154 ceph-mon[35693]: pgmap v400: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 118 op/s 2024-03-19T02:40:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:35 smithi116 ceph-mon[32263]: pgmap v400: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 118 op/s 2024-03-19T02:40:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:35 smithi162 ceph-mon[35761]: pgmap v400: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 118 op/s 2024-03-19T02:40:36.798 INFO:tasks.workunit.client.0.smithi116.stdout: 1 416216 3.08 MB/sec execute 599 sec latency 17.841 ms 2024-03-19T02:40:37.799 INFO:tasks.workunit.client.0.smithi116.stdout: 1 cleanup 600 sec 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: 0 cleanup 600 sec 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Operation Count AvgLat MaxLat 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: ---------------------------------------- 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: NTCreateX 59328 2.698 64.251 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Close 43542 0.045 3.805 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Rename 2516 7.815 53.728 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Unlink 12009 3.299 87.109 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Qpathinfo 53779 1.907 1961.046 2024-03-19T02:40:38.216 INFO:tasks.workunit.client.0.smithi116.stdout: Qfileinfo 9371 0.068 1.373 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: Qfsinfo 9875 0.288 5.758 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: Sfileinfo 4834 0.611 34.618 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: Find 20791 11.938 217.761 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: WriteX 29307 0.156 35.237 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: ReadX 93078 0.003 0.530 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: LockX 194 0.661 1.532 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: UnlockX 194 0.724 2.140 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: Flush 4164 3.722 60.831 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:40:38.217 INFO:tasks.workunit.client.0.smithi116.stdout:Throughput 3.08163 MB/sec 1 clients 1 procs max_latency=1961.055 ms 2024-03-19T02:40:38.226 INFO:tasks.workunit.client.0.smithi116.stderr:+ dbench 10 2024-03-19T02:40:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-19T02:40:38.227 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:40:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-19T02:40:38.227 INFO:tasks.workunit.client.0.smithi116.stdout:9 of 10 processes prepared for launch 0 sec 2024-03-19T02:40:38.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:37 smithi154 ceph-mon[35693]: pgmap v401: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:37 smithi116 ceph-mon[32263]: pgmap v401: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:38.318 INFO:tasks.workunit.client.0.smithi116.stdout:10 of 10 processes prepared for launch 0 sec 2024-03-19T02:40:38.318 INFO:tasks.workunit.client.0.smithi116.stdout:releasing clients 2024-03-19T02:40:38.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:37 smithi162 ceph-mon[35761]: pgmap v401: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:38 smithi154 ceph-mon[35693]: pgmap v402: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:38 smithi116 ceph-mon[32263]: pgmap v402: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:39.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 244 93.20 MB/sec warmup 1 sec latency 125.013 ms 2024-03-19T02:40:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:38 smithi162 ceph-mon[35761]: pgmap v402: 129 pgs: 129 active+clean; 490 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-03-19T02:40:40.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 424 74.98 MB/sec warmup 2 sec latency 168.302 ms 2024-03-19T02:40:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:40 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:40 smithi154 ceph-mon[35693]: pgmap v403: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 149 op/s 2024-03-19T02:40:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:40 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:40 smithi116 ceph-mon[32263]: pgmap v403: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 149 op/s 2024-03-19T02:40:41.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 642 68.40 MB/sec warmup 3 sec latency 135.368 ms 2024-03-19T02:40:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:40 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:40 smithi162 ceph-mon[35761]: pgmap v403: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 149 op/s 2024-03-19T02:40:42.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 769 53.63 MB/sec warmup 4 sec latency 152.792 ms 2024-03-19T02:40:43.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 902 43.93 MB/sec warmup 5 sec latency 162.624 ms 2024-03-19T02:40:43.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:40:43.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 -- 172.21.15.116:0/4217368384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 msgr2=0x7f487c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 --2- 172.21.15.116:0/4217368384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c107ba0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f4878009900 tx=0x7f487802bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 -- 172.21.15.116:0/4217368384 shutdown_connections 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 --2- 172.21.15.116:0/4217368384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c100680 0x7f487c10e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 --2- 172.21.15.116:0/4217368384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c0ffcc0 0x7f487c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 --2- 172.21.15.116:0/4217368384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c107ba0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 -- 172.21.15.116:0/4217368384 >> 172.21.15.116:0/4217368384 conn(0x7f487c076c40 msgr2=0x7f487c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 -- 172.21.15.116:0/4217368384 shutdown_connections 2024-03-19T02:40:43.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.683+0000 7f488331b640 1 -- 172.21.15.116:0/4217368384 wait complete. 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 Processor -- start 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 -- start start 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c0ffcc0 0x7f487c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f487c111a60 con 0x7f487c1077c0 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1118e0 con 0x7f487c0ffcc0 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f488331b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f487c111be0 con 0x7f487c100680 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48818b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48808b6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48808b6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41040/0 (socket says 172.21.15.116:41040) 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48810b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c0ffcc0 0x7f487c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48808b6640 1 -- 172.21.15.116:0/1208225924 learned_addr learned my addr 172.21.15.116:0/1208225924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.684+0000 7f48818b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59162/0 (socket says 172.21.15.116:59162) 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 -- 172.21.15.116:0/1208225924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c0ffcc0 msgr2=0x7f487c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c0ffcc0 0x7f487c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 -- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 msgr2=0x7f487c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 -- 172.21.15.116:0/1208225924 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c19d010 con 0x7f487c100680 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48808b6640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f486c00e9f0 tx=0x7f486c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48727fc640 1 -- 172.21.15.116:0/1208225924 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c00cd80 con 0x7f487c100680 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48727fc640 1 -- 172.21.15.116:0/1208225924 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f486c0054d0 con 0x7f487c100680 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487c19d300 con 0x7f487c100680 2024-03-19T02:40:43.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f48727fc640 1 -- 172.21.15.116:0/1208225924 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c010490 con 0x7f487c100680 2024-03-19T02:40:43.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.685+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487c1a4b90 con 0x7f487c100680 2024-03-19T02:40:43.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.688+0000 7f48727fc640 1 -- 172.21.15.116:0/1208225924 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f486c005020 con 0x7f487c100680 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.689+0000 7f48727fc640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 0x7f48680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 Processor -- start 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48810b7640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 0x7f48680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48727fc640 1 -- 172.21.15.116:0/1208225924 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f486c0cf530 con 0x7f487c100680 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 -- start start 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5ea30 0x562549f5ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f6a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562549ec2500 con 0x562549f5f350 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48810b7640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 0x7f48680a4eb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4878006010 tx=0x7f4878005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562549e6abe0 con 0x562549f67f30 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48808b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f4885fe3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562549ebe5a0 con 0x562549f5ea30 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48808b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59178/0 (socket says 172.21.15.116:59178) 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48808b6640 1 -- 172.21.15.116:0/2517723855 learned_addr learned my addr 172.21.15.116:0/2517723855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48810b7640 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5ea30 0x562549f5ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.690+0000 7f48818b8640 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f6a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 -- 172.21.15.116:0/2517723855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5ea30 msgr2=0x562549f5ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5ea30 0x562549f5ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 -- 172.21.15.116:0/2517723855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 msgr2=0x562549f6a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f6a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 -- 172.21.15.116:0/2517723855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562549e8c000 con 0x562549f5f350 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48808b6640 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f486c0a3a20 tx=0x7f486c09b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c83a2689a53df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.691+0000 7f48677fe640 1 -- 172.21.15.116:0/2517723855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c09beb0 con 0x562549f5f350 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.692+0000 7f48677fe640 1 -- 172.21.15.116:0/2517723855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f486c09b7b0 con 0x562549f5f350 2024-03-19T02:40:43.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.692+0000 7f48677fe640 1 -- 172.21.15.116:0/2517723855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c097e20 con 0x562549f5f350 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 -- 172.21.15.116:0/2517723855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 msgr2=0x562549f679f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f486c0a3a20 tx=0x7f486c09b6b0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 -- 172.21.15.116:0/2517723855 shutdown_connections 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f6a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5f350 0x562549f679f0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2517723855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5ea30 0x562549f5ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 -- 172.21.15.116:0/2517723855 >> 172.21.15.116:0/2517723855 conn(0x562549f59160 msgr2=0x562549f59540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 -- 172.21.15.116:0/2517723855 shutdown_connections 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.693+0000 7f4885fe3740 1 -- 172.21.15.116:0/2517723855 wait complete. 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.694+0000 7f4885fe3740 1 -- start start 2024-03-19T02:40:43.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.694+0000 7f4885fe3740 1 Processor -- start 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.694+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.694+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f65510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.694+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f4885fe3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562549f6ad20 con 0x562549f5ea30 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f4885fe3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562549e7e160 con 0x562549f67f30 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f4885fe3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562549f6aea0 con 0x562549f5f350 2024-03-19T02:40:43.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f4885fe3740 10 client.0 _ll_register_callbacks cb 0x562549ebaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59188/0 (socket says 172.21.15.116:59188) 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f48810b7640 1 -- 172.21.15.116:0/2885654483 learned_addr learned my addr 172.21.15.116:0/2885654483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.695+0000 7f48808b6640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f65510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 -- 172.21.15.116:0/2885654483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 msgr2=0x562549f65510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f65510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 -- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 msgr2=0x562549f65a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 -- 172.21.15.116:0/2885654483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562549ca07d0 con 0x562549f5ea30 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f48810b7640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4878009900 tx=0x7f4878009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f6e13a969fcfc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.696+0000 7f4864ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878050030 con 0x562549f5ea30 2024-03-19T02:40:43.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f487803a040 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f486c09b7b0 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f48780046c0 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4885fe3740 10 client.19263 fetch_fsmap learned FSMap version 31 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562549bd63f0 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.697+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4878045040 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4878003ad0 con 0x562549f5ea30 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 10 client.19263 fetch_fsmap finished waiting for FSMap version 31 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 10 client.19263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:43.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 20 client.19263 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549f78360 0x562549f7a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 4 client.19263 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f486c09b7b0 con 0x562549f78360 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.698+0000 7f48808b6640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549f78360 0x562549f7a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.700+0000 7f4864ff9640 10 client.19263 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:43.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.700+0000 7f48808b6640 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549f78360 0x562549f7a7c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7679292bbf192c7d server_cookie=76418147b418f85c in_seq=0 out_seq=0 2024-03-19T02:40:43.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4864ff9640 1 -- 172.21.15.116:0/2885654483 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f486c09b7b0 con 0x562549f78360 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4864ff9640 10 client.19263 handle_command_reply: tid=0 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 client.19263 shutdown 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 msgr2=0x562549f64fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f4878009900 tx=0x7f4878009260 comp rx=0 tx=0).stop 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 shutdown_connections 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549f78360 0x562549f7a7c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f65510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 --2- 172.21.15.116:0/2885654483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f64fd0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 >> 172.21.15.116:0/2885654483 conn(0x562549f59160 msgr2=0x562549f5aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.702+0000 7f4864ff9640 0 client.19263 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:43.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.703+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 shutdown_connections 2024-03-19T02:40:43.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.703+0000 7f4885fe3740 1 -- 172.21.15.116:0/2885654483 wait complete. 2024-03-19T02:40:43.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.703+0000 7f4885fe3740 20 client.19263 trim_cache size 0 max 16384 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 Processor -- start 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 -- start start 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1aedf0 con 0x562549f5ea30 2024-03-19T02:40:43.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1b0990 con 0x562549f67f30 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f4885fe3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1af4c0 con 0x562549f5f350 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.725+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59202/0 (socket says 172.21.15.116:59202) 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 -- 172.21.15.116:0/1094520093 learned_addr learned my addr 172.21.15.116:0/1094520093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48818b8640 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48808b6640 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 -- 172.21.15.116:0/1094520093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 msgr2=0x562549f581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 -- 172.21.15.116:0/1094520093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 msgr2=0x562549f65870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 -- 172.21.15.116:0/1094520093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562549f5f180 con 0x562549f5ea30 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f48810b7640 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f4878009850 tx=0x7f48780058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3bbc08448089d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f4866ffd640 1 -- 172.21.15.116:0/1094520093 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878063030 con 0x562549f5ea30 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f4866ffd640 1 -- 172.21.15.116:0/1094520093 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4878045040 con 0x562549f5ea30 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.726+0000 7f4866ffd640 1 -- 172.21.15.116:0/1094520093 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878050070 con 0x562549f5ea30 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- 172.21.15.116:0/1094520093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 msgr2=0x562549f73ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f4878009850 tx=0x7f48780058d0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- 172.21.15.116:0/1094520093 shutdown_connections 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f67f30 0x562549f65870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f5f350 0x562549f581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1094520093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f73ac0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- 172.21.15.116:0/1094520093 >> 172.21.15.116:0/1094520093 conn(0x562549f59160 msgr2=0x562549f63f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- 172.21.15.116:0/1094520093 shutdown_connections 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- 172.21.15.116:0/1094520093 wait complete. 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 -- start start 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.727+0000 7f4885fe3740 1 Processor -- start 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f5f350 0x562549f6b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 0x562549f6bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1b0260 con 0x562549f5ea30 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1afb90 con 0x562549f5f350 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f487c1acdf0 con 0x562549f67f30 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4885fe3740 10 client.0 _ll_register_callbacks cb 0x7f4878009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59204/0 (socket says 172.21.15.116:59204) 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48818b8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 0x562549f6bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 -- 172.21.15.116:0/1545661743 learned_addr learned my addr 172.21.15.116:0/1545661743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48808b6640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f5f350 0x562549f6b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 -- 172.21.15.116:0/1545661743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 msgr2=0x562549f6bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 0x562549f6bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 -- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f5f350 msgr2=0x562549f6b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f5f350 0x562549f6b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 -- 172.21.15.116:0/1545661743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562549b960f0 con 0x562549f5ea30 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48818b8640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 0x562549f6bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f48810b7640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4878004ee0 tx=0x7f48780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9577b4bf76984b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.728+0000 7f4864ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878063070 con 0x562549f5ea30 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f4878003800 con 0x562549f5ea30 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4878045040 con 0x562549f5ea30 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4878071050 con 0x562549f5ea30 2024-03-19T02:40:43.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 10 client.19269 fetch_fsmap learned FSMap version 31 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562549fde5c0 con 0x562549f5ea30 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4878045040 con 0x562549f5ea30 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7f4878041c50 con 0x562549f5ea30 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 10 client.19269 fetch_fsmap finished waiting for FSMap version 31 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 10 client.19269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 20 client.19269 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.729+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549fdec30 0x562549fe1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.730+0000 7f4885fe3740 4 client.19269 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.730+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562549ecfa40 con 0x562549fdec30 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.730+0000 7f48808b6640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549fdec30 0x562549fe1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.730+0000 7f4864ff9640 10 client.19269 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.730+0000 7f48808b6640 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549fdec30 0x562549fe1050 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e34e17ceabe89a server_cookie=e8011e6ba2e78db0 in_seq=0 out_seq=0 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.731+0000 7f4864ff9640 1 -- 172.21.15.116:0/1545661743 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562549ecfa40 con 0x562549fdec30 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.731+0000 7f4864ff9640 10 client.19269 handle_command_reply: tid=0 2024-03-19T02:40:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.731+0000 7f4885fe3740 1 client.19269 shutdown 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.731+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 msgr2=0x562549f70460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.731+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4878004ee0 tx=0x7f48780057f0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 shutdown_connections 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562549fdec30 0x562549fe1050 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4864ff9640 0 client.19269 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562549f67f30 0x562549f6bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562549f5f350 0x562549f6b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 --2- 172.21.15.116:0/1545661743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562549f5ea30 0x562549f70460 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 >> 172.21.15.116:0/1545661743 conn(0x562549f59160 msgr2=0x562549f581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 shutdown_connections 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 1 -- 172.21.15.116:0/1545661743 wait complete. 2024-03-19T02:40:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.732+0000 7f4885fe3740 20 client.19269 trim_cache size 0 max 16384 2024-03-19T02:40:43.742 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:43.742 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:40:43.742 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:40:43.742 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 msgr2=0x7f48680a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 0x7f48680a4eb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4878006010 tx=0x7f4878005e40 comp rx=0 tx=0).stop 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 msgr2=0x7f487c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f486c00e9f0 tx=0x7f486c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 shutdown_connections 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48680a29f0 0x7f48680a4eb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c1077c0 0x7f487c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487c100680 0x7f487c198560 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 --2- 172.21.15.116:0/1208225924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f487c0ffcc0 0x7f487c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 >> 172.21.15.116:0/1208225924 conn(0x7f487c076c40 msgr2=0x7f487c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 shutdown_connections 2024-03-19T02:40:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.733+0000 7f488331b640 1 -- 172.21.15.116:0/1208225924 wait complete. 2024-03-19T02:40:43.745 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:40:43.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:40:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- 172.21.15.116:0/442002254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c1010e0 msgr2=0x7fe65c10d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- 172.21.15.116:0/442002254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c1010e0 0x7fe65c10d5f0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fe654009cd0 tx=0x7fe65402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- 172.21.15.116:0/442002254 shutdown_connections 2024-03-19T02:40:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- 172.21.15.116:0/442002254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c1010e0 0x7fe65c10d5f0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- 172.21.15.116:0/442002254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c100720 0x7fe65c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- 172.21.15.116:0/442002254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c106730 0x7fe65c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- 172.21.15.116:0/442002254 >> 172.21.15.116:0/442002254 conn(0x7fe65c077820 msgr2=0x7fe65c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- 172.21.15.116:0/442002254 shutdown_connections 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- 172.21.15.116:0/442002254 wait complete. 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 Processor -- start 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- start start 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 0x7fe65c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c106730 0x7fe65c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c1109b0 con 0x7fe65c100720 2024-03-19T02:40:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c110830 con 0x7fe65c106730 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.879+0000 7fe663d37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c110b30 con 0x7fe65c1010e0 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59220/0 (socket says 172.21.15.116:59220) 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 -- 172.21.15.116:0/3793689430 learned_addr learned my addr 172.21.15.116:0/3793689430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe6612d2640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 0x7fe65c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe6622d4640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c106730 0x7fe65c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 -- 172.21.15.116:0/3793689430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 msgr2=0x7fe65c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 0x7fe65c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 -- 172.21.15.116:0/3793689430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c106730 msgr2=0x7fe65c19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c106730 0x7fe65c19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 -- 172.21.15.116:0/3793689430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65c19f1d0 con 0x7fe65c100720 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe6612d2640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 0x7fe65c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe661ad3640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe65800e9f0 tx=0x7fe65800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65800cd80 con 0x7fe65c100720 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6580108e0 con 0x7fe65c100720 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65c19f4c0 con 0x7fe65c100720 2024-03-19T02:40:43.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658018920 con 0x7fe65c100720 2024-03-19T02:40:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.880+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65c1a6d50 con 0x7fe65c100720 2024-03-19T02:40:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.882+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65c068610 con 0x7fe65c100720 2024-03-19T02:40:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.883+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe658010a90 con 0x7fe65c100720 2024-03-19T02:40:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.883+0000 7fe652ffd640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 0x7fe6480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.884+0000 7fe6612d2640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 0x7fe6480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.884+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fe6580cf4f0 con 0x7fe65c100720 2024-03-19T02:40:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.884+0000 7fe6612d2640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 0x7fe6480a4e30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe65c10da10 tx=0x7fe64c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:43.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:43.889+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe65809b820 con 0x7fe65c100720 2024-03-19T02:40:44.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.204+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe65c1a6f20 con 0x7fe65c100720 2024-03-19T02:40:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.205+0000 7fe652ffd640 1 -- 172.21.15.116:0/3793689430 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11954 (secure 0 0 0) 0x7fe65809f8c0 con 0x7fe65c100720 2024-03-19T02:40:44.204 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:40:44.205 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:40:43.038554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:40:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 msgr2=0x7fe6480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 0x7fe6480a4e30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe65c10da10 tx=0x7fe64c005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 msgr2=0x7fe65c19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe65800e9f0 tx=0x7fe65800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 shutdown_connections 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6480a2970 0x7fe6480a4e30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe65c106730 0x7fe65c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe65c1010e0 0x7fe65c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 --2- 172.21.15.116:0/3793689430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe65c100720 0x7fe65c19a230 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 >> 172.21.15.116:0/3793689430 conn(0x7fe65c077820 msgr2=0x7fe65c068b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 shutdown_connections 2024-03-19T02:40:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.207+0000 7fe663d37640 1 -- 172.21.15.116:0/3793689430 wait complete. 2024-03-19T02:40:44.217 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:43 smithi116 ceph-mon[32263]: pgmap v404: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 11 MiB/s wr, 110 op/s 2024-03-19T02:40:44.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:40:44.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:43 smithi154 ceph-mon[35693]: pgmap v404: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 11 MiB/s wr, 110 op/s 2024-03-19T02:40:44.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1024 37.03 MB/sec warmup 6 sec latency 151.320 ms 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 -- 172.21.15.116:0/2675269936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 msgr2=0x7f0e8c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/2675269936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c10d570 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009cd0 tx=0x7f0e8402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 -- 172.21.15.116:0/2675269936 shutdown_connections 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/2675269936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c10d570 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/2675269936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/2675269936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 -- 172.21.15.116:0/2675269936 >> 172.21.15.116:0/2675269936 conn(0x7f0e8c0778d0 msgr2=0x7f0e8c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 -- 172.21.15.116:0/2675269936 shutdown_connections 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.332+0000 7f0e93ba9640 1 -- 172.21.15.116:0/2675269936 wait complete. 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 Processor -- start 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 -- start start 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c110970 con 0x7f0e8c101060 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1107f0 con 0x7f0e8c1066b0 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.333+0000 7f0e93ba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c110af0 con 0x7f0e8c1006a0 2024-03-19T02:40:44.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59250/0 (socket says 172.21.15.116:59250) 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91945640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 -- 172.21.15.116:0/7477137 learned_addr learned my addr 172.21.15.116:0/7477137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e92146640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 -- 172.21.15.116:0/7477137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 msgr2=0x7f0e8c19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 -- 172.21.15.116:0/7477137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 msgr2=0x7f0e8c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 -- 172.21.15.116:0/7477137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c19f260 con 0x7f0e8c101060 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e92146640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91945640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e91144640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00e9f0 tx=0x7f0e7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e82ffd640 1 -- 172.21.15.116:0/7477137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c00cd80 con 0x7f0e8c101060 2024-03-19T02:40:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e82ffd640 1 -- 172.21.15.116:0/7477137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e7c0108e0 con 0x7f0e8c101060 2024-03-19T02:40:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e82ffd640 1 -- 172.21.15.116:0/7477137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c018920 con 0x7f0e8c101060 2024-03-19T02:40:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c19f550 con 0x7f0e8c101060 2024-03-19T02:40:44.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.334+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c1a6de0 con 0x7f0e8c101060 2024-03-19T02:40:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.337+0000 7f0e82ffd640 1 -- 172.21.15.116:0/7477137 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0e7c010a90 con 0x7f0e8c101060 2024-03-19T02:40:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.338+0000 7f0e82ffd640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 0x7f0e780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.338+0000 7f0e9678d740 1 Processor -- start 2024-03-19T02:40:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.338+0000 7f0e91945640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 0x7f0e780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.338+0000 7f0e82ffd640 1 -- 172.21.15.116:0/7477137 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f0e7c0cf4a0 con 0x7f0e8c101060 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.338+0000 7f0e9678d740 1 -- start start 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561ade93f4b0 con 0x561ade9dcbf0 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561ade8e7d70 con 0x561ade9dc2d0 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e9678d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561ade93b550 con 0x561ade9e57d0 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91945640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 0x7f0e780a4e30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009f40 tx=0x7f0e88005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91945640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59262/0 (socket says 172.21.15.116:59262) 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91945640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49706/0 (socket says 172.21.15.116:49706) 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 -- 172.21.15.116:0/410093095 learned_addr learned my addr 172.21.15.116:0/410093095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e92146640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 -- 172.21.15.116:0/410093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 msgr2=0x561ade9e7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 -- 172.21.15.116:0/410093095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 msgr2=0x561ade9dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91144640 1 -- 172.21.15.116:0/410093095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ade90a210 con 0x561ade9dcbf0 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e92146640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.339+0000 7f0e91945640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e91144640 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00c6a0 tx=0x7f0e7c09b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ffc79aed16747e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e77fff640 1 -- 172.21.15.116:0/410093095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0ccb30 con 0x561ade9dcbf0 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e77fff640 1 -- 172.21.15.116:0/410093095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e7c097ee0 con 0x561ade9dcbf0 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e77fff640 1 -- 172.21.15.116:0/410093095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0afba0 con 0x561ade9dcbf0 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 -- 172.21.15.116:0/410093095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 msgr2=0x561ade9e5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00c6a0 tx=0x7f0e7c09b870 comp rx=0 tx=0).stop 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 -- 172.21.15.116:0/410093095 shutdown_connections 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dcbf0 0x561ade9e5290 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 --2- 172.21.15.116:0/410093095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.340+0000 7f0e9678d740 1 -- 172.21.15.116:0/410093095 >> 172.21.15.116:0/410093095 conn(0x561ade9d6a00 msgr2=0x561ade9d6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 -- 172.21.15.116:0/410093095 shutdown_connections 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 -- 172.21.15.116:0/410093095 wait complete. 2024-03-19T02:40:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 -- start start 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 Processor -- start 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 0x561ade9e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561ade9e8560 con 0x561ade9dc2d0 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.341+0000 7f0e9678d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561ade8fb310 con 0x561ade9dcbf0 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e9678d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561ade9e86e0 con 0x561ade9e57d0 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e9678d740 10 client.0 _ll_register_callbacks cb 0x561ade8ef2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59276/0 (socket says 172.21.15.116:59276) 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 -- 172.21.15.116:0/1286254361 learned_addr learned my addr 172.21.15.116:0/1286254361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91144640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 0x561ade9e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e92146640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 -- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 msgr2=0x561ade9e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 -- 172.21.15.116:0/1286254361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 msgr2=0x561ade9e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 0x561ade9e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 -- 172.21.15.116:0/1286254361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ade6533f0 con 0x561ade9dc2d0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91144640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 0x561ade9e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e91945640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800fda0 tx=0x7f0e88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36715dcbbfc53c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8801db20 con 0x561ade9dc2d0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f0e8801dcc0 con 0x561ade9dc2d0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.342+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e7c097ee0 con 0x561ade9dc2d0 2024-03-19T02:40:44.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0e88031050 con 0x561ade9dc2d0 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 10 client.19287 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561ade71d7d0 con 0x561ade9dc2d0 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e7c097ee0 con 0x561ade9dc2d0 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f0e88003ba0 con 0x561ade9dc2d0 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 10 client.19287 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 10 client.19287 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 20 client.19287 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561ade9f5bd0 0x561ade9f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 4 client.19287 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ade99d3e0 con 0x561ade9f5bd0 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.343+0000 7f0e91144640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561ade9f5bd0 0x561ade9f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.344+0000 7f0e91144640 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561ade9f5bd0 0x561ade9f8000 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b8e112091d4cde14 server_cookie=c9e445142b58935c in_seq=0 out_seq=0 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.344+0000 7f0e757fa640 10 client.19287 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e757fa640 1 -- 172.21.15.116:0/1286254361 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ade99d3e0 con 0x561ade9f5bd0 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e757fa640 10 client.19287 handle_command_reply: tid=0 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 client.19287 shutdown 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 msgr2=0x561ade9e28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800fda0 tx=0x7f0e88004730 comp rx=0 tx=0).stop 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 shutdown_connections 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561ade9f5bd0 0x561ade9f8000 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9e57d0 0x561ade9e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dcbf0 0x561ade9e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 --2- 172.21.15.116:0/1286254361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9e28d0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 >> 172.21.15.116:0/1286254361 conn(0x561ade9d6a00 msgr2=0x561ade9d8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e757fa640 0 client.19287 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 shutdown_connections 2024-03-19T02:40:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 1 -- 172.21.15.116:0/1286254361 wait complete. 2024-03-19T02:40:44.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.345+0000 7f0e9678d740 20 client.19287 trim_cache size 0 max 16384 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 Processor -- start 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 -- start start 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9f1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1b0ee0 con 0x561ade9e57d0 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1b2ab0 con 0x561ade9dc2d0 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e9678d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1b15b0 con 0x561ade9dcbf0 2024-03-19T02:40:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e92146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e92146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59278/0 (socket says 172.21.15.116:59278) 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e92146640 1 -- 172.21.15.116:0/2865902085 learned_addr learned my addr 172.21.15.116:0/2865902085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e91945640 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.361+0000 7f0e91144640 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 -- 172.21.15.116:0/2865902085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 msgr2=0x561ade9e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 -- 172.21.15.116:0/2865902085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 msgr2=0x561ade9f1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9f1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 -- 172.21.15.116:0/2865902085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ade9e0e20 con 0x561ade9e57d0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e92146640 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009cd0 tx=0x7f0e84005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5836b88b819f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e777fe640 1 -- 172.21.15.116:0/2865902085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84045070 con 0x561ade9e57d0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e777fe640 1 -- 172.21.15.116:0/2865902085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e84003e20 con 0x561ade9e57d0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.362+0000 7f0e777fe640 1 -- 172.21.15.116:0/2865902085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84039040 con 0x561ade9e57d0 2024-03-19T02:40:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- 172.21.15.116:0/2865902085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 msgr2=0x561ade9fd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009cd0 tx=0x7f0e84005c50 comp rx=0 tx=0).stop 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- 172.21.15.116:0/2865902085 shutdown_connections 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9e57d0 0x561ade9fd8d0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 --2- 172.21.15.116:0/2865902085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9dc2d0 0x561ade9f1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- 172.21.15.116:0/2865902085 >> 172.21.15.116:0/2865902085 conn(0x561ade9d6a00 msgr2=0x561ade9d5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- 172.21.15.116:0/2865902085 shutdown_connections 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- 172.21.15.116:0/2865902085 wait complete. 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 -- start start 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.363+0000 7f0e9678d740 1 Processor -- start 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9e57d0 0x561ade9e9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1b2380 con 0x561ade9dc2d0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1b1cb0 con 0x561ade9e57d0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c1aeee0 con 0x561ade9dcbf0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 10 client.0 _ll_register_callbacks cb 0x7f0e8800f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59294/0 (socket says 172.21.15.116:59294) 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 -- 172.21.15.116:0/4276609216 learned_addr learned my addr 172.21.15.116:0/4276609216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e92146640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9e57d0 0x561ade9e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91144640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 -- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 msgr2=0x561ade9e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 -- 172.21.15.116:0/4276609216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9e57d0 msgr2=0x561ade9e9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9e57d0 0x561ade9e9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 -- 172.21.15.116:0/4276609216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ade9e9a10 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e91945640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f0e88016040 tx=0x7f0e8801ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214e2e0aa22953ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88039070 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f0e88009040 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e84003e20 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0e88045050 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 10 client.19293 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.364+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561adea5bcb0 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e84003e20 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f0e88003b20 con 0x561ade9dc2d0 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 10 client.19293 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 10 client.19293 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:40:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 20 client.19293 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561adea5c460 0x561adea5e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 4 client.19293 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ade94c9f0 con 0x561adea5c460 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.365+0000 7f0e91144640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561adea5c460 0x561adea5e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e757fa640 10 client.19293 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e91144640 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561adea5c460 0x561adea5e880 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=91ea87ff31652ded server_cookie=afeaa2886b019698 in_seq=0 out_seq=0 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e757fa640 1 -- 172.21.15.116:0/4276609216 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ade94c9f0 con 0x561adea5c460 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e757fa640 10 client.19293 handle_command_reply: tid=0 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 client.19293 shutdown 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 msgr2=0x561ade9ed9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f0e88016040 tx=0x7f0e8801ddb0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 shutdown_connections 2024-03-19T02:40:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561adea5c460 0x561adea5e880 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ade9e57d0 0x561ade9e9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e757fa640 0 client.19293 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ade9dcbf0 0x561ade9e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 --2- 172.21.15.116:0/4276609216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561ade9dc2d0 0x561ade9ed9b0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 >> 172.21.15.116:0/4276609216 conn(0x561ade9d6a00 msgr2=0x561ade9d5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 shutdown_connections 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 1 -- 172.21.15.116:0/4276609216 wait complete. 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.366+0000 7f0e9678d740 20 client.19293 trim_cache size 0 max 16384 2024-03-19T02:40:44.373 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 msgr2=0x7f0e780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 0x7f0e780a4e30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009f40 tx=0x7f0e88005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 msgr2=0x7f0e8c19a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00e9f0 tx=0x7f0e7c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 shutdown_connections 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e780a2970 0x7f0e780a4e30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e8c1066b0 0x7f0e8c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e8c101060 0x7f0e8c19a7b0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 --2- 172.21.15.116:0/7477137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e8c1006a0 0x7f0e8c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 >> 172.21.15.116:0/7477137 conn(0x7f0e8c0778d0 msgr2=0x7f0e8c10c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 shutdown_connections 2024-03-19T02:40:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.367+0000 7f0e93ba9640 1 -- 172.21.15.116:0/7477137 wait complete. 2024-03-19T02:40:44.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:40:44.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:43 smithi162 ceph-mon[35761]: pgmap v404: 129 pgs: 129 active+clean; 603 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 11 MiB/s wr, 110 op/s 2024-03-19T02:40:44.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 -- 172.21.15.116:0/2644469993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c102a40 msgr2=0x7fed0c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 --2- 172.21.15.116:0/2644469993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c102a40 0x7fed0c102e20 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fed08009900 tx=0x7fed0802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 -- 172.21.15.116:0/2644469993 shutdown_connections 2024-03-19T02:40:44.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 --2- 172.21.15.116:0/2644469993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c10bd80 0x7fed0c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 --2- 172.21.15.116:0/2644469993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c103360 0x7fed0c10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 --2- 172.21.15.116:0/2644469993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c102a40 0x7fed0c102e20 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 -- 172.21.15.116:0/2644469993 >> 172.21.15.116:0/2644469993 conn(0x7fed0c0778d0 msgr2=0x7fed0c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 -- 172.21.15.116:0/2644469993 shutdown_connections 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.494+0000 7fed1347e640 1 -- 172.21.15.116:0/2644469993 wait complete. 2024-03-19T02:40:44.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 Processor -- start 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 -- start start 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c102a40 0x7fed0c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c10bd80 0x7fed0c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c111a30 con 0x7fed0c103360 2024-03-19T02:40:44.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1118b0 con 0x7fed0c102a40 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1347e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c111bb0 con 0x7fed0c10bd80 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59310/0 (socket says 172.21.15.116:59310) 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed10a19640 1 -- 172.21.15.116:0/1684064671 learned_addr learned my addr 172.21.15.116:0/1684064671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed11a1b640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c10bd80 0x7fed0c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.495+0000 7fed1121a640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c102a40 0x7fed0c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 -- 172.21.15.116:0/1684064671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c10bd80 msgr2=0x7fed0c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c10bd80 0x7fed0c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 -- 172.21.15.116:0/1684064671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c102a40 msgr2=0x7fed0c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c102a40 0x7fed0c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 -- 172.21.15.116:0/1684064671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c19f170 con 0x7fed0c103360 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed10a19640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e9f0 tx=0x7fed0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fecfe7fc640 1 -- 172.21.15.116:0/1684064671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000cd80 con 0x7fed0c103360 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fecfe7fc640 1 -- 172.21.15.116:0/1684064671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed000108e0 con 0x7fed0c103360 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c19f460 con 0x7fed0c103360 2024-03-19T02:40:44.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fecfe7fc640 1 -- 172.21.15.116:0/1684064671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00018920 con 0x7fed0c103360 2024-03-19T02:40:44.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.496+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c1a6cf0 con 0x7fed0c103360 2024-03-19T02:40:44.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fecfe7fc640 1 -- 172.21.15.116:0/1684064671 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fed00010a90 con 0x7fed0c103360 2024-03-19T02:40:44.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 Processor -- start 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 -- start start 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 0x55728b1f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f97d0 0x55728b1fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55728b1534b0 con 0x55728b1f02d0 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55728b0fbd70 con 0x55728b1f97d0 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.499+0000 7fed16063740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55728b14f550 con 0x55728b1f0bf0 2024-03-19T02:40:44.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59322/0 (socket says 172.21.15.116:59322) 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed1121a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 0x55728b1f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 -- 172.21.15.116:0/4165923400 learned_addr learned my addr 172.21.15.116:0/4165923400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed10a19640 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f97d0 0x55728b1fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fecfe7fc640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 0x7fecf80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 -- 172.21.15.116:0/4165923400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 msgr2=0x55728b1f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 0x55728b1f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 -- 172.21.15.116:0/4165923400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f97d0 msgr2=0x55728b1fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f97d0 0x55728b1fbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 -- 172.21.15.116:0/4165923400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728b11e210 con 0x55728b1f02d0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed1121a640 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 0x55728b1f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fecfe7fc640 1 -- 172.21.15.116:0/1684064671 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fed000cf4c0 con 0x7fed0c103360 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 0x7fecf80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fed040104a0 tx=0x7fed04010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b166c1f05cf4e42e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fecf77fe640 1 -- 172.21.15.116:0/4165923400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0401d910 con 0x55728b1f02d0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fed11a1b640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 0x7fecf80a4e30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed04021000 tx=0x7fed0401e000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fecf77fe640 1 -- 172.21.15.116:0/4165923400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed04011440 con 0x55728b1f02d0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.500+0000 7fecf77fe640 1 -- 172.21.15.116:0/4165923400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0401c6a0 con 0x55728b1f02d0 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- 172.21.15.116:0/4165923400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 msgr2=0x55728b1f06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fed040104a0 tx=0x7fed04010960 comp rx=0 tx=0).stop 2024-03-19T02:40:44.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- 172.21.15.116:0/4165923400 shutdown_connections 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f97d0 0x55728b1fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f0bf0 0x55728b1f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 --2- 172.21.15.116:0/4165923400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1f06b0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- 172.21.15.116:0/4165923400 >> 172.21.15.116:0/4165923400 conn(0x55728b1eaa00 msgr2=0x55728b1eade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- 172.21.15.116:0/4165923400 shutdown_connections 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- 172.21.15.116:0/4165923400 wait complete. 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.501+0000 7fed16063740 1 -- start start 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 Processor -- start 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b1f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55728b1fc5c0 con 0x55728b1f0bf0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55728b10f310 con 0x55728b1f02d0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55728b1fc740 con 0x55728b1f97d0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed16063740 10 client.0 _ll_register_callbacks cb 0x55728b1032d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59324/0 (socket says 172.21.15.116:59324) 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 -- 172.21.15.116:0/1352941153 learned_addr learned my addr 172.21.15.116:0/1352941153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed11a1b640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed1121a640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b1f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 -- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 msgr2=0x55728b1f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 -- 172.21.15.116:0/1352941153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 msgr2=0x55728b1f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b1f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.502+0000 7fed10a19640 1 -- 172.21.15.116:0/1352941153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728ae673f0 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed10a19640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c6a0 tx=0x7fed0009b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4310cf9efd1bac5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000a3f80 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fed000860f0 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed04011440 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fed000a3af0 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 10 client.19305 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55728af317d0 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed04011440 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7fed0009ace0 con 0x55728b1f0bf0 2024-03-19T02:40:44.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 10 client.19305 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 10 client.19305 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 20 client.19305 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b209c60 0x55728b20c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 4 client.19305 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55728b1b13e0 con 0x55728b209c60 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.503+0000 7fed1121a640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b209c60 0x55728b20c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.504+0000 7fecf4ff9640 10 client.19305 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.504+0000 7fed1121a640 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b209c60 0x55728b20c090 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4e3efdd8da1cc296 server_cookie=af7f1f95c7eba491 in_seq=0 out_seq=0 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fecf4ff9640 1 -- 172.21.15.116:0/1352941153 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55728b1b13e0 con 0x55728b209c60 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fecf4ff9640 10 client.19305 handle_command_reply: tid=0 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 client.19305 shutdown 2024-03-19T02:40:44.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 msgr2=0x55728b1f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c6a0 tx=0x7fed0009b2e0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 shutdown_connections 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b209c60 0x55728b20c090 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b1f6df0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 --2- 172.21.15.116:0/1352941153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b1f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fecf4ff9640 0 client.19305 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 >> 172.21.15.116:0/1352941153 conn(0x55728b1eaa00 msgr2=0x55728b1ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 shutdown_connections 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 1 -- 172.21.15.116:0/1352941153 wait complete. 2024-03-19T02:40:44.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.505+0000 7fed16063740 20 client.19305 trim_cache size 0 max 16384 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 Processor -- start 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 -- start start 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b205360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1b0f40 con 0x55728b1f0bf0 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1b2b10 con 0x55728b1f02d0 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed16063740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1b1610 con 0x55728b1f97d0 2024-03-19T02:40:44.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.519+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59328/0 (socket says 172.21.15.116:59328) 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 -- 172.21.15.116:0/670603538 learned_addr learned my addr 172.21.15.116:0/670603538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed11a1b640 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed1121a640 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b205360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 -- 172.21.15.116:0/670603538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 msgr2=0x55728b1e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1e9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 -- 172.21.15.116:0/670603538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 msgr2=0x55728b205360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b205360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 -- 172.21.15.116:0/670603538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728b1eb5d0 con 0x55728b1f0bf0 2024-03-19T02:40:44.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fed10a19640 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fed00098260 tx=0x7fed0009fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2086f78d77b15a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fecf6ffd640 1 -- 172.21.15.116:0/670603538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0009b890 con 0x55728b1f0bf0 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fecf6ffd640 1 -- 172.21.15.116:0/670603538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed000a3c90 con 0x55728b1f0bf0 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.520+0000 7fecf6ffd640 1 -- 172.21.15.116:0/670603538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0009cb10 con 0x55728b1f0bf0 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- 172.21.15.116:0/670603538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 msgr2=0x55728b211960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fed00098260 tx=0x7fed0009fab0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- 172.21.15.116:0/670603538 shutdown_connections 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f0bf0 0x55728b211960 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 --2- 172.21.15.116:0/670603538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f02d0 0x55728b205360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- 172.21.15.116:0/670603538 >> 172.21.15.116:0/670603538 conn(0x55728b1eaa00 msgr2=0x55728b1f1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- 172.21.15.116:0/670603538 shutdown_connections 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- 172.21.15.116:0/670603538 wait complete. 2024-03-19T02:40:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 -- start start 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.521+0000 7fed16063740 1 Processor -- start 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 0x55728b1fd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1fdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1b23e0 con 0x55728b1f02d0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1b1d10 con 0x55728b1f0bf0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c1aef40 con 0x55728b1f97d0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed16063740 10 client.0 _ll_register_callbacks cb 0x7fed0009b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed1121a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1fdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed10a19640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 0x55728b1fd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed1121a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59330/0 (socket says 172.21.15.116:59330) 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed1121a640 1 -- 172.21.15.116:0/3377454951 learned_addr learned my addr 172.21.15.116:0/3377454951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 -- 172.21.15.116:0/3377454951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 msgr2=0x55728b1fd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 0x55728b1fd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 -- 172.21.15.116:0/3377454951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 msgr2=0x55728b1fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed11a1b640 1 -- 172.21.15.116:0/3377454951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728b26efe0 con 0x55728b1f97d0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed1121a640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.522+0000 7fed10a19640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 0x55728b1fd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed11a1b640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1fdd70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fed040096e0 tx=0x7fed04010fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=336e07042b65cb35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04013050 con 0x55728b1f97d0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed000a3c90 con 0x55728b1f97d0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7fed040131f0 con 0x55728b1f97d0 2024-03-19T02:40:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fed04040050 con 0x55728b1f97d0 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 10 client.29041 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55728b270330 con 0x55728b1f97d0 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed000a3c90 con 0x55728b1f97d0 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7fed0401c960 con 0x55728b1f97d0 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 10 client.29041 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 10 client.29041 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 20 client.29041 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b270890 0x55728b272cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 4 client.29041 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.523+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55728b1609f0 con 0x55728b270890 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed1121a640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b270890 0x55728b272cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fecf4ff9640 10 client.29041 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed1121a640 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b270890 0x55728b272cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5fe04915fc3ccb4d server_cookie=87ae6df57ab2c02f in_seq=0 out_seq=0 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fecf4ff9640 1 -- 172.21.15.116:0/3377454951 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55728b1609f0 con 0x55728b270890 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fecf4ff9640 10 client.29041 handle_command_reply: tid=0 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 client.29041 shutdown 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 msgr2=0x55728b1fdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1fdd70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fed040096e0 tx=0x7fed04010fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 shutdown_connections 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55728b1f97d0 0x55728b1fdd70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55728b1f0bf0 0x55728b1fd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55728b270890 0x55728b272cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 --2- 172.21.15.116:0/3377454951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728b1f02d0 0x55728b1fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fecf4ff9640 0 client.29041 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.524+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 >> 172.21.15.116:0/3377454951 conn(0x55728b1eaa00 msgr2=0x55728b1f6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 shutdown_connections 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed16063740 1 -- 172.21.15.116:0/3377454951 wait complete. 2024-03-19T02:40:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed16063740 20 client.29041 trim_cache size 0 max 16384 2024-03-19T02:40:44.537 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 msgr2=0x7fecf80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 0x7fecf80a4e30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed04021000 tx=0x7fed0401e000 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 msgr2=0x7fed0c19a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e9f0 tx=0x7fed0000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 shutdown_connections 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fecf80a2970 0x7fecf80a4e30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed0c10bd80 0x7fed0c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c103360 0x7fed0c19a710 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 --2- 172.21.15.116:0/1684064671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed0c102a40 0x7fed0c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 >> 172.21.15.116:0/1684064671 conn(0x7fed0c0778d0 msgr2=0x7fed0c107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 shutdown_connections 2024-03-19T02:40:44.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.525+0000 7fed1347e640 1 -- 172.21.15.116:0/1684064671 wait complete. 2024-03-19T02:40:44.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 -- 172.21.15.116:0/3573412296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848105670 msgr2=0x7f7848105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 --2- 172.21.15.116:0/3573412296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848105670 0x7f7848105a50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7838009900 tx=0x7f783802bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 -- 172.21.15.116:0/3573412296 shutdown_connections 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 --2- 172.21.15.116:0/3573412296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78481017f0 0x7f784810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 --2- 172.21.15.116:0/3573412296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7848100e30 0x7f78481012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 --2- 172.21.15.116:0/3573412296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848105670 0x7f7848105a50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 -- 172.21.15.116:0/3573412296 >> 172.21.15.116:0/3573412296 conn(0x7f7848076c50 msgr2=0x7f78480fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 -- 172.21.15.116:0/3573412296 shutdown_connections 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.669+0000 7f784cdb3640 1 -- 172.21.15.116:0/3573412296 wait complete. 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.670+0000 7f784cdb3640 1 Processor -- start 2024-03-19T02:40:44.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.670+0000 7f784cdb3640 1 -- start start 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.670+0000 7f784cdb3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784cdb3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78481017f0 0x7f7848197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784cdb3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7848105670 0x7f7848197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784cdb3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7848110920 con 0x7f7848100e30 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784cdb3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78481107a0 con 0x7f7848105670 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784cdb3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7848110aa0 con 0x7f78481017f0 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59354/0 (socket says 172.21.15.116:59354) 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 -- 172.21.15.116:0/4269106182 learned_addr learned my addr 172.21.15.116:0/4269106182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f7846d9d640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7848105670 0x7f7848197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f7845d9b640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78481017f0 0x7f7848197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 -- 172.21.15.116:0/4269106182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78481017f0 msgr2=0x7f7848197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78481017f0 0x7f7848197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 -- 172.21.15.116:0/4269106182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7848105670 msgr2=0x7f7848197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7848105670 0x7f7848197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 -- 172.21.15.116:0/4269106182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78481a63b0 con 0x7f7848100e30 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f784659c640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f7838036040 tx=0x7f7838005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.671+0000 7f78377fe640 1 -- 172.21.15.116:0/4269106182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838043020 con 0x7f7848100e30 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.672+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78481a6640 con 0x7f7848100e30 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.672+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78481a6be0 con 0x7f7848100e30 2024-03-19T02:40:44.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.672+0000 7f78377fe640 1 -- 172.21.15.116:0/4269106182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7838004350 con 0x7f7848100e30 2024-03-19T02:40:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.672+0000 7f78377fe640 1 -- 172.21.15.116:0/4269106182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838003a10 con 0x7f7848100e30 2024-03-19T02:40:44.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.674+0000 7f784f996740 1 Processor -- start 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.674+0000 7f78377fe640 1 -- 172.21.15.116:0/4269106182 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7838046020 con 0x7f7848100e30 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 -- start start 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b746d0d4b0 con 0x55b746db37d0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b746cb5d70 con 0x55b746daa2d0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784f996740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b746d09550 con 0x55b746daabf0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f784659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f7846d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f7845d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f7846d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49842/0 (socket says 172.21.15.116:49842) 2024-03-19T02:40:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f7845d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59366/0 (socket says 172.21.15.116:59366) 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f7846d9d640 1 -- 172.21.15.116:0/3178587430 learned_addr learned my addr 172.21.15.116:0/3178587430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.675+0000 7f78377fe640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 0x7f78300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 -- 172.21.15.116:0/3178587430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 msgr2=0x55b746daa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 -- 172.21.15.116:0/3178587430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 msgr2=0x55b746db5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 -- 172.21.15.116:0/3178587430 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b746cd8210 con 0x55b746daabf0 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7846d9d640 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7845d9b640 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7846d9d640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 0x7f78300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f78377fe640 1 -- 172.21.15.116:0/4269106182 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f78380f08d0 con 0x7f7848100e30 2024-03-19T02:40:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f784659c640 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db3290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78380edcf0 tx=0x7f7838005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5c476a19382bd6a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7834ff9640 1 -- 172.21.15.116:0/3178587430 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380bd300 con 0x55b746daabf0 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7846d9d640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 0x7f78300a4e30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7840003df0 tx=0x7f7840004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7834ff9640 1 -- 172.21.15.116:0/3178587430 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78380c53a0 con 0x55b746daabf0 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.676+0000 7f7834ff9640 1 -- 172.21.15.116:0/3178587430 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380bcc00 con 0x55b746daabf0 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- 172.21.15.116:0/3178587430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 msgr2=0x55b746db3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db3290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f78380edcf0 tx=0x7f7838005e80 comp rx=0 tx=0).stop 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- 172.21.15.116:0/3178587430 shutdown_connections 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746db5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db3290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 --2- 172.21.15.116:0/3178587430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daa2d0 0x55b746daa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- 172.21.15.116:0/3178587430 >> 172.21.15.116:0/3178587430 conn(0x55b746da4a00 msgr2=0x55b746da4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- 172.21.15.116:0/3178587430 shutdown_connections 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- 172.21.15.116:0/3178587430 wait complete. 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.677+0000 7f784f996740 1 -- start start 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 Processor -- start 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746db1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b746d0d4b0 con 0x55b746daa2d0 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b746cc9310 con 0x55b746db37d0 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b746d09550 con 0x55b746daabf0 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784f996740 10 client.0 _ll_register_callbacks cb 0x55b746cbd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59380/0 (socket says 172.21.15.116:59380) 2024-03-19T02:40:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 -- 172.21.15.116:0/1415238935 learned_addr learned my addr 172.21.15.116:0/1415238935 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f7845d9b640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f7846d9d640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746db1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 -- 172.21.15.116:0/1415238935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 msgr2=0x55b746db0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 -- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 msgr2=0x55b746db1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746db1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 -- 172.21.15.116:0/1415238935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b746a213f0 con 0x55b746daa2d0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f784659c640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f78380ad2a0 tx=0x7f783802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50410bb27bcb1204 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.678+0000 7f782dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380bd300 con 0x55b746daa2d0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f783808b5a0 con 0x55b746daa2d0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78380c53a0 con 0x55b746daa2d0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f78380c0d50 con 0x55b746daa2d0 2024-03-19T02:40:44.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 10 client.19323 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b746aeb7d0 con 0x55b746daa2d0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78380c53a0 con 0x55b746daa2d0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f78380bce60 con 0x55b746daa2d0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 10 client.19323 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 10 client.19323 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 20 client.19323 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746dc36a0 0x55b746dc5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 4 client.19323 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.679+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b746d6b3e0 con 0x55b746dc36a0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.680+0000 7f7845d9b640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746dc36a0 0x55b746dc5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.680+0000 7f782dffb640 10 client.19323 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.680+0000 7f7845d9b640 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746dc36a0 0x55b746dc5ac0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba43ff84a36bb7b3 server_cookie=5fcc692fcaeb60bd in_seq=0 out_seq=0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f782dffb640 1 -- 172.21.15.116:0/1415238935 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b746d6b3e0 con 0x55b746dc36a0 2024-03-19T02:40:44.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f782dffb640 10 client.19323 handle_command_reply: tid=0 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 client.19323 shutdown 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 msgr2=0x55b746db07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f78380ad2a0 tx=0x7f783802bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 shutdown_connections 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746dc36a0 0x55b746dc5ac0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746db1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daabf0 0x55b746db0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 --2- 172.21.15.116:0/1415238935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daa2d0 0x55b746db07e0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 >> 172.21.15.116:0/1415238935 conn(0x55b746da4a00 msgr2=0x55b746da6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f782dffb640 0 client.19323 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 shutdown_connections 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 1 -- 172.21.15.116:0/1415238935 wait complete. 2024-03-19T02:40:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.681+0000 7f784f996740 20 client.19323 trim_cache size 0 max 16384 2024-03-19T02:40:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.701+0000 7f784f996740 1 Processor -- start 2024-03-19T02:40:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.701+0000 7f784f996740 1 -- start start 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daabf0 0x55b746dbee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746dcb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78481b0cc0 con 0x55b746db37d0 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78481b2890 con 0x55b746daabf0 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784f996740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78481b1390 con 0x55b746daa2d0 2024-03-19T02:40:44.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41238/0 (socket says 172.21.15.116:41238) 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 -- 172.21.15.116:0/1949007765 learned_addr learned my addr 172.21.15.116:0/1949007765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f7845d9b640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daabf0 0x55b746dbee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f7846d9d640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746dcb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 -- 172.21.15.116:0/1949007765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daabf0 msgr2=0x55b746dbee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daabf0 0x55b746dbee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.702+0000 7f784659c640 1 -- 172.21.15.116:0/1949007765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 msgr2=0x55b746dcb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f784659c640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746dcb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f784659c640 1 -- 172.21.15.116:0/1949007765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b746d9efb0 con 0x55b746daa2d0 2024-03-19T02:40:44.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f7846d9d640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746dcb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f784659c640 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f78380baeb0 tx=0x7f78380bc620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac9660c6d7008cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f782ffff640 1 -- 172.21.15.116:0/1949007765 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380bd300 con 0x55b746daa2d0 2024-03-19T02:40:44.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f782ffff640 1 -- 172.21.15.116:0/1949007765 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78380c53a0 con 0x55b746daa2d0 2024-03-19T02:40:44.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.703+0000 7f782ffff640 1 -- 172.21.15.116:0/1949007765 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380bce60 con 0x55b746daa2d0 2024-03-19T02:40:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 -- 172.21.15.116:0/1949007765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 msgr2=0x55b746daf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f78380baeb0 tx=0x7f78380bc620 comp rx=0 tx=0).stop 2024-03-19T02:40:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 -- 172.21.15.116:0/1949007765 shutdown_connections 2024-03-19T02:40:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746db37d0 0x55b746dcb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746daabf0 0x55b746dbee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 --2- 172.21.15.116:0/1949007765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746daf940 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 -- 172.21.15.116:0/1949007765 >> 172.21.15.116:0/1949007765 conn(0x55b746da4a00 msgr2=0x55b746da3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 -- 172.21.15.116:0/1949007765 shutdown_connections 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.704+0000 7f784f996740 1 -- 172.21.15.116:0/1949007765 wait complete. 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 -- start start 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 Processor -- start 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746dbb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746dad050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78481b2160 con 0x55b746daabf0 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78481b1a90 con 0x55b746db37d0 2024-03-19T02:40:44.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78481aecc0 con 0x55b746daa2d0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f7845d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f7845d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59404/0 (socket says 172.21.15.116:59404) 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746dbb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f7845d9b640 1 -- 172.21.15.116:0/3173639997 learned_addr learned my addr 172.21.15.116:0/3173639997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f7846d9d640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746dad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784f996740 10 client.0 _ll_register_callbacks cb 0x7f783c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784659c640 1 -- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 msgr2=0x55b746dad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784659c640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746dad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784659c640 1 -- 172.21.15.116:0/3173639997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 msgr2=0x55b746dacb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.705+0000 7f784659c640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f784659c640 1 -- 172.21.15.116:0/3173639997 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b746dbbbc0 con 0x55b746daa2d0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f7845d9b640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f784659c640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746dbb480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f78380bcb90 tx=0x7f78380bd1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=431b3d83565b1c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f782dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78380d9030 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f78380d5050 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78380c53a0 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f78380bcc00 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f784f996740 10 client.29059 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.706+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b746dbbbc0 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78380c53a0 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f7838083060 con 0x55b746daa2d0 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 10 client.29059 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 10 client.29059 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 20 client.29059 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746e297b0 0x55b746e2bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 4 client.29059 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b746d1a9f0 con 0x55b746e297b0 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f7845d9b640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746e297b0 0x55b746e2bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f7845d9b640 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746e297b0 0x55b746e2bbd0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f489967117d64b2 server_cookie=861389df41354c88 in_seq=0 out_seq=0 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.707+0000 7f782dffb640 10 client.29059 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f782dffb640 1 -- 172.21.15.116:0/3173639997 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b746d1a9f0 con 0x55b746e297b0 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f782dffb640 10 client.29059 handle_command_reply: tid=0 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 client.29059 shutdown 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 msgr2=0x55b746dbb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746dbb480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f78380bcb90 tx=0x7f78380bd1f0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 shutdown_connections 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b746e297b0 0x55b746e2bbd0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b746db37d0 0x55b746dad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b746daabf0 0x55b746dacb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f782dffb640 0 client.29059 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 --2- 172.21.15.116:0/3173639997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b746daa2d0 0x55b746dbb480 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 >> 172.21.15.116:0/3173639997 conn(0x55b746da4a00 msgr2=0x55b746daf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 shutdown_connections 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 1 -- 172.21.15.116:0/3173639997 wait complete. 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.709+0000 7f784f996740 20 client.29059 trim_cache size 0 max 16384 2024-03-19T02:40:44.717 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:44.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 msgr2=0x7f78300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 0x7f78300a4e30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7840003df0 tx=0x7f7840004a40 comp rx=0 tx=0).stop 2024-03-19T02:40:44.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 msgr2=0x7f7848196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f7838036040 tx=0x7f7838005d90 comp rx=0 tx=0).stop 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 shutdown_connections 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78300a2970 0x7f78300a4e30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7848105670 0x7f7848197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78481017f0 0x7f7848197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 --2- 172.21.15.116:0/4269106182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7848100e30 0x7f7848196bd0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 >> 172.21.15.116:0/4269106182 conn(0x7f7848076c50 msgr2=0x7f7848103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 shutdown_connections 2024-03-19T02:40:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.710+0000 7f784cdb3640 1 -- 172.21.15.116:0/4269106182 wait complete. 2024-03-19T02:40:44.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:40:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 -- 172.21.15.116:0/4201611155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a41077d0 msgr2=0x7f21a4107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 --2- 172.21.15.116:0/4201611155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a41077d0 0x7f21a4107bb0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2198009930 tx=0x7f219802bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 -- 172.21.15.116:0/4201611155 shutdown_connections 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 --2- 172.21.15.116:0/4201611155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a4102180 0x7f21a410e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 --2- 172.21.15.116:0/4201611155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 --2- 172.21.15.116:0/4201611155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a41077d0 0x7f21a4107bb0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 -- 172.21.15.116:0/4201611155 >> 172.21.15.116:0/4201611155 conn(0x7f21a40fd2d0 msgr2=0x7f21a40ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 -- 172.21.15.116:0/4201611155 shutdown_connections 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.844+0000 7f21aac36640 1 -- 172.21.15.116:0/4201611155 wait complete. 2024-03-19T02:40:44.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 Processor -- start 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 -- start start 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4111a50 con 0x7f21a4102180 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41118d0 con 0x7f21a41017c0 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21aac36640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4111bd0 con 0x7f21a41077d0 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a91d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59422/0 (socket says 172.21.15.116:59422) 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49876/0 (socket says 172.21.15.116:49876) 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a91d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41270/0 (socket says 172.21.15.116:41270) 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.845+0000 7f21a3fff640 1 -- 172.21.15.116:0/1051406716 learned_addr learned my addr 172.21.15.116:0/1051406716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 -- 172.21.15.116:0/1051406716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 msgr2=0x7f21a419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 -- 172.21.15.116:0/1051406716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 msgr2=0x7f21a4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 -- 172.21.15.116:0/1051406716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a419cfa0 con 0x7f21a4102180 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a91d3640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f219000e9f0 tx=0x7f219000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a1ffb640 1 -- 172.21.15.116:0/1051406716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219000cd80 con 0x7f21a4102180 2024-03-19T02:40:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a419d290 con 0x7f21a4102180 2024-03-19T02:40:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a1ffb640 1 -- 172.21.15.116:0/1051406716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21900108e0 con 0x7f21a4102180 2024-03-19T02:40:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21a1ffb640 1 -- 172.21.15.116:0/1051406716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190018920 con 0x7f21a4102180 2024-03-19T02:40:44.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.846+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a41a4b20 con 0x7f21a4102180 2024-03-19T02:40:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21a1ffb640 1 -- 172.21.15.116:0/1051406716 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2190010a90 con 0x7f21a4102180 2024-03-19T02:40:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 Processor -- start 2024-03-19T02:40:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 -- start start 2024-03-19T02:40:44.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981936b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d9819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d980f64b0 con 0x560d981932d0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560d9809ed70 con 0x560d98193bf0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21ad81e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d980f2550 con 0x560d9819c7d0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21a1ffb640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 0x7f21940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21a91d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.849+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d9819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49890/0 (socket says 172.21.15.116:49890) 2024-03-19T02:40:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 -- 172.21.15.116:0/1051364000 learned_addr learned my addr 172.21.15.116:0/1051364000 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a91d3640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 0x7f21940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a1ffb640 1 -- 172.21.15.116:0/1051406716 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f21900cf4a0 con 0x7f21a4102180 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 -- 172.21.15.116:0/1051364000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 msgr2=0x560d9819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d9819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 -- 172.21.15.116:0/1051364000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 msgr2=0x560d981936b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981936b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 -- 172.21.15.116:0/1051364000 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d980c1210 con 0x560d98193bf0 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a91d3640 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981936b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a89d2640 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2198036040 tx=0x7f2198009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4058c26e06e5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f21a91d3640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 0x7f21940a4e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f219c0094d0 tx=0x7f219c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f218effd640 1 -- 172.21.15.116:0/1051364000 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198045020 con 0x560d98193bf0 2024-03-19T02:40:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f218effd640 1 -- 172.21.15.116:0/1051364000 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f219802c7e0 con 0x560d98193bf0 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.850+0000 7f218effd640 1 -- 172.21.15.116:0/1051364000 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980054c0 con 0x560d98193bf0 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 -- 172.21.15.116:0/1051364000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 msgr2=0x560d9819c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2198036040 tx=0x7f2198009560 comp rx=0 tx=0).stop 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 -- 172.21.15.116:0/1051364000 shutdown_connections 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d9819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d98193bf0 0x560d9819c290 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1051364000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981936b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 -- 172.21.15.116:0/1051364000 >> 172.21.15.116:0/1051364000 conn(0x560d9818da00 msgr2=0x560d9818dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 -- 172.21.15.116:0/1051364000 shutdown_connections 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.851+0000 7f21ad81e740 1 -- 172.21.15.116:0/1051364000 wait complete. 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 -- start start 2024-03-19T02:40:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 Processor -- start 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d98193bf0 0x560d98199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d9819a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59442/0 (socket says 172.21.15.116:59442) 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d9819f560 con 0x560d981932d0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21a89d2640 1 -- 172.21.15.116:0/448080189 learned_addr learned my addr 172.21.15.116:0/448080189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560d980b2310 con 0x560d9819c7d0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560d9819f6e0 con 0x560d98193bf0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.852+0000 7f21a3fff640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d98193bf0 0x560d98199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21ad81e740 10 client.0 _ll_register_callbacks cb 0x560d980a62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a91d3640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d9819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 -- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d98193bf0 msgr2=0x560d98199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d98193bf0 0x560d98199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 -- 172.21.15.116:0/448080189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 msgr2=0x560d9819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d9819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d97e0a3f0 con 0x560d981932d0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21a89d2640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f21980054b0 tx=0x7f21980054e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6eda7e4fdfa8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f2183fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198050070 con 0x560d981932d0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f219802c7e0 con 0x560d981932d0 2024-03-19T02:40:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f2198004450 con 0x560d981932d0 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2198059050 con 0x560d981932d0 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21ad81e740 10 client.19341 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.853+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d97ed47d0 con 0x560d981932d0 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f219802c7e0 con 0x560d981932d0 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f2198053020 con 0x560d981932d0 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 10 client.19341 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 10 client.19341 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 20 client.19341 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d981ac910 0x560d981aed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 4 client.19341 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d981543e0 con 0x560d981ac910 2024-03-19T02:40:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21a3fff640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d981ac910 0x560d981aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f21a3fff640 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d981ac910 0x560d981aed00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=650c2d3270a797fe server_cookie=ee0db10cd1929b14 in_seq=0 out_seq=0 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.854+0000 7f2183fff640 10 client.19341 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.855+0000 7f2183fff640 1 -- 172.21.15.116:0/448080189 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d981543e0 con 0x560d981ac910 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.855+0000 7f2183fff640 10 client.19341 handle_command_reply: tid=0 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.855+0000 7f21ad81e740 1 client.19341 shutdown 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 msgr2=0x560d981997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f21980054b0 tx=0x7f21980054e0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 shutdown_connections 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d981ac910 0x560d981aed00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d9819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d98193bf0 0x560d98199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 --2- 172.21.15.116:0/448080189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d981932d0 0x560d981997b0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 >> 172.21.15.116:0/448080189 conn(0x560d9818da00 msgr2=0x560d9818f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f2183fff640 0 client.19341 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 shutdown_connections 2024-03-19T02:40:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 1 -- 172.21.15.116:0/448080189 wait complete. 2024-03-19T02:40:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.856+0000 7f21ad81e740 20 client.19341 trim_cache size 0 max 16384 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 Processor -- start 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 -- start start 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 0x560d981a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d981b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41aebe0 con 0x560d98193bf0 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b07b0 con 0x560d9819c7d0 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21ad81e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41af2b0 con 0x560d981932d0 2024-03-19T02:40:44.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 0x560d981a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59446/0 (socket says 172.21.15.116:59446) 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21a3fff640 1 -- 172.21.15.116:0/1622825903 learned_addr learned my addr 172.21.15.116:0/1622825903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.872+0000 7f21a91d3640 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d981b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 -- 172.21.15.116:0/1622825903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 msgr2=0x560d981a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 0x560d981a8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 -- 172.21.15.116:0/1622825903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 msgr2=0x560d981b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d981b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 -- 172.21.15.116:0/1622825903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d98193a20 con 0x560d98193bf0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a89d2640 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 0x560d981a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f21a3fff640 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f219000c6a0 tx=0x7f21900d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2adb59b0a289ade server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f218e7fc640 1 -- 172.21.15.116:0/1622825903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219009f910 con 0x560d98193bf0 2024-03-19T02:40:44.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f218e7fc640 1 -- 172.21.15.116:0/1622825903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21900a3f60 con 0x560d98193bf0 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.873+0000 7f218e7fc640 1 -- 172.21.15.116:0/1622825903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900cf120 con 0x560d98193bf0 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- 172.21.15.116:0/1622825903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 msgr2=0x560d9819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f219000c6a0 tx=0x7f21900d1040 comp rx=0 tx=0).stop 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- 172.21.15.116:0/1622825903 shutdown_connections 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d9819c7d0 0x560d981b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d9819a050 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 --2- 172.21.15.116:0/1622825903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d981932d0 0x560d981a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- 172.21.15.116:0/1622825903 >> 172.21.15.116:0/1622825903 conn(0x560d9818da00 msgr2=0x560d9818ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- 172.21.15.116:0/1622825903 shutdown_connections 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- 172.21.15.116:0/1622825903 wait complete. 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 -- start start 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.874+0000 7f21ad81e740 1 Processor -- start 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 0x560d981a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d98196580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41b0080 con 0x560d98193bf0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41af9b0 con 0x560d981932d0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41acbe0 con 0x560d9819c7d0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21ad81e740 10 client.0 _ll_register_callbacks cb 0x7f219c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 0x560d981a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59460/0 (socket says 172.21.15.116:59460) 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a89d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 0x560d981a4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49916/0 (socket says 172.21.15.116:49916) 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 -- 172.21.15.116:0/3751025841 learned_addr learned my addr 172.21.15.116:0/3751025841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a91d3640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d98196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 -- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 msgr2=0x560d98196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d98196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 -- 172.21.15.116:0/3751025841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 msgr2=0x560d981a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 0x560d981a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 -- 172.21.15.116:0/3751025841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d98196bc0 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a91d3640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d98196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f21a3fff640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f21900a3a30 tx=0x7f2190097c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4946910ccaa0a4df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f2183fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.875+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219009f910 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f21900cf120 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21900a3f60 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2190086070 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 10 client.19347 fetch_fsmap learned FSMap version 32 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d98212890 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21900a3f60 con 0x560d98193bf0 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8374+0+0 (secure 0 0 0) 0x7f21900afba0 con 0x560d98193bf0 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 10 client.19347 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 10 client.19347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 20 client.19347 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d98213040 0x560d98215460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 4 client.19347 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d981039f0 con 0x560d98213040 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.876+0000 7f21a89d2640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d98213040 0x560d98215460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f2183fff640 10 client.19347 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21a89d2640 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d98213040 0x560d98215460 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=38b012347c36de1f server_cookie=34ace48532eb186c in_seq=0 out_seq=0 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f2183fff640 1 -- 172.21.15.116:0/3751025841 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d981039f0 con 0x560d98213040 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f2183fff640 10 client.19347 handle_command_reply: tid=0 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 client.19347 shutdown 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 msgr2=0x560d98196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f21900a3a30 tx=0x7f2190097c40 comp rx=0 tx=0).stop 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 shutdown_connections 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x560d98213040 0x560d98215460 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560d9819c7d0 0x560d98196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d98193bf0 0x560d98196040 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 --2- 172.21.15.116:0/3751025841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560d981932d0 0x560d981a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 >> 172.21.15.116:0/3751025841 conn(0x560d9818da00 msgr2=0x560d9818cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f2183fff640 0 client.19347 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 shutdown_connections 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 1 -- 172.21.15.116:0/3751025841 wait complete. 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.877+0000 7f21ad81e740 20 client.19347 trim_cache size 0 max 16384 2024-03-19T02:40:44.885 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 msgr2=0x7f21940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 0x7f21940a4e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f219c0094d0 tx=0x7f219c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 msgr2=0x7f21a41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f219000e9f0 tx=0x7f219000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 shutdown_connections 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f21940a2970 0x7f21940a4e30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21a41077d0 0x7f21a419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a4102180 0x7f21a41984f0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 --2- 172.21.15.116:0/1051406716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21a41017c0 0x7f21a4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 >> 172.21.15.116:0/1051406716 conn(0x7f21a40fd2d0 msgr2=0x7f21a40fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 shutdown_connections 2024-03-19T02:40:44.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:44.878+0000 7f21aac36640 1 -- 172.21.15.116:0/1051406716 wait complete. 2024-03-19T02:40:44.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:40:45.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 -- 172.21.15.116:0/910136500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff0100290 msgr2=0x7efff010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 --2- 172.21.15.116:0/910136500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff0100290 0x7efff010e590 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7effe8009ca0 tx=0x7effe802c1d0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 -- 172.21.15.116:0/910136500 shutdown_connections 2024-03-19T02:40:45.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 --2- 172.21.15.116:0/910136500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff0100290 0x7efff010e590 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 --2- 172.21.15.116:0/910136500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff00ff8f0 0x7efff00ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 --2- 172.21.15.116:0/910136500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff01068d0 0x7efff0106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 -- 172.21.15.116:0/910136500 >> 172.21.15.116:0/910136500 conn(0x7efff00768b0 msgr2=0x7efff00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 -- 172.21.15.116:0/910136500 shutdown_connections 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.029+0000 7efff668a640 1 -- 172.21.15.116:0/910136500 wait complete. 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 Processor -- start 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 -- start start 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff0100290 0x7efff019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 0x7efff019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efff0111a50 con 0x7efff00ff8f0 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efff01118d0 con 0x7efff01068d0 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7efff668a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efff0111bd0 con 0x7efff0100290 2024-03-19T02:40:45.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.030+0000 7effeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7effeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59476/0 (socket says 172.21.15.116:59476) 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7effeffff640 1 -- 172.21.15.116:0/3748401252 learned_addr learned my addr 172.21.15.116:0/3748401252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7effef7fe640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff0100290 0x7efff019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 0x7efff019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 -- 172.21.15.116:0/3748401252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff0100290 msgr2=0x7efff019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff0100290 0x7efff019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 -- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 msgr2=0x7efff019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 -- 172.21.15.116:0/3748401252 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff019f180 con 0x7efff01068d0 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7effeffff640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.031+0000 7efff4c27640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 0x7efff019ea50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7effe8009bf0 tx=0x7effe80323b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.032+0000 7effed7fa640 1 -- 172.21.15.116:0/3748401252 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8039040 con 0x7efff01068d0 2024-03-19T02:40:45.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.032+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff019f410 con 0x7efff01068d0 2024-03-19T02:40:45.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.032+0000 7effed7fa640 1 -- 172.21.15.116:0/3748401252 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe8044070 con 0x7efff01068d0 2024-03-19T02:40:45.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.032+0000 7effed7fa640 1 -- 172.21.15.116:0/3748401252 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe80327e0 con 0x7efff01068d0 2024-03-19T02:40:45.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.032+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff019f860 con 0x7efff01068d0 2024-03-19T02:40:45.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.035+0000 7effed7fa640 1 -- 172.21.15.116:0/3748401252 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7effe8032980 con 0x7efff01068d0 2024-03-19T02:40:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.035+0000 7effed7fa640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 0x7effd80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.035+0000 7effed7fa640 1 -- 172.21.15.116:0/3748401252 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7effe80f13a0 con 0x7efff01068d0 2024-03-19T02:40:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effeffff640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 0x7effd80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 Processor -- start 2024-03-19T02:40:45.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 -- start start 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 0x555ab2a60aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a60fe0 0x555ab2a633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ab29b24b0 con 0x555ab2a57ae0 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555ab295ad70 con 0x555ab2a60fe0 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff9271740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ab29ae550 con 0x555ab2a58400 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 0x555ab2a60aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59490/0 (socket says 172.21.15.116:59490) 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effeffff640 1 -- 172.21.15.116:0/3986541925 learned_addr learned my addr 172.21.15.116:0/3986541925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7efff4c27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a60fe0 0x555ab2a633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effeffff640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 0x7effd80a4eb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effe0011490 tx=0x7effe0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.036+0000 7effef7fe640 1 -- 172.21.15.116:0/3986541925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a60fe0 msgr2=0x555ab2a633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effef7fe640 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a60fe0 0x555ab2a633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effef7fe640 1 -- 172.21.15.116:0/3986541925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 msgr2=0x555ab2a57ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effef7fe640 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effef7fe640 1 -- 172.21.15.116:0/3986541925 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ab297d210 con 0x555ab2a58400 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effeffff640 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effef7fe640 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 0x555ab2a60aa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7effe4009bf0 tx=0x7effe4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80d44a90df87cef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effd67fc640 1 -- 172.21.15.116:0/3986541925 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4004050 con 0x555ab2a58400 2024-03-19T02:40:45.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.037+0000 7effd67fc640 1 -- 172.21.15.116:0/3986541925 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe40054d0 con 0x555ab2a58400 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7effd67fc640 1 -- 172.21.15.116:0/3986541925 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4004d10 con 0x555ab2a58400 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 -- 172.21.15.116:0/3986541925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 msgr2=0x555ab2a60aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 0x555ab2a60aa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7effe4009bf0 tx=0x7effe4011810 comp rx=0 tx=0).stop 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 -- 172.21.15.116:0/3986541925 shutdown_connections 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a60fe0 0x555ab2a633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a58400 0x555ab2a60aa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 --2- 172.21.15.116:0/3986541925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a57ae0 0x555ab2a57ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 -- 172.21.15.116:0/3986541925 >> 172.21.15.116:0/3986541925 conn(0x555ab2a52210 msgr2=0x555ab2a525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 -- 172.21.15.116:0/3986541925 shutdown_connections 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.038+0000 7efff9271740 1 -- 172.21.15.116:0/3986541925 wait complete. 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.039+0000 7efff9271740 1 -- start start 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.039+0000 7efff9271740 1 Processor -- start 2024-03-19T02:40:45.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.039+0000 7efff9271740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a5df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ab2a63e20 con 0x555ab2a58400 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a5df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59506/0 (socket says 172.21.15.116:59506) 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 -- 172.21.15.116:0/750763491 learned_addr learned my addr 172.21.15.116:0/750763491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff4c27640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555ab296e310 con 0x555ab2a57ae0 2024-03-19T02:40:45.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ab2a63fa0 con 0x555ab2a60fe0 2024-03-19T02:40:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7efff9271740 10 client.0 _ll_register_callbacks cb 0x555ab29622d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 -- 172.21.15.116:0/750763491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 msgr2=0x555ab2a5e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 -- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 msgr2=0x555ab2a5df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a5df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.040+0000 7effef7fe640 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ab26c63f0 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7effeffff640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a5df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7effef7fe640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7effe4009730 tx=0x7effe4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3909a2156aba9885 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7effcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4029070 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7effe4003a00 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.041+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effe40054d0 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7effe4035050 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 10 client.19359 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555ab27907d0 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe40046c0 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7effe402e440 con 0x555ab2a58400 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 10 client.19359 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:45.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 10 client.19359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 20 client.19359 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2a711d0 0x555ab2a73620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 4 client.19359 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7effe40054d0 con 0x555ab2a711d0 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.042+0000 7effeffff640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2a711d0 0x555ab2a73620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.043+0000 7effcf7fe640 10 client.19359 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.043+0000 7effeffff640 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2a711d0 0x555ab2a73620 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4814dad3823b788 server_cookie=3b7b7b81e1e5f9e4 in_seq=0 out_seq=0 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.044+0000 7effcf7fe640 1 -- 172.21.15.116:0/750763491 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7effe40054d0 con 0x555ab2a711d0 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7effcf7fe640 10 client.19359 handle_command_reply: tid=0 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 client.19359 shutdown 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 msgr2=0x555ab2a5e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7effe4009730 tx=0x7effe4009fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 shutdown_connections 2024-03-19T02:40:45.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2a711d0 0x555ab2a73620 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5e470 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 --2- 172.21.15.116:0/750763491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a5df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 >> 172.21.15.116:0/750763491 conn(0x555ab2a52210 msgr2=0x555ab2a48ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7effcf7fe640 0 client.19359 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 shutdown_connections 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 1 -- 172.21.15.116:0/750763491 wait complete. 2024-03-19T02:40:45.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.045+0000 7efff9271740 20 client.19359 trim_cache size 0 max 16384 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 Processor -- start 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 -- start start 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 0x555ab2a6cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 0x555ab2a5d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efff01ac960 con 0x555ab2a60fe0 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efff01ae590 con 0x555ab2a58400 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff9271740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efff01ad030 con 0x555ab2a57ae0 2024-03-19T02:40:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7effeffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7effeffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41326/0 (socket says 172.21.15.116:41326) 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7effeffff640 1 -- 172.21.15.116:0/3717930541 learned_addr learned my addr 172.21.15.116:0/3717930541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7efff4c27640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 0x555ab2a5d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.069+0000 7effef7fe640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 0x555ab2a6cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 -- 172.21.15.116:0/3717930541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 msgr2=0x555ab2a6cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 0x555ab2a6cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 -- 172.21.15.116:0/3717930541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 msgr2=0x555ab2a5d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 0x555ab2a5d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 -- 172.21.15.116:0/3717930541 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ab2a52040 con 0x555ab2a57ae0 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effef7fe640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 0x555ab2a6cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7efff4c27640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 0x555ab2a5d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effeffff640 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7effe0010010 tx=0x7effe0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16d29efae03cbe9e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effd5ffb640 1 -- 172.21.15.116:0/3717930541 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0031070 con 0x555ab2a57ae0 2024-03-19T02:40:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effd5ffb640 1 -- 172.21.15.116:0/3717930541 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe0028410 con 0x555ab2a57ae0 2024-03-19T02:40:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.070+0000 7effd5ffb640 1 -- 172.21.15.116:0/3717930541 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe00275d0 con 0x555ab2a57ae0 2024-03-19T02:40:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 -- 172.21.15.116:0/3717930541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 msgr2=0x555ab2a60660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7effe0010010 tx=0x7effe0014000 comp rx=0 tx=0).stop 2024-03-19T02:40:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 -- 172.21.15.116:0/3717930541 shutdown_connections 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a60fe0 0x555ab2a5d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a58400 0x555ab2a6cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 --2- 172.21.15.116:0/3717930541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a57ae0 0x555ab2a60660 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 -- 172.21.15.116:0/3717930541 >> 172.21.15.116:0/3717930541 conn(0x555ab2a52210 msgr2=0x555ab2a48310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 -- 172.21.15.116:0/3717930541 shutdown_connections 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.071+0000 7efff9271740 1 -- 172.21.15.116:0/3717930541 wait complete. 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 -- start start 2024-03-19T02:40:45.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 Processor -- start 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efff01ade00 con 0x555ab2a58400 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efff01ad730 con 0x555ab2a57ae0 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efff01aa9a0 con 0x555ab2a60fe0 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7effeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.072+0000 7efff9271740 10 client.0 _ll_register_callbacks cb 0x7effe0009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49972/0 (socket says 172.21.15.116:49972) 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7efff4c27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 -- 172.21.15.116:0/2917275243 learned_addr learned my addr 172.21.15.116:0/2917275243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effef7fe640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 -- 172.21.15.116:0/2917275243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 msgr2=0x555ab2a5aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 -- 172.21.15.116:0/2917275243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 msgr2=0x555ab2a5a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7effeffff640 1 -- 172.21.15.116:0/2917275243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ab2a5aff0 con 0x555ab2a57ae0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.073+0000 7efff4c27640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7effeffff640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7effe0005750 tx=0x7effe000fa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d2ab1d6a795830 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7effcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0031070 con 0x555ab2a57ae0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effe0028410 con 0x555ab2a57ae0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7effe0028780 con 0x555ab2a57ae0 2024-03-19T02:40:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7effe0040050 con 0x555ab2a57ae0 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7efff9271740 10 client.29141 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.074+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555ab2ad7160 con 0x555ab2a57ae0 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effe0028410 con 0x555ab2a57ae0 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7effe0035520 con 0x555ab2a57ae0 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 10 client.29141 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 10 client.29141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 20 client.29141 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2ad77d0 0x555ab2ad9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 4 client.29141 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555ab2a103e0 con 0x555ab2ad77d0 2024-03-19T02:40:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7effef7fe640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2ad77d0 0x555ab2ad9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.075+0000 7effcf7fe640 10 client.29141 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.076+0000 7effef7fe640 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2ad77d0 0x555ab2ad9bf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68445b0c9e5b6a4f server_cookie=fa3ead8368ec4556 in_seq=0 out_seq=0 2024-03-19T02:40:45.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7effcf7fe640 1 -- 172.21.15.116:0/2917275243 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555ab2a103e0 con 0x555ab2ad77d0 2024-03-19T02:40:45.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7effcf7fe640 10 client.29141 handle_command_reply: tid=0 2024-03-19T02:40:45.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 client.29141 shutdown 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 msgr2=0x555ab2a69240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7effe0005750 tx=0x7effe000fa90 comp rx=0 tx=0).stop 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 shutdown_connections 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x555ab2ad77d0 0x555ab2ad9bf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ab2a60fe0 0x555ab2a5aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ab2a58400 0x555ab2a5a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 --2- 172.21.15.116:0/2917275243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ab2a57ae0 0x555ab2a69240 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 >> 172.21.15.116:0/2917275243 conn(0x555ab2a52210 msgr2=0x555ab2a48ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7effcf7fe640 0 client.29141 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 shutdown_connections 2024-03-19T02:40:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 1 -- 172.21.15.116:0/2917275243 wait complete. 2024-03-19T02:40:45.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.078+0000 7efff9271740 20 client.29141 trim_cache size 0 max 16384 2024-03-19T02:40:45.084 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 msgr2=0x7effd80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 0x7effd80a4eb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7effe0011490 tx=0x7effe0005f90 comp rx=0 tx=0).stop 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 msgr2=0x7efff019ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 0x7efff019ea50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7effe8009bf0 tx=0x7effe80323b0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 shutdown_connections 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7effd80a29f0 0x7effd80a4eb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efff01068d0 0x7efff019ea50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff0100290 0x7efff019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.080+0000 7efff668a640 1 --2- 172.21.15.116:0/3748401252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efff00ff8f0 0x7efff019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.081+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 >> 172.21.15.116:0/3748401252 conn(0x7efff00768b0 msgr2=0x7efff010d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.081+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 shutdown_connections 2024-03-19T02:40:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:45.081+0000 7efff668a640 1 -- 172.21.15.116:0/3748401252 wait complete. 2024-03-19T02:40:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:44 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3793689430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:44 smithi154 ceph-mon[35693]: pgmap v405: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 19 MiB/s wr, 215 op/s 2024-03-19T02:40:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:44 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3793689430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:44 smithi116 ceph-mon[32263]: pgmap v405: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 19 MiB/s wr, 215 op/s 2024-03-19T02:40:45.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1135 32.81 MB/sec warmup 7 sec latency 191.972 ms 2024-03-19T02:40:45.420 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:44 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:45.420 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3793689430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:40:45.420 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:44 smithi162 ceph-mon[35761]: pgmap v405: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 19 MiB/s wr, 215 op/s 2024-03-19T02:40:46.096 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:40:46.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:40:46.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:40:46.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 -- 172.21.15.116:0/2582384575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 msgr2=0x7f55cc10e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 --2- 172.21.15.116:0/2582384575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc10e640 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009ca0 tx=0x7f55c402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 -- 172.21.15.116:0/2582384575 shutdown_connections 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 --2- 172.21.15.116:0/2582384575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc10e640 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 --2- 172.21.15.116:0/2582384575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 --2- 172.21.15.116:0/2582384575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc1077c0 0x7f55cc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 -- 172.21.15.116:0/2582384575 >> 172.21.15.116:0/2582384575 conn(0x7f55cc076c40 msgr2=0x7f55cc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 -- 172.21.15.116:0/2582384575 shutdown_connections 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.239+0000 7f55d2adc640 1 -- 172.21.15.116:0/2582384575 wait complete. 2024-03-19T02:40:46.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 Processor -- start 2024-03-19T02:40:46.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 -- start start 2024-03-19T02:40:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc1077c0 0x7f55cc101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc111a60 con 0x7f55cc100680 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1118e0 con 0x7f55cc1077c0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.240+0000 7f55d2adc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc111be0 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41356/0 (socket says 172.21.15.116:41356) 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59528/0 (socket says 172.21.15.116:59528) 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 -- 172.21.15.116:0/1101015735 learned_addr learned my addr 172.21.15.116:0/1101015735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d1079640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc1077c0 0x7f55cc101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 -- 172.21.15.116:0/1101015735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc1077c0 msgr2=0x7f55cc101920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc1077c0 0x7f55cc101920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 -- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 msgr2=0x7f55cc1013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 -- 172.21.15.116:0/1101015735 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc1a6460 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55cbfff640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d0878640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55c000e9f0 tx=0x7f55c000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55c9ffb640 1 -- 172.21.15.116:0/1101015735 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c000cd80 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55cc1a6750 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55c9ffb640 1 -- 172.21.15.116:0/1101015735 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55c00108e0 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.241+0000 7f55c9ffb640 1 -- 172.21.15.116:0/1101015735 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0018920 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.242+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55cc1a6cf0 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.244+0000 7f55c9ffb640 1 -- 172.21.15.116:0/1101015735 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f55c0010a90 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 Processor -- start 2024-03-19T02:40:46.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 -- start start 2024-03-19T02:40:46.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55c9ffb640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 0x7f55bc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55c9ffb640 1 -- 172.21.15.116:0/1101015735 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f55c00cf530 con 0x7f55cc0ffcc0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d1079640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 0x7f55bc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030380 0x55e46d030760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d039880 0x55e46d03bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e46cf94160 con 0x55e46d030ca0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e46cf3ce10 con 0x55e46d030380 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55d57a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e46cf90200 con 0x55e46d039880 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.245+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59544/0 (socket says 172.21.15.116:59544) 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 -- 172.21.15.116:0/1157920561 learned_addr learned my addr 172.21.15.116:0/1157920561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55d1079640 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d039880 0x55e46d03bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55d0878640 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030380 0x55e46d030760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 -- 172.21.15.116:0/1157920561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d039880 msgr2=0x55e46d03bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d039880 0x55e46d03bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 -- 172.21.15.116:0/1157920561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030380 msgr2=0x55e46d030760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030380 0x55e46d030760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 -- 172.21.15.116:0/1157920561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46cd707e0 con 0x55e46d030ca0 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55d1079640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 0x7f55bc0a4eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005cc0 tx=0x7f55c4032dd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55cbfff640 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f55b8011470 tx=0x7f55b8011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e301221a32cae480 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55b6ffd640 1 -- 172.21.15.116:0/1157920561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b801e070 con 0x55e46d030ca0 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55b6ffd640 1 -- 172.21.15.116:0/1157920561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b80035f0 con 0x55e46d030ca0 2024-03-19T02:40:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.246+0000 7f55b6ffd640 1 -- 172.21.15.116:0/1157920561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b800f740 con 0x55e46d030ca0 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- 172.21.15.116:0/1157920561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 msgr2=0x55e46d039340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f55b8011470 tx=0x7f55b8011930 comp rx=0 tx=0).stop 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- 172.21.15.116:0/1157920561 shutdown_connections 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d039880 0x55e46d03bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d039340 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1157920561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030380 0x55e46d030760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- 172.21.15.116:0/1157920561 >> 172.21.15.116:0/1157920561 conn(0x55e46d02aab0 msgr2=0x55e46d02ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- 172.21.15.116:0/1157920561 shutdown_connections 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- 172.21.15.116:0/1157920561 wait complete. 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.247+0000 7f55d57a1740 1 -- start start 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 Processor -- start 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d036d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e46d03c670 con 0x55e46d030ca0 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e46cf501a0 con 0x55e46d039880 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e46d03c7f0 con 0x55e46d030380 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d57a1740 10 client.0 _ll_register_callbacks cb 0x55e46cca7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d036d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d1079640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d1079640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:50020/0 (socket says 172.21.15.116:50020) 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41368/0 (socket says 172.21.15.116:41368) 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d036d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59558/0 (socket says 172.21.15.116:59558) 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.248+0000 7f55d1079640 1 -- 172.21.15.116:0/31364880 learned_addr learned my addr 172.21.15.116:0/31364880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 -- 172.21.15.116:0/31364880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 msgr2=0x55e46d0367e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 -- 172.21.15.116:0/31364880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 msgr2=0x55e46d036d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d036d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 -- 172.21.15.116:0/31364880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46cdd0f70 con 0x55e46d039880 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d0878640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d1079640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f55c4032780 tx=0x7f55c402c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c80ea6e356c6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55abfff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4050020 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b80035f0 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f55c4004240 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f55c405b050 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d57a1740 10 client.29153 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.249+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e46cc68170 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b80035f0 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f55c4034a20 con 0x55e46d039880 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 10 client.29153 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 10 client.29153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 20 client.29153 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d049c30 0x55e46d04c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 4 client.29153 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e46cfa16a0 con 0x55e46d049c30 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.250+0000 7f55d0878640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d049c30 0x55e46d04c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.251+0000 7f55abfff640 10 client.29153 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.251+0000 7f55d0878640 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d049c30 0x55e46d04c090 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c35e83245670ad4b server_cookie=25be3ed0c32383c8 in_seq=0 out_seq=0 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.251+0000 7f55abfff640 1 -- 172.21.15.116:0/31364880 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e46cfa16a0 con 0x55e46d049c30 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.251+0000 7f55abfff640 10 client.29153 handle_command_reply: tid=0 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.251+0000 7f55d57a1740 1 client.29153 shutdown 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 msgr2=0x55e46d037260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f55c4032780 tx=0x7f55c402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 shutdown_connections 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d049c30 0x55e46d04c090 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d037260 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d036d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 --2- 172.21.15.116:0/31364880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 >> 172.21.15.116:0/31364880 conn(0x55e46d02aab0 msgr2=0x55e46d02c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55abfff640 0 client.29153 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 shutdown_connections 2024-03-19T02:40:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 1 -- 172.21.15.116:0/31364880 wait complete. 2024-03-19T02:40:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.252+0000 7f55d57a1740 20 client.29153 trim_cache size 0 max 16384 2024-03-19T02:40:46.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:45 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:45 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 Processor -- start 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 -- start start 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d038f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d051960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1b0dd0 con 0x55e46d030ca0 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1b2a00 con 0x55e46d039880 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d57a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1b14a0 con 0x55e46d030380 2024-03-19T02:40:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d038f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.274+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59566/0 (socket says 172.21.15.116:59566) 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d038f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41370/0 (socket says 172.21.15.116:41370) 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 -- 172.21.15.116:0/3606771895 learned_addr learned my addr 172.21.15.116:0/3606771895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55d1079640 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d051960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 -- 172.21.15.116:0/3606771895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 msgr2=0x55e46d038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d038f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 -- 172.21.15.116:0/3606771895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 msgr2=0x55e46d051960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d051960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 -- 172.21.15.116:0/3606771895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46d048700 con 0x55e46d030ca0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55cbfff640 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f55b801c040 tx=0x7f55b8005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82327891764b5e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55b67fc640 1 -- 172.21.15.116:0/3606771895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8027020 con 0x55e46d030ca0 2024-03-19T02:40:46.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55b67fc640 1 -- 172.21.15.116:0/3606771895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b801f040 con 0x55e46d030ca0 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.275+0000 7f55b67fc640 1 -- 172.21.15.116:0/3606771895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b801e040 con 0x55e46d030ca0 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- 172.21.15.116:0/3606771895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 msgr2=0x55e46d045410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f55b801c040 tx=0x7f55b8005e10 comp rx=0 tx=0).stop 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- 172.21.15.116:0/3606771895 shutdown_connections 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d039880 0x55e46d051960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d030ca0 0x55e46d045410 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 --2- 172.21.15.116:0/3606771895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d038f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- 172.21.15.116:0/3606771895 >> 172.21.15.116:0/3606771895 conn(0x55e46d02aab0 msgr2=0x55e46d02c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- 172.21.15.116:0/3606771895 shutdown_connections 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- 172.21.15.116:0/3606771895 wait complete. 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 -- start start 2024-03-19T02:40:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.276+0000 7f55d57a1740 1 Processor -- start 2024-03-19T02:40:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030ca0 0x55e46d0a8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1b2270 con 0x55e46d039880 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1b1ba0 con 0x55e46d030ca0 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc1aedd0 con 0x55e46d030380 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 10 client.0 _ll_register_callbacks cb 0x7f55c4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d0878640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59582/0 (socket says 172.21.15.116:59582) 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55cbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030ca0 0x55e46d0a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 -- 172.21.15.116:0/1156741191 learned_addr learned my addr 172.21.15.116:0/1156741191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 -- 172.21.15.116:0/1156741191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 msgr2=0x55e46d0a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 -- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030ca0 msgr2=0x55e46d0a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030ca0 0x55e46d0a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 -- 172.21.15.116:0/1156741191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46d02a8e0 con 0x55e46d039880 2024-03-19T02:40:46.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d1079640 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009600 tx=0x7f55c4032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7a6693fbe12dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4064060 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f55c405f430 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.277+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b801f040 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f55c4050040 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 10 client.19383 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e46d0b09c0 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b801f040 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f55c404c410 con 0x55e46d039880 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 10 client.19383 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:46.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 10 client.19383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 20 client.19383 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d0b11a0 0x55e46d0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 4 client.19383 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e46cf9b9f0 con 0x55e46d0b11a0 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d0878640 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d0b11a0 0x55e46d0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55abfff640 10 client.19383 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.278+0000 7f55d0878640 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d0b11a0 0x55e46d0b35c0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=672886af8ea7832 server_cookie=24bf59a70a310844 in_seq=0 out_seq=0 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55abfff640 1 -- 172.21.15.116:0/1156741191 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e46cf9b9f0 con 0x55e46d0b11a0 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55abfff640 10 client.19383 handle_command_reply: tid=0 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 client.19383 shutdown 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 msgr2=0x55e46d03dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009600 tx=0x7f55c4032d60 comp rx=0 tx=0).stop 2024-03-19T02:40:46.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 shutdown_connections 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e46d0b11a0 0x55e46d0b35c0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e46d039880 0x55e46d03dc10 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e46d030ca0 0x55e46d0a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 --2- 172.21.15.116:0/1156741191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e46d030380 0x55e46d0a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 >> 172.21.15.116:0/1156741191 conn(0x55e46d02aab0 msgr2=0x55e46d0292a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.280+0000 7f55abfff640 0 client.19383 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 shutdown_connections 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d57a1740 1 -- 172.21.15.116:0/1156741191 wait complete. 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d57a1740 20 client.19383 trim_cache size 0 max 16384 2024-03-19T02:40:46.287 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:46.288 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:40:46.288 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "a58fc1e5-fc9d-4f33-a307-19a8c7a52acb", 2024-03-19T02:40:46.288 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:40:46.288 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 msgr2=0x7f55bc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 0x7f55bc0a4eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005cc0 tx=0x7f55c4032dd0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 msgr2=0x7f55cc100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55c000e9f0 tx=0x7f55c000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 shutdown_connections 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55bc0a29f0 0x7f55bc0a4eb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55cc1077c0 0x7f55cc101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc100680 0x7f55cc1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 --2- 172.21.15.116:0/1101015735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55cc0ffcc0 0x7f55cc100ea0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 >> 172.21.15.116:0/1101015735 conn(0x7f55cc076c40 msgr2=0x7f55cc0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 shutdown_connections 2024-03-19T02:40:46.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.281+0000 7f55d2adc640 1 -- 172.21.15.116:0/1101015735 wait complete. 2024-03-19T02:40:46.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:40:46.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:40:46.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1231 29.29 MB/sec warmup 8 sec latency 208.465 ms 2024-03-19T02:40:46.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 -- 172.21.15.116:0/2663793011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34541066a0 msgr2=0x7f3454106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 --2- 172.21.15.116:0/2663793011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34541066a0 0x7f3454106a80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3450009930 tx=0x7f345002bc30 comp rx=0 tx=0).stop 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 -- 172.21.15.116:0/2663793011 shutdown_connections 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 --2- 172.21.15.116:0/2663793011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454101050 0x7f345410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 --2- 172.21.15.116:0/2663793011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 0x7f3454100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 --2- 172.21.15.116:0/2663793011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34541066a0 0x7f3454106a80 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 -- 172.21.15.116:0/2663793011 >> 172.21.15.116:0/2663793011 conn(0x7f3454077820 msgr2=0x7f3454077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 -- 172.21.15.116:0/2663793011 shutdown_connections 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.410+0000 7f345b89a640 1 -- 172.21.15.116:0/2663793011 wait complete. 2024-03-19T02:40:46.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 Processor -- start 2024-03-19T02:40:46.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 -- start start 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 0x7f3454198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34541066a0 0x7f34541a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3454110920 con 0x7f3454101050 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34541107a0 con 0x7f34541066a0 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f345b89a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3454110aa0 con 0x7f3454100690 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f3458e35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f3458e35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59592/0 (socket says 172.21.15.116:59592) 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f3458e35640 1 -- 172.21.15.116:0/3921996742 learned_addr learned my addr 172.21.15.116:0/3921996742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f3459e37640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34541066a0 0x7f34541a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.411+0000 7f3459636640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 0x7f3454198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 -- 172.21.15.116:0/3921996742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 msgr2=0x7f3454198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 0x7f3454198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 -- 172.21.15.116:0/3921996742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34541066a0 msgr2=0x7f34541a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34541066a0 0x7f34541a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 -- 172.21.15.116:0/3921996742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34541a43c0 con 0x7f3454101050 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f3458e35640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f344400e9f0 tx=0x7f344400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f344a7fc640 1 -- 172.21.15.116:0/3921996742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344400cd80 con 0x7f3454101050 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f344a7fc640 1 -- 172.21.15.116:0/3921996742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34440108e0 con 0x7f3454101050 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f344a7fc640 1 -- 172.21.15.116:0/3921996742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444018920 con 0x7f3454101050 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34541a4590 con 0x7f3454101050 2024-03-19T02:40:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.412+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34541a4b30 con 0x7f3454101050 2024-03-19T02:40:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.414+0000 7f344a7fc640 1 -- 172.21.15.116:0/3921996742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3444010a90 con 0x7f3454101050 2024-03-19T02:40:46.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:45 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.415+0000 7f345e563740 1 Processor -- start 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.415+0000 7f345e563740 1 -- start start 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.415+0000 7f345e563740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61812e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.415+0000 7f345e563740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd6181b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.415+0000 7f345e563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd6181e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f345e563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd61776500 con 0x55dd6181bf30 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f345e563740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd6171ebe0 con 0x55dd61812a30 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f345e563740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd617725a0 con 0x55dd61813350 2024-03-19T02:40:46.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f344a7fc640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 0x7f34400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3458e35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd6181e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459e37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61812e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd6181b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459e37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61812e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:50062/0 (socket says 172.21.15.116:50062) 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459e37640 1 -- 172.21.15.116:0/2243499049 learned_addr learned my addr 172.21.15.116:0/2243499049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459e37640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 0x7f34400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f344a7fc640 1 -- 172.21.15.116:0/3921996742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f34440cf540 con 0x7f3454101050 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 -- 172.21.15.116:0/2243499049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 msgr2=0x55dd61812e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61812e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 -- 172.21.15.116:0/2243499049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 msgr2=0x55dd6181e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd6181e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 -- 172.21.15.116:0/2243499049 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd61740000 con 0x55dd61813350 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459e37640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 0x7f34400a4eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f344c005c40 tx=0x7f344c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.416+0000 7f3459636640 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd6181b9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3450009230 tx=0x7f3450009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7f714713cfcb0f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.417+0000 7f343f7fe640 1 -- 172.21.15.116:0/2243499049 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450004040 con 0x55dd61813350 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.417+0000 7f343f7fe640 1 -- 172.21.15.116:0/2243499049 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f345002cc90 con 0x55dd61813350 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.417+0000 7f343f7fe640 1 -- 172.21.15.116:0/2243499049 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500054d0 con 0x55dd61813350 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.417+0000 7f345e563740 1 -- 172.21.15.116:0/2243499049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 msgr2=0x55dd6181b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.417+0000 7f345e563740 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd6181b9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3450009230 tx=0x7f3450009560 comp rx=0 tx=0).stop 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 -- 172.21.15.116:0/2243499049 shutdown_connections 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd6181e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd6181b9f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 --2- 172.21.15.116:0/2243499049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61812e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 -- 172.21.15.116:0/2243499049 >> 172.21.15.116:0/2243499049 conn(0x55dd6180d160 msgr2=0x55dd6180d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 -- 172.21.15.116:0/2243499049 shutdown_connections 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 -- 172.21.15.116:0/2243499049 wait complete. 2024-03-19T02:40:46.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 -- start start 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 Processor -- start 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.418+0000 7f345e563740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61819490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618199d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd6181ecc0 con 0x55dd6181bf30 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd61732160 con 0x55dd61812a30 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd6181ee40 con 0x55dd61813350 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:50070/0 (socket says 172.21.15.116:50070) 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 -- 172.21.15.116:0/446338198 learned_addr learned my addr 172.21.15.116:0/446338198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 10 client.0 _ll_register_callbacks cb 0x55dd6176ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3458e35640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61819490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459e37640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 -- 172.21.15.116:0/446338198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 msgr2=0x55dd61819490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61819490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 -- 172.21.15.116:0/446338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 msgr2=0x55dd618199d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618199d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 -- 172.21.15.116:0/446338198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd615547d0 con 0x55dd61812a30 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f3459636640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3450036040 tx=0x7f34500044c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9383c89b93c1f24e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f343cff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450051070 con 0x55dd61812a30 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.419+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f345002cc90 con 0x55dd61812a30 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.420+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f3450005560 con 0x55dd61812a30 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.420+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f345002caf0 con 0x55dd61812a30 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.420+0000 7f345e563740 10 client.29171 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:46.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.420+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd6148a3f0 con 0x55dd61812a30 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.420+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f345002cc90 con 0x55dd61812a30 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f3450003ae0 con 0x55dd61812a30 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 10 client.29171 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 10 client.29171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 20 client.29171 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd6182e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 4 client.29171 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd61783a40 con 0x55dd6182be90 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f3458e35640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd6182e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f3458e35640 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd6182e280 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c85eeb4afddac2b3 server_cookie=e94c1dd063461136 in_seq=0 out_seq=0 2024-03-19T02:40:46.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.421+0000 7f343cff9640 10 client.29171 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f343cff9640 1 -- 172.21.15.116:0/446338198 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd61783a40 con 0x55dd6182be90 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f343cff9640 10 client.29171 handle_command_reply: tid=0 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 client.29171 shutdown 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 msgr2=0x55dd61818f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3450036040 tx=0x7f34500044c0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 shutdown_connections 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd6182e280 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618199d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f343cff9640 0 client.29171 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61819490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 --2- 172.21.15.116:0/446338198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd61818f50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.425+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 >> 172.21.15.116:0/446338198 conn(0x55dd6180d160 msgr2=0x55dd6180eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.426+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 shutdown_connections 2024-03-19T02:40:46.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.426+0000 7f345e563740 1 -- 172.21.15.116:0/446338198 wait complete. 2024-03-19T02:40:46.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.426+0000 7f345e563740 20 client.29171 trim_cache size 0 max 16384 2024-03-19T02:40:46.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.446+0000 7f345e563740 1 Processor -- start 2024-03-19T02:40:46.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.446+0000 7f345e563740 1 -- start start 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd6181b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618197a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34541ac9d0 con 0x55dd6181bf30 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34541ae600 con 0x55dd61812a30 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f345e563740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f34541ad0a0 con 0x55dd61813350 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3459e37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41418/0 (socket says 172.21.15.116:41418) 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3459636640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd6181b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 -- 172.21.15.116:0/4091129559 learned_addr learned my addr 172.21.15.116:0/4091129559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3459e37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618197a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59628/0 (socket says 172.21.15.116:59628) 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 -- 172.21.15.116:0/4091129559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 msgr2=0x55dd6181b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd6181b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 -- 172.21.15.116:0/4091129559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 msgr2=0x55dd618197a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618197a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 -- 172.21.15.116:0/4091129559 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd6144a0f0 con 0x55dd61813350 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.447+0000 7f3458e35640 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3444097c70 tx=0x7f3444097af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4974e5380e5018 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f343effd640 1 -- 172.21.15.116:0/4091129559 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444026040 con 0x55dd61813350 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f343effd640 1 -- 172.21.15.116:0/4091129559 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3444099b10 con 0x55dd61813350 2024-03-19T02:40:46.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f343effd640 1 -- 172.21.15.116:0/4091129559 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34440ab490 con 0x55dd61813350 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 -- 172.21.15.116:0/4091129559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 msgr2=0x55dd61827a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3444097c70 tx=0x7f3444097af0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 -- 172.21.15.116:0/4091129559 shutdown_connections 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd618197a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61813350 0x55dd61827a60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 --2- 172.21.15.116:0/4091129559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61812a30 0x55dd6181b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 -- 172.21.15.116:0/4091129559 >> 172.21.15.116:0/4091129559 conn(0x55dd6180d160 msgr2=0x55dd61817e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 -- 172.21.15.116:0/4091129559 shutdown_connections 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.448+0000 7f345e563740 1 -- 172.21.15.116:0/4091129559 wait complete. 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 -- start start 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 Processor -- start 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61813350 0x55dd6188b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd61820180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34541ade70 con 0x55dd6181bf30 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34541ad7a0 con 0x55dd61813350 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f34541aa9d0 con 0x55dd61812a30 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f345e563740 10 client.0 _ll_register_callbacks cb 0x7f344409b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3459e37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd61820180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3459636640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3459636640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41434/0 (socket says 172.21.15.116:41434) 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3459636640 1 -- 172.21.15.116:0/2798178137 learned_addr learned my addr 172.21.15.116:0/2798178137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3459e37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd61820180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59632/0 (socket says 172.21.15.116:59632) 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.449+0000 7f3458e35640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61813350 0x55dd6188b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 -- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61813350 msgr2=0x55dd6188b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61813350 0x55dd6188b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 -- 172.21.15.116:0/2798178137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 msgr2=0x55dd61820180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd61820180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 -- 172.21.15.116:0/2798178137 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd61820870 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f3459636640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3450042010 tx=0x7f3450005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a74800ea334bc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f343cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450051040 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3444099b10 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9510+0+0 (secure 0 0 0) 0x7f3450004790 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f345002cc90 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f345e563740 10 client.29131 fetch_fsmap learned FSMap version 33 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd61892a80 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.450+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3444099b10 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f345006b030 con 0x55dd61812a30 2024-03-19T02:40:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 10 client.29131 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 10 client.29131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 20 client.29131 populate_metadata read hostname 'smithi116' 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd61895120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 4 client.29131 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd6177dd90 con 0x55dd6182be90 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f3458e35640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd61895120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f3458e35640 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd61895120 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97e4d44e010041e server_cookie=69d2eea107171255 in_seq=0 out_seq=0 2024-03-19T02:40:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.451+0000 7f343cff9640 10 client.29131 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.455+0000 7f343cff9640 1 -- 172.21.15.116:0/2798178137 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55dd6177dd90 con 0x55dd6182be90 2024-03-19T02:40:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.455+0000 7f343cff9640 10 client.29131 handle_command_reply: tid=0 2024-03-19T02:40:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.455+0000 7f345e563740 1 client.29131 shutdown 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 msgr2=0x55dd6188b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3450042010 tx=0x7f3450005fd0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 shutdown_connections 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd6182be90 0x55dd61895120 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd6181bf30 0x55dd61820180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd61813350 0x55dd6188b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 --2- 172.21.15.116:0/2798178137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd61812a30 0x55dd6188b320 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f343cff9640 0 client.29131 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 >> 172.21.15.116:0/2798178137 conn(0x55dd6180d160 msgr2=0x55dd6180c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 shutdown_connections 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 1 -- 172.21.15.116:0/2798178137 wait complete. 2024-03-19T02:40:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345e563740 20 client.29131 trim_cache size 0 max 16384 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (138 inodes in the stack)", 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "a58fc1e5-fc9d-4f33-a307-19a8c7a52acb": { 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "a58fc1e5-fc9d-4f33-a307-19a8c7a52acb", 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:40:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:40:46.462 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 msgr2=0x7f34400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 0x7f34400a4eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f344c005c40 tx=0x7f344c009500 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 msgr2=0x7f3454199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f344400e9f0 tx=0x7f344400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 shutdown_connections 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f34400a29f0 0x7f34400a4eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34541066a0 0x7f34541a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3454101050 0x7f3454199340 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 --2- 172.21.15.116:0/3921996742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3454100690 0x7f3454198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 >> 172.21.15.116:0/3921996742 conn(0x7f3454077820 msgr2=0x7f3454102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:40:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 shutdown_connections 2024-03-19T02:40:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:40:46.456+0000 7f345b89a640 1 -- 172.21.15.116:0/3921996742 wait complete. 2024-03-19T02:40:46.468 INFO:tasks.cephfs.filesystem:scrub status for tag:a58fc1e5-fc9d-4f33-a307-19a8c7a52acb - {'path': '/', 'tag': 'a58fc1e5-fc9d-4f33-a307-19a8c7a52acb', 'options': 'recursive,force'} 2024-03-19T02:40:47.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1369 26.93 MB/sec warmup 9 sec latency 235.574 ms 2024-03-19T02:40:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:40:47.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:47 smithi162 ceph-mon[35761]: pgmap v406: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:47.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:40:47.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:47 smithi154 ceph-mon[35693]: pgmap v406: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:40:47.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:47.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:47 smithi116 ceph-mon[32263]: pgmap v406: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:48.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1480 24.64 MB/sec warmup 10 sec latency 146.980 ms 2024-03-19T02:40:48.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:48 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:48.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:48 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:48.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:48 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:40:49.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1619 24.54 MB/sec warmup 11 sec latency 148.667 ms 2024-03-19T02:40:49.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:49 smithi162 ceph-mon[35761]: pgmap v407: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:49.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:49 smithi154 ceph-mon[35693]: pgmap v407: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:49.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:49 smithi116 ceph-mon[32263]: pgmap v407: 129 pgs: 129 active+clean; 698 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 18 MiB/s wr, 176 op/s 2024-03-19T02:40:50.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1746 22.87 MB/sec warmup 12 sec latency 121.581 ms 2024-03-19T02:40:51.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1852 21.19 MB/sec warmup 13 sec latency 173.739 ms 2024-03-19T02:40:52.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:51 smithi154 ceph-mon[35693]: pgmap v408: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 245 op/s 2024-03-19T02:40:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:51 smithi116 ceph-mon[32263]: pgmap v408: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 245 op/s 2024-03-19T02:40:52.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 1990 19.79 MB/sec warmup 14 sec latency 133.845 ms 2024-03-19T02:40:52.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:51 smithi162 ceph-mon[35761]: pgmap v408: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 20 MiB/s wr, 245 op/s 2024-03-19T02:40:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:53 smithi154 ceph-mon[35693]: pgmap v409: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 10 MiB/s wr, 174 op/s 2024-03-19T02:40:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:53 smithi116 ceph-mon[32263]: pgmap v409: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 10 MiB/s wr, 174 op/s 2024-03-19T02:40:53.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2132 18.62 MB/sec warmup 15 sec latency 127.791 ms 2024-03-19T02:40:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:53 smithi162 ceph-mon[35761]: pgmap v409: 129 pgs: 129 active+clean; 705 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 10 MiB/s wr, 174 op/s 2024-03-19T02:40:54.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2283 17.71 MB/sec warmup 16 sec latency 138.807 ms 2024-03-19T02:40:55.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2409 16.77 MB/sec warmup 17 sec latency 133.550 ms 2024-03-19T02:40:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:55 smithi154 ceph-mon[35693]: pgmap v410: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 266 op/s 2024-03-19T02:40:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:55 smithi116 ceph-mon[32263]: pgmap v410: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 266 op/s 2024-03-19T02:40:56.318 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2546 15.98 MB/sec warmup 18 sec latency 131.805 ms 2024-03-19T02:40:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:55 smithi162 ceph-mon[35761]: pgmap v410: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 266 op/s 2024-03-19T02:40:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:56 smithi154 ceph-mon[35693]: pgmap v411: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:40:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:56 smithi116 ceph-mon[32263]: pgmap v411: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:40:57.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2695 15.80 MB/sec warmup 19 sec latency 130.100 ms 2024-03-19T02:40:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:56 smithi162 ceph-mon[35761]: pgmap v411: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:40:58.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2796 15.22 MB/sec warmup 20 sec latency 215.504 ms 2024-03-19T02:40:59.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2872 14.63 MB/sec warmup 21 sec latency 146.614 ms 2024-03-19T02:41:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:59 smithi154 ceph-mon[35693]: pgmap v412: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:41:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:59 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:40:59 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:59 smithi116 ceph-mon[32263]: pgmap v412: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:41:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:59 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:00.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:40:59 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:00.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 2978 14.38 MB/sec warmup 22 sec latency 206.290 ms 2024-03-19T02:41:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:59 smithi162 ceph-mon[35761]: pgmap v412: 129 pgs: 129 active+clean; 707 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-03-19T02:41:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:59 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:40:59 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:00 smithi154 ceph-mon[35693]: pgmap v413: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 233 op/s 2024-03-19T02:41:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:00 smithi116 ceph-mon[32263]: pgmap v413: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 233 op/s 2024-03-19T02:41:01.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3018 13.83 MB/sec warmup 23 sec latency 848.730 ms 2024-03-19T02:41:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:00 smithi162 ceph-mon[35761]: pgmap v413: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.1 MiB/s wr, 233 op/s 2024-03-19T02:41:02.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3097 13.33 MB/sec warmup 24 sec latency 161.187 ms 2024-03-19T02:41:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:03 smithi154 ceph-mon[35693]: pgmap v414: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-03-19T02:41:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:03 smithi116 ceph-mon[32263]: pgmap v414: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-03-19T02:41:03.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3316 13.83 MB/sec warmup 25 sec latency 202.798 ms 2024-03-19T02:41:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:03 smithi162 ceph-mon[35761]: pgmap v414: 129 pgs: 129 active+clean; 689 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-03-19T02:41:04.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3470 13.79 MB/sec warmup 26 sec latency 142.009 ms 2024-03-19T02:41:05.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3692 13.74 MB/sec warmup 27 sec latency 183.640 ms 2024-03-19T02:41:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:05 smithi162 ceph-mon[35761]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T02:41:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:05 smithi162 ceph-mon[35761]: pgmap v416: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 4.4 MiB/s wr, 174 op/s 2024-03-19T02:41:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:05 smithi154 ceph-mon[35693]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T02:41:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:05 smithi154 ceph-mon[35693]: pgmap v416: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 4.4 MiB/s wr, 174 op/s 2024-03-19T02:41:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:05 smithi116 ceph-mon[32263]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T02:41:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:05 smithi116 ceph-mon[32263]: pgmap v416: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 4.4 MiB/s wr, 174 op/s 2024-03-19T02:41:06.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3826 13.61 MB/sec warmup 28 sec latency 136.633 ms 2024-03-19T02:41:07.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3900 13.21 MB/sec warmup 29 sec latency 157.789 ms 2024-03-19T02:41:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:07 smithi162 ceph-mon[35761]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T02:41:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:07 smithi162 ceph-mon[35761]: pgmap v418: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 218 op/s 2024-03-19T02:41:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:07 smithi154 ceph-mon[35693]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T02:41:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:07 smithi154 ceph-mon[35693]: pgmap v418: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 218 op/s 2024-03-19T02:41:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:07 smithi116 ceph-mon[32263]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T02:41:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:07 smithi116 ceph-mon[32263]: pgmap v418: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.4 MiB/s wr, 218 op/s 2024-03-19T02:41:08.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 3948 12.85 MB/sec warmup 30 sec latency 247.638 ms 2024-03-19T02:41:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:09 smithi116 ceph-mon[32263]: pgmap v419: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 111 op/s 2024-03-19T02:41:09.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4023 12.64 MB/sec warmup 31 sec latency 158.790 ms 2024-03-19T02:41:09.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:09 smithi162 ceph-mon[35761]: pgmap v419: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 111 op/s 2024-03-19T02:41:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:09 smithi154 ceph-mon[35693]: pgmap v419: 129 pgs: 129 active+clean; 706 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 111 op/s 2024-03-19T02:41:10.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4043 12.25 MB/sec warmup 32 sec latency 171.258 ms 2024-03-19T02:41:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:11 smithi116 ceph-mon[32263]: pgmap v420: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:41:11.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4099 11.99 MB/sec warmup 33 sec latency 333.426 ms 2024-03-19T02:41:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:11 smithi162 ceph-mon[35761]: pgmap v420: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:41:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:11 smithi154 ceph-mon[35693]: pgmap v420: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:41:12.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4245 12.34 MB/sec warmup 34 sec latency 1333.455 ms 2024-03-19T02:41:13.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:12 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:41:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:12 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:41:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:12 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:41:13.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4352 12.06 MB/sec warmup 35 sec latency 2333.480 ms 2024-03-19T02:41:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:13 smithi162 ceph-mon[35761]: pgmap v421: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-19T02:41:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:13 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:41:14.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:13 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:41:14.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:13 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:41:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:13 smithi154 ceph-mon[35693]: pgmap v421: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-19T02:41:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:13 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:41:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:13 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:41:14.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:13 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:41:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:13 smithi116 ceph-mon[32263]: pgmap v421: 129 pgs: 129 active+clean; 715 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-03-19T02:41:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:13 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:41:14.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:13 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:41:14.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:13 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:41:14.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4463 12.04 MB/sec warmup 36 sec latency 3333.511 ms 2024-03-19T02:41:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:15 smithi116 ceph-mon[32263]: pgmap v422: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.3 MiB/s wr, 124 op/s 2024-03-19T02:41:15.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4526 11.71 MB/sec warmup 37 sec latency 4333.537 ms 2024-03-19T02:41:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:15 smithi162 ceph-mon[35761]: pgmap v422: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.3 MiB/s wr, 124 op/s 2024-03-19T02:41:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:15 smithi154 ceph-mon[35693]: pgmap v422: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 3.3 MiB/s wr, 124 op/s 2024-03-19T02:41:16.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4598 11.51 MB/sec warmup 38 sec latency 4857.871 ms 2024-03-19T02:41:16.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:41:16.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 -- 172.21.15.116:0/4240531286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4100690 msgr2=0x7f0de4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 --2- 172.21.15.116:0/4240531286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4100690 0x7f0de4100b10 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009930 tx=0x7f0dd402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 -- 172.21.15.116:0/4240531286 shutdown_connections 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 --2- 172.21.15.116:0/4240531286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de4101050 0x7f0de410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 --2- 172.21.15.116:0/4240531286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4100690 0x7f0de4100b10 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 --2- 172.21.15.116:0/4240531286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de41066a0 0x7f0de4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.594+0000 7f0deb885640 1 -- 172.21.15.116:0/4240531286 >> 172.21.15.116:0/4240531286 conn(0x7f0de4077820 msgr2=0x7f0de4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.595+0000 7f0deb885640 1 -- 172.21.15.116:0/4240531286 shutdown_connections 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.595+0000 7f0deb885640 1 -- 172.21.15.116:0/4240531286 wait complete. 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.595+0000 7f0deb885640 1 Processor -- start 2024-03-19T02:41:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.595+0000 7f0deb885640 1 -- start start 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de41066a0 0x7f0de419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110960 con 0x7f0de4101050 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41107e0 con 0x7f0de41066a0 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0deb885640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110ae0 con 0x7f0de4100690 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de9621640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de9621640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58446/0 (socket says 172.21.15.116:58446) 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de9621640 1 -- 172.21.15.116:0/1221670651 learned_addr learned my addr 172.21.15.116:0/1221670651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43844/0 (socket says 172.21.15.116:43844) 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de9e22640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de41066a0 0x7f0de419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 -- 172.21.15.116:0/1221670651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 msgr2=0x7f0de419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 -- 172.21.15.116:0/1221670651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de41066a0 msgr2=0x7f0de419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de41066a0 0x7f0de419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 -- 172.21.15.116:0/1221670651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de419f200 con 0x7f0de4101050 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de9621640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0de8e20640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009230 tx=0x7f0dd40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.596+0000 7f0dda7fc640 1 -- 172.21.15.116:0/1221670651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4043020 con 0x7f0de4101050 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.597+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de419f490 con 0x7f0de4101050 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.597+0000 7f0dda7fc640 1 -- 172.21.15.116:0/1221670651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd402ca70 con 0x7f0de4101050 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.597+0000 7f0dda7fc640 1 -- 172.21.15.116:0/1221670651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40039b0 con 0x7f0de4101050 2024-03-19T02:41:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.597+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de41a6d80 con 0x7f0de4101050 2024-03-19T02:41:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.599+0000 7f0dda7fc640 1 -- 172.21.15.116:0/1221670651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0dd4046020 con 0x7f0de4101050 2024-03-19T02:41:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 Processor -- start 2024-03-19T02:41:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 -- start start 2024-03-19T02:41:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dda7fc640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 0x7f0dd00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b3049b6500 con 0x55b304a53350 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b30495ebe0 con 0x55b304a52a30 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dee477740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3049b25a0 con 0x55b304a5bf30 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de9e22640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 0x7f0dd00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de9621640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43846/0 (socket says 172.21.15.116:43846) 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de9621640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40506/0 (socket says 172.21.15.116:40506) 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0dda7fc640 1 -- 172.21.15.116:0/1221670651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f0dd40f1030 con 0x7f0de4101050 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 -- 172.21.15.116:0/21767572 learned_addr learned my addr 172.21.15.116:0/21767572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de9e22640 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 -- 172.21.15.116:0/21767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 msgr2=0x55b304a5e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 -- 172.21.15.116:0/21767572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 msgr2=0x55b304a52e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de8e20640 1 -- 172.21.15.116:0/21767572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b304980000 con 0x55b304a53350 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.600+0000 7f0de9621640 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.601+0000 7f0de9e22640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 0x7f0dd00a4e30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005fd0 tx=0x7f0ddc009f10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.601+0000 7f0de8e20640 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40c5630 tx=0x7f0dd402c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0dacb51b5631fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.601+0000 7f0dcf7fe640 1 -- 172.21.15.116:0/21767572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd409a9e0 con 0x55b304a53350 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.601+0000 7f0dcf7fe640 1 -- 172.21.15.116:0/21767572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd40bdac0 con 0x55b304a53350 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.601+0000 7f0dcf7fe640 1 -- 172.21.15.116:0/21767572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd409bf80 con 0x55b304a53350 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- 172.21.15.116:0/21767572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 msgr2=0x55b304a5b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40c5630 tx=0x7f0dd402c740 comp rx=0 tx=0).stop 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- 172.21.15.116:0/21767572 shutdown_connections 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5b9f0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 --2- 172.21.15.116:0/21767572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a52e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- 172.21.15.116:0/21767572 >> 172.21.15.116:0/21767572 conn(0x55b304a4d160 msgr2=0x55b304a4d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- 172.21.15.116:0/21767572 shutdown_connections 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- 172.21.15.116:0/21767572 wait complete. 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 -- start start 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.602+0000 7f0dee477740 1 Processor -- start 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a59360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 0x55b304a598a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b304a5eae0 con 0x55b304a5bf30 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b304972160 con 0x55b304a52a30 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b304a5ec60 con 0x55b304a53350 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dee477740 10 client.0 _ll_register_callbacks cb 0x55b3049aea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9e22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 0x55b304a598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40518/0 (socket says 172.21.15.116:40518) 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 -- 172.21.15.116:0/6457619 learned_addr learned my addr 172.21.15.116:0/6457619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de8e20640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a59360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 -- 172.21.15.116:0/6457619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 msgr2=0x55b304a59360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a59360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 -- 172.21.15.116:0/6457619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 msgr2=0x55b304a598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 0x55b304a598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 -- 172.21.15.116:0/6457619 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3047947d0 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9e22640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 0x55b304a598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0de9621640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012d20 tx=0x7f0de000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad348e34f5f8f92d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.603+0000 7f0dccff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de000fe20 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f0de0010430 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd40bdac0 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0de0010630 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 10 client.29189 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b3046ca3f0 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd40bdac0 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f0de001c7c0 con 0x55b304a52a30 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 10 client.29189 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 10 client.29189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 20 client.29189 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304a6bfe0 0x55b304a6e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 4 client.29189 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.604+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3049c3a40 con 0x55b304a6bfe0 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.605+0000 7f0de8e20640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304a6bfe0 0x55b304a6e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.605+0000 7f0de8e20640 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304a6bfe0 0x55b304a6e430 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c45d8aeaa1238132 server_cookie=e2f8b003ddf86676 in_seq=0 out_seq=0 2024-03-19T02:41:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.605+0000 7f0dccff9640 10 client.29189 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dccff9640 1 -- 172.21.15.116:0/6457619 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3049c3a40 con 0x55b304a6bfe0 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dccff9640 10 client.29189 handle_command_reply: tid=0 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 client.29189 shutdown 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 msgr2=0x55b304a58e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0de0012d20 tx=0x7f0de000f790 comp rx=0 tx=0).stop 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 shutdown_connections 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304a6bfe0 0x55b304a6e430 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a5bf30 0x55b304a598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a59360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 --2- 172.21.15.116:0/6457619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a58e20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dccff9640 0 client.29189 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 >> 172.21.15.116:0/6457619 conn(0x55b304a4d160 msgr2=0x55b304a4eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 shutdown_connections 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 1 -- 172.21.15.116:0/6457619 wait complete. 2024-03-19T02:41:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.608+0000 7f0dee477740 20 client.29189 trim_cache size 0 max 16384 2024-03-19T02:41:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.632+0000 7f0dee477740 1 Processor -- start 2024-03-19T02:41:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.632+0000 7f0dee477740 1 -- start start 2024-03-19T02:41:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 0x55b304a5b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a5bf30 0x55b304a73d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41b0e30 con 0x55b304a52a30 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41b2a60 con 0x55b304a5bf30 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0dee477740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41b1500 con 0x55b304a53350 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de9621640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 0x55b304a5b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de9621640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 0x55b304a5b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43866/0 (socket says 172.21.15.116:43866) 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de9621640 1 -- 172.21.15.116:0/3284581502 learned_addr learned my addr 172.21.15.116:0/3284581502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de9e22640 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a5bf30 0x55b304a73d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 -- 172.21.15.116:0/3284581502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a5bf30 msgr2=0x55b304a73d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a5bf30 0x55b304a73d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 -- 172.21.15.116:0/3284581502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 msgr2=0x55b304a5b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 0x55b304a5b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 -- 172.21.15.116:0/3284581502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b30468a0f0 con 0x55b304a53350 2024-03-19T02:41:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.633+0000 7f0de8e20640 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a677f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009900 tx=0x7f0dd40b9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e13dfd22d4778ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dceffd640 1 -- 172.21.15.116:0/3284581502 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd409c590 con 0x55b304a53350 2024-03-19T02:41:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dceffd640 1 -- 172.21.15.116:0/3284581502 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd40bdac0 con 0x55b304a53350 2024-03-19T02:41:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dceffd640 1 -- 172.21.15.116:0/3284581502 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd409a800 con 0x55b304a53350 2024-03-19T02:41:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 -- 172.21.15.116:0/3284581502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 msgr2=0x55b304a677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a677f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009900 tx=0x7f0dd40b9040 comp rx=0 tx=0).stop 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 -- 172.21.15.116:0/3284581502 shutdown_connections 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a5bf30 0x55b304a73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a53350 0x55b304a677f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 --2- 172.21.15.116:0/3284581502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a52a30 0x55b304a5b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.634+0000 7f0dee477740 1 -- 172.21.15.116:0/3284581502 >> 172.21.15.116:0/3284581502 conn(0x55b304a4d160 msgr2=0x55b304a58de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- 172.21.15.116:0/3284581502 shutdown_connections 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- 172.21.15.116:0/3284581502 wait complete. 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- start start 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 Processor -- start 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a63f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41b22d0 con 0x55b304a53350 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41b1c00 con 0x55b304a52a30 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41aee30 con 0x55b304a5bf30 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0dee477740 10 client.0 _ll_register_callbacks cb 0x7f0de0012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0de8e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43882/0 (socket says 172.21.15.116:43882) 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0de8e20640 1 -- 172.21.15.116:0/2180154963 learned_addr learned my addr 172.21.15.116:0/2180154963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0de9e22640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.635+0000 7f0de9621640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a63f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 -- 172.21.15.116:0/2180154963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 msgr2=0x55b304a5f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 -- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 msgr2=0x55b304a63f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a63f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 -- 172.21.15.116:0/2180154963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b304a60000 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0de8e20640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40b9010 tx=0x7f0dd402c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a718463f38f943cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dccff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40bd3c0 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f0dd40ee620 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd40bdac0 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0dd40c1510 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dee477740 10 client.19419 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b304ad2210 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.636+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd40bdac0 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f0dd40c10d0 con 0x55b304a53350 2024-03-19T02:41:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 10 client.19419 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 10 client.19419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 20 client.19419 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304ad23f0 0x55b304ad4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 4 client.19419 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b3049bdd90 con 0x55b304ad23f0 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0de9621640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304ad23f0 0x55b304ad4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0de9621640 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304ad23f0 0x55b304ad4810 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eff8fbc746d3a675 server_cookie=5d0279b0a6fe262b in_seq=0 out_seq=0 2024-03-19T02:41:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.637+0000 7f0dccff9640 10 client.19419 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.642+0000 7f0dccff9640 1 -- 172.21.15.116:0/2180154963 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b3049bdd90 con 0x55b304ad23f0 2024-03-19T02:41:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.642+0000 7f0dccff9640 10 client.19419 handle_command_reply: tid=0 2024-03-19T02:41:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.642+0000 7f0dee477740 1 client.19419 shutdown 2024-03-19T02:41:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 msgr2=0x55b304a5f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40b9010 tx=0x7f0dd402c740 comp rx=0 tx=0).stop 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 shutdown_connections 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b304ad23f0 0x55b304ad4810 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b304a5bf30 0x55b304a5f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b304a53350 0x55b304a5f400 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dccff9640 0 client.19419 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 --2- 172.21.15.116:0/2180154963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b304a52a30 0x55b304a63f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 >> 172.21.15.116:0/2180154963 conn(0x55b304a4d160 msgr2=0x55b304a4c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 shutdown_connections 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 1 -- 172.21.15.116:0/2180154963 wait complete. 2024-03-19T02:41:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0dee477740 20 client.19419 trim_cache size 0 max 16384 2024-03-19T02:41:16.646 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:16.646 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:41:16.646 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:41:16.646 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 msgr2=0x7f0dd00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 0x7f0dd00a4e30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005fd0 tx=0x7f0ddc009f10 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 msgr2=0x7f0de419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.643+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009230 tx=0x7f0dd40045e0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 shutdown_connections 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0dd00a2970 0x7f0dd00a4e30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de41066a0 0x7f0de419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0de4101050 0x7f0de419a750 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 --2- 172.21.15.116:0/1221670651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0de4100690 0x7f0de419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 >> 172.21.15.116:0/1221670651 conn(0x7f0de4077820 msgr2=0x7f0de410c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 shutdown_connections 2024-03-19T02:41:16.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.644+0000 7f0deb885640 1 -- 172.21.15.116:0/1221670651 wait complete. 2024-03-19T02:41:16.656 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:41:16.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:41:16.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 -- 172.21.15.116:0/1442235483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 msgr2=0x7f3dc4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/1442235483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4100ba0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009930 tx=0x7f3db802bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 -- 172.21.15.116:0/1442235483 shutdown_connections 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/1442235483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/1442235483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4100ba0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/1442235483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 0x7f3dc4106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 -- 172.21.15.116:0/1442235483 >> 172.21.15.116:0/1442235483 conn(0x7f3dc4077820 msgr2=0x7f3dc4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 -- 172.21.15.116:0/1442235483 shutdown_connections 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 -- 172.21.15.116:0/1442235483 wait complete. 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.778+0000 7f3dc8f97640 1 Processor -- start 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 -- start start 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 0x7f3dc419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc41109f0 con 0x7f3dc4100720 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4110870 con 0x7f3dc4106730 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc8f97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4110b70 con 0x7f3dc41010e0 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc1d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43898/0 (socket says 172.21.15.116:43898) 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 -- 172.21.15.116:0/2796267933 learned_addr learned my addr 172.21.15.116:0/2796267933 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc2d9d640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 0x7f3dc419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 -- 172.21.15.116:0/2796267933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 msgr2=0x7f3dc4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 -- 172.21.15.116:0/2796267933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 msgr2=0x7f3dc419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 0x7f3dc419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 -- 172.21.15.116:0/2796267933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc419d030 con 0x7f3dc4100720 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc1d9b640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.779+0000 7f3dc259c640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9f0 tx=0x7f3db400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.780+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db400cd80 con 0x7f3dc4100720 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.780+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3db40108e0 con 0x7f3dc4100720 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.780+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4018920 con 0x7f3dc4100720 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.780+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc419d320 con 0x7f3dc4100720 2024-03-19T02:41:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.780+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc41a4bb0 con 0x7f3dc4100720 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.781+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc4068610 con 0x7f3dc4100720 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.782+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3db4010a90 con 0x7f3dc4100720 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.782+0000 7f3db37fe640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 0x7f3dac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.783+0000 7f3dc1d9b640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 0x7f3dac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.783+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f3db40cf450 con 0x7f3dc4100720 2024-03-19T02:41:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.783+0000 7f3dc1d9b640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 0x7f3dac0a4eb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005ab0 tx=0x7f3db8005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:16.788+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3db409b780 con 0x7f3dc4100720 2024-03-19T02:41:17.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.085+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dc4199390 con 0x7f3dc4100720 2024-03-19T02:41:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.086+0000 7f3db37fe640 1 -- 172.21.15.116:0/2796267933 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11958 (secure 0 0 0) 0x7f3db409f820 con 0x7f3dc4100720 2024-03-19T02:41:17.090 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:41:17.090 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:40:44.485117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 msgr2=0x7f3dac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 0x7f3dac0a4eb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005ab0 tx=0x7f3db8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 msgr2=0x7f3dc4198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9f0 tx=0x7f3db400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 shutdown_connections 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3dac0a29f0 0x7f3dac0a4eb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dc4106730 0x7f3dc419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dc41010e0 0x7f3dc4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 --2- 172.21.15.116:0/2796267933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc4100720 0x7f3dc4198040 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 >> 172.21.15.116:0/2796267933 conn(0x7f3dc4077820 msgr2=0x7f3dc4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 shutdown_connections 2024-03-19T02:41:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.088+0000 7f3dc8f97640 1 -- 172.21.15.116:0/2796267933 wait complete. 2024-03-19T02:41:17.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 -- 172.21.15.116:0/2204718550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 msgr2=0x7f17a4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 --2- 172.21.15.116:0/2204718550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4100b20 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f1798009930 tx=0x7f179802bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 -- 172.21.15.116:0/2204718550 shutdown_connections 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 --2- 172.21.15.116:0/2204718550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a4101060 0x7f17a410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 --2- 172.21.15.116:0/2204718550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4100b20 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 --2- 172.21.15.116:0/2204718550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a41066b0 0x7f17a4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 -- 172.21.15.116:0/2204718550 >> 172.21.15.116:0/2204718550 conn(0x7f17a40778d0 msgr2=0x7f17a4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 -- 172.21.15.116:0/2204718550 shutdown_connections 2024-03-19T02:41:17.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 -- 172.21.15.116:0/2204718550 wait complete. 2024-03-19T02:41:17.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.237+0000 7f17aa043640 1 Processor -- start 2024-03-19T02:41:17.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 -- start start 2024-03-19T02:41:17.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a4101060 0x7f17a41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a41066b0 0x7f17a419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17a4110970 con 0x7f17a41006a0 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17a41107f0 con 0x7f17a41066b0 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17aa043640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17a4110af0 con 0x7f17a4101060 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43908/0 (socket says 172.21.15.116:43908) 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 -- 172.21.15.116:0/2121792540 learned_addr learned my addr 172.21.15.116:0/2121792540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a4101060 0x7f17a41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a3fff640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a41066b0 0x7f17a419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 -- 172.21.15.116:0/2121792540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a4101060 msgr2=0x7f17a41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a4101060 0x7f17a41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 -- 172.21.15.116:0/2121792540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a41066b0 msgr2=0x7f17a419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a41066b0 0x7f17a419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 -- 172.21.15.116:0/2121792540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a419d020 con 0x7f17a41006a0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.238+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9f0 tx=0x7f179400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.239+0000 7f17a0ff9640 1 -- 172.21.15.116:0/2121792540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179400cd80 con 0x7f17a41006a0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.239+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a419d310 con 0x7f17a41006a0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.239+0000 7f17a0ff9640 1 -- 172.21.15.116:0/2121792540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17940054d0 con 0x7f17a41006a0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.239+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a41a4ba0 con 0x7f17a41006a0 2024-03-19T02:41:17.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.239+0000 7f17a0ff9640 1 -- 172.21.15.116:0/2121792540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794010490 con 0x7f17a41006a0 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.241+0000 7f17a0ff9640 1 -- 172.21.15.116:0/2121792540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1794005020 con 0x7f17a41006a0 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17a0ff9640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 0x7f178c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17acc2a740 1 Processor -- start 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 0x7f178c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17acc2a740 1 -- start start 2024-03-19T02:41:17.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17a0ff9640 1 -- 172.21.15.116:0/2121792540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f17940cf340 con 0x7f17a41006a0 2024-03-19T02:41:17.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.242+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a29e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17acc2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c7a2014b0 con 0x560c7a2a77d0 2024-03-19T02:41:17.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17acc2a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c7a1a9d70 con 0x560c7a29ebf0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17acc2a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c7a1fd550 con 0x560c7a29e2d0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a29e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40588/0 (socket says 172.21.15.116:40588) 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a2ffd640 1 -- 172.21.15.116:0/3453269561 learned_addr learned my addr 172.21.15.116:0/3453269561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 0x7f178c0a4eb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1798005ab0 tx=0x7f1798005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 -- 172.21.15.116:0/3453269561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 msgr2=0x560c7a2a7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 -- 172.21.15.116:0/3453269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 msgr2=0x560c7a2a9bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a9bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 -- 172.21.15.116:0/3453269561 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c7a1cc210 con 0x560c7a29e2d0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.243+0000 7f17a37fe640 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a29e6b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17940cc6b0 tx=0x7f1794097990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9aba5278124c627 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.244+0000 7f1789ffb640 1 -- 172.21.15.116:0/3453269561 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940b4070 con 0x560c7a29e2d0 2024-03-19T02:41:17.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.244+0000 7f1789ffb640 1 -- 172.21.15.116:0/3453269561 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17940cf170 con 0x560c7a29e2d0 2024-03-19T02:41:17.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.244+0000 7f1789ffb640 1 -- 172.21.15.116:0/3453269561 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794097c30 con 0x560c7a29e2d0 2024-03-19T02:41:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- 172.21.15.116:0/3453269561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 msgr2=0x560c7a29e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a29e6b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17940cc6b0 tx=0x7f1794097990 comp rx=0 tx=0).stop 2024-03-19T02:41:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- 172.21.15.116:0/3453269561 shutdown_connections 2024-03-19T02:41:17.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3453269561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a29e6b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- 172.21.15.116:0/3453269561 >> 172.21.15.116:0/3453269561 conn(0x560c7a298a00 msgr2=0x560c7a298de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- 172.21.15.116:0/3453269561 shutdown_connections 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- 172.21.15.116:0/3453269561 wait complete. 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 -- start start 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.245+0000 7f17acc2a740 1 Processor -- start 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a2a4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c7a2aa610 con 0x560c7a2a77d0 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560c7a1bd310 con 0x560c7a29ebf0 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c7a2aa790 con 0x560c7a29e2d0 2024-03-19T02:41:17.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17acc2a740 10 client.0 _ll_register_callbacks cb 0x560c7a1b12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43912/0 (socket says 172.21.15.116:43912) 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a3fff640 1 -- 172.21.15.116:0/2525485434 learned_addr learned my addr 172.21.15.116:0/2525485434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a2a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 -- 172.21.15.116:0/2525485434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 msgr2=0x560c7a2a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 -- 172.21.15.116:0/2525485434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 msgr2=0x560c7a2a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a37fe640 1 -- 172.21.15.116:0/2525485434 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c79f153f0 con 0x560c7a29e2d0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.246+0000 7f17a3fff640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17a37fe640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a2a4820 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9c0 tx=0x7f179409b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7a4810fa3b4dca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940b4040 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f17940a38d0 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17940cf170 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f17940ab050 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17acc2a740 10 client.29164 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560c79fdf7d0 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17940cf170 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f1794084b00 con 0x560c7a29e2d0 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17acc2a740 10 client.29164 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.247+0000 7f17acc2a740 10 client.29164 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17acc2a740 20 client.29164 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a2bbe90 0x560c7a2be280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17acc2a740 4 client.29164 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:17.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c7a25f3e0 con 0x560c7a2bbe90 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a2bbe90 0x560c7a2be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f1782ffd640 10 client.29164 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.248+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a2bbe90 0x560c7a2be280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d42b50937ec42f42 server_cookie=2a761803b46d6a5d in_seq=0 out_seq=0 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f1782ffd640 1 -- 172.21.15.116:0/2525485434 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c7a25f3e0 con 0x560c7a2bbe90 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f1782ffd640 10 client.29164 handle_command_reply: tid=0 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 client.29164 shutdown 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 msgr2=0x560c7a2a4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a2a4820 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9c0 tx=0x7f179409b020 comp rx=0 tx=0).stop 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 shutdown_connections 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a2bbe90 0x560c7a2be280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 --2- 172.21.15.116:0/2525485434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29e2d0 0x560c7a2a4820 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f1782ffd640 0 client.29164 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:17.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.249+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 >> 172.21.15.116:0/2525485434 conn(0x560c7a298a00 msgr2=0x560c7a29a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.250+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 shutdown_connections 2024-03-19T02:41:17.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.250+0000 7f17acc2a740 1 -- 172.21.15.116:0/2525485434 wait complete. 2024-03-19T02:41:17.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.250+0000 7f17acc2a740 20 client.29164 trim_cache size 0 max 16384 2024-03-19T02:41:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:17 smithi116 ceph-mon[32263]: pgmap v423: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.2 MiB/s wr, 123 op/s 2024-03-19T02:41:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 Processor -- start 2024-03-19T02:41:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 -- start start 2024-03-19T02:41:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 0x560c7a2a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f179000ff10 con 0x560c7a2a77d0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1790011b40 con 0x560c7a29e2d0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.268+0000 7f17acc2a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17900105e0 con 0x560c7a29ebf0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 0x560c7a2a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58552/0 (socket says 172.21.15.116:58552) 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 0x560c7a2a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40610/0 (socket says 172.21.15.116:40610) 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 -- 172.21.15.116:0/3746875307 learned_addr learned my addr 172.21.15.116:0/3746875307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43926/0 (socket says 172.21.15.116:43926) 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 -- 172.21.15.116:0/3746875307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 msgr2=0x560c7a2a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 0x560c7a2a6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 -- 172.21.15.116:0/3746875307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 msgr2=0x560c7a2c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 -- 172.21.15.116:0/3746875307 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c7a298530 con 0x560c7a29ebf0 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f179802c170 tx=0x7f1798044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6480aa6fe46a0210 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17897fa640 1 -- 172.21.15.116:0/3746875307 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179806c070 con 0x560c7a29ebf0 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17897fa640 1 -- 172.21.15.116:0/3746875307 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17980673f0 con 0x560c7a29ebf0 2024-03-19T02:41:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.269+0000 7f17897fa640 1 -- 172.21.15.116:0/3746875307 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798005260 con 0x560c7a29ebf0 2024-03-19T02:41:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 -- 172.21.15.116:0/3746875307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 msgr2=0x560c7a2a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f179802c170 tx=0x7f1798044000 comp rx=0 tx=0).stop 2024-03-19T02:41:17.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 -- 172.21.15.116:0/3746875307 shutdown_connections 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a2a77d0 0x560c7a2c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a29ebf0 0x560c7a2a50c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 --2- 172.21.15.116:0/3746875307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29e2d0 0x560c7a2a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 -- 172.21.15.116:0/3746875307 >> 172.21.15.116:0/3746875307 conn(0x560c7a298a00 msgr2=0x560c7a297ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 -- 172.21.15.116:0/3746875307 shutdown_connections 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.270+0000 7f17acc2a740 1 -- 172.21.15.116:0/3746875307 wait complete. 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 -- start start 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 Processor -- start 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 0x560c7a2be280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.271+0000 7f17acc2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17900113b0 con 0x560c7a29e2d0 2024-03-19T02:41:17.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17acc2a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1790010ce0 con 0x560c7a29ebf0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 0x560c7a2be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43940/0 (socket says 172.21.15.116:43940) 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a37fe640 1 -- 172.21.15.116:0/651832311 learned_addr learned my addr 172.21.15.116:0/651832311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f179000df10 con 0x560c7a2a77d0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17acc2a740 10 client.0 _ll_register_callbacks cb 0x7f17940b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 -- 172.21.15.116:0/651832311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 msgr2=0x560c7a2c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2c1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 -- 172.21.15.116:0/651832311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 msgr2=0x560c7a2c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 -- 172.21.15.116:0/651832311 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c7a2bea80 con 0x560c7a2a77d0 2024-03-19T02:41:17.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a2ffd640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2c1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a37fe640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f17a3fff640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 0x560c7a2be280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x560c7a2014d0 tx=0x7f179c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f3de83595b9d7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.272+0000 7f1782ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c004b20 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17980673f0 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f179c004cc0 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f179c028050 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f17acc2a740 10 client.29170 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560c7a321900 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17980673f0 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.273+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f179c005140 con 0x560c7a2a77d0 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 10 client.29170 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 10 client.29170 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:41:17.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 20 client.29170 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a322080 0x560c7a3244a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 4 client.29170 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c7a20e9f0 con 0x560c7a322080 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17a37fe640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a322080 0x560c7a3244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f1782ffd640 10 client.29170 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17a37fe640 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a322080 0x560c7a3244a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cbf1cb5faefa09ca server_cookie=918a61fc6226e952 in_seq=0 out_seq=0 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f1782ffd640 1 -- 172.21.15.116:0/651832311 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c7a20e9f0 con 0x560c7a322080 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f1782ffd640 10 client.29170 handle_command_reply: tid=0 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.274+0000 7f17acc2a740 1 client.29170 shutdown 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 msgr2=0x560c7a2be280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 0x560c7a2be280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x560c7a2014d0 tx=0x7f179c011810 comp rx=0 tx=0).stop 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 shutdown_connections 2024-03-19T02:41:17.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560c7a322080 0x560c7a3244a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c7a2a77d0 0x560c7a2be280 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560c7a29ebf0 0x560c7a2c1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f1782ffd640 0 client.29170 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 --2- 172.21.15.116:0/651832311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c7a29e2d0 0x560c7a2c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 >> 172.21.15.116:0/651832311 conn(0x560c7a298a00 msgr2=0x560c7a2b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 shutdown_connections 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 1 -- 172.21.15.116:0/651832311 wait complete. 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17acc2a740 20 client.29170 trim_cache size 0 max 16384 2024-03-19T02:41:17.281 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.275+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 msgr2=0x7f178c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 0x7f178c0a4eb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1798005ab0 tx=0x7f1798005fd0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 msgr2=0x7f17a4198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f179400e9f0 tx=0x7f179400edf0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 shutdown_connections 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f178c0a29f0 0x7f178c0a4eb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17a41066b0 0x7f17a419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17a4101060 0x7f17a41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 --2- 172.21.15.116:0/2121792540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a41006a0 0x7f17a4198080 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 >> 172.21.15.116:0/2121792540 conn(0x7f17a40778d0 msgr2=0x7f17a40ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 shutdown_connections 2024-03-19T02:41:17.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.276+0000 7f17aa043640 1 -- 172.21.15.116:0/2121792540 wait complete. 2024-03-19T02:41:17.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:41:17.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4679 11.25 MB/sec warmup 39 sec latency 157.297 ms 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 -- 172.21.15.116:0/1115808233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3058101060 msgr2=0x7f305810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 --2- 172.21.15.116:0/1115808233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3058101060 0x7f305810d570 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f3050009cd0 tx=0x7f305002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 -- 172.21.15.116:0/1115808233 shutdown_connections 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 --2- 172.21.15.116:0/1115808233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3058101060 0x7f305810d570 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 --2- 172.21.15.116:0/1115808233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581006a0 0x7f3058100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 --2- 172.21.15.116:0/1115808233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30581066b0 0x7f3058106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 -- 172.21.15.116:0/1115808233 >> 172.21.15.116:0/1115808233 conn(0x7f30580778d0 msgr2=0x7f3058077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 -- 172.21.15.116:0/1115808233 shutdown_connections 2024-03-19T02:41:17.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.413+0000 7f305cf3b640 1 -- 172.21.15.116:0/1115808233 wait complete. 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 Processor -- start 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 -- start start 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 0x7f30581984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581066b0 0x7f305819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3058110970 con 0x7f30581006a0 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30581107f0 con 0x7f30581066b0 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305cf3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3058110af0 con 0x7f3058101060 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f304ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 0x7f30581984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43960/0 (socket says 172.21.15.116:43960) 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.414+0000 7f305659c640 1 -- 172.21.15.116:0/3443249110 learned_addr learned my addr 172.21.15.116:0/3443249110 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f3056d9d640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581066b0 0x7f305819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 -- 172.21.15.116:0/3443249110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581066b0 msgr2=0x7f305819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581066b0 0x7f305819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 -- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 msgr2=0x7f3058197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 -- 172.21.15.116:0/3443249110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305819cf50 con 0x7f3058101060 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f305659c640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304ffff640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 0x7f30581984a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f304000e9f0 tx=0x7f304000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304f7fe640 1 -- 172.21.15.116:0/3443249110 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304000ccc0 con 0x7f3058101060 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304f7fe640 1 -- 172.21.15.116:0/3443249110 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30400108e0 con 0x7f3058101060 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f304f7fe640 1 -- 172.21.15.116:0/3443249110 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040018880 con 0x7f3058101060 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.415+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305819d240 con 0x7f3058101060 2024-03-19T02:41:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.416+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3058102c40 con 0x7f3058101060 2024-03-19T02:41:17.420 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:17 smithi162 ceph-mon[35761]: pgmap v423: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.2 MiB/s wr, 123 op/s 2024-03-19T02:41:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.418+0000 7f304f7fe640 1 -- 172.21.15.116:0/3443249110 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3040010a90 con 0x7f3058101060 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.418+0000 7f305fc0a740 1 Processor -- start 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 -- start start 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f304f7fe640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 0x7f30440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c1bf0 0x5617087ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 0x5617087ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617087244b0 con 0x5617087ca7d0 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617086ccd70 con 0x5617087c1bf0 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305fc0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561708720550 con 0x5617087c12d0 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f3056d9d640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 0x7f30440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f304f7fe640 1 -- 172.21.15.116:0/3443249110 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f30400cf4f0 con 0x7f3058101060 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58580/0 (socket says 172.21.15.116:58580) 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f304ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c1bf0 0x5617087ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 -- 172.21.15.116:0/4173002643 learned_addr learned my addr 172.21.15.116:0/4173002643 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f3056d9d640 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 0x5617087ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 -- 172.21.15.116:0/4173002643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c1bf0 msgr2=0x5617087ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c1bf0 0x5617087ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 -- 172.21.15.116:0/4173002643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 msgr2=0x5617087ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 0x5617087ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f305659c640 1 -- 172.21.15.116:0/4173002643 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617086ef210 con 0x5617087c12d0 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f3056d9d640 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 0x5617087ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.419+0000 7f3056d9d640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 0x7f30440a4e30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3050005cc0 tx=0x7f3050032e00 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f305659c640 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f30480115f0 tx=0x7f3048011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=577e60a2f5fcb876 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f304cff9640 1 -- 172.21.15.116:0/4173002643 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304801e070 con 0x5617087c12d0 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f304cff9640 1 -- 172.21.15.116:0/4173002643 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f304800fda0 con 0x5617087c12d0 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f304cff9640 1 -- 172.21.15.116:0/4173002643 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048019e50 con 0x5617087c12d0 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f305fc0a740 1 -- 172.21.15.116:0/4173002643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 msgr2=0x5617087c16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.420+0000 7f305fc0a740 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f30480115f0 tx=0x7f3048011ab0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 -- 172.21.15.116:0/4173002643 shutdown_connections 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087ca7d0 0x5617087ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c1bf0 0x5617087ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 --2- 172.21.15.116:0/4173002643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c12d0 0x5617087c16b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 -- 172.21.15.116:0/4173002643 >> 172.21.15.116:0/4173002643 conn(0x5617087bba00 msgr2=0x5617087bbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 -- 172.21.15.116:0/4173002643 shutdown_connections 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 -- 172.21.15.116:0/4173002643 wait complete. 2024-03-19T02:41:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 -- start start 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.421+0000 7f305fc0a740 1 Processor -- start 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617087cd5c0 con 0x5617087c1bf0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617086e0310 con 0x5617087c12d0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5617087cd740 con 0x5617087ca7d0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305fc0a740 10 client.0 _ll_register_callbacks cb 0x5617086d42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58590/0 (socket says 172.21.15.116:58590) 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 -- 172.21.15.116:0/586438436 learned_addr learned my addr 172.21.15.116:0/586438436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f305659c640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f304ffff640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 -- 172.21.15.116:0/586438436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 msgr2=0x5617087c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 -- 172.21.15.116:0/586438436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 msgr2=0x5617087c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.422+0000 7f3056d9d640 1 -- 172.21.15.116:0/586438436 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617084383f0 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f3056d9d640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30500328e0 tx=0x7f305002c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ae9fc4963bddd30 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050052060 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f3050004250 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f304800fda0 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3050059050 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 10 client.29182 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5617085027d0 con 0x5617087ca7d0 2024-03-19T02:41:17.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f304800fda0 con 0x5617087ca7d0 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f305003f070 con 0x5617087ca7d0 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 10 client.29182 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 10 client.29182 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 20 client.29182 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5617087daba0 0x5617087dcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.423+0000 7f305fc0a740 4 client.29182 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.424+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617087823e0 con 0x5617087daba0 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.424+0000 7f305659c640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5617087daba0 0x5617087dcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.424+0000 7f303dffb640 10 client.29182 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.424+0000 7f305659c640 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5617087daba0 0x5617087dcfd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1c0e5783f2ee3696 server_cookie=a492a53d9cf4f247 in_seq=0 out_seq=0 2024-03-19T02:41:17.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f303dffb640 1 -- 172.21.15.116:0/586438436 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617087823e0 con 0x5617087daba0 2024-03-19T02:41:17.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f303dffb640 10 client.29182 handle_command_reply: tid=0 2024-03-19T02:41:17.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 client.29182 shutdown 2024-03-19T02:41:17.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 msgr2=0x5617087c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30500328e0 tx=0x7f305002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 shutdown_connections 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x5617087c82d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5617087daba0 0x5617087dcfd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 --2- 172.21.15.116:0/586438436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f303dffb640 0 client.29182 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 >> 172.21.15.116:0/586438436 conn(0x5617087bba00 msgr2=0x5617087bd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 shutdown_connections 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.425+0000 7f305fc0a740 1 -- 172.21.15.116:0/586438436 wait complete. 2024-03-19T02:41:17.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.426+0000 7f305fc0a740 20 client.29182 trim_cache size 0 max 16384 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 Processor -- start 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 -- start start 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c1bf0 0x5617087d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 0x5617087baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b0e60 con 0x5617087c12d0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b2a90 con 0x5617087ca7d0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305fc0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b1530 con 0x5617087c1bf0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43994/0 (socket says 172.21.15.116:43994) 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f3056d9d640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 0x5617087baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305659c640 1 -- 172.21.15.116:0/1820420690 learned_addr learned my addr 172.21.15.116:0/1820420690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f304ffff640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c1bf0 0x5617087d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305659c640 1 -- 172.21.15.116:0/1820420690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c1bf0 msgr2=0x5617087d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.442+0000 7f305659c640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c1bf0 0x5617087d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f305659c640 1 -- 172.21.15.116:0/1820420690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 msgr2=0x5617087baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f305659c640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 0x5617087baa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f305659c640 1 -- 172.21.15.116:0/1820420690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617087bc5d0 con 0x5617087c12d0 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f3056d9d640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 0x5617087baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f305659c640 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3048011510 tx=0x7f3048009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fad84819301467 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f303ffff640 1 -- 172.21.15.116:0/1820420690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304801e040 con 0x5617087c12d0 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f303ffff640 1 -- 172.21.15.116:0/1820420690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f304802a070 con 0x5617087c12d0 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.443+0000 7f303ffff640 1 -- 172.21.15.116:0/1820420690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048005600 con 0x5617087c12d0 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- 172.21.15.116:0/1820420690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 msgr2=0x5617087e28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3048011510 tx=0x7f3048009dc0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- 172.21.15.116:0/1820420690 shutdown_connections 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087ca7d0 0x5617087baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087c1bf0 0x5617087d2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- 172.21.15.116:0/1820420690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c12d0 0x5617087e28a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- 172.21.15.116:0/1820420690 >> 172.21.15.116:0/1820420690 conn(0x5617087bba00 msgr2=0x5617087ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- 172.21.15.116:0/1820420690 shutdown_connections 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- 172.21.15.116:0/1820420690 wait complete. 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- start start 2024-03-19T02:41:17.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 Processor -- start 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x56170883fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b2300 con 0x5617087c1bf0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30581b1c30 con 0x5617087c12d0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30581aee60 con 0x5617087ca7d0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305fc0a740 10 client.0 _ll_register_callbacks cb 0x7f3040005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f304ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40680/0 (socket says 172.21.15.116:40680) 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f305659c640 1 -- 172.21.15.116:0/3144609215 learned_addr learned my addr 172.21.15.116:0/3144609215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f3056d9d640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x56170883fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.444+0000 7f304ffff640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44006/0 (socket says 172.21.15.116:44006) 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 -- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 msgr2=0x56170883fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x56170883fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 -- 172.21.15.116:0/3144609215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 msgr2=0x5617087c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 -- 172.21.15.116:0/3144609215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561708840410 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305659c640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f304801c040 tx=0x7f3048009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4865eb7e94f91eed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f303dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304801e030 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f304802a070 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f3048005380 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f304803d050 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305fc0a740 10 client.29237 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561708841550 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.445+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f304802a070 con 0x5617087c12d0 2024-03-19T02:41:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f304800f6c0 con 0x5617087c12d0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 10 client.29237 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 10 client.29237 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 20 client.29237 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x561708841ae0 0x561708843f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 4 client.29237 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617087319f0 con 0x561708841ae0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f304ffff640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x561708841ae0 0x561708843f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f303dffb640 10 client.29237 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f304ffff640 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x561708841ae0 0x561708843f00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4d469152f2c4d0db server_cookie=45d111a02a754676 in_seq=0 out_seq=0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f303dffb640 1 -- 172.21.15.116:0/3144609215 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617087319f0 con 0x561708841ae0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f303dffb640 10 client.29237 handle_command_reply: tid=0 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 1 client.29237 shutdown 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 msgr2=0x5617087c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.446+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f304801c040 tx=0x7f3048009ec0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 shutdown_connections 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5617087ca7d0 0x56170883fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x561708841ae0 0x561708843f00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617087c1bf0 0x5617087c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 --2- 172.21.15.116:0/3144609215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617087c12d0 0x5617087c4950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f303dffb640 0 client.29237 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 >> 172.21.15.116:0/3144609215 conn(0x5617087bba00 msgr2=0x5617087ba310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 shutdown_connections 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 1 -- 172.21.15.116:0/3144609215 wait complete. 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305fc0a740 20 client.29237 trim_cache size 0 max 16384 2024-03-19T02:41:17.454 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 msgr2=0x7f30440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 0x7f30440a4e30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3050005cc0 tx=0x7f3050032e00 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 msgr2=0x7f30581984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 0x7f30581984a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f304000e9f0 tx=0x7f304000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 shutdown_connections 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30440a2970 0x7f30440a4e30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581066b0 0x7f305819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3058101060 0x7f30581984a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 --2- 172.21.15.116:0/3443249110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30581006a0 0x7f3058197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 >> 172.21.15.116:0/3443249110 conn(0x7f30580778d0 msgr2=0x7f3058068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 shutdown_connections 2024-03-19T02:41:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.447+0000 7f305cf3b640 1 -- 172.21.15.116:0/3443249110 wait complete. 2024-03-19T02:41:17.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:41:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:17 smithi154 ceph-mon[35693]: pgmap v423: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 3.2 MiB/s wr, 123 op/s 2024-03-19T02:41:17.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- 172.21.15.116:0/1647650976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b0102150 msgr2=0x7f59b010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 --2- 172.21.15.116:0/1647650976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b0102150 0x7f59b010e660 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009cd0 tx=0x7f59a802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- 172.21.15.116:0/1647650976 shutdown_connections 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 --2- 172.21.15.116:0/1647650976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b0102150 0x7f59b010e660 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 --2- 172.21.15.116:0/1647650976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0101790 0x7f59b0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 --2- 172.21.15.116:0/1647650976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b01077a0 0x7f59b0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- 172.21.15.116:0/1647650976 >> 172.21.15.116:0/1647650976 conn(0x7f59b00fd2c0 msgr2=0x7f59b00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- 172.21.15.116:0/1647650976 shutdown_connections 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- 172.21.15.116:0/1647650976 wait complete. 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 Processor -- start 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.633+0000 7f59b82d1640 1 -- start start 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 0x7f59b0074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 0x7f59b00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0111a60 con 0x7f59b01077a0 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01118e0 con 0x7f59b0101790 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b82d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59b0111be0 con 0x7f59b0102150 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39988/0 (socket says 172.21.15.116:39988) 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b686e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 0x7f59b00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b586c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 0x7f59b0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 -- 172.21.15.116:0/63109698 learned_addr learned my addr 172.21.15.116:0/63109698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 -- 172.21.15.116:0/63109698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 msgr2=0x7f59b0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 0x7f59b0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 -- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 msgr2=0x7f59b00745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 0x7f59b00745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 -- 172.21.15.116:0/63109698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b01a6480 con 0x7f59b0101790 2024-03-19T02:41:17.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b686e640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 0x7f59b00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b586c640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 0x7f59b0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.634+0000 7f59b606d640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00ea60 tx=0x7f59ac00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.635+0000 7f59a77fe640 1 -- 172.21.15.116:0/63109698 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac00cde0 con 0x7f59b0101790 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.635+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b01a6770 con 0x7f59b0101790 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.635+0000 7f59a77fe640 1 -- 172.21.15.116:0/63109698 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac0035f0 con 0x7f59b0101790 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.635+0000 7f59a77fe640 1 -- 172.21.15.116:0/63109698 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac010690 con 0x7f59b0101790 2024-03-19T02:41:17.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.635+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b01a6cc0 con 0x7f59b0101790 2024-03-19T02:41:17.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.637+0000 7f59a77fe640 1 -- 172.21.15.116:0/63109698 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f59ac0037a0 con 0x7f59b0101790 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 Processor -- start 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59a77fe640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 0x7f599c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 -- start start 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59b686e640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 0x7f599c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59a77fe640 1 -- 172.21.15.116:0/63109698 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f59ac0cf450 con 0x7f59b0101790 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2ec290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c2464b0 con 0x55ba9c2ec7d0 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c1eed70 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59baf96740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c242550 con 0x55ba9c2e32d0 2024-03-19T02:41:17.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59b606d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59b586c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59b606d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55876/0 (socket says 172.21.15.116:55876) 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.638+0000 7f59b606d640 1 -- 172.21.15.116:0/1267610341 learned_addr learned my addr 172.21.15.116:0/1267610341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b686e640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b686e640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 0x7f599c0a4e30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f59a8005cc0 tx=0x7f59a8005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 -- 172.21.15.116:0/1267610341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 msgr2=0x55ba9c2ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2ec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 -- 172.21.15.116:0/1267610341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 msgr2=0x55ba9c2eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 -- 172.21.15.116:0/1267610341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba9c211210 con 0x55ba9c2e32d0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b586c640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b686e640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59b606d640 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59ac09b3d0 tx=0x7f59ac09b210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7358e2f953d9072f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59a4bfe640 1 -- 172.21.15.116:0/1267610341 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b4070 con 0x55ba9c2e32d0 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59a4bfe640 1 -- 172.21.15.116:0/1267610341 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac0860c0 con 0x55ba9c2e32d0 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.639+0000 7f59a4bfe640 1 -- 172.21.15.116:0/1267610341 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0cf170 con 0x55ba9c2e32d0 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- 172.21.15.116:0/1267610341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 msgr2=0x55ba9c2e36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59ac09b3d0 tx=0x7f59ac09b210 comp rx=0 tx=0).stop 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- 172.21.15.116:0/1267610341 shutdown_connections 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 --2- 172.21.15.116:0/1267610341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e36b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- 172.21.15.116:0/1267610341 >> 172.21.15.116:0/1267610341 conn(0x55ba9c2dda00 msgr2=0x55ba9c2ddde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- 172.21.15.116:0/1267610341 shutdown_connections 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- 172.21.15.116:0/1267610341 wait complete. 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 -- start start 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.640+0000 7f59baf96740 1 Processor -- start 2024-03-19T02:41:17.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2ea350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c2ef5c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c202310 con 0x55ba9c2e32d0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba9c2ef740 con 0x55ba9c2ec7d0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 10 client.0 _ll_register_callbacks cb 0x55ba9c1f62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b606d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b606d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40014/0 (socket says 172.21.15.116:40014) 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b606d640 1 -- 172.21.15.116:0/3474307064 learned_addr learned my addr 172.21.15.116:0/3474307064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b686e640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2ea350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 -- 172.21.15.116:0/3474307064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2ec7d0 msgr2=0x55ba9c2ea350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2ea350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 -- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 msgr2=0x55ba9c2e98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 -- 172.21.15.116:0/3474307064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba9bf5a3f0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59b586c640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e9e10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f59b00755b0 tx=0x7f59a000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b0ca3fba9aa87c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f5999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0012110 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f59a0004050 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.641+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ac0860c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f59a00122b0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 10 client.19470 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ba9c0247d0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac0860c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f59a0010430 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 10 client.19470 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 10 client.19470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 20 client.19470 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c2fcce0 0x55ba9c2ff100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 4 client.19470 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba9c2a43e0 con 0x55ba9c2fcce0 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.642+0000 7f59b606d640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c2fcce0 0x55ba9c2ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.643+0000 7f5999ffb640 10 client.19470 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:17.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.643+0000 7f59b606d640 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c2fcce0 0x55ba9c2ff100 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce88a68fa0bef852 server_cookie=ad6c9b60b4ea0a79 in_seq=0 out_seq=0 2024-03-19T02:41:17.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f5999ffb640 1 -- 172.21.15.116:0/3474307064 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ba9c2a43e0 con 0x55ba9c2fcce0 2024-03-19T02:41:17.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f5999ffb640 10 client.19470 handle_command_reply: tid=0 2024-03-19T02:41:17.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 client.19470 shutdown 2024-03-19T02:41:17.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 msgr2=0x55ba9c2e9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e9e10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f59b00755b0 tx=0x7f59a000f790 comp rx=0 tx=0).stop 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 shutdown_connections 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c2fcce0 0x55ba9c2ff100 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2ea350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f5999ffb640 0 client.19470 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e9e10 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 --2- 172.21.15.116:0/3474307064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2e98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 >> 172.21.15.116:0/3474307064 conn(0x55ba9c2dda00 msgr2=0x55ba9c2df750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 shutdown_connections 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 1 -- 172.21.15.116:0/3474307064 wait complete. 2024-03-19T02:41:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.644+0000 7f59baf96740 20 client.19470 trim_cache size 0 max 16384 2024-03-19T02:41:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 Processor -- start 2024-03-19T02:41:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 -- start start 2024-03-19T02:41:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2f8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01b0ef0 con 0x55ba9c2e32d0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01b2b20 con 0x55ba9c2ec7d0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59baf96740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01b15c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59b606d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59b606d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40778/0 (socket says 172.21.15.116:40778) 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59b606d640 1 -- 172.21.15.116:0/877278801 learned_addr learned my addr 172.21.15.116:0/877278801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59b586c640 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.661+0000 7f59b686e640 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 -- 172.21.15.116:0/877278801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e3bf0 msgr2=0x55ba9c2f8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2f8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 -- 172.21.15.116:0/877278801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 msgr2=0x55ba9c2e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 -- 172.21.15.116:0/877278801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba9c2eef30 con 0x55ba9c2e32d0 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f59b606d640 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0980a0 tx=0x7f59ac086050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b508cdd7d5f6461 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f599bfff640 1 -- 172.21.15.116:0/877278801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b4040 con 0x55ba9c2e32d0 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f599bfff640 1 -- 172.21.15.116:0/877278801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac0860c0 con 0x55ba9c2e32d0 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.662+0000 7f599bfff640 1 -- 172.21.15.116:0/877278801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0cf170 con 0x55ba9c2e32d0 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- 172.21.15.116:0/877278801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 msgr2=0x55ba9c2ec000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0980a0 tx=0x7f59ac086050 comp rx=0 tx=0).stop 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- 172.21.15.116:0/877278801 shutdown_connections 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2f8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- 172.21.15.116:0/877278801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2ec000 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- 172.21.15.116:0/877278801 >> 172.21.15.116:0/877278801 conn(0x55ba9c2dda00 msgr2=0x55ba9c2dc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- 172.21.15.116:0/877278801 shutdown_connections 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- 172.21.15.116:0/877278801 wait complete. 2024-03-19T02:41:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- start start 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 Processor -- start 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01b2390 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01b1cc0 con 0x55ba9c2ec7d0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59b01aeef0 con 0x55ba9c2e32d0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.663+0000 7f59baf96740 10 client.0 _ll_register_callbacks cb 0x7f59a0009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40780/0 (socket says 172.21.15.116:40780) 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 -- 172.21.15.116:0/609948801 learned_addr learned my addr 172.21.15.116:0/609948801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b686e640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b606d640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 -- 172.21.15.116:0/609948801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 msgr2=0x55ba9c2f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 -- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 msgr2=0x55ba9c2e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 -- 172.21.15.116:0/609948801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba9c362040 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59b586c640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x55ba9c2eaea0 tx=0x7f59a0010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce16fca15610a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f5999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a002d030 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f59a0018040 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ac0860c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f59a00128a0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59baf96740 10 client.19476 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.664+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ba9c363270 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac0860c0 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f59a0003c20 con 0x55ba9c2e3bf0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 10 client.19476 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 10 client.19476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 20 client.19476 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c363aa0 0x55ba9c365ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 4 client.19476 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba9c2539f0 con 0x55ba9c363aa0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59b606d640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c363aa0 0x55ba9c365ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f5999ffb640 10 client.19476 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59b606d640 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c363aa0 0x55ba9c365ec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e9f08eafb8a9f0e server_cookie=81590e2ef6023529 in_seq=0 out_seq=0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f5999ffb640 1 -- 172.21.15.116:0/609948801 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba9c2539f0 con 0x55ba9c363aa0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f5999ffb640 10 client.19476 handle_command_reply: tid=0 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.665+0000 7f59baf96740 1 client.19476 shutdown 2024-03-19T02:41:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 msgr2=0x55ba9c2e6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x55ba9c2eaea0 tx=0x7f59a0010cf0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 shutdown_connections 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55ba9c363aa0 0x55ba9c365ec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ba9c2ec7d0 0x55ba9c2e6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9c2e3bf0 0x55ba9c2e6330 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 --2- 172.21.15.116:0/609948801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9c2e32d0 0x55ba9c2f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 >> 172.21.15.116:0/609948801 conn(0x55ba9c2dda00 msgr2=0x55ba9c2dcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f5999ffb640 0 client.19476 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 shutdown_connections 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 1 -- 172.21.15.116:0/609948801 wait complete. 2024-03-19T02:41:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59baf96740 20 client.19476 trim_cache size 0 max 16384 2024-03-19T02:41:17.674 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 msgr2=0x7f599c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 0x7f599c0a4e30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f59a8005cc0 tx=0x7f59a8005c50 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 msgr2=0x7f59b0073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f59ac00ea60 tx=0x7f59ac00ef20 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 shutdown_connections 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f599c0a2970 0x7f599c0a4e30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59b01077a0 0x7f59b00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59b0102150 0x7f59b0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 --2- 172.21.15.116:0/63109698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101790 0x7f59b0073b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 >> 172.21.15.116:0/63109698 conn(0x7f59b00fd2c0 msgr2=0x7f59b00fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 shutdown_connections 2024-03-19T02:41:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.666+0000 7f59b82d1640 1 -- 172.21.15.116:0/63109698 wait complete. 2024-03-19T02:41:17.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 -- 172.21.15.116:0/2817442148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc101790 msgr2=0x7f41dc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 --2- 172.21.15.116:0/2817442148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc101790 0x7f41dc101c10 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009890 tx=0x7f41d002bb70 comp rx=0 tx=0).stop 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 -- 172.21.15.116:0/2817442148 shutdown_connections 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 --2- 172.21.15.116:0/2817442148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc102150 0x7f41dc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 --2- 172.21.15.116:0/2817442148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc101790 0x7f41dc101c10 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 --2- 172.21.15.116:0/2817442148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 0x7f41dc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 -- 172.21.15.116:0/2817442148 >> 172.21.15.116:0/2817442148 conn(0x7f41dc0fd2c0 msgr2=0x7f41dc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 -- 172.21.15.116:0/2817442148 shutdown_connections 2024-03-19T02:41:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 -- 172.21.15.116:0/2817442148 wait complete. 2024-03-19T02:41:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.805+0000 7f41e1827640 1 Processor -- start 2024-03-19T02:41:17.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 -- start start 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 0x7f41dc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 0x7f41dc19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc111a20 con 0x7f41dc102150 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1118a0 con 0x7f41dc101790 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41e1827640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc111ba0 con 0x7f41dc1077a0 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41da7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41da7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40808/0 (socket says 172.21.15.116:40808) 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 0x7f41dc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41da7fc640 1 -- 172.21.15.116:0/1489812376 learned_addr learned my addr 172.21.15.116:0/1489812376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41db7fe640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 0x7f41dc19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 -- 172.21.15.116:0/1489812376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 msgr2=0x7f41dc19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 0x7f41dc19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 -- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 msgr2=0x7f41dc198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.806+0000 7f41daffd640 1 -- 172.21.15.116:0/1489812376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc19cfc0 con 0x7f41dc101790 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41da7fc640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41daffd640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 0x7f41dc197fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00e9f0 tx=0x7f41cc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41e0825640 1 -- 172.21.15.116:0/1489812376 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00cd80 con 0x7f41dc101790 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc19d2b0 con 0x7f41dc101790 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41e0825640 1 -- 172.21.15.116:0/1489812376 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41cc0035f0 con 0x7f41dc101790 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1a4b40 con 0x7f41dc101790 2024-03-19T02:41:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.807+0000 7f41e0825640 1 -- 172.21.15.116:0/1489812376 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc010690 con 0x7f41dc101790 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.809+0000 7f41e0825640 1 -- 172.21.15.116:0/1489812376 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f41cc004000 con 0x7f41dc101790 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.810+0000 7f41e4413740 1 Processor -- start 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.810+0000 7f41e0825640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 0x7f41c40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.810+0000 7f41e4413740 1 -- start start 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.810+0000 7f41e0825640 1 -- 172.21.15.116:0/1489812376 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f41cc0cf370 con 0x7f41dc101790 2024-03-19T02:41:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41db7fe640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 0x7f41c40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 0x55c7f27a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 0x55c7f27ab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7f27054b0 con 0x55c7f27a2bf0 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c7f26add70 con 0x55c7f27a22d0 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.811+0000 7f41e4413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7f2701550 con 0x55c7f27ab7d0 2024-03-19T02:41:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.812+0000 7f41db7fe640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 0x7f41c40a4eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005fd0 tx=0x7f41d4010040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.812+0000 7f41db7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.812+0000 7f41db7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55920/0 (socket says 172.21.15.116:55920) 2024-03-19T02:41:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.812+0000 7f41db7fe640 1 -- 172.21.15.116:0/2942263384 learned_addr learned my addr 172.21.15.116:0/2942263384 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.812+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 0x55c7f27ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 -- 172.21.15.116:0/2942263384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 msgr2=0x55c7f27a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41daffd640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 0x55c7f27a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 0x55c7f27a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 -- 172.21.15.116:0/2942263384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 msgr2=0x55c7f27ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 0x55c7f27ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 -- 172.21.15.116:0/2942263384 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7f26d0210 con 0x55c7f27ab7d0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 0x55c7f27ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41daffd640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 0x55c7f27a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.813+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f41d400d820 tx=0x7f41d400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb3c8ae19a959f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.814+0000 7f41c1ffb640 1 -- 172.21.15.116:0/2942263384 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4019b10 con 0x55c7f27ab7d0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.814+0000 7f41c1ffb640 1 -- 172.21.15.116:0/2942263384 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d4028070 con 0x55c7f27ab7d0 2024-03-19T02:41:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.814+0000 7f41c1ffb640 1 -- 172.21.15.116:0/2942263384 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d40165a0 con 0x55c7f27ab7d0 2024-03-19T02:41:17.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.815+0000 7f41e4413740 1 -- 172.21.15.116:0/2942263384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 msgr2=0x55c7f27adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.815+0000 7f41e4413740 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f41d400d820 tx=0x7f41d400ef90 comp rx=0 tx=0).stop 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 -- 172.21.15.116:0/2942263384 shutdown_connections 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f27adbc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a2bf0 0x55c7f27ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 --2- 172.21.15.116:0/2942263384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a22d0 0x55c7f27a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 -- 172.21.15.116:0/2942263384 >> 172.21.15.116:0/2942263384 conn(0x55c7f279ca00 msgr2=0x55c7f279cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.816+0000 7f41e4413740 1 -- 172.21.15.116:0/2942263384 shutdown_connections 2024-03-19T02:41:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.817+0000 7f41e4413740 1 -- 172.21.15.116:0/2942263384 wait complete. 2024-03-19T02:41:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.817+0000 7f41e4413740 1 -- start start 2024-03-19T02:41:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.817+0000 7f41e4413740 1 Processor -- start 2024-03-19T02:41:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27ab7d0 0x55c7f27a9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41daffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41daffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55922/0 (socket says 172.21.15.116:55922) 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41daffd640 1 -- 172.21.15.116:0/2111304572 learned_addr learned my addr 172.21.15.116:0/2111304572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7f27b2940 con 0x55c7f27ab7d0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c7f26c1310 con 0x55c7f27a2bf0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7f27b2ac0 con 0x55c7f27a22d0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27ab7d0 0x55c7f27a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.818+0000 7f41e4413740 10 client.0 _ll_register_callbacks cb 0x55c7f26b52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 -- 172.21.15.116:0/2111304572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 msgr2=0x55c7f27a8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 -- 172.21.15.116:0/2111304572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27ab7d0 msgr2=0x55c7f27a9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27ab7d0 0x55c7f27a9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7f24193f0 con 0x55c7f27a22d0 2024-03-19T02:41:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d4028070 con 0x55c7f27a22d0 2024-03-19T02:41:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41daffd640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097a50 tx=0x7f41cc09be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=509e908b71ce841e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.819+0000 7f41baffd640 10 client.29218 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41baffd640 1 -- 172.21.15.116:0/2111304572 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0b4060 con 0x55c7f27a22d0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41baffd640 1 -- 172.21.15.116:0/2111304572 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f41cc0cc960 con 0x55c7f27a22d0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41baffd640 1 -- 172.21.15.116:0/2111304572 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41cc097c60 con 0x55c7f27a22d0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 10 client.29218 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55c7f24e37d0 con 0x55c7f27a22d0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41baffd640 1 -- 172.21.15.116:0/2111304572 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f41cc0b4200 con 0x55c7f27a22d0 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 10 client.29218 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 10 client.29218 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 20 client.29218 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 4 client.29218 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.820+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7f27633e0 con 0x55c7f27bfcd0 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.821+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.822+0000 7f41baffd640 10 client.29218 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:17.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.822+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27c2190 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65772650cd0df517 server_cookie=aaf700e3d0dd68d1 in_seq=0 out_seq=0 2024-03-19T02:41:17.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41baffd640 1 -- 172.21.15.116:0/2111304572 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7f27633e0 con 0x55c7f27bfcd0 2024-03-19T02:41:17.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41baffd640 10 client.29218 handle_command_reply: tid=0 2024-03-19T02:41:17.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 client.29218 shutdown 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 msgr2=0x55c7f27a87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097a50 tx=0x7f41cc09be60 comp rx=0 tx=0).stop 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 shutdown_connections 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27c2190 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41baffd640 0 client.29218 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27ab7d0 0x55c7f27a9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 --2- 172.21.15.116:0/2111304572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a22d0 0x55c7f27a87c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.823+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 >> 172.21.15.116:0/2111304572 conn(0x55c7f279ca00 msgr2=0x55c7f279e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.824+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 shutdown_connections 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.824+0000 7f41e4413740 1 -- 172.21.15.116:0/2111304572 wait complete. 2024-03-19T02:41:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.824+0000 7f41e4413740 20 client.29218 trim_cache size 0 max 16384 2024-03-19T02:41:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 Processor -- start 2024-03-19T02:41:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 -- start start 2024-03-19T02:41:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a2bf0 0x55c7f27bb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27ab7d0 0x55c7f27c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8010020 con 0x55c7f27a22d0 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8011c50 con 0x55c7f27ab7d0 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.847+0000 7f41e4413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80106f0 con 0x55c7f27a2bf0 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41da7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a2bf0 0x55c7f27bb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40828/0 (socket says 172.21.15.116:40828) 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 -- 172.21.15.116:0/2350609512 learned_addr learned my addr 172.21.15.116:0/2350609512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27ab7d0 0x55c7f27c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 -- 172.21.15.116:0/2350609512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a2bf0 msgr2=0x55c7f27bb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a2bf0 0x55c7f27bb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 -- 172.21.15.116:0/2350609512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27ab7d0 msgr2=0x55c7f27c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27ab7d0 0x55c7f27c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 -- 172.21.15.116:0/2350609512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7f27a6d30 con 0x55c7f27a22d0 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.848+0000 7f41daffd640 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0ccce0 tx=0x7f41cc0ce000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb6b0f95d6142f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.849+0000 7f41c17fa640 1 -- 172.21.15.116:0/2350609512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0b4040 con 0x55c7f27a22d0 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.849+0000 7f41c17fa640 1 -- 172.21.15.116:0/2350609512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d4028070 con 0x55c7f27a22d0 2024-03-19T02:41:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.849+0000 7f41c17fa640 1 -- 172.21.15.116:0/2350609512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc09aa40 con 0x55c7f27a22d0 2024-03-19T02:41:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.850+0000 7f41e4413740 1 -- 172.21.15.116:0/2350609512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 msgr2=0x55c7f27aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.850+0000 7f41e4413740 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0ccce0 tx=0x7f41cc0ce000 comp rx=0 tx=0).stop 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 -- 172.21.15.116:0/2350609512 shutdown_connections 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27ab7d0 0x55c7f27c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27a2bf0 0x55c7f27bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 --2- 172.21.15.116:0/2350609512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27aaef0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 -- 172.21.15.116:0/2350609512 >> 172.21.15.116:0/2350609512 conn(0x55c7f279ca00 msgr2=0x55c7f279bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 -- 172.21.15.116:0/2350609512 shutdown_connections 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 -- 172.21.15.116:0/2350609512 wait complete. 2024-03-19T02:41:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.851+0000 7f41e4413740 1 -- start start 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 Processor -- start 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f2825090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80114c0 con 0x55c7f27a22d0 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8010df0 con 0x55c7f27a2bf0 2024-03-19T02:41:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f41c800e020 con 0x55c7f27ab7d0 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41e4413740 10 client.0 _ll_register_callbacks cb 0x7f41cc09b3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41da7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41da7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40076/0 (socket says 172.21.15.116:40076) 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27a5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40838/0 (socket says 172.21.15.116:40838) 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41da7fc640 1 -- 172.21.15.116:0/2439193304 learned_addr learned my addr 172.21.15.116:0/2439193304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.852+0000 7f41db7fe640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f2825090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 -- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 msgr2=0x55c7f2825090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f2825090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 -- 172.21.15.116:0/2439193304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 msgr2=0x55c7f27a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 -- 172.21.15.116:0/2439193304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7f2825620 con 0x55c7f27a2bf0 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41da7fc640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41d0000c00 tx=0x7f41d002c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ee18216d2cd8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:17.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41baffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0039040 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d4028070 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f41d0034c00 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.853+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41d005e050 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.854+0000 7f41e4413740 10 client.29270 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.854+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c7f27ae3b0 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.854+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d4028070 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.854+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f41d002c930 con 0x55c7f27a2bf0 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 10 client.29270 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 10 client.29270 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 20 client.29270 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 4 client.29270 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7f27129f0 con 0x55c7f27bfcd0 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.855+0000 7f41daffd640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41daffd640 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27b0ba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86202e32bcf38738 server_cookie=28791232a74fc6e9 in_seq=0 out_seq=0 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41baffd640 10 client.29270 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41baffd640 1 -- 172.21.15.116:0/2439193304 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7f27129f0 con 0x55c7f27bfcd0 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41baffd640 10 client.29270 handle_command_reply: tid=0 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 client.29270 shutdown 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 msgr2=0x55c7f27a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41d0000c00 tx=0x7f41d002c310 comp rx=0 tx=0).stop 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 shutdown_connections 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c7f27bfcd0 0x55c7f27b0ba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7f27ab7d0 0x55c7f2825090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7f27a2bf0 0x55c7f27a6000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 --2- 172.21.15.116:0/2439193304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f27a22d0 0x55c7f27a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 >> 172.21.15.116:0/2439193304 conn(0x55c7f279ca00 msgr2=0x55c7f279b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.856+0000 7f41baffd640 0 client.29270 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:17.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.857+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 shutdown_connections 2024-03-19T02:41:17.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.857+0000 7f41e4413740 1 -- 172.21.15.116:0/2439193304 wait complete. 2024-03-19T02:41:17.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.857+0000 7f41e4413740 20 client.29270 trim_cache size 0 max 16384 2024-03-19T02:41:17.860 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:17.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 msgr2=0x7f41c40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 0x7f41c40a4eb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005fd0 tx=0x7f41d4010040 comp rx=0 tx=0).stop 2024-03-19T02:41:17.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 msgr2=0x7f41dc197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:17.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 0x7f41dc197fd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00e9f0 tx=0x7f41cc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 shutdown_connections 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f41c40a29f0 0x7f41c40a4eb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41dc1077a0 0x7f41dc19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41dc102150 0x7f41dc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 --2- 172.21.15.116:0/1489812376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41dc101790 0x7f41dc197fd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 >> 172.21.15.116:0/1489812376 conn(0x7f41dc0fd2c0 msgr2=0x7f41dc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 shutdown_connections 2024-03-19T02:41:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:17.859+0000 7f41e1827640 1 -- 172.21.15.116:0/1489812376 wait complete. 2024-03-19T02:41:17.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/3670181691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24102150 msgr2=0x7f1b2410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/3670181691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24102150 0x7f1b2410e660 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009cd0 tx=0x7f1b1c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/3670181691 shutdown_connections 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/3670181691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24102150 0x7f1b2410e660 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/3670181691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24101790 0x7f1b24101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/3670181691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 0x7f1b24107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/3670181691 >> 172.21.15.116:0/3670181691 conn(0x7f1b240fd2c0 msgr2=0x7f1b240ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/3670181691 shutdown_connections 2024-03-19T02:41:18.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.002+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/3670181691 wait complete. 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 Processor -- start 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 -- start start 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 0x7f1b2419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24111a60 con 0x7f1b24101790 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241118e0 con 0x7f1b241077a0 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2b8ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24111be0 con 0x7f1b24102150 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2969b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b28e9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2969b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40854/0 (socket says 172.21.15.116:40854) 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b2969b640 1 -- 172.21.15.116:0/694249539 learned_addr learned my addr 172.21.15.116:0/694249539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b28e9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55968/0 (socket says 172.21.15.116:55968) 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.003+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 0x7f1b2419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 -- 172.21.15.116:0/694249539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 msgr2=0x7f1b24198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 -- 172.21.15.116:0/694249539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 msgr2=0x7f1b2419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 0x7f1b2419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 -- 172.21.15.116:0/694249539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2419cfd0 con 0x7f1b24101790 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2969b640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000e9f0 tx=0x7f1b2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b1a7fc640 1 -- 172.21.15.116:0/694249539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2000cd80 con 0x7f1b24101790 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b1a7fc640 1 -- 172.21.15.116:0/694249539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b200108e0 con 0x7f1b24101790 2024-03-19T02:41:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b1a7fc640 1 -- 172.21.15.116:0/694249539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20018920 con 0x7f1b24101790 2024-03-19T02:41:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2419d2c0 con 0x7f1b24101790 2024-03-19T02:41:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.004+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b24073a50 con 0x7f1b24101790 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.007+0000 7f1b1a7fc640 1 -- 172.21.15.116:0/694249539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1b20010a90 con 0x7f1b24101790 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 Processor -- start 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 -- start start 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501525a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015263290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650151bd4b0 con 0x5650152637d0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565015165d70 con 0x56501525a2d0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5650151b9550 con 0x56501525abf0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b1a7fc640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 0x7f1b100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b28e9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b28e9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40868/0 (socket says 172.21.15.116:40868) 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b28e9a640 1 -- 172.21.15.116:0/2086607112 learned_addr learned my addr 172.21.15.116:0/2086607112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b2969b640 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015263290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501525a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.008+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 0x7f1b100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 -- 172.21.15.116:0/2086607112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 msgr2=0x565015263290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015263290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 -- 172.21.15.116:0/2086607112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 msgr2=0x56501525a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501525a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 -- 172.21.15.116:0/2086607112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565015188210 con 0x5650152637d0 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b1a7fc640 1 -- 172.21.15.116:0/694249539 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1b200cf4d0 con 0x7f1b24101790 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b140096f0 tx=0x7f1b14005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b8efeb323e5cce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.009+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 0x7f1b100a4e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0098d0 tx=0x7f1b1c005dc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.010+0000 7f1b0f7fe640 1 -- 172.21.15.116:0/2086607112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1401e030 con 0x5650152637d0 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.010+0000 7f1b0f7fe640 1 -- 172.21.15.116:0/2086607112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b14009d90 con 0x5650152637d0 2024-03-19T02:41:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.010+0000 7f1b0f7fe640 1 -- 172.21.15.116:0/2086607112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003850 con 0x5650152637d0 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2086607112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 msgr2=0x565015265bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f1b140096f0 tx=0x7f1b14005f90 comp rx=0 tx=0).stop 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2086607112 shutdown_connections 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x565015265bc0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015263290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2086607112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501525a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2086607112 >> 172.21.15.116:0/2086607112 conn(0x565015254a00 msgr2=0x565015254de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2086607112 shutdown_connections 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.011+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2086607112 wait complete. 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.012+0000 7f1b2e5c6740 1 -- start start 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.012+0000 7f1b2e5c6740 1 Processor -- start 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x565015260870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x565015260db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650152665c0 con 0x5650152637d0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565015179310 con 0x56501525abf0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565015266740 con 0x56501525a2d0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2e5c6740 10 client.0 _ll_register_callbacks cb 0x56501516d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40874/0 (socket says 172.21.15.116:40874) 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 -- 172.21.15.116:0/3348550423 learned_addr learned my addr 172.21.15.116:0/3348550423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x565015260db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b2969b640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x565015260870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 -- 172.21.15.116:0/3348550423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 msgr2=0x565015260870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x565015260870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 -- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 msgr2=0x565015260db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x565015260db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 -- 172.21.15.116:0/3348550423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565014ed13f0 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036000 tx=0x7f1b1c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c620e0961a278c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.013+0000 7f1b0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c054060 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1b1c004040 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b14009d90 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1b1c039040 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b2e5c6740 10 client.19506 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565014f9b7d0 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b14009d90 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f1b1c04f930 con 0x5650152637d0 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b2e5c6740 10 client.19506 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.014+0000 7f1b2e5c6740 10 client.19506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2e5c6740 20 client.19506 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:18.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152740b0 0x565015276550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2e5c6740 4 client.19506 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56501521b3e0 con 0x5650152740b0 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2969b640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152740b0 0x565015276550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b0cff9640 10 client.19506 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.015+0000 7f1b2969b640 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152740b0 0x565015276550 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f50533117ca68506 server_cookie=6619bf21b0411528 in_seq=0 out_seq=0 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b0cff9640 1 -- 172.21.15.116:0/3348550423 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56501521b3e0 con 0x5650152740b0 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b0cff9640 10 client.19506 handle_command_reply: tid=0 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b2e5c6740 1 client.19506 shutdown 2024-03-19T02:41:18.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 msgr2=0x5650152612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036000 tx=0x7f1b1c0095c0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.017+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 shutdown_connections 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152740b0 0x565015276550 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x5650152612f0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x565015260db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/3348550423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x565015260870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 >> 172.21.15.116:0/3348550423 conn(0x565015254a00 msgr2=0x565015256750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b0cff9640 0 client.19506 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 shutdown_connections 2024-03-19T02:41:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/3348550423 wait complete. 2024-03-19T02:41:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.018+0000 7f1b2e5c6740 20 client.19506 trim_cache size 0 max 16384 2024-03-19T02:41:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 Processor -- start 2024-03-19T02:41:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 -- start start 2024-03-19T02:41:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501526f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015253a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241acb10 con 0x5650152637d0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ae740 con 0x56501525a2d0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ad1e0 con 0x56501525abf0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40886/0 (socket says 172.21.15.116:40886) 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 -- 172.21.15.116:0/2030598823 learned_addr learned my addr 172.21.15.116:0/2030598823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015253a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b2969b640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501526f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 -- 172.21.15.116:0/2030598823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 msgr2=0x565015253a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015253a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 -- 172.21.15.116:0/2030598823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 msgr2=0x56501526f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501526f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b29e9c640 1 -- 172.21.15.116:0/2030598823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56501522a880 con 0x5650152637d0 2024-03-19T02:41:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.033+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015253a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.034+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009a60 tx=0x7f1b1c005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec6dbf4880b0994 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.034+0000 7f1b0effd640 1 -- 172.21.15.116:0/2030598823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c054040 con 0x5650152637d0 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.034+0000 7f1b0effd640 1 -- 172.21.15.116:0/2030598823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b1c04f920 con 0x5650152637d0 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.034+0000 7f1b0effd640 1 -- 172.21.15.116:0/2030598823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c034ab0 con 0x5650152637d0 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2030598823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 msgr2=0x56501527be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009a60 tx=0x7f1b1c005cd0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2030598823 shutdown_connections 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501527be20 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525abf0 0x565015253a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/2030598823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525a2d0 0x56501526f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2030598823 >> 172.21.15.116:0/2030598823 conn(0x565015254a00 msgr2=0x565015253310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2030598823 shutdown_connections 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/2030598823 wait complete. 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 -- start start 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.035+0000 7f1b2e5c6740 1 Processor -- start 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x56501525cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501525d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241adfb0 con 0x5650152637d0 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ad8e0 con 0x56501525abf0 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241aab10 con 0x56501525a2d0 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2e5c6740 10 client.0 _ll_register_callbacks cb 0x7f1b14005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b29e9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501525d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56008/0 (socket says 172.21.15.116:56008) 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 -- 172.21.15.116:0/1267255238 learned_addr learned my addr 172.21.15.116:0/1267255238 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x56501525cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 -- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 msgr2=0x56501525cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x56501525cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 -- 172.21.15.116:0/1267255238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 msgr2=0x56501525d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501525d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b2969b640 1 -- 172.21.15.116:0/1267255238 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56501525d780 con 0x56501525a2d0 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.036+0000 7f1b29e9c640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501525d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2969b640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b200cc8a0 tx=0x7f1b200d19e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b40398ab602ca9ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b200b3070 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b1c04f920 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1b200ccac0 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1b2006ce60 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 10 client.29242 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5650152da080 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b1c04f920 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f1b200cf120 con 0x56501525a2d0 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 10 client.29242 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 10 client.29242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 20 client.29242 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152da7e0 0x5650152dcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 4 client.29242 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650151ca9f0 con 0x5650152da7e0 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.037+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152da7e0 0x5650152dcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.038+0000 7f1b0cff9640 10 client.29242 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.038+0000 7f1b28e9a640 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152da7e0 0x5650152dcc00 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65883de2910e7b5d server_cookie=821078e406a2309 in_seq=0 out_seq=0 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b0cff9640 1 -- 172.21.15.116:0/1267255238 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650151ca9f0 con 0x5650152da7e0 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b0cff9640 10 client.29242 handle_command_reply: tid=0 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 client.29242 shutdown 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 msgr2=0x56501526b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b200cc8a0 tx=0x7f1b200d19e0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 shutdown_connections 2024-03-19T02:41:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650152da7e0 0x5650152dcc00 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650152637d0 0x56501525d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56501525abf0 0x56501525cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b0cff9640 0 client.29242 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 --2- 172.21.15.116:0/1267255238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56501525a2d0 0x56501526b960 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.039+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 >> 172.21.15.116:0/1267255238 conn(0x565015254a00 msgr2=0x5650152610c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 shutdown_connections 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2e5c6740 1 -- 172.21.15.116:0/1267255238 wait complete. 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2e5c6740 20 client.29242 trim_cache size 0 max 16384 2024-03-19T02:41:18.047 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:18.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 msgr2=0x7f1b100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 0x7f1b100a4e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0098d0 tx=0x7f1b1c005dc0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 msgr2=0x7f1b24197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.040+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000e9f0 tx=0x7f1b2000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 shutdown_connections 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1b100a2970 0x7f1b100a4e30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b241077a0 0x7f1b2419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b24102150 0x7f1b24198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 --2- 172.21.15.116:0/694249539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b24101790 0x7f1b24197fe0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 >> 172.21.15.116:0/694249539 conn(0x7f1b240fd2c0 msgr2=0x7f1b240fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 shutdown_connections 2024-03-19T02:41:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:18.041+0000 7f1b2b8ff640 1 -- 172.21.15.116:0/694249539 wait complete. 2024-03-19T02:41:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:18 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2796267933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:18.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4745 10.97 MB/sec warmup 40 sec latency 187.399 ms 2024-03-19T02:41:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:18 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2796267933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:18.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:18 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2796267933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:19.054 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:41:19.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:41:19.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 -- 172.21.15.116:0/1338155866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901017b0 msgr2=0x7fe490101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 --2- 172.21.15.116:0/1338155866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901017b0 0x7fe490101c30 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe484009930 tx=0x7fe48402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 -- 172.21.15.116:0/1338155866 shutdown_connections 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 --2- 172.21.15.116:0/1338155866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 0x7fe49010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 --2- 172.21.15.116:0/1338155866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901017b0 0x7fe490101c30 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 --2- 172.21.15.116:0/1338155866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901077c0 0x7fe490107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 -- 172.21.15.116:0/1338155866 >> 172.21.15.116:0/1338155866 conn(0x7fe490078be0 msgr2=0x7fe490078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 -- 172.21.15.116:0/1338155866 shutdown_connections 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.179+0000 7fe4966dc640 1 -- 172.21.15.116:0/1338155866 wait complete. 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 Processor -- start 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 -- start start 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 0x7fe4901985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe490111a80 con 0x7fe4901077c0 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe490111900 con 0x7fe490102170 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe4966dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe490111c00 con 0x7fe4901017b0 2024-03-19T02:41:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe494c79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56030/0 (socket says 172.21.15.116:56030) 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe494c79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40924/0 (socket says 172.21.15.116:40924) 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe48ffff640 1 -- 172.21.15.116:0/3790078999 learned_addr learned my addr 172.21.15.116:0/3790078999 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.180+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 0x7fe4901985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 -- 172.21.15.116:0/3790078999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 msgr2=0x7fe490198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 -- 172.21.15.116:0/3790078999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 msgr2=0x7fe4901985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 0x7fe4901985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 -- 172.21.15.116:0/3790078999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49019d050 con 0x7fe4901077c0 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe48ffff640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe494c79640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe488009fd0 tx=0x7fe48800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe48d7fa640 1 -- 172.21.15.116:0/3790078999 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48800ce20 con 0x7fe4901077c0 2024-03-19T02:41:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe48d7fa640 1 -- 172.21.15.116:0/3790078999 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe488005020 con 0x7fe4901077c0 2024-03-19T02:41:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe49019d340 con 0x7fe4901077c0 2024-03-19T02:41:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe48d7fa640 1 -- 172.21.15.116:0/3790078999 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488010690 con 0x7fe4901077c0 2024-03-19T02:41:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.181+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4901a4bd0 con 0x7fe4901077c0 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe48d7fa640 1 -- 172.21.15.116:0/3790078999 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe4880051d0 con 0x7fe4901077c0 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe48d7fa640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 0x7fe4780a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe49939e740 1 Processor -- start 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe48d7fa640 1 -- 172.21.15.116:0/3790078999 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7fe4880cf8a0 con 0x7fe4901077c0 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe48ffff640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 0x7fe4780a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.184+0000 7fe49939e740 1 -- start start 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b8c119160 con 0x562b8c1be880 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b8c0c1e10 con 0x562b8c1b5380 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe49939e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b8c115200 con 0x562b8c1b5ca0 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48ffff640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 0x7fe4780a4eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe480011490 tx=0x7fe480005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56034/0 (socket says 172.21.15.116:56034) 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 -- 172.21.15.116:0/1363439300 learned_addr learned my addr 172.21.15.116:0/1363439300 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe494c79640 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48ffff640 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 -- 172.21.15.116:0/1363439300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 msgr2=0x562b8c1b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 -- 172.21.15.116:0/1363439300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 msgr2=0x562b8c1c0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1c0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 -- 172.21.15.116:0/1363439300 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8bef57e0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.185+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe484036040 tx=0x7fe484009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402a7fdecbb91a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe4767fc640 1 -- 172.21.15.116:0/1363439300 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484045020 con 0x562b8c1b5ca0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe4767fc640 1 -- 172.21.15.116:0/1363439300 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe48402c7e0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe4767fc640 1 -- 172.21.15.116:0/1363439300 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4840054a0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe49939e740 1 -- 172.21.15.116:0/1363439300 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 msgr2=0x562b8c1be340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe49939e740 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe484036040 tx=0x7fe484009560 comp rx=0 tx=0).stop 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe49939e740 1 -- 172.21.15.116:0/1363439300 shutdown_connections 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.186+0000 7fe49939e740 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1be340 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 --2- 172.21.15.116:0/1363439300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 -- 172.21.15.116:0/1363439300 >> 172.21.15.116:0/1363439300 conn(0x562b8c1afab0 msgr2=0x562b8c1afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 -- 172.21.15.116:0/1363439300 shutdown_connections 2024-03-19T02:41:19.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 -- 172.21.15.116:0/1363439300 wait complete. 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 -- start start 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.187+0000 7fe49939e740 1 Processor -- start 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 0x562b8c1bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 0x562b8c1bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1be880 0x562b8c1bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b8c1c1610 con 0x562b8c1b5ca0 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b8c0d51a0 con 0x562b8c1be880 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b8c1c1790 con 0x562b8c1b5380 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 0x562b8c1bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 0x562b8c1bb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56046/0 (socket says 172.21.15.116:56046) 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48ffff640 1 -- 172.21.15.116:0/2750139109 learned_addr learned my addr 172.21.15.116:0/2750139109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 0x562b8c1bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe494c79640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1be880 0x562b8c1bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe49939e740 10 client.0 _ll_register_callbacks cb 0x562b8be2cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 -- 172.21.15.116:0/2750139109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 msgr2=0x562b8c1bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 0x562b8c1bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 -- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1be880 msgr2=0x562b8c1bc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1be880 0x562b8c1bc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.188+0000 7fe48f7fe640 1 -- 172.21.15.116:0/2750139109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8bf55f70 con 0x562b8c1b5ca0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 0x562b8c1bbd00 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fe484009900 tx=0x7fe4840054b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a13b401ded0d40b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484045030 con 0x562b8c1b5ca0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7fe484004450 con 0x562b8c1b5ca0 2024-03-19T02:41:19.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe48402c7e0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe48405e050 con 0x562b8c1b5ca0 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe49939e740 10 client.19524 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.189+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562b8bded170 con 0x562b8c1b5ca0 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe48402c7e0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fe4840037b0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 10 client.19524 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:19.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 10 client.19524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 20 client.19524 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c1ce950 0x562b8c1d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 4 client.19524 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b8c1266a0 con 0x562b8c1ce950 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe48ffff640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c1ce950 0x562b8c1d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe48ffff640 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c1ce950 0x562b8c1d0d40 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7051a6953fda632c server_cookie=b21515a2c42d6dfa in_seq=0 out_seq=0 2024-03-19T02:41:19.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.190+0000 7fe46f7fe640 10 client.19524 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2750139109 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b8c1266a0 con 0x562b8c1ce950 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe46f7fe640 10 client.19524 handle_command_reply: tid=0 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 client.19524 shutdown 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 msgr2=0x562b8c1bbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 0x562b8c1bbd00 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fe484009900 tx=0x7fe4840054b0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 shutdown_connections 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c1ce950 0x562b8c1d0d40 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1be880 0x562b8c1bc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe46f7fe640 0 client.19524 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5ca0 0x562b8c1bbd00 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 --2- 172.21.15.116:0/2750139109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5380 0x562b8c1bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.193+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 >> 172.21.15.116:0/2750139109 conn(0x562b8c1afab0 msgr2=0x562b8c1b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.194+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 shutdown_connections 2024-03-19T02:41:19.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.194+0000 7fe49939e740 1 -- 172.21.15.116:0/2750139109 wait complete. 2024-03-19T02:41:19.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.194+0000 7fe49939e740 20 client.19524 trim_cache size 0 max 16384 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 Processor -- start 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 -- start start 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1bdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1d63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901ac9e0 con 0x562b8c1be880 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901ae610 con 0x562b8c1b5380 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe49939e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901ad0b0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.217+0000 7fe494c79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1d63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1bdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56056/0 (socket says 172.21.15.116:56056) 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48f7fe640 1 -- 172.21.15.116:0/1309075893 learned_addr learned my addr 172.21.15.116:0/1309075893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 -- 172.21.15.116:0/1309075893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 msgr2=0x562b8c1ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 -- 172.21.15.116:0/1309075893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 msgr2=0x562b8c1bdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1bdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 -- 172.21.15.116:0/1309075893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8c1b5ad0 con 0x562b8c1be880 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48ffff640 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1bdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe494c79640 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1d63f0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe4880ccc10 tx=0x7fe48809bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98f73a02ba5ab7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe475ffb640 1 -- 172.21.15.116:0/1309075893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48809de60 con 0x562b8c1be880 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe475ffb640 1 -- 172.21.15.116:0/1309075893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4880ba300 con 0x562b8c1be880 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.218+0000 7fe475ffb640 1 -- 172.21.15.116:0/1309075893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4880b4e20 con 0x562b8c1be880 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 -- 172.21.15.116:0/1309075893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 msgr2=0x562b8c1d63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1d63f0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe4880ccc10 tx=0x7fe48809bc50 comp rx=0 tx=0).stop 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 -- 172.21.15.116:0/1309075893 shutdown_connections 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1be880 0x562b8c1d63f0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1b5ca0 0x562b8c1ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 --2- 172.21.15.116:0/1309075893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5380 0x562b8c1bdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 -- 172.21.15.116:0/1309075893 >> 172.21.15.116:0/1309075893 conn(0x562b8c1afab0 msgr2=0x562b8c1bb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 -- 172.21.15.116:0/1309075893 shutdown_connections 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.219+0000 7fe49939e740 1 -- 172.21.15.116:0/1309075893 wait complete. 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 -- start start 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 Processor -- start 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5ca0 0x562b8c1b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1be880 0x562b8c233a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901ade80 con 0x562b8c1b5380 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901ad7b0 con 0x562b8c1b5ca0 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901aa9e0 con 0x562b8c1be880 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe49939e740 10 client.0 _ll_register_callbacks cb 0x7fe488097dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.220+0000 7fe48f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5ca0 0x562b8c1b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40954/0 (socket says 172.21.15.116:40954) 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 -- 172.21.15.116:0/2999616335 learned_addr learned my addr 172.21.15.116:0/2999616335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe494c79640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1be880 0x562b8c233a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 -- 172.21.15.116:0/2999616335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1be880 msgr2=0x562b8c233a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1be880 0x562b8c233a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 -- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5ca0 msgr2=0x562b8c1b8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5ca0 0x562b8c1b8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 -- 172.21.15.116:0/2999616335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8c233f60 con 0x562b8c1b5380 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe48ffff640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x562b8c1d1360 tx=0x7fe480004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e188c22aace79943 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480028620 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7fe4800287c0 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.221+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4880ba300 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe480039050 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 10 client.19530 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562b8c235160 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4880ba300 con 0x562b8c1b5380 2024-03-19T02:41:19.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fe4800317f0 con 0x562b8c1b5380 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 10 client.19530 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 10 client.19530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 20 client.19530 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c235970 0x562b8c237d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 4 client.19530 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562b8c1209f0 con 0x562b8c235970 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.222+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c235970 0x562b8c237d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.223+0000 7fe48f7fe640 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c235970 0x562b8c237d90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe28dd81e12d646b server_cookie=7f5be968d903fe3c in_seq=0 out_seq=0 2024-03-19T02:41:19.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.223+0000 7fe46f7fe640 10 client.19530 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe46f7fe640 1 -- 172.21.15.116:0/2999616335 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562b8c1209f0 con 0x562b8c235970 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe46f7fe640 10 client.19530 handle_command_reply: tid=0 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 client.19530 shutdown 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 msgr2=0x562b8c1b8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x562b8c1d1360 tx=0x7fe480004730 comp rx=0 tx=0).stop 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 shutdown_connections 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562b8c235970 0x562b8c237d90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b8c1be880 0x562b8c233a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b8c1b5ca0 0x562b8c1b8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 --2- 172.21.15.116:0/2999616335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b8c1b5380 0x562b8c1b8750 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 >> 172.21.15.116:0/2999616335 conn(0x562b8c1afab0 msgr2=0x562b8c1aeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe46f7fe640 0 client.19530 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 shutdown_connections 2024-03-19T02:41:19.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.227+0000 7fe49939e740 1 -- 172.21.15.116:0/2999616335 wait complete. 2024-03-19T02:41:19.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.228+0000 7fe49939e740 20 client.19530 trim_cache size 0 max 16384 2024-03-19T02:41:19.236 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:41:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "f9b5cb67-3339-410a-a869-e16cc20e0623", 2024-03-19T02:41:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:41:19.236 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.228+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 msgr2=0x7fe4780a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.228+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 0x7fe4780a4eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe480011490 tx=0x7fe480005f90 comp rx=0 tx=0).stop 2024-03-19T02:41:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.228+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 msgr2=0x7fe49019c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.228+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe488009fd0 tx=0x7fe48800ef10 comp rx=0 tx=0).stop 2024-03-19T02:41:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 shutdown_connections 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe4780a29f0 0x7fe4780a4eb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4901077c0 0x7fe49019c920 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe490102170 0x7fe4901985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 --2- 172.21.15.116:0/3790078999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4901017b0 0x7fe490198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 >> 172.21.15.116:0/3790078999 conn(0x7fe490078be0 msgr2=0x7fe490104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 shutdown_connections 2024-03-19T02:41:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.229+0000 7fe4966dc640 1 -- 172.21.15.116:0/3790078999 wait complete. 2024-03-19T02:41:19.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:41:19.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:41:19.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:19 smithi116 ceph-mon[32263]: pgmap v424: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 103 op/s 2024-03-19T02:41:19.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4849 10.76 MB/sec warmup 41 sec latency 178.513 ms 2024-03-19T02:41:19.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 -- 172.21.15.116:0/2961599791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4068a20 msgr2=0x7f5ab4068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 --2- 172.21.15.116:0/2961599791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4068a20 0x7f5ab4068ea0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009930 tx=0x7f5aa402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 -- 172.21.15.116:0/2961599791 shutdown_connections 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 --2- 172.21.15.116:0/2961599791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab40693e0 0x7f5ab410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 --2- 172.21.15.116:0/2961599791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4068a20 0x7f5ab4068ea0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 --2- 172.21.15.116:0/2961599791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab4107870 0x7f5ab4107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 -- 172.21.15.116:0/2961599791 >> 172.21.15.116:0/2961599791 conn(0x7f5ab4077820 msgr2=0x7f5ab4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 -- 172.21.15.116:0/2961599791 shutdown_connections 2024-03-19T02:41:19.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.364+0000 7f5aba480640 1 -- 172.21.15.116:0/2961599791 wait complete. 2024-03-19T02:41:19.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.365+0000 7f5aba480640 1 Processor -- start 2024-03-19T02:41:19.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.365+0000 7f5aba480640 1 -- start start 2024-03-19T02:41:19.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 0x7f5ab4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab40693e0 0x7f5ab4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4111a70 con 0x7f5ab4107870 2024-03-19T02:41:19.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41118f0 con 0x7f5ab40693e0 2024-03-19T02:41:19.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5aba480640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4111bf0 con 0x7f5ab4068a20 2024-03-19T02:41:19.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab40693e0 0x7f5ab4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 0x7f5ab4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40972/0 (socket says 172.21.15.116:40972) 2024-03-19T02:41:19.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 -- 172.21.15.116:0/1578926291 learned_addr learned my addr 172.21.15.116:0/1578926291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 -- 172.21.15.116:0/1578926291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 msgr2=0x7f5ab4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 0x7f5ab4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 -- 172.21.15.116:0/1578926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab40693e0 msgr2=0x7f5ab4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab40693e0 0x7f5ab4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 -- 172.21.15.116:0/1578926291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab419cf70 con 0x7f5ab4107870 2024-03-19T02:41:19.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 0x7f5ab4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.366+0000 7f5ab8a1d640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eb10 tx=0x7f5aac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.367+0000 7f5ab17fa640 1 -- 172.21.15.116:0/1578926291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac004b20 con 0x7f5ab4107870 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.367+0000 7f5ab17fa640 1 -- 172.21.15.116:0/1578926291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aac004cc0 con 0x7f5ab4107870 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.367+0000 7f5ab17fa640 1 -- 172.21.15.116:0/1578926291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac010690 con 0x7f5ab4107870 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.367+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab419d260 con 0x7f5ab4107870 2024-03-19T02:41:19.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.367+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a4af0 con 0x7f5ab4107870 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.369+0000 7f5ab17fa640 1 -- 172.21.15.116:0/1578926291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5aac005020 con 0x7f5ab4107870 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 Processor -- start 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 -- start start 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac860e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac86c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563aac7c4500 con 0x563aac860a30 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563aac76cbe0 con 0x563aac861350 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5abd05f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563aac7c05a0 con 0x563aac869f30 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab17fa640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 0x7f5a9c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac860e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac860e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40984/0 (socket says 172.21.15.116:40984) 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab37fe640 1 -- 172.21.15.116:0/2404935363 learned_addr learned my addr 172.21.15.116:0/2404935363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40220/0 (socket says 172.21.15.116:40220) 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab8a1d640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 0x7f5a9c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.370+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac86c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab17fa640 1 -- 172.21.15.116:0/1578926291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f5aac0cf740 con 0x7f5ab4107870 2024-03-19T02:41:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 -- 172.21.15.116:0/2404935363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 msgr2=0x563aac86c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac86c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 -- 172.21.15.116:0/2404935363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 msgr2=0x563aac860e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac860e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 -- 172.21.15.116:0/2404935363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aac78e000 con 0x563aac861350 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac86c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab8a1d640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 0x7f5a9c0a4eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aac097e20 tx=0x7f5aac066bd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009fd0 tx=0x7f5aa80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88af1d611ea978c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5a9a7fc640 1 -- 172.21.15.116:0/2404935363 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004030 con 0x563aac861350 2024-03-19T02:41:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5a9a7fc640 1 -- 172.21.15.116:0/2404935363 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa80054d0 con 0x563aac861350 2024-03-19T02:41:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.371+0000 7f5a9a7fc640 1 -- 172.21.15.116:0/2404935363 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004c40 con 0x563aac861350 2024-03-19T02:41:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- 172.21.15.116:0/2404935363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 msgr2=0x563aac8699f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009fd0 tx=0x7f5aa80118d0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- 172.21.15.116:0/2404935363 shutdown_connections 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac86c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac861350 0x563aac8699f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 --2- 172.21.15.116:0/2404935363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac860e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- 172.21.15.116:0/2404935363 >> 172.21.15.116:0/2404935363 conn(0x563aac85b160 msgr2=0x563aac85b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- 172.21.15.116:0/2404935363 shutdown_connections 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- 172.21.15.116:0/2404935363 wait complete. 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 -- start start 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.372+0000 7f5abd05f740 1 Processor -- start 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac866e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac8678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563aac86cd20 con 0x563aac861350 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563aac780160 con 0x563aac860a30 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563aac86cea0 con 0x563aac869f30 2024-03-19T02:41:19.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 10 client.0 _ll_register_callbacks cb 0x563aac7bca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41000/0 (socket says 172.21.15.116:41000) 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 -- 172.21.15.116:0/1374180714 learned_addr learned my addr 172.21.15.116:0/1374180714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac866e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac8678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 -- 172.21.15.116:0/1374180714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 msgr2=0x563aac8678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac8678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 -- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 msgr2=0x563aac866e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac866e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 -- 172.21.15.116:0/1374180714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aac5a27d0 con 0x563aac861350 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac866e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009700 tx=0x7f5aa800fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41f2bc00469b20b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5a937fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8029020 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f5aa8025680 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa80054d0 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5aa8034050 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 10 client.19542 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.373+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563aac4d83f0 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa80054d0 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f5aa8004a60 con 0x563aac861350 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 10 client.19542 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 10 client.19542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 20 client.19542 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac87a170 0x563aac87c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 4 client.19542 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aac7d1a40 con 0x563aac87a170 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac87a170 0x563aac87c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5a937fe640 10 client.19542 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.374+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac87a170 0x563aac87c560 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=307f940dcda4ab18 server_cookie=81d1468f9d9c24af in_seq=0 out_seq=0 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.378+0000 7f5a937fe640 1 -- 172.21.15.116:0/1374180714 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563aac7d1a40 con 0x563aac87a170 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.378+0000 7f5a937fe640 10 client.19542 handle_command_reply: tid=0 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.378+0000 7f5abd05f740 1 client.19542 shutdown 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 msgr2=0x563aac8673b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009700 tx=0x7f5aa800fcd0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 shutdown_connections 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac87a170 0x563aac87c560 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac869f30 0x563aac8678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac861350 0x563aac8673b0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5a937fe640 0 client.19542 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 --2- 172.21.15.116:0/1374180714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac866e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 >> 172.21.15.116:0/1374180714 conn(0x563aac85b160 msgr2=0x563aac85ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 shutdown_connections 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 1 -- 172.21.15.116:0/1374180714 wait complete. 2024-03-19T02:41:19.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.379+0000 7f5abd05f740 20 client.19542 trim_cache size 0 max 16384 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 Processor -- start 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 -- start start 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac8695a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac875ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41aecb0 con 0x563aac869f30 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41b08e0 con 0x563aac860a30 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5abd05f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41af380 con 0x563aac861350 2024-03-19T02:41:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41012/0 (socket says 172.21.15.116:41012) 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3912910575 learned_addr learned my addr 172.21.15.116:0/3912910575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac875ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.397+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac8695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3912910575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 msgr2=0x563aac875ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac875ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3912910575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 msgr2=0x563aac8695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac8695a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3912910575 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aac4980f0 con 0x563aac869f30 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4005ab0 tx=0x7f5aa4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768c375104d83242 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5a99ffb640 1 -- 172.21.15.116:0/3912910575 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa404fba0 con 0x563aac869f30 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5a99ffb640 1 -- 172.21.15.116:0/3912910575 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa404fd40 con 0x563aac869f30 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.398+0000 7f5a99ffb640 1 -- 172.21.15.116:0/3912910575 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003de0 con 0x563aac869f30 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- 172.21.15.116:0/3912910575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 msgr2=0x563aac865fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4005ab0 tx=0x7f5aa4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- 172.21.15.116:0/3912910575 shutdown_connections 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac869f30 0x563aac865fd0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac875ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3912910575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac860a30 0x563aac8695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- 172.21.15.116:0/3912910575 >> 172.21.15.116:0/3912910575 conn(0x563aac85b160 msgr2=0x563aac85c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- 172.21.15.116:0/3912910575 shutdown_connections 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- 172.21.15.116:0/3912910575 wait complete. 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- start start 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 Processor -- start 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac8636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac869f30 0x563aac863c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41b0150 con 0x563aac860a30 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41afa80 con 0x563aac869f30 2024-03-19T02:41:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.399+0000 7f5abd05f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41accb0 con 0x563aac861350 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5abd05f740 10 client.0 _ll_register_callbacks cb 0x7f5aac0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41014/0 (socket says 172.21.15.116:41014) 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3695403690 learned_addr learned my addr 172.21.15.116:0/3695403690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac8636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac869f30 0x563aac863c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3695403690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 msgr2=0x563aac8636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac8636c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac869f30 msgr2=0x563aac863c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac869f30 0x563aac863c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 -- 172.21.15.116:0/3695403690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aac8642c0 con 0x563aac860a30 2024-03-19T02:41:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac8636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5ab37fe640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f5aa402cb60 tx=0x7f5aa40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc925982861f098d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5a937fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa405f070 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f5aa404f650 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa404fd40 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5aa404fba0 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5abd05f740 10 client.19548 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563aac8e0310 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.400+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa404fd40 con 0x563aac860a30 2024-03-19T02:41:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f5aa4058030 con 0x563aac860a30 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 10 client.19548 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 10 client.19548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 20 client.19548 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac8e09c0 0x563aac8e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 4 client.19548 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563aac7cbd90 con 0x563aac8e09c0 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac8e09c0 0x563aac8e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5a937fe640 10 client.19548 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.401+0000 7f5ab3fff640 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac8e09c0 0x563aac8e2de0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2974861e9c88b3e server_cookie=9afbd4052a4c849c in_seq=0 out_seq=0 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5a937fe640 1 -- 172.21.15.116:0/3695403690 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x563aac7cbd90 con 0x563aac8e09c0 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5a937fe640 10 client.19548 handle_command_reply: tid=0 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 client.19548 shutdown 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 msgr2=0x563aac8722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f5aa402cb60 tx=0x7f5aa40039c0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 shutdown_connections 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aac8e09c0 0x563aac8e2de0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aac869f30 0x563aac863c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aac861350 0x563aac8636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5a937fe640 0 client.19548 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 --2- 172.21.15.116:0/3695403690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aac860a30 0x563aac8722e0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 >> 172.21.15.116:0/3695403690 conn(0x563aac85b160 msgr2=0x563aac865e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 shutdown_connections 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 1 -- 172.21.15.116:0/3695403690 wait complete. 2024-03-19T02:41:19.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.407+0000 7f5abd05f740 20 client.19548 trim_cache size 0 max 16384 2024-03-19T02:41:19.421 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:19.421 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (376 inodes in the stack)", 2024-03-19T02:41:19.421 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:41:19.421 INFO:teuthology.orchestra.run.smithi116.stdout: "f9b5cb67-3339-410a-a869-e16cc20e0623": { 2024-03-19T02:41:19.422 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:41:19.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "f9b5cb67-3339-410a-a869-e16cc20e0623", 2024-03-19T02:41:19.422 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:41:19.422 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:41:19.422 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:41:19.423 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:19.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:19 smithi162 ceph-mon[35761]: pgmap v424: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 103 op/s 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 msgr2=0x7f5a9c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 0x7f5a9c0a4eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aac097e20 tx=0x7f5aac066bd0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 msgr2=0x7f5ab419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eb10 tx=0x7f5aac00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 shutdown_connections 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a9c0a29f0 0x7f5a9c0a4eb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab4107870 0x7f5ab419c840 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab40693e0 0x7f5ab4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 --2- 172.21.15.116:0/1578926291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ab4068a20 0x7f5ab4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 >> 172.21.15.116:0/1578926291 conn(0x7f5ab4077820 msgr2=0x7f5ab40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 shutdown_connections 2024-03-19T02:41:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:19.408+0000 7f5aba480640 1 -- 172.21.15.116:0/1578926291 wait complete. 2024-03-19T02:41:19.427 INFO:tasks.cephfs.filesystem:scrub status for tag:f9b5cb67-3339-410a-a869-e16cc20e0623 - {'path': '/', 'tag': 'f9b5cb67-3339-410a-a869-e16cc20e0623', 'options': 'recursive,force'} 2024-03-19T02:41:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:19 smithi154 ceph-mon[35693]: pgmap v424: 129 pgs: 129 active+clean; 728 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 103 op/s 2024-03-19T02:41:20.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 4936 10.57 MB/sec warmup 42 sec latency 336.347 ms 2024-03-19T02:41:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:20 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:41:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:20 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:41:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:20.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:20.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:20.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:20 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:41:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:20.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5040 10.37 MB/sec warmup 43 sec latency 160.992 ms 2024-03-19T02:41:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:21 smithi162 ceph-mon[35761]: pgmap v425: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 149 op/s 2024-03-19T02:41:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:21 smithi154 ceph-mon[35693]: pgmap v425: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 149 op/s 2024-03-19T02:41:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:21.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:21 smithi116 ceph-mon[32263]: pgmap v425: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 149 op/s 2024-03-19T02:41:22.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5172 10.20 MB/sec warmup 44 sec latency 174.676 ms 2024-03-19T02:41:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:23 smithi116 ceph-mon[32263]: pgmap v426: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.1 MiB/s wr, 102 op/s 2024-03-19T02:41:23.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:23 smithi154 ceph-mon[35693]: pgmap v426: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.1 MiB/s wr, 102 op/s 2024-03-19T02:41:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5288 10.04 MB/sec warmup 45 sec latency 134.046 ms 2024-03-19T02:41:23.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:23 smithi162 ceph-mon[35761]: pgmap v426: 129 pgs: 129 active+clean; 740 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.1 MiB/s wr, 102 op/s 2024-03-19T02:41:24.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5408 9.97 MB/sec warmup 46 sec latency 149.414 ms 2024-03-19T02:41:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:25 smithi116 ceph-mon[32263]: pgmap v427: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-03-19T02:41:25.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5527 9.99 MB/sec warmup 47 sec latency 164.050 ms 2024-03-19T02:41:25.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:25 smithi162 ceph-mon[35761]: pgmap v427: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-03-19T02:41:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:25 smithi154 ceph-mon[35693]: pgmap v427: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 210 op/s 2024-03-19T02:41:26.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5644 9.86 MB/sec warmup 48 sec latency 142.659 ms 2024-03-19T02:41:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:27 smithi116 ceph-mon[32263]: pgmap v428: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:27.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5749 9.71 MB/sec warmup 49 sec latency 148.271 ms 2024-03-19T02:41:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:27 smithi162 ceph-mon[35761]: pgmap v428: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:27 smithi154 ceph-mon[35693]: pgmap v428: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:28.319 INFO:tasks.workunit.client.0.smithi116.stdout: 10 5894 10.01 MB/sec warmup 50 sec latency 149.720 ms 2024-03-19T02:41:29.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:29 smithi116 ceph-mon[32263]: pgmap v429: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:29.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6011 9.83 MB/sec warmup 51 sec latency 131.615 ms 2024-03-19T02:41:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:29 smithi162 ceph-mon[35761]: pgmap v429: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:29 smithi154 ceph-mon[35693]: pgmap v429: 129 pgs: 129 active+clean; 730 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-03-19T02:41:30.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6130 9.69 MB/sec warmup 52 sec latency 152.439 ms 2024-03-19T02:41:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:30.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:30.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:30.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:30.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:31.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6270 9.63 MB/sec warmup 53 sec latency 132.227 ms 2024-03-19T02:41:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:41:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:31 smithi162 ceph-mon[35761]: pgmap v430: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:41:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:41:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:31 smithi154 ceph-mon[35693]: pgmap v430: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:41:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:41:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:31 smithi116 ceph-mon[32263]: pgmap v430: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:41:32.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6431 9.48 MB/sec warmup 54 sec latency 128.732 ms 2024-03-19T02:41:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:33 smithi116 ceph-mon[32263]: pgmap v431: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-19T02:41:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6544 9.33 MB/sec warmup 55 sec latency 197.277 ms 2024-03-19T02:41:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:33 smithi162 ceph-mon[35761]: pgmap v431: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-19T02:41:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:33 smithi154 ceph-mon[35693]: pgmap v431: 129 pgs: 129 active+clean; 734 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-19T02:41:34.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6686 9.22 MB/sec warmup 56 sec latency 143.104 ms 2024-03-19T02:41:35.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6824 9.16 MB/sec warmup 57 sec latency 153.550 ms 2024-03-19T02:41:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:35 smithi162 ceph-mon[35761]: pgmap v432: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 276 op/s 2024-03-19T02:41:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:35 smithi154 ceph-mon[35693]: pgmap v432: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 276 op/s 2024-03-19T02:41:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:35 smithi116 ceph-mon[32263]: pgmap v432: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 276 op/s 2024-03-19T02:41:36.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 6972 9.15 MB/sec warmup 58 sec latency 129.479 ms 2024-03-19T02:41:37.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7062 9.09 MB/sec warmup 59 sec latency 131.017 ms 2024-03-19T02:41:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:37 smithi162 ceph-mon[35761]: pgmap v433: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:37 smithi154 ceph-mon[35693]: pgmap v433: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:37 smithi116 ceph-mon[32263]: pgmap v433: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:38.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7147 8.99 MB/sec warmup 60 sec latency 152.262 ms 2024-03-19T02:41:39.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:39 smithi116 ceph-mon[32263]: pgmap v434: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:39.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7252 8.94 MB/sec warmup 61 sec latency 140.900 ms 2024-03-19T02:41:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:39 smithi162 ceph-mon[35761]: pgmap v434: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:39 smithi154 ceph-mon[35693]: pgmap v434: 129 pgs: 129 active+clean; 736 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-03-19T02:41:40.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7366 8.86 MB/sec warmup 62 sec latency 131.402 ms 2024-03-19T02:41:41.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7510 9.02 MB/sec warmup 63 sec latency 149.917 ms 2024-03-19T02:41:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:41 smithi162 ceph-mon[35761]: pgmap v435: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 240 op/s 2024-03-19T02:41:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:41 smithi154 ceph-mon[35693]: pgmap v435: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 240 op/s 2024-03-19T02:41:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:41 smithi116 ceph-mon[32263]: pgmap v435: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 240 op/s 2024-03-19T02:41:42.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7707 9.17 MB/sec warmup 64 sec latency 190.164 ms 2024-03-19T02:41:43.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 7871 9.12 MB/sec warmup 65 sec latency 178.407 ms 2024-03-19T02:41:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:43 smithi162 ceph-mon[35761]: pgmap v436: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-03-19T02:41:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:43 smithi154 ceph-mon[35693]: pgmap v436: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-03-19T02:41:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:43 smithi116 ceph-mon[32263]: pgmap v436: 129 pgs: 129 active+clean; 749 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.6 MiB/s wr, 161 op/s 2024-03-19T02:41:44.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8082 9.18 MB/sec warmup 66 sec latency 133.127 ms 2024-03-19T02:41:45.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8160 9.17 MB/sec warmup 67 sec latency 129.321 ms 2024-03-19T02:41:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:45 smithi162 ceph-mon[35761]: pgmap v437: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.5 MiB/s wr, 227 op/s 2024-03-19T02:41:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:45 smithi154 ceph-mon[35693]: pgmap v437: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.5 MiB/s wr, 227 op/s 2024-03-19T02:41:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:45 smithi116 ceph-mon[32263]: pgmap v437: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.5 MiB/s wr, 227 op/s 2024-03-19T02:41:46.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8248 9.11 MB/sec warmup 68 sec latency 152.104 ms 2024-03-19T02:41:47.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8303 9.03 MB/sec warmup 69 sec latency 122.932 ms 2024-03-19T02:41:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:47 smithi162 ceph-mon[35761]: pgmap v438: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:47 smithi154 ceph-mon[35693]: pgmap v438: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:47 smithi116 ceph-mon[32263]: pgmap v438: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:48.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8352 8.94 MB/sec warmup 70 sec latency 129.811 ms 2024-03-19T02:41:49.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8430 8.87 MB/sec warmup 71 sec latency 119.388 ms 2024-03-19T02:41:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:49 smithi162 ceph-mon[35761]: pgmap v439: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:49.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:41:49.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:41:49.455 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:49 smithi116 ceph-mon[32263]: pgmap v439: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:49 smithi154 ceph-mon[35693]: pgmap v439: 129 pgs: 129 active+clean; 764 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 3.8 MiB/s wr, 137 op/s 2024-03-19T02:41:49.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 -- 172.21.15.116:0/932195323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e8068a20 msgr2=0x7f03e8068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/932195323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e8068a20 0x7f03e8068ea0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009930 tx=0x7f03d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:49.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 -- 172.21.15.116:0/932195323 shutdown_connections 2024-03-19T02:41:49.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/932195323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e80693e0 0x7f03e810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/932195323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e8068a20 0x7f03e8068ea0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/932195323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8107870 0x7f03e8107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 -- 172.21.15.116:0/932195323 >> 172.21.15.116:0/932195323 conn(0x7f03e8077820 msgr2=0x7f03e8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.554+0000 7f03ef5ba640 1 -- 172.21.15.116:0/932195323 shutdown_connections 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 -- 172.21.15.116:0/932195323 wait complete. 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 Processor -- start 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 -- start start 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8068a20 0x7f03e8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e8107870 0x7f03e819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03e8111a70 con 0x7f03e80693e0 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81118f0 con 0x7f03e8068a20 2024-03-19T02:41:49.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ef5ba640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03e8111bf0 con 0x7f03e8107870 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.555+0000 7f03ecb55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45468/0 (socket says 172.21.15.116:45468) 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 -- 172.21.15.116:0/3925739370 learned_addr learned my addr 172.21.15.116:0/3925739370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ed356640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8068a20 0x7f03e8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03edb57640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e8107870 0x7f03e819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 -- 172.21.15.116:0/3925739370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e8107870 msgr2=0x7f03e819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e8107870 0x7f03e819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 -- 172.21.15.116:0/3925739370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8068a20 msgr2=0x7f03e8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8068a20 0x7f03e8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 -- 172.21.15.116:0/3925739370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e819cf90 con 0x7f03e80693e0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005940 tx=0x7f03d8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03de7fc640 1 -- 172.21.15.116:0/3925739370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8043020 con 0x7f03e80693e0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03de7fc640 1 -- 172.21.15.116:0/3925739370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03d8004350 con 0x7f03e80693e0 2024-03-19T02:41:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e819d220 con 0x7f03e80693e0 2024-03-19T02:41:49.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03de7fc640 1 -- 172.21.15.116:0/3925739370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8003a80 con 0x7f03e80693e0 2024-03-19T02:41:49.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.556+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e81a4b10 con 0x7f03e80693e0 2024-03-19T02:41:49.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.558+0000 7f03de7fc640 1 -- 172.21.15.116:0/3925739370 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f03d8046020 con 0x7f03e80693e0 2024-03-19T02:41:49.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.559+0000 7f03f219d740 1 Processor -- start 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.559+0000 7f03f219d740 1 -- start start 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03de7fc640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 0x7f03d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afc3660 0x55742afc5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55742af15500 con 0x55742afb1e50 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55742aebdbe0 con 0x55742afc3660 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03f219d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55742af115a0 con 0x55742afb14a0 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47922/0 (socket says 172.21.15.116:47922) 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 -- 172.21.15.116:0/2324606018 learned_addr learned my addr 172.21.15.116:0/2324606018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 0x7f03d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03ed356640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03ecb55640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afc3660 0x55742afc5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03de7fc640 1 -- 172.21.15.116:0/3925739370 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f03d80f0ab0 con 0x7f03e80693e0 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 -- 172.21.15.116:0/2324606018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afc3660 msgr2=0x55742afc5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afc3660 0x55742afc5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 -- 172.21.15.116:0/2324606018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 msgr2=0x55742afb22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03edb57640 1 -- 172.21.15.116:0/2324606018 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742aedf000 con 0x55742afb14a0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.560+0000 7f03ed356640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.561+0000 7f03edb57640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 0x7f03d40a4e30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03e000fc90 tx=0x7f03e0018000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.561+0000 7f03edb57640 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f03e0011de0 tx=0x7f03e000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f542317fc48c4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.561+0000 7f03d37fe640 1 -- 172.21.15.116:0/2324606018 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0022070 con 0x55742afb14a0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.561+0000 7f03d37fe640 1 -- 172.21.15.116:0/2324606018 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03e0011600 con 0x55742afb14a0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.561+0000 7f03d37fe640 1 -- 172.21.15.116:0/2324606018 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0004290 con 0x55742afb14a0 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- 172.21.15.116:0/2324606018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 msgr2=0x55742afb1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f03e0011de0 tx=0x7f03e000f790 comp rx=0 tx=0).stop 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- 172.21.15.116:0/2324606018 shutdown_connections 2024-03-19T02:41:49.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afc3660 0x55742afc5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- 172.21.15.116:0/2324606018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb14a0 0x55742afb1880 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- 172.21.15.116:0/2324606018 >> 172.21.15.116:0/2324606018 conn(0x55742afac160 msgr2=0x55742afac540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- 172.21.15.116:0/2324606018 shutdown_connections 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- 172.21.15.116:0/2324606018 wait complete. 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- start start 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 Processor -- start 2024-03-19T02:41:49.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afb39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afb4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55742afc64b0 con 0x55742afb1e50 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55742aed1160 con 0x55742afb14a0 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.562+0000 7f03f219d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55742afc6630 con 0x55742afc3660 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03f219d740 10 client.0 _ll_register_callbacks cb 0x55742af0da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ed356640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afb39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45492/0 (socket says 172.21.15.116:45492) 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 -- 172.21.15.116:0/4021389900 learned_addr learned my addr 172.21.15.116:0/4021389900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03edb57640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afb4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 -- 172.21.15.116:0/4021389900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 msgr2=0x55742afb4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afb4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 -- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 msgr2=0x55742afb39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afb39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 -- 172.21.15.116:0/4021389900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742acf37d0 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ed356640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afb39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03ecb55640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f03d80edf30 tx=0x7f03d80bcc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b0f536e8a51f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d809a400 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f03d80ee040 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03e0011600 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.563+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f03d809a220 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 10 client.19560 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55742ac293f0 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03e0011600 con 0x55742afb1e50 2024-03-19T02:41:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f03d809a6a0 con 0x55742afb1e50 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 10 client.19560 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 10 client.19560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 20 client.19560 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742afd3a60 0x55742afd5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 4 client.19560 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55742af22a40 con 0x55742afd3a60 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.564+0000 7f03ed356640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742afd3a60 0x55742afd5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.565+0000 7f03d0ff9640 10 client.19560 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.565+0000 7f03ed356640 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742afd3a60 0x55742afd5ee0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec917e3ad5db6dc server_cookie=79770bf7878ae0e1 in_seq=0 out_seq=0 2024-03-19T02:41:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03d0ff9640 1 -- 172.21.15.116:0/4021389900 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55742af22a40 con 0x55742afd3a60 2024-03-19T02:41:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03d0ff9640 10 client.19560 handle_command_reply: tid=0 2024-03-19T02:41:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 client.19560 shutdown 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 msgr2=0x55742afb3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f03d80edf30 tx=0x7f03d80bcc70 comp rx=0 tx=0).stop 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 shutdown_connections 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742afd3a60 0x55742afd5ee0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03d0ff9640 0 client.19560 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afb4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb1e50 0x55742afb3f20 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 --2- 172.21.15.116:0/4021389900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afb39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 >> 172.21.15.116:0/4021389900 conn(0x55742afac160 msgr2=0x55742afadeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 shutdown_connections 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 1 -- 172.21.15.116:0/4021389900 wait complete. 2024-03-19T02:41:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.567+0000 7f03f219d740 20 client.19560 trim_cache size 0 max 16384 2024-03-19T02:41:49.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.592+0000 7f03f219d740 1 Processor -- start 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 -- start start 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afbf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb1e50 0x55742afcf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81ac920 con 0x55742afc3660 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81ae550 con 0x55742afb14a0 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03f219d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81acff0 con 0x55742afb1e50 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45496/0 (socket says 172.21.15.116:45496) 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 -- 172.21.15.116:0/683031964 learned_addr learned my addr 172.21.15.116:0/683031964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03ecb55640 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb1e50 0x55742afcf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03ed356640 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afbf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 -- 172.21.15.116:0/683031964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb1e50 msgr2=0x55742afcf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb1e50 0x55742afcf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 -- 172.21.15.116:0/683031964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 msgr2=0x55742afbf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afbf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 -- 172.21.15.116:0/683031964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742abe90f0 con 0x55742afc3660 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03edb57640 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f03e000ece0 tx=0x7f03e0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b747de72c1a49a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.593+0000 7f03d2ffd640 1 -- 172.21.15.116:0/683031964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0022070 con 0x55742afc3660 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03d2ffd640 1 -- 172.21.15.116:0/683031964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03e002b040 con 0x55742afc3660 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03d2ffd640 1 -- 172.21.15.116:0/683031964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e00112b0 con 0x55742afc3660 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 -- 172.21.15.116:0/683031964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 msgr2=0x55742afdb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f03e000ece0 tx=0x7f03e0009500 comp rx=0 tx=0).stop 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 -- 172.21.15.116:0/683031964 shutdown_connections 2024-03-19T02:41:49.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afc3660 0x55742afdb7b0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afb1e50 0x55742afcf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 --2- 172.21.15.116:0/683031964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb14a0 0x55742afbf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 -- 172.21.15.116:0/683031964 >> 172.21.15.116:0/683031964 conn(0x55742afac160 msgr2=0x55742afaa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.594+0000 7f03f219d740 1 -- 172.21.15.116:0/683031964 shutdown_connections 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 -- 172.21.15.116:0/683031964 wait complete. 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 -- start start 2024-03-19T02:41:49.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 Processor -- start 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb1e50 0x55742afc68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afc6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81addc0 con 0x55742afb14a0 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81ad6f0 con 0x55742afb1e50 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81aa920 con 0x55742afc3660 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.595+0000 7f03f219d740 10 client.0 _ll_register_callbacks cb 0x7f03d8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45510/0 (socket says 172.21.15.116:45510) 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 -- 172.21.15.116:0/3113206465 learned_addr learned my addr 172.21.15.116:0/3113206465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03edb57640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afc6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb1e50 0x55742afc68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 -- 172.21.15.116:0/3113206465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 msgr2=0x55742afc6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afc6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 -- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb1e50 msgr2=0x55742afc68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb1e50 0x55742afc68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 -- 172.21.15.116:0/3113206465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742afc7330 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03ed356640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x55742afd6720 tx=0x7f03e4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3630babc99a52adf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4024020 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f03e4010db0 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03e002b040 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.596+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f03e402f050 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 10 client.19566 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55742b03ab50 con 0x55742afb14a0 2024-03-19T02:41:49.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03e002b040 con 0x55742afb14a0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f03e401e840 con 0x55742afb14a0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 10 client.19566 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 10 client.19566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 20 client.19566 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742b03b3e0 0x55742b03d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 4 client.19566 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55742af1cd90 con 0x55742b03b3e0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742b03b3e0 0x55742b03d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03d0ff9640 10 client.19566 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.597+0000 7f03ecb55640 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742b03b3e0 0x55742b03d800 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78a7f9d0d5683c5 server_cookie=1d72b41e35540c2d in_seq=0 out_seq=0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.598+0000 7f03d0ff9640 1 -- 172.21.15.116:0/3113206465 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55742af1cd90 con 0x55742b03b3e0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.598+0000 7f03d0ff9640 10 client.19566 handle_command_reply: tid=0 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.598+0000 7f03f219d740 1 client.19566 shutdown 2024-03-19T02:41:49.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.598+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 msgr2=0x55742afcb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.598+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x55742afd6720 tx=0x7f03e4012ba0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 shutdown_connections 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55742b03b3e0 0x55742b03d800 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55742afc3660 0x55742afc6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55742afb1e50 0x55742afc68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 --2- 172.21.15.116:0/3113206465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55742afb14a0 0x55742afcb900 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 >> 172.21.15.116:0/3113206465 conn(0x55742afac160 msgr2=0x55742afab1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03d0ff9640 0 client.19566 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 shutdown_connections 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 1 -- 172.21.15.116:0/3113206465 wait complete. 2024-03-19T02:41:49.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03f219d740 20 client.19566 trim_cache size 0 max 16384 2024-03-19T02:41:49.606 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:49.606 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:41:49.606 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:41:49.606 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 msgr2=0x7f03d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 0x7f03d40a4e30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03e000fc90 tx=0x7f03e0018000 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 msgr2=0x7f03e8198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f03d8005940 tx=0x7f03d8005dc0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 shutdown_connections 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03d40a2970 0x7f03d40a4e30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03e8107870 0x7f03e819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03e80693e0 0x7f03e8198530 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 --2- 172.21.15.116:0/3925739370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e8068a20 0x7f03e8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.599+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 >> 172.21.15.116:0/3925739370 conn(0x7f03e8077820 msgr2=0x7f03e80fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.600+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 shutdown_connections 2024-03-19T02:41:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.600+0000 7f03ef5ba640 1 -- 172.21.15.116:0/3925739370 wait complete. 2024-03-19T02:41:49.614 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:41:49.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:41:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 -- 172.21.15.116:0/46829038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81066c0 msgr2=0x7fded8106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 --2- 172.21.15.116:0/46829038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81066c0 0x7fded8106aa0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fded4009930 tx=0x7fded402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 -- 172.21.15.116:0/46829038 shutdown_connections 2024-03-19T02:41:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 --2- 172.21.15.116:0/46829038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 --2- 172.21.15.116:0/46829038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fded81006b0 0x7fded8100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 --2- 172.21.15.116:0/46829038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81066c0 0x7fded8106aa0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 -- 172.21.15.116:0/46829038 >> 172.21.15.116:0/46829038 conn(0x7fded80777f0 msgr2=0x7fded8077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 -- 172.21.15.116:0/46829038 shutdown_connections 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 -- 172.21.15.116:0/46829038 wait complete. 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.784+0000 7fdee0644640 1 Processor -- start 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 -- start start 2024-03-19T02:41:49.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fded81066c0 0x7fded819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fded8111a80 con 0x7fded81006b0 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fded8111900 con 0x7fded81066c0 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdee0644640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fded8111c00 con 0x7fded8101070 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45522/0 (socket says 172.21.15.116:45522) 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 -- 172.21.15.116:0/1696555544 learned_addr learned my addr 172.21.15.116:0/1696555544 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdeddbdf640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 -- 172.21.15.116:0/1696555544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 msgr2=0x7fded8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 -- 172.21.15.116:0/1696555544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fded81066c0 msgr2=0x7fded819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fded81066c0 0x7fded819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdede3e0640 1 -- 172.21.15.116:0/1696555544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded819cfe0 con 0x7fded81006b0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.785+0000 7fdeddbdf640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.786+0000 7fdede3e0640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fded4009230 tx=0x7fded402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.786+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4043020 con 0x7fded81006b0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.786+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded819d270 con 0x7fded81006b0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.786+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fded402ca70 con 0x7fded81006b0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.787+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded81a4b60 con 0x7fded81006b0 2024-03-19T02:41:49.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.787+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4003a50 con 0x7fded81006b0 2024-03-19T02:41:49.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.788+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded8101df0 con 0x7fded81006b0 2024-03-19T02:41:49.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.789+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fded4046020 con 0x7fded81006b0 2024-03-19T02:41:49.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.789+0000 7fdecf7fe640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 0x7fdec40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:49.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.790+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7fded40f11c0 con 0x7fded81006b0 2024-03-19T02:41:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.790+0000 7fdeddbdf640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 0x7fdec40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.790+0000 7fdeddbdf640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 0x7fdec40a4e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fded8104df0 tx=0x7fdec8005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:49.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:49.796+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fded404d020 con 0x7fded81006b0 2024-03-19T02:41:50.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.119+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fded8068a60 con 0x7fded81006b0 2024-03-19T02:41:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.120+0000 7fdecf7fe640 1 -- 172.21.15.116:0/1696555544 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11958 (secure 0 0 0) 0x7fded40bdc50 con 0x7fded81006b0 2024-03-19T02:41:50.121 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:41:50.121 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:40:44.485117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:41:50.121 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 msgr2=0x7fdec40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 0x7fdec40a4e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fded8104df0 tx=0x7fdec8005f90 comp rx=0 tx=0).stop 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 msgr2=0x7fded8197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fded4009230 tx=0x7fded402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 shutdown_connections 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdec40a2970 0x7fdec40a4e30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fded81066c0 0x7fded819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fded8101070 0x7fded8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 --2- 172.21.15.116:0/1696555544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fded81006b0 0x7fded8197ff0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 >> 172.21.15.116:0/1696555544 conn(0x7fded80777f0 msgr2=0x7fded8102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 shutdown_connections 2024-03-19T02:41:50.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.123+0000 7fdee0644640 1 -- 172.21.15.116:0/1696555544 wait complete. 2024-03-19T02:41:50.138 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 -- 172.21.15.116:0/693777421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 msgr2=0x7f1c0c100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 --2- 172.21.15.116:0/693777421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c100b30 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009930 tx=0x7f1bfc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 -- 172.21.15.116:0/693777421 shutdown_connections 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 --2- 172.21.15.116:0/693777421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c101070 0x7f1c0c10d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 --2- 172.21.15.116:0/693777421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c100b30 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 --2- 172.21.15.116:0/693777421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 -- 172.21.15.116:0/693777421 >> 172.21.15.116:0/693777421 conn(0x7f1c0c077810 msgr2=0x7f1c0c077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 -- 172.21.15.116:0/693777421 shutdown_connections 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.269+0000 7f1c134dc640 1 -- 172.21.15.116:0/693777421 wait complete. 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 Processor -- start 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 -- start start 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c101070 0x7f1c0c197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c110980 con 0x7f1c0c1006b0 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c110800 con 0x7f1c0c1066c0 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c134dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c110b00 con 0x7f1c0c101070 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c11278640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c11278640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45544/0 (socket says 172.21.15.116:45544) 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c11278640 1 -- 172.21.15.116:0/365697126 learned_addr learned my addr 172.21.15.116:0/365697126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c10a77640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c101070 0x7f1c0c197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.270+0000 7f1c11a79640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 -- 172.21.15.116:0/365697126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c101070 msgr2=0x7f1c0c197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c101070 0x7f1c0c197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 -- 172.21.15.116:0/365697126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 msgr2=0x7f1c0c19b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c19b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 -- 172.21.15.116:0/365697126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c19beb0 con 0x7f1c0c1006b0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11a79640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c11278640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c027fc640 1 -- 172.21.15.116:0/365697126 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800cd80 con 0x7f1c0c1006b0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c19c1a0 con 0x7f1c0c1006b0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c027fc640 1 -- 172.21.15.116:0/365697126 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c080054d0 con 0x7f1c0c1006b0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c1a3a30 con 0x7f1c0c1006b0 2024-03-19T02:41:50.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.271+0000 7f1c027fc640 1 -- 172.21.15.116:0/365697126 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08010490 con 0x7f1c0c1006b0 2024-03-19T02:41:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.274+0000 7f1c027fc640 1 -- 172.21.15.116:0/365697126 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1c08005020 con 0x7f1c0c1006b0 2024-03-19T02:41:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c027fc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 0x7f1bf8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 Processor -- start 2024-03-19T02:41:50.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c10a77640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 0x7f1bf8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 -- start start 2024-03-19T02:41:50.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c027fc640 1 -- 172.21.15.116:0/365697126 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1c080cf530 con 0x7f1c0c1006b0 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fcecc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fcecefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b9fce224b0 con 0x55b9fceccbe0 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b9fcdcad70 con 0x55b9fcec36e0 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c1619f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9fce1e550 con 0x55b9fcec4000 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c11278640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.275+0000 7f1c11278640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52468/0 (socket says 172.21.15.116:52468) 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 -- 172.21.15.116:0/3636836163 learned_addr learned my addr 172.21.15.116:0/3636836163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c10a77640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 0x7f1bf8003df0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc005ab0 tx=0x7f1bfc005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11a79640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fcecefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c10a77640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fcecc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 -- 172.21.15.116:0/3636836163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 msgr2=0x55b9fcecc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fcecc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 -- 172.21.15.116:0/3636836163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 msgr2=0x55b9fcecefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fcecefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 -- 172.21.15.116:0/3636836163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9fcded210 con 0x55b9fcec36e0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c10a77640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fcecc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11a79640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fcecefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.276+0000 7f1c11278640 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1c080106e0 tx=0x7f1c0809b400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a7db165ba348ed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.277+0000 7f1bf77fe640 1 -- 172.21.15.116:0/3636836163 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08005680 con 0x55b9fcec36e0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.277+0000 7f1bf77fe640 1 -- 172.21.15.116:0/3636836163 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c0809beb0 con 0x55b9fcec36e0 2024-03-19T02:41:50.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.277+0000 7f1bf77fe640 1 -- 172.21.15.116:0/3636836163 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080ccb20 con 0x55b9fcec36e0 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- 172.21.15.116:0/3636836163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 msgr2=0x55b9fcec3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1c080106e0 tx=0x7f1c0809b400 comp rx=0 tx=0).stop 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- 172.21.15.116:0/3636836163 shutdown_connections 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fcecefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fcecc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3636836163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec3ac0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- 172.21.15.116:0/3636836163 >> 172.21.15.116:0/3636836163 conn(0x55b9fcebde10 msgr2=0x55b9fcebe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- 172.21.15.116:0/3636836163 shutdown_connections 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- 172.21.15.116:0/3636836163 wait complete. 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.278+0000 7f1c1619f740 1 -- start start 2024-03-19T02:41:50.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.279+0000 7f1c1619f740 1 Processor -- start 2024-03-19T02:41:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.279+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.279+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.279+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c1619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b9fcecfa20 con 0x55b9fceccbe0 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c1619f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b9fcdde310 con 0x55b9fcec36e0 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c1619f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9fcecfba0 con 0x55b9fcec4000 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c11a79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c1619f740 10 client.0 _ll_register_callbacks cb 0x55b9fcdd22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47996/0 (socket says 172.21.15.116:47996) 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 -- 172.21.15.116:0/1485504842 learned_addr learned my addr 172.21.15.116:0/1485504842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c11278640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 -- 172.21.15.116:0/1485504842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 msgr2=0x55b9fcec9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 -- 172.21.15.116:0/1485504842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 msgr2=0x55b9fceca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c10a77640 1 -- 172.21.15.116:0/1485504842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9fcb363f0 con 0x55b9fcec4000 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.280+0000 7f1c11278640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1c10a77640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009930 tx=0x7f1bfc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8820c0eb1c325c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1bf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc04f030 con 0x55b9fcec4000 2024-03-19T02:41:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1bfc03a040 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c0809beb0 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1bfc05c050 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1c1619f740 10 client.29311 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.281+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b9fcc007d0 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bfc045040 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f1bfc04a040 con 0x55b9fcec4000 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 10 client.29311 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 10 client.29311 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 20 client.29311 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcedf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 4 client.29311 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.282+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1c0809beb0 con 0x55b9fcedcf20 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.283+0000 7f1c11278640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcedf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.285+0000 7f1c11278640 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcedf380 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=41535009840b48da server_cookie=941db3dd03a2785b in_seq=0 out_seq=0 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.285+0000 7f1bf4ff9640 10 client.29311 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/1485504842 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f1c0809beb0 con 0x55b9fcedcf20 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1bf4ff9640 10 client.29311 handle_command_reply: tid=0 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 client.29311 shutdown 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 msgr2=0x55b9fceca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009930 tx=0x7f1bfc009260 comp rx=0 tx=0).stop 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 shutdown_connections 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcedf380 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1bf4ff9640 0 client.29311 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fceca150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 --2- 172.21.15.116:0/1485504842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcec9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.286+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 >> 172.21.15.116:0/1485504842 conn(0x55b9fcebde10 msgr2=0x55b9fceb8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.287+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 shutdown_connections 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.287+0000 7f1c1619f740 1 -- 172.21.15.116:0/1485504842 wait complete. 2024-03-19T02:41:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.287+0000 7f1c1619f740 20 client.29311 trim_cache size 0 max 16384 2024-03-19T02:41:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.310+0000 7f1c1619f740 1 Processor -- start 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.310+0000 7f1c1619f740 1 -- start start 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcecc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fced87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1aca70 con 0x55b9fceccbe0 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1ae6a0 con 0x55b9fcec36e0 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c1619f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1ad140 con 0x55b9fcec4000 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c11a79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fced87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c11a79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45584/0 (socket says 172.21.15.116:45584) 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c11a79640 1 -- 172.21.15.116:0/393456420 learned_addr learned my addr 172.21.15.116:0/393456420 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c11278640 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcecc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 -- 172.21.15.116:0/393456420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 msgr2=0x55b9fcecc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcecc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 -- 172.21.15.116:0/393456420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 msgr2=0x55b9fceb8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 -- 172.21.15.116:0/393456420 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9fcebdc40 con 0x55b9fcec4000 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c11a79640 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.311+0000 7f1c10a77640 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fced87c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc003690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b74747b87450ee05 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.312+0000 7f1bf6ffd640 1 -- 172.21.15.116:0/393456420 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc04f030 con 0x55b9fcec4000 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.312+0000 7f1bf6ffd640 1 -- 172.21.15.116:0/393456420 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bfc045040 con 0x55b9fcec4000 2024-03-19T02:41:50.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.312+0000 7f1bf6ffd640 1 -- 172.21.15.116:0/393456420 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0046c0 con 0x55b9fcec4000 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- 172.21.15.116:0/393456420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 msgr2=0x55b9fced87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fced87c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc003690 comp rx=0 tx=0).stop 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- 172.21.15.116:0/393456420 shutdown_connections 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fceccbe0 0x55b9fceb8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec4000 0x55b9fced87c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 --2- 172.21.15.116:0/393456420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fcec36e0 0x55b9fcecc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- 172.21.15.116:0/393456420 >> 172.21.15.116:0/393456420 conn(0x55b9fcebde10 msgr2=0x55b9fceb8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- 172.21.15.116:0/393456420 shutdown_connections 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- 172.21.15.116:0/393456420 wait complete. 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 -- start start 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.313+0000 7f1c1619f740 1 Processor -- start 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fcec4000 0x55b9fced4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 0x55b9fced0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1adf10 con 0x55b9fcec4000 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1ad840 con 0x55b9fceccbe0 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c1aaa70 con 0x55b9fcec36e0 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c1619f740 10 client.0 _ll_register_callbacks cb 0x7f1c08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48014/0 (socket says 172.21.15.116:48014) 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c10a77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fcec4000 0x55b9fced4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 -- 172.21.15.116:0/3060872450 learned_addr learned my addr 172.21.15.116:0/3060872450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11a79640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 0x55b9fced0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 -- 172.21.15.116:0/3060872450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 msgr2=0x55b9fced0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 0x55b9fced0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 -- 172.21.15.116:0/3060872450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fcec4000 msgr2=0x55b9fced4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fcec4000 0x55b9fced4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 -- 172.21.15.116:0/3060872450 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9fced1500 con 0x55b9fcec36e0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11a79640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 0x55b9fced0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1c11278640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c0809b310 tx=0x7f1c0800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e88c7a771b6268b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.314+0000 7f1bf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08005680 con 0x55b9fcec36e0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f1c080ccb20 con 0x55b9fcec36e0 2024-03-19T02:41:50.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bfc045040 con 0x55b9fcec36e0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1c0808e050 con 0x55b9fcec36e0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 10 client.29317 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b9fcf43c70 con 0x55b9fcec36e0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bfc045040 con 0x55b9fcec36e0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f1c08097e20 con 0x55b9fcec36e0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 10 client.29317 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 10 client.29317 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 20 client.29317 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcf45f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 4 client.29317 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9fce803e0 con 0x55b9fcedcf20 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.315+0000 7f1c10a77640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcf45f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1bf4ff9640 10 client.29317 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:50.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c10a77640 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcf45f10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1549258f155d5fd2 server_cookie=b76ea0aaa2347a21 in_seq=0 out_seq=0 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1bf4ff9640 1 -- 172.21.15.116:0/3060872450 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9fce803e0 con 0x55b9fcedcf20 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1bf4ff9640 10 client.29317 handle_command_reply: tid=0 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 client.29317 shutdown 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 msgr2=0x55b9fced4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1c0809b310 tx=0x7f1c0800cf60 comp rx=0 tx=0).stop 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 shutdown_connections 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b9fcedcf20 0x55b9fcf45f10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9fceccbe0 0x55b9fced0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1bf4ff9640 0 client.29317 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9fcec4000 0x55b9fced4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 --2- 172.21.15.116:0/3060872450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9fcec36e0 0x55b9fced4970 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.316+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 >> 172.21.15.116:0/3060872450 conn(0x55b9fcebde10 msgr2=0x55b9fceb8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 shutdown_connections 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c1619f740 1 -- 172.21.15.116:0/3060872450 wait complete. 2024-03-19T02:41:50.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c1619f740 20 client.29317 trim_cache size 0 max 16384 2024-03-19T02:41:50.323 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 msgr2=0x7f1bf8003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 0x7f1bf8003df0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc005ab0 tx=0x7f1bfc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 msgr2=0x7f1c0c196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.317+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800edf0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 shutdown_connections 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1bf8003940 0x7f1bf8003df0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c0c1066c0 0x7f1c0c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c0c101070 0x7f1c0c197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 --2- 172.21.15.116:0/365697126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0c1006b0 0x7f1c0c196ec0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 >> 172.21.15.116:0/365697126 conn(0x7f1c0c077810 msgr2=0x7f1c0c1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 shutdown_connections 2024-03-19T02:41:50.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.318+0000 7f1c134dc640 1 -- 172.21.15.116:0/365697126 wait complete. 2024-03-19T02:41:50.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8598 9.05 MB/sec warmup 72 sec latency 161.317 ms 2024-03-19T02:41:50.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:41:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.461+0000 7f37ed53b640 1 -- 172.21.15.116:0/3516054752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 msgr2=0x7f37e8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.461+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3516054752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e8106a90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009930 tx=0x7f37d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 -- 172.21.15.116:0/3516054752 shutdown_connections 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3516054752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 0x7f37e810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3516054752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3516054752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e8106a90 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 -- 172.21.15.116:0/3516054752 >> 172.21.15.116:0/3516054752 conn(0x7f37e80778d0 msgr2=0x7f37e8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 -- 172.21.15.116:0/3516054752 shutdown_connections 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.462+0000 7f37ed53b640 1 -- 172.21.15.116:0/3516054752 wait complete. 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 Processor -- start 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 -- start start 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 0x7f37e8197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8110930 con 0x7f37e81066b0 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81107b0 con 0x7f37e8101060 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37ed53b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8110ab0 con 0x7f37e81006a0 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45614/0 (socket says 172.21.15.116:45614) 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.463+0000 7f37e77fe640 1 -- 172.21.15.116:0/3019915676 learned_addr learned my addr 172.21.15.116:0/3019915676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 0x7f37e8197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 -- 172.21.15.116:0/3019915676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 msgr2=0x7f37e8196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 -- 172.21.15.116:0/3019915676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 msgr2=0x7f37e8197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 0x7f37e8197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 -- 172.21.15.116:0/3019915676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e819bec0 con 0x7f37e81066b0 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f37e000efd0 tx=0x7f37e000d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37cffff640 1 -- 172.21.15.116:0/3019915676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0003970 con 0x7f37e81066b0 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37cffff640 1 -- 172.21.15.116:0/3019915676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37e0004020 con 0x7f37e81066b0 2024-03-19T02:41:50.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e819c1b0 con 0x7f37e81066b0 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37cffff640 1 -- 172.21.15.116:0/3019915676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e00166b0 con 0x7f37e81066b0 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.464+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e8068980 con 0x7f37e81066b0 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 Processor -- start 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37cffff640 1 -- 172.21.15.116:0/3019915676 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f37e001d020 con 0x7f37e81066b0 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 -- start start 2024-03-19T02:41:50.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fb2d0 0x5593f41fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f42047d0 0x5593f4206bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593f415e4b0 con 0x5593f42047d0 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593f4106d70 con 0x5593f41fbbf0 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37f0201740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593f415a550 con 0x5593f41fb2d0 2024-03-19T02:41:50.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52518/0 (socket says 172.21.15.116:52518) 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3734889437 learned_addr learned my addr 172.21.15.116:0/3734889437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f42047d0 0x5593f4206bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.467+0000 7f37cffff640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 0x7f37d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fb2d0 0x5593f41fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3734889437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fb2d0 msgr2=0x5593f41fb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fb2d0 0x5593f41fb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3734889437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f42047d0 msgr2=0x5593f4206bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f42047d0 0x5593f4206bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3734889437 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f4129210 con 0x5593f41fbbf0 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 0x7f37d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37cffff640 1 -- 172.21.15.116:0/3019915676 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f37e00ceef0 con 0x7f37e81066b0 2024-03-19T02:41:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37d8036040 tx=0x7f37d80095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=579c818169841466 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3734889437 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8004240 con 0x5593f41fbbf0 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 0x7f37d00a4e30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f37e000dbb0 tx=0x7f37e00bb040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3734889437 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d802cc90 con 0x5593f41fbbf0 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.468+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3734889437 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8003d60 con 0x5593f41fbbf0 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 -- 172.21.15.116:0/3734889437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 msgr2=0x5593f4204290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37d8036040 tx=0x7f37d80095f0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 -- 172.21.15.116:0/3734889437 shutdown_connections 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f42047d0 0x5593f4206bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4204290 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 --2- 172.21.15.116:0/3734889437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fb2d0 0x5593f41fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 -- 172.21.15.116:0/3734889437 >> 172.21.15.116:0/3734889437 conn(0x5593f41f5a00 msgr2=0x5593f41f5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 -- 172.21.15.116:0/3734889437 shutdown_connections 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.469+0000 7f37f0201740 1 -- 172.21.15.116:0/3734889437 wait complete. 2024-03-19T02:41:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 -- start start 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 Processor -- start 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42018e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4202360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593f420fd70 con 0x5593f41fb2d0 2024-03-19T02:41:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593f411a310 con 0x5593f41fbbf0 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593f420fef0 con 0x5593f42047d0 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.470+0000 7f37f0201740 10 client.0 _ll_register_callbacks cb 0x5593f410e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52534/0 (socket says 172.21.15.116:52534) 2024-03-19T02:41:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3975331612 learned_addr learned my addr 172.21.15.116:0/3975331612 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42018e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45634/0 (socket says 172.21.15.116:45634) 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 msgr2=0x5593f4202360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4202360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3975331612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 msgr2=0x5593f42018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3975331612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f3e723f0 con 0x5593f41fbbf0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006010 tx=0x7f37d802cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95858e5610738bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37caff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d804e050 con 0x5593f41fbbf0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f37d8004030 con 0x5593f41fbbf0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.471+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37d802cc90 con 0x5593f41fbbf0 2024-03-19T02:41:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f37d805d050 con 0x5593f41fbbf0 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 10 client.29372 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5593f3f3c7d0 con 0x5593f41fbbf0 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37d802cc90 con 0x5593f41fbbf0 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f37d8048ca0 con 0x5593f41fbbf0 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 10 client.29372 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 10 client.29372 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 20 client.29372 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f421d260 0x5593f421f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 4 client.29372 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593f41bc3e0 con 0x5593f421d260 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.472+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f421d260 0x5593f421f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.473+0000 7f37caff5640 10 client.29372 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.473+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f421d260 0x5593f421f650 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1f1ab9c6aacaea15 server_cookie=34f9f1c136f6a013 in_seq=0 out_seq=0 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37caff5640 1 -- 172.21.15.116:0/3975331612 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593f41bc3e0 con 0x5593f421d260 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37caff5640 10 client.29372 handle_command_reply: tid=0 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 client.29372 shutdown 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 msgr2=0x5593f4201e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006010 tx=0x7f37d802cbd0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 shutdown_connections 2024-03-19T02:41:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4202360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fbbf0 0x5593f4201e20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f421d260 0x5593f421f650 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 --2- 172.21.15.116:0/3975331612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 >> 172.21.15.116:0/3975331612 conn(0x5593f41f5a00 msgr2=0x5593f41f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37caff5640 0 client.29372 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 shutdown_connections 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.474+0000 7f37f0201740 1 -- 172.21.15.116:0/3975331612 wait complete. 2024-03-19T02:41:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.475+0000 7f37f0201740 20 client.29372 trim_cache size 0 max 16384 2024-03-19T02:41:50.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:50 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1696555544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.491+0000 7f37f0201740 1 Processor -- start 2024-03-19T02:41:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 -- start start 2024-03-19T02:41:50.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 0x5593f4218b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f42047d0 0x5593f4224bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81ac930 con 0x5593f41fb2d0 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81ae560 con 0x5593f42047d0 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37f0201740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81ad000 con 0x5593f41fbbf0 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45644/0 (socket says 172.21.15.116:45644) 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 -- 172.21.15.116:0/4211875996 learned_addr learned my addr 172.21.15.116:0/4211875996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 0x5593f4218b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f42047d0 0x5593f4224bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 -- 172.21.15.116:0/4211875996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 msgr2=0x5593f4218b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 0x5593f4218b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 -- 172.21.15.116:0/4211875996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f42047d0 msgr2=0x5593f4224bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f42047d0 0x5593f4224bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 -- 172.21.15.116:0/4211875996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f41faca0 con 0x5593f41fb2d0 2024-03-19T02:41:50.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 0x5593f4218b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.492+0000 7f37e67fc640 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0095b0 tx=0x7f37dc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29189ea37a4dfed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37ccff9640 1 -- 172.21.15.116:0/4211875996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc028070 con 0x5593f41fb2d0 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37ccff9640 1 -- 172.21.15.116:0/4211875996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37dc021810 con 0x5593f41fb2d0 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37ccff9640 1 -- 172.21.15.116:0/4211875996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004a00 con 0x5593f41fb2d0 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 -- 172.21.15.116:0/4211875996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 msgr2=0x5593f42011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0095b0 tx=0x7f37dc005f90 comp rx=0 tx=0).stop 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 -- 172.21.15.116:0/4211875996 shutdown_connections 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f42047d0 0x5593f4224bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f41fbbf0 0x5593f4218b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 --2- 172.21.15.116:0/4211875996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fb2d0 0x5593f42011e0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.493+0000 7f37f0201740 1 -- 172.21.15.116:0/4211875996 >> 172.21.15.116:0/4211875996 conn(0x5593f41f5a00 msgr2=0x5593f41f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- 172.21.15.116:0/4211875996 shutdown_connections 2024-03-19T02:41:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- 172.21.15.116:0/4211875996 wait complete. 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- start start 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 Processor -- start 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 0x5593f4215110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 0x5593f42104d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4210a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81addd0 con 0x5593f41fbbf0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81ad700 con 0x5593f41fb2d0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81aa930 con 0x5593f42047d0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37f0201740 10 client.0 _ll_register_callbacks cb 0x7f37d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 0x5593f4215110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 0x5593f42104d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 0x5593f4215110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52544/0 (socket says 172.21.15.116:52544) 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37e67fc640 1 -- 172.21.15.116:0/2057856070 learned_addr learned my addr 172.21.15.116:0/2057856070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.494+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4210a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 msgr2=0x5593f4210a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4210a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2057856070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 msgr2=0x5593f4215110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 0x5593f4215110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2057856070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f4210f50 con 0x5593f41fbbf0 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 0x5593f42104d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034510 tx=0x7f37d8034930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5a2ecfe409baa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8003a40 con 0x5593f41fbbf0 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f37d8034ce0 con 0x5593f41fbbf0 2024-03-19T02:41:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc021810 con 0x5593f41fbbf0 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f37d802c5d0 con 0x5593f41fbbf0 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37f0201740 10 client.19608 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5593f4282f50 con 0x5593f41fbbf0 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37dc021810 con 0x5593f41fbbf0 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.495+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f37d8048740 con 0x5593f41fbbf0 2024-03-19T02:41:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 10 client.19608 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 10 client.19608 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 20 client.19608 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f4283650 0x5593f4285a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 4 client.19608 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593f416b9f0 con 0x5593f4283650 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f4283650 0x5593f4285a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37caff5640 10 client.19608 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f4283650 0x5593f4285a70 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=56bf4fc4f9203d86 server_cookie=5d96117ed89796 in_seq=0 out_seq=0 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37caff5640 1 -- 172.21.15.116:0/2057856070 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593f416b9f0 con 0x5593f4283650 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37caff5640 10 client.19608 handle_command_reply: tid=0 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 client.19608 shutdown 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 msgr2=0x5593f42104d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 0x5593f42104d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034510 tx=0x7f37d8034930 comp rx=0 tx=0).stop 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 shutdown_connections 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593f42047d0 0x5593f4210a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5593f4283650 0x5593f4285a70 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593f41fbbf0 0x5593f42104d0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.496+0000 7f37f0201740 1 --2- 172.21.15.116:0/2057856070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593f41fb2d0 0x5593f4215110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 >> 172.21.15.116:0/2057856070 conn(0x5593f41f5a00 msgr2=0x5593f42021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37caff5640 0 client.19608 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:41:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 shutdown_connections 2024-03-19T02:41:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37f0201740 1 -- 172.21.15.116:0/2057856070 wait complete. 2024-03-19T02:41:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37f0201740 20 client.19608 trim_cache size 0 max 16384 2024-03-19T02:41:50.521 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 msgr2=0x7f37d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 0x7f37d00a4e30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f37e000dbb0 tx=0x7f37e00bb040 comp rx=0 tx=0).stop 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 msgr2=0x7f37e819b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f37e000efd0 tx=0x7f37e000d6b0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 shutdown_connections 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f37d00a2970 0x7f37d00a4e30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e81066b0 0x7f37e819b790 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e8101060 0x7f37e8197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 --2- 172.21.15.116:0/3019915676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e81006a0 0x7f37e8196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.497+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 >> 172.21.15.116:0/3019915676 conn(0x7f37e80778d0 msgr2=0x7f37e81030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.498+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 shutdown_connections 2024-03-19T02:41:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.498+0000 7f37ed53b640 1 -- 172.21.15.116:0/3019915676 wait complete. 2024-03-19T02:41:50.524 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:50 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1696555544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:50.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2718545851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d010bd80 msgr2=0x7f80d010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2718545851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d010bd80 0x7f80d010e220 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009be0 tx=0x7f80c802bf80 comp rx=0 tx=0).stop 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2718545851 shutdown_connections 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2718545851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d010bd80 0x7f80d010e220 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2718545851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d0103360 0x7f80d010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2718545851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0102a40 0x7f80d0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2718545851 >> 172.21.15.116:0/2718545851 conn(0x7f80d00778d0 msgr2=0x7f80d0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2718545851 shutdown_connections 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.662+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2718545851 wait complete. 2024-03-19T02:41:50.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 Processor -- start 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 -- start start 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 0x7f80d0100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d010bd80 0x7f80d0101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0111a30 con 0x7f80d0102a40 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01118b0 con 0x7f80d0103360 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d7a1f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0111bb0 con 0x7f80d010bd80 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45666/0 (socket says 172.21.15.116:45666) 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 -- 172.21.15.116:0/2963548540 learned_addr learned my addr 172.21.15.116:0/2963548540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d010bd80 0x7f80d0101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d4fba640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 0x7f80d0100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 -- 172.21.15.116:0/2963548540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d010bd80 msgr2=0x7f80d0101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d010bd80 0x7f80d0101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 -- 172.21.15.116:0/2963548540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 msgr2=0x7f80d0100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 0x7f80d0100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 -- 172.21.15.116:0/2963548540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d01a6450 con 0x7f80d0102a40 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d4fba640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 0x7f80d0100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.663+0000 7f80d57bb640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00e9f0 tx=0x7f80cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.664+0000 7f80c67fc640 1 -- 172.21.15.116:0/2963548540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00cd80 con 0x7f80d0102a40 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.664+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d01a6740 con 0x7f80d0102a40 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.664+0000 7f80c67fc640 1 -- 172.21.15.116:0/2963548540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80cc0108e0 con 0x7f80d0102a40 2024-03-19T02:41:50.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.664+0000 7f80c67fc640 1 -- 172.21.15.116:0/2963548540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc018920 con 0x7f80d0102a40 2024-03-19T02:41:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.664+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d01a6ce0 con 0x7f80d0102a40 2024-03-19T02:41:50.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:50 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1696555544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:41:50.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.666+0000 7f80c67fc640 1 -- 172.21.15.116:0/2963548540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f80cc010a90 con 0x7f80d0102a40 2024-03-19T02:41:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.667+0000 7f80da605740 1 Processor -- start 2024-03-19T02:41:50.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.667+0000 7f80da605740 1 -- start start 2024-03-19T02:41:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80c67fc640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 0x7f80bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cb8ed64b0 con 0x561cb8f732d0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561cb8e7ed70 con 0x561cb8f7c7d0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80da605740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cb8ed2550 con 0x561cb8f73bf0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.668+0000 7f80d5fbc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 0x7f80bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48096/0 (socket says 172.21.15.116:48096) 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 -- 172.21.15.116:0/3195709 learned_addr learned my addr 172.21.15.116:0/3195709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d57bb640 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 -- 172.21.15.116:0/3195709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f7c7d0 msgr2=0x561cb8f7ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 -- 172.21.15.116:0/3195709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 msgr2=0x561cb8f736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 -- 172.21.15.116:0/3195709 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb8ea1210 con 0x561cb8f73bf0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d57bb640 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80c67fc640 1 -- 172.21.15.116:0/2963548540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f80cc0cf670 con 0x7f80d0102a40 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.669+0000 7f80d4fba640 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009cc0 tx=0x7f80c00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65764997dbdb7c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.670+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 0x7f80bc0a4e30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009370 tx=0x7f80c802bf80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.670+0000 7f80bb7fe640 1 -- 172.21.15.116:0/3195709 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c001c480 con 0x561cb8f73bf0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.670+0000 7f80bb7fe640 1 -- 172.21.15.116:0/3195709 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80c001cad0 con 0x561cb8f73bf0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80bb7fe640 1 -- 172.21.15.116:0/3195709 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0011b80 con 0x561cb8f73bf0 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 -- 172.21.15.116:0/3195709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 msgr2=0x561cb8f7c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009cc0 tx=0x7f80c00107a0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 -- 172.21.15.116:0/3195709 shutdown_connections 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f7c290 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 --2- 172.21.15.116:0/3195709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 -- 172.21.15.116:0/3195709 >> 172.21.15.116:0/3195709 conn(0x561cb8f6da00 msgr2=0x561cb8f6dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 -- 172.21.15.116:0/3195709 shutdown_connections 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.671+0000 7f80da605740 1 -- 172.21.15.116:0/3195709 wait complete. 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 -- start start 2024-03-19T02:41:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 Processor -- start 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f797a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cb8f87e20 con 0x561cb8f73bf0 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561cb8e92310 con 0x561cb8f732d0 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cb8f87fa0 con 0x561cb8f7c7d0 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80da605740 10 client.0 _ll_register_callbacks cb 0x561cb8e862d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80d4fba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80d4fba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45670/0 (socket says 172.21.15.116:45670) 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80d4fba640 1 -- 172.21.15.116:0/4236944971 learned_addr learned my addr 172.21.15.116:0/4236944971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.672+0000 7f80d57bb640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 -- 172.21.15.116:0/4236944971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 msgr2=0x561cb8f7a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 -- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 msgr2=0x561cb8f797a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f797a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 -- 172.21.15.116:0/4236944971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb8bea3f0 con 0x561cb8f73bf0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80d4fba640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009ff0 tx=0x7f80c0011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e9f35ff72b24f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:50.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c002b060 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f80c001c890 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80c001cad0 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.673+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f80c0032050 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 10 client.19620 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561cb8cb47d0 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80c001cad0 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f80c0025070 con 0x561cb8f73bf0 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 10 client.19620 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 10 client.19620 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 20 client.19620 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8f95290 0x561cb8f97680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 4 client.19620 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cb8f343e0 con 0x561cb8f95290 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.674+0000 7f80d57bb640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8f95290 0x561cb8f97680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.675+0000 7f80d57bb640 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8f95290 0x561cb8f97680 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=718fb4a35cfb9e7e server_cookie=181070575e3d9a3 in_seq=0 out_seq=0 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.675+0000 7f80b8ff9640 10 client.19620 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80b8ff9640 1 -- 172.21.15.116:0/4236944971 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cb8f343e0 con 0x561cb8f95290 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80b8ff9640 10 client.19620 handle_command_reply: tid=0 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 client.19620 shutdown 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 msgr2=0x561cb8f79ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009ff0 tx=0x7f80c0011630 comp rx=0 tx=0).stop 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 shutdown_connections 2024-03-19T02:41:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8f95290 0x561cb8f97680 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f7a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f73bf0 0x561cb8f79ce0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80b8ff9640 0 client.19620 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 --2- 172.21.15.116:0/4236944971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f797a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.676+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 >> 172.21.15.116:0/4236944971 conn(0x561cb8f6da00 msgr2=0x561cb8f6f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.677+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 shutdown_connections 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.677+0000 7f80da605740 1 -- 172.21.15.116:0/4236944971 wait complete. 2024-03-19T02:41:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.677+0000 7f80da605740 20 client.19620 trim_cache size 0 max 16384 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 Processor -- start 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 -- start start 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f7bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f90c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 0x561cb8f6ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01acb30 con 0x561cb8f7c7d0 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01ae760 con 0x561cb8f732d0 2024-03-19T02:41:50.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.694+0000 7f80da605740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01ad200 con 0x561cb8f73bf0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f7bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 0x561cb8f6ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f7bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52574/0 (socket says 172.21.15.116:52574) 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d57bb640 1 -- 172.21.15.116:0/3020680184 learned_addr learned my addr 172.21.15.116:0/3020680184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d4fba640 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f90c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 -- 172.21.15.116:0/3020680184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 msgr2=0x561cb8f90c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f90c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 -- 172.21.15.116:0/3020680184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 msgr2=0x561cb8f7bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f7bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 -- 172.21.15.116:0/3020680184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cb8f6d830 con 0x561cb8f7c7d0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 0x561cb8f6ca20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f80c802c4c0 tx=0x7f80c8045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d9c80e4e53a61e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80baffd640 1 -- 172.21.15.116:0/3020680184 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8033e10 con 0x561cb8f7c7d0 2024-03-19T02:41:50.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80baffd640 1 -- 172.21.15.116:0/3020680184 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80c8054070 con 0x561cb8f7c7d0 2024-03-19T02:41:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.695+0000 7f80baffd640 1 -- 172.21.15.116:0/3020680184 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c802d610 con 0x561cb8f7c7d0 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- 172.21.15.116:0/3020680184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 msgr2=0x561cb8f6ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 0x561cb8f6ca20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f80c802c4c0 tx=0x7f80c8045070 comp rx=0 tx=0).stop 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- 172.21.15.116:0/3020680184 shutdown_connections 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f7c7d0 0x561cb8f6ca20 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f73bf0 0x561cb8f90c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 --2- 172.21.15.116:0/3020680184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f732d0 0x561cb8f7bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- 172.21.15.116:0/3020680184 >> 172.21.15.116:0/3020680184 conn(0x561cb8f6da00 msgr2=0x561cb8f6c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- 172.21.15.116:0/3020680184 shutdown_connections 2024-03-19T02:41:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- 172.21.15.116:0/3020680184 wait complete. 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 -- start start 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.696+0000 7f80da605740 1 Processor -- start 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f73bf0 0x561cb8f89000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f89540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01adfd0 con 0x561cb8f732d0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01ad900 con 0x561cb8f73bf0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01aab30 con 0x561cb8f7c7d0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80da605740 10 client.0 _ll_register_callbacks cb 0x7f80c0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45688/0 (socket says 172.21.15.116:45688) 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 -- 172.21.15.116:0/1411006206 learned_addr learned my addr 172.21.15.116:0/1411006206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d5fbc640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f89540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d4fba640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f73bf0 0x561cb8f89000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 -- 172.21.15.116:0/1411006206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 msgr2=0x561cb8f89540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f89540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 -- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f73bf0 msgr2=0x561cb8f89000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f73bf0 0x561cb8f89000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 -- 172.21.15.116:0/1411006206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cb8ffa420 con 0x561cb8f732d0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80d57bb640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f80cc09b820 tx=0x7f80cc097480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57d0f8c551dd968 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.697+0000 7f80b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0a4030 con 0x561cb8f732d0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f80cc097f00 con 0x561cb8f732d0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80c8054070 con 0x561cb8f732d0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f80cc0860b0 con 0x561cb8f732d0 2024-03-19T02:41:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 10 client.19626 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561cb8ffb6e0 con 0x561cb8f732d0 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80c8054070 con 0x561cb8f732d0 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f80cc0a3ba0 con 0x561cb8f732d0 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 10 client.19626 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 10 client.19626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 20 client.19626 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8ffbe10 0x561cb8ffe230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 4 client.19626 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561cb8ee39f0 con 0x561cb8ffbe10 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.698+0000 7f80d4fba640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8ffbe10 0x561cb8ffe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.699+0000 7f80b8ff9640 10 client.19626 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.699+0000 7f80d4fba640 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8ffbe10 0x561cb8ffe230 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dac875cbd23c5ea server_cookie=8d281368cf86b3b0 in_seq=0 out_seq=0 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.699+0000 7f80b8ff9640 1 -- 172.21.15.116:0/1411006206 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561cb8ee39f0 con 0x561cb8ffbe10 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.699+0000 7f80b8ff9640 10 client.19626 handle_command_reply: tid=0 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.699+0000 7f80da605740 1 client.19626 shutdown 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 msgr2=0x561cb8f88ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f80cc09b820 tx=0x7f80cc097480 comp rx=0 tx=0).stop 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 shutdown_connections 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561cb8ffbe10 0x561cb8ffe230 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cb8f7c7d0 0x561cb8f89540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561cb8f73bf0 0x561cb8f89000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80b8ff9640 0 client.19626 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 --2- 172.21.15.116:0/1411006206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cb8f732d0 0x561cb8f88ac0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 >> 172.21.15.116:0/1411006206 conn(0x561cb8f6da00 msgr2=0x561cb8f6cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 shutdown_connections 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 1 -- 172.21.15.116:0/1411006206 wait complete. 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80da605740 20 client.19626 trim_cache size 0 max 16384 2024-03-19T02:41:50.707 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:50.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 msgr2=0x7f80bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 0x7f80bc0a4e30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009370 tx=0x7f80c802bf80 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 msgr2=0x7f80d0100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.700+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f80cc00e9f0 tx=0x7f80cc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 shutdown_connections 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80bc0a2970 0x7f80bc0a4e30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80d010bd80 0x7f80d0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80d0103360 0x7f80d0100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 --2- 172.21.15.116:0/2963548540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80d0102a40 0x7f80d0100690 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 >> 172.21.15.116:0/2963548540 conn(0x7f80d00778d0 msgr2=0x7f80d00fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 shutdown_connections 2024-03-19T02:41:50.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.701+0000 7f80d7a1f640 1 -- 172.21.15.116:0/2963548540 wait complete. 2024-03-19T02:41:50.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:41:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.841+0000 7f48637c6640 1 -- 172.21.15.116:0/1769044948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c06a830 msgr2=0x7f485c10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.841+0000 7f48637c6640 1 --2- 172.21.15.116:0/1769044948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c06a830 0x7f485c10e670 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4854009ca0 tx=0x7f485402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 -- 172.21.15.116:0/1769044948 shutdown_connections 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 --2- 172.21.15.116:0/1769044948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c06a830 0x7f485c10e670 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 --2- 172.21.15.116:0/1769044948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c069e90 0x7f485c06a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 --2- 172.21.15.116:0/1769044948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 0x7f485c107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 -- 172.21.15.116:0/1769044948 >> 172.21.15.116:0/1769044948 conn(0x7f485c077500 msgr2=0x7f485c077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 -- 172.21.15.116:0/1769044948 shutdown_connections 2024-03-19T02:41:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 -- 172.21.15.116:0/1769044948 wait complete. 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 Processor -- start 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.842+0000 7f48637c6640 1 -- start start 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c06a830 0x7f485c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 0x7f485c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f485c111a60 con 0x7f485c069e90 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1118e0 con 0x7f485c06a830 2024-03-19T02:41:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f48637c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f485c111be0 con 0x7f485c107890 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45700/0 (socket says 172.21.15.116:45700) 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 -- 172.21.15.116:0/2384969360 learned_addr learned my addr 172.21.15.116:0/2384969360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4860d61640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c06a830 0x7f485c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861d63640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 0x7f485c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 -- 172.21.15.116:0/2384969360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 msgr2=0x7f485c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 0x7f485c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 -- 172.21.15.116:0/2384969360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c06a830 msgr2=0x7f485c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c06a830 0x7f485c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 -- 172.21.15.116:0/2384969360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c19ceb0 con 0x7f485c069e90 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.843+0000 7f4861562640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f485800e9f0 tx=0x7f485800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.844+0000 7f48527fc640 1 -- 172.21.15.116:0/2384969360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485800cd80 con 0x7f485c069e90 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.844+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485c19d1a0 con 0x7f485c069e90 2024-03-19T02:41:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.844+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f485c1a4a30 con 0x7f485c069e90 2024-03-19T02:41:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.844+0000 7f48527fc640 1 -- 172.21.15.116:0/2384969360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48580054d0 con 0x7f485c069e90 2024-03-19T02:41:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.844+0000 7f48527fc640 1 -- 172.21.15.116:0/2384969360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858018630 con 0x7f485c069e90 2024-03-19T02:41:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.846+0000 7f48527fc640 1 -- 172.21.15.116:0/2384969360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f48580046c0 con 0x7f485c069e90 2024-03-19T02:41:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 Processor -- start 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 -- start start 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 0x55703eff96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55703ef5c4b0 con 0x55703f0027d0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55703ef04d70 con 0x55703eff9bf0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f48663b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55703ef58550 con 0x55703eff92d0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4861562640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4860d61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4861d63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 0x55703eff96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4860d61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45716/0 (socket says 172.21.15.116:45716) 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4861562640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52602/0 (socket says 172.21.15.116:52602) 2024-03-19T02:41:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.847+0000 7f4860d61640 1 -- 172.21.15.116:0/3204909383 learned_addr learned my addr 172.21.15.116:0/3204909383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f48527fc640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 0x7f48480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 -- 172.21.15.116:0/3204909383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 msgr2=0x55703eff96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 0x55703eff96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 -- 172.21.15.116:0/3204909383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 msgr2=0x55703f002290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 -- 172.21.15.116:0/3204909383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55703ef27210 con 0x55703f0027d0 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4861562640 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4861d63640 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 0x55703eff96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4861d63640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 0x7f48480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f48527fc640 1 -- 172.21.15.116:0/2384969360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f48580cf5b0 con 0x7f485c069e90 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4860d61640 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f484c009c90 tx=0x7f484c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682a02ae9b85c508 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.848+0000 7f4861d63640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 0x7f48480a4eb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4854004af0 tx=0x7f485402e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.849+0000 7f48477fe640 1 -- 172.21.15.116:0/3204909383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c01e070 con 0x55703f0027d0 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.849+0000 7f48477fe640 1 -- 172.21.15.116:0/3204909383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f484c019ae0 con 0x55703f0027d0 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.849+0000 7f48477fe640 1 -- 172.21.15.116:0/3204909383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c00ea40 con 0x55703f0027d0 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- 172.21.15.116:0/3204909383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 msgr2=0x55703f004bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f484c009c90 tx=0x7f484c011810 comp rx=0 tx=0).stop 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- 172.21.15.116:0/3204909383 shutdown_connections 2024-03-19T02:41:50.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703f0027d0 0x55703f004bc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f002290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 --2- 172.21.15.116:0/3204909383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff92d0 0x55703eff96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- 172.21.15.116:0/3204909383 >> 172.21.15.116:0/3204909383 conn(0x55703eff3a00 msgr2=0x55703eff3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- 172.21.15.116:0/3204909383 shutdown_connections 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- 172.21.15.116:0/3204909383 wait complete. 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 -- start start 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.850+0000 7f48663b2740 1 Processor -- start 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703efffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f000210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55703f0099d0 con 0x55703eff92d0 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55703ef18310 con 0x55703eff9bf0 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55703f009b50 con 0x55703f0027d0 2024-03-19T02:41:50.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f48663b2740 10 client.0 _ll_register_callbacks cb 0x55703ef0c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45728/0 (socket says 172.21.15.116:45728) 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 -- 172.21.15.116:0/1938657118 learned_addr learned my addr 172.21.15.116:0/1938657118 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861d63640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f000210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 -- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 msgr2=0x55703f000210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f000210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 -- 172.21.15.116:0/1938657118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 msgr2=0x55703efffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4860d61640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703efffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703efffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.851+0000 7f4861562640 1 -- 172.21.15.116:0/1938657118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55703ec703f0 con 0x55703eff92d0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4861562640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f48580a3f40 tx=0x7f485809b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28870f7fd3f96639 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:50.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858005680 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f48580cf120 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f484c019ae0 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f485809bf30 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f48663b2740 10 client.19638 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55703ed3a7d0 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f484c019ae0 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f485809cb60 con 0x55703eff92d0 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f48663b2740 10 client.19638 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.852+0000 7f48663b2740 10 client.19638 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f48663b2740 20 client.19638 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f017040 0x55703f019500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f48663b2740 4 client.19638 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55703efba3e0 con 0x55703f017040 2024-03-19T02:41:50.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f4860d61640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f017040 0x55703f019500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f4844ff9640 10 client.19638 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.853+0000 7f4860d61640 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f017040 0x55703f019500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0a72ec07d437c12 server_cookie=d8e998aabca9116b in_seq=0 out_seq=0 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f4844ff9640 1 -- 172.21.15.116:0/1938657118 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55703efba3e0 con 0x55703f017040 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f4844ff9640 10 client.19638 handle_command_reply: tid=0 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 client.19638 shutdown 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 msgr2=0x55703efff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f48580a3f40 tx=0x7f485809b600 comp rx=0 tx=0).stop 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 shutdown_connections 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f017040 0x55703f019500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f000210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703efffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 --2- 172.21.15.116:0/1938657118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703efff790 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 >> 172.21.15.116:0/1938657118 conn(0x55703eff3a00 msgr2=0x55703eff5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f4844ff9640 0 client.19638 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:50.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 shutdown_connections 2024-03-19T02:41:50.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 1 -- 172.21.15.116:0/1938657118 wait complete. 2024-03-19T02:41:50.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.854+0000 7f48663b2740 20 client.19638 trim_cache size 0 max 16384 2024-03-19T02:41:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 Processor -- start 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 -- start start 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 0x55703eff2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703f0027d0 0x55703f01edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f484000ff10 con 0x55703eff92d0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4840011b40 con 0x55703f0027d0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f48663b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48400105e0 con 0x55703eff9bf0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45732/0 (socket says 172.21.15.116:45732) 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 -- 172.21.15.116:0/11314210 learned_addr learned my addr 172.21.15.116:0/11314210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4860d61640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 0x55703eff2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861d63640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703f0027d0 0x55703f01edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 -- 172.21.15.116:0/11314210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 msgr2=0x55703eff2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 0x55703eff2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 -- 172.21.15.116:0/11314210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703f0027d0 msgr2=0x55703f01edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703f0027d0 0x55703f01edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 -- 172.21.15.116:0/11314210 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55703eff3830 con 0x55703eff92d0 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4860d61640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 0x55703eff2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.873+0000 7f4861562640 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4858097c90 tx=0x7f4858099d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609a405f5b88e3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.874+0000 7f4846ffd640 1 -- 172.21.15.116:0/11314210 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48580cf410 con 0x55703eff92d0 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.874+0000 7f4846ffd640 1 -- 172.21.15.116:0/11314210 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48580b9030 con 0x55703eff92d0 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.874+0000 7f4846ffd640 1 -- 172.21.15.116:0/11314210 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858097580 con 0x55703eff92d0 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.874+0000 7f48663b2740 1 -- 172.21.15.116:0/11314210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 msgr2=0x55703f0127d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.874+0000 7f48663b2740 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f4858097c90 tx=0x7f4858099d40 comp rx=0 tx=0).stop 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 -- 172.21.15.116:0/11314210 shutdown_connections 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703f0027d0 0x55703f01edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703eff9bf0 0x55703eff2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 --2- 172.21.15.116:0/11314210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f0127d0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 -- 172.21.15.116:0/11314210 >> 172.21.15.116:0/11314210 conn(0x55703eff3a00 msgr2=0x55703eff2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 -- 172.21.15.116:0/11314210 shutdown_connections 2024-03-19T02:41:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 -- 172.21.15.116:0/11314210 wait complete. 2024-03-19T02:41:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 -- start start 2024-03-19T02:41:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.875+0000 7f48663b2740 1 Processor -- start 2024-03-19T02:41:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f00b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f07c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f48400113b0 con 0x55703eff92d0 2024-03-19T02:41:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4840010ce0 con 0x55703eff9bf0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f484000df10 con 0x55703f0027d0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 10 client.0 _ll_register_callbacks cb 0x7f48580d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45742/0 (socket says 172.21.15.116:45742) 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 -- 172.21.15.116:0/3155862685 learned_addr learned my addr 172.21.15.116:0/3155862685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861d63640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f07c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4860d61640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f00b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 -- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 msgr2=0x55703f07c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f07c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 -- 172.21.15.116:0/3155862685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 msgr2=0x55703f00b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f00b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 -- 172.21.15.116:0/3155862685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55703efc9880 con 0x55703eff92d0 2024-03-19T02:41:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4861562640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4858086010 tx=0x7f485809cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1b4cfbf64ba397 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48580ad060 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f48580af740 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.876+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48580b9030 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f485808b050 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 10 client.19644 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55703f07d970 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48580b9030 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f48580af2b0 con 0x55703eff92d0 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 10 client.19644 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 10 client.19644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:41:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 20 client.19644 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f07ddf0 0x55703f0801e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 4 client.19644 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55703ef699f0 con 0x55703f07ddf0 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.877+0000 7f4860d61640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f07ddf0 0x55703f0801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f4860d61640 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f07ddf0 0x55703f0801e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa9b1b8dcff5c32 server_cookie=9a4f229219935fad in_seq=0 out_seq=0 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f4844ff9640 10 client.19644 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f4844ff9640 1 -- 172.21.15.116:0/3155862685 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55703ef699f0 con 0x55703f07ddf0 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f4844ff9640 10 client.19644 handle_command_reply: tid=0 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 client.19644 shutdown 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 msgr2=0x55703f00ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4858086010 tx=0x7f485809cf20 comp rx=0 tx=0).stop 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 shutdown_connections 2024-03-19T02:41:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55703f07ddf0 0x55703f0801e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55703f0027d0 0x55703f07c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55703eff9bf0 0x55703f00b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 --2- 172.21.15.116:0/3155862685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55703eff92d0 0x55703f00ad80 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 >> 172.21.15.116:0/3155862685 conn(0x55703eff3a00 msgr2=0x55703eff2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f4844ff9640 0 client.19644 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 shutdown_connections 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 1 -- 172.21.15.116:0/3155862685 wait complete. 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.878+0000 7f48663b2740 20 client.19644 trim_cache size 0 max 16384 2024-03-19T02:41:50.886 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 msgr2=0x7f48480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 0x7f48480a4eb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4854004af0 tx=0x7f485402e040 comp rx=0 tx=0).stop 2024-03-19T02:41:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 msgr2=0x7f485c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f485800e9f0 tx=0x7f485800edf0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 shutdown_connections 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f48480a29f0 0x7f48480a4eb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f485c107890 0x7f485c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f485c06a830 0x7f485c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 --2- 172.21.15.116:0/2384969360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f485c069e90 0x7f485c197ec0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 >> 172.21.15.116:0/2384969360 conn(0x7f485c077500 msgr2=0x7f485c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 shutdown_connections 2024-03-19T02:41:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:50.879+0000 7f48637c6640 1 -- 172.21.15.116:0/2384969360 wait complete. 2024-03-19T02:41:50.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:41:51.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 -- 172.21.15.116:0/3231645945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 msgr2=0x7f9638101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 --2- 172.21.15.116:0/3231645945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638101c10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f962c009930 tx=0x7f962c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:51.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 -- 172.21.15.116:0/3231645945 shutdown_connections 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 --2- 172.21.15.116:0/3231645945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9638102150 0x7f963810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 --2- 172.21.15.116:0/3231645945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638101c10 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 --2- 172.21.15.116:0/3231645945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96381077a0 0x7f9638107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 -- 172.21.15.116:0/3231645945 >> 172.21.15.116:0/3231645945 conn(0x7f96380fd2c0 msgr2=0x7f96380ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 -- 172.21.15.116:0/3231645945 shutdown_connections 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.023+0000 7f963ceea640 1 -- 172.21.15.116:0/3231645945 wait complete. 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 Processor -- start 2024-03-19T02:41:51.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 -- start start 2024-03-19T02:41:51.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 0x7f9638198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96381077a0 0x7f963819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9638111a60 con 0x7f9638101790 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96381118e0 con 0x7f9638102150 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963ceea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9638111be0 con 0x7f96381077a0 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45766/0 (socket says 172.21.15.116:45766) 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f963659c640 1 -- 172.21.15.116:0/3360813453 learned_addr learned my addr 172.21.15.116:0/3360813453 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f9635d9b640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 0x7f9638198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.024+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96381077a0 0x7f963819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 -- 172.21.15.116:0/3360813453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96381077a0 msgr2=0x7f963819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96381077a0 0x7f963819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 -- 172.21.15.116:0/3360813453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 msgr2=0x7f9638198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 0x7f9638198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 -- 172.21.15.116:0/3360813453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963819cfb0 con 0x7f9638101790 2024-03-19T02:41:51.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f9635d9b640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 0x7f9638198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963659c640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f962800e9f0 tx=0x7f962800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f96277fe640 1 -- 172.21.15.116:0/3360813453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962800cd80 con 0x7f9638101790 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f963819d2a0 con 0x7f9638101790 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f96277fe640 1 -- 172.21.15.116:0/3360813453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96280108e0 con 0x7f9638101790 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96381a4b30 con 0x7f9638101790 2024-03-19T02:41:51.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.025+0000 7f96277fe640 1 -- 172.21.15.116:0/3360813453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628018920 con 0x7f9638101790 2024-03-19T02:41:51.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.028+0000 7f963fbb4740 1 Processor -- start 2024-03-19T02:41:51.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 -- start start 2024-03-19T02:41:51.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f96277fe640 1 -- 172.21.15.116:0/3360813453 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9628010a90 con 0x7f9638101790 2024-03-19T02:41:51.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e1d40 0x55aba19e2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aba19454b0 con 0x55aba19e26f0 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aba18edd70 con 0x55aba19f3f00 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963fbb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aba1941550 con 0x55aba19e1d40 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e1d40 0x55aba19e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45776/0 (socket says 172.21.15.116:45776) 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f963659c640 1 -- 172.21.15.116:0/537149804 learned_addr learned my addr 172.21.15.116:0/537149804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.029+0000 7f9635d9b640 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 -- 172.21.15.116:0/537149804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e1d40 msgr2=0x55aba19e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e1d40 0x55aba19e2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 -- 172.21.15.116:0/537149804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 msgr2=0x55aba19f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 -- 172.21.15.116:0/537149804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aba1910210 con 0x55aba19e26f0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f963659c640 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f96280cc780 tx=0x7f96280d5b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc0f3b2767b7b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f9635d9b640 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f9624ff9640 1 -- 172.21.15.116:0/537149804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280e2070 con 0x55aba19e26f0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f96277fe640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 0x7f96200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f9624ff9640 1 -- 172.21.15.116:0/537149804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96280cfd30 con 0x55aba19e26f0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.030+0000 7f9624ff9640 1 -- 172.21.15.116:0/537149804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96280d7a80 con 0x55aba19e26f0 2024-03-19T02:41:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.031+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 0x7f96200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.031+0000 7f96277fe640 1 -- 172.21.15.116:0/3360813453 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f96280cf530 con 0x7f9638101790 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.031+0000 7f963fbb4740 1 -- 172.21.15.116:0/537149804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 msgr2=0x55aba19e2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.031+0000 7f963fbb4740 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f96280cc780 tx=0x7f96280d5b00 comp rx=0 tx=0).stop 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 0x7f96200a4e30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9630005fd0 tx=0x7f9630009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 -- 172.21.15.116:0/537149804 shutdown_connections 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e26f0 0x55aba19e2b70 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 --2- 172.21.15.116:0/537149804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e1d40 0x55aba19e2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 -- 172.21.15.116:0/537149804 >> 172.21.15.116:0/537149804 conn(0x55aba19dca00 msgr2=0x55aba19dcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 -- 172.21.15.116:0/537149804 shutdown_connections 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 -- 172.21.15.116:0/537149804 wait complete. 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 -- start start 2024-03-19T02:41:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.032+0000 7f963fbb4740 1 Processor -- start 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 0x55aba19e4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 0x55aba19e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aba19f6db0 con 0x55aba19e1d40 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aba1901310 con 0x55aba19e26f0 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aba19f6f30 con 0x55aba19f3f00 2024-03-19T02:41:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963fbb4740 10 client.0 _ll_register_callbacks cb 0x55aba18f52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9635d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 0x55aba19e4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 0x55aba19e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9635d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 0x55aba19e4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52668/0 (socket says 172.21.15.116:52668) 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45780/0 (socket says 172.21.15.116:45780) 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9635d9b640 1 -- 172.21.15.116:0/2496161391 learned_addr learned my addr 172.21.15.116:0/2496161391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 -- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 msgr2=0x55aba19e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 0x55aba19e4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 -- 172.21.15.116:0/2496161391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 msgr2=0x55aba19e4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 -- 172.21.15.116:0/2496161391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aba16593f0 con 0x55aba19f3f00 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f963659c640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f9636d9d640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 0x55aba19e4dc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9630003f60 tx=0x7f9630004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0327b42e3adfe5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.033+0000 7f961dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963000f8f0 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96280cfd30 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f963000fa90 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9630031050 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 10 client.29377 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aba17237d0 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96280cfd30 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f9630026020 con 0x55aba19f3f00 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 10 client.29377 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 10 client.29377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 20 client.29377 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a06650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 4 client.29377 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:51.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aba19a33e0 con 0x55aba1a04210 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.034+0000 7f963659c640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a06650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.035+0000 7f963659c640 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a06650 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99a41e1764d9226f server_cookie=b8a10089c1d39077 in_seq=0 out_seq=0 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.035+0000 7f961dffb640 10 client.29377 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f961dffb640 1 -- 172.21.15.116:0/2496161391 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aba19a33e0 con 0x55aba1a04210 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f961dffb640 10 client.29377 handle_command_reply: tid=0 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 client.29377 shutdown 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 msgr2=0x55aba19e4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 0x55aba19e4dc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9630003f60 tx=0x7f9630004ab0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 shutdown_connections 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a06650 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f961dffb640 0 client.29377 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19f3f00 0x55aba19e4dc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e26f0 0x55aba19e4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 --2- 172.21.15.116:0/2496161391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19e4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 >> 172.21.15.116:0/2496161391 conn(0x55aba19dca00 msgr2=0x55aba19de750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 shutdown_connections 2024-03-19T02:41:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.036+0000 7f963fbb4740 1 -- 172.21.15.116:0/2496161391 wait complete. 2024-03-19T02:41:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.037+0000 7f963fbb4740 20 client.29377 trim_cache size 0 max 16384 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.056+0000 7f963fbb4740 1 Processor -- start 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.056+0000 7f963fbb4740 1 -- start start 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e1d40 0x55aba19e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96381acbb0 con 0x55aba19f3f00 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ae7e0 con 0x55aba19e1d40 2024-03-19T02:41:51.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963fbb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ad280 con 0x55aba19e26f0 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45782/0 (socket says 172.21.15.116:45782) 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 -- 172.21.15.116:0/3931057623 learned_addr learned my addr 172.21.15.116:0/3931057623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f963659c640 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e1d40 0x55aba19e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9635d9b640 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 -- 172.21.15.116:0/3931057623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 msgr2=0x55aba19ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 -- 172.21.15.116:0/3931057623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e1d40 msgr2=0x55aba19e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e1d40 0x55aba19e6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 -- 172.21.15.116:0/3931057623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aba19b2880 con 0x55aba19f3f00 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f9636d9d640 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f96300114f0 tx=0x7f9630009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db2897211f4955 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.057+0000 7f961ffff640 1 -- 172.21.15.116:0/3931057623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630004910 con 0x55aba19f3f00 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f961ffff640 1 -- 172.21.15.116:0/3931057623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f963000fda0 con 0x55aba19f3f00 2024-03-19T02:41:51.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f961ffff640 1 -- 172.21.15.116:0/3931057623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963001e400 con 0x55aba19f3f00 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 -- 172.21.15.116:0/3931057623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 msgr2=0x55aba19dcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f96300114f0 tx=0x7f9630009f10 comp rx=0 tx=0).stop 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 -- 172.21.15.116:0/3931057623 shutdown_connections 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19f3f00 0x55aba19dcde0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 --2- 172.21.15.116:0/3931057623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19e1d40 0x55aba19e6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 -- 172.21.15.116:0/3931057623 >> 172.21.15.116:0/3931057623 conn(0x55aba19dca00 msgr2=0x55aba19dbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 -- 172.21.15.116:0/3931057623 shutdown_connections 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.058+0000 7f963fbb4740 1 -- 172.21.15.116:0/3931057623 wait complete. 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 -- start start 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 Processor -- start 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19f7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ae050 con 0x55aba19e1d40 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ad980 con 0x55aba19f3f00 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96381aabb0 con 0x55aba19e26f0 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963fbb4740 10 client.0 _ll_register_callbacks cb 0x7f9630009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45790/0 (socket says 172.21.15.116:45790) 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 -- 172.21.15.116:0/4128000554 learned_addr learned my addr 172.21.15.116:0/4128000554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f9635d9b640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f9636d9d640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 -- 172.21.15.116:0/4128000554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 msgr2=0x55aba19f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 -- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 msgr2=0x55aba19f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:51.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.059+0000 7f963659c640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f963659c640 1 -- 172.21.15.116:0/4128000554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aba19f7c00 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f963659c640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f962809b790 tx=0x7f9628000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57a0092c391d8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f961dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628097aa0 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f96280d5150 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f963000fda0 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f962809b160 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f963fbb4740 10 client.19662 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55aba1a6b0d0 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.060+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f963000fda0 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f96280d2cd0 con 0x55aba19e1d40 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 10 client.19662 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 10 client.19662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:51.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 20 client.19662 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a6d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 4 client.19662 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aba19529f0 con 0x55aba1a04210 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f9635d9b640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a6d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f961dffb640 10 client.19662 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:51.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.061+0000 7f9635d9b640 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a6d3f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66648f908e8a228b server_cookie=359ccc293bccfaca in_seq=0 out_seq=0 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f961dffb640 1 -- 172.21.15.116:0/4128000554 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aba19529f0 con 0x55aba1a04210 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f961dffb640 10 client.19662 handle_command_reply: tid=0 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 client.19662 shutdown 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 msgr2=0x55aba19fc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f962809b790 tx=0x7f9628000bc0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 shutdown_connections 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55aba1a04210 0x55aba1a6d3f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aba19f3f00 0x55aba19f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aba19e26f0 0x55aba19f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f961dffb640 0 client.19662 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 --2- 172.21.15.116:0/4128000554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aba19e1d40 0x55aba19fc1d0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 >> 172.21.15.116:0/4128000554 conn(0x55aba19dca00 msgr2=0x55aba19dbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 shutdown_connections 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 1 -- 172.21.15.116:0/4128000554 wait complete. 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.065+0000 7f963fbb4740 20 client.19662 trim_cache size 0 max 16384 2024-03-19T02:41:51.069 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 msgr2=0x7f96200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 0x7f96200a4e30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9630005fd0 tx=0x7f9630009500 comp rx=0 tx=0).stop 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 msgr2=0x7f9638197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f962800e9f0 tx=0x7f962800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 shutdown_connections 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96200a2970 0x7f96200a4e30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96381077a0 0x7f963819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638102150 0x7f9638198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 --2- 172.21.15.116:0/3360813453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9638101790 0x7f9638197fc0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:51.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 >> 172.21.15.116:0/3360813453 conn(0x7f96380fd2c0 msgr2=0x7f96380fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:51.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 shutdown_connections 2024-03-19T02:41:51.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:51.066+0000 7f963ceea640 1 -- 172.21.15.116:0/3360813453 wait complete. 2024-03-19T02:41:51.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8726 9.00 MB/sec warmup 73 sec latency 147.485 ms 2024-03-19T02:41:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:51 smithi154 ceph-mon[35693]: pgmap v440: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.3 MiB/s wr, 191 op/s 2024-03-19T02:41:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:51 smithi116 ceph-mon[32263]: pgmap v440: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.3 MiB/s wr, 191 op/s 2024-03-19T02:41:51.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:51 smithi162 ceph-mon[35761]: pgmap v440: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.3 MiB/s wr, 191 op/s 2024-03-19T02:41:52.079 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:41:52.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:41:52.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:41:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 -- 172.21.15.116:0/3293032018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80041066a0 msgr2=0x7f8004106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 --2- 172.21.15.116:0/3293032018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80041066a0 0x7f8004106a80 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009930 tx=0x7f7ff402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 -- 172.21.15.116:0/3293032018 shutdown_connections 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 --2- 172.21.15.116:0/3293032018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8004101050 0x7f800410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 --2- 172.21.15.116:0/3293032018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 0x7f8004100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 --2- 172.21.15.116:0/3293032018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80041066a0 0x7f8004106a80 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 -- 172.21.15.116:0/3293032018 >> 172.21.15.116:0/3293032018 conn(0x7f8004077820 msgr2=0x7f8004077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 -- 172.21.15.116:0/3293032018 shutdown_connections 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.217+0000 7f800a630640 1 -- 172.21.15.116:0/3293032018 wait complete. 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 Processor -- start 2024-03-19T02:41:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 -- start start 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 0x7f8004198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80041066a0 0x7f80041a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8004110920 con 0x7f8004101050 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80041107a0 con 0x7f8004100690 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f800a630640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8004110aa0 con 0x7f80041066a0 2024-03-19T02:41:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f80037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f80037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45810/0 (socket says 172.21.15.116:45810) 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.218+0000 7f80037fe640 1 -- 172.21.15.116:0/4159738480 learned_addr learned my addr 172.21.15.116:0/4159738480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f8003fff640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 0x7f8004198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f8008bcd640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80041066a0 0x7f80041a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 -- 172.21.15.116:0/4159738480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80041066a0 msgr2=0x7f80041a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80041066a0 0x7f80041a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 -- 172.21.15.116:0/4159738480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 msgr2=0x7f8004198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 0x7f8004198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 -- 172.21.15.116:0/4159738480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80041a4400 con 0x7f8004101050 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80037fe640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d950 tx=0x7f7ff800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80017fa640 1 -- 172.21.15.116:0/4159738480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8017030 con 0x7f8004101050 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80041a45d0 con 0x7f8004101050 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80041a4b40 con 0x7f8004101050 2024-03-19T02:41:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80017fa640 1 -- 172.21.15.116:0/4159738480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ff8004020 con 0x7f8004101050 2024-03-19T02:41:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.219+0000 7f80017fa640 1 -- 172.21.15.116:0/4159738480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80047a0 con 0x7f8004101050 2024-03-19T02:41:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f80017fa640 1 -- 172.21.15.116:0/4159738480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7ff80041d0 con 0x7f8004101050 2024-03-19T02:41:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 Processor -- start 2024-03-19T02:41:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 -- start start 2024-03-19T02:41:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea0f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adea053160 con 0x55adea0f8880 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ade9ffbe10 con 0x55adea0efca0 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f800d217740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55adea04f200 con 0x55adea0ef380 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f8008bcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f80037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f80037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45816/0 (socket says 172.21.15.116:45816) 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f8008bcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48278/0 (socket says 172.21.15.116:48278) 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f80037fe640 1 -- 172.21.15.116:0/1959394021 learned_addr learned my addr 172.21.15.116:0/1959394021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f8003fff640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea0f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.222+0000 7f80017fa640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 0x7f7fec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 -- 172.21.15.116:0/1959394021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 msgr2=0x55adea0ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 -- 172.21.15.116:0/1959394021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 msgr2=0x55adea0f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea0f8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80017fa640 1 -- 172.21.15.116:0/4159738480 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f7ff80ceba0 con 0x7f8004101050 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 -- 172.21.15.116:0/1959394021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ade9e2f7e0 con 0x55adea0f8880 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f8008bcd640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f8003fff640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea0f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f8008bcd640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 0x7f7fec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f80037fe640 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8097280 tx=0x7f7ff8005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c42fb85b2140e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f7fea7fc640 1 -- 172.21.15.116:0/1959394021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff809a780 con 0x55adea0f8880 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f7fea7fc640 1 -- 172.21.15.116:0/1959394021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ff804d3d0 con 0x55adea0f8880 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.223+0000 7f7fea7fc640 1 -- 172.21.15.116:0/1959394021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80cc190 con 0x55adea0f8880 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- 172.21.15.116:0/1959394021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 msgr2=0x55adea0fac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f8008bcd640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 0x7f7fec0a4eb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005fd0 tx=0x7f7ffc004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8097280 tx=0x7f7ff8005bf0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- 172.21.15.116:0/1959394021 shutdown_connections 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea0fac70 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea0f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 --2- 172.21.15.116:0/1959394021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0ef380 0x55adea0ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- 172.21.15.116:0/1959394021 >> 172.21.15.116:0/1959394021 conn(0x55adea0e9ab0 msgr2=0x55adea0e9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- 172.21.15.116:0/1959394021 shutdown_connections 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- 172.21.15.116:0/1959394021 wait complete. 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.224+0000 7f800d217740 1 -- start start 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 Processor -- start 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea0f5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea0f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0f8880 0x55adea0f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adea0fb5b0 con 0x55adea0ef380 2024-03-19T02:41:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55adea00f1a0 con 0x55adea0f8880 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55adea0fb730 con 0x55adea0efca0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f800d217740 10 client.0 _ll_register_callbacks cb 0x55ade9d66c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea0f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea0f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea0f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45826/0 (socket says 172.21.15.116:45826) 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f8003fff640 1 -- 172.21.15.116:0/2744292408 learned_addr learned my addr 172.21.15.116:0/2744292408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f8008bcd640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0f8880 0x55adea0f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 -- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0f8880 msgr2=0x55adea0f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0f8880 0x55adea0f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 -- 172.21.15.116:0/2744292408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 msgr2=0x55adea0f5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea0f5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.225+0000 7f80037fe640 1 -- 172.21.15.116:0/2744292408 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ade9e8ff70 con 0x55adea0efca0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f80037fe640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea0f5d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55adea0fb240 tx=0x7f7ff8005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cef0e2d915dbdc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff809a780 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ff804d3d0 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f7ff8097490 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7ff809ae20 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 10 client.29395 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ade9d27170 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ff804d3d0 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f7ff80b2070 con 0x55adea0efca0 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 10 client.29395 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 10 client.29395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 20 client.29395 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea108930 0x55adea10ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 4 client.29395 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.226+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adea0606a0 con 0x55adea108930 2024-03-19T02:41:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.227+0000 7f8003fff640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea108930 0x55adea10ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.227+0000 7f8003fff640 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea108930 0x55adea10ad20 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4ae819946c66d85 server_cookie=55f9a5b599f725c0 in_seq=0 out_seq=0 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.227+0000 7f7fe37fe640 10 client.29395 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f7fe37fe640 1 -- 172.21.15.116:0/2744292408 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adea0606a0 con 0x55adea108930 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f7fe37fe640 10 client.29395 handle_command_reply: tid=0 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 client.29395 shutdown 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 msgr2=0x55adea0f5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea0f5d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x55adea0fb240 tx=0x7f7ff8005d30 comp rx=0 tx=0).stop 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 shutdown_connections 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea108930 0x55adea10ad20 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0f8880 0x55adea0f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea0f5d60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f7fe37fe640 0 client.29395 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 --2- 172.21.15.116:0/2744292408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea0f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 >> 172.21.15.116:0/2744292408 conn(0x55adea0e9ab0 msgr2=0x55adea0eb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 shutdown_connections 2024-03-19T02:41:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 1 -- 172.21.15.116:0/2744292408 wait complete. 2024-03-19T02:41:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.229+0000 7f800d217740 20 client.29395 trim_cache size 0 max 16384 2024-03-19T02:41:52.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.256+0000 7f800d217740 1 Processor -- start 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 -- start start 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 0x55adea0f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea104350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea1102c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80041aec40 con 0x55adea0f8880 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80041b0870 con 0x55adea0ef380 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f800d217740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80041af310 con 0x55adea0efca0 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f8008bcd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea1102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f8003fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 0x55adea0f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f8008bcd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea1102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45842/0 (socket says 172.21.15.116:45842) 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f8008bcd640 1 -- 172.21.15.116:0/1884218505 learned_addr learned my addr 172.21.15.116:0/1884218505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea104350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 -- 172.21.15.116:0/1884218505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 msgr2=0x55adea0f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 0x55adea0f7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 -- 172.21.15.116:0/1884218505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 msgr2=0x55adea1102c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea1102c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f80037fe640 1 -- 172.21.15.116:0/1884218505 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adea0eeb20 con 0x55adea0efca0 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.257+0000 7f8003fff640 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 0x55adea0f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.258+0000 7f80037fe640 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea104350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7ff809adb0 tx=0x7f7ff80ceea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=423edef3d244576f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.258+0000 7f7fe9ffb640 1 -- 172.21.15.116:0/1884218505 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff809a780 con 0x55adea0efca0 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.258+0000 7f7fe9ffb640 1 -- 172.21.15.116:0/1884218505 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ff804d3d0 con 0x55adea0efca0 2024-03-19T02:41:52.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.258+0000 7f7fe9ffb640 1 -- 172.21.15.116:0/1884218505 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff809ef70 con 0x55adea0efca0 2024-03-19T02:41:52.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- 172.21.15.116:0/1884218505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 msgr2=0x55adea104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea104350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7ff809adb0 tx=0x7f7ff80ceea0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- 172.21.15.116:0/1884218505 shutdown_connections 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0f8880 0x55adea1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0efca0 0x55adea104350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 --2- 172.21.15.116:0/1884218505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0ef380 0x55adea0f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- 172.21.15.116:0/1884218505 >> 172.21.15.116:0/1884218505 conn(0x55adea0e9ab0 msgr2=0x55adea0eb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- 172.21.15.116:0/1884218505 shutdown_connections 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- 172.21.15.116:0/1884218505 wait complete. 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 -- start start 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.259+0000 7f800d217740 1 Processor -- start 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea100a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0f8880 0x55adea0fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80041b00e0 con 0x55adea0ef380 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80041afa10 con 0x55adea0efca0 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80041acc40 con 0x55adea0f8880 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f800d217740 10 client.0 _ll_register_callbacks cb 0x7f7ff4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45852/0 (socket says 172.21.15.116:45852) 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 -- 172.21.15.116:0/96976470 learned_addr learned my addr 172.21.15.116:0/96976470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f80037fe640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8008bcd640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0f8880 0x55adea0fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 -- 172.21.15.116:0/96976470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0f8880 msgr2=0x55adea0fc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0f8880 0x55adea0fc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 -- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 msgr2=0x55adea100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f8003fff640 1 -- 172.21.15.116:0/96976470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adea0fd160 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.260+0000 7f80037fe640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f8003fff640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4005e00 tx=0x7f7ff4004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc64d6de2f1d1851 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f7fe37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff402c9f0 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f7ff403a040 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ff804d3d0 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7ff4057050 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f800d217740 10 client.19680 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55adea177bf0 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.261+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ff402cb90 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f7ff4004a90 con 0x55adea0ef380 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 10 client.19680 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:52.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 10 client.19680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 20 client.19680 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea178400 0x55adea17a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 4 client.19680 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f7ff804d3d0 con 0x55adea178400 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.262+0000 7f80037fe640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea178400 0x55adea17a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.263+0000 7f80037fe640 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea178400 0x55adea17a820 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad3d8bb9924d0f03 server_cookie=2317ddd8db6c9160 in_seq=0 out_seq=0 2024-03-19T02:41:52.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.263+0000 7f7fe37fe640 10 client.19680 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.265+0000 7f7fe37fe640 1 -- 172.21.15.116:0/96976470 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f7ff804d3d0 con 0x55adea178400 2024-03-19T02:41:52.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.265+0000 7f7fe37fe640 10 client.19680 handle_command_reply: tid=0 2024-03-19T02:41:52.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.265+0000 7f800d217740 1 client.19680 shutdown 2024-03-19T02:41:52.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 msgr2=0x55adea100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4005e00 tx=0x7f7ff4004590 comp rx=0 tx=0).stop 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 shutdown_connections 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55adea178400 0x55adea17a820 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adea0f8880 0x55adea0fc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55adea0efca0 0x55adea100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 --2- 172.21.15.116:0/96976470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adea0ef380 0x55adea100500 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 >> 172.21.15.116:0/96976470 conn(0x55adea0e9ab0 msgr2=0x55adea0e82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f7fe37fe640 0 client.19680 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 shutdown_connections 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 1 -- 172.21.15.116:0/96976470 wait complete. 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800d217740 20 client.19680 trim_cache size 0 max 16384 2024-03-19T02:41:52.269 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:52.270 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:41:52.270 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "756d19d6-c31f-4c2c-99ea-500fba104c1d", 2024-03-19T02:41:52.270 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:41:52.270 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 msgr2=0x7f7fec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 0x7f7fec0a4eb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005fd0 tx=0x7f7ffc004a40 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 msgr2=0x7f8004199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.266+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800d950 tx=0x7f7ff800de10 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 shutdown_connections 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7fec0a29f0 0x7f7fec0a4eb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80041066a0 0x7f80041a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8004101050 0x7f8004199380 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 --2- 172.21.15.116:0/4159738480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8004100690 0x7f8004198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 >> 172.21.15.116:0/4159738480 conn(0x7f8004077820 msgr2=0x7f8004103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 shutdown_connections 2024-03-19T02:41:52.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.267+0000 7f800a630640 1 -- 172.21.15.116:0/4159738480 wait complete. 2024-03-19T02:41:52.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:41:52.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:41:52.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8837 8.99 MB/sec warmup 74 sec latency 155.643 ms 2024-03-19T02:41:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 -- 172.21.15.116:0/2207180789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 msgr2=0x7f8140100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 --2- 172.21.15.116:0/2207180789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140100140 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8134009900 tx=0x7f813402bc30 comp rx=0 tx=0).stop 2024-03-19T02:41:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 -- 172.21.15.116:0/2207180789 shutdown_connections 2024-03-19T02:41:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 --2- 172.21.15.116:0/2207180789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 0x7f814010e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 --2- 172.21.15.116:0/2207180789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140100140 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 --2- 172.21.15.116:0/2207180789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 0x7f8140107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 -- 172.21.15.116:0/2207180789 >> 172.21.15.116:0/2207180789 conn(0x7f8140076c40 msgr2=0x7f81400fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 -- 172.21.15.116:0/2207180789 shutdown_connections 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.397+0000 7f81459cd640 1 -- 172.21.15.116:0/2207180789 wait complete. 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 Processor -- start 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 -- start start 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 0x7f81401984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 0x7f814019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8140111a20 con 0x7f81400ffcc0 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81401118a0 con 0x7f8140100680 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f81459cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8140111ba0 con 0x7f81401077c0 2024-03-19T02:41:52.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f813effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f813effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45876/0 (socket says 172.21.15.116:45876) 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f813effd640 1 -- 172.21.15.116:0/3610780560 learned_addr learned my addr 172.21.15.116:0/3610780560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f813e7fc640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 0x7f81401984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.398+0000 7f813f7fe640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 0x7f814019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 -- 172.21.15.116:0/3610780560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 msgr2=0x7f814019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 0x7f814019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 -- 172.21.15.116:0/3610780560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 msgr2=0x7f81401984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 0x7f81401984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 -- 172.21.15.116:0/3610780560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814019cf70 con 0x7f81400ffcc0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f813effd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f813000c920 tx=0x7f813000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f81449cb640 1 -- 172.21.15.116:0/3610780560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130007c70 con 0x7f81400ffcc0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814019d260 con 0x7f81400ffcc0 2024-03-19T02:41:52.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f81449cb640 1 -- 172.21.15.116:0/3610780560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8130007e10 con 0x7f81400ffcc0 2024-03-19T02:41:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f81449cb640 1 -- 172.21.15.116:0/3610780560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130018820 con 0x7f81400ffcc0 2024-03-19T02:41:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.399+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81401a4af0 con 0x7f81400ffcc0 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.401+0000 7f81449cb640 1 -- 172.21.15.116:0/3610780560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f81300189c0 con 0x7f81400ffcc0 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 Processor -- start 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 -- start start 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81449cb640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 0x7f81280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d58089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d580b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a6d5763500 con 0x55a6d57ffa30 2024-03-19T02:41:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a6d570bbe0 con 0x55a6d5800350 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81485b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a6d575f5a0 con 0x55a6d5808f30 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813f7fe640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 0x7f81280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d58089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d580b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45886/0 (socket says 172.21.15.116:45886) 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 -- 172.21.15.116:0/2660440957 learned_addr learned my addr 172.21.15.116:0/2660440957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f81449cb640 1 -- 172.21.15.116:0/3610780560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f81300cf280 con 0x7f81400ffcc0 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d58089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52740/0 (socket says 172.21.15.116:52740) 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 -- 172.21.15.116:0/2660440957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 msgr2=0x55a6d580b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d580b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 -- 172.21.15.116:0/2660440957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 msgr2=0x55a6d58089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d58089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.402+0000 7f813effd640 1 -- 172.21.15.116:0/2660440957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6d572d000 con 0x55a6d57ffa30 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f813f7fe640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 0x7f81280a4e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f8138009e00 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f813effd640 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f8130097ed0 tx=0x7f81300978b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49e98733ddea6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f8125ffb640 1 -- 172.21.15.116:0/2660440957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130064830 con 0x55a6d57ffa30 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f8125ffb640 1 -- 172.21.15.116:0/2660440957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f813000fb00 con 0x55a6d57ffa30 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f8125ffb640 1 -- 172.21.15.116:0/2660440957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130018cb0 con 0x55a6d57ffa30 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 -- 172.21.15.116:0/2660440957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 msgr2=0x55a6d57ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f8130097ed0 tx=0x7f81300978b0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 -- 172.21.15.116:0/2660440957 shutdown_connections 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d580b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d58089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.403+0000 7f81485b0740 1 --2- 172.21.15.116:0/2660440957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d57ffa30 0x55a6d57ffe10 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 -- 172.21.15.116:0/2660440957 >> 172.21.15.116:0/2660440957 conn(0x55a6d57fa160 msgr2=0x55a6d57fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 -- 172.21.15.116:0/2660440957 shutdown_connections 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 -- 172.21.15.116:0/2660440957 wait complete. 2024-03-19T02:41:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 -- start start 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 Processor -- start 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.404+0000 7f81485b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5806490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a6d580bd20 con 0x55a6d5808f30 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a6d571f160 con 0x55a6d5800350 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a6d580bea0 con 0x55a6d57ffa30 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 10 client.0 _ll_register_callbacks cb 0x55a6d575ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5806490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48348/0 (socket says 172.21.15.116:48348) 2024-03-19T02:41:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813effd640 1 -- 172.21.15.116:0/345246035 learned_addr learned my addr 172.21.15.116:0/345246035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45890/0 (socket says 172.21.15.116:45890) 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5806490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:52750/0 (socket says 172.21.15.116:52750) 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 -- 172.21.15.116:0/345246035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 msgr2=0x55a6d5805f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 -- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 msgr2=0x55a6d5806490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5806490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 -- 172.21.15.116:0/345246035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6d55417d0 con 0x55a6d5808f30 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813effd640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f813f7fe640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f8138009dd0 tx=0x7f813801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e96b6be5716ac5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f811effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138004b20 con 0x55a6d5808f30 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f813801c040 con 0x55a6d5808f30 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.405+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6d5770a40 con 0x55a6d5808f30 2024-03-19T02:41:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8138031050 con 0x55a6d5808f30 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 10 client.19692 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a6d54773f0 con 0x55a6d5808f30 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a6d5770a40 con 0x55a6d5808f30 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f8138028070 con 0x55a6d5808f30 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 10 client.19692 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:52.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 10 client.19692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 20 client.19692 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d5821b40 0x55a6d5823f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 4 client.19692 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6d576ad90 con 0x55a6d5821b40 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.406+0000 7f813effd640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d5821b40 0x55a6d5823f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.407+0000 7f811effd640 10 client.19692 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.407+0000 7f813effd640 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d5821b40 0x55a6d5823f70 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c9faaac7a800790 server_cookie=947f142ff41518a in_seq=0 out_seq=0 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f811effd640 1 -- 172.21.15.116:0/345246035 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a6d576ad90 con 0x55a6d5821b40 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f811effd640 10 client.19692 handle_command_reply: tid=0 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 client.19692 shutdown 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 msgr2=0x55a6d58069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f8138009dd0 tx=0x7f813801d000 comp rx=0 tx=0).stop 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 shutdown_connections 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d5821b40 0x55a6d5823f70 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58069d0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5806490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f811effd640 0 client.19692 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 --2- 172.21.15.116:0/345246035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5805f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 >> 172.21.15.116:0/345246035 conn(0x55a6d57fa160 msgr2=0x55a6d57fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 shutdown_connections 2024-03-19T02:41:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.410+0000 7f81485b0740 1 -- 172.21.15.116:0/345246035 wait complete. 2024-03-19T02:41:52.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.411+0000 7f81485b0740 20 client.19692 trim_cache size 0 max 16384 2024-03-19T02:41:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 Processor -- start 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 -- start start 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d57ffa30 0x55a6d5808680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 0x55a6d581d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ac9d0 con 0x55a6d5800350 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ae600 con 0x55a6d57ffa30 2024-03-19T02:41:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f81485b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ad0a0 con 0x55a6d5808f30 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 0x55a6d581d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 0x55a6d581d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45904/0 (socket says 172.21.15.116:45904) 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48360/0 (socket says 172.21.15.116:48360) 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813e7fc640 1 -- 172.21.15.116:0/2006304230 learned_addr learned my addr 172.21.15.116:0/2006304230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813effd640 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d57ffa30 0x55a6d5808680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 -- 172.21.15.116:0/2006304230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d57ffa30 msgr2=0x55a6d5808680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d57ffa30 0x55a6d5808680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 -- 172.21.15.116:0/2006304230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 msgr2=0x55a6d581d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 0x55a6d581d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.434+0000 7f813f7fe640 1 -- 172.21.15.116:0/2006304230 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6d54370f0 con 0x55a6d5808f30 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.435+0000 7f813f7fe640 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f813800fc20 tx=0x7f8138005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4c8e08ced10084 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.435+0000 7f81257fa640 1 -- 172.21.15.116:0/2006304230 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138039020 con 0x55a6d5808f30 2024-03-19T02:41:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.435+0000 7f81257fa640 1 -- 172.21.15.116:0/2006304230 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8138013970 con 0x55a6d5808f30 2024-03-19T02:41:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.435+0000 7f81257fa640 1 -- 172.21.15.116:0/2006304230 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813802e070 con 0x55a6d5808f30 2024-03-19T02:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- 172.21.15.116:0/2006304230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 msgr2=0x55a6d58050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f813800fc20 tx=0x7f8138005bd0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- 172.21.15.116:0/2006304230 shutdown_connections 2024-03-19T02:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d5808f30 0x55a6d58050b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5800350 0x55a6d581d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 --2- 172.21.15.116:0/2006304230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d57ffa30 0x55a6d5808680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- 172.21.15.116:0/2006304230 >> 172.21.15.116:0/2006304230 conn(0x55a6d57fa160 msgr2=0x55a6d5805f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- 172.21.15.116:0/2006304230 shutdown_connections 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- 172.21.15.116:0/2006304230 wait complete. 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 -- start start 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.436+0000 7f81485b0740 1 Processor -- start 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5814fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ade70 con 0x55a6d5808f30 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ad7a0 con 0x55a6d5800350 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f81401aa9d0 con 0x55a6d57ffa30 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f81485b0740 10 client.0 _ll_register_callbacks cb 0x7f8138009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:41:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48376/0 (socket says 172.21.15.116:48376) 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 -- 172.21.15.116:0/2783307822 learned_addr learned my addr 172.21.15.116:0/2783307822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813e7fc640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5814fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 -- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 msgr2=0x55a6d5814fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5814fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 -- 172.21.15.116:0/2783307822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 msgr2=0x55a6d58154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 -- 172.21.15.116:0/2783307822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6d5815cc0 con 0x55a6d57ffa30 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813f7fe640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f813effd640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f81300cc6a0 tx=0x7f813000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63569dfd89e411eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.437+0000 7f811effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813006d3e0 con 0x55a6d57ffa30 2024-03-19T02:41:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8138013970 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9510+0+0 (secure 0 0 0) 0x7f813009b760 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f813006de50 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 10 client.29416 fetch_fsmap learned FSMap version 33 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a6d5887ed0 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8138013970 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f81300b4150 con 0x55a6d57ffa30 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 10 client.29416 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 10 client.29416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.438+0000 7f81485b0740 20 client.29416 populate_metadata read hostname 'smithi116' 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.439+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d58882d0 0x55a6d588a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.439+0000 7f81485b0740 4 client.29416 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.439+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6d56e8540 con 0x55a6d58882d0 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.439+0000 7f813e7fc640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d58882d0 0x55a6d588a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:41:52.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.441+0000 7f811effd640 10 client.29416 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.441+0000 7f813e7fc640 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d58882d0 0x55a6d588a6c0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=894a6be544bbb790 server_cookie=c258a28708cd67dd in_seq=0 out_seq=0 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.442+0000 7f811effd640 1 -- 172.21.15.116:0/2783307822 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a6d56e8540 con 0x55a6d58882d0 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.442+0000 7f811effd640 10 client.29416 handle_command_reply: tid=0 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.442+0000 7f81485b0740 1 client.29416 shutdown 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 msgr2=0x55a6d5819c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f81300cc6a0 tx=0x7f813000cf90 comp rx=0 tx=0).stop 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 shutdown_connections 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a6d58882d0 0x55a6d588a6c0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6d5808f30 0x55a6d58154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a6d5800350 0x55a6d5814fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f811effd640 0 client.29416 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:41:52.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 --2- 172.21.15.116:0/2783307822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a6d57ffa30 0x55a6d5819c30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 >> 172.21.15.116:0/2783307822 conn(0x55a6d57fa160 msgr2=0x55a6d57fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 shutdown_connections 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 1 -- 172.21.15.116:0/2783307822 wait complete. 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.443+0000 7f81485b0740 20 client.29416 trim_cache size 0 max 16384 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (634 inodes in the stack)", 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "756d19d6-c31f-4c2c-99ea-500fba104c1d": { 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "756d19d6-c31f-4c2c-99ea-500fba104c1d", 2024-03-19T02:41:52.447 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:41:52.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:41:52.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:41:52.448 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 msgr2=0x7f81280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 0x7f81280a4e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f8138009e00 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 msgr2=0x7f8140197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f813000c920 tx=0x7f813000cde0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 shutdown_connections 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f81280a2970 0x7f81280a4e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81401077c0 0x7f814019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8140100680 0x7f81401984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 --2- 172.21.15.116:0/3610780560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81400ffcc0 0x7f8140197f80 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 >> 172.21.15.116:0/3610780560 conn(0x7f8140076c40 msgr2=0x7f81400fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 shutdown_connections 2024-03-19T02:41:52.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:41:52.444+0000 7f81459cd640 1 -- 172.21.15.116:0/3610780560 wait complete. 2024-03-19T02:41:52.458 INFO:tasks.cephfs.filesystem:scrub status for tag:756d19d6-c31f-4c2c-99ea-500fba104c1d - {'path': '/', 'tag': '756d19d6-c31f-4c2c-99ea-500fba104c1d', 'options': 'recursive,force'} 2024-03-19T02:41:52.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:52 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:41:52.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:52 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:52.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:52 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:52.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:52 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:41:52.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:52 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:52.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:52 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:52 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:41:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:52 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:52 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:53.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 8920 8.89 MB/sec warmup 75 sec latency 227.123 ms 2024-03-19T02:41:53.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: pgmap v441: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:41:53.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:53 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: pgmap v441: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:53 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: pgmap v441: 129 pgs: 129 active+clean; 771 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:41:54.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:53 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:41:54.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9007 8.80 MB/sec warmup 76 sec latency 177.516 ms 2024-03-19T02:41:55.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9101 8.76 MB/sec warmup 77 sec latency 199.408 ms 2024-03-19T02:41:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:55 smithi162 ceph-mon[35761]: pgmap v442: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 186 op/s 2024-03-19T02:41:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:55 smithi154 ceph-mon[35693]: pgmap v442: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 186 op/s 2024-03-19T02:41:55.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:55 smithi116 ceph-mon[32263]: pgmap v442: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 186 op/s 2024-03-19T02:41:56.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9203 8.71 MB/sec warmup 78 sec latency 176.981 ms 2024-03-19T02:41:57.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9288 8.63 MB/sec warmup 79 sec latency 151.603 ms 2024-03-19T02:41:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:57 smithi162 ceph-mon[35761]: pgmap v443: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:41:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:57 smithi154 ceph-mon[35693]: pgmap v443: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:41:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:57 smithi116 ceph-mon[32263]: pgmap v443: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:41:58.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9383 8.57 MB/sec warmup 80 sec latency 176.929 ms 2024-03-19T02:41:59.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9485 8.50 MB/sec warmup 81 sec latency 146.425 ms 2024-03-19T02:41:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:41:59 smithi162 ceph-mon[35761]: pgmap v444: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:41:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:41:59 smithi154 ceph-mon[35693]: pgmap v444: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:41:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:41:59 smithi116 ceph-mon[32263]: pgmap v444: 129 pgs: 129 active+clean; 778 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-03-19T02:42:00.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9595 8.43 MB/sec warmup 82 sec latency 187.649 ms 2024-03-19T02:42:01.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9687 8.35 MB/sec warmup 83 sec latency 200.101 ms 2024-03-19T02:42:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:01 smithi162 ceph-mon[35761]: pgmap v445: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 183 op/s 2024-03-19T02:42:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:01 smithi154 ceph-mon[35693]: pgmap v445: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 183 op/s 2024-03-19T02:42:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:01 smithi116 ceph-mon[32263]: pgmap v445: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 183 op/s 2024-03-19T02:42:02.320 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9796 8.30 MB/sec warmup 84 sec latency 150.502 ms 2024-03-19T02:42:03.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 9917 8.32 MB/sec warmup 85 sec latency 141.710 ms 2024-03-19T02:42:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:03 smithi162 ceph-mon[35761]: pgmap v446: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 130 op/s 2024-03-19T02:42:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:03 smithi154 ceph-mon[35693]: pgmap v446: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 130 op/s 2024-03-19T02:42:03.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:03 smithi116 ceph-mon[32263]: pgmap v446: 129 pgs: 129 active+clean; 784 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 130 op/s 2024-03-19T02:42:04.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10040 8.25 MB/sec warmup 86 sec latency 148.124 ms 2024-03-19T02:42:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:04 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:04.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:04 smithi162 ceph-mon[35761]: osdmap e112: 12 total, 12 up, 12 in 2024-03-19T02:42:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:04 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:05.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:04 smithi154 ceph-mon[35693]: osdmap e112: 12 total, 12 up, 12 in 2024-03-19T02:42:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:04 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:04 smithi116 ceph-mon[32263]: osdmap e112: 12 total, 12 up, 12 in 2024-03-19T02:42:05.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10148 8.17 MB/sec warmup 87 sec latency 178.116 ms 2024-03-19T02:42:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:05 smithi162 ceph-mon[35761]: pgmap v448: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 213 op/s 2024-03-19T02:42:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:05 smithi154 ceph-mon[35693]: pgmap v448: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 213 op/s 2024-03-19T02:42:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:05 smithi116 ceph-mon[32263]: pgmap v448: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 213 op/s 2024-03-19T02:42:06.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10288 8.27 MB/sec warmup 88 sec latency 210.516 ms 2024-03-19T02:42:07.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10433 8.33 MB/sec warmup 89 sec latency 143.863 ms 2024-03-19T02:42:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:07 smithi162 ceph-mon[35761]: osdmap e113: 12 total, 12 up, 12 in 2024-03-19T02:42:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:07 smithi162 ceph-mon[35761]: pgmap v450: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-19T02:42:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:07 smithi154 ceph-mon[35693]: osdmap e113: 12 total, 12 up, 12 in 2024-03-19T02:42:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:07 smithi154 ceph-mon[35693]: pgmap v450: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-19T02:42:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:07 smithi116 ceph-mon[32263]: osdmap e113: 12 total, 12 up, 12 in 2024-03-19T02:42:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:07 smithi116 ceph-mon[32263]: pgmap v450: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-19T02:42:08.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10544 8.27 MB/sec warmup 90 sec latency 187.215 ms 2024-03-19T02:42:09.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10649 8.23 MB/sec warmup 91 sec latency 161.473 ms 2024-03-19T02:42:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:09 smithi162 ceph-mon[35761]: pgmap v451: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:09 smithi154 ceph-mon[35693]: pgmap v451: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:09 smithi116 ceph-mon[32263]: pgmap v451: 129 pgs: 129 active+clean; 778 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:10.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10795 8.16 MB/sec warmup 92 sec latency 170.152 ms 2024-03-19T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 10921 8.09 MB/sec warmup 93 sec latency 139.664 ms 2024-03-19T02:42:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:11 smithi162 ceph-mon[35761]: pgmap v452: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-03-19T02:42:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:11 smithi154 ceph-mon[35693]: pgmap v452: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-03-19T02:42:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:11 smithi116 ceph-mon[32263]: pgmap v452: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 296 op/s 2024-03-19T02:42:12.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11050 8.05 MB/sec warmup 94 sec latency 138.123 ms 2024-03-19T02:42:13.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11165 7.98 MB/sec warmup 95 sec latency 161.455 ms 2024-03-19T02:42:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:13 smithi162 ceph-mon[35761]: pgmap v453: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-03-19T02:42:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:13 smithi154 ceph-mon[35693]: pgmap v453: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-03-19T02:42:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:13 smithi116 ceph-mon[32263]: pgmap v453: 129 pgs: 129 active+clean; 787 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 282 op/s 2024-03-19T02:42:14.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11280 7.94 MB/sec warmup 96 sec latency 154.712 ms 2024-03-19T02:42:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:14 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:42:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:14 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:42:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:14 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:42:15.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11420 7.95 MB/sec warmup 97 sec latency 135.304 ms 2024-03-19T02:42:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:42:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:42:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:42:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:15 smithi162 ceph-mon[35761]: pgmap v454: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 202 op/s 2024-03-19T02:42:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:42:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:42:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:42:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:15 smithi154 ceph-mon[35693]: pgmap v454: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 202 op/s 2024-03-19T02:42:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:42:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:42:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:42:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:15 smithi116 ceph-mon[32263]: pgmap v454: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 202 op/s 2024-03-19T02:42:16.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11526 7.89 MB/sec warmup 98 sec latency 144.469 ms 2024-03-19T02:42:17.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11626 7.85 MB/sec warmup 99 sec latency 148.394 ms 2024-03-19T02:42:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:17 smithi162 ceph-mon[35761]: pgmap v455: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-03-19T02:42:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:17 smithi154 ceph-mon[35693]: pgmap v455: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-03-19T02:42:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:17 smithi116 ceph-mon[32263]: pgmap v455: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-03-19T02:42:18.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11762 7.88 MB/sec warmup 100 sec latency 157.733 ms 2024-03-19T02:42:19.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 11882 7.89 MB/sec warmup 101 sec latency 156.979 ms 2024-03-19T02:42:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:19 smithi162 ceph-mon[35761]: pgmap v456: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-19T02:42:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:19 smithi154 ceph-mon[35693]: pgmap v456: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-19T02:42:19.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:19 smithi116 ceph-mon[32263]: pgmap v456: 129 pgs: 129 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-19T02:42:20.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12010 7.98 MB/sec warmup 102 sec latency 259.174 ms 2024-03-19T02:42:21.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12194 8.04 MB/sec warmup 103 sec latency 154.749 ms 2024-03-19T02:42:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:21 smithi162 ceph-mon[35761]: pgmap v457: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:42:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:21 smithi154 ceph-mon[35693]: pgmap v457: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:42:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:21 smithi116 ceph-mon[32263]: pgmap v457: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 233 op/s 2024-03-19T02:42:22.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12384 8.07 MB/sec warmup 104 sec latency 134.157 ms 2024-03-19T02:42:22.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:42:22.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:42:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 -- 172.21.15.116:0/3478409495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b801006e0 msgr2=0x7f8b80100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 --2- 172.21.15.116:0/3478409495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b801006e0 0x7f8b80100b40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009900 tx=0x7f8b7002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 -- 172.21.15.116:0/3478409495 shutdown_connections 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 --2- 172.21.15.116:0/3478409495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b80101080 0x7f8b8010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 --2- 172.21.15.116:0/3478409495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b801006e0 0x7f8b80100b40 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 --2- 172.21.15.116:0/3478409495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801055c0 0x7f8b801059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 -- 172.21.15.116:0/3478409495 >> 172.21.15.116:0/3478409495 conn(0x7f8b800777b0 msgr2=0x7f8b80077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 -- 172.21.15.116:0/3478409495 shutdown_connections 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.584+0000 7f8b8831b640 1 -- 172.21.15.116:0/3478409495 wait complete. 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 Processor -- start 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 -- start start 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801006e0 0x7f8b80197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b801055c0 0x7f8b8019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80110960 con 0x7f8b80101080 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801107e0 con 0x7f8b801006e0 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b8831b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80110ae0 con 0x7f8b801055c0 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33144/0 (socket says 172.21.15.116:33144) 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b858b6640 1 -- 172.21.15.116:0/2096772200 learned_addr learned my addr 172.21.15.116:0/2096772200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b860b7640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801006e0 0x7f8b80197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.585+0000 7f8b868b8640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b801055c0 0x7f8b8019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 -- 172.21.15.116:0/2096772200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b801055c0 msgr2=0x7f8b8019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b801055c0 0x7f8b8019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 -- 172.21.15.116:0/2096772200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801006e0 msgr2=0x7f8b80197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801006e0 0x7f8b80197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 -- 172.21.15.116:0/2096772200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8019cf70 con 0x7f8b80101080 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b858b6640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f8b70005940 tx=0x7f8b7002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b777fe640 1 -- 172.21.15.116:0/2096772200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70043020 con 0x7f8b80101080 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b777fe640 1 -- 172.21.15.116:0/2096772200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b70004e00 con 0x7f8b80101080 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b777fe640 1 -- 172.21.15.116:0/2096772200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70003970 con 0x7f8b80101080 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8019d200 con 0x7f8b80101080 2024-03-19T02:42:22.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.586+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b801a4af0 con 0x7f8b80101080 2024-03-19T02:42:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.588+0000 7f8b777fe640 1 -- 172.21.15.116:0/2096772200 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8b70046020 con 0x7f8b80101080 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 Processor -- start 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 -- start start 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb5ba30 0x56233cb5be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b777fe640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 0x7f8b6c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb67320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56233cabf500 con 0x56233cb64f30 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56233ca67be0 con 0x56233cb5ba30 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56233cabb5a0 con 0x56233cb5c350 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b868b8640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 0x7f8b6c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58264/0 (socket says 172.21.15.116:58264) 2024-03-19T02:42:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 -- 172.21.15.116:0/4185865247 learned_addr learned my addr 172.21.15.116:0/4185865247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.589+0000 7f8b860b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb5ba30 0x56233cb5be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b868b8640 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb67320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b777fe640 1 -- 172.21.15.116:0/2096772200 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f8b700f10b0 con 0x7f8b80101080 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 -- 172.21.15.116:0/4185865247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb5ba30 msgr2=0x56233cb5be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb5ba30 0x56233cb5be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 -- 172.21.15.116:0/4185865247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 msgr2=0x56233cb67320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb67320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 -- 172.21.15.116:0/4185865247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56233ca89000 con 0x56233cb5c350 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b868b8640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 0x7f8b6c0a4e30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b78005fd0 tx=0x7f8b78009f10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b858b6640 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8b700c56b0 tx=0x7f8b70009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb90e43f3cc4301 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b74bfe640 1 -- 172.21.15.116:0/4185865247 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b700bdb40 con 0x56233cb5c350 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b74bfe640 1 -- 172.21.15.116:0/4185865247 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b700c6aa0 con 0x56233cb5c350 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.590+0000 7f8b74bfe640 1 -- 172.21.15.116:0/4185865247 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7002e040 con 0x56233cb5c350 2024-03-19T02:42:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 -- 172.21.15.116:0/4185865247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 msgr2=0x56233cb649f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8b700c56b0 tx=0x7f8b70009680 comp rx=0 tx=0).stop 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 -- 172.21.15.116:0/4185865247 shutdown_connections 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb67320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb649f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/4185865247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb5ba30 0x56233cb5be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 -- 172.21.15.116:0/4185865247 >> 172.21.15.116:0/4185865247 conn(0x56233cb56160 msgr2=0x56233cb56540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 -- 172.21.15.116:0/4185865247 shutdown_connections 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.591+0000 7f8b8afe1740 1 -- 172.21.15.116:0/4185865247 wait complete. 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 -- start start 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 Processor -- start 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb62920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56233cb67ae0 con 0x56233cb5ba30 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56233ca7b160 con 0x56233cb64f30 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56233cb67c60 con 0x56233cb5c350 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b8afe1740 10 client.0 _ll_register_callbacks cb 0x56233cab7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33154/0 (socket says 172.21.15.116:33154) 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 -- 172.21.15.116:0/3638509989 learned_addr learned my addr 172.21.15.116:0/3638509989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b868b8640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb62920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 -- 172.21.15.116:0/3638509989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 msgr2=0x56233cb623e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb623e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 -- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 msgr2=0x56233cb62920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb62920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.592+0000 7f8b860b7640 1 -- 172.21.15.116:0/3638509989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56233c89d7d0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c012740 tx=0x7f8b7c012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee6104d88d9b66e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00f920 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b700c6aa0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f8b7c00fac0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b7c00fcc0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 10 client.19710 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56233c7d33f0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b700c6aa0 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f8b7c01c420 con 0x56233cb5ba30 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 10 client.19710 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 10 client.19710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:22.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.593+0000 7f8b8afe1740 20 client.19710 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cb75090 0x56233cb77530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b8afe1740 4 client.19710 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56233cacca40 con 0x56233cb75090 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cb75090 0x56233cb77530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b69ffb640 10 client.19710 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:22.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.594+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cb75090 0x56233cb77530 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259b7508ebd4197a server_cookie=f4533be537bc1e2 in_seq=0 out_seq=0 2024-03-19T02:42:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3638509989 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56233cacca40 con 0x56233cb75090 2024-03-19T02:42:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b69ffb640 10 client.19710 handle_command_reply: tid=0 2024-03-19T02:42:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 client.19710 shutdown 2024-03-19T02:42:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 msgr2=0x56233cb61ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c012740 tx=0x7f8b7c012c00 comp rx=0 tx=0).stop 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 shutdown_connections 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cb75090 0x56233cb77530 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb62920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb623e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b69ffb640 0 client.19710 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3638509989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb61ea0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 >> 172.21.15.116:0/3638509989 conn(0x56233cb56160 msgr2=0x56233cb57eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 shutdown_connections 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3638509989 wait complete. 2024-03-19T02:42:22.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.598+0000 7f8b8afe1740 20 client.19710 trim_cache size 0 max 16384 2024-03-19T02:42:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.623+0000 7f8b8afe1740 1 Processor -- start 2024-03-19T02:42:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.623+0000 7f8b8afe1740 1 -- start start 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb60d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801aec70 con 0x56233cb5ba30 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801b08a0 con 0x56233cb64f30 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801af340 con 0x56233cb5c350 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33162/0 (socket says 172.21.15.116:33162) 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 -- 172.21.15.116:0/2703584314 learned_addr learned my addr 172.21.15.116:0/2703584314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b868b8640 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb60d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b858b6640 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 -- 172.21.15.116:0/2703584314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 msgr2=0x56233cb707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 -- 172.21.15.116:0/2703584314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 msgr2=0x56233cb60d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb60d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 -- 172.21.15.116:0/2703584314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56233c7930f0 con 0x56233cb5ba30 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.624+0000 7f8b860b7640 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009f90 tx=0x7f8b7c009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c3fe49f7cb2a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.625+0000 7f8b6bfff640 1 -- 172.21.15.116:0/2703584314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00f920 con 0x56233cb5ba30 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.625+0000 7f8b6bfff640 1 -- 172.21.15.116:0/2703584314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b7c012e50 con 0x56233cb5ba30 2024-03-19T02:42:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.625+0000 7f8b6bfff640 1 -- 172.21.15.116:0/2703584314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c005350 con 0x56233cb5ba30 2024-03-19T02:42:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- 172.21.15.116:0/2703584314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 msgr2=0x56233cb645d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009f90 tx=0x7f8b7c009fc0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- 172.21.15.116:0/2703584314 shutdown_connections 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cb64f30 0x56233cb60d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/2703584314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb5ba30 0x56233cb645d0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- 172.21.15.116:0/2703584314 >> 172.21.15.116:0/2703584314 conn(0x56233cb56160 msgr2=0x56233cb60640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- 172.21.15.116:0/2703584314 shutdown_connections 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- 172.21.15.116:0/2703584314 wait complete. 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 -- start start 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.626+0000 7f8b8afe1740 1 Processor -- start 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb5f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cbd8420 0x56233cbda810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801b0110 con 0x56233cb64f30 2024-03-19T02:42:22.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801afa40 con 0x56233cbd8420 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b801acc70 con 0x56233cb5c350 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b8afe1740 10 client.0 _ll_register_callbacks cb 0x7f8b7c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33172/0 (socket says 172.21.15.116:33172) 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 -- 172.21.15.116:0/3764500803 learned_addr learned my addr 172.21.15.116:0/3764500803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb5f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 -- 172.21.15.116:0/3764500803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 msgr2=0x56233cb5f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb5f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 -- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cbd8420 msgr2=0x56233cbda810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cbd8420 0x56233cbda810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.627+0000 7f8b858b6640 1 -- 172.21.15.116:0/3764500803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56233cbdad50 con 0x56233cb64f30 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b858b6640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8b700eea20 tx=0x7f8b700b9810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0365d3c4170b186 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b700ca7e0 con 0x56233cb64f30 2024-03-19T02:42:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f8b700b9ab0 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b7c012e50 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8b700c1590 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b8afe1740 10 client.19716 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.628+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56233cbdbfd0 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b7c012e50 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f8b700c1180 con 0x56233cb64f30 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 10 client.19716 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 10 client.19716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 20 client.19716 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cbdc600 0x56233cbdea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 4 client.19716 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56233cac6d90 con 0x56233cbdc600 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.629+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cbdc600 0x56233cbdea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.630+0000 7f8b69ffb640 10 client.19716 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.630+0000 7f8b860b7640 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cbdc600 0x56233cbdea20 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb340d7a602c9b9 server_cookie=198001445fa6c71c in_seq=0 out_seq=0 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.633+0000 7f8b69ffb640 1 -- 172.21.15.116:0/3764500803 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56233cac6d90 con 0x56233cbdc600 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.633+0000 7f8b69ffb640 10 client.19716 handle_command_reply: tid=0 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.633+0000 7f8b8afe1740 1 client.19716 shutdown 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 msgr2=0x56233cb5f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8b700eea20 tx=0x7f8b700b9810 comp rx=0 tx=0).stop 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 shutdown_connections 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56233cbdc600 0x56233cbdea20 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56233cbd8420 0x56233cbda810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56233cb64f30 0x56233cb5f670 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b69ffb640 0 client.19716 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 --2- 172.21.15.116:0/3764500803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56233cb5c350 0x56233cb5f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 >> 172.21.15.116:0/3764500803 conn(0x56233cb56160 msgr2=0x56233cb60ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 shutdown_connections 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 1 -- 172.21.15.116:0/3764500803 wait complete. 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.634+0000 7f8b8afe1740 20 client.19716 trim_cache size 0 max 16384 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:42:22.636 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:42:22.637 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 msgr2=0x7f8b6c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 0x7f8b6c0a4e30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b78005fd0 tx=0x7f8b78009f10 comp rx=0 tx=0).stop 2024-03-19T02:42:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 msgr2=0x7f8b801984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f8b70005940 tx=0x7f8b7002c690 comp rx=0 tx=0).stop 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 shutdown_connections 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8b6c0a2970 0x7f8b6c0a4e30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b801055c0 0x7f8b8019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80101080 0x7f8b801984c0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 --2- 172.21.15.116:0/2096772200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b801006e0 0x7f8b80197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 >> 172.21.15.116:0/2096772200 conn(0x7f8b800777b0 msgr2=0x7f8b80104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 shutdown_connections 2024-03-19T02:42:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.635+0000 7f8b8831b640 1 -- 172.21.15.116:0/2096772200 wait complete. 2024-03-19T02:42:22.648 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:42:22.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 -- 172.21.15.116:0/1944410333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 msgr2=0x7f22ac10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 --2- 172.21.15.116:0/1944410333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac10e680 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f22a4009cd0 tx=0x7f22a402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 -- 172.21.15.116:0/1944410333 shutdown_connections 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 --2- 172.21.15.116:0/1944410333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac10e680 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 --2- 172.21.15.116:0/1944410333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1017b0 0x7f22ac101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 --2- 172.21.15.116:0/1944410333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1077c0 0x7f22ac107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 -- 172.21.15.116:0/1944410333 >> 172.21.15.116:0/1944410333 conn(0x7f22ac078c20 msgr2=0x7f22ac079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 -- 172.21.15.116:0/1944410333 shutdown_connections 2024-03-19T02:42:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.769+0000 7f22b084b640 1 -- 172.21.15.116:0/1944410333 wait complete. 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 Processor -- start 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 -- start start 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 0x7f22ac073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1077c0 0x7f22ac0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.770+0000 7f22b084b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac111a40 con 0x7f22ac102170 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22b084b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac1118c0 con 0x7f22ac1017b0 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22b084b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac111bc0 con 0x7f22ac1077c0 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22aa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 0x7f22ac073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22aa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 0x7f22ac073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45330/0 (socket says 172.21.15.116:45330) 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33192/0 (socket says 172.21.15.116:33192) 2024-03-19T02:42:22.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22aa59c640 1 -- 172.21.15.116:0/1791341331 learned_addr learned my addr 172.21.15.116:0/1791341331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22aad9d640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1077c0 0x7f22ac0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 -- 172.21.15.116:0/1791341331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1077c0 msgr2=0x7f22ac0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1077c0 0x7f22ac0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 -- 172.21.15.116:0/1791341331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 msgr2=0x7f22ac073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 0x7f22ac073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 -- 172.21.15.116:0/1791341331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ac1a6440 con 0x7f22ac102170 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a3fff640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f229400e9f0 tx=0x7f229400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229400ccc0 con 0x7f22ac102170 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22940108e0 con 0x7f22ac102170 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ac1a6730 con 0x7f22ac102170 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294018880 con 0x7f22ac102170 2024-03-19T02:42:22.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.771+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ac1a6c80 con 0x7f22ac102170 2024-03-19T02:42:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.773+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22ac067d80 con 0x7f22ac102170 2024-03-19T02:42:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.774+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2294010a90 con 0x7f22ac102170 2024-03-19T02:42:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.774+0000 7f22a37fe640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:22.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.775+0000 7f22aa59c640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:22.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.775+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f22940cf4f0 con 0x7f22ac102170 2024-03-19T02:42:22.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.775+0000 7f22aa59c640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 0x7f22980a4e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f229c009c10 tx=0x7f229c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:22.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:22.780+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f229409b8c0 con 0x7f22ac102170 2024-03-19T02:42:23.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.069+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22ac109d40 con 0x7f22ac102170 2024-03-19T02:42:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.069+0000 7f22a37fe640 1 -- 172.21.15.116:0/1791341331 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11958 (secure 0 0 0) 0x7f229409f960 con 0x7f22ac102170 2024-03-19T02:42:23.070 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:42:23.070 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:40:44.485117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:42:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 msgr2=0x7f22980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 0x7f22980a4e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f229c009c10 tx=0x7f229c005f90 comp rx=0 tx=0).stop 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 msgr2=0x7f22ac074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f229400e9f0 tx=0x7f229400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 shutdown_connections 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22ac1077c0 0x7f22ac0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22ac102170 0x7f22ac074060 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 --2- 172.21.15.116:0/1791341331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22ac1017b0 0x7f22ac073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 >> 172.21.15.116:0/1791341331 conn(0x7f22ac078c20 msgr2=0x7f22ac104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 shutdown_connections 2024-03-19T02:42:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.072+0000 7f22b084b640 1 -- 172.21.15.116:0/1791341331 wait complete. 2024-03-19T02:42:23.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:42:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 -- 172.21.15.116:0/1601095228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441066b0 msgr2=0x7fc144106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 --2- 172.21.15.116:0/1601095228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441066b0 0x7fc144106a90 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc134009930 tx=0x7fc13402bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 -- 172.21.15.116:0/1601095228 shutdown_connections 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 --2- 172.21.15.116:0/1601095228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc14410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 --2- 172.21.15.116:0/1601095228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441006a0 0x7fc144100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 --2- 172.21.15.116:0/1601095228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441066b0 0x7fc144106a90 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 -- 172.21.15.116:0/1601095228 >> 172.21.15.116:0/1601095228 conn(0x7fc1440778d0 msgr2=0x7fc144077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 -- 172.21.15.116:0/1601095228 shutdown_connections 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.212+0000 7fc149adc640 1 -- 172.21.15.116:0/1601095228 wait complete. 2024-03-19T02:42:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.213+0000 7fc149adc640 1 Processor -- start 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.213+0000 7fc149adc640 1 -- start start 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.213+0000 7fc149adc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.213+0000 7fc149adc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc144199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.213+0000 7fc149adc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441066b0 0x7fc1441a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc149adc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc144110970 con 0x7fc1441006a0 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc149adc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441107f0 con 0x7fc144101060 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc149adc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc144110af0 con 0x7fc1441066b0 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33206/0 (socket says 172.21.15.116:33206) 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc1437fe640 1 -- 172.21.15.116:0/1851699699 learned_addr learned my addr 172.21.15.116:0/1851699699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc144199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc143fff640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441066b0 0x7fc1441a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 -- 172.21.15.116:0/1851699699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441066b0 msgr2=0x7fc1441a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441066b0 0x7fc1441a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 -- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 msgr2=0x7fc144198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 -- 172.21.15.116:0/1851699699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1441a43f0 con 0x7fc144101060 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc144199370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9f0 tx=0x7fc12c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc140ff9640 1 -- 172.21.15.116:0/1851699699 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00ccc0 con 0x7fc144101060 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1441a45c0 con 0x7fc144101060 2024-03-19T02:42:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc140ff9640 1 -- 172.21.15.116:0/1851699699 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc12c005020 con 0x7fc144101060 2024-03-19T02:42:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc140ff9640 1 -- 172.21.15.116:0/1851699699 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c004a00 con 0x7fc144101060 2024-03-19T02:42:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.214+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1441a4ab0 con 0x7fc144101060 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 Processor -- start 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 -- start start 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 0x561d61254290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61256bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc140ff9640 1 -- 172.21.15.116:0/1851699699 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc12c004ba0 con 0x7fc144101060 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561d611ae4b0 con 0x561d6124b2d0 2024-03-19T02:42:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d61156d70 con 0x561d612547d0 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d611aa550 con 0x561d6124bbf0 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 0x561d61254290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 0x561d61254290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58342/0 (socket says 172.21.15.116:58342) 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc1437fe640 1 -- 172.21.15.116:0/843950024 learned_addr learned my addr 172.21.15.116:0/843950024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc143fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33214/0 (socket says 172.21.15.116:33214) 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.217+0000 7fc142ffd640 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61256bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 -- 172.21.15.116:0/843950024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 msgr2=0x561d61254290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 0x561d61254290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 -- 172.21.15.116:0/843950024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 msgr2=0x561d61256bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61256bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 -- 172.21.15.116:0/843950024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d61179210 con 0x561d6124b2d0 2024-03-19T02:42:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc143fff640 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009f00 tx=0x7fc13c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273cc667f1193d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc140ff9640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 0x7fc1300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc139ffb640 1 -- 172.21.15.116:0/843950024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c004d20 con 0x561d6124b2d0 2024-03-19T02:42:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 0x7fc1300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc140ff9640 1 -- 172.21.15.116:0/1851699699 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc12c0cf1b0 con 0x7fc144101060 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc139ffb640 1 -- 172.21.15.116:0/843950024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc13c004050 con 0x561d6124b2d0 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.218+0000 7fc139ffb640 1 -- 172.21.15.116:0/843950024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c005280 con 0x561d6124b2d0 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- 172.21.15.116:0/843950024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 msgr2=0x561d6124b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 0x7fc1300a4e30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1340096c0 tx=0x7fc1340058d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009f00 tx=0x7fc13c011810 comp rx=0 tx=0).stop 2024-03-19T02:42:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- 172.21.15.116:0/843950024 shutdown_connections 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61256bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124bbf0 0x561d61254290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/843950024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d6124b6b0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- 172.21.15.116:0/843950024 >> 172.21.15.116:0/843950024 conn(0x561d61245a00 msgr2=0x561d61245de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- 172.21.15.116:0/843950024 shutdown_connections 2024-03-19T02:42:23.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- 172.21.15.116:0/843950024 wait complete. 2024-03-19T02:42:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 -- start start 2024-03-19T02:42:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.219+0000 7fc14c7a4740 1 Processor -- start 2024-03-19T02:42:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124b2d0 0x561d61251820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d612522a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561d612575c0 con 0x561d6124bbf0 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d6116a310 con 0x561d6124b2d0 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d61257740 con 0x561d612547d0 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc14c7a4740 10 client.0 _ll_register_callbacks cb 0x561d6115e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33220/0 (socket says 172.21.15.116:33220) 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124b2d0 0x561d61251820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 -- 172.21.15.116:0/1241528 learned_addr learned my addr 172.21.15.116:0/1241528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc143fff640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d612522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 -- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 msgr2=0x561d612522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d612522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 -- 172.21.15.116:0/1241528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124b2d0 msgr2=0x561d61251820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124b2d0 0x561d61251820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 -- 172.21.15.116:0/1241528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d60ec23f0 con 0x561d6124bbf0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc12c09b3c0 tx=0x7fc12c0096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312f33bb5f1e9c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.220+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c01c070 con 0x561d6124bbf0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc12c097aa0 con 0x561d6124bbf0 2024-03-19T02:42:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc13c004050 con 0x561d6124bbf0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc12c09af70 con 0x561d6124bbf0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 10 client.19734 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561d60f8c7d0 con 0x561d6124bbf0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc13c004050 con 0x561d6124bbf0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc12c09f180 con 0x561d6124bbf0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 10 client.19734 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 10 client.19734 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 20 client.19734 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d61266f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 4 client.19734 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d6120c3e0 con 0x561d61264b70 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.221+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d61266f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.222+0000 7fc122ffd640 10 client.19734 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.222+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d61266f60 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be44acfaa46141ea server_cookie=681322486014b575 in_seq=0 out_seq=0 2024-03-19T02:42:23.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc122ffd640 1 -- 172.21.15.116:0/1241528 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d6120c3e0 con 0x561d61264b70 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc122ffd640 10 client.19734 handle_command_reply: tid=0 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 client.19734 shutdown 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 msgr2=0x561d61251d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fc12c09b3c0 tx=0x7fc12c0096e0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 shutdown_connections 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d61266f60 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc122ffd640 0 client.19734 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d612522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61251d60 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1241528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124b2d0 0x561d61251820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 >> 172.21.15.116:0/1241528 conn(0x561d61245a00 msgr2=0x561d61247750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 shutdown_connections 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1241528 wait complete. 2024-03-19T02:42:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.223+0000 7fc14c7a4740 20 client.19734 trim_cache size 0 max 16384 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 Processor -- start 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 -- start start 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124bbf0 0x561d61260360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d6126c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441aebe0 con 0x561d6124b2d0 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441b0810 con 0x561d6124bbf0 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441af2b0 con 0x561d612547d0 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33228/0 (socket says 172.21.15.116:33228) 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 -- 172.21.15.116:0/1258381710 learned_addr learned my addr 172.21.15.116:0/1258381710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc142ffd640 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124bbf0 0x561d61260360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc143fff640 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d6126c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 -- 172.21.15.116:0/1258381710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 msgr2=0x561d6126c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d6126c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 -- 172.21.15.116:0/1258381710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124bbf0 msgr2=0x561d61260360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124bbf0 0x561d61260360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.242+0000 7fc1437fe640 1 -- 172.21.15.116:0/1258381710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d61256bc0 con 0x561d6124b2d0 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.243+0000 7fc1437fe640 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc134005940 tx=0x7fc134042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7e35f51c18995f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.243+0000 7fc1397fa640 1 -- 172.21.15.116:0/1258381710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134046070 con 0x561d6124b2d0 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.243+0000 7fc1397fa640 1 -- 172.21.15.116:0/1258381710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc13402cae0 con 0x561d6124b2d0 2024-03-19T02:42:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.243+0000 7fc1397fa640 1 -- 172.21.15.116:0/1258381710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134003970 con 0x561d6124b2d0 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1258381710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 msgr2=0x561d61244ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc134005940 tx=0x7fc134042000 comp rx=0 tx=0).stop 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1258381710 shutdown_connections 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d612547d0 0x561d6126c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d6124bbf0 0x561d61260360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/1258381710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124b2d0 0x561d61244ba0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1258381710 >> 172.21.15.116:0/1258381710 conn(0x561d61245a00 msgr2=0x561d612517e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1258381710 shutdown_connections 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- 172.21.15.116:0/1258381710 wait complete. 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.244+0000 7fc14c7a4740 1 -- start start 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 Processor -- start 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 0x561d6125cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61259ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441b0080 con 0x561d6124bbf0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441af9b0 con 0x561d612547d0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441acbe0 con 0x561d6124b2d0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc14c7a4740 10 client.0 _ll_register_callbacks cb 0x7fc12c0d29f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc1437fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 0x561d6125cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33240/0 (socket says 172.21.15.116:33240) 2024-03-19T02:42:23.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 -- 172.21.15.116:0/4070028986 learned_addr learned my addr 172.21.15.116:0/4070028986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc143fff640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61259ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 -- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 msgr2=0x561d6125cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 0x561d6125cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 -- 172.21.15.116:0/4070028986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 msgr2=0x561d61259ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61259ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 -- 172.21.15.116:0/4070028986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d6125a4b0 con 0x561d6124bbf0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc1437fe640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 0x561d6125cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc143fff640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61259ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc142ffd640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cf180 tx=0x7fc12c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da30ce570bb59a72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.245+0000 7fc122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c01c040 con 0x561d6124bbf0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc12c0bb0d0 con 0x561d6124bbf0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc13402cae0 con 0x561d6124bbf0 2024-03-19T02:42:23.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc12c09ad90 con 0x561d6124bbf0 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 10 client.19740 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561d612cbb10 con 0x561d6124bbf0 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc13402cae0 con 0x561d6124bbf0 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc12c097180 con 0x561d6124bbf0 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 10 client.19740 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 10 client.19740 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 20 client.19740 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d612ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 4 client.19740 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d611bb9f0 con 0x561d61264b70 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.246+0000 7fc1437fe640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d612ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc122ffd640 10 client.19740 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc1437fe640 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d612ce000 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6c6d44a141f0aa8 server_cookie=bb089dd0f9425e74 in_seq=0 out_seq=0 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc122ffd640 1 -- 172.21.15.116:0/4070028986 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d611bb9f0 con 0x561d61264b70 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc122ffd640 10 client.19740 handle_command_reply: tid=0 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 client.19740 shutdown 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 msgr2=0x561d61259960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cf180 tx=0x7fc12c005bd0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 shutdown_connections 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x561d61264b70 0x561d612ce000 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d612547d0 0x561d61259ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d6124bbf0 0x561d61259960 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.247+0000 7fc14c7a4740 1 --2- 172.21.15.116:0/4070028986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d6124b2d0 0x561d6125cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.248+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 >> 172.21.15.116:0/4070028986 conn(0x561d61245a00 msgr2=0x561d61244a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.248+0000 7fc122ffd640 0 client.19740 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.248+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 shutdown_connections 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.248+0000 7fc14c7a4740 1 -- 172.21.15.116:0/4070028986 wait complete. 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.248+0000 7fc14c7a4740 20 client.19740 trim_cache size 0 max 16384 2024-03-19T02:42:23.255 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 msgr2=0x7fc1300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 0x7fc1300a4e30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1340096c0 tx=0x7fc1340058d0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 msgr2=0x7fc144199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc144199370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9f0 tx=0x7fc12c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 shutdown_connections 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc1300a2970 0x7fc1300a4e30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1441066b0 0x7fc1441a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144101060 0x7fc144199370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 --2- 172.21.15.116:0/1851699699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1441006a0 0x7fc144198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 >> 172.21.15.116:0/1851699699 conn(0x7fc1440778d0 msgr2=0x7fc144102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 shutdown_connections 2024-03-19T02:42:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.249+0000 7fc149adc640 1 -- 172.21.15.116:0/1851699699 wait complete. 2024-03-19T02:42:23.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:42:23.290 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:23 smithi116 ceph-mon[32263]: pgmap v458: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-03-19T02:42:23.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:23 smithi154 ceph-mon[35693]: pgmap v458: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-03-19T02:42:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12513 8.07 MB/sec warmup 105 sec latency 136.960 ms 2024-03-19T02:42:23.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 -- 172.21.15.116:0/2725338705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c1066b0 msgr2=0x7fc84c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 --2- 172.21.15.116:0/2725338705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c1066b0 0x7fc84c106a90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc848009930 tx=0x7fc84802bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 -- 172.21.15.116:0/2725338705 shutdown_connections 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 --2- 172.21.15.116:0/2725338705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c101060 0x7fc84c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 --2- 172.21.15.116:0/2725338705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1006a0 0x7fc84c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 --2- 172.21.15.116:0/2725338705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c1066b0 0x7fc84c106a90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 -- 172.21.15.116:0/2725338705 >> 172.21.15.116:0/2725338705 conn(0x7fc84c0778d0 msgr2=0x7fc84c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 -- 172.21.15.116:0/2725338705 shutdown_connections 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.386+0000 7fc8546c6640 1 -- 172.21.15.116:0/2725338705 wait complete. 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 Processor -- start 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 -- start start 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c1006a0 0x7fc84c198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1066b0 0x7fc84c1a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c110970 con 0x7fc84c101060 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1107f0 con 0x7fc84c1066b0 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc8546c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c110af0 con 0x7fc84c1006a0 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc851c61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc851c61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33250/0 (socket says 172.21.15.116:33250) 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc851c61640 1 -- 172.21.15.116:0/3042055524 learned_addr learned my addr 172.21.15.116:0/3042055524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc852462640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c1006a0 0x7fc84c198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.387+0000 7fc852c63640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1066b0 0x7fc84c1a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 -- 172.21.15.116:0/3042055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c1006a0 msgr2=0x7fc84c198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c1006a0 0x7fc84c198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 -- 172.21.15.116:0/3042055524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1066b0 msgr2=0x7fc84c1a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1066b0 0x7fc84c1a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 -- 172.21.15.116:0/3042055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c1a4390 con 0x7fc84c101060 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc851c61640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00d950 tx=0x7fc83c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc8437fe640 1 -- 172.21.15.116:0/3042055524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c017030 con 0x7fc84c101060 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc8437fe640 1 -- 172.21.15.116:0/3042055524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc83c004020 con 0x7fc84c101060 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c1a4560 con 0x7fc84c101060 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc8437fe640 1 -- 172.21.15.116:0/3042055524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0047a0 con 0x7fc84c101060 2024-03-19T02:42:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.388+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1a4ab0 con 0x7fc84c101060 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc8437fe640 1 -- 172.21.15.116:0/3042055524 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc83c0041d0 con 0x7fc84c101060 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 Processor -- start 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 -- start start 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa920 0x55608b7076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b707be0 0x55608b709fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55608b65d4b0 con 0x55608b6fa000 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55608b605d70 con 0x55608b6fa920 2024-03-19T02:42:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc85738d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55608b659550 con 0x55608b707be0 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc852c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc852c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33252/0 (socket says 172.21.15.116:33252) 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc852c63640 1 -- 172.21.15.116:0/3974805806 learned_addr learned my addr 172.21.15.116:0/3974805806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc851c61640 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b707be0 0x55608b709fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.391+0000 7fc852462640 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa920 0x55608b7076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc8437fe640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 0x7fc8380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 -- 172.21.15.116:0/3974805806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b707be0 msgr2=0x55608b709fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b707be0 0x55608b709fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 -- 172.21.15.116:0/3974805806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa920 msgr2=0x55608b7076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa920 0x55608b7076a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 -- 172.21.15.116:0/3974805806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55608b628210 con 0x55608b6fa000 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc8437fe640 1 -- 172.21.15.116:0/3042055524 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc83c0cec40 con 0x7fc84c101060 2024-03-19T02:42:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fc844009f50 tx=0x7fc8440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebe640bf03bc2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc852c63640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 0x7fc8380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc840ff9640 1 -- 172.21.15.116:0/3974805806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84401e070 con 0x55608b6fa000 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc840ff9640 1 -- 172.21.15.116:0/3974805806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc84400fc30 con 0x55608b6fa000 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.392+0000 7fc840ff9640 1 -- 172.21.15.116:0/3974805806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84400e3b0 con 0x55608b6fa000 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc852c63640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 0x7fc8380a4e30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc84401b000 tx=0x7fc84400ea60 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 -- 172.21.15.116:0/3974805806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 msgr2=0x55608b6fa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fc844009f50 tx=0x7fc8440118d0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 -- 172.21.15.116:0/3974805806 shutdown_connections 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b707be0 0x55608b709fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa920 0x55608b7076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 --2- 172.21.15.116:0/3974805806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b6fa3e0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 -- 172.21.15.116:0/3974805806 >> 172.21.15.116:0/3974805806 conn(0x55608b6f4a00 msgr2=0x55608b6f4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 -- 172.21.15.116:0/3974805806 shutdown_connections 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.393+0000 7fc85738d740 1 -- 172.21.15.116:0/3974805806 wait complete. 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 -- start start 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 Processor -- start 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b704bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b7050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55608b70a9a0 con 0x55608b707be0 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55608b619310 con 0x55608b6fa000 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55608b70ab20 con 0x55608b6fa920 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc85738d740 10 client.0 _ll_register_callbacks cb 0x55608b60d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33262/0 (socket says 172.21.15.116:33262) 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc851c61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b7050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 -- 172.21.15.116:0/3517696059 learned_addr learned my addr 172.21.15.116:0/3517696059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852462640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b704bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 -- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 msgr2=0x55608b7050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b7050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 -- 172.21.15.116:0/3517696059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 msgr2=0x55608b704bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b704bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc852c63640 1 -- 172.21.15.116:0/3517696059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55608b3713f0 con 0x55608b707be0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.394+0000 7fc851c61640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b7050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc852c63640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fc844019af0 tx=0x7fc844019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347b62db4f419f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84401e040 con 0x55608b707be0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc844003a30 con 0x55608b707be0 2024-03-19T02:42:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc84400fc30 con 0x55608b707be0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc84403d050 con 0x55608b707be0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 10 client.19752 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55608b43b7d0 con 0x55608b707be0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc84400fc30 con 0x55608b707be0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc844004500 con 0x55608b707be0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 10 client.19752 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 10 client.19752 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 20 client.19752 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b717eb0 0x55608b71a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 4 client.19752 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.395+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55608b6bb3e0 con 0x55608b717eb0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.396+0000 7fc852462640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b717eb0 0x55608b71a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.396+0000 7fc852462640 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b717eb0 0x55608b71a2a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b859d447419abd75 server_cookie=b74d0414e55cb648 in_seq=0 out_seq=0 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.396+0000 7fc835ffb640 10 client.19752 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc835ffb640 1 -- 172.21.15.116:0/3517696059 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55608b6bb3e0 con 0x55608b717eb0 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc835ffb640 10 client.19752 handle_command_reply: tid=0 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 client.19752 shutdown 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 msgr2=0x55608b705630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fc844019af0 tx=0x7fc844019000 comp rx=0 tx=0).stop 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 shutdown_connections 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b717eb0 0x55608b71a2a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b705630 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc835ffb640 0 client.19752 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b7050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 --2- 172.21.15.116:0/3517696059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b704bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 >> 172.21.15.116:0/3517696059 conn(0x55608b6f4a00 msgr2=0x55608b6f6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 shutdown_connections 2024-03-19T02:42:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 1 -- 172.21.15.116:0/3517696059 wait complete. 2024-03-19T02:42:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.397+0000 7fc85738d740 20 client.19752 trim_cache size 0 max 16384 2024-03-19T02:42:23.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:23 smithi162 ceph-mon[35761]: pgmap v458: 129 pgs: 129 active+clean; 804 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 Processor -- start 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 -- start start 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b713780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b707be0 0x55608b6f3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1acab0 con 0x55608b6fa000 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ae6e0 con 0x55608b707be0 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc85738d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad180 con 0x55608b6fa920 2024-03-19T02:42:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc852462640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc852462640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33270/0 (socket says 172.21.15.116:33270) 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc852462640 1 -- 172.21.15.116:0/3149111043 learned_addr learned my addr 172.21.15.116:0/3149111043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc851c61640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b713780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.413+0000 7fc852c63640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b707be0 0x55608b6f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 -- 172.21.15.116:0/3149111043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 msgr2=0x55608b713780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b713780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 -- 172.21.15.116:0/3149111043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b707be0 msgr2=0x55608b6f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b707be0 0x55608b6f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 -- 172.21.15.116:0/3149111043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55608b71a860 con 0x55608b6fa000 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc851c61640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b713780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc852462640 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fc848005ab0 tx=0x7fc848005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e918abb48102c0e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc837fff640 1 -- 172.21.15.116:0/3149111043 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848049050 con 0x55608b6fa000 2024-03-19T02:42:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc837fff640 1 -- 172.21.15.116:0/3149111043 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8480042c0 con 0x55608b6fa000 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.414+0000 7fc837fff640 1 -- 172.21.15.116:0/3149111043 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84804b2f0 con 0x55608b6fa000 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- 172.21.15.116:0/3149111043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 msgr2=0x55608b704490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fc848005ab0 tx=0x7fc848005fd0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- 172.21.15.116:0/3149111043 shutdown_connections 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b707be0 0x55608b6f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b713780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- 172.21.15.116:0/3149111043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b6fa000 0x55608b704490 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- 172.21.15.116:0/3149111043 >> 172.21.15.116:0/3149111043 conn(0x55608b6f4a00 msgr2=0x55608b6f66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- 172.21.15.116:0/3149111043 shutdown_connections 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- 172.21.15.116:0/3149111043 wait complete. 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- start start 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 Processor -- start 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b70fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b70b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b70b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1adf50 con 0x55608b707be0 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad880 con 0x55608b6fa000 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1aaab0 con 0x55608b6fa920 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc85738d740 10 client.0 _ll_register_callbacks cb 0x7fc84401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc852c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b70b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc852462640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b70fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc852462640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b70fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45452/0 (socket says 172.21.15.116:45452) 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc852462640 1 -- 172.21.15.116:0/3785458528 learned_addr learned my addr 172.21.15.116:0/3785458528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.415+0000 7fc851c61640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b70b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 -- 172.21.15.116:0/3785458528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 msgr2=0x55608b70fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b70fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 -- 172.21.15.116:0/3785458528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 msgr2=0x55608b70b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b70b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 -- 172.21.15.116:0/3785458528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55608b6f4830 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc852c63640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b70b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc851c61640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b70b330 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0cbfb0 tx=0x7fc83c09ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2f47a9e956df631 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc835ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0a33d0 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fc83c09a310 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8480042c0 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc83c09b5c0 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc85738d740 10 client.29476 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.416+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55608b77dd20 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8480042c0 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc83c009a30 con 0x55608b6fa920 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 10 client.29476 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 10 client.29476 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 20 client.29476 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b77e4f0 0x55608b780910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 4 client.29476 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55608b66a9f0 con 0x55608b77e4f0 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc852462640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b77e4f0 0x55608b780910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc852462640 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b77e4f0 0x55608b780910 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2ee86e71ffb4f65e server_cookie=77690fe34781c4fe in_seq=0 out_seq=0 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.417+0000 7fc835ffb640 10 client.29476 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc835ffb640 1 -- 172.21.15.116:0/3785458528 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55608b66a9f0 con 0x55608b77e4f0 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc835ffb640 10 client.29476 handle_command_reply: tid=0 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 client.29476 shutdown 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 msgr2=0x55608b70b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b70b330 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0cbfb0 tx=0x7fc83c09ad50 comp rx=0 tx=0).stop 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 shutdown_connections 2024-03-19T02:42:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55608b77e4f0 0x55608b780910 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55608b707be0 0x55608b70b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55608b6fa920 0x55608b70b330 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc835ffb640 0 client.29476 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 --2- 172.21.15.116:0/3785458528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55608b6fa000 0x55608b70fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 >> 172.21.15.116:0/3785458528 conn(0x55608b6f4a00 msgr2=0x55608b706210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 shutdown_connections 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 1 -- 172.21.15.116:0/3785458528 wait complete. 2024-03-19T02:42:23.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc85738d740 20 client.29476 trim_cache size 0 max 16384 2024-03-19T02:42:23.427 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 msgr2=0x7fc8380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 0x7fc8380a4e30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc84401b000 tx=0x7fc84400ea60 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 msgr2=0x7fc84c199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.418+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00d950 tx=0x7fc83c00de10 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 shutdown_connections 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8380a2970 0x7fc8380a4e30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c1066b0 0x7fc84c1a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c101060 0x7fc84c199310 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 --2- 172.21.15.116:0/3042055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc84c1006a0 0x7fc84c198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 >> 172.21.15.116:0/3042055524 conn(0x7fc84c0778d0 msgr2=0x7fc84c103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 shutdown_connections 2024-03-19T02:42:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.419+0000 7fc8546c6640 1 -- 172.21.15.116:0/3042055524 wait complete. 2024-03-19T02:42:23.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:42:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 -- 172.21.15.116:0/846200179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 msgr2=0x7f91e0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 --2- 172.21.15.116:0/846200179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e0106a90 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009930 tx=0x7f91dc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 -- 172.21.15.116:0/846200179 shutdown_connections 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 --2- 172.21.15.116:0/846200179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 0x7f91e010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 --2- 172.21.15.116:0/846200179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 --2- 172.21.15.116:0/846200179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e0106a90 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 -- 172.21.15.116:0/846200179 >> 172.21.15.116:0/846200179 conn(0x7f91e00778d0 msgr2=0x7f91e0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 -- 172.21.15.116:0/846200179 shutdown_connections 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.597+0000 7f91e771b640 1 -- 172.21.15.116:0/846200179 wait complete. 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 Processor -- start 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 -- start start 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 0x7f91e01984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91e0110930 con 0x7f91e01066b0 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01107b0 con 0x7f91e0101060 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e771b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91e0110ab0 con 0x7f91e01006a0 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33290/0 (socket says 172.21.15.116:33290) 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e4cb6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 0x7f91e01984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.598+0000 7f91e5cb8640 1 -- 172.21.15.116:0/3049499385 learned_addr learned my addr 172.21.15.116:0/3049499385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 -- 172.21.15.116:0/3049499385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 msgr2=0x7f91e01984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 0x7f91e01984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 -- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 msgr2=0x7f91e019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 -- 172.21.15.116:0/3049499385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e019cf90 con 0x7f91e01006a0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0197fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009230 tx=0x7f91dc0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91d67fc640 1 -- 172.21.15.116:0/3049499385 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc043020 con 0x7f91e01006a0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e019d220 con 0x7f91e01006a0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91d67fc640 1 -- 172.21.15.116:0/3049499385 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91dc02ca70 con 0x7f91e01006a0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91d67fc640 1 -- 172.21.15.116:0/3049499385 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0039f0 con 0x7f91e01006a0 2024-03-19T02:42:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.599+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e01a4b10 con 0x7f91e01006a0 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91d67fc640 1 -- 172.21.15.116:0/3049499385 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f91dc046020 con 0x7f91e01006a0 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 Processor -- start 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 -- start start 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314abf0 0x55cdb3153290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb31537d0 0x55cdb3155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cdb30ad4b0 con 0x55cdb314a2d0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3055d70 con 0x55cdb314abf0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cdb30a9550 con 0x55cdb31537d0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33292/0 (socket says 172.21.15.116:33292) 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.602+0000 7f91e5cb8640 1 -- 172.21.15.116:0/702442080 learned_addr learned my addr 172.21.15.116:0/702442080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e54b7640 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314abf0 0x55cdb3153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb31537d0 0x55cdb3155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91d67fc640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 0x7f91cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 -- 172.21.15.116:0/702442080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb31537d0 msgr2=0x55cdb3155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb31537d0 0x55cdb3155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 -- 172.21.15.116:0/702442080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314abf0 msgr2=0x55cdb3153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314abf0 0x55cdb3153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 -- 172.21.15.116:0/702442080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb3078210 con 0x55cdb314a2d0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91d67fc640 1 -- 172.21.15.116:0/3049499385 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f91dc0f1050 con 0x7f91e01006a0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 0x7f91cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f91d80119d0 tx=0x7f91d8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4961f62457e4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91cb7fe640 1 -- 172.21.15.116:0/702442080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8017070 con 0x55cdb314a2d0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91cb7fe640 1 -- 172.21.15.116:0/702442080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91d800fc40 con 0x55cdb314a2d0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.603+0000 7f91cb7fe640 1 -- 172.21.15.116:0/702442080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d800e3b0 con 0x55cdb314a2d0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- 172.21.15.116:0/702442080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 msgr2=0x55cdb314a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f91d80119d0 tx=0x7f91d8011e90 comp rx=0 tx=0).stop 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 0x7f91cc0a4e30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f91d8023000 tx=0x7f91d8024000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- 172.21.15.116:0/702442080 shutdown_connections 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb31537d0 0x55cdb3155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314abf0 0x55cdb3153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/702442080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb314a6b0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- 172.21.15.116:0/702442080 >> 172.21.15.116:0/702442080 conn(0x55cdb3144a00 msgr2=0x55cdb3144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- 172.21.15.116:0/702442080 shutdown_connections 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- 172.21.15.116:0/702442080 wait complete. 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 -- start start 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.604+0000 7f91ea3e9740 1 Processor -- start 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb31511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cdb31565c0 con 0x55cdb314abf0 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3069310 con 0x55cdb31537d0 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cdb3156740 con 0x55cdb314a2d0 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91ea3e9740 10 client.0 _ll_register_callbacks cb 0x55cdb305d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e4cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58428/0 (socket says 172.21.15.116:58428) 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e4cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33296/0 (socket says 172.21.15.116:33296) 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 -- 172.21.15.116:0/3014471544 learned_addr learned my addr 172.21.15.116:0/3014471544 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb31511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 -- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 msgr2=0x55cdb31511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb31511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 -- 172.21.15.116:0/3014471544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 msgr2=0x55cdb3150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 -- 172.21.15.116:0/3014471544 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb2dc13f0 con 0x55cdb314a2d0 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91e54b7640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0b9840 tx=0x7f91dc0b9050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e4c734a413caa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.605+0000 7f91c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc02cc20 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f91dc005340 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d800fc40 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f91dc0bd3e0 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 10 client.29488 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cdb2e8b7d0 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91d800fc40 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f91dc0d60f0 con 0x55cdb314a2d0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 10 client.29488 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 10 client.29488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 20 client.29488 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb3163ac0 0x55cdb3165f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 4 client.29488 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdb310b3e0 con 0x55cdb3163ac0 2024-03-19T02:42:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.606+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb3163ac0 0x55cdb3165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.607+0000 7f91c8ff9640 10 client.29488 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.607+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb3163ac0 0x55cdb3165f80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4893feca362a556f server_cookie=513d7970f0eb4798 in_seq=0 out_seq=0 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91c8ff9640 1 -- 172.21.15.116:0/3014471544 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cdb310b3e0 con 0x55cdb3163ac0 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91c8ff9640 10 client.29488 handle_command_reply: tid=0 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91ea3e9740 1 client.29488 shutdown 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 msgr2=0x55cdb3150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0b9840 tx=0x7f91dc0b9050 comp rx=0 tx=0).stop 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 shutdown_connections 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.608+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb3163ac0 0x55cdb3165f80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb31511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314abf0 0x55cdb3150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91c8ff9640 0 client.29488 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/3014471544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314a2d0 0x55cdb3150730 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 >> 172.21.15.116:0/3014471544 conn(0x55cdb3144a00 msgr2=0x55cdb3146750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 shutdown_connections 2024-03-19T02:42:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 1 -- 172.21.15.116:0/3014471544 wait complete. 2024-03-19T02:42:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.609+0000 7f91ea3e9740 20 client.29488 trim_cache size 0 max 16384 2024-03-19T02:42:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.623+0000 7f91ea3e9740 1 Processor -- start 2024-03-19T02:42:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.623+0000 7f91ea3e9740 1 -- start start 2024-03-19T02:42:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb313fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb3150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01aebe0 con 0x55cdb314a2d0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01b0810 con 0x55cdb31537d0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01af2b0 con 0x55cdb314abf0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33310/0 (socket says 172.21.15.116:33310) 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 -- 172.21.15.116:0/396458598 learned_addr learned my addr 172.21.15.116:0/396458598 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb313fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb3150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 -- 172.21.15.116:0/396458598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 msgr2=0x55cdb313fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb313fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 -- 172.21.15.116:0/396458598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 msgr2=0x55cdb3150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb3150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 -- 172.21.15.116:0/396458598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb314f6e0 con 0x55cdb314a2d0 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb313fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91e54b7640 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f91dc006010 tx=0x7f91dc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6daab4060d78d478 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.624+0000 7f91caffd640 1 -- 172.21.15.116:0/396458598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc02cc20 con 0x55cdb314a2d0 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91caffd640 1 -- 172.21.15.116:0/396458598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91dc0c1530 con 0x55cdb314a2d0 2024-03-19T02:42:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91caffd640 1 -- 172.21.15.116:0/396458598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0d60f0 con 0x55cdb314a2d0 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 -- 172.21.15.116:0/396458598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 msgr2=0x55cdb315f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f91dc006010 tx=0x7f91dc009260 comp rx=0 tx=0).stop 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 -- 172.21.15.116:0/396458598 shutdown_connections 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb31537d0 0x55cdb3150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb313fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/396458598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb314a2d0 0x55cdb315f360 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 -- 172.21.15.116:0/396458598 >> 172.21.15.116:0/396458598 conn(0x55cdb3144a00 msgr2=0x55cdb3143ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 -- 172.21.15.116:0/396458598 shutdown_connections 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.625+0000 7f91ea3e9740 1 -- 172.21.15.116:0/396458598 wait complete. 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 -- start start 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 Processor -- start 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 0x55cdb3157430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb3157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01b0080 con 0x55cdb31537d0 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01af9b0 con 0x55cdb314a2d0 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01acbe0 con 0x55cdb314abf0 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91ea3e9740 10 client.0 _ll_register_callbacks cb 0x7f91dc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 0x55cdb3157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e54b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 0x55cdb3157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45496/0 (socket says 172.21.15.116:45496) 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e5cb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33316/0 (socket says 172.21.15.116:33316) 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e54b7640 1 -- 172.21.15.116:0/1838056955 learned_addr learned my addr 172.21.15.116:0/1838056955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.626+0000 7f91e4cb6640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb3157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 -- 172.21.15.116:0/1838056955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 msgr2=0x55cdb3157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb3157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 -- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 msgr2=0x55cdb3157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 0x55cdb3157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 -- 172.21.15.116:0/1838056955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb314aa20 con 0x55cdb31537d0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91e5cb8640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f91d800f790 tx=0x7f91d800ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35b97cc372b5260 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8017040 con 0x55cdb31537d0 2024-03-19T02:42:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f91d801cdd0 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91dc0c1530 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f91d8011550 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 10 client.19776 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cdb31ca130 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91dc0c1530 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f91d801b070 con 0x55cdb31537d0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 10 client.19776 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 10 client.19776 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.627+0000 7f91ea3e9740 20 client.19776 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb31ca770 0x55cdb31ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 4 client.19776 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdb30ba9f0 con 0x55cdb31ca770 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91e54b7640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb31ca770 0x55cdb31ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91c8ff9640 10 client.19776 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91e54b7640 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb31ca770 0x55cdb31ccb90 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8e21a7244e7597f server_cookie=d94ce522a97e9481 in_seq=0 out_seq=0 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91c8ff9640 1 -- 172.21.15.116:0/1838056955 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdb30ba9f0 con 0x55cdb31ca770 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91c8ff9640 10 client.19776 handle_command_reply: tid=0 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 client.19776 shutdown 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 msgr2=0x55cdb3157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f91d800f790 tx=0x7f91d800ff50 comp rx=0 tx=0).stop 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 shutdown_connections 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55cdb31ca770 0x55cdb31ccb90 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb31537d0 0x55cdb3157eb0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cdb314abf0 0x55cdb3157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 --2- 172.21.15.116:0/1838056955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cdb314a2d0 0x55cdb3157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91c8ff9640 0 client.19776 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.628+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 >> 172.21.15.116:0/1838056955 conn(0x55cdb3144a00 msgr2=0x55cdb3145e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 shutdown_connections 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91ea3e9740 1 -- 172.21.15.116:0/1838056955 wait complete. 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91ea3e9740 20 client.19776 trim_cache size 0 max 16384 2024-03-19T02:42:23.637 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 msgr2=0x7f91cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 0x7f91cc0a4e30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f91d8023000 tx=0x7f91d8024000 comp rx=0 tx=0).stop 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 msgr2=0x7f91e0197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0197fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009230 tx=0x7f91dc0045e0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 shutdown_connections 2024-03-19T02:42:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f91cc0a2970 0x7f91cc0a4e30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e01066b0 0x7f91e019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91e0101060 0x7f91e01984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 --2- 172.21.15.116:0/3049499385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e01006a0 0x7f91e0197fa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 >> 172.21.15.116:0/3049499385 conn(0x7f91e00778d0 msgr2=0x7f91e0068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 shutdown_connections 2024-03-19T02:42:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.629+0000 7f91e771b640 1 -- 172.21.15.116:0/3049499385 wait complete. 2024-03-19T02:42:23.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:42:23.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/315900561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41006a0 msgr2=0x7fd3e4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/315900561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41006a0 0x7fd3e4100b20 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009930 tx=0x7fd3cc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:23.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/315900561 shutdown_connections 2024-03-19T02:42:23.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/315900561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 0x7fd3e410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/315900561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41006a0 0x7fd3e4100b20 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/315900561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41066b0 0x7fd3e4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/315900561 >> 172.21.15.116:0/315900561 conn(0x7fd3e40778d0 msgr2=0x7fd3e4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/315900561 shutdown_connections 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.761+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/315900561 wait complete. 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 Processor -- start 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 -- start start 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41006a0 0x7fd3e4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 0x7fd3e4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4110970 con 0x7fd3e41066b0 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41107f0 con 0x7fd3e41006a0 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e8c8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4110af0 con 0x7fd3e4101060 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41006a0 0x7fd3e4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 0x7fd3e4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33344/0 (socket says 172.21.15.116:33344) 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/854634363 learned_addr learned my addr 172.21.15.116:0/854634363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/854634363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 msgr2=0x7fd3e4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 0x7fd3e4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/854634363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41006a0 msgr2=0x7fd3e4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41006a0 0x7fd3e4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.762+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/854634363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e419cfd0 con 0x7fd3e41066b0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00eb10 tx=0x7fd3dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3db7fe640 1 -- 172.21.15.116:0/854634363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc004b20 con 0x7fd3e41066b0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3db7fe640 1 -- 172.21.15.116:0/854634363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3dc004cc0 con 0x7fd3e41066b0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e419d2c0 con 0x7fd3e41066b0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3db7fe640 1 -- 172.21.15.116:0/854634363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc010690 con 0x7fd3e41066b0 2024-03-19T02:42:23.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.763+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e41a4b50 con 0x7fd3e41066b0 2024-03-19T02:42:23.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.765+0000 7fd3db7fe640 1 -- 172.21.15.116:0/854634363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd3dc005020 con 0x7fd3e41066b0 2024-03-19T02:42:23.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 Processor -- start 2024-03-19T02:42:23.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 -- start start 2024-03-19T02:42:23.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 0x7fd3d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 0x5583dd58b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd596bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5583dd4ee4b0 con 0x5583dd58b2d0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5583dd496d70 con 0x5583dd5947d0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3eb872740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5583dd4ea550 con 0x5583dd58bbf0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 0x7fd3d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3db7fe640 1 -- 172.21.15.116:0/854634363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd3dc0cf900 con 0x7fd3e41066b0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58480/0 (socket says 172.21.15.116:58480) 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3dbfff640 1 -- 172.21.15.116:0/1514039862 learned_addr learned my addr 172.21.15.116:0/1514039862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3e259c640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 0x5583dd58b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.766+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd596bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 -- 172.21.15.116:0/1514039862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 msgr2=0x5583dd596bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd596bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 -- 172.21.15.116:0/1514039862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 msgr2=0x5583dd58b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 0x5583dd58b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 -- 172.21.15.116:0/1514039862 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583dd4b9210 con 0x5583dd58bbf0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3e259c640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 0x5583dd58b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd596bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 0x7fd3d00a4e30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009900 tx=0x7fd3cc02c660 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009930 tx=0x7fd3cc047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=900fa00023669f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3d8ff9640 1 -- 172.21.15.116:0/1514039862 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc005100 con 0x5583dd58bbf0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3d8ff9640 1 -- 172.21.15.116:0/1514039862 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3cc0046c0 con 0x5583dd58bbf0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.767+0000 7fd3d8ff9640 1 -- 172.21.15.116:0/1514039862 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc003810 con 0x5583dd58bbf0 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 -- 172.21.15.116:0/1514039862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 msgr2=0x5583dd594290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009930 tx=0x7fd3cc047000 comp rx=0 tx=0).stop 2024-03-19T02:42:23.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 -- 172.21.15.116:0/1514039862 shutdown_connections 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd596bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd594290 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 --2- 172.21.15.116:0/1514039862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58b2d0 0x5583dd58b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 -- 172.21.15.116:0/1514039862 >> 172.21.15.116:0/1514039862 conn(0x5583dd585a00 msgr2=0x5583dd585de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 -- 172.21.15.116:0/1514039862 shutdown_connections 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.768+0000 7fd3eb872740 1 -- 172.21.15.116:0/1514039862 wait complete. 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 -- start start 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 Processor -- start 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd5917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 0x5583dd591d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd592270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5583dd59b970 con 0x5583dd58bbf0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5583dd4aa310 con 0x5583dd5947d0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5583dd59baf0 con 0x5583dd58b2d0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3eb872740 10 client.0 _ll_register_callbacks cb 0x5583dd49e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 0x5583dd591d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd5917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 0x5583dd591d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33360/0 (socket says 172.21.15.116:33360) 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3dbfff640 1 -- 172.21.15.116:0/2644460770 learned_addr learned my addr 172.21.15.116:0/2644460770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd5917f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58496/0 (socket says 172.21.15.116:58496) 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.769+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd592270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 msgr2=0x5583dd5917f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd5917f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/2644460770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 msgr2=0x5583dd591d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 0x5583dd591d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/2644460770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583dd2023f0 con 0x5583dd5947d0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd592270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0ccc70 tx=0x7fd3dc09b4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e89eb86bf64453f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0b3070 con 0x5583dd5947d0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd3dc005510 con 0x5583dd5947d0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.770+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3cc0046c0 con 0x5583dd5947d0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.771+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd3dc09fcd0 con 0x5583dd5947d0 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.771+0000 7fd3eb872740 10 client.29549 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.771+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5583dd2cc7d0 con 0x5583dd5947d0 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.771+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3dc0a4320 con 0x5583dd5947d0 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fd3dc0978d0 con 0x5583dd5947d0 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 10 client.29549 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 10 client.29549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 20 client.29549 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd5a9000 0x5583dd5ab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 4 client.29549 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd3cc0046c0 con 0x5583dd5a9000 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3e259c640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd5a9000 0x5583dd5ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3c9ffb640 10 client.29549 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.772+0000 7fd3e259c640 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd5a9000 0x5583dd5ab420 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c0607cbcfbc13c7 server_cookie=6623e097e24e234e in_seq=0 out_seq=0 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/2644460770 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd3cc0046c0 con 0x5583dd5a9000 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3c9ffb640 10 client.29549 handle_command_reply: tid=0 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 client.29549 shutdown 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 msgr2=0x5583dd592270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd592270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0ccc70 tx=0x7fd3dc09b4e0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 shutdown_connections 2024-03-19T02:42:23.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd5a9000 0x5583dd5ab420 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd5947d0 0x5583dd592270 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3c9ffb640 0 client.29549 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd58bbf0 0x5583dd591d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 --2- 172.21.15.116:0/2644460770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd5917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 >> 172.21.15.116:0/2644460770 conn(0x5583dd585a00 msgr2=0x5583dd587750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 shutdown_connections 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 1 -- 172.21.15.116:0/2644460770 wait complete. 2024-03-19T02:42:23.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.773+0000 7fd3eb872740 20 client.29549 trim_cache size 0 max 16384 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 Processor -- start 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 -- start start 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 0x5583dd593f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd590970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41aca40 con 0x5583dd5947d0 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ae670 con 0x5583dd58b2d0 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3eb872740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ad110 con 0x5583dd58bbf0 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 0x5583dd593f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3dbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58498/0 (socket says 172.21.15.116:58498) 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd590970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 0x5583dd593f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45540/0 (socket says 172.21.15.116:45540) 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3dbfff640 1 -- 172.21.15.116:0/3895311628 learned_addr learned my addr 172.21.15.116:0/3895311628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/3895311628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 msgr2=0x5583dd5a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.791+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/3895311628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 msgr2=0x5583dd593f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 0x5583dd593f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/3895311628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583dd55b880 con 0x5583dd5947d0 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd590970 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc098550 tx=0x7fd3dc09b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a8ed0eea238a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3cbfff640 1 -- 172.21.15.116:0/3895311628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0b3070 con 0x5583dd5947d0 2024-03-19T02:42:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3cbfff640 1 -- 172.21.15.116:0/3895311628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3dc0a4320 con 0x5583dd5947d0 2024-03-19T02:42:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.792+0000 7fd3cbfff640 1 -- 172.21.15.116:0/3895311628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0ba120 con 0x5583dd5947d0 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- 172.21.15.116:0/3895311628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 msgr2=0x5583dd590970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd590970 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc098550 tx=0x7fd3dc09b610 comp rx=0 tx=0).stop 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- 172.21.15.116:0/3895311628 shutdown_connections 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd590970 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58bbf0 0x5583dd5a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- 172.21.15.116:0/3895311628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58b2d0 0x5583dd593f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- 172.21.15.116:0/3895311628 >> 172.21.15.116:0/3895311628 conn(0x5583dd585a00 msgr2=0x5583dd586e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- 172.21.15.116:0/3895311628 shutdown_connections 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- 172.21.15.116:0/3895311628 wait complete. 2024-03-19T02:42:23.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- start start 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 Processor -- start 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58bbf0 0x5583dd58ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41adee0 con 0x5583dd5947d0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41ad810 con 0x5583dd58bbf0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e41aaa40 con 0x5583dd58b2d0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3eb872740 10 client.0 _ll_register_callbacks cb 0x7fd3dc097e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.793+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33376/0 (socket says 172.21.15.116:33376) 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/410958536 learned_addr learned my addr 172.21.15.116:0/410958536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e259c640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58510/0 (socket says 172.21.15.116:58510) 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3dbfff640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58bbf0 0x5583dd58ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 msgr2=0x5583dd58e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/410958536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58bbf0 msgr2=0x5583dd58ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58bbf0 0x5583dd58ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 -- 172.21.15.116:0/410958536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583dd596bc0 con 0x5583dd5947d0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e259c640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3e2d9d640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc005ba0 tx=0x7fd3dc09b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb763237ac04e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0b3070 con 0x5583dd5947d0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3dc0a4320 con 0x5583dd5947d0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd3dc0ba120 con 0x5583dd5947d0 2024-03-19T02:42:23.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd3dc09b6c0 con 0x5583dd5947d0 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3eb872740 10 client.19794 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5583dd60f530 con 0x5583dd5947d0 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.794+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3dc0a4320 con 0x5583dd5947d0 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fd3dc0a0d60 con 0x5583dd5947d0 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 10 client.19794 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 10 client.19794 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 20 client.19794 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd60fa00 0x5583dd611df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 4 client.19794 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5583dd54c3e0 con 0x5583dd60fa00 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3e259c640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd60fa00 0x5583dd611df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3e259c640 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd60fa00 0x5583dd611df0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d907c6f037100221 server_cookie=5faadfae17dc5cac in_seq=0 out_seq=0 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3c9ffb640 10 client.19794 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3c9ffb640 1 -- 172.21.15.116:0/410958536 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5583dd54c3e0 con 0x5583dd60fa00 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3c9ffb640 10 client.19794 handle_command_reply: tid=0 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.795+0000 7fd3eb872740 1 client.19794 shutdown 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 msgr2=0x5583dd58ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc005ba0 tx=0x7fd3dc09b9c0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 shutdown_connections 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5583dd60fa00 0x5583dd611df0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583dd5947d0 0x5583dd58ef40 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3c9ffb640 0 client.19794 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583dd58bbf0 0x5583dd58ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 --2- 172.21.15.116:0/410958536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5583dd58b2d0 0x5583dd58e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 >> 172.21.15.116:0/410958536 conn(0x5583dd585a00 msgr2=0x5583dd584310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 shutdown_connections 2024-03-19T02:42:23.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 1 -- 172.21.15.116:0/410958536 wait complete. 2024-03-19T02:42:23.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3eb872740 20 client.19794 trim_cache size 0 max 16384 2024-03-19T02:42:23.804 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 msgr2=0x7fd3d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 0x7fd3d00a4e30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009900 tx=0x7fd3cc02c660 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 msgr2=0x7fd3e419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00eb10 tx=0x7fd3dc00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 shutdown_connections 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3d00a2970 0x7fd3d00a4e30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e41066b0 0x7fd3e419c8a0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3e4101060 0x7fd3e4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 --2- 172.21.15.116:0/854634363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e41006a0 0x7fd3e4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.796+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 >> 172.21.15.116:0/854634363 conn(0x7fd3e40778d0 msgr2=0x7fd3e4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.797+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 shutdown_connections 2024-03-19T02:42:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.797+0000 7fd3e8c8c640 1 -- 172.21.15.116:0/854634363 wait complete. 2024-03-19T02:42:23.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:42:23.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/3960089763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 msgr2=0x7f0a3410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/3960089763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3410d520 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009be0 tx=0x7f0a2c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/3960089763 shutdown_connections 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/3960089763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3410d520 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/3960089763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34100970 0x7f0a34100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/3960089763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34105690 0x7f0a34105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/3960089763 >> 172.21.15.116:0/3960089763 conn(0x7f0a340774b0 msgr2=0x7f0a340778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/3960089763 shutdown_connections 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.938+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/3960089763 wait complete. 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 Processor -- start 2024-03-19T02:42:23.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 -- start start 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 0x7f0a3419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34105690 0x7f0a3419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3410f910 con 0x7f0a34101330 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3410f790 con 0x7f0a34105690 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a3a4ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3410fa90 con 0x7f0a34100970 2024-03-19T02:42:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a33fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 0x7f0a3419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33400/0 (socket says 172.21.15.116:33400) 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 -- 172.21.15.116:0/2155280133 learned_addr learned my addr 172.21.15.116:0/2155280133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34105690 0x7f0a3419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 -- 172.21.15.116:0/2155280133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 msgr2=0x7f0a3419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 0x7f0a3419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 -- 172.21.15.116:0/2155280133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34105690 msgr2=0x7f0a3419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.939+0000 7f0a337fe640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34105690 0x7f0a3419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a337fe640 1 -- 172.21.15.116:0/2155280133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3419f1c0 con 0x7f0a34101330 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a33fff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 0x7f0a3419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a337fe640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800e9f0 tx=0x7f0a2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a317fa640 1 -- 172.21.15.116:0/2155280133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800ccc0 con 0x7f0a34101330 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3419f4b0 con 0x7f0a34101330 2024-03-19T02:42:23.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a317fa640 1 -- 172.21.15.116:0/2155280133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a280108e0 con 0x7f0a34101330 2024-03-19T02:42:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a317fa640 1 -- 172.21.15.116:0/2155280133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280187c0 con 0x7f0a34101330 2024-03-19T02:42:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.940+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a341a6d40 con 0x7f0a34101330 2024-03-19T02:42:23.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.942+0000 7f0a317fa640 1 -- 172.21.15.116:0/2155280133 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0a28010a90 con 0x7f0a34101330 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 Processor -- start 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 -- start start 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed12d0 0x564d52ed16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 0x564d52eda290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d52e344b0 con 0x564d52eda7d0 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564d52ddcd70 con 0x564d52ed1bf0 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564d52e30550 con 0x564d52ed12d0 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a38a9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed12d0 0x564d52ed16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 0x564d52eda290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33404/0 (socket says 172.21.15.116:33404) 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 -- 172.21.15.116:0/638802149 learned_addr learned my addr 172.21.15.116:0/638802149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a317fa640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 0x7f0a1c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 -- 172.21.15.116:0/638802149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed12d0 msgr2=0x564d52ed16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed12d0 0x564d52ed16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 -- 172.21.15.116:0/638802149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 msgr2=0x564d52eda290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 0x564d52eda290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a337fe640 1 -- 172.21.15.116:0/638802149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d52dff210 con 0x564d52eda7d0 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a33fff640 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 0x564d52eda290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.943+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 0x7f0a1c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.944+0000 7f0a317fa640 1 -- 172.21.15.116:0/2155280133 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f0a280cf6a0 con 0x7f0a34101330 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.944+0000 7f0a337fe640 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f0a280cc8f0 tx=0x7f0a280c6310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f165a9c8cd600ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.944+0000 7f0a1a7fc640 1 -- 172.21.15.116:0/638802149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a28026030 con 0x564d52eda7d0 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.944+0000 7f0a1a7fc640 1 -- 172.21.15.116:0/638802149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a280980c0 con 0x564d52eda7d0 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.944+0000 7f0a1a7fc640 1 -- 172.21.15.116:0/638802149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2806c9d0 con 0x564d52eda7d0 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/638802149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 msgr2=0x564d52edcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 0x7f0a1c0a4e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005f00 tx=0x7f0a2c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f0a280cc8f0 tx=0x7f0a280c6310 comp rx=0 tx=0).stop 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/638802149 shutdown_connections 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52edcbc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed1bf0 0x564d52eda290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/638802149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed12d0 0x564d52ed16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/638802149 >> 172.21.15.116:0/638802149 conn(0x564d52ecba00 msgr2=0x564d52ecbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/638802149 shutdown_connections 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/638802149 wait complete. 2024-03-19T02:42:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 -- start start 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.945+0000 7f0a3d1c9740 1 Processor -- start 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ed7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52ed7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ed8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d52edd500 con 0x564d52ed1bf0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564d52df0310 con 0x564d52ed12d0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564d52edd680 con 0x564d52eda7d0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a3d1c9740 10 client.0 _ll_register_callbacks cb 0x564d52de42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ed7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52ed7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a38a9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ed8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ed7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45590/0 (socket says 172.21.15.116:45590) 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a33fff640 1 -- 172.21.15.116:0/220791346 learned_addr learned my addr 172.21.15.116:0/220791346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 -- 172.21.15.116:0/220791346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 msgr2=0x564d52ed8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ed8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 -- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 msgr2=0x564d52ed7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ed7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 -- 172.21.15.116:0/220791346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d52b483f0 con 0x564d52ed1bf0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ed8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a337fe640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52ed7d50 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0a28098290 tx=0x7f0a280cc950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87024c4b33b6c4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.946+0000 7f0a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2809b400 con 0x564d52ed1bf0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f0a280d5170 con 0x564d52ed1bf0 2024-03-19T02:42:23.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a280980c0 con 0x564d52ed1bf0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0a2809b220 con 0x564d52ed1bf0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 10 client.19806 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564d52c127d0 con 0x564d52ed1bf0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a28010e00 con 0x564d52ed1bf0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f0a28097610 con 0x564d52ed1bf0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 10 client.19806 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 10 client.19806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 20 client.19806 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52eea940 0x564d52eecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 4 client.19806 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.947+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0a280980c0 con 0x564d52eea940 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.948+0000 7f0a33fff640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52eea940 0x564d52eecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.948+0000 7f0a33fff640 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52eea940 0x564d52eecd30 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1accfcbea497295b server_cookie=b9f307dc1d48e72b in_seq=0 out_seq=0 2024-03-19T02:42:23.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.948+0000 7f0a137fe640 10 client.19806 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.949+0000 7f0a137fe640 1 -- 172.21.15.116:0/220791346 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f0a280980c0 con 0x564d52eea940 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.949+0000 7f0a137fe640 10 client.19806 handle_command_reply: tid=0 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 client.19806 shutdown 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 msgr2=0x564d52ed7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52ed7d50 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0a28098290 tx=0x7f0a280cc950 comp rx=0 tx=0).stop 2024-03-19T02:42:23.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 shutdown_connections 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52eea940 0x564d52eecd30 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ed8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52ed7d50 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a137fe640 0 client.19806 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/220791346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ed7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 >> 172.21.15.116:0/220791346 conn(0x564d52ecba00 msgr2=0x564d52ecd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 shutdown_connections 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/220791346 wait complete. 2024-03-19T02:42:23.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.950+0000 7f0a3d1c9740 20 client.19806 trim_cache size 0 max 16384 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 Processor -- start 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 -- start start 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed1bf0 0x564d52ed80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b0e40 con 0x564d52eda7d0 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b2a70 con 0x564d52ed12d0 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b1510 con 0x564d52ed1bf0 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a38a9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a38a9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33436/0 (socket says 172.21.15.116:33436) 2024-03-19T02:42:23.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a337fe640 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed1bf0 0x564d52ed80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a38a9c640 1 -- 172.21.15.116:0/712443089 learned_addr learned my addr 172.21.15.116:0/712443089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 -- 172.21.15.116:0/712443089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed1bf0 msgr2=0x564d52ed80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed1bf0 0x564d52ed80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 -- 172.21.15.116:0/712443089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 msgr2=0x564d52ef2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.966+0000 7f0a33fff640 1 -- 172.21.15.116:0/712443089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d52ed5d60 con 0x564d52ed12d0 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.967+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.967+0000 7f0a33fff640 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee62a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009b60 tx=0x7f0a24009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3bbcf49ef60437a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.967+0000 7f0a19ffb640 1 -- 172.21.15.116:0/712443089 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240120d0 con 0x564d52ed12d0 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.967+0000 7f0a19ffb640 1 -- 172.21.15.116:0/712443089 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a24024020 con 0x564d52ed12d0 2024-03-19T02:42:23.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.967+0000 7f0a19ffb640 1 -- 172.21.15.116:0/712443089 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24020790 con 0x564d52ed12d0 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/712443089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 msgr2=0x564d52ee62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee62a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009b60 tx=0x7f0a24009260 comp rx=0 tx=0).stop 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/712443089 shutdown_connections 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52eda7d0 0x564d52ef2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52ed1bf0 0x564d52ed80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/712443089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee62a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/712443089 >> 172.21.15.116:0/712443089 conn(0x564d52ecba00 msgr2=0x564d52eca310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/712443089 shutdown_connections 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/712443089 wait complete. 2024-03-19T02:42:23.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.968+0000 7f0a3d1c9740 1 -- start start 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 Processor -- start 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ede1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b22e0 con 0x564d52ed1bf0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341b1c10 con 0x564d52ed12d0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341aee40 con 0x564d52eda7d0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33442/0 (socket says 172.21.15.116:33442) 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 -- 172.21.15.116:0/3216525041 learned_addr learned my addr 172.21.15.116:0/3216525041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a33fff640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a3d1c9740 10 client.0 _ll_register_callbacks cb 0x7f0a24009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:23.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a38a9c640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ede1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 -- 172.21.15.116:0/3216525041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 msgr2=0x564d52ede1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ede1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 -- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 msgr2=0x564d52ee2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a337fe640 1 -- 172.21.15.116:0/3216525041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d52ede700 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.969+0000 7f0a33fff640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a337fe640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0a2809b850 tx=0x7f0a280862e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad5ed85da62e9f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2809b400 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f0a28098070 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a24024020 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0a28010e00 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a3d1c9740 10 client.19812 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564d52ede700 con 0x564d52ed1bf0 2024-03-19T02:42:23.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a24024020 con 0x564d52ed1bf0 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.970+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f0a2809bb20 con 0x564d52ed1bf0 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 10 client.19812 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 10 client.19812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 20 client.19812 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52f50b30 0x564d52f52f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 4 client.19812 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d52e923e0 con 0x564d52f50b30 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a33fff640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52f50b30 0x564d52f52f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a137fe640 10 client.19812 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.971+0000 7f0a33fff640 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52f50b30 0x564d52f52f50 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afa831e5d82cf3b server_cookie=46491dc2bb5e706a in_seq=0 out_seq=0 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a137fe640 1 -- 172.21.15.116:0/3216525041 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d52e923e0 con 0x564d52f50b30 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a137fe640 10 client.19812 handle_command_reply: tid=0 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a3d1c9740 1 client.19812 shutdown 2024-03-19T02:42:23.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 msgr2=0x564d52eddc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0a2809b850 tx=0x7f0a280862e0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 shutdown_connections 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.972+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564d52f50b30 0x564d52f52f50 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a137fe640 0 client.19812 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564d52eda7d0 0x564d52ede1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d52ed1bf0 0x564d52eddc80 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 --2- 172.21.15.116:0/3216525041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d52ed12d0 0x564d52ee2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 >> 172.21.15.116:0/3216525041 conn(0x564d52ecba00 msgr2=0x564d52ecaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 shutdown_connections 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 1 -- 172.21.15.116:0/3216525041 wait complete. 2024-03-19T02:42:23.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3d1c9740 20 client.19812 trim_cache size 0 max 16384 2024-03-19T02:42:23.979 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 msgr2=0x7f0a1c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 0x7f0a1c0a4e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c005f00 tx=0x7f0a2c009500 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 msgr2=0x7f0a3419ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800e9f0 tx=0x7f0a2800edf0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 shutdown_connections 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0a1c0a2970 0x7f0a1c0a4e30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a34105690 0x7f0a3419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a34101330 0x7f0a3419ad00 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 --2- 172.21.15.116:0/2155280133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a34100970 0x7f0a3419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 >> 172.21.15.116:0/2155280133 conn(0x7f0a340774b0 msgr2=0x7f0a340fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 shutdown_connections 2024-03-19T02:42:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:23.973+0000 7f0a3a4ff640 1 -- 172.21.15.116:0/2155280133 wait complete. 2024-03-19T02:42:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:24 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1791341331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:24.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12579 8.03 MB/sec warmup 106 sec latency 141.430 ms 2024-03-19T02:42:24.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:24 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1791341331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:24 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1791341331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:24.987 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:42:24.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:42:24.988 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:42:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 -- 172.21.15.116:0/2775866582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b81006e0 msgr2=0x7f53b8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2775866582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b81006e0 0x7f53b8100b40 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009900 tx=0x7f53a002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 -- 172.21.15.116:0/2775866582 shutdown_connections 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2775866582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b8101080 0x7f53b810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2775866582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b81006e0 0x7f53b8100b40 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2775866582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 0x7f53b81059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 -- 172.21.15.116:0/2775866582 >> 172.21.15.116:0/2775866582 conn(0x7f53b80777b0 msgr2=0x7f53b8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 -- 172.21.15.116:0/2775866582 shutdown_connections 2024-03-19T02:42:25.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.114+0000 7f53bd52f640 1 -- 172.21.15.116:0/2775866582 wait complete. 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 Processor -- start 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 -- start start 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b81006e0 0x7f53b8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 0x7f53b819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53b8111a80 con 0x7f53b8101080 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53b8111900 con 0x7f53b81055c0 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53bd52f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53b8111c00 con 0x7f53b81006e0 2024-03-19T02:42:25.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b81006e0 0x7f53b8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33458/0 (socket says 172.21.15.116:33458) 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 -- 172.21.15.116:0/2846257946 learned_addr learned my addr 172.21.15.116:0/2846257946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53b77fe640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 0x7f53b819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 -- 172.21.15.116:0/2846257946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b81006e0 msgr2=0x7f53b8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b81006e0 0x7f53b8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 -- 172.21.15.116:0/2846257946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 msgr2=0x7f53b819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 0x7f53b819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.115+0000 7f53affff640 1 -- 172.21.15.116:0/2846257946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b819cff0 con 0x7f53b8101080 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53affff640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005940 tx=0x7f53a002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53b4ff9640 1 -- 172.21.15.116:0/2846257946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0043020 con 0x7f53b8101080 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b819d280 con 0x7f53b8101080 2024-03-19T02:42:25.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53b4ff9640 1 -- 172.21.15.116:0/2846257946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53a0004e00 con 0x7f53b8101080 2024-03-19T02:42:25.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53b4ff9640 1 -- 172.21.15.116:0/2846257946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0003970 con 0x7f53b8101080 2024-03-19T02:42:25.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.116+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b81a4b70 con 0x7f53b8101080 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.118+0000 7f53b4ff9640 1 -- 172.21.15.116:0/2846257946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f53a0046020 con 0x7f53b8101080 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53b4ff9640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 0x7f53a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 Processor -- start 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 -- start start 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 0x7f53a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53b4ff9640 1 -- 172.21.15.116:0/2846257946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f53a00f1070 con 0x7f53b8101080 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e44e160 con 0x55dd0e4ea030 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e3f6e10 con 0x55dd0e4f7c90 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.119+0000 7f53c01f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e44a200 con 0x55dd0e4ea950 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33472/0 (socket says 172.21.15.116:33472) 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b6ffd640 1 -- 172.21.15.116:0/1643062409 learned_addr learned my addr 172.21.15.116:0/1643062409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b77fe640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 0x7f53a40a4e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009b50 tx=0x7f53a8005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 -- 172.21.15.116:0/1643062409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 msgr2=0x55dd0e4fa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 -- 172.21.15.116:0/1643062409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 msgr2=0x55dd0e4ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 -- 172.21.15.116:0/1643062409 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd0e22a7e0 con 0x55dd0e4ea950 2024-03-19T02:42:25.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53affff640 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f7750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f53a00ee4f0 tx=0x7f53a00bce20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffd47789aca58ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.120+0000 7f53ad7fa640 1 -- 172.21.15.116:0/1643062409 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00bd400 con 0x55dd0e4ea950 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.121+0000 7f53ad7fa640 1 -- 172.21.15.116:0/1643062409 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53a00c1550 con 0x55dd0e4ea950 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.121+0000 7f53ad7fa640 1 -- 172.21.15.116:0/1643062409 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0003b10 con 0x55dd0e4ea950 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.121+0000 7f53c01f1740 1 -- 172.21.15.116:0/1643062409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 msgr2=0x55dd0e4f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.121+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f7750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f53a00ee4f0 tx=0x7f53a00bce20 comp rx=0 tx=0).stop 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 -- 172.21.15.116:0/1643062409 shutdown_connections 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f7750 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1643062409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e4ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 -- 172.21.15.116:0/1643062409 >> 172.21.15.116:0/1643062409 conn(0x55dd0e4e4ab0 msgr2=0x55dd0e4e4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 -- 172.21.15.116:0/1643062409 shutdown_connections 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 -- 172.21.15.116:0/1643062409 wait complete. 2024-03-19T02:42:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 -- start start 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.122+0000 7f53c01f1740 1 Processor -- start 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4f5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e4fa890 con 0x55dd0e4f7c90 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e40a1a0 con 0x55dd0e4ea030 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd0e4faa10 con 0x55dd0e4ea950 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45644/0 (socket says 172.21.15.116:45644) 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3066138111 learned_addr learned my addr 172.21.15.116:0/3066138111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53c01f1740 10 client.0 _ll_register_callbacks cb 0x55dd0e161c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3066138111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 msgr2=0x55dd0e4f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3066138111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4f7c90 msgr2=0x55dd0e4f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3066138111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0e28af70 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.123+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003970 tx=0x7f53a8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=923175f191aa97c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8005020 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a00c1550 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f53a80051c0 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f53a8030050 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 10 client.29585 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd0e122170 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53a00c1550 con 0x55dd0e4ea030 2024-03-19T02:42:25.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f53a800e750 con 0x55dd0e4ea030 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 10 client.29585 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 10 client.29585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 20 client.29585 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e507d70 0x55dd0e50a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 4 client.29585 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.124+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd0e45b6a0 con 0x55dd0e507d70 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.125+0000 7f53affff640 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e507d70 0x55dd0e50a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.125+0000 7f539e7fc640 10 client.29585 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.125+0000 7f53affff640 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e507d70 0x55dd0e50a1e0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f9c9030010b026c server_cookie=e08d734f4ae87a4e in_seq=0 out_seq=0 2024-03-19T02:42:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f539e7fc640 1 -- 172.21.15.116:0/3066138111 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd0e45b6a0 con 0x55dd0e507d70 2024-03-19T02:42:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f539e7fc640 10 client.29585 handle_command_reply: tid=0 2024-03-19T02:42:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 client.29585 shutdown 2024-03-19T02:42:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 msgr2=0x55dd0e4f4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003970 tx=0x7f53a8004730 comp rx=0 tx=0).stop 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 shutdown_connections 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e507d70 0x55dd0e50a1e0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e4f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3066138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4f4c10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f539e7fc640 0 client.29585 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 >> 172.21.15.116:0/3066138111 conn(0x55dd0e4e4ab0 msgr2=0x55dd0e4e6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 shutdown_connections 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.126+0000 7f53c01f1740 1 -- 172.21.15.116:0/3066138111 wait complete. 2024-03-19T02:42:25.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.127+0000 7f53c01f1740 20 client.29585 trim_cache size 0 max 16384 2024-03-19T02:42:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 Processor -- start 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 -- start start 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e50fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4ffc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81aca80 con 0x55dd0e4ea030 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.147+0000 7f53c01f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81ae6b0 con 0x55dd0e4f7c90 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53c01f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81ad150 con 0x55dd0e4ea950 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53affff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e50fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4ffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33498/0 (socket says 172.21.15.116:33498) 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3700100638 learned_addr learned my addr 172.21.15.116:0/3700100638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3700100638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 msgr2=0x55dd0e50fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e50fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3700100638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 msgr2=0x55dd0e4ffc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4ffc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 -- 172.21.15.116:0/3700100638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd0e4efa60 con 0x55dd0e4ea030 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009a10 tx=0x7f53a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458ebff9a7817891 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53acff9640 1 -- 172.21.15.116:0/3700100638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a800ed90 con 0x55dd0e4ea030 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53acff9640 1 -- 172.21.15.116:0/3700100638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53a80054e0 con 0x55dd0e4ea030 2024-03-19T02:42:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.148+0000 7f53acff9640 1 -- 172.21.15.116:0/3700100638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8004c00 con 0x55dd0e4ea030 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 -- 172.21.15.116:0/3700100638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 msgr2=0x55dd0e503630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009a10 tx=0x7f53a80046c0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 -- 172.21.15.116:0/3700100638 shutdown_connections 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4ea950 0x55dd0e50fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 --2- 172.21.15.116:0/3700100638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea030 0x55dd0e503630 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 -- 172.21.15.116:0/3700100638 >> 172.21.15.116:0/3700100638 conn(0x55dd0e4e4ab0 msgr2=0x55dd0e4f4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 -- 172.21.15.116:0/3700100638 shutdown_connections 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.149+0000 7f53c01f1740 1 -- 172.21.15.116:0/3700100638 wait complete. 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 -- start start 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 Processor -- start 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81adf20 con 0x55dd0e4ea950 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81ad850 con 0x55dd0e4ea030 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53b81aaa80 con 0x55dd0e4f7c90 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33508/0 (socket says 172.21.15.116:33508) 2024-03-19T02:42:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53c01f1740 10 client.0 _ll_register_callbacks cb 0x7f53a8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53b77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53affff640 1 -- 172.21.15.116:0/1375147282 learned_addr learned my addr 172.21.15.116:0/1375147282 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.150+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 -- 172.21.15.116:0/1375147282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4f7c90 msgr2=0x55dd0e4fd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 -- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 msgr2=0x55dd0e4fcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4fcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 -- 172.21.15.116:0/1375147282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd0e4ea780 con 0x55dd0e4ea950 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53affff640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f53a00ee5a0 tx=0x7f53a00b97d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6b3526333393a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f539e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00bd400 con 0x55dd0e4ea950 2024-03-19T02:42:25.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f53a00bb700 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a80054e0 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f53a00c1550 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53c01f1740 10 client.19830 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.151+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55dd0e56f2e0 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53a80054e0 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f53a0003b10 con 0x55dd0e4ea950 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 10 client.19830 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 10 client.19830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 20 client.19830 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e56f8a0 0x55dd0e571cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 4 client.19830 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd0e4559f0 con 0x55dd0e56f8a0 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.152+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e56f8a0 0x55dd0e571cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.153+0000 7f539e7fc640 10 client.19830 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.153+0000 7f53b6ffd640 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e56f8a0 0x55dd0e571cc0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6055162de2f35cb5 server_cookie=6214d5c00368b65a in_seq=0 out_seq=0 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.155+0000 7f539e7fc640 1 -- 172.21.15.116:0/1375147282 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd0e4559f0 con 0x55dd0e56f8a0 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.155+0000 7f539e7fc640 10 client.19830 handle_command_reply: tid=0 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.155+0000 7f53c01f1740 1 client.19830 shutdown 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 msgr2=0x55dd0e4fd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f53a00ee5a0 tx=0x7f53a00b97d0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 shutdown_connections 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd0e56f8a0 0x55dd0e571cc0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd0e4f7c90 0x55dd0e4fd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e4ea950 0x55dd0e4fd170 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 --2- 172.21.15.116:0/1375147282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd0e4ea030 0x55dd0e4fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f539e7fc640 0 client.19830 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 >> 172.21.15.116:0/1375147282 conn(0x55dd0e4e4ab0 msgr2=0x55dd0e4e3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 shutdown_connections 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 1 -- 172.21.15.116:0/1375147282 wait complete. 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.156+0000 7f53c01f1740 20 client.19830 trim_cache size 0 max 16384 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "849e6162-0c72-4f6e-a200-8d1074699557", 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:42:25.160 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:25.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 msgr2=0x7f53a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 0x7f53a40a4e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009b50 tx=0x7f53a8005f90 comp rx=0 tx=0).stop 2024-03-19T02:42:25.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 msgr2=0x7f53b8198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005940 tx=0x7f53a002c690 comp rx=0 tx=0).stop 2024-03-19T02:42:25.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 shutdown_connections 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f53a40a2970 0x7f53a40a4e30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b81055c0 0x7f53b819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53b8101080 0x7f53b8198540 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 --2- 172.21.15.116:0/2846257946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53b81006e0 0x7f53b8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 >> 172.21.15.116:0/2846257946 conn(0x7f53b80777b0 msgr2=0x7f53b80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 shutdown_connections 2024-03-19T02:42:25.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.157+0000 7f53bd52f640 1 -- 172.21.15.116:0/2846257946 wait complete. 2024-03-19T02:42:25.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:42:25.170 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 -- 172.21.15.116:0/3629815859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d00100680 msgr2=0x7f9d0010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 --2- 172.21.15.116:0/3629815859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d00100680 0x7f9d0010e640 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009ca0 tx=0x7f9cf802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 -- 172.21.15.116:0/3629815859 shutdown_connections 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 --2- 172.21.15.116:0/3629815859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d00100680 0x7f9d0010e640 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 --2- 172.21.15.116:0/3629815859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 --2- 172.21.15.116:0/3629815859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 0x7f9d00107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 -- 172.21.15.116:0/3629815859 >> 172.21.15.116:0/3629815859 conn(0x7f9d00076c40 msgr2=0x7f9d000fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 -- 172.21.15.116:0/3629815859 shutdown_connections 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.299+0000 7f9d04f90640 1 -- 172.21.15.116:0/3629815859 wait complete. 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 Processor -- start 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 -- start start 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d00100680 0x7f9d001013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 0x7f9d00101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d00111a60 con 0x7f9d000ffcc0 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001118e0 con 0x7f9d001077c0 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9d04f90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d00111be0 con 0x7f9d00100680 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33530/0 (socket says 172.21.15.116:33530) 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 -- 172.21.15.116:0/2441958750 learned_addr learned my addr 172.21.15.116:0/2441958750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 0x7f9d00101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d00100680 0x7f9d001013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 -- 172.21.15.116:0/2441958750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d00100680 msgr2=0x7f9d001013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d00100680 0x7f9d001013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 -- 172.21.15.116:0/2441958750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 msgr2=0x7f9d00101920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 0x7f9d00101920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 -- 172.21.15.116:0/2441958750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d001a6480 con 0x7f9d000ffcc0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e9f0 tx=0x7f9cf000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.300+0000 7f9cf77fe640 1 -- 172.21.15.116:0/2441958750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf000cd80 con 0x7f9d000ffcc0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.301+0000 7f9cf77fe640 1 -- 172.21.15.116:0/2441958750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf00108e0 con 0x7f9d000ffcc0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.301+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d001a6770 con 0x7f9d000ffcc0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.301+0000 7f9cf77fe640 1 -- 172.21.15.116:0/2441958750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0018920 con 0x7f9d000ffcc0 2024-03-19T02:42:25.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.301+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d001a6cc0 con 0x7f9d000ffcc0 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9cf77fe640 1 -- 172.21.15.116:0/2441958750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9cf0010a90 con 0x7f9d000ffcc0 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 Processor -- start 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 -- start start 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 0x5586e1b18320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5586e1a70500 con 0x5586e1b0ca30 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5586e1a18be0 con 0x5586e1b15f30 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9d07b76740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5586e1a6c5a0 con 0x5586e1b0d350 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9cfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9cfe59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9cfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33546/0 (socket says 172.21.15.116:33546) 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.303+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1520911519 learned_addr learned my addr 172.21.15.116:0/1520911519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 0x5586e1b18320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf77fe640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 0x7f9cec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1520911519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 msgr2=0x5586e1b159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1520911519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 msgr2=0x5586e1b18320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 0x5586e1b18320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1520911519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586e1a3a000 con 0x5586e1b0ca30 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 0x5586e1b18320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 0x7f9cec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf77fe640 1 -- 172.21.15.116:0/2441958750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f9cf00cf500 con 0x7f9d000ffcc0 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8037040 tx=0x7f9cf80323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32db1160cf00d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf4ff9640 1 -- 172.21.15.116:0/1520911519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8039040 con 0x5586e1b0ca30 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf4ff9640 1 -- 172.21.15.116:0/1520911519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf8003b70 con 0x5586e1b0ca30 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.304+0000 7f9cf4ff9640 1 -- 172.21.15.116:0/1520911519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80346a0 con 0x5586e1b0ca30 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- 172.21.15.116:0/1520911519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 msgr2=0x5586e1b0ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 0x7f9cec0a4e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9cf804c000 tx=0x7f9cf8045000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8037040 tx=0x7f9cf80323c0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- 172.21.15.116:0/1520911519 shutdown_connections 2024-03-19T02:42:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b15f30 0x5586e1b18320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1520911519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0ca30 0x5586e1b0ce10 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- 172.21.15.116:0/1520911519 >> 172.21.15.116:0/1520911519 conn(0x5586e1b07160 msgr2=0x5586e1b07540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- 172.21.15.116:0/1520911519 shutdown_connections 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- 172.21.15.116:0/1520911519 wait complete. 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 -- start start 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.305+0000 7f9d07b76740 1 Processor -- start 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0ca30 0x5586e1b12e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0d350 0x5586e1b133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5586e1b18d20 con 0x5586e1b15f30 2024-03-19T02:42:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5586e1a2c160 con 0x5586e1b0d350 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5586e1b18ea0 con 0x5586e1b0ca30 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 10 client.0 _ll_register_callbacks cb 0x5586e1a68a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33548/0 (socket says 172.21.15.116:33548) 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1740536505 learned_addr learned my addr 172.21.15.116:0/1740536505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0ca30 0x5586e1b12e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0d350 0x5586e1b133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1740536505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0ca30 msgr2=0x5586e1b12e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0ca30 0x5586e1b12e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0d350 msgr2=0x5586e1b133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0d350 0x5586e1b133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 -- 172.21.15.116:0/1740536505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586e184e7d0 con 0x5586e1b15f30 2024-03-19T02:42:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f9cf802c130 tx=0x7f9cf80098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc88b188725ef989 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8039070 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f9cf8003d20 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.306+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cf8003b70 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9cf8064050 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 10 client.19842 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5586e17843f0 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf8003b70 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f9cf803f040 con 0x5586e1b15f30 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 10 client.19842 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 10 client.19842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 20 client.19842 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b26060 0x5586e1b28450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 4 client.19842 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586e1a7da40 con 0x5586e1b26060 2024-03-19T02:42:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.307+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b26060 0x5586e1b28450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.308+0000 7f9ce5ffb640 10 client.19842 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.308+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b26060 0x5586e1b28450 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f875cdae1a650e4 server_cookie=fbde40648935df0a in_seq=0 out_seq=0 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/1740536505 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5586e1a7da40 con 0x5586e1b26060 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9ce5ffb640 10 client.19842 handle_command_reply: tid=0 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 client.19842 shutdown 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 msgr2=0x5586e1b138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f9cf802c130 tx=0x7f9cf80098e0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 shutdown_connections 2024-03-19T02:42:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b26060 0x5586e1b28450 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b138f0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9ce5ffb640 0 client.19842 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0d350 0x5586e1b133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 --2- 172.21.15.116:0/1740536505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0ca30 0x5586e1b12e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 >> 172.21.15.116:0/1740536505 conn(0x5586e1b07160 msgr2=0x5586e1b08eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 shutdown_connections 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 1 -- 172.21.15.116:0/1740536505 wait complete. 2024-03-19T02:42:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.311+0000 7f9d07b76740 20 client.19842 trim_cache size 0 max 16384 2024-03-19T02:42:25.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12638 7.98 MB/sec warmup 107 sec latency 190.808 ms 2024-03-19T02:42:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 Processor -- start 2024-03-19T02:42:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 -- start start 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b21ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 0x5586e1b08cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001b0e80 con 0x5586e1b0d350 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001b2ab0 con 0x5586e1b0ca30 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9d07b76740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001b1550 con 0x5586e1b15f30 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cfed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 0x5586e1b08cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33552/0 (socket says 172.21.15.116:33552) 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 -- 172.21.15.116:0/780620641 learned_addr learned my addr 172.21.15.116:0/780620641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b21ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 -- 172.21.15.116:0/780620641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 msgr2=0x5586e1b08cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 0x5586e1b08cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 -- 172.21.15.116:0/780620641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 msgr2=0x5586e1b21ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b21ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cf7fff640 1 -- 172.21.15.116:0/780620641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586e17440f0 con 0x5586e1b0d350 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.333+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 0x5586e1b08cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009bf0 tx=0x7f9ce8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1909794b15cecc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9ce7fff640 1 -- 172.21.15.116:0/780620641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80120d0 con 0x5586e1b0d350 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9ce7fff640 1 -- 172.21.15.116:0/780620641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ce8012270 con 0x5586e1b0d350 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9ce7fff640 1 -- 172.21.15.116:0/780620641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8010430 con 0x5586e1b0d350 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9d07b76740 1 -- 172.21.15.116:0/780620641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 msgr2=0x5586e1b13710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.334+0000 7f9d07b76740 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009bf0 tx=0x7f9ce8005f90 comp rx=0 tx=0).stop 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- 172.21.15.116:0/780620641 shutdown_connections 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b15f30 0x5586e1b08cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b0d350 0x5586e1b13710 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- 172.21.15.116:0/780620641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b21ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- 172.21.15.116:0/780620641 >> 172.21.15.116:0/780620641 conn(0x5586e1b07160 msgr2=0x5586e1b11db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- 172.21.15.116:0/780620641 shutdown_connections 2024-03-19T02:42:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- 172.21.15.116:0/780620641 wait complete. 2024-03-19T02:42:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- start start 2024-03-19T02:42:25.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 Processor -- start 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b1e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b0fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001b2320 con 0x5586e1b15f30 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001b1c50 con 0x5586e1b0ca30 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9d07b76740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d001aee80 con 0x5586e1b0d350 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.335+0000 7f9cf7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cf7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58654/0 (socket says 172.21.15.116:58654) 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cf7fff640 1 -- 172.21.15.116:0/2955847721 learned_addr learned my addr 172.21.15.116:0/2955847721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9d07b76740 10 client.0 _ll_register_callbacks cb 0x7f9cf00d2f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b0fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b1e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 -- 172.21.15.116:0/2955847721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 msgr2=0x5586e1b0f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 -- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 msgr2=0x5586e1b1e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b1e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 -- 172.21.15.116:0/2955847721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586e1b10530 con 0x5586e1b15f30 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cf7fff640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9cfed9d640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b0fd00 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8037040 tx=0x7f9cf80328c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcecba97c0158eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf806d060 con 0x5586e1b15f30 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f9cf8034be0 con 0x5586e1b15f30 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.336+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce8012270 con 0x5586e1b15f30 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9cf8039040 con 0x5586e1b15f30 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 10 client.19848 fetch_fsmap learned FSMap version 33 2024-03-19T02:42:25.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5586e1b94990 con 0x5586e1b15f30 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ce8012270 con 0x5586e1b15f30 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f9cf8003b10 con 0x5586e1b15f30 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 10 client.19848 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 10 client.19848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 20 client.19848 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b95080 0x5586e1b974a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 4 client.19848 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5586e1a77d90 con 0x5586e1b95080 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.337+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b95080 0x5586e1b974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.338+0000 7f9ce5ffb640 10 client.19848 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.338+0000 7f9cfe59c640 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b95080 0x5586e1b974a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42309e4a034233a server_cookie=5cc1ee060cdd457a in_seq=0 out_seq=0 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9ce5ffb640 1 -- 172.21.15.116:0/2955847721 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5586e1a77d90 con 0x5586e1b95080 2024-03-19T02:42:25.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9ce5ffb640 10 client.19848 handle_command_reply: tid=0 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 client.19848 shutdown 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 msgr2=0x5586e1b0fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b0fd00 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8037040 tx=0x7f9cf80328c0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 shutdown_connections 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5586e1b95080 0x5586e1b974a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5586e1b15f30 0x5586e1b0fd00 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5586e1b0d350 0x5586e1b0f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 --2- 172.21.15.116:0/2955847721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586e1b0ca30 0x5586e1b1e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9ce5ffb640 0 client.19848 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 >> 172.21.15.116:0/2955847721 conn(0x5586e1b07160 msgr2=0x5586e1b08cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 shutdown_connections 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 1 -- 172.21.15.116:0/2955847721 wait complete. 2024-03-19T02:42:25.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.340+0000 7f9d07b76740 20 client.19848 trim_cache size 0 max 16384 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (555 inodes in the stack)", 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "849e6162-0c72-4f6e-a200-8d1074699557": { 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "849e6162-0c72-4f6e-a200-8d1074699557", 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:42:25.346 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 msgr2=0x7f9cec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 0x7f9cec0a4e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9cf804c000 tx=0x7f9cf8045000 comp rx=0 tx=0).stop 2024-03-19T02:42:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 msgr2=0x7f9d00100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:25.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000e9f0 tx=0x7f9cf000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 shutdown_connections 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9cec0a2970 0x7f9cec0a4e30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d001077c0 0x7f9d00101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d00100680 0x7f9d001013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 --2- 172.21.15.116:0/2441958750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d000ffcc0 0x7f9d00100ea0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 >> 172.21.15.116:0/2441958750 conn(0x7f9d00076c40 msgr2=0x7f9d000fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 shutdown_connections 2024-03-19T02:42:25.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:25.341+0000 7f9d04f90640 1 -- 172.21.15.116:0/2441958750 wait complete. 2024-03-19T02:42:25.351 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:25 smithi116 ceph-mon[32263]: pgmap v459: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:25.352 INFO:tasks.cephfs.filesystem:scrub status for tag:849e6162-0c72-4f6e-a200-8d1074699557 - {'path': '/', 'tag': '849e6162-0c72-4f6e-a200-8d1074699557', 'options': 'recursive,force'} 2024-03-19T02:42:25.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:25 smithi162 ceph-mon[35761]: pgmap v459: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:25 smithi154 ceph-mon[35693]: pgmap v459: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:26.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12704 7.94 MB/sec warmup 108 sec latency 224.313 ms 2024-03-19T02:42:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:26 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:42:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:26 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:26 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:26 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:26 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:26 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:42:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:26 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:26 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:26 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:26 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:26 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:42:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:26 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:26 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:26 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:26 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12779 7.90 MB/sec warmup 109 sec latency 140.821 ms 2024-03-19T02:42:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:27 smithi162 ceph-mon[35761]: pgmap v460: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:27 smithi154 ceph-mon[35693]: pgmap v460: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:27 smithi116 ceph-mon[32263]: pgmap v460: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:28.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 12886 7.92 MB/sec warmup 110 sec latency 131.636 ms 2024-03-19T02:42:29.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13071 8.01 MB/sec warmup 111 sec latency 149.173 ms 2024-03-19T02:42:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:29 smithi162 ceph-mon[35761]: pgmap v461: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:29 smithi154 ceph-mon[35693]: pgmap v461: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:29.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:29 smithi116 ceph-mon[32263]: pgmap v461: 129 pgs: 129 active+clean; 821 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-03-19T02:42:30.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13182 8.02 MB/sec warmup 112 sec latency 167.828 ms 2024-03-19T02:42:31.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13296 8.02 MB/sec warmup 113 sec latency 171.726 ms 2024-03-19T02:42:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:31 smithi162 ceph-mon[35761]: pgmap v462: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-03-19T02:42:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:31 smithi154 ceph-mon[35693]: pgmap v462: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-03-19T02:42:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:31 smithi116 ceph-mon[32263]: pgmap v462: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.3 MiB/s wr, 189 op/s 2024-03-19T02:42:32.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13384 7.99 MB/sec warmup 114 sec latency 157.974 ms 2024-03-19T02:42:33.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13461 7.93 MB/sec warmup 115 sec latency 158.197 ms 2024-03-19T02:42:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:33 smithi162 ceph-mon[35761]: pgmap v463: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.4 MiB/s wr, 125 op/s 2024-03-19T02:42:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:33 smithi154 ceph-mon[35693]: pgmap v463: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.4 MiB/s wr, 125 op/s 2024-03-19T02:42:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:33 smithi116 ceph-mon[32263]: pgmap v463: 129 pgs: 129 active+clean; 827 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 3.4 MiB/s wr, 125 op/s 2024-03-19T02:42:34.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13543 7.87 MB/sec warmup 116 sec latency 164.439 ms 2024-03-19T02:42:35.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13630 7.82 MB/sec warmup 117 sec latency 184.157 ms 2024-03-19T02:42:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:35 smithi162 ceph-mon[35761]: pgmap v464: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:42:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:35 smithi154 ceph-mon[35693]: pgmap v464: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:42:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:35 smithi116 ceph-mon[32263]: pgmap v464: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:42:36.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13738 7.80 MB/sec warmup 118 sec latency 157.459 ms 2024-03-19T02:42:37.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 13843 7.75 MB/sec warmup 119 sec latency 235.631 ms 2024-03-19T02:42:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:37 smithi162 ceph-mon[35761]: pgmap v465: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:37 smithi154 ceph-mon[35693]: pgmap v465: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:37 smithi116 ceph-mon[32263]: pgmap v465: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:39.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:39 smithi116 ceph-mon[32263]: pgmap v466: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:39.321 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14065 3.70 MB/sec execute 1 sec latency 168.799 ms 2024-03-19T02:42:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:39 smithi162 ceph-mon[35761]: pgmap v466: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:39 smithi154 ceph-mon[35693]: pgmap v466: 129 pgs: 129 active+clean; 795 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 3.4 MiB/s wr, 119 op/s 2024-03-19T02:42:40.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14210 6.14 MB/sec execute 2 sec latency 163.948 ms 2024-03-19T02:42:40.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:40.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:40.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:40.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:41.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14283 5.12 MB/sec execute 3 sec latency 148.080 ms 2024-03-19T02:42:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:41 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:41 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:41 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:42:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:41 smithi162 ceph-mon[35761]: pgmap v467: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:41 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:41 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:41 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:42:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:41 smithi154 ceph-mon[35693]: pgmap v467: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:41 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:41 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:41 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:42:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:41 smithi116 ceph-mon[32263]: pgmap v467: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 213 op/s 2024-03-19T02:42:42.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14396 6.20 MB/sec execute 4 sec latency 155.849 ms 2024-03-19T02:42:43.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14507 5.60 MB/sec execute 5 sec latency 150.798 ms 2024-03-19T02:42:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:43 smithi162 ceph-mon[35761]: pgmap v468: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 151 op/s 2024-03-19T02:42:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:43 smithi154 ceph-mon[35693]: pgmap v468: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 151 op/s 2024-03-19T02:42:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:43 smithi116 ceph-mon[32263]: pgmap v468: 129 pgs: 129 active+clean; 784 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 151 op/s 2024-03-19T02:42:44.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14623 5.74 MB/sec execute 6 sec latency 139.171 ms 2024-03-19T02:42:45.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14754 6.72 MB/sec execute 7 sec latency 153.155 ms 2024-03-19T02:42:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:45 smithi162 ceph-mon[35761]: pgmap v469: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-03-19T02:42:45.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:45 smithi154 ceph-mon[35693]: pgmap v469: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-03-19T02:42:45.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:45 smithi116 ceph-mon[32263]: pgmap v469: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 223 op/s 2024-03-19T02:42:46.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 14881 6.83 MB/sec execute 8 sec latency 146.282 ms 2024-03-19T02:42:47.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15006 6.52 MB/sec execute 9 sec latency 149.366 ms 2024-03-19T02:42:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:47 smithi162 ceph-mon[35761]: pgmap v470: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:47 smithi154 ceph-mon[35693]: pgmap v470: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:47 smithi116 ceph-mon[32263]: pgmap v470: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:48.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15152 6.00 MB/sec execute 10 sec latency 139.794 ms 2024-03-19T02:42:49.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15298 5.68 MB/sec execute 11 sec latency 115.944 ms 2024-03-19T02:42:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:49 smithi162 ceph-mon[35761]: pgmap v471: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:49 smithi154 ceph-mon[35693]: pgmap v471: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:49.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:49 smithi116 ceph-mon[32263]: pgmap v471: 129 pgs: 129 active+clean; 788 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-19T02:42:50.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15431 5.63 MB/sec execute 12 sec latency 159.556 ms 2024-03-19T02:42:51.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15578 5.63 MB/sec execute 13 sec latency 165.306 ms 2024-03-19T02:42:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:51 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:51 smithi162 ceph-mon[35761]: pgmap v472: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 261 op/s 2024-03-19T02:42:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:51 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:51 smithi154 ceph-mon[35693]: pgmap v472: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 261 op/s 2024-03-19T02:42:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:51 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:51 smithi116 ceph-mon[32263]: pgmap v472: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 261 op/s 2024-03-19T02:42:52.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15692 5.45 MB/sec execute 14 sec latency 140.284 ms 2024-03-19T02:42:53.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:53 smithi154 ceph-mon[35693]: pgmap v473: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-03-19T02:42:53.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15830 5.81 MB/sec execute 15 sec latency 135.353 ms 2024-03-19T02:42:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:53 smithi162 ceph-mon[35761]: pgmap v473: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-03-19T02:42:53.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:53 smithi116 ceph-mon[32263]: pgmap v473: 129 pgs: 129 active+clean; 789 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-03-19T02:42:54.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 15931 5.72 MB/sec execute 16 sec latency 137.238 ms 2024-03-19T02:42:55.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16014 5.53 MB/sec execute 17 sec latency 182.882 ms 2024-03-19T02:42:55.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:42:55.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:42:55.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:55 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:55.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:55 smithi116 ceph-mon[32263]: pgmap v474: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 243 op/s 2024-03-19T02:42:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:55 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:55 smithi162 ceph-mon[35761]: pgmap v474: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 243 op/s 2024-03-19T02:42:55.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3052099834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 msgr2=0x7fe4ac101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3052099834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac101c40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe498009890 tx=0x7fe49802bb70 comp rx=0 tx=0).stop 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3052099834 shutdown_connections 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3052099834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3052099834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac101c40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3052099834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 0x7fe4ac107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3052099834 >> 172.21.15.116:0/3052099834 conn(0x7fe4ac0fd250 msgr2=0x7fe4ac0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3052099834 shutdown_connections 2024-03-19T02:42:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.482+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3052099834 wait complete. 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 Processor -- start 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 -- start start 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 0x7fe4ac0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac111a90 con 0x7fe4ac102180 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac111910 con 0x7fe4ac1077d0 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b2fc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac111c10 con 0x7fe4ac1017c0 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33578/0 (socket says 172.21.15.116:33578) 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b0d60640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4abfff640 1 -- 172.21.15.116:0/3304221197 learned_addr learned my addr 172.21.15.116:0/3304221197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.483+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 0x7fe4ac0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 -- 172.21.15.116:0/3304221197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 msgr2=0x7fe4ac073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 -- 172.21.15.116:0/3304221197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 msgr2=0x7fe4ac0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 0x7fe4ac0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 -- 172.21.15.116:0/3304221197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac1a6560 con 0x7fe4ac102180 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4abfff640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe498035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4a9ffb640 1 -- 172.21.15.116:0/3304221197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498041020 con 0x7fe4ac102180 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ac1a67f0 con 0x7fe4ac102180 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4a9ffb640 1 -- 172.21.15.116:0/3304221197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe49802c9c0 con 0x7fe4ac102180 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4a9ffb640 1 -- 172.21.15.116:0/3304221197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498039040 con 0x7fe4ac102180 2024-03-19T02:42:55.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.484+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ac1a6d30 con 0x7fe4ac102180 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.487+0000 7fe4a9ffb640 1 -- 172.21.15.116:0/3304221197 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe498044020 con 0x7fe4ac102180 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.487+0000 7fe4b5bac740 1 Processor -- start 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 -- start start 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 0x562839c2c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c2eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562839b7e500 con 0x562839c23240 2024-03-19T02:42:55.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562839b26be0 con 0x562839c23b60 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562839b7a5a0 con 0x562839c2c740 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b1561640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b1561640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33584/0 (socket says 172.21.15.116:33584) 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b1561640 1 -- 172.21.15.116:0/3025034090 learned_addr learned my addr 172.21.15.116:0/3025034090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4abfff640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c2eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4a9ffb640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 0x7fe49c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.488+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 0x562839c2c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4a9ffb640 1 -- 172.21.15.116:0/3304221197 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fe4980f08c0 con 0x7fe4ac102180 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 -- 172.21.15.116:0/3025034090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 msgr2=0x562839c2eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c2eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 -- 172.21.15.116:0/3025034090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 msgr2=0x562839c2c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 0x562839c2c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 -- 172.21.15.116:0/3025034090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562839b48000 con 0x562839c23240 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 0x7fe49c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 0x562839c2c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e520 tx=0x7fe4a400e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66b711fb5f4db8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe496ffd640 1 -- 172.21.15.116:0/3025034090 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a401f040 con 0x562839c23240 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe496ffd640 1 -- 172.21.15.116:0/3025034090 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4a4020020 con 0x562839c23240 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.489+0000 7fe496ffd640 1 -- 172.21.15.116:0/3025034090 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4004300 con 0x562839c23240 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- 172.21.15.116:0/3025034090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 msgr2=0x562839c23620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b1561640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 0x7fe49c0a4e30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4021000 tx=0x7fe4a4022000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400e520 tx=0x7fe4a400e9e0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- 172.21.15.116:0/3025034090 shutdown_connections 2024-03-19T02:42:55.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c2eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23b60 0x562839c2c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/3025034090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23240 0x562839c23620 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- 172.21.15.116:0/3025034090 >> 172.21.15.116:0/3025034090 conn(0x562839c1d970 msgr2=0x562839c1dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- 172.21.15.116:0/3025034090 shutdown_connections 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- 172.21.15.116:0/3025034090 wait complete. 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 -- start start 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.490+0000 7fe4b5bac740 1 Processor -- start 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c29700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 0x562839c2a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562839c2f530 con 0x562839c2c740 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562839b3a160 con 0x562839c23240 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562839c2f6b0 con 0x562839c23b60 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b5bac740 10 client.0 _ll_register_callbacks cb 0x562839b76a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b1561640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 0x562839c2a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b0d60640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c29700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53292/0 (socket says 172.21.15.116:53292) 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b0d60640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c29700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:58908/0 (socket says 172.21.15.116:58908) 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4b1561640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 0x562839c2a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33592/0 (socket says 172.21.15.116:33592) 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 -- 172.21.15.116:0/339614401 learned_addr learned my addr 172.21.15.116:0/339614401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 -- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 msgr2=0x562839c29700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c29700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 -- 172.21.15.116:0/339614401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 msgr2=0x562839c2a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 0x562839c2a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.491+0000 7fe4abfff640 1 -- 172.21.15.116:0/339614401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56283995c7d0 con 0x562839c23b60 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4abfff640 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe4980097b0 tx=0x7fe4980056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8383d55e7a4d5b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4980bd350 con 0x562839c23b60 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4a4020020 con 0x562839c23b60 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fe498003680 con 0x562839c23b60 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe4980bcc50 con 0x562839c23b60 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 10 client.29578 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:55.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5628398923f0 con 0x562839c23b60 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4a4020020 con 0x562839c23b60 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fe4980da020 con 0x562839c23b60 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 10 client.29578 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 10 client.29578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 20 client.29578 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839c3c970 0x562839c3edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 4 client.29578 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.492+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562839b8ba40 con 0x562839c3c970 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.493+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839c3c970 0x562839c3edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.493+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839c3c970 0x562839c3edd0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41a2024ce02ad91 server_cookie=1a1eb1d1c1de3926 in_seq=0 out_seq=0 2024-03-19T02:42:55.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.493+0000 7fe48bfff640 10 client.29578 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe48bfff640 1 -- 172.21.15.116:0/339614401 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562839b8ba40 con 0x562839c3c970 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe48bfff640 10 client.29578 handle_command_reply: tid=0 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe4b5bac740 1 client.29578 shutdown 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 msgr2=0x562839c29c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe4980097b0 tx=0x7fe4980056f0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 shutdown_connections 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.494+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839c3c970 0x562839c3edd0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c2c740 0x562839c2a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23b60 0x562839c29c40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/339614401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c29700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 >> 172.21.15.116:0/339614401 conn(0x562839c1d970 msgr2=0x562839c1f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe48bfff640 0 client.29578 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 shutdown_connections 2024-03-19T02:42:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 1 -- 172.21.15.116:0/339614401 wait complete. 2024-03-19T02:42:55.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.495+0000 7fe4b5bac740 20 client.29578 trim_cache size 0 max 16384 2024-03-19T02:42:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:55 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:55 smithi154 ceph-mon[35693]: pgmap v474: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 243 op/s 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.516+0000 7fe4b5bac740 1 Processor -- start 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 -- start start 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c2c740 0x562839c24a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1aca30 con 0x562839c23b60 2024-03-19T02:42:55.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1ae5d0 con 0x562839c2c740 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1ad100 con 0x562839c23240 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53306/0 (socket says 172.21.15.116:53306) 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 -- 172.21.15.116:0/716454227 learned_addr learned my addr 172.21.15.116:0/716454227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33604/0 (socket says 172.21.15.116:33604) 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b1561640 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c2c740 0x562839c24a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 -- 172.21.15.116:0/716454227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c2c740 msgr2=0x562839c24a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c2c740 0x562839c24a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 -- 172.21.15.116:0/716454227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 msgr2=0x562839c28660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 -- 172.21.15.116:0/716454227 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562839c23990 con 0x562839c23240 2024-03-19T02:42:55.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4abfff640 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.517+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009cf0 tx=0x7fe4a0014840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad1d630414701e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.518+0000 7fe4967fc640 1 -- 172.21.15.116:0/716454227 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00142d0 con 0x562839c23240 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.518+0000 7fe4967fc640 1 -- 172.21.15.116:0/716454227 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4a00054d0 con 0x562839c23240 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.518+0000 7fe4967fc640 1 -- 172.21.15.116:0/716454227 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0004d30 con 0x562839c23240 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- 172.21.15.116:0/716454227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 msgr2=0x562839c2be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009cf0 tx=0x7fe4a0014840 comp rx=0 tx=0).stop 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- 172.21.15.116:0/716454227 shutdown_connections 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c2c740 0x562839c24a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c28660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/716454227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c23240 0x562839c2be30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- 172.21.15.116:0/716454227 >> 172.21.15.116:0/716454227 conn(0x562839c1d970 msgr2=0x562839c141e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- 172.21.15.116:0/716454227 shutdown_connections 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- 172.21.15.116:0/716454227 wait complete. 2024-03-19T02:42:55.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 -- start start 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 Processor -- start 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.519+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c26f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1adea0 con 0x562839c23b60 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1ad7d0 con 0x562839c23240 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac1aaa50 con 0x562839c2c740 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 10 client.0 _ll_register_callbacks cb 0x7fe498005a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33618/0 (socket says 172.21.15.116:33618) 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 -- 172.21.15.116:0/1543331936 learned_addr learned my addr 172.21.15.116:0/1543331936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b1561640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c26f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 -- 172.21.15.116:0/1543331936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 msgr2=0x562839c26f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c26f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 -- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 msgr2=0x562839c264c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c264c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 -- 172.21.15.116:0/1543331936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628398520f0 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4abfff640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe4980b90b0 tx=0x7fe4980038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d4914c49ce9d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe48bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4980bd350 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fe49802c510 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4a00054d0 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe4980c0da0 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 10 client.19866 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.520+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562839ca2ff0 con 0x562839c23b60 2024-03-19T02:42:55.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4a00054d0 con 0x562839c23b60 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fe4980da040 con 0x562839c23b60 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 10 client.19866 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 10 client.19866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 20 client.19866 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839ca3550 0x562839ca5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 4 client.19866 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562839b85d90 con 0x562839ca3550 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839ca3550 0x562839ca5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe48bfff640 10 client.19866 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.521+0000 7fe4b0d60640 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839ca3550 0x562839ca5970 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71556a3078094004 server_cookie=a4b429ef4d8c2238 in_seq=0 out_seq=0 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe48bfff640 1 -- 172.21.15.116:0/1543331936 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562839b85d90 con 0x562839ca3550 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe48bfff640 10 client.19866 handle_command_reply: tid=0 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 client.19866 shutdown 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 msgr2=0x562839c26a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fe4980b90b0 tx=0x7fe4980038b0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 shutdown_connections 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562839ca3550 0x562839ca5970 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562839c2c740 0x562839c26f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562839c23b60 0x562839c26a00 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe48bfff640 0 client.19866 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 --2- 172.21.15.116:0/1543331936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562839c23240 0x562839c264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 >> 172.21.15.116:0/1543331936 conn(0x562839c1d970 msgr2=0x562839c1ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 shutdown_connections 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 1 -- 172.21.15.116:0/1543331936 wait complete. 2024-03-19T02:42:55.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.523+0000 7fe4b5bac740 20 client.19866 trim_cache size 0 max 16384 2024-03-19T02:42:55.535 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:55.535 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:42:55.535 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:42:55.535 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 msgr2=0x7fe49c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 0x7fe49c0a4e30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4021000 tx=0x7fe4a4022000 comp rx=0 tx=0).stop 2024-03-19T02:42:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 msgr2=0x7fe4ac074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe498035990 comp rx=0 tx=0).stop 2024-03-19T02:42:55.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 shutdown_connections 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe49c0a2970 0x7fe49c0a4e30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4ac1077d0 0x7fe4ac0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4ac102180 0x7fe4ac074060 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 --2- 172.21.15.116:0/3304221197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4ac1017c0 0x7fe4ac073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 >> 172.21.15.116:0/3304221197 conn(0x7fe4ac0fd250 msgr2=0x7fe4ac0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 shutdown_connections 2024-03-19T02:42:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.524+0000 7fe4b2fc4640 1 -- 172.21.15.116:0/3304221197 wait complete. 2024-03-19T02:42:55.537 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:42:55.537 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:42:55.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.701+0000 7f9c6de12640 1 -- 172.21.15.116:0/3390929000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 msgr2=0x7f9c68100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 --2- 172.21.15.116:0/3390929000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68100ba0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009930 tx=0x7f9c5c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 -- 172.21.15.116:0/3390929000 shutdown_connections 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 --2- 172.21.15.116:0/3390929000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c681010e0 0x7f9c6810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 --2- 172.21.15.116:0/3390929000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68100ba0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 --2- 172.21.15.116:0/3390929000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c68106730 0x7f9c68106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 -- 172.21.15.116:0/3390929000 >> 172.21.15.116:0/3390929000 conn(0x7f9c68077820 msgr2=0x7f9c68077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 -- 172.21.15.116:0/3390929000 shutdown_connections 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 -- 172.21.15.116:0/3390929000 wait complete. 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 Processor -- start 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.702+0000 7f9c6de12640 1 -- start start 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c681010e0 0x7f9c681993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 0x7f9c681a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681109f0 con 0x7f9c68100720 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68110870 con 0x7f9c681010e0 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c6de12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68110b70 con 0x7f9c68106730 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33642/0 (socket says 172.21.15.116:33642) 2024-03-19T02:42:55.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c66ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c681010e0 0x7f9c681993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 0x7f9c681a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 -- 172.21.15.116:0/588031026 learned_addr learned my addr 172.21.15.116:0/588031026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 -- 172.21.15.116:0/588031026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 msgr2=0x7f9c681a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 0x7f9c681a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 -- 172.21.15.116:0/588031026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c681010e0 msgr2=0x7f9c681993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c681010e0 0x7f9c681993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 -- 172.21.15.116:0/588031026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c681a4430 con 0x7f9c68100720 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c67fff640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 0x7f9c681a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c677fe640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800e9f0 tx=0x7f9c5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.703+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5800cd80 con 0x7f9c68100720 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.704+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681a4660 con 0x7f9c68100720 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.704+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c580108e0 con 0x7f9c68100720 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.704+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58018920 con 0x7f9c68100720 2024-03-19T02:42:55.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.704+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c681a4bd0 con 0x7f9c68100720 2024-03-19T02:42:55.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.705+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c68068610 con 0x7f9c68100720 2024-03-19T02:42:55.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.706+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9c58010a90 con 0x7f9c68100720 2024-03-19T02:42:55.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.706+0000 7f9c64ff9640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 0x7f9c500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:55.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.707+0000 7f9c66ffd640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 0x7f9c500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:55.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.707+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f9c580cf4f0 con 0x7f9c68100720 2024-03-19T02:42:55.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.707+0000 7f9c66ffd640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 0x7f9c500a4e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c005ab0 tx=0x7f9c5c005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:55.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:55.711+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c5809b820 con 0x7f9c68100720 2024-03-19T02:42:56.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.006+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c68108cb0 con 0x7f9c68100720 2024-03-19T02:42:56.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.007+0000 7f9c64ff9640 1 -- 172.21.15.116:0/588031026 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11953 (secure 0 0 0) 0x7f9c5809f8c0 con 0x7f9c68100720 2024-03-19T02:42:56.008 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:42:56.008 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:42:54.492490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:42:56.008 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 36 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 msgr2=0x7f9c500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 0x7f9c500a4e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c005ab0 tx=0x7f9c5c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 msgr2=0x7f9c68198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800e9f0 tx=0x7f9c5800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 shutdown_connections 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9c500a2970 0x7f9c500a4e30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c68106730 0x7f9c681a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c681010e0 0x7f9c681993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 --2- 172.21.15.116:0/588031026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c68100720 0x7f9c68198e70 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 >> 172.21.15.116:0/588031026 conn(0x7f9c68077820 msgr2=0x7f9c68102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 shutdown_connections 2024-03-19T02:42:56.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.009+0000 7f9c6de12640 1 -- 172.21.15.116:0/588031026 wait complete. 2024-03-19T02:42:56.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:42:56.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 -- 172.21.15.116:0/3201561899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c1006a0 msgr2=0x7fd23c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 --2- 172.21.15.116:0/3201561899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c1006a0 0x7fd23c100b20 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd230009930 tx=0x7fd23002bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 -- 172.21.15.116:0/3201561899 shutdown_connections 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 --2- 172.21.15.116:0/3201561899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c101060 0x7fd23c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 --2- 172.21.15.116:0/3201561899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c1006a0 0x7fd23c100b20 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 --2- 172.21.15.116:0/3201561899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 0x7fd23c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 -- 172.21.15.116:0/3201561899 >> 172.21.15.116:0/3201561899 conn(0x7fd23c0778d0 msgr2=0x7fd23c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 -- 172.21.15.116:0/3201561899 shutdown_connections 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.145+0000 7fd241ab5640 1 -- 172.21.15.116:0/3201561899 wait complete. 2024-03-19T02:42:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.146+0000 7fd241ab5640 1 Processor -- start 2024-03-19T02:42:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.146+0000 7fd241ab5640 1 -- start start 2024-03-19T02:42:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.146+0000 7fd241ab5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 0x7fd23c196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.146+0000 7fd241ab5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd241ab5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 0x7fd23c19b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd241ab5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c110980 con 0x7fd23c101060 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd241ab5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c110800 con 0x7fd23c1006a0 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd241ab5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c110b00 con 0x7fd23c1066b0 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33670/0 (socket says 172.21.15.116:33670) 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 0x7fd23c196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 -- 172.21.15.116:0/2001455130 learned_addr learned my addr 172.21.15.116:0/2001455130 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 0x7fd23c19b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 -- 172.21.15.116:0/2001455130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 msgr2=0x7fd23c19b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 0x7fd23c19b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 -- 172.21.15.116:0/2001455130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 msgr2=0x7fd23c196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 0x7fd23c196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 -- 172.21.15.116:0/2001455130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c19be90 con 0x7fd23c101060 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 0x7fd23c196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd23affd640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd230000c00 tx=0x7fd23002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.147+0000 7fd238ff9640 1 -- 172.21.15.116:0/2001455130 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd230043020 con 0x7fd23c101060 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.148+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd23c19c120 con 0x7fd23c101060 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.148+0000 7fd238ff9640 1 -- 172.21.15.116:0/2001455130 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd23002caf0 con 0x7fd23c101060 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.148+0000 7fd238ff9640 1 -- 172.21.15.116:0/2001455130 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd230003a80 con 0x7fd23c101060 2024-03-19T02:42:56.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.148+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd23c1a3a10 con 0x7fd23c101060 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.150+0000 7fd238ff9640 1 -- 172.21.15.116:0/2001455130 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd230046020 con 0x7fd23c101060 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 Processor -- start 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 -- start start 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd238ff9640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 0x7fd2240a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 0x55fe201321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe200854b0 con 0x55fe20121b30 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe2002dd70 con 0x55fe2012fdd0 2024-03-19T02:42:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd244776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe20081550 con 0x55fe20122450 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 0x7fd2240a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd238ff9640 1 -- 172.21.15.116:0/2001455130 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd2300f1090 con 0x7fd23c101060 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 0x55fe201321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 0x55fe201321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:58966/0 (socket says 172.21.15.116:58966) 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33674/0 (socket says 172.21.15.116:33674) 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23bfff640 1 -- 172.21.15.116:0/3483085792 learned_addr learned my addr 172.21.15.116:0/3483085792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.151+0000 7fd23affd640 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3483085792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 msgr2=0x55fe2012f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3483085792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 msgr2=0x55fe201321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 0x55fe201321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3483085792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe20050210 con 0x55fe20121b30 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23affd640 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 0x7fd2240a4eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd2340138b0 tx=0x7fd234009b80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009f50 tx=0x7fd22c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdba279b9a1716e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd221ffb640 1 -- 172.21.15.116:0/3483085792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c004230 con 0x55fe20121b30 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd221ffb640 1 -- 172.21.15.116:0/3483085792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd22c0043d0 con 0x55fe20121b30 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.152+0000 7fd221ffb640 1 -- 172.21.15.116:0/3483085792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0035f0 con 0x55fe20121b30 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- 172.21.15.116:0/3483085792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 msgr2=0x55fe20121f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009f50 tx=0x7fd22c011810 comp rx=0 tx=0).stop 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- 172.21.15.116:0/3483085792 shutdown_connections 2024-03-19T02:42:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe2012fdd0 0x55fe201321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 --2- 172.21.15.116:0/3483085792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe20121b30 0x55fe20121f10 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- 172.21.15.116:0/3483085792 >> 172.21.15.116:0/3483085792 conn(0x55fe2011ca00 msgr2=0x55fe2011cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- 172.21.15.116:0/3483085792 shutdown_connections 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- 172.21.15.116:0/3483085792 wait complete. 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 -- start start 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.153+0000 7fd244776740 1 Processor -- start 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe20132a00 con 0x55fe2012fdd0 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe20041310 con 0x55fe20121b30 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fe20132b80 con 0x55fe20122450 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd244776740 10 client.0 _ll_register_callbacks cb 0x55fe200352d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:58976/0 (socket says 172.21.15.116:58976) 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23b7fe640 1 -- 172.21.15.116:0/175898022 learned_addr learned my addr 172.21.15.116:0/175898022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23bfff640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 -- 172.21.15.116:0/175898022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 msgr2=0x55fe2012cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 -- 172.21.15.116:0/175898022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 msgr2=0x55fe2012d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23affd640 1 -- 172.21.15.116:0/175898022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1fd993f0 con 0x55fe20122450 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23bfff640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.154+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd23affd640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012d2a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd2300ee510 tx=0x7fd230005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f490d4facee8d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2300c1320 con 0x55fe20122450 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd22c0043d0 con 0x55fe20122450 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd2300bd680 con 0x55fe20122450 2024-03-19T02:42:56.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd2300c1570 con 0x55fe20122450 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 10 client.29602 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fe1fe637d0 con 0x55fe20122450 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd22c0043d0 con 0x55fe20122450 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fd2300c5730 con 0x55fe20122450 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 10 client.29602 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 10 client.29602 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 20 client.29602 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe2013fed0 0x55fe20142340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 4 client.29602 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:56.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.155+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe200e33e0 con 0x55fe2013fed0 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.156+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe2013fed0 0x55fe20142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.156+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe2013fed0 0x55fe20142340 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=967816a9f251f107 server_cookie=94eebecb38a7ada3 in_seq=0 out_seq=0 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.156+0000 7fd21affd640 10 client.29602 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd21affd640 1 -- 172.21.15.116:0/175898022 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe200e33e0 con 0x55fe2013fed0 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd21affd640 10 client.29602 handle_command_reply: tid=0 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 client.29602 shutdown 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 msgr2=0x55fe2012d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012d2a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd2300ee510 tx=0x7fd230005fd0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 shutdown_connections 2024-03-19T02:42:56.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe2013fed0 0x55fe20142340 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe2012d2a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd21affd640 0 client.29602 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 --2- 172.21.15.116:0/175898022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe2012cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 >> 172.21.15.116:0/175898022 conn(0x55fe2011ca00 msgr2=0x55fe2011e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 shutdown_connections 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 1 -- 172.21.15.116:0/175898022 wait complete. 2024-03-19T02:42:56.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.157+0000 7fd244776740 20 client.29602 trim_cache size 0 max 16384 2024-03-19T02:42:56.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.175+0000 7fd244776740 1 Processor -- start 2024-03-19T02:42:56.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.175+0000 7fd244776740 1 -- start start 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20122450 0x55fe2013b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1acad0 con 0x55fe2012fdd0 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1ae670 con 0x55fe20122450 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd244776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1ad1a0 con 0x55fe20121b30 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53366/0 (socket says 172.21.15.116:53366) 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3932415635 learned_addr learned my addr 172.21.15.116:0/3932415635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23affd640 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20122450 0x55fe2013b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3932415635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20122450 msgr2=0x55fe2013b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20122450 0x55fe2013b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3932415635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 msgr2=0x55fe2012a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 -- 172.21.15.116:0/3932415635 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe201429c0 con 0x55fe20121b30 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.176+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009730 tx=0x7fd22c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc4c077455cd9383 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd2217fa640 1 -- 172.21.15.116:0/3932415635 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c01e040 con 0x55fe20121b30 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd2217fa640 1 -- 172.21.15.116:0/3932415635 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd22c01f040 con 0x55fe20121b30 2024-03-19T02:42:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd2217fa640 1 -- 172.21.15.116:0/3932415635 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c005240 con 0x55fe20121b30 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 -- 172.21.15.116:0/3932415635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 msgr2=0x55fe2012f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009730 tx=0x7fd22c011810 comp rx=0 tx=0).stop 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 -- 172.21.15.116:0/3932415635 shutdown_connections 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe2012a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20122450 0x55fe2013b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 --2- 172.21.15.116:0/3932415635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20121b30 0x55fe2012f490 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 -- 172.21.15.116:0/3932415635 >> 172.21.15.116:0/3932415635 conn(0x55fe2011ca00 msgr2=0x55fe2011b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 -- 172.21.15.116:0/3932415635 shutdown_connections 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.177+0000 7fd244776740 1 -- 172.21.15.116:0/3932415635 wait complete. 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 -- start start 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 Processor -- start 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe20147050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe20147590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe20147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1adf40 con 0x55fe2012fdd0 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1ad870 con 0x55fe20121b30 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c1aaad0 con 0x55fe20122450 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd244776740 10 client.0 _ll_register_callbacks cb 0x7fd2300bd1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe20147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe20147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd23b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe20147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59000/0 (socket says 172.21.15.116:59000) 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd23b7fe640 1 -- 172.21.15.116:0/2628224979 learned_addr learned my addr 172.21.15.116:0/2628224979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.178+0000 7fd23bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe20147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 -- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 msgr2=0x55fe20147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe20147590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 -- 172.21.15.116:0/2628224979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 msgr2=0x55fe20147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe20147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 -- 172.21.15.116:0/2628224979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe20122ae0 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23affd640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe20147590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd23bfff640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe20147ad0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd234009e20 tx=0x7fd234004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e956817abb698 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd21affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234004b20 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fd234026050 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd22c01f040 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd234037050 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd244776740 10 client.19890 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.179+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fe201a6340 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd22c01f040 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fd23401f070 con 0x55fe2012fdd0 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 10 client.19890 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 10 client.19890 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:42:56.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 20 client.19890 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe201a6b10 0x55fe201a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 4 client.19890 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe200929f0 con 0x55fe201a6b10 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe201a6b10 0x55fe201a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd21affd640 10 client.19890 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.180+0000 7fd23b7fe640 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe201a6b10 0x55fe201a8f30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1dd02a2da3c21b88 server_cookie=5d149702713b544c in_seq=0 out_seq=0 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd21affd640 1 -- 172.21.15.116:0/2628224979 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe200929f0 con 0x55fe201a6b10 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd21affd640 10 client.19890 handle_command_reply: tid=0 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 client.19890 shutdown 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 msgr2=0x55fe20147ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe20147ad0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fd234009e20 tx=0x7fd234004ab0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 shutdown_connections 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55fe201a6b10 0x55fe201a8f30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2012fdd0 0x55fe20147ad0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd21affd640 0 client.19890 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fe20122450 0x55fe20147590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 --2- 172.21.15.116:0/2628224979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe20121b30 0x55fe20147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 >> 172.21.15.116:0/2628224979 conn(0x55fe2011ca00 msgr2=0x55fe2012d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 shutdown_connections 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 1 -- 172.21.15.116:0/2628224979 wait complete. 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd244776740 20 client.19890 trim_cache size 0 max 16384 2024-03-19T02:42:56.188 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 msgr2=0x7fd2240a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.181+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 0x7fd2240a4eb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd2340138b0 tx=0x7fd234009b80 comp rx=0 tx=0).stop 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 msgr2=0x7fd23c1973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd230000c00 tx=0x7fd23002c660 comp rx=0 tx=0).stop 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 shutdown_connections 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2240a29f0 0x7fd2240a4eb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd23c1066b0 0x7fd23c19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd23c101060 0x7fd23c1973e0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 --2- 172.21.15.116:0/2001455130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd23c1006a0 0x7fd23c196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 >> 172.21.15.116:0/2001455130 conn(0x7fd23c0778d0 msgr2=0x7fd23c102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 shutdown_connections 2024-03-19T02:42:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.182+0000 7fd241ab5640 1 -- 172.21.15.116:0/2001455130 wait complete. 2024-03-19T02:42:56.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 -- 172.21.15.116:0/2260093646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41066b0 msgr2=0x7f1de4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 --2- 172.21.15.116:0/2260093646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41066b0 0x7f1de4106a90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009930 tx=0x7f1dd402bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 -- 172.21.15.116:0/2260093646 shutdown_connections 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 --2- 172.21.15.116:0/2260093646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de4101060 0x7f1de410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 --2- 172.21.15.116:0/2260093646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de41006a0 0x7f1de4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 --2- 172.21.15.116:0/2260093646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41066b0 0x7f1de4106a90 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 -- 172.21.15.116:0/2260093646 >> 172.21.15.116:0/2260093646 conn(0x7f1de40778d0 msgr2=0x7f1de4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 -- 172.21.15.116:0/2260093646 shutdown_connections 2024-03-19T02:42:56.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.320+0000 7f1dea212640 1 -- 172.21.15.116:0/2260093646 wait complete. 2024-03-19T02:42:56.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.321+0000 7f1dea212640 1 Processor -- start 2024-03-19T02:42:56.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.321+0000 7f1dea212640 1 -- start start 2024-03-19T02:42:56.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16118 5.52 MB/sec execute 18 sec latency 146.906 ms 2024-03-19T02:42:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 0x7f1de4107b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de41066b0 0x7f1de4108090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4110970 con 0x7f1de41006a0 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41107f0 con 0x7f1de4101060 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1dea212640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1de4110af0 con 0x7f1de41066b0 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33708/0 (socket says 172.21.15.116:33708) 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 0x7f1de4107b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 -- 172.21.15.116:0/2288287124 learned_addr learned my addr 172.21.15.116:0/2288287124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de3fff640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de41066b0 0x7f1de4108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 -- 172.21.15.116:0/2288287124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de41066b0 msgr2=0x7f1de4108090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de41066b0 0x7f1de4108090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 -- 172.21.15.116:0/2288287124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 msgr2=0x7f1de4107b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 0x7f1de4107b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 -- 172.21.15.116:0/2288287124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de41a6490 con 0x7f1de41006a0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 0x7f1de4107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.322+0000 7f1de37fe640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009230 tx=0x7f1dd402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.323+0000 7f1de0ff9640 1 -- 172.21.15.116:0/2288287124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4043020 con 0x7f1de41006a0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.323+0000 7f1de0ff9640 1 -- 172.21.15.116:0/2288287124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dd402ca70 con 0x7f1de41006a0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.323+0000 7f1de0ff9640 1 -- 172.21.15.116:0/2288287124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4003a50 con 0x7f1de41006a0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.323+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de41a6720 con 0x7f1de41006a0 2024-03-19T02:42:56.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.323+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de41a6c20 con 0x7f1de41006a0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 Processor -- start 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de0ff9640 1 -- 172.21.15.116:0/2288287124 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1dd4046020 con 0x7f1de41006a0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 -- start start 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da68290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da6abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55736d9c24b0 con 0x55736da687d0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55736d96ad70 con 0x55736da5fbf0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1dece01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55736d9be550 con 0x55736da5f2d0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da6abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da68290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53392/0 (socket says 172.21.15.116:53392) 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de3fff640 1 -- 172.21.15.116:0/882001389 learned_addr learned my addr 172.21.15.116:0/882001389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.326+0000 7f1de3fff640 1 -- 172.21.15.116:0/882001389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 msgr2=0x55736da68290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de3fff640 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da68290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de3fff640 1 -- 172.21.15.116:0/882001389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 msgr2=0x55736da6abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de3fff640 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da6abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de0ff9640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 0x7f1dcc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de3fff640 1 -- 172.21.15.116:0/882001389 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736d98d210 con 0x55736da5f2d0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de37fe640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 0x7f1dcc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de3fff640 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009f50 tx=0x7f1ddc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3468f888dd89bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1de0ff9640 1 -- 172.21.15.116:0/2288287124 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f1dd40f10e0 con 0x7f1de41006a0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1dc9ffb640 1 -- 172.21.15.116:0/882001389 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc004d20 con 0x55736da5f2d0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1dc9ffb640 1 -- 172.21.15.116:0/882001389 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ddc004050 con 0x55736da5f2d0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.327+0000 7f1dc9ffb640 1 -- 172.21.15.116:0/882001389 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc005280 con 0x55736da5f2d0 2024-03-19T02:42:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 -- 172.21.15.116:0/882001389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 msgr2=0x55736da5f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009f50 tx=0x7f1ddc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1de37fe640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 0x7f1dcc0a4e60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40ee610 tx=0x7f1dd40b9040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 -- 172.21.15.116:0/882001389 shutdown_connections 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da6abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da68290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 --2- 172.21.15.116:0/882001389 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da5f6b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 -- 172.21.15.116:0/882001389 >> 172.21.15.116:0/882001389 conn(0x55736da59a00 msgr2=0x55736da59de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 -- 172.21.15.116:0/882001389 shutdown_connections 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.328+0000 7f1dece01740 1 -- 172.21.15.116:0/882001389 wait complete. 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.329+0000 7f1dece01740 1 -- start start 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.329+0000 7f1dece01740 1 Processor -- start 2024-03-19T02:42:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.329+0000 7f1dece01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5f2d0 0x55736da65860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.329+0000 7f1dece01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dece01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dece01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55736da6b5c0 con 0x55736da687d0 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dece01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55736d97e310 con 0x55736da5f2d0 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53408/0 (socket says 172.21.15.116:53408) 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de2ffd640 1 -- 172.21.15.116:0/2530122088 learned_addr learned my addr 172.21.15.116:0/2530122088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dece01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55736da6b740 con 0x55736da5fbf0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dece01740 10 client.0 _ll_register_callbacks cb 0x55736d9722d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5f2d0 0x55736da65860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 -- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 msgr2=0x55736da65da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 -- 172.21.15.116:0/2530122088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5f2d0 msgr2=0x55736da65860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5f2d0 0x55736da65860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 -- 172.21.15.116:0/2530122088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736d6d63f0 con 0x55736da687d0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1de3fff640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da662e0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc003fd0 tx=0x7f1ddc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6fa5744ff44539 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.330+0000 7f1dc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc004a00 con 0x55736da687d0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ddc004050 con 0x55736da687d0 2024-03-19T02:42:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f1ddc004ba0 con 0x55736da687d0 2024-03-19T02:42:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1ddc028450 con 0x55736da687d0 2024-03-19T02:42:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dece01740 10 client.19902 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55736d7a07d0 con 0x55736da687d0 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.331+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ddc004050 con 0x55736da687d0 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f1ddc027640 con 0x55736da687d0 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 10 client.19902 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 10 client.19902 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 20 client.19902 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736da812e0 0x55736da83710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 4 client.19902 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:56.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55736da203e0 con 0x55736da812e0 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.332+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736da812e0 0x55736da83710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.333+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736da812e0 0x55736da83710 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bda9a481223aa1ac server_cookie=c66127510f372c64 in_seq=0 out_seq=0 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.333+0000 7f1dc2ffd640 10 client.19902 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.334+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/2530122088 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55736da203e0 con 0x55736da812e0 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.334+0000 7f1dc2ffd640 10 client.19902 handle_command_reply: tid=0 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.334+0000 7f1dece01740 1 client.19902 shutdown 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.335+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 msgr2=0x55736da662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.335+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da662e0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc003fd0 tx=0x7f1ddc0047b0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 shutdown_connections 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736da812e0 0x55736da83710 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da662e0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da65da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dc2ffd640 0 client.19902 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 --2- 172.21.15.116:0/2530122088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5f2d0 0x55736da65860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 >> 172.21.15.116:0/2530122088 conn(0x55736da59a00 msgr2=0x55736da5b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 shutdown_connections 2024-03-19T02:42:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 1 -- 172.21.15.116:0/2530122088 wait complete. 2024-03-19T02:42:56.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.336+0000 7f1dece01740 20 client.19902 trim_cache size 0 max 16384 2024-03-19T02:42:56.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 Processor -- start 2024-03-19T02:42:56.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 -- start start 2024-03-19T02:42:56.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da67690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41aca70 con 0x55736da687d0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41ae610 con 0x55736da5fbf0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1dece01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41ad140 con 0x55736da5f2d0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1de2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da67690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33744/0 (socket says 172.21.15.116:33744) 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1de3fff640 1 -- 172.21.15.116:0/347742847 learned_addr learned my addr 172.21.15.116:0/347742847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.354+0000 7f1de3fff640 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 -- 172.21.15.116:0/347742847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 msgr2=0x55736da7cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da7cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 -- 172.21.15.116:0/347742847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 msgr2=0x55736da67690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da67690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 -- 172.21.15.116:0/347742847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55736da5a5d0 con 0x55736da687d0 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1de3fff640 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc003fa0 tx=0x7f1ddc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47346ae567dba67a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1dc97fa640 1 -- 172.21.15.116:0/347742847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc009d90 con 0x55736da687d0 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1dc97fa640 1 -- 172.21.15.116:0/347742847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ddc037040 con 0x55736da687d0 2024-03-19T02:42:56.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.355+0000 7f1dc97fa640 1 -- 172.21.15.116:0/347742847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc027440 con 0x55736da687d0 2024-03-19T02:42:56.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 -- 172.21.15.116:0/347742847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 msgr2=0x55736da88fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc003fa0 tx=0x7f1ddc009500 comp rx=0 tx=0).stop 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 -- 172.21.15.116:0/347742847 shutdown_connections 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da687d0 0x55736da88fe0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da5fbf0 0x55736da67690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 --2- 172.21.15.116:0/347742847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5f2d0 0x55736da7cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 -- 172.21.15.116:0/347742847 >> 172.21.15.116:0/347742847 conn(0x55736da59a00 msgr2=0x55736da66f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 -- 172.21.15.116:0/347742847 shutdown_connections 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.356+0000 7f1dece01740 1 -- 172.21.15.116:0/347742847 wait complete. 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 -- start start 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 Processor -- start 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da61af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da687d0 0x55736da62030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41adee0 con 0x55736da5f2d0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41ad810 con 0x55736da687d0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1de41aaaf0 con 0x55736da5fbf0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1dece01740 10 client.0 _ll_register_callbacks cb 0x7f1dd8009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33758/0 (socket says 172.21.15.116:33758) 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1de3fff640 1 -- 172.21.15.116:0/3038305423 learned_addr learned my addr 172.21.15.116:0/3038305423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da61af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.357+0000 7f1de3fff640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da687d0 0x55736da62030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 -- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 msgr2=0x55736da61af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da61af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 -- 172.21.15.116:0/3038305423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da687d0 msgr2=0x55736da62030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da687d0 0x55736da62030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 -- 172.21.15.116:0/3038305423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55736da63680 con 0x55736da5f2d0 2024-03-19T02:42:56.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da61af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1de3fff640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc011ed0 tx=0x7f1ddc009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414c90438df58797 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc027cb0 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f1ddc0285f0 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ddc037040 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1ddc027e50 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dece01740 10 client.19908 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.358+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55736dae6f10 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ddc037040 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f1ddc048070 con 0x55736da5f2d0 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 10 client.19908 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 10 client.19908 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:42:56.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 20 client.19908 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736dae7560 0x55736dae9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 4 client.19908 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55736d9cf9f0 con 0x55736dae7560 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736dae7560 0x55736dae9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1dc2ffd640 10 client.19908 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.359+0000 7f1de2ffd640 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736dae7560 0x55736dae9950 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=711ed927fc0bf9d7 server_cookie=a327a82b7487939f in_seq=0 out_seq=0 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dc2ffd640 1 -- 172.21.15.116:0/3038305423 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55736d9cf9f0 con 0x55736dae7560 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dc2ffd640 10 client.19908 handle_command_reply: tid=0 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 client.19908 shutdown 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 msgr2=0x55736da79220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc011ed0 tx=0x7f1ddc009d90 comp rx=0 tx=0).stop 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 shutdown_connections 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55736da687d0 0x55736da62030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55736da5fbf0 0x55736da61af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55736dae7560 0x55736dae9950 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 --2- 172.21.15.116:0/3038305423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55736da5f2d0 0x55736da79220 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dc2ffd640 0 client.19908 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 >> 172.21.15.116:0/3038305423 conn(0x55736da59a00 msgr2=0x55736da660b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 shutdown_connections 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 1 -- 172.21.15.116:0/3038305423 wait complete. 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.360+0000 7f1dece01740 20 client.19908 trim_cache size 0 max 16384 2024-03-19T02:42:56.368 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 msgr2=0x7f1dcc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 0x7f1dcc0a4e60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40ee610 tx=0x7f1dd40b9040 comp rx=0 tx=0).stop 2024-03-19T02:42:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 msgr2=0x7f1de4107610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009230 tx=0x7f1dd402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 shutdown_connections 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1dcc0a29a0 0x7f1dcc0a4e60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1de41066b0 0x7f1de4108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1de4101060 0x7f1de4107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 --2- 172.21.15.116:0/2288287124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de41006a0 0x7f1de4107610 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 >> 172.21.15.116:0/2288287124 conn(0x7f1de40778d0 msgr2=0x7f1de4103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 shutdown_connections 2024-03-19T02:42:56.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.361+0000 7f1dea212640 1 -- 172.21.15.116:0/2288287124 wait complete. 2024-03-19T02:42:56.374 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:56 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:56.374 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:56 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/588031026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:56.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:42:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:56 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:56 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/588031026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 -- 172.21.15.116:0/3767885329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 msgr2=0x7f4390106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 --2- 172.21.15.116:0/3767885329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f4390106a90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f438c009930 tx=0x7f438c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 -- 172.21.15.116:0/3767885329 shutdown_connections 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 --2- 172.21.15.116:0/3767885329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 0x7f439010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 --2- 172.21.15.116:0/3767885329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 --2- 172.21.15.116:0/3767885329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f4390106a90 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.497+0000 7f4397b59640 1 -- 172.21.15.116:0/3767885329 >> 172.21.15.116:0/3767885329 conn(0x7f43900778d0 msgr2=0x7f4390077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- 172.21.15.116:0/3767885329 shutdown_connections 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- 172.21.15.116:0/3767885329 wait complete. 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 Processor -- start 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- start start 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 0x7f4390199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4390110970 con 0x7f43901066b0 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43901107f0 con 0x7f4390101060 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f4397b59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4390110af0 con 0x7f43901006a0 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f43960f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f43958f5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.498+0000 7f43960f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33784/0 (socket says 172.21.15.116:33784) 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43960f6640 1 -- 172.21.15.116:0/3749866822 learned_addr learned my addr 172.21.15.116:0/3749866822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43950f4640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 0x7f4390199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 -- 172.21.15.116:0/3749866822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 msgr2=0x7f4390199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 0x7f4390199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 -- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 msgr2=0x7f43901a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 -- 172.21.15.116:0/3749866822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43901a4410 con 0x7f43901006a0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43960f6640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f43958f5640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390198e50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f438c009230 tx=0x7f438c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f4386ffd640 1 -- 172.21.15.116:0/3749866822 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c043020 con 0x7f43901006a0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43901a45e0 con 0x7f43901006a0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f4386ffd640 1 -- 172.21.15.116:0/3749866822 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f438c02ca70 con 0x7f43901006a0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43901a4aa0 con 0x7f43901006a0 2024-03-19T02:42:56.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.499+0000 7f4386ffd640 1 -- 172.21.15.116:0/3749866822 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c003a60 con 0x7f43901006a0 2024-03-19T02:42:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:56 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:42:56.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:56 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/588031026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:42:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.502+0000 7f4386ffd640 1 -- 172.21.15.116:0/3749866822 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f438c046020 con 0x7f43901006a0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.502+0000 7f439a812740 1 Processor -- start 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.502+0000 7f439a812740 1 -- start start 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x556883050290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x556883052bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556882faa4b0 con 0x5568830472d0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556882f52d70 con 0x556883047bf0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f439a812740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556882fa6550 con 0x5568830507d0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33796/0 (socket says 172.21.15.116:33796) 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43958f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x556883050290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 -- 172.21.15.116:0/1377514361 learned_addr learned my addr 172.21.15.116:0/1377514361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43950f4640 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x556883052bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f4386ffd640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 0x7f437c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 -- 172.21.15.116:0/1377514361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 msgr2=0x556883052bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x556883052bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 -- 172.21.15.116:0/1377514361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 msgr2=0x556883050290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x556883050290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 -- 172.21.15.116:0/1377514361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556882f75210 con 0x5568830472d0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f4386ffd640 1 -- 172.21.15.116:0/3749866822 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f438c0f10d0 con 0x7f43901006a0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43958f5640 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x556883050290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 0x7f437c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.503+0000 7f43960f6640 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f43880118e0 tx=0x7f4388011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4279833ee54a3659 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f437bfff640 1 -- 172.21.15.116:0/1377514361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388014c00 con 0x5568830472d0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f437bfff640 1 -- 172.21.15.116:0/1377514361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4388014da0 con 0x5568830472d0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f437bfff640 1 -- 172.21.15.116:0/1377514361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438801c410 con 0x5568830472d0 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f43960f6640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 0x7f437c0a4e30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4388022000 tx=0x7f438801f000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 -- 172.21.15.116:0/1377514361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 msgr2=0x5568830476b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f43880118e0 tx=0x7f4388011da0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 -- 172.21.15.116:0/1377514361 shutdown_connections 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x556883052bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x556883050290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 --2- 172.21.15.116:0/1377514361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830476b0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.504+0000 7f439a812740 1 -- 172.21.15.116:0/1377514361 >> 172.21.15.116:0/1377514361 conn(0x556883041a00 msgr2=0x556883041de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- 172.21.15.116:0/1377514361 shutdown_connections 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- 172.21.15.116:0/1377514361 wait complete. 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- start start 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 Processor -- start 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x55688304dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x55688304e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568830535c0 con 0x5568830472d0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556882f66310 con 0x556883047bf0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556883053740 con 0x5568830507d0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f439a812740 10 client.0 _ll_register_callbacks cb 0x556882f5a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33798/0 (socket says 172.21.15.116:33798) 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.505+0000 7f43958f5640 1 -- 172.21.15.116:0/2657748270 learned_addr learned my addr 172.21.15.116:0/2657748270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43950f4640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x55688304dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43960f6640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x55688304e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 -- 172.21.15.116:0/2657748270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 msgr2=0x55688304e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x55688304e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 -- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 msgr2=0x55688304dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x55688304dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 -- 172.21.15.116:0/2657748270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556882cbe3f0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43958f5640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f438c006010 tx=0x7f438c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29f7d4d5f3bf761 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bcdc0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f438c0053b0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4388014da0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f438c0bdb60 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f439a812740 10 client.19920 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556882d887d0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4388014da0 con 0x5568830472d0 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.506+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f438c0c1150 con 0x5568830472d0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 10 client.19920 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 10 client.19920 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 20 client.19920 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556883060ac0 0x556883062f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 4 client.19920 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568830083e0 con 0x556883060ac0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f43950f4640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556883060ac0 0x556883062f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f43797fa640 10 client.19920 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.507+0000 7f43950f4640 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556883060ac0 0x556883062f10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ea03113136bedad server_cookie=cfee9c2034e0e706 in_seq=0 out_seq=0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f43797fa640 1 -- 172.21.15.116:0/2657748270 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568830083e0 con 0x556883060ac0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f43797fa640 10 client.19920 handle_command_reply: tid=0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 client.19920 shutdown 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 msgr2=0x55688304d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f438c006010 tx=0x7f438c009560 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 shutdown_connections 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x556883060ac0 0x556883062f10 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x55688304e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x55688304dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f43797fa640 0 client.19920 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 --2- 172.21.15.116:0/2657748270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304d7d0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 >> 172.21.15.116:0/2657748270 conn(0x556883041a00 msgr2=0x556883043750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 shutdown_connections 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 1 -- 172.21.15.116:0/2657748270 wait complete. 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.508+0000 7f439a812740 20 client.19920 trim_cache size 0 max 16384 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.521+0000 7f439a812740 1 Processor -- start 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 -- start start 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556883047bf0 0x55688305c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 0x556883068790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43901aebb0 con 0x5568830472d0 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b0750 con 0x5568830507d0 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f439a812740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f43901af280 con 0x556883047bf0 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33804/0 (socket says 172.21.15.116:33804) 2024-03-19T02:42:56.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 -- 172.21.15.116:0/3176853205 learned_addr learned my addr 172.21.15.116:0/3176853205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43950f4640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556883047bf0 0x55688305c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43960f6640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 0x556883068790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 -- 172.21.15.116:0/3176853205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556883047bf0 msgr2=0x55688305c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556883047bf0 0x55688305c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 -- 172.21.15.116:0/3176853205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 msgr2=0x556883068790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 0x556883068790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 -- 172.21.15.116:0/3176853205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556883047a20 con 0x5568830472d0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43960f6640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 0x556883068790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f43958f5640 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b9e30 tx=0x7f438c0bda30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdb0b9a6c421e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f437b7fe640 1 -- 172.21.15.116:0/3176853205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bcdc0 con 0x5568830472d0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f437b7fe640 1 -- 172.21.15.116:0/3176853205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f438c0bd460 con 0x5568830472d0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.522+0000 7f437b7fe640 1 -- 172.21.15.116:0/3176853205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bd230 con 0x5568830472d0 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- 172.21.15.116:0/3176853205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 msgr2=0x55688304fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b9e30 tx=0x7f438c0bda30 comp rx=0 tx=0).stop 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- 172.21.15.116:0/3176853205 shutdown_connections 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568830507d0 0x556883068790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556883047bf0 0x55688305c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 --2- 172.21.15.116:0/3176853205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x55688304fe90 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- 172.21.15.116:0/3176853205 >> 172.21.15.116:0/3176853205 conn(0x556883041a00 msgr2=0x556883040ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- 172.21.15.116:0/3176853205 shutdown_connections 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- 172.21.15.116:0/3176853205 wait complete. 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 -- start start 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.523+0000 7f439a812740 1 Processor -- start 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x5568830c0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x5568830549e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b0020 con 0x5568830472d0 2024-03-19T02:42:56.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43901af950 con 0x556883047bf0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f43901acbb0 con 0x5568830507d0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f439a812740 10 client.0 _ll_register_callbacks cb 0x7f438c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43960f6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x5568830549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33820/0 (socket says 172.21.15.116:33820) 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 -- 172.21.15.116:0/994467636 learned_addr learned my addr 172.21.15.116:0/994467636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43950f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x5568830c0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 -- 172.21.15.116:0/994467636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 msgr2=0x5568830549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x5568830549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 -- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 msgr2=0x5568830c0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x5568830c0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.524+0000 7f43958f5640 1 -- 172.21.15.116:0/994467636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568830550a0 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43958f5640 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c170 tx=0x7f438c0bcdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d22f714908bed15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0c15b0 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f438c0053b0 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438c0bd460 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f438c0c5c00 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 10 client.19926 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5568830550a0 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f438c0bd460 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f438c0dd0b0 con 0x5568830472d0 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 10 client.19926 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 10 client.19926 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 20 client.19926 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5568830c7aa0 0x5568830c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 4 client.19926 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556882fb79f0 con 0x5568830c7aa0 2024-03-19T02:42:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.525+0000 7f43950f4640 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5568830c7aa0 0x5568830c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.526+0000 7f43797fa640 10 client.19926 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:56.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.526+0000 7f43950f4640 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5568830c7aa0 0x5568830c9e90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=888b1266a2a3309 server_cookie=2dbb8c65310ef667 in_seq=0 out_seq=0 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f43797fa640 1 -- 172.21.15.116:0/994467636 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556882fb79f0 con 0x5568830c7aa0 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f43797fa640 10 client.19926 handle_command_reply: tid=0 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 client.19926 shutdown 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 msgr2=0x5568830c00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c170 tx=0x7f438c0bcdc0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 shutdown_connections 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5568830c7aa0 0x5568830c9e90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568830507d0 0x5568830549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556883047bf0 0x5568830c0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f43797fa640 0 client.19926 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 --2- 172.21.15.116:0/994467636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568830472d0 0x5568830c00d0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 >> 172.21.15.116:0/994467636 conn(0x556883041a00 msgr2=0x55688304d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 shutdown_connections 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 1 -- 172.21.15.116:0/994467636 wait complete. 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.527+0000 7f439a812740 20 client.19926 trim_cache size 0 max 16384 2024-03-19T02:42:56.534 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 msgr2=0x7f437c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 0x7f437c0a4e30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4388022000 tx=0x7f438801f000 comp rx=0 tx=0).stop 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 msgr2=0x7f4390198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390198e50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f438c009230 tx=0x7f438c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 shutdown_connections 2024-03-19T02:42:56.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f437c0a2970 0x7f437c0a4e30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43901066b0 0x7f43901a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390101060 0x7f4390199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 --2- 172.21.15.116:0/3749866822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43901006a0 0x7f4390198e50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 >> 172.21.15.116:0/3749866822 conn(0x7f43900778d0 msgr2=0x7f4390102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 shutdown_connections 2024-03-19T02:42:56.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.528+0000 7f4397b59640 1 -- 172.21.15.116:0/3749866822 wait complete. 2024-03-19T02:42:56.541 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:42:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.668+0000 7f261ac36640 1 -- 172.21.15.116:0/3107353391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26141066b0 msgr2=0x7f2614106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.668+0000 7f261ac36640 1 --2- 172.21.15.116:0/3107353391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26141066b0 0x7f2614106a90 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f2608009930 tx=0x7f260802bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- 172.21.15.116:0/3107353391 shutdown_connections 2024-03-19T02:42:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- 172.21.15.116:0/3107353391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2614101060 0x7f261410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- 172.21.15.116:0/3107353391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141006a0 0x7f2614100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- 172.21.15.116:0/3107353391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26141066b0 0x7f2614106a90 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- 172.21.15.116:0/3107353391 >> 172.21.15.116:0/3107353391 conn(0x7f26140778d0 msgr2=0x7f2614077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- 172.21.15.116:0/3107353391 shutdown_connections 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- 172.21.15.116:0/3107353391 wait complete. 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 Processor -- start 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- start start 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26141006a0 0x7f2614196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 0x7f26141976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2614110970 con 0x7f2614101060 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26141107f0 con 0x7f26141006a0 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f261ac36640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2614110af0 con 0x7f26141066b0 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.669+0000 7f2613fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f26191d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 0x7f26141976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33848/0 (socket says 172.21.15.116:33848) 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 -- 172.21.15.116:0/194774421 learned_addr learned my addr 172.21.15.116:0/194774421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f26191d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 0x7f26141976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53516/0 (socket says 172.21.15.116:53516) 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f26189d2640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26141006a0 0x7f2614196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 -- 172.21.15.116:0/194774421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 msgr2=0x7f26141976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 0x7f26141976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 -- 172.21.15.116:0/194774421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26141006a0 msgr2=0x7f2614196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26141006a0 0x7f2614196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 -- 172.21.15.116:0/194774421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26141a6420 con 0x7f2614101060 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2613fff640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f260000ea20 tx=0x7f260000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2611ffb640 1 -- 172.21.15.116:0/194774421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260000cd80 con 0x7f2614101060 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2611ffb640 1 -- 172.21.15.116:0/194774421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26000054d0 con 0x7f2614101060 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26141a6710 con 0x7f2614101060 2024-03-19T02:42:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f2611ffb640 1 -- 172.21.15.116:0/194774421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600010430 con 0x7f2614101060 2024-03-19T02:42:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.670+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141a6c10 con 0x7f2614101060 2024-03-19T02:42:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.672+0000 7f2611ffb640 1 -- 172.21.15.116:0/194774421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2600005020 con 0x7f2614101060 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 Processor -- start 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f2611ffb640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 0x7f26040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 -- start start 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26191d3640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 0x7f26040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f2611ffb640 1 -- 172.21.15.116:0/194774421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f26000cf590 con 0x7f2614101060 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e97290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e99bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561eb5df14b0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561eb5d99d70 con 0x561eb5e8ebf0 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f261d827740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561eb5ded550 con 0x561eb5e977d0 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f2613fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e97290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33864/0 (socket says 172.21.15.116:33864) 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 -- 172.21.15.116:0/2284206231 learned_addr learned my addr 172.21.15.116:0/2284206231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26191d3640 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e99bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26191d3640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 0x7f26040a4e30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f260c005fd0 tx=0x7f260c011810 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 -- 172.21.15.116:0/2284206231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 msgr2=0x561eb5e99bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e99bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 -- 172.21.15.116:0/2284206231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 msgr2=0x561eb5e97290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e97290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f26189d2640 1 -- 172.21.15.116:0/2284206231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eb5dbc210 con 0x561eb5e8e2d0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.673+0000 7f2613fff640 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e97290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.674+0000 7f26189d2640 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2608036040 tx=0x7f2608009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b235b152fd514ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.674+0000 7f25feffd640 1 -- 172.21.15.116:0/2284206231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608045020 con 0x561eb5e8e2d0 2024-03-19T02:42:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.674+0000 7f25feffd640 1 -- 172.21.15.116:0/2284206231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f260802c7e0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.674+0000 7f25feffd640 1 -- 172.21.15.116:0/2284206231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26080054c0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- 172.21.15.116:0/2284206231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 msgr2=0x561eb5e8e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f2608036040 tx=0x7f2608009560 comp rx=0 tx=0).stop 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- 172.21.15.116:0/2284206231 shutdown_connections 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e99bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e97290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 --2- 172.21.15.116:0/2284206231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e8e6b0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- 172.21.15.116:0/2284206231 >> 172.21.15.116:0/2284206231 conn(0x561eb5e88a00 msgr2=0x561eb5e88de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- 172.21.15.116:0/2284206231 shutdown_connections 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- 172.21.15.116:0/2284206231 wait complete. 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 -- start start 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.675+0000 7f261d827740 1 Processor -- start 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e94da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 0x561eb5e952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561eb5e9a560 con 0x561eb5e977d0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561eb5dad310 con 0x561eb5e8ebf0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561eb5e9a6e0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26191d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 0x561eb5e952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53524/0 (socket says 172.21.15.116:53524) 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 -- 172.21.15.116:0/1290692575 learned_addr learned my addr 172.21.15.116:0/1290692575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f2613fff640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e94da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f261d827740 10 client.0 _ll_register_callbacks cb 0x561eb5da12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 -- 172.21.15.116:0/1290692575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 msgr2=0x561eb5e94da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e94da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 -- 172.21.15.116:0/1290692575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 msgr2=0x561eb5e952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 0x561eb5e952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 -- 172.21.15.116:0/1290692575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eb5b053f0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f2613fff640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e94da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26191d3640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 0x561eb5e952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f26189d2640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f26080054e0 tx=0x7f2608005510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74aa393a547f3a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f25f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608045030 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f260802c7e0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.676+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f2608004450 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f260805b050 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 10 client.29656 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561eb5bcf7d0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f260802c7e0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f2608004880 con 0x561eb5e8e2d0 2024-03-19T02:42:56.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 10 client.29656 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 10 client.29656 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 20 client.29656 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5ea79a0 0x561eb5ea9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 4 client.29656 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561eb5e4f3e0 con 0x561eb5ea79a0 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.677+0000 7f2613fff640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5ea79a0 0x561eb5ea9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.678+0000 7f25f3fff640 10 client.29656 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.678+0000 7f2613fff640 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5ea79a0 0x561eb5ea9d90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55e0a3c2c6bf080 server_cookie=459721da7403e6a4 in_seq=0 out_seq=0 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f25f3fff640 1 -- 172.21.15.116:0/1290692575 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561eb5e4f3e0 con 0x561eb5ea79a0 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f25f3fff640 10 client.29656 handle_command_reply: tid=0 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 client.29656 shutdown 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 msgr2=0x561eb5e94860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f26080054e0 tx=0x7f2608005510 comp rx=0 tx=0).stop 2024-03-19T02:42:56.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 shutdown_connections 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5ea79a0 0x561eb5ea9d90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e977d0 0x561eb5e952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e94da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 --2- 172.21.15.116:0/1290692575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e8e2d0 0x561eb5e94860 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f25f3fff640 0 client.29656 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 >> 172.21.15.116:0/1290692575 conn(0x561eb5e88a00 msgr2=0x561eb5e8a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 shutdown_connections 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 1 -- 172.21.15.116:0/1290692575 wait complete. 2024-03-19T02:42:56.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.679+0000 7f261d827740 20 client.29656 trim_cache size 0 max 16384 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 Processor -- start 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 -- start start 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 0x561eb5e96f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e95100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f26141b0df0 con 0x561eb5e8ebf0 2024-03-19T02:42:56.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26141b2990 con 0x561eb5e8e2d0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f261d827740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26141b14c0 con 0x561eb5e977d0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f2613fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f26189d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 0x561eb5e96f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f2613fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33880/0 (socket says 172.21.15.116:33880) 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f2613fff640 1 -- 172.21.15.116:0/3074249270 learned_addr learned my addr 172.21.15.116:0/3074249270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.695+0000 7f26191d3640 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e95100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 -- 172.21.15.116:0/3074249270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 msgr2=0x561eb5e95100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e95100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 -- 172.21.15.116:0/3074249270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 msgr2=0x561eb5e96f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 0x561eb5e96f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 -- 172.21.15.116:0/3074249270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eb5e99f30 con 0x561eb5e8ebf0 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f26189d2640 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 0x561eb5e96f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f2613fff640 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f260000e9f0 tx=0x7f26000097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606d920e716b0d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f25fe7fc640 1 -- 172.21.15.116:0/3074249270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26000b9070 con 0x561eb5e8ebf0 2024-03-19T02:42:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f25fe7fc640 1 -- 172.21.15.116:0/3074249270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26000b3030 con 0x561eb5e8ebf0 2024-03-19T02:42:56.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.696+0000 7f25fe7fc640 1 -- 172.21.15.116:0/3074249270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26000ccb80 con 0x561eb5e8ebf0 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- 172.21.15.116:0/3074249270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 msgr2=0x561eb5ea3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f260000e9f0 tx=0x7f26000097b0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- 172.21.15.116:0/3074249270 shutdown_connections 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e95100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8ebf0 0x561eb5ea3300 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 --2- 172.21.15.116:0/3074249270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8e2d0 0x561eb5e96f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- 172.21.15.116:0/3074249270 >> 172.21.15.116:0/3074249270 conn(0x561eb5e88a00 msgr2=0x561eb5e93890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- 172.21.15.116:0/3074249270 shutdown_connections 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- 172.21.15.116:0/3074249270 wait complete. 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 -- start start 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.697+0000 7f261d827740 1 Processor -- start 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e9f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e9a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f26141b2260 con 0x561eb5e8e2d0 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26141b1b90 con 0x561eb5e8ebf0 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26141aedf0 con 0x561eb5e977d0 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f261d827740 10 client.0 _ll_register_callbacks cb 0x7f2600004a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33894/0 (socket says 172.21.15.116:33894) 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 -- 172.21.15.116:0/1786955224 learned_addr learned my addr 172.21.15.116:0/1786955224 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26191d3640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e9a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f2613fff640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e9f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 -- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 msgr2=0x561eb5e9a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e9a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 -- 172.21.15.116:0/1786955224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 msgr2=0x561eb5e9f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e9f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 -- 172.21.15.116:0/1786955224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eb5e9b0f0 con 0x561eb5e8e2d0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26191d3640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e9a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f26189d2640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f2608005db0 tx=0x7f260802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733863552922f400 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.698+0000 7f25f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608045040 con 0x561eb5e8e2d0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f2608004450 con 0x561eb5e8e2d0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26000b3030 con 0x561eb5e8e2d0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2608005050 con 0x561eb5e8e2d0 2024-03-19T02:42:56.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 10 client.19944 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561eb5f0d520 con 0x561eb5e8e2d0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26000b3030 con 0x561eb5e8e2d0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f2608003a80 con 0x561eb5e8e2d0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 10 client.19944 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 10 client.19944 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 20 client.19944 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5f0dad0 0x561eb5f0fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 4 client.19944 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561eb5dfe9f0 con 0x561eb5f0dad0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.699+0000 7f2613fff640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5f0dad0 0x561eb5f0fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f25f3fff640 10 client.19944 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f2613fff640 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5f0dad0 0x561eb5f0fef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=737c4bc87cb74029 server_cookie=a1108f5488245e54 in_seq=0 out_seq=0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f25f3fff640 1 -- 172.21.15.116:0/1786955224 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561eb5dfe9f0 con 0x561eb5f0dad0 2024-03-19T02:42:56.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f25f3fff640 10 client.19944 handle_command_reply: tid=0 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 client.19944 shutdown 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 msgr2=0x561eb5e9f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f2608005db0 tx=0x7f260802c7e0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 shutdown_connections 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561eb5f0dad0 0x561eb5f0fef0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561eb5e977d0 0x561eb5e9a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561eb5e8ebf0 0x561eb5e9f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 --2- 172.21.15.116:0/1786955224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561eb5e8e2d0 0x561eb5e9f450 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f25f3fff640 0 client.19944 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 >> 172.21.15.116:0/1786955224 conn(0x561eb5e88a00 msgr2=0x561eb5e87310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 shutdown_connections 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 1 -- 172.21.15.116:0/1786955224 wait complete. 2024-03-19T02:42:56.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.700+0000 7f261d827740 20 client.19944 trim_cache size 0 max 16384 2024-03-19T02:42:56.708 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 msgr2=0x7f26040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 0x7f26040a4e30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f260c005fd0 tx=0x7f260c011810 comp rx=0 tx=0).stop 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 msgr2=0x7f2614197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f260000ea20 tx=0x7f260000ee20 comp rx=0 tx=0).stop 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 shutdown_connections 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26040a2970 0x7f26040a4e30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26141066b0 0x7f26141976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2614101060 0x7f2614197180 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 --2- 172.21.15.116:0/194774421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26141006a0 0x7f2614196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 >> 172.21.15.116:0/194774421 conn(0x7f26140778d0 msgr2=0x7f2614102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 shutdown_connections 2024-03-19T02:42:56.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.701+0000 7f261ac36640 1 -- 172.21.15.116:0/194774421 wait complete. 2024-03-19T02:42:56.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:42:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 -- 172.21.15.116:0/1334714744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c105690 msgr2=0x7f6a5c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1334714744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c105690 0x7f6a5c105a70 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009ca0 tx=0x7f6a5402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 -- 172.21.15.116:0/1334714744 shutdown_connections 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1334714744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c101330 0x7f6a5c10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1334714744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c100970 0x7f6a5c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1334714744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c105690 0x7f6a5c105a70 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 -- 172.21.15.116:0/1334714744 >> 172.21.15.116:0/1334714744 conn(0x7f6a5c077580 msgr2=0x7f6a5c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 -- 172.21.15.116:0/1334714744 shutdown_connections 2024-03-19T02:42:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.842+0000 7f6a60a19640 1 -- 172.21.15.116:0/1334714744 wait complete. 2024-03-19T02:42:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 Processor -- start 2024-03-19T02:42:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 -- start start 2024-03-19T02:42:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 0x7f6a5c196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c105690 0x7f6a5c19b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c110980 con 0x7f6a5c101330 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c110800 con 0x7f6a5c105690 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.843+0000 7f6a60a19640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c110b00 con 0x7f6a5c100970 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a59d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a59d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33918/0 (socket says 172.21.15.116:33918) 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a59d9b640 1 -- 172.21.15.116:0/1336868853 learned_addr learned my addr 172.21.15.116:0/1336868853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 0x7f6a5c196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c105690 0x7f6a5c19b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 -- 172.21.15.116:0/1336868853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c105690 msgr2=0x7f6a5c19b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c105690 0x7f6a5c19b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 -- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 msgr2=0x7f6a5c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 -- 172.21.15.116:0/1336868853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5c19be30 con 0x7f6a5c100970 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 0x7f6a5c196e40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006010 tx=0x7f6a54035d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a4b7fe640 1 -- 172.21.15.116:0/1336868853 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54039040 con 0x7f6a5c100970 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a5c19c0c0 con 0x7f6a5c100970 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a4b7fe640 1 -- 172.21.15.116:0/1336868853 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a540329a0 con 0x7f6a5c100970 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a4b7fe640 1 -- 172.21.15.116:0/1336868853 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a540042a0 con 0x7f6a5c100970 2024-03-19T02:42:56.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.844+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a5c1a39b0 con 0x7f6a5c100970 2024-03-19T02:42:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.847+0000 7f6a4b7fe640 1 -- 172.21.15.116:0/1336868853 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6a54046030 con 0x7f6a5c100970 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.847+0000 7f6a63606740 1 Processor -- start 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 -- start start 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a4b7fe640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 0x7f6a440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee48290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 0x55d70ee4abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d70eda24b0 con 0x55d70ee3f2d0 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d70ed4ad70 con 0x55d70ee487d0 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a63606740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d70ed9e550 con 0x55d70ee3fbf0 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33926/0 (socket says 172.21.15.116:33926) 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 -- 172.21.15.116:0/4259757853 learned_addr learned my addr 172.21.15.116:0/4259757853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a4b7fe640 1 -- 172.21.15.116:0/1336868853 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f6a540f0b30 con 0x7f6a5c100970 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 0x7f6a440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee48290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 -- 172.21.15.116:0/4259757853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 msgr2=0x55d70ee48290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee48290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 -- 172.21.15.116:0/4259757853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 msgr2=0x55d70ee4abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 0x55d70ee4abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 0x55d70ee4abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5a59c640 1 -- 172.21.15.116:0/4259757853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70ed6d210 con 0x55d70ee3f2d0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.848+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 0x55d70ee4abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.849+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f6a5402c630 tx=0x7f6a54035f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e416db0da404c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.849+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 0x7f6a440a4e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55d70ee3ab90 tx=0x7f6a4c009ad0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.849+0000 7f6a48ff9640 1 -- 172.21.15.116:0/4259757853 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a540bcd20 con 0x55d70ee3f2d0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.849+0000 7f6a48ff9640 1 -- 172.21.15.116:0/4259757853 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a540bd650 con 0x55d70ee3f2d0 2024-03-19T02:42:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.849+0000 7f6a48ff9640 1 -- 172.21.15.116:0/4259757853 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a540bc8b0 con 0x55d70ee3f2d0 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- 172.21.15.116:0/4259757853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 msgr2=0x55d70ee3f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f6a5402c630 tx=0x7f6a54035f50 comp rx=0 tx=0).stop 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- 172.21.15.116:0/4259757853 shutdown_connections 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee487d0 0x55d70ee4abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee48290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 --2- 172.21.15.116:0/4259757853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee3f2d0 0x55d70ee3f6b0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- 172.21.15.116:0/4259757853 >> 172.21.15.116:0/4259757853 conn(0x55d70ee39a00 msgr2=0x55d70ee39de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- 172.21.15.116:0/4259757853 shutdown_connections 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- 172.21.15.116:0/4259757853 wait complete. 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 -- start start 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.850+0000 7f6a63606740 1 Processor -- start 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee45cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee46210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d70eda24b0 con 0x55d70ee487d0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d70ed5e310 con 0x55d70ee3f2d0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d70ee4f790 con 0x55d70ee3fbf0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a59d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee45cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59184/0 (socket says 172.21.15.116:59184) 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 -- 172.21.15.116:0/2027322081 learned_addr learned my addr 172.21.15.116:0/2027322081 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee46210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a63606740 10 client.0 _ll_register_callbacks cb 0x55d70ed522d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 -- 172.21.15.116:0/2027322081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 msgr2=0x55d70ee45cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee45cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 -- 172.21.15.116:0/2027322081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 msgr2=0x55d70ee46210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee46210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 -- 172.21.15.116:0/2027322081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70eab63f0 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee45cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6a540ee4a0 tx=0x7f6a54038f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b198f3b29af999f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.851+0000 7f6a41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a540bca90 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a540bd650 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f6a54003970 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6a540bcf50 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 10 client.29711 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d70eb807d0 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a540bd650 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f6a5403c030 con 0x55d70ee3f2d0 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 10 client.29711 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 10 client.29711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 20 client.29711 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70ee5c9d0 0x55d70ee5edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 4 client.29711 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d70ee003e0 con 0x55d70ee5c9d0 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.852+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70ee5c9d0 0x55d70ee5edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.853+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70ee5c9d0 0x55d70ee5edc0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3d201ccd1a3e66f server_cookie=32c6c7d565b8bb99 in_seq=0 out_seq=0 2024-03-19T02:42:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.853+0000 7f6a41ffb640 10 client.29711 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a41ffb640 1 -- 172.21.15.116:0/2027322081 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d70ee003e0 con 0x55d70ee5c9d0 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a41ffb640 10 client.29711 handle_command_reply: tid=0 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 client.29711 shutdown 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 msgr2=0x55d70ee45790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6a540ee4a0 tx=0x7f6a54038f70 comp rx=0 tx=0).stop 2024-03-19T02:42:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 shutdown_connections 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70ee5c9d0 0x55d70ee5edc0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee46210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3fbf0 0x55d70ee45cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 --2- 172.21.15.116:0/2027322081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3f2d0 0x55d70ee45790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a41ffb640 0 client.29711 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 >> 172.21.15.116:0/2027322081 conn(0x55d70ee39a00 msgr2=0x55d70ee3b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 shutdown_connections 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 1 -- 172.21.15.116:0/2027322081 wait complete. 2024-03-19T02:42:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.855+0000 7f6a63606740 20 client.29711 trim_cache size 0 max 16384 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 Processor -- start 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 -- start start 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee47ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1ac990 con 0x55d70ee487d0 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1ae530 con 0x55d70ee3fbf0 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a63606740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1ad060 con 0x55d70ee3f2d0 2024-03-19T02:42:56.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33944/0 (socket says 172.21.15.116:33944) 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1494433037 learned_addr learned my addr 172.21.15.116:0/1494433037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.871+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee47ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1494433037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 msgr2=0x55d70ee47ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee47ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1494433037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 msgr2=0x55d70ee584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1494433037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70ee5f3f0 con 0x55d70ee487d0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c01f780 tx=0x7f6a4c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4fe3093045ec5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a43fff640 1 -- 172.21.15.116:0/1494433037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c005220 con 0x55d70ee487d0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a43fff640 1 -- 172.21.15.116:0/1494433037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a4c004c50 con 0x55d70ee487d0 2024-03-19T02:42:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.872+0000 7f6a43fff640 1 -- 172.21.15.116:0/1494433037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c01d9b0 con 0x55d70ee487d0 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 -- 172.21.15.116:0/1494433037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 msgr2=0x55d70ee38a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c01f780 tx=0x7f6a4c013040 comp rx=0 tx=0).stop 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 -- 172.21.15.116:0/1494433037 shutdown_connections 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee38a20 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 --2- 172.21.15.116:0/1494433037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee47ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 -- 172.21.15.116:0/1494433037 >> 172.21.15.116:0/1494433037 conn(0x55d70ee39a00 msgr2=0x55d70ee38310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 -- 172.21.15.116:0/1494433037 shutdown_connections 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.873+0000 7f6a63606740 1 -- 172.21.15.116:0/1494433037 wait complete. 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 -- start start 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 Processor -- start 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee41f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1ade00 con 0x55d70ee487d0 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1ad730 con 0x55d70ee3fbf0 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5c1aa990 con 0x55d70ee3f2d0 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a63606740 10 client.0 _ll_register_callbacks cb 0x7f6a540c1110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33946/0 (socket says 172.21.15.116:33946) 2024-03-19T02:42:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1809926074 learned_addr learned my addr 172.21.15.116:0/1809926074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.874+0000 7f6a59d9b640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee41f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1809926074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 msgr2=0x55d70ee54cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee54cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 msgr2=0x55d70ee41f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee41f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 -- 172.21.15.116:0/1809926074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70ee4abc0 con 0x55d70ee487d0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a5ad9d640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c01f490 tx=0x7f6a4c01f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59afbcf35a7e9b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c035020 con 0x55d70ee487d0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a4c004c50 con 0x55d70ee487d0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f6a4c005210 con 0x55d70ee487d0 2024-03-19T02:42:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6a4c040050 con 0x55d70ee487d0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a63606740 10 client.19962 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d70eec2910 con 0x55d70ee487d0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.875+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a4c004c50 con 0x55d70ee487d0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f6a4c01b050 con 0x55d70ee487d0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 10 client.19962 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 10 client.19962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 20 client.19962 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70eec3000 0x55d70eec5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 4 client.19962 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d70edaf9f0 con 0x55d70eec3000 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70eec3000 0x55d70eec5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a5a59c640 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70eec3000 0x55d70eec5420 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2e88bd1f6c8cce server_cookie=4976f5741bfea17c in_seq=0 out_seq=0 2024-03-19T02:42:56.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.876+0000 7f6a41ffb640 10 client.19962 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a41ffb640 1 -- 172.21.15.116:0/1809926074 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d70edaf9f0 con 0x55d70eec3000 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a41ffb640 10 client.19962 handle_command_reply: tid=0 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 client.19962 shutdown 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 msgr2=0x55d70ee424a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c01f490 tx=0x7f6a4c01f4c0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 shutdown_connections 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d70eec3000 0x55d70eec5420 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a41ffb640 0 client.19962 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d70ee487d0 0x55d70ee424a0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d70ee3fbf0 0x55d70ee41f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 --2- 172.21.15.116:0/1809926074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d70ee3f2d0 0x55d70ee54cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 >> 172.21.15.116:0/1809926074 conn(0x55d70ee39a00 msgr2=0x55d70ee43c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 shutdown_connections 2024-03-19T02:42:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.878+0000 7f6a63606740 1 -- 172.21.15.116:0/1809926074 wait complete. 2024-03-19T02:42:56.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a63606740 20 client.19962 trim_cache size 0 max 16384 2024-03-19T02:42:56.884 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 msgr2=0x7f6a440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 0x7f6a440a4e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55d70ee3ab90 tx=0x7f6a4c009ad0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 msgr2=0x7f6a5c196e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 0x7f6a5c196e40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006010 tx=0x7f6a54035d60 comp rx=0 tx=0).stop 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 shutdown_connections 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a440a2970 0x7f6a440a4e30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5c105690 0x7f6a5c19b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a5c101330 0x7f6a5c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 --2- 172.21.15.116:0/1336868853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a5c100970 0x7f6a5c196e40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 >> 172.21.15.116:0/1336868853 conn(0x7f6a5c077580 msgr2=0x7f6a5c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 shutdown_connections 2024-03-19T02:42:56.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:56.879+0000 7f6a60a19640 1 -- 172.21.15.116:0/1336868853 wait complete. 2024-03-19T02:42:57.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16241 5.82 MB/sec execute 19 sec latency 199.993 ms 2024-03-19T02:42:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:57 smithi162 ceph-mon[35761]: pgmap v475: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:57 smithi154 ceph-mon[35693]: pgmap v475: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:57 smithi116 ceph-mon[32263]: pgmap v475: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:57.893 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:42:57.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:42:57.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:42:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 -- 172.21.15.116:0/3325719435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc14101050 msgr2=0x7fcc1410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/3325719435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc14101050 0x7fcc1410d560 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009cd0 tx=0x7fcc0c02c0e0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 -- 172.21.15.116:0/3325719435 shutdown_connections 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/3325719435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc14101050 0x7fcc1410d560 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/3325719435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14100690 0x7fcc14100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/3325719435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc141066a0 0x7fcc14106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.024+0000 7fcc1b51d640 1 -- 172.21.15.116:0/3325719435 >> 172.21.15.116:0/3325719435 conn(0x7fcc14077820 msgr2=0x7fcc14077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 -- 172.21.15.116:0/3325719435 shutdown_connections 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 -- 172.21.15.116:0/3325719435 wait complete. 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 Processor -- start 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 -- start start 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 0x7fcc14197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14101050 0x7fcc141984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14110920 con 0x7fcc141066a0 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.025+0000 7fcc1b51d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141107a0 con 0x7fcc14101050 2024-03-19T02:42:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc1b51d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14110aa0 con 0x7fcc14100690 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc192b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 0x7fcc14197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc18ab8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14101050 0x7fcc141984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc192b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 0x7fcc14197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44896/0 (socket says 172.21.15.116:44896) 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47540/0 (socket says 172.21.15.116:47540) 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 -- 172.21.15.116:0/1508066582 learned_addr learned my addr 172.21.15.116:0/1508066582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 -- 172.21.15.116:0/1508066582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 msgr2=0x7fcc14197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 0x7fcc14197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 -- 172.21.15.116:0/1508066582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14101050 msgr2=0x7fcc141984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14101050 0x7fcc141984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 -- 172.21.15.116:0/1508066582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1419cf80 con 0x7fcc141066a0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004af0 tx=0x7fcc0c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc0a7fc640 1 -- 172.21.15.116:0/1508066582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c039040 con 0x7fcc141066a0 2024-03-19T02:42:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1419d210 con 0x7fcc141066a0 2024-03-19T02:42:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc0a7fc640 1 -- 172.21.15.116:0/1508066582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc0c044070 con 0x7fcc141066a0 2024-03-19T02:42:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc0a7fc640 1 -- 172.21.15.116:0/1508066582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0042a0 con 0x7fcc141066a0 2024-03-19T02:42:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.026+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc14102be0 con 0x7fcc141066a0 2024-03-19T02:42:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.029+0000 7fcc0a7fc640 1 -- 172.21.15.116:0/1508066582 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcc0c004440 con 0x7fcc141066a0 2024-03-19T02:42:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 Processor -- start 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 -- start start 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7db90 0x55b015d7df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d86b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc0a7fc640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 0x7fcc000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b015cd9160 con 0x55b015d7e4b0 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b015c81e10 con 0x55b015d87090 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.030+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b015cd5200 con 0x55b015d7db90 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 0x7fcc000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41940/0 (socket says 172.21.15.116:41940) 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 -- 172.21.15.116:0/1741831683 learned_addr learned my addr 172.21.15.116:0/1741831683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7db90 0x55b015d7df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc0a7fc640 1 -- 172.21.15.116:0/1508066582 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fcc0c0f15c0 con 0x7fcc141066a0 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 -- 172.21.15.116:0/1741831683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7db90 msgr2=0x55b015d7df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7db90 0x55b015d7df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc192b9640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d86b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 -- 172.21.15.116:0/1741831683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 msgr2=0x55b015d86b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d86b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 -- 172.21.15.116:0/1741831683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b015ab57e0 con 0x55b015d87090 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc192b9640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d86b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 0x7fcc000a4eb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcc141002e0 tx=0x7fcc0400fbe0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.031+0000 7fcc19aba640 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b9d00 tx=0x7fcc0c02c0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47b38721180c42af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.032+0000 7fcbff7fe640 1 -- 172.21.15.116:0/1741831683 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0bdff0 con 0x55b015d87090 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.032+0000 7fcbff7fe640 1 -- 172.21.15.116:0/1741831683 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc0c0098d0 con 0x55b015d87090 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.032+0000 7fcbff7fe640 1 -- 172.21.15.116:0/1741831683 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0bd250 con 0x55b015d87090 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.032+0000 7fcc1e1df740 1 -- 172.21.15.116:0/1741831683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 msgr2=0x55b015d89480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.032+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b9d00 tx=0x7fcc0c02c0e0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 -- 172.21.15.116:0/1741831683 shutdown_connections 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015d89480 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d86b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/1741831683 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7db90 0x55b015d7df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 -- 172.21.15.116:0/1741831683 >> 172.21.15.116:0/1741831683 conn(0x55b015d782c0 msgr2=0x55b015d786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 -- 172.21.15.116:0/1741831683 shutdown_connections 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 -- 172.21.15.116:0/1741831683 wait complete. 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 -- start start 2024-03-19T02:42:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.033+0000 7fcc1e1df740 1 Processor -- start 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d84070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b015d9ac50 con 0x55b015d87090 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b015c951a0 con 0x55b015d7db90 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b015d9add0 con 0x55b015d7e4b0 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc1e1df740 10 client.0 _ll_register_callbacks cb 0x55b0159ecc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47566/0 (socket says 172.21.15.116:47566) 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 -- 172.21.15.116:0/3144468973 learned_addr learned my addr 172.21.15.116:0/3144468973 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc192b9640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d84070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 -- 172.21.15.116:0/3144468973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 msgr2=0x55b015d845b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d845b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 -- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 msgr2=0x55b015d84070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d84070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 -- 172.21.15.116:0/3144468973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b015b15f70 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcc19aba640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b9d50 tx=0x7fcc0c0b9470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73da7ddc4b91f652 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.034+0000 7fcbfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0326f0 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fcc0c0a8110 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b015ce66a0 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcc0c0c6f50 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 10 client.19971 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b0159ad170 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b015ce66a0 con 0x55b015d87090 2024-03-19T02:42:58.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fcc0c03c030 con 0x55b015d87090 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 10 client.19971 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 10 client.19971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 20 client.19971 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015da7e80 0x55b015daa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 4 client.19971 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.035+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b015ce09f0 con 0x55b015da7e80 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.036+0000 7fcc192b9640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015da7e80 0x55b015daa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.036+0000 7fcbfcff9640 10 client.19971 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.036+0000 7fcc192b9640 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015da7e80 0x55b015daa270 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9112aae1fcd96bae server_cookie=1299bb2f0b397d51 in_seq=0 out_seq=0 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcbfcff9640 1 -- 172.21.15.116:0/3144468973 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b015ce09f0 con 0x55b015da7e80 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcbfcff9640 10 client.19971 handle_command_reply: tid=0 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 client.19971 shutdown 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 msgr2=0x55b015d84af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0b9d50 tx=0x7fcc0c0b9470 comp rx=0 tx=0).stop 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 shutdown_connections 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015da7e80 0x55b015daa270 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d87090 0x55b015d84af0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcbfcff9640 0 client.19971 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/3144468973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d84070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 >> 172.21.15.116:0/3144468973 conn(0x55b015d782c0 msgr2=0x55b015d7a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 shutdown_connections 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 1 -- 172.21.15.116:0/3144468973 wait complete. 2024-03-19T02:42:58.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.038+0000 7fcc1e1df740 20 client.19971 trim_cache size 0 max 16384 2024-03-19T02:42:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 Processor -- start 2024-03-19T02:42:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 -- start start 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d867a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d87090 0x55b015da38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14073050 con 0x55b015d7e4b0 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141a5ca0 con 0x55b015d7db90 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14073720 con 0x55b015d87090 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47568/0 (socket says 172.21.15.116:47568) 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 -- 172.21.15.116:0/4219246151 learned_addr learned my addr 172.21.15.116:0/4219246151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc19aba640 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d87090 0x55b015da38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc192b9640 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d867a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 -- 172.21.15.116:0/4219246151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d87090 msgr2=0x55b015da38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d87090 0x55b015da38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 -- 172.21.15.116:0/4219246151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 msgr2=0x55b015d867a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d867a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 -- 172.21.15.116:0/4219246151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b015da3de0 con 0x55b015d7e4b0 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.062+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401c690 tx=0x7fcc0401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f3fca3b2d0522c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcbfeffd640 1 -- 172.21.15.116:0/4219246151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0401d600 con 0x55b015d7e4b0 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcbfeffd640 1 -- 172.21.15.116:0/4219246151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc0402b070 con 0x55b015d7e4b0 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcbfeffd640 1 -- 172.21.15.116:0/4219246151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04026890 con 0x55b015d7e4b0 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 -- 172.21.15.116:0/4219246151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 msgr2=0x55b015d82fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401c690 tx=0x7fcc0401d000 comp rx=0 tx=0).stop 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 -- 172.21.15.116:0/4219246151 shutdown_connections 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d87090 0x55b015da38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7e4b0 0x55b015d82fb0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/4219246151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d7db90 0x55b015d867a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 -- 172.21.15.116:0/4219246151 >> 172.21.15.116:0/4219246151 conn(0x55b015d782c0 msgr2=0x55b015d6eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 -- 172.21.15.116:0/4219246151 shutdown_connections 2024-03-19T02:42:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.063+0000 7fcc1e1df740 1 -- 172.21.15.116:0/4219246151 wait complete. 2024-03-19T02:42:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 -- start start 2024-03-19T02:42:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 Processor -- start 2024-03-19T02:42:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d819a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015e0cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141a5570 con 0x55b015d7db90 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc141a4ea0 con 0x55b015d87090 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14071050 con 0x55b015d7e4b0 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc1e1df740 10 client.0 _ll_register_callbacks cb 0x7fcc0c0c0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc192b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc192b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47574/0 (socket says 172.21.15.116:47574) 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc192b9640 1 -- 172.21.15.116:0/2843456324 learned_addr learned my addr 172.21.15.116:0/2843456324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc18ab8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d819a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.064+0000 7fcc19aba640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015e0cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 -- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 msgr2=0x55b015e0cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015e0cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 -- 172.21.15.116:0/2843456324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 msgr2=0x55b015d81460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 -- 172.21.15.116:0/2843456324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b015e0d5b0 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc192b9640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc18ab8640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d819a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401cfd0 tx=0x7fcc0401dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d278b7479cb1f98c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcbfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04033070 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7fcc04018040 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc0402b070 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcc0403e050 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc1e1df740 10 client.29698 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b015e0e8d0 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.065+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc0402b070 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fcc04026990 con 0x55b015d7e4b0 2024-03-19T02:42:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 10 client.29698 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 10 client.29698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 20 client.29698 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015e0efd0 0x55b015e113f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 4 client.29698 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b015c5ebe0 con 0x55b015e0efd0 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc192b9640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015e0efd0 0x55b015e113f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcc192b9640 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015e0efd0 0x55b015e113f0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d697dd35be3f63c server_cookie=3b16128d5fe0854f in_seq=0 out_seq=0 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.066+0000 7fcbfcff9640 10 client.29698 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcbfcff9640 1 -- 172.21.15.116:0/2843456324 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b015c5ebe0 con 0x55b015e0efd0 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcbfcff9640 10 client.29698 handle_command_reply: tid=0 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 client.29698 shutdown 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 msgr2=0x55b015d819a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d819a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401cfd0 tx=0x7fcc0401dc60 comp rx=0 tx=0).stop 2024-03-19T02:42:58.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 shutdown_connections 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b015e0efd0 0x55b015e113f0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b015d87090 0x55b015e0cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b015d7e4b0 0x55b015d819a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 --2- 172.21.15.116:0/2843456324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b015d7db90 0x55b015d81460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 >> 172.21.15.116:0/2843456324 conn(0x55b015d782c0 msgr2=0x55b015d84030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.069+0000 7fcbfcff9640 0 client.29698 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 shutdown_connections 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1e1df740 1 -- 172.21.15.116:0/2843456324 wait complete. 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1e1df740 20 client.29698 trim_cache size 0 max 16384 2024-03-19T02:42:58.074 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:58.075 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:42:58.075 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "48376ece-2f58-43c7-b3ae-6d8a563e7fca", 2024-03-19T02:42:58.075 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:42:58.075 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 msgr2=0x7fcc000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 0x7fcc000a4eb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcc141002e0 tx=0x7fcc0400fbe0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 msgr2=0x7fcc1419c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004af0 tx=0x7fcc0c035ce0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 shutdown_connections 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcc000a29f0 0x7fcc000a4eb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc141066a0 0x7fcc1419c850 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc14101050 0x7fcc141984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 --2- 172.21.15.116:0/1508066582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc14100690 0x7fcc14197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 >> 172.21.15.116:0/1508066582 conn(0x7fcc14077820 msgr2=0x7fcc14068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.070+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 shutdown_connections 2024-03-19T02:42:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.071+0000 7fcc1b51d640 1 -- 172.21.15.116:0/1508066582 wait complete. 2024-03-19T02:42:58.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:42:58.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 -- 172.21.15.116:0/3820881897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5600100690 msgr2=0x7f5600100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 --2- 172.21.15.116:0/3820881897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5600100690 0x7f5600100b10 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009930 tx=0x7f55f402bc30 comp rx=0 tx=0).stop 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 -- 172.21.15.116:0/3820881897 shutdown_connections 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 --2- 172.21.15.116:0/3820881897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600101050 0x7f560010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 --2- 172.21.15.116:0/3820881897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5600100690 0x7f5600100b10 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 --2- 172.21.15.116:0/3820881897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56001066a0 0x7f5600106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 -- 172.21.15.116:0/3820881897 >> 172.21.15.116:0/3820881897 conn(0x7f5600077820 msgr2=0x7f5600077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 -- 172.21.15.116:0/3820881897 shutdown_connections 2024-03-19T02:42:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 -- 172.21.15.116:0/3820881897 wait complete. 2024-03-19T02:42:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.208+0000 7f56058e8640 1 Processor -- start 2024-03-19T02:42:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 -- start start 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 0x7f5600196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5600101050 0x7f5600197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5600110960 con 0x7f56001066a0 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56001107e0 con 0x7f5600101050 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f56058e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5600110ae0 con 0x7f5600100690 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55feffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 0x7f5600196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47592/0 (socket says 172.21.15.116:47592) 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 -- 172.21.15.116:0/1111913198 learned_addr learned my addr 172.21.15.116:0/1111913198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5600101050 0x7f5600197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 -- 172.21.15.116:0/1111913198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 msgr2=0x7f5600196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 0x7f5600196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 -- 172.21.15.116:0/1111913198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5600101050 msgr2=0x7f5600197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5600101050 0x7f5600197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55ff7fe640 1 -- 172.21.15.116:0/1111913198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56001a64c0 con 0x7f56001066a0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.209+0000 7f55feffd640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 0x7f5600196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f55ff7fe640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f55f800eb10 tx=0x7f55f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f56048e6640 1 -- 172.21.15.116:0/1111913198 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8004b20 con 0x7f56001066a0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f56048e6640 1 -- 172.21.15.116:0/1111913198 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55f8004cc0 con 0x7f56001066a0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f56048e6640 1 -- 172.21.15.116:0/1111913198 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8010690 con 0x7f56001066a0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56001a67b0 con 0x7f56001066a0 2024-03-19T02:42:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.210+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56001a6cd0 con 0x7f56001066a0 2024-03-19T02:42:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56048e6640 1 -- 172.21.15.116:0/1111913198 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f55f8005020 con 0x7f56001066a0 2024-03-19T02:42:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 Processor -- start 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 -- start start 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e29e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e35320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a825d8d500 con 0x55a825e32f30 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a825d35be0 con 0x55a825e29a30 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f56085aa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a825d895a0 con 0x55a825e2a350 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e35320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e35320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47604/0 (socket says 172.21.15.116:47604) 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f55fe7fc640 1 -- 172.21.15.116:0/514235502 learned_addr learned my addr 172.21.15.116:0/514235502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f55feffd640 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.213+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e29e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f56048e6640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 0x7f55e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 -- 172.21.15.116:0/514235502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 msgr2=0x55a825e29e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e29e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 -- 172.21.15.116:0/514235502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 msgr2=0x55a825e35320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e35320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 -- 172.21.15.116:0/514235502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a825d57000 con 0x55a825e2a350 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55ff7fe640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 0x7f55e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55feffd640 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e329f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009f50 tx=0x7f55f00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3382f7d6e8e6c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f56048e6640 1 -- 172.21.15.116:0/1111913198 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f55f80cf940 con 0x7f56001066a0 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55e5ffb640 1 -- 172.21.15.116:0/514235502 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0004030 con 0x55a825e2a350 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55e5ffb640 1 -- 172.21.15.116:0/514235502 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55f00054d0 con 0x55a825e2a350 2024-03-19T02:42:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.214+0000 7f55e5ffb640 1 -- 172.21.15.116:0/514235502 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0004bc0 con 0x55a825e2a350 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 -- 172.21.15.116:0/514235502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 msgr2=0x55a825e329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e329f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009f50 tx=0x7f55f00118d0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f55ff7fe640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 0x7f55e80a4e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f55f809bb50 tx=0x7f55f806d040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 -- 172.21.15.116:0/514235502 shutdown_connections 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e35320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e329f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 --2- 172.21.15.116:0/514235502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e29e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 -- 172.21.15.116:0/514235502 >> 172.21.15.116:0/514235502 conn(0x55a825e24160 msgr2=0x55a825e24540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 -- 172.21.15.116:0/514235502 shutdown_connections 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.215+0000 7f56085aa740 1 -- 172.21.15.116:0/514235502 wait complete. 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 -- start start 2024-03-19T02:42:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 Processor -- start 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 0x55a825e2fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e30910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a825e35d20 con 0x55a825e29a30 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a825d49160 con 0x55a825e32f30 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a825e35ea0 con 0x55a825e2a350 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f56085aa740 10 client.0 _ll_register_callbacks cb 0x55a825d85a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f55feffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 0x55a825e2fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f55feffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44972/0 (socket says 172.21.15.116:44972) 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 0x55a825e2fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47606/0 (socket says 172.21.15.116:47606) 2024-03-19T02:42:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.216+0000 7f55feffd640 1 -- 172.21.15.116:0/1420826755 learned_addr learned my addr 172.21.15.116:0/1420826755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e30910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 -- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 msgr2=0x55a825e30910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e30910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 -- 172.21.15.116:0/1420826755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 msgr2=0x55a825e2fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 0x55a825e2fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 -- 172.21.15.116:0/1420826755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a825b6b7d0 con 0x55a825e2a350 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e30910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55feffd640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f55f00058c0 tx=0x7f55f000fba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3129b4588d5820cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55deffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0029020 con 0x55a825e2a350 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f55f0025470 con 0x55a825e2a350 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f00054d0 con 0x55a825e2a350 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.217+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f55f0025670 con 0x55a825e2a350 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 10 client.29710 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a825aa13f0 con 0x55a825e2a350 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55f00054d0 con 0x55a825e2a350 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f55f00049b0 con 0x55a825e2a350 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 10 client.29710 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 10 client.29710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 20 client.29710 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825e434c0 0x55a825e45940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 4 client.29710 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a825d9aa40 con 0x55a825e434c0 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.218+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825e434c0 0x55a825e45940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.219+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825e434c0 0x55a825e45940 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24753f35893d4ea9 server_cookie=4ab0500bcfee5908 in_seq=0 out_seq=0 2024-03-19T02:42:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.219+0000 7f55deffd640 10 client.29710 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f55deffd640 1 -- 172.21.15.116:0/1420826755 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a825d9aa40 con 0x55a825e434c0 2024-03-19T02:42:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f55deffd640 10 client.29710 handle_command_reply: tid=0 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 client.29710 shutdown 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 msgr2=0x55a825e303d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f55f00058c0 tx=0x7f55f000fba0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 shutdown_connections 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825e434c0 0x55a825e45940 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e30910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e303d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 --2- 172.21.15.116:0/1420826755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e29a30 0x55a825e2fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f55deffd640 0 client.29710 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.221+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 >> 172.21.15.116:0/1420826755 conn(0x55a825e24160 msgr2=0x55a825e25eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.222+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 shutdown_connections 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.222+0000 7f56085aa740 1 -- 172.21.15.116:0/1420826755 wait complete. 2024-03-19T02:42:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.222+0000 7f56085aa740 20 client.29710 trim_cache size 0 max 16384 2024-03-19T02:42:58.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 Processor -- start 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 -- start start 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e29a30 0x55a825e306e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e4b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b0fd0 con 0x55a825e2a350 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b2b70 con 0x55a825e32f30 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.249+0000 7f56085aa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b16a0 con 0x55a825e29a30 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47610/0 (socket says 172.21.15.116:47610) 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 -- 172.21.15.116:0/2386151611 learned_addr learned my addr 172.21.15.116:0/2386151611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e29a30 0x55a825e306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e4b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 -- 172.21.15.116:0/2386151611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e29a30 msgr2=0x55a825e306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e29a30 0x55a825e306e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 -- 172.21.15.116:0/2386151611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 msgr2=0x55a825e4b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e4b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 -- 172.21.15.116:0/2386151611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a825a610f0 con 0x55a825e2a350 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55feffd640 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f55f0011dd0 tx=0x7f55f0025600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb05de54907b558c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55e57fa640 1 -- 172.21.15.116:0/2386151611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0029040 con 0x55a825e2a350 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55e57fa640 1 -- 172.21.15.116:0/2386151611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55f0025cc0 con 0x55a825e2a350 2024-03-19T02:42:58.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.250+0000 7f55e57fa640 1 -- 172.21.15.116:0/2386151611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00035f0 con 0x55a825e2a350 2024-03-19T02:42:58.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 -- 172.21.15.116:0/2386151611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 msgr2=0x55a825e3eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f55f0011dd0 tx=0x7f55f0025600 comp rx=0 tx=0).stop 2024-03-19T02:42:58.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 -- 172.21.15.116:0/2386151611 shutdown_connections 2024-03-19T02:42:58.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e32f30 0x55a825e4b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e2a350 0x55a825e3eac0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 --2- 172.21.15.116:0/2386151611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e29a30 0x55a825e306e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.251+0000 7f56085aa740 1 -- 172.21.15.116:0/2386151611 >> 172.21.15.116:0/2386151611 conn(0x55a825e24160 msgr2=0x55a825e22950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- 172.21.15.116:0/2386151611 shutdown_connections 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- 172.21.15.116:0/2386151611 wait complete. 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- start start 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 Processor -- start 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e3b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e380c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b2440 con 0x55a825e32f30 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56001b1d70 con 0x55a825e29a30 2024-03-19T02:42:58.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56001aefd0 con 0x55a825e2a350 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.252+0000 7f56085aa740 10 client.0 _ll_register_callbacks cb 0x7f55f80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55feffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47614/0 (socket says 172.21.15.116:47614) 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 -- 172.21.15.116:0/3369480500 learned_addr learned my addr 172.21.15.116:0/3369480500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e3b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 -- 172.21.15.116:0/3369480500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 msgr2=0x55a825e380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 -- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 msgr2=0x55a825e3b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e3b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 -- 172.21.15.116:0/3369480500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a825e3b6b0 con 0x55a825e32f30 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55feffd640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55fe7fc640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f55f4036040 tx=0x7f55f000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d61922e0c136a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:58.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55deffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.253+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f404a070 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9510+0+0 (secure 0 0 0) 0x7f55f404eac0 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55f0025cc0 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f55f404d040 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 10 client.19995 fetch_fsmap learned FSMap version 36 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a825eaa280 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55f0025cc0 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f55f40037b0 con 0x55a825e32f30 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 10 client.19995 fetch_fsmap finished waiting for FSMap version 36 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 10 client.19995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 20 client.19995 populate_metadata read hostname 'smithi116' 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825eaa9d0 0x55a825eacdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 4 client.19995 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:42:58.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.254+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a825d94d90 con 0x55a825eaa9d0 2024-03-19T02:42:58.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.255+0000 7f55feffd640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825eaa9d0 0x55a825eacdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:58.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.255+0000 7f55deffd640 10 client.19995 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.255+0000 7f55feffd640 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825eaa9d0 0x55a825eacdf0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e101144197b1185e server_cookie=4752d66781aa402d in_seq=0 out_seq=0 2024-03-19T02:42:58.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:58 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:58.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:58 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:58.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f55deffd640 1 -- 172.21.15.116:0/3369480500 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a825d94d90 con 0x55a825eaa9d0 2024-03-19T02:42:58.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f55deffd640 10 client.19995 handle_command_reply: tid=0 2024-03-19T02:42:58.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 client.19995 shutdown 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 msgr2=0x55a825e38600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f55f4036040 tx=0x7f55f000f850 comp rx=0 tx=0).stop 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 shutdown_connections 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a825eaa9d0 0x55a825eacdf0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a825e32f30 0x55a825e38600 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a825e2a350 0x55a825e380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 --2- 172.21.15.116:0/3369480500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a825e29a30 0x55a825e3b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f55deffd640 0 client.19995 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 >> 172.21.15.116:0/3369480500 conn(0x55a825e24160 msgr2=0x55a825e30540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 shutdown_connections 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 1 -- 172.21.15.116:0/3369480500 wait complete. 2024-03-19T02:42:58.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.258+0000 7f56085aa740 20 client.19995 trim_cache size 0 max 16384 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (462 inodes in the stack)", 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "48376ece-2f58-43c7-b3ae-6d8a563e7fca": { 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "48376ece-2f58-43c7-b3ae-6d8a563e7fca", 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:42:58.263 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 msgr2=0x7f55e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 0x7f55e80a4e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f55f809bb50 tx=0x7f55f806d040 comp rx=0 tx=0).stop 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 msgr2=0x7f56001976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f55f800eb10 tx=0x7f55f800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 shutdown_connections 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55e80a2970 0x7f55e80a4e30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56001066a0 0x7f56001976d0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5600101050 0x7f5600197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 --2- 172.21.15.116:0/1111913198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5600100690 0x7f5600196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:58.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 >> 172.21.15.116:0/1111913198 conn(0x7f5600077820 msgr2=0x7f5600102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:58.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 shutdown_connections 2024-03-19T02:42:58.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:42:58.259+0000 7f56058e8640 1 -- 172.21.15.116:0/1111913198 wait complete. 2024-03-19T02:42:58.273 INFO:tasks.cephfs.filesystem:scrub status for tag:48376ece-2f58-43c7-b3ae-6d8a563e7fca - {'path': '/', 'tag': '48376ece-2f58-43c7-b3ae-6d8a563e7fca', 'options': 'recursive,force'} 2024-03-19T02:42:58.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16388 6.13 MB/sec execute 20 sec latency 195.897 ms 2024-03-19T02:42:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:58 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:58 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:58 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:58 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16530 6.17 MB/sec execute 21 sec latency 159.733 ms 2024-03-19T02:42:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:42:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:42:59 smithi162 ceph-mon[35761]: pgmap v476: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:42:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:42:59 smithi154 ceph-mon[35693]: pgmap v476: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:42:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:42:59.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:42:59 smithi116 ceph-mon[32263]: pgmap v476: 129 pgs: 129 active+clean; 797 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.4 MiB/s wr, 171 op/s 2024-03-19T02:43:00.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16700 6.56 MB/sec execute 22 sec latency 194.721 ms 2024-03-19T02:43:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:00 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:00 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:00.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:00 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:01.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16912 6.97 MB/sec execute 23 sec latency 154.019 ms 2024-03-19T02:43:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:01 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:01 smithi154 ceph-mon[35693]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 243 op/s 2024-03-19T02:43:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:01 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:01 smithi116 ceph-mon[32263]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 243 op/s 2024-03-19T02:43:01.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:01 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:01.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:01 smithi162 ceph-mon[35761]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 243 op/s 2024-03-19T02:43:02.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17013 6.87 MB/sec execute 24 sec latency 158.338 ms 2024-03-19T02:43:03.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17081 6.82 MB/sec execute 25 sec latency 129.608 ms 2024-03-19T02:43:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:03 smithi162 ceph-mon[35761]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 3.4 MiB/s wr, 147 op/s 2024-03-19T02:43:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:03 smithi154 ceph-mon[35693]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 3.4 MiB/s wr, 147 op/s 2024-03-19T02:43:03.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:03 smithi116 ceph-mon[32263]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 3.4 MiB/s wr, 147 op/s 2024-03-19T02:43:04.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17125 6.62 MB/sec execute 26 sec latency 169.417 ms 2024-03-19T02:43:05.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17170 6.38 MB/sec execute 27 sec latency 172.235 ms 2024-03-19T02:43:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:05 smithi162 ceph-mon[35761]: osdmap e114: 12 total, 12 up, 12 in 2024-03-19T02:43:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:05 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:05 smithi162 ceph-mon[35761]: pgmap v480: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-19T02:43:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:05 smithi154 ceph-mon[35693]: osdmap e114: 12 total, 12 up, 12 in 2024-03-19T02:43:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:05 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:05 smithi154 ceph-mon[35693]: pgmap v480: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-19T02:43:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:05 smithi116 ceph-mon[32263]: osdmap e114: 12 total, 12 up, 12 in 2024-03-19T02:43:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:05 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:43:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:05 smithi116 ceph-mon[32263]: pgmap v480: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-19T02:43:06.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17264 6.46 MB/sec execute 28 sec latency 136.300 ms 2024-03-19T02:43:07.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17394 6.69 MB/sec execute 29 sec latency 154.818 ms 2024-03-19T02:43:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:07 smithi162 ceph-mon[35761]: osdmap e115: 12 total, 12 up, 12 in 2024-03-19T02:43:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:07 smithi162 ceph-mon[35761]: pgmap v482: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 208 op/s 2024-03-19T02:43:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:07 smithi154 ceph-mon[35693]: osdmap e115: 12 total, 12 up, 12 in 2024-03-19T02:43:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:07 smithi154 ceph-mon[35693]: pgmap v482: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 208 op/s 2024-03-19T02:43:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:07 smithi116 ceph-mon[32263]: osdmap e115: 12 total, 12 up, 12 in 2024-03-19T02:43:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:07 smithi116 ceph-mon[32263]: pgmap v482: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 208 op/s 2024-03-19T02:43:08.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17533 6.78 MB/sec execute 30 sec latency 151.107 ms 2024-03-19T02:43:09.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17685 6.93 MB/sec execute 31 sec latency 138.923 ms 2024-03-19T02:43:09.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:09 smithi162 ceph-mon[35761]: pgmap v483: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:43:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:09 smithi154 ceph-mon[35693]: pgmap v483: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:43:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:09 smithi116 ceph-mon[32263]: pgmap v483: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-03-19T02:43:10.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17775 6.82 MB/sec execute 32 sec latency 164.113 ms 2024-03-19T02:43:10.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:10.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:11.322 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17914 6.91 MB/sec execute 33 sec latency 151.359 ms 2024-03-19T02:43:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:11.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:11 smithi162 ceph-mon[35761]: pgmap v484: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 201 op/s 2024-03-19T02:43:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:11 smithi154 ceph-mon[35693]: pgmap v484: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 201 op/s 2024-03-19T02:43:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:11.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:11 smithi116 ceph-mon[32263]: pgmap v484: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.0 MiB/s wr, 201 op/s 2024-03-19T02:43:12.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 17998 6.79 MB/sec execute 34 sec latency 219.992 ms 2024-03-19T02:43:13.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18069 6.70 MB/sec execute 35 sec latency 161.194 ms 2024-03-19T02:43:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:13 smithi162 ceph-mon[35761]: pgmap v485: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 192 op/s 2024-03-19T02:43:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:13 smithi154 ceph-mon[35693]: pgmap v485: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 192 op/s 2024-03-19T02:43:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:13 smithi116 ceph-mon[32263]: pgmap v485: 129 pgs: 129 active+clean; 836 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 192 op/s 2024-03-19T02:43:14.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18138 6.53 MB/sec execute 36 sec latency 188.840 ms 2024-03-19T02:43:15.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:14 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:14 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:14 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:15.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18217 6.40 MB/sec execute 37 sec latency 226.395 ms 2024-03-19T02:43:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:15 smithi154 ceph-mon[35693]: pgmap v486: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 141 op/s 2024-03-19T02:43:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:43:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:43:16.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:15 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:43:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:15 smithi116 ceph-mon[32263]: pgmap v486: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 141 op/s 2024-03-19T02:43:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:43:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:43:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:15 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:43:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:15 smithi162 ceph-mon[35761]: pgmap v486: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 141 op/s 2024-03-19T02:43:16.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:43:16.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:43:16.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:15 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:43:16.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18306 6.25 MB/sec execute 38 sec latency 200.901 ms 2024-03-19T02:43:17.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18404 6.14 MB/sec execute 39 sec latency 185.837 ms 2024-03-19T02:43:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:17 smithi162 ceph-mon[35761]: pgmap v487: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-03-19T02:43:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:17 smithi154 ceph-mon[35693]: pgmap v487: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-03-19T02:43:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:17 smithi116 ceph-mon[32263]: pgmap v487: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 140 op/s 2024-03-19T02:43:18.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18517 6.07 MB/sec execute 40 sec latency 160.722 ms 2024-03-19T02:43:19.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18649 6.14 MB/sec execute 41 sec latency 162.619 ms 2024-03-19T02:43:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:19 smithi162 ceph-mon[35761]: pgmap v488: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 117 op/s 2024-03-19T02:43:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:19 smithi154 ceph-mon[35693]: pgmap v488: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 117 op/s 2024-03-19T02:43:19.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:19 smithi116 ceph-mon[32263]: pgmap v488: 129 pgs: 129 active+clean; 850 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 117 op/s 2024-03-19T02:43:20.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18766 6.10 MB/sec execute 42 sec latency 139.087 ms 2024-03-19T02:43:21.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 18873 6.06 MB/sec execute 43 sec latency 138.031 ms 2024-03-19T02:43:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:21 smithi162 ceph-mon[35761]: pgmap v489: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 222 op/s 2024-03-19T02:43:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:21 smithi154 ceph-mon[35693]: pgmap v489: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 222 op/s 2024-03-19T02:43:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:21 smithi116 ceph-mon[32263]: pgmap v489: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 222 op/s 2024-03-19T02:43:22.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19011 6.19 MB/sec execute 44 sec latency 148.082 ms 2024-03-19T02:43:23.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:23 smithi154 ceph-mon[35693]: pgmap v490: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-03-19T02:43:23.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19136 6.18 MB/sec execute 45 sec latency 151.762 ms 2024-03-19T02:43:23.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:23 smithi162 ceph-mon[35761]: pgmap v490: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-03-19T02:43:23.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:23 smithi116 ceph-mon[32263]: pgmap v490: 129 pgs: 129 active+clean; 851 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-03-19T02:43:24.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19257 6.37 MB/sec execute 46 sec latency 142.216 ms 2024-03-19T02:43:25.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19366 6.40 MB/sec execute 47 sec latency 184.932 ms 2024-03-19T02:43:25.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:25 smithi162 ceph-mon[35761]: pgmap v491: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-03-19T02:43:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:25 smithi154 ceph-mon[35693]: pgmap v491: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-03-19T02:43:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:25 smithi116 ceph-mon[32263]: pgmap v491: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 222 op/s 2024-03-19T02:43:26.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19473 6.28 MB/sec execute 48 sec latency 159.848 ms 2024-03-19T02:43:27.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19591 6.17 MB/sec execute 49 sec latency 155.287 ms 2024-03-19T02:43:28.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:27 smithi162 ceph-mon[35761]: pgmap v492: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:28.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:27 smithi154 ceph-mon[35693]: pgmap v492: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:27 smithi116 ceph-mon[32263]: pgmap v492: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:28.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:43:28.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:43:28.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19730 6.09 MB/sec execute 50 sec latency 133.026 ms 2024-03-19T02:43:28.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 -- 172.21.15.116:0/1846616742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 msgr2=0x7fd9b010d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 --2- 172.21.15.116:0/1846616742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b010d540 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009ca0 tx=0x7fd9a802c180 comp rx=0 tx=0).stop 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 -- 172.21.15.116:0/1846616742 shutdown_connections 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 --2- 172.21.15.116:0/1846616742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b010d540 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 --2- 172.21.15.116:0/1846616742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 0x7fd9b0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 --2- 172.21.15.116:0/1846616742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b01059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 -- 172.21.15.116:0/1846616742 >> 172.21.15.116:0/1846616742 conn(0x7fd9b00777b0 msgr2=0x7fd9b0077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 -- 172.21.15.116:0/1846616742 shutdown_connections 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.415+0000 7fd9b5661640 1 -- 172.21.15.116:0/1846616742 wait complete. 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 Processor -- start 2024-03-19T02:43:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 -- start start 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 0x7fd9b0196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b0110960 con 0x7fd9b0101080 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01107e0 con 0x7fd9b01006e0 2024-03-19T02:43:28.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9b5661640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b0110ae0 con 0x7fd9b01055c0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46910/0 (socket says 172.21.15.116:46910) 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36770/0 (socket says 172.21.15.116:36770) 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9a7fff640 1 -- 172.21.15.116:0/43841517 learned_addr learned my addr 172.21.15.116:0/43841517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.416+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 0x7fd9b0196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 -- 172.21.15.116:0/43841517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 msgr2=0x7fd9b0196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 0x7fd9b0196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 -- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 msgr2=0x7fd9b0197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 -- 172.21.15.116:0/43841517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b019be30 con 0x7fd9b01055c0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a802c790 tx=0x7fd9a8035dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9acff9640 1 -- 172.21.15.116:0/43841517 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8039040 con 0x7fd9b01055c0 2024-03-19T02:43:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b019c0c0 con 0x7fd9b01055c0 2024-03-19T02:43:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9acff9640 1 -- 172.21.15.116:0/43841517 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a8044070 con 0x7fd9b01055c0 2024-03-19T02:43:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9acff9640 1 -- 172.21.15.116:0/43841517 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8032850 con 0x7fd9b01055c0 2024-03-19T02:43:28.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.417+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b01a39b0 con 0x7fd9b01055c0 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.420+0000 7fd9acff9640 1 -- 172.21.15.116:0/43841517 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd9a80329f0 con 0x7fd9b01055c0 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.420+0000 7fd9b8252740 1 Processor -- start 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.420+0000 7fd9b8252740 1 -- start start 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9acff9640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 0x7fd99c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 0x7fd99c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9acff9640 1 -- 172.21.15.116:0/43841517 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd9a80f1430 con 0x7fd9b01055c0 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb89e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb929f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599afaed500 con 0x5599afb92f30 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5599afa95be0 con 0x5599afb89a30 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9b8252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5599afae95a0 con 0x5599afb8a350 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46926/0 (socket says 172.21.15.116:46926) 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 -- 172.21.15.116:0/1980972962 learned_addr learned my addr 172.21.15.116:0/1980972962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb89e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 0x7fd99c0a4eb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0077e90 tx=0x7fd998011a90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 -- 172.21.15.116:0/1980972962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 msgr2=0x5599afb929f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb929f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 -- 172.21.15.116:0/1980972962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 msgr2=0x5599afb89e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb89e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9af7fe640 1 -- 172.21.15.116:0/1980972962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599afab7000 con 0x5599afb92f30 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb89e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.421+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb929f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.422+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80c5a30 tx=0x7fd9a80bd3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef2c3decc6a7c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.422+0000 7fd9a57fa640 1 -- 172.21.15.116:0/1980972962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80bdec0 con 0x5599afb92f30 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.422+0000 7fd9a57fa640 1 -- 172.21.15.116:0/1980972962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd7c0 con 0x5599afb92f30 2024-03-19T02:43:28.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.422+0000 7fd9a57fa640 1 -- 172.21.15.116:0/1980972962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80f1250 con 0x5599afb92f30 2024-03-19T02:43:28.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- 172.21.15.116:0/1980972962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 msgr2=0x5599afb95320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80c5a30 tx=0x7fd9a80bd3c0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- 172.21.15.116:0/1980972962 shutdown_connections 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb95320 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb929f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 --2- 172.21.15.116:0/1980972962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb89e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- 172.21.15.116:0/1980972962 >> 172.21.15.116:0/1980972962 conn(0x5599afb84160 msgr2=0x5599afb84540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- 172.21.15.116:0/1980972962 shutdown_connections 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- 172.21.15.116:0/1980972962 wait complete. 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.423+0000 7fd9b8252740 1 -- start start 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 Processor -- start 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb89a30 0x5599afb8e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599afb959e0 con 0x5599afb92f30 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5599afaa9160 con 0x5599afb8a350 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5599afb95b60 con 0x5599afb89a30 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9b8252740 10 client.0 _ll_register_callbacks cb 0x5599afae5a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46930/0 (socket says 172.21.15.116:46930) 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 -- 172.21.15.116:0/3507237669 learned_addr learned my addr 172.21.15.116:0/3507237669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 -- 172.21.15.116:0/3507237669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb89a30 msgr2=0x5599afb8e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb89a30 0x5599afb8e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 -- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 msgr2=0x5599afb8e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.424+0000 7fd9af7fe640 1 -- 172.21.15.116:0/3507237669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599af8cb7d0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fd9a802c760 tx=0x7fd9a80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0947dcb474ee5c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80c5f60 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd9a80a80e0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a80bd7c0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd9a80bdec0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9b8252740 10 client.20007 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5599af8013f0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.425+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd7c0 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd9a803c030 con 0x5599afb92f30 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 10 client.20007 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 10 client.20007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 20 client.20007 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:28.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afba7240 0x5599afba9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 4 client.20007 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:28.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.426+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599afafaa40 con 0x5599afba7240 2024-03-19T02:43:28.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.427+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afba7240 0x5599afba9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.427+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afba7240 0x5599afba9630 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f161bbfa49754fa5 server_cookie=28959afac98da8d6 in_seq=0 out_seq=0 2024-03-19T02:43:28.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.428+0000 7fd9967fc640 10 client.20007 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9967fc640 1 -- 172.21.15.116:0/3507237669 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599afafaa40 con 0x5599afba7240 2024-03-19T02:43:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9967fc640 10 client.20007 handle_command_reply: tid=0 2024-03-19T02:43:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9b8252740 1 client.20007 shutdown 2024-03-19T02:43:28.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 msgr2=0x5599afb8ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fd9a802c760 tx=0x7fd9a80095c0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.429+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 shutdown_connections 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afba7240 0x5599afba9630 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9967fc640 0 client.20007 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb92f30 0x5599afb8ebd0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3507237669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb89a30 0x5599afb8e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 >> 172.21.15.116:0/3507237669 conn(0x5599afb84160 msgr2=0x5599afb85eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 shutdown_connections 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 1 -- 172.21.15.116:0/3507237669 wait complete. 2024-03-19T02:43:28.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.430+0000 7fd9b8252740 20 client.20007 trim_cache size 0 max 16384 2024-03-19T02:43:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.455+0000 7fd9b8252740 1 Processor -- start 2024-03-19T02:43:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.455+0000 7fd9b8252740 1 -- start start 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb84160 0x5599afb8d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01ac9d0 con 0x5599afb89a30 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01ae510 con 0x5599afb8a350 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9b8252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01ad0a0 con 0x5599afb84160 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46944/0 (socket says 172.21.15.116:46944) 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9af7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 -- 172.21.15.116:0/3934272585 learned_addr learned my addr 172.21.15.116:0/3934272585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb84160 0x5599afb8d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 -- 172.21.15.116:0/3934272585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb84160 msgr2=0x5599afb8d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb84160 0x5599afb8d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 -- 172.21.15.116:0/3934272585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 msgr2=0x5599afb8b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9a7fff640 1 -- 172.21.15.116:0/3934272585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599afb8a180 con 0x5599afb89a30 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.456+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.457+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fd998011f90 tx=0x7fd99801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed11874e7f520d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.457+0000 7fd9a4ff9640 1 -- 172.21.15.116:0/3934272585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998005020 con 0x5599afb89a30 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.457+0000 7fd9a4ff9640 1 -- 172.21.15.116:0/3934272585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9980047a0 con 0x5599afb89a30 2024-03-19T02:43:28.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.457+0000 7fd9a4ff9640 1 -- 172.21.15.116:0/3934272585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998027650 con 0x5599afb89a30 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- 172.21.15.116:0/3934272585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 msgr2=0x5599afbaee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fd998011f90 tx=0x7fd99801d000 comp rx=0 tx=0).stop 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- 172.21.15.116:0/3934272585 shutdown_connections 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb8a350 0x5599afb8b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb89a30 0x5599afbaee60 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3934272585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb84160 0x5599afb8d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- 172.21.15.116:0/3934272585 >> 172.21.15.116:0/3934272585 conn(0x5599afb8e7d0 msgr2=0x5599afb82950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- 172.21.15.116:0/3934272585 shutdown_connections 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- 172.21.15.116:0/3934272585 wait complete. 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 -- start start 2024-03-19T02:43:28.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.458+0000 7fd9b8252740 1 Processor -- start 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb96640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb96b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01ade40 con 0x5599afb84160 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01ad770 con 0x5599afb89a30 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b01aa9d0 con 0x5599afb8a350 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9b8252740 10 client.0 _ll_register_callbacks cb 0x7fd9a80c1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46960/0 (socket says 172.21.15.116:46960) 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9a7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb96640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 -- 172.21.15.116:0/3513193113 learned_addr learned my addr 172.21.15.116:0/3513193113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9af7fe640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb96b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 -- 172.21.15.116:0/3513193113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 msgr2=0x5599afb96b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb96b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 -- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 msgr2=0x5599afb96640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb96640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 -- 172.21.15.116:0/3513193113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599af7c10f0 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9aeffd640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x5599afba9dd0 tx=0x7fd9a00044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de39a680bce198ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.459+0000 7fd9967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a00035f0 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9980047a0 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd9a0003790 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd9a002d050 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 10 client.20013 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5599afc0e320 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9980047a0 con 0x5599afb84160 2024-03-19T02:43:28.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd9a0010a00 con 0x5599afb84160 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 10 client.20013 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 10 client.20013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 20 client.20013 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afb92f30 0x5599afc10b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 4 client.20013 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.460+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5599afaf4d90 con 0x5599afb92f30 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.461+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afb92f30 0x5599afc10b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.461+0000 7fd9a7fff640 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afb92f30 0x5599afc10b10 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2cfc93c040ca3f3 server_cookie=a3644a188ceaa95f in_seq=0 out_seq=0 2024-03-19T02:43:28.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.461+0000 7fd9967fc640 10 client.20013 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.463+0000 7fd9967fc640 1 -- 172.21.15.116:0/3513193113 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5599afaf4d90 con 0x5599afb92f30 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.463+0000 7fd9967fc640 10 client.20013 handle_command_reply: tid=0 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.463+0000 7fd9b8252740 1 client.20013 shutdown 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.463+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 msgr2=0x5599afba29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.463+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x5599afba9dd0 tx=0x7fd9a00044f0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 shutdown_connections 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5599afb92f30 0x5599afc10b10 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5599afb8a350 0x5599afb96b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599afb89a30 0x5599afb96640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9967fc640 0 client.20013 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 --2- 172.21.15.116:0/3513193113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599afb84160 0x5599afba29f0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 >> 172.21.15.116:0/3513193113 conn(0x5599afb8e7d0 msgr2=0x5599afb831e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 shutdown_connections 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 1 -- 172.21.15.116:0/3513193113 wait complete. 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.464+0000 7fd9b8252740 20 client.20013 trim_cache size 0 max 16384 2024-03-19T02:43:28.468 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:43:28.469 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:43:28.469 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:43:28.469 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 msgr2=0x7fd99c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 0x7fd99c0a4eb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0077e90 tx=0x7fd998011a90 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 msgr2=0x7fd9b019b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a802c790 tx=0x7fd9a8035dd0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 shutdown_connections 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd99c0a29f0 0x7fd99c0a4eb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9b01055c0 0x7fd9b019b700 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b0101080 0x7fd9b0197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 --2- 172.21.15.116:0/43841517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9b01006e0 0x7fd9b0196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 >> 172.21.15.116:0/43841517 conn(0x7fd9b00777b0 msgr2=0x7fd9b00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 shutdown_connections 2024-03-19T02:43:28.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.465+0000 7fd9b5661640 1 -- 172.21.15.116:0/43841517 wait complete. 2024-03-19T02:43:28.478 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:43:28.478 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2102866684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 msgr2=0x7f0eac10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2102866684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac10b850 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009900 tx=0x7f0e9c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2102866684 shutdown_connections 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2102866684 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 0x7f0eac10e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2102866684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac10b850 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2102866684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 0x7f0eac102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2102866684 >> 172.21.15.116:0/2102866684 conn(0x7f0eac076aa0 msgr2=0x7f0eac076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2102866684 shutdown_connections 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.625+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2102866684 wait complete. 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 Processor -- start 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 -- start start 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 0x7f0eac197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 0x7f0eac19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac111a60 con 0x7f0eac103370 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1118e0 con 0x7f0eac102a50 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb44dd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac111be0 con 0x7f0eac10be20 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46980/0 (socket says 172.21.15.116:46980) 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb2279640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 0x7f0eac197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 -- 172.21.15.116:0/2116050028 learned_addr learned my addr 172.21.15.116:0/2116050028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb2a7a640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 0x7f0eac19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 -- 172.21.15.116:0/2116050028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 msgr2=0x7f0eac19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 0x7f0eac19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 -- 172.21.15.116:0/2116050028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 msgr2=0x7f0eac197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 0x7f0eac197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.626+0000 7f0eb1a78640 1 -- 172.21.15.116:0/2116050028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac19cfd0 con 0x7f0eac103370 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0eb1a78640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c005940 tx=0x7f0e9c02c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c043020 con 0x7f0eac103370 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac19d260 con 0x7f0eac103370 2024-03-19T02:43:28.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e9c004e00 con 0x7f0eac103370 2024-03-19T02:43:28.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c003970 con 0x7f0eac103370 2024-03-19T02:43:28.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.627+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1a4b50 con 0x7f0eac103370 2024-03-19T02:43:28.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.628+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac06a670 con 0x7f0eac103370 2024-03-19T02:43:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.629+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0e9c046020 con 0x7f0eac103370 2024-03-19T02:43:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.630+0000 7f0ea37fe640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 0x7f0e980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.630+0000 7f0eb2279640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 0x7f0e980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.630+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f0e9c0f10b0 con 0x7f0eac103370 2024-03-19T02:43:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.631+0000 7f0eb2279640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 0x7f0e980a4e30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009bd0 tx=0x7f0ea8005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.636+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e9c04d020 con 0x7f0eac103370 2024-03-19T02:43:28.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.956+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eac106f40 con 0x7f0eac103370 2024-03-19T02:43:28.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.957+0000 7f0ea37fe640 1 -- 172.21.15.116:0/2116050028 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f0e9c0bdb40 con 0x7f0eac103370 2024-03-19T02:43:28.958 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:43:28.958 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:43:28.958 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:43:28.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 msgr2=0x7f0e980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 0x7f0e980a4e30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009bd0 tx=0x7f0ea8005f90 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 msgr2=0x7f0eac198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c005940 tx=0x7f0e9c02c690 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 shutdown_connections 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0e980a2970 0x7f0e980a4e30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0eac10be20 0x7f0eac19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eac103370 0x7f0eac198520 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 --2- 172.21.15.116:0/2116050028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eac102a50 0x7f0eac197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.959+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 >> 172.21.15.116:0/2116050028 conn(0x7f0eac076aa0 msgr2=0x7f0eac0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.960+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 shutdown_connections 2024-03-19T02:43:28.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:28.960+0000 7f0eb44dd640 1 -- 172.21.15.116:0/2116050028 wait complete. 2024-03-19T02:43:28.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 -- 172.21.15.116:0/629368436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c100970 msgr2=0x7fb23c100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 --2- 172.21.15.116:0/629368436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c100970 0x7fb23c100dd0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb230009900 tx=0x7fb23002bc30 comp rx=0 tx=0).stop 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 -- 172.21.15.116:0/629368436 shutdown_connections 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 --2- 172.21.15.116:0/629368436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c101310 0x7fb23c10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 --2- 172.21.15.116:0/629368436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c100970 0x7fb23c100dd0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 --2- 172.21.15.116:0/629368436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c1056a0 0x7fb23c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 -- 172.21.15.116:0/629368436 >> 172.21.15.116:0/629368436 conn(0x7fb23c0774f0 msgr2=0x7fb23c077900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 -- 172.21.15.116:0/629368436 shutdown_connections 2024-03-19T02:43:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 -- 172.21.15.116:0/629368436 wait complete. 2024-03-19T02:43:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.130+0000 7fb24167e640 1 Processor -- start 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 -- start start 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 0x7fb23c10e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46998/0 (socket says 172.21.15.116:46998) 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57052/0 (socket says 172.21.15.116:57052) 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23affd640 1 -- 172.21.15.116:0/2234016426 learned_addr learned my addr 172.21.15.116:0/2234016426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c111bd0 con 0x7fb23c101310 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c111a50 con 0x7fb23c100970 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c111d50 con 0x7fb23c1056a0 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.131+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 0x7fb23c10e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 -- 172.21.15.116:0/2234016426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 msgr2=0x7fb23c10de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 -- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 msgr2=0x7fb23c10e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c1aa9d0 con 0x7fb23c1056a0 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23affd640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 0x7fb23c10e8e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb234009fd0 tx=0x7fb23400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb223fff640 1 -- 172.21.15.116:0/2234016426 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23400ce20 con 0x7fb23c1056a0 2024-03-19T02:43:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c1aac00 con 0x7fb23c1056a0 2024-03-19T02:43:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.132+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c1ab170 con 0x7fb23c1056a0 2024-03-19T02:43:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.133+0000 7fb223fff640 1 -- 172.21.15.116:0/2234016426 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb234010430 con 0x7fb23c1056a0 2024-03-19T02:43:29.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.133+0000 7fb223fff640 1 -- 172.21.15.116:0/2234016426 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23400f620 con 0x7fb23c1056a0 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.135+0000 7fb223fff640 1 -- 172.21.15.116:0/2234016426 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb234004a40 con 0x7fb23c1056a0 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.135+0000 7fb244267740 1 Processor -- start 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 -- start start 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 0x55d55a90a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a913290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a915bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d55a86d4b0 con 0x55d55a9137d0 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d55a815d70 con 0x55d55a90abf0 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb244267740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d55a869550 con 0x55d55a90a2d0 2024-03-19T02:43:29.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb223fff640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 0x7fb2240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 0x55d55a90a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a913290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 0x55d55a90a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36846/0 (socket says 172.21.15.116:36846) 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23a7fc640 1 -- 172.21.15.116:0/3754800632 learned_addr learned my addr 172.21.15.116:0/3754800632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a915bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 0x7fb2240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 -- 172.21.15.116:0/3754800632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 msgr2=0x55d55a90a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 0x55d55a90a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 -- 172.21.15.116:0/3754800632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 msgr2=0x55d55a915bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a915bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb23affd640 1 -- 172.21.15.116:0/3754800632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d55a838210 con 0x55d55a90abf0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.136+0000 7fb223fff640 1 -- 172.21.15.116:0/2234016426 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fb2340cf490 con 0x7fb23c1056a0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb23affd640 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a913290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0058c0 tx=0x7fb22c010820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d221a0c9f530c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb23b7fe640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 0x7fb2240a4e30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb2340980e0 tx=0x7fb23406e3f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb2217fa640 1 -- 172.21.15.116:0/3754800632 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c011700 con 0x55d55a90abf0 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb2217fa640 1 -- 172.21.15.116:0/3754800632 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb22c011d50 con 0x55d55a90abf0 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb2217fa640 1 -- 172.21.15.116:0/3754800632 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c01bdc0 con 0x55d55a90abf0 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb244267740 1 -- 172.21.15.116:0/3754800632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 msgr2=0x55d55a913290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.137+0000 7fb244267740 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a913290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0058c0 tx=0x7fb22c010820 comp rx=0 tx=0).stop 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 -- 172.21.15.116:0/3754800632 shutdown_connections 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a915bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a913290 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 --2- 172.21.15.116:0/3754800632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90a2d0 0x55d55a90a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 -- 172.21.15.116:0/3754800632 >> 172.21.15.116:0/3754800632 conn(0x55d55a904a00 msgr2=0x55d55a904de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 -- 172.21.15.116:0/3754800632 shutdown_connections 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 -- 172.21.15.116:0/3754800632 wait complete. 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 -- start start 2024-03-19T02:43:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.138+0000 7fb244267740 1 Processor -- start 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 0x55d55a910710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a911190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d55a9165c0 con 0x55d55a9137d0 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d55a829310 con 0x55d55a90a2d0 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d55a916740 con 0x55d55a90abf0 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 0x55d55a910710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36850/0 (socket says 172.21.15.116:36850) 2024-03-19T02:43:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 -- 172.21.15.116:0/115230589 learned_addr learned my addr 172.21.15.116:0/115230589 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb244267740 10 client.0 _ll_register_callbacks cb 0x55d55a81d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a911190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 -- 172.21.15.116:0/115230589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 msgr2=0x55d55a910710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 0x55d55a910710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 -- 172.21.15.116:0/115230589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 msgr2=0x55d55a911190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a911190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23affd640 1 -- 172.21.15.116:0/115230589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d55a5813f0 con 0x55d55a90abf0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.139+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 0x55d55a910710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb23affd640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb22c01baa0 tx=0x7fb22c01b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1945bd5bfe4d7cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb21eff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c02a060 con 0x55d55a90abf0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fb22c004250 con 0x55d55a90abf0 2024-03-19T02:43:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb22c011d50 con 0x55d55a90abf0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb22c031050 con 0x55d55a90abf0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb244267740 10 client.29749 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d55a64b7d0 con 0x55d55a90abf0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.140+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb22c011d50 con 0x55d55a90abf0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fb22c0116c0 con 0x55d55a90abf0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 10 client.29749 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 10 client.29749 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 20 client.29749 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a923a20 0x55d55a925e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 4 client.29749 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d55a8cb3e0 con 0x55d55a923a20 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a923a20 0x55d55a925e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a923a20 0x55d55a925e10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b27b0dd88fc4f9 server_cookie=a6778b30cacac576 in_seq=0 out_seq=0 2024-03-19T02:43:29.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.141+0000 7fb21eff5640 10 client.29749 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb21eff5640 1 -- 172.21.15.116:0/115230589 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d55a8cb3e0 con 0x55d55a923a20 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb21eff5640 10 client.29749 handle_command_reply: tid=0 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 client.29749 shutdown 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 msgr2=0x55d55a910c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb22c01baa0 tx=0x7fb22c01b660 comp rx=0 tx=0).stop 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 shutdown_connections 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a923a20 0x55d55a925e10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb21eff5640 0 client.29749 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a9137d0 0x55d55a911190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a90abf0 0x55d55a910c50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.142+0000 7fb244267740 1 --2- 172.21.15.116:0/115230589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90a2d0 0x55d55a910710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.143+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 >> 172.21.15.116:0/115230589 conn(0x55d55a904a00 msgr2=0x55d55a906750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.143+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 shutdown_connections 2024-03-19T02:43:29.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.143+0000 7fb244267740 1 -- 172.21.15.116:0/115230589 wait complete. 2024-03-19T02:43:29.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.143+0000 7fb244267740 20 client.29749 trim_cache size 0 max 16384 2024-03-19T02:43:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.162+0000 7fb244267740 1 Processor -- start 2024-03-19T02:43:29.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.162+0000 7fb244267740 1 -- start start 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a91f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a910fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb22800ff60 con 0x55d55a90a2d0 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb228011aa0 con 0x55d55a90abf0 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb244267740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb228010630 con 0x55d55a9137d0 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47036/0 (socket says 172.21.15.116:47036) 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 -- 172.21.15.116:0/3328689520 learned_addr learned my addr 172.21.15.116:0/3328689520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23affd640 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a91f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a910fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 -- 172.21.15.116:0/3328689520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 msgr2=0x55d55a910fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a910fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 -- 172.21.15.116:0/3328689520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 msgr2=0x55d55a91f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a91f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.163+0000 7fb23a7fc640 1 -- 172.21.15.116:0/3328689520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d55a8da880 con 0x55d55a90a2d0 2024-03-19T02:43:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.164+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb23002bc70 tx=0x7fb230005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34759114d33a682e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.165+0000 7fb220ff9640 1 -- 172.21.15.116:0/3328689520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23002cbb0 con 0x55d55a90a2d0 2024-03-19T02:43:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.165+0000 7fb220ff9640 1 -- 172.21.15.116:0/3328689520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb230052810 con 0x55d55a90a2d0 2024-03-19T02:43:29.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.165+0000 7fb220ff9640 1 -- 172.21.15.116:0/3328689520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230003c50 con 0x55d55a90a2d0 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- 172.21.15.116:0/3328689520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 msgr2=0x55d55a912e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb23002bc70 tx=0x7fb230005a20 comp rx=0 tx=0).stop 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- 172.21.15.116:0/3328689520 shutdown_connections 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a910fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a90abf0 0x55d55a91f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 --2- 172.21.15.116:0/3328689520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90a2d0 0x55d55a912e40 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- 172.21.15.116:0/3328689520 >> 172.21.15.116:0/3328689520 conn(0x55d55a904a00 msgr2=0x55d55a903ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- 172.21.15.116:0/3328689520 shutdown_connections 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- 172.21.15.116:0/3328689520 wait complete. 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 -- start start 2024-03-19T02:43:29.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.166+0000 7fb244267740 1 Processor -- start 2024-03-19T02:43:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 0x55d55a91bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a917590 0x55d55a9179d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2280113d0 con 0x55d55a90abf0 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb228010d00 con 0x55d55a917590 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb22800df60 con 0x55d55a9137d0 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb244267740 10 client.0 _ll_register_callbacks cb 0x7fb2300041d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 0x55d55a91bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36870/0 (socket says 172.21.15.116:36870) 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 -- 172.21.15.116:0/4107600792 learned_addr learned my addr 172.21.15.116:0/4107600792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 -- 172.21.15.116:0/4107600792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a917590 msgr2=0x55d55a9179d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a917590 0x55d55a9179d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 -- 172.21.15.116:0/4107600792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 msgr2=0x55d55a91bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 0x55d55a91bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 -- 172.21.15.116:0/4107600792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d55a988c70 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 0x55d55a91bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb23affd640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb22c010860 tx=0x7fb22c01be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02b126d76e41e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.167+0000 7fb21eff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c02a040 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb230052810 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fb22c004680 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb22c01b6d0 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb244267740 10 client.29755 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d55a989fa0 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.168+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb230052810 con 0x55d55a9137d0 2024-03-19T02:43:29.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fb22c004290 con 0x55d55a9137d0 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 10 client.29755 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 10 client.29755 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 20 client.29755 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a98a6c0 0x55d55a98cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 4 client.29755 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d55a87a9f0 con 0x55d55a98a6c0 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.169+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a98a6c0 0x55d55a98cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.170+0000 7fb21eff5640 10 client.29755 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.170+0000 7fb23a7fc640 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a98a6c0 0x55d55a98cae0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7f0a95b923aefeae server_cookie=1d2c4d9b9a85a2ac in_seq=0 out_seq=0 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.170+0000 7fb21eff5640 1 -- 172.21.15.116:0/4107600792 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d55a87a9f0 con 0x55d55a98a6c0 2024-03-19T02:43:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.170+0000 7fb21eff5640 10 client.29755 handle_command_reply: tid=0 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.170+0000 7fb244267740 1 client.29755 shutdown 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 msgr2=0x55d55a917050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb22c010860 tx=0x7fb22c01be10 comp rx=0 tx=0).stop 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 shutdown_connections 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d55a98a6c0 0x55d55a98cae0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb21eff5640 0 client.29755 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d55a917590 0x55d55a9179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d55a9137d0 0x55d55a917050 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 --2- 172.21.15.116:0/4107600792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d55a90abf0 0x55d55a91bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 >> 172.21.15.116:0/4107600792 conn(0x55d55a904a00 msgr2=0x55d55a90eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 shutdown_connections 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 1 -- 172.21.15.116:0/4107600792 wait complete. 2024-03-19T02:43:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.171+0000 7fb244267740 20 client.29755 trim_cache size 0 max 16384 2024-03-19T02:43:29.175 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 msgr2=0x7fb2240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 0x7fb2240a4e30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb2340980e0 tx=0x7fb23406e3f0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 msgr2=0x7fb23c10e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 0x7fb23c10e8e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb234009fd0 tx=0x7fb23400ef10 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 shutdown_connections 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb2240a2970 0x7fb2240a4e30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb23c1056a0 0x7fb23c10e8e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb23c101310 0x7fb23c10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 --2- 172.21.15.116:0/2234016426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c100970 0x7fb23c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 >> 172.21.15.116:0/2234016426 conn(0x7fb23c0774f0 msgr2=0x7fb23c0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 shutdown_connections 2024-03-19T02:43:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.174+0000 7fb24167e640 1 -- 172.21.15.116:0/2234016426 wait complete. 2024-03-19T02:43:29.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:43:29.224 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:29 smithi116 ceph-mon[32263]: pgmap v493: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:29.224 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:29 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2116050028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:43:29.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19870 6.02 MB/sec execute 51 sec latency 124.646 ms 2024-03-19T02:43:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 -- 172.21.15.116:0/717660004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 msgr2=0x7f6b1c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 --2- 172.21.15.116:0/717660004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c10d570 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009cd0 tx=0x7f6b1402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 -- 172.21.15.116:0/717660004 shutdown_connections 2024-03-19T02:43:29.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 --2- 172.21.15.116:0/717660004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c10d570 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 --2- 172.21.15.116:0/717660004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 --2- 172.21.15.116:0/717660004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 -- 172.21.15.116:0/717660004 >> 172.21.15.116:0/717660004 conn(0x7f6b1c0778d0 msgr2=0x7f6b1c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 -- 172.21.15.116:0/717660004 shutdown_connections 2024-03-19T02:43:29.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.338+0000 7f6b22359640 1 -- 172.21.15.116:0/717660004 wait complete. 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 Processor -- start 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 -- start start 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1109f0 con 0x7f6b1c101060 2024-03-19T02:43:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.339+0000 7f6b22359640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c110870 con 0x7f6b1c1066b0 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36874/0 (socket says 172.21.15.116:36874) 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1bfff640 1 -- 172.21.15.116:0/2228439900 learned_addr learned my addr 172.21.15.116:0/2228439900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b208f6640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c110b70 con 0x7f6b1c1006a0 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2228439900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 msgr2=0x7f6b1c196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2228439900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 msgr2=0x7f6b1c19bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2228439900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c19c1f0 con 0x7f6b1c101060 2024-03-19T02:43:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c1971a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b3f0 tx=0x7f6b1000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.340+0000 7f6b197fa640 1 -- 172.21.15.116:0/2228439900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b10015020 con 0x7f6b1c101060 2024-03-19T02:43:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.341+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1c19c4e0 con 0x7f6b1c101060 2024-03-19T02:43:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.341+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c19ca80 con 0x7f6b1c101060 2024-03-19T02:43:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.341+0000 7f6b197fa640 1 -- 172.21.15.116:0/2228439900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b10011ce0 con 0x7f6b1c101060 2024-03-19T02:43:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.341+0000 7f6b197fa640 1 -- 172.21.15.116:0/2228439900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b10010d80 con 0x7f6b1c101060 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.343+0000 7f6b197fa640 1 -- 172.21.15.116:0/2228439900 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6b10018020 con 0x7f6b1c101060 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.343+0000 7f6b197fa640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 0x7f6b040a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b197fa640 1 -- 172.21.15.116:0/2228439900 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6b100ce7b0 con 0x7f6b1c101060 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 Processor -- start 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 -- start start 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 0x7f6b040a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x5609806326b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098063dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609805954b0 con 0x560980632bf0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56098053dd70 con 0x56098063b7d0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b25013740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560980591550 con 0x5609806322d0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47048/0 (socket says 172.21.15.116:47048) 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2508061510 learned_addr learned my addr 172.21.15.116:0/2508061510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x5609806326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b208f6640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098063dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 0x7f6b040a4eb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009a90 tx=0x7f6b0c009e90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.344+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2508061510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 msgr2=0x5609806326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x5609806326b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2508061510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 msgr2=0x56098063dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098063dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2508061510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560980560210 con 0x560980632bf0 2024-03-19T02:43:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x5609806326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f6b1009a840 tx=0x7f6b1009b240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b46e2f988ce278 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b027fc640 1 -- 172.21.15.116:0/2508061510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1009ec90 con 0x560980632bf0 2024-03-19T02:43:29.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b027fc640 1 -- 172.21.15.116:0/2508061510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b100a32e0 con 0x560980632bf0 2024-03-19T02:43:29.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.345+0000 7f6b027fc640 1 -- 172.21.15.116:0/2508061510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b10010380 con 0x560980632bf0 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- 172.21.15.116:0/2508061510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 msgr2=0x56098063b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f6b1009a840 tx=0x7f6b1009b240 comp rx=0 tx=0).stop 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- 172.21.15.116:0/2508061510 shutdown_connections 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098063dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x56098063b290 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 --2- 172.21.15.116:0/2508061510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x5609806326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- 172.21.15.116:0/2508061510 >> 172.21.15.116:0/2508061510 conn(0x56098062ca00 msgr2=0x56098062cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- 172.21.15.116:0/2508061510 shutdown_connections 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- 172.21.15.116:0/2508061510 wait complete. 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 -- start start 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.346+0000 7f6b25013740 1 Processor -- start 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980638800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 0x560980638d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56098064f5d0 con 0x56098063b7d0 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47056/0 (socket says 172.21.15.116:47056) 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980638800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 0x560980638d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 -- 172.21.15.116:0/30991882 learned_addr learned my addr 172.21.15.116:0/30991882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560980551310 con 0x560980632bf0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56098064f750 con 0x5609806322d0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b25013740 10 client.0 _ll_register_callbacks cb 0x5609805452d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 -- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 msgr2=0x560980638800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980638800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 -- 172.21.15.116:0/30991882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 msgr2=0x560980638d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 0x560980638d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b208f6640 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609802a93f0 con 0x56098063b7d0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980638800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.347+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 0x560980638d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6b208f6640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009ca0 tx=0x7f6b14032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5696b20824aaedba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14044070 con 0x56098063b7d0 2024-03-19T02:43:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6b140327e0 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b100a32e0 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6b14051050 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6b25013740 10 client.20049 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5609803737d0 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b100a32e0 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.348+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6b140046e0 con 0x56098063b7d0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 10 client.20049 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 10 client.20049 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 20 client.20049 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56098065cbf0 0x56098065f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 4 client.20049 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609805f33e0 con 0x56098065cbf0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56098065cbf0 0x56098065f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6afb7fe640 10 client.20049 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.349+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56098065cbf0 0x56098065f060 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8c54650f401e3ed2 server_cookie=59614d0ca08f4f0a in_seq=0 out_seq=0 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6afb7fe640 1 -- 172.21.15.116:0/30991882 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609805f33e0 con 0x56098065cbf0 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6afb7fe640 10 client.20049 handle_command_reply: tid=0 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 client.20049 shutdown 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 msgr2=0x560980639280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009ca0 tx=0x7f6b14032670 comp rx=0 tx=0).stop 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 shutdown_connections 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56098065cbf0 0x56098065f060 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980639280 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560980632bf0 0x560980638d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 --2- 172.21.15.116:0/30991882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980638800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6afb7fe640 0 client.20049 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 >> 172.21.15.116:0/30991882 conn(0x56098062ca00 msgr2=0x56098062e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 shutdown_connections 2024-03-19T02:43:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 1 -- 172.21.15.116:0/30991882 wait complete. 2024-03-19T02:43:29.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.350+0000 7f6b25013740 20 client.20049 trim_cache size 0 max 16384 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 Processor -- start 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 -- start start 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609806322d0 0x5609806380e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560980632bf0 0x5609806583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0800ff60 con 0x56098063b7d0 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08011aa0 con 0x5609806322d0 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b25013740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08010630 con 0x560980632bf0 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47060/0 (socket says 172.21.15.116:47060) 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 -- 172.21.15.116:0/442411361 learned_addr learned my addr 172.21.15.116:0/442411361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609806322d0 0x5609806380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560980632bf0 0x5609806583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 -- 172.21.15.116:0/442411361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560980632bf0 msgr2=0x5609806583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560980632bf0 0x5609806583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 -- 172.21.15.116:0/442411361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609806322d0 msgr2=0x5609806380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609806322d0 0x5609806380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 -- 172.21.15.116:0/442411361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56098063dbc0 con 0x56098063b7d0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b208f6640 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f6b140098d0 tx=0x7f6b14005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908935cdfe345759 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.367+0000 7f6b01ffb640 1 -- 172.21.15.116:0/442411361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14044070 con 0x56098063b7d0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b01ffb640 1 -- 172.21.15.116:0/442411361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b14032480 con 0x56098063b7d0 2024-03-19T02:43:29.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b01ffb640 1 -- 172.21.15.116:0/442411361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14039040 con 0x56098063b7d0 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 -- 172.21.15.116:0/442411361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 msgr2=0x560980664930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f6b140098d0 tx=0x7f6b14005f70 comp rx=0 tx=0).stop 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 -- 172.21.15.116:0/442411361 shutdown_connections 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098063b7d0 0x560980664930 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560980632bf0 0x5609806583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 --2- 172.21.15.116:0/442411361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609806322d0 0x5609806380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.368+0000 7f6b25013740 1 -- 172.21.15.116:0/442411361 >> 172.21.15.116:0/442411361 conn(0x56098062ca00 msgr2=0x56098062b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- 172.21.15.116:0/442411361 shutdown_connections 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- 172.21.15.116:0/442411361 wait complete. 2024-03-19T02:43:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- start start 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 Processor -- start 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980654570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098064fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b080113d0 con 0x560980632bf0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08010d00 con 0x56098063b7d0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0800df60 con 0x5609806322d0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b25013740 10 client.0 _ll_register_callbacks cb 0x7f6b140323a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980654570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47074/0 (socket says 172.21.15.116:47074) 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/2514059670 learned_addr learned my addr 172.21.15.116:0/2514059670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.369+0000 7f6b208f6640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098064fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 -- 172.21.15.116:0/2514059670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 msgr2=0x56098064fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098064fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 -- 172.21.15.116:0/2514059670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 msgr2=0x560980654ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 -- 172.21.15.116:0/2514059670 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560980650230 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980654570 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x560980628410 tx=0x7f6b0c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff87a94148d749ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6afb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c01a070 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b14032480 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6b0c028d30 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6b0c03a050 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b25013740 10 client.29770 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.370+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5609806c2a30 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b14032480 con 0x5609806322d0 2024-03-19T02:43:29.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6b0c01d040 con 0x5609806322d0 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 10 client.29770 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 10 client.29770 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 20 client.29770 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5609806c3160 0x5609806c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 4 client.29770 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609805a29f0 con 0x5609806c3160 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5609806c3160 0x5609806c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6afb7fe640 10 client.29770 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.371+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5609806c3160 0x5609806c5580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d4f70377f653309 server_cookie=5c82a14e164f31af in_seq=0 out_seq=0 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6afb7fe640 1 -- 172.21.15.116:0/2514059670 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609805a29f0 con 0x5609806c3160 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6afb7fe640 10 client.29770 handle_command_reply: tid=0 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 client.29770 shutdown 2024-03-19T02:43:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 msgr2=0x560980654570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980654570 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x560980628410 tx=0x7f6b0c013000 comp rx=0 tx=0).stop 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 shutdown_connections 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56098063b7d0 0x56098064fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5609806c3160 0x5609806c5580 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560980632bf0 0x560980654ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 --2- 172.21.15.116:0/2514059670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609806322d0 0x560980654570 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 >> 172.21.15.116:0/2514059670 conn(0x56098062ca00 msgr2=0x560980638770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6afb7fe640 0 client.29770 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 shutdown_connections 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 1 -- 172.21.15.116:0/2514059670 wait complete. 2024-03-19T02:43:29.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.372+0000 7f6b25013740 20 client.29770 trim_cache size 0 max 16384 2024-03-19T02:43:29.380 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:43:29.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 msgr2=0x7f6b040a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 0x7f6b040a4eb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009a90 tx=0x7f6b0c009e90 comp rx=0 tx=0).stop 2024-03-19T02:43:29.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 msgr2=0x7f6b1c1971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c1971a0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b3f0 tx=0x7f6b1000b8b0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 shutdown_connections 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6b040a29f0 0x7f6b040a4eb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1c1066b0 0x7f6b1c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c101060 0x7f6b1c1971a0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 --2- 172.21.15.116:0/2228439900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b1c1006a0 0x7f6b1c196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 >> 172.21.15.116:0/2228439900 conn(0x7f6b1c0778d0 msgr2=0x7f6b1c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 shutdown_connections 2024-03-19T02:43:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.373+0000 7f6b22359640 1 -- 172.21.15.116:0/2228439900 wait complete. 2024-03-19T02:43:29.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:43:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:29 smithi162 ceph-mon[35761]: pgmap v493: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:29 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2116050028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:43:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:29 smithi154 ceph-mon[35693]: pgmap v493: 129 pgs: 129 active+clean; 807 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 172 op/s 2024-03-19T02:43:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:29 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2116050028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:43:29.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 -- 172.21.15.116:0/526432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc102150 msgr2=0x7fd3dc10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/526432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc102150 0x7fd3dc10e660 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009cd0 tx=0x7fd3d402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 -- 172.21.15.116:0/526432100 shutdown_connections 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/526432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc102150 0x7fd3dc10e660 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/526432100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc101790 0x7fd3dc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/526432100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc1077a0 0x7fd3dc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 -- 172.21.15.116:0/526432100 >> 172.21.15.116:0/526432100 conn(0x7fd3dc0fd2c0 msgr2=0x7fd3dc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 -- 172.21.15.116:0/526432100 shutdown_connections 2024-03-19T02:43:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.524+0000 7fd3e12c6640 1 -- 172.21.15.116:0/526432100 wait complete. 2024-03-19T02:43:29.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.525+0000 7fd3e12c6640 1 Processor -- start 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.525+0000 7fd3e12c6640 1 -- start start 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.525+0000 7fd3e12c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc101790 0x7fd3dc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.525+0000 7fd3e12c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3e12c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc1077a0 0x7fd3dc19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3e12c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc111a60 con 0x7fd3dc1077a0 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3e12c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc1118e0 con 0x7fd3dc101790 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3e12c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc111be0 con 0x7fd3dc102150 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc101790 0x7fd3dc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36924/0 (socket says 172.21.15.116:36924) 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 -- 172.21.15.116:0/148862655 learned_addr learned my addr 172.21.15.116:0/148862655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc1077a0 0x7fd3dc19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 -- 172.21.15.116:0/148862655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc101790 msgr2=0x7fd3dc19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc101790 0x7fd3dc19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 -- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc1077a0 msgr2=0x7fd3dc19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc1077a0 0x7fd3dc19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 -- 172.21.15.116:0/148862655 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc19f170 con 0x7fd3dc102150 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.526+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000e9f0 tx=0x7fd3d000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.527+0000 7fd3c3fff640 1 -- 172.21.15.116:0/148862655 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000cd80 con 0x7fd3dc102150 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.527+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc19f460 con 0x7fd3dc102150 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.527+0000 7fd3c3fff640 1 -- 172.21.15.116:0/148862655 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3d00054d0 con 0x7fd3dc102150 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.527+0000 7fd3c3fff640 1 -- 172.21.15.116:0/148862655 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0010490 con 0x7fd3dc102150 2024-03-19T02:43:29.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.527+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc1a6cf0 con 0x7fd3dc102150 2024-03-19T02:43:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.530+0000 7fd3e3f90740 1 Processor -- start 2024-03-19T02:43:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.530+0000 7fd3e3f90740 1 -- start start 2024-03-19T02:43:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f5680a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f5680cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560f567534b0 con 0x560f567eff80 2024-03-19T02:43:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f566fbd70 con 0x560f5680a7e0 2024-03-19T02:43:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f5674f550 con 0x560f567f08a0 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47104/0 (socket says 172.21.15.116:47104) 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 -- 172.21.15.116:0/4275912662 learned_addr learned my addr 172.21.15.116:0/4275912662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 -- 172.21.15.116:0/4275912662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 msgr2=0x560f5680a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f5680a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 -- 172.21.15.116:0/4275912662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 msgr2=0x560f5680cbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f5680cbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3db7fe640 1 -- 172.21.15.116:0/4275912662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5671e210 con 0x560f567eff80 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.531+0000 7fd3c3fff640 1 -- 172.21.15.116:0/148862655 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd3d0005020 con 0x7fd3dc102150 2024-03-19T02:43:29.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.532+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd3d402c630 tx=0x7fd3d4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85af68276705372 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3c17fa640 1 -- 172.21.15.116:0/4275912662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4043030 con 0x560f567eff80 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3c3fff640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 0x7fd3c40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3c17fa640 1 -- 172.21.15.116:0/4275912662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3d4003e20 con 0x560f567eff80 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3c17fa640 1 -- 172.21.15.116:0/4275912662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4039040 con 0x560f567eff80 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3daffd640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 0x7fd3c40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.533+0000 7fd3c3fff640 1 -- 172.21.15.116:0/148862655 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd3d00cf530 con 0x7fd3dc102150 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3daffd640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 0x7fd3c40a4eb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009c20 tx=0x7fd3cc0097c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- 172.21.15.116:0/4275912662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 msgr2=0x560f567f0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd3d402c630 tx=0x7fd3d4005ef0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- 172.21.15.116:0/4275912662 shutdown_connections 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f5680cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f5680a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/4275912662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f567f0360 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- 172.21.15.116:0/4275912662 >> 172.21.15.116:0/4275912662 conn(0x560f567eaa00 msgr2=0x560f567eade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- 172.21.15.116:0/4275912662 shutdown_connections 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- 172.21.15.116:0/4275912662 wait complete. 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 -- start start 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.534+0000 7fd3e3f90740 1 Processor -- start 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f56807de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f56808320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560f5680d5c0 con 0x560f567eff80 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f5670f310 con 0x560f5680a7e0 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560f5680d740 con 0x560f567f08a0 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3e3f90740 10 client.0 _ll_register_callbacks cb 0x560f567032d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47114/0 (socket says 172.21.15.116:47114) 2024-03-19T02:43:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3da7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f56807de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3daffd640 1 -- 172.21.15.116:0/3443467777 learned_addr learned my addr 172.21.15.116:0/3443467777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.535+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f56808320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 -- 172.21.15.116:0/3443467777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 msgr2=0x560f56807de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f56807de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 -- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 msgr2=0x560f56808320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f56808320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 -- 172.21.15.116:0/3443467777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f564673f0 con 0x560f567eff80 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f56808320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:43:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f56807de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc004570 tx=0x7fd3cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3764d1a89a6e9053 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3beff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc005220 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd3cc0053c0 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.536+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d4003e20 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd3cc02e050 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 10 client.20067 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560f565317d0 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3d4003e20 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd3cc01e920 con 0x560f567eff80 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 10 client.20067 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 10 client.20067 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 20 client.20067 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f568234c0 0x560f568258d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 4 client.20067 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.537+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f567b13e0 con 0x560f568234c0 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.538+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f568234c0 0x560f568258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.538+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f568234c0 0x560f568258d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=996249e4a23bf413 server_cookie=45c6d3a469c7a939 in_seq=0 out_seq=0 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.538+0000 7fd3beff5640 10 client.20067 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3beff5640 1 -- 172.21.15.116:0/3443467777 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f567b13e0 con 0x560f568234c0 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3beff5640 10 client.20067 handle_command_reply: tid=0 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 client.20067 shutdown 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 msgr2=0x560f568078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc004570 tx=0x7fd3cc005f90 comp rx=0 tx=0).stop 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 shutdown_connections 2024-03-19T02:43:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f568234c0 0x560f568258d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f5680a7e0 0x560f56808320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f567f08a0 0x560f56807de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3443467777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f568078a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 >> 172.21.15.116:0/3443467777 conn(0x560f567eaa00 msgr2=0x560f567ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3beff5640 0 client.20067 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 shutdown_connections 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.539+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3443467777 wait complete. 2024-03-19T02:43:29.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.540+0000 7fd3e3f90740 20 client.20067 trim_cache size 0 max 16384 2024-03-19T02:43:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 Processor -- start 2024-03-19T02:43:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 -- start start 2024-03-19T02:43:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56808140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f5682b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc070d00 con 0x560f567eff80 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc072840 con 0x560f567f08a0 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc0713d0 con 0x560f5680a7e0 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.559+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47116/0 (socket says 172.21.15.116:47116) 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 -- 172.21.15.116:0/1137791212 learned_addr learned my addr 172.21.15.116:0/1137791212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56808140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f5682b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 -- 172.21.15.116:0/1137791212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 msgr2=0x560f5682b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f5682b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 -- 172.21.15.116:0/1137791212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 msgr2=0x560f56808140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56808140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 -- 172.21.15.116:0/1137791212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f56820280 con 0x560f567eff80 2024-03-19T02:43:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3daffd640 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc000c00 tx=0x7fd3cc003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b766bebe97750d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3c0ff9640 1 -- 172.21.15.116:0/1137791212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc01e630 con 0x560f567eff80 2024-03-19T02:43:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3c0ff9640 1 -- 172.21.15.116:0/1137791212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3cc03c410 con 0x560f567eff80 2024-03-19T02:43:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.560+0000 7fd3c0ff9640 1 -- 172.21.15.116:0/1137791212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc032660 con 0x560f567eff80 2024-03-19T02:43:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 -- 172.21.15.116:0/1137791212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 msgr2=0x560f5681ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc000c00 tx=0x7fd3cc003ed0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 -- 172.21.15.116:0/1137791212 shutdown_connections 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f5682b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56808140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/1137791212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681ec60 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 -- 172.21.15.116:0/1137791212 >> 172.21.15.116:0/1137791212 conn(0x560f567eaa00 msgr2=0x560f567e9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 -- 172.21.15.116:0/1137791212 shutdown_connections 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.561+0000 7fd3e3f90740 1 -- 172.21.15.116:0/1137791212 wait complete. 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 -- start start 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 Processor -- start 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56804110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f56804650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc072170 con 0x560f567eff80 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc071aa0 con 0x560f567f08a0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc06ed40 con 0x560f5680a7e0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3e3f90740 10 client.0 _ll_register_callbacks cb 0x7fd3cc009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47130/0 (socket says 172.21.15.116:47130) 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 -- 172.21.15.116:0/3293143533 learned_addr learned my addr 172.21.15.116:0/3293143533 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f56804650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56804110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 -- 172.21.15.116:0/3293143533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 msgr2=0x560f56804650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f56804650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 -- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 msgr2=0x560f56804110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56804110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3daffd640 1 -- 172.21.15.116:0/3293143533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f56804dc0 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.562+0000 7fd3db7fe640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f56804650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3daffd640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc016040 tx=0x7fd3cc003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2494a5a423bb763a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc040070 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd3cc03c5c0 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3cc03c410 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd3cc04c050 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 10 client.20073 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560f56889770 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3cc03c410 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd3cc026b00 con 0x560f567eff80 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 10 client.20073 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 10 client.20073 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:43:29.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 20 client.20073 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f56889e30 0x560f5688c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 4 client.20073 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.563+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f567609f0 con 0x560f56889e30 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f56889e30 0x560f5688c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3beff5640 10 client.20073 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3da7fc640 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f56889e30 0x560f5688c220 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b2dfa3c6257a885 server_cookie=ac7c055652978c9f in_seq=0 out_seq=0 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3beff5640 1 -- 172.21.15.116:0/3293143533 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f567609f0 con 0x560f56889e30 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3beff5640 10 client.20073 handle_command_reply: tid=0 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 client.20073 shutdown 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 msgr2=0x560f5681b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc016040 tx=0x7fd3cc003fd0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 shutdown_connections 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x560f56889e30 0x560f5688c220 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560f5680a7e0 0x560f56804650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f567f08a0 0x560f56804110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 --2- 172.21.15.116:0/3293143533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f567eff80 0x560f5681b480 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 >> 172.21.15.116:0/3293143533 conn(0x560f567eaa00 msgr2=0x560f567e9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3beff5640 0 client.20073 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 shutdown_connections 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.564+0000 7fd3e3f90740 1 -- 172.21.15.116:0/3293143533 wait complete. 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e3f90740 20 client.20073 trim_cache size 0 max 16384 2024-03-19T02:43:29.572 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 msgr2=0x7fd3c40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 0x7fd3c40a4eb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009c20 tx=0x7fd3cc0097c0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 msgr2=0x7fd3dc19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000e9f0 tx=0x7fd3d000edf0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 shutdown_connections 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3c40a29f0 0x7fd3c40a4eb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3dc1077a0 0x7fd3dc19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3dc102150 0x7fd3dc19a6c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 --2- 172.21.15.116:0/148862655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3dc101790 0x7fd3dc19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 >> 172.21.15.116:0/148862655 conn(0x7fd3dc0fd2c0 msgr2=0x7fd3dc10cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 shutdown_connections 2024-03-19T02:43:29.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.565+0000 7fd3e12c6640 1 -- 172.21.15.116:0/148862655 wait complete. 2024-03-19T02:43:29.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:43:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 -- 172.21.15.116:0/1510993447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102a40 msgr2=0x7f1218102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 --2- 172.21.15.116:0/1510993447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102a40 0x7f1218102e20 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f1208009900 tx=0x7f120802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 -- 172.21.15.116:0/1510993447 shutdown_connections 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 --2- 172.21.15.116:0/1510993447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f121810bd80 0x7f121810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 --2- 172.21.15.116:0/1510993447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1218103360 0x7f121810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 --2- 172.21.15.116:0/1510993447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102a40 0x7f1218102e20 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 -- 172.21.15.116:0/1510993447 >> 172.21.15.116:0/1510993447 conn(0x7f12180778d0 msgr2=0x7f1218077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 -- 172.21.15.116:0/1510993447 shutdown_connections 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.719+0000 7f121ddb2640 1 -- 172.21.15.116:0/1510993447 wait complete. 2024-03-19T02:43:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 Processor -- start 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 -- start start 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 0x7f1218100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f121810bd80 0x7f1218101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1218111a30 con 0x7f1218103360 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181118b0 con 0x7f1218102a40 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f121ddb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1218111bb0 con 0x7f121810bd80 2024-03-19T02:43:29.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f12177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 0x7f1218100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47140/0 (socket says 172.21.15.116:47140) 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f1216ffd640 1 -- 172.21.15.116:0/2192449872 learned_addr learned my addr 172.21.15.116:0/2192449872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.720+0000 7f1217fff640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f121810bd80 0x7f1218101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 -- 172.21.15.116:0/2192449872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f121810bd80 msgr2=0x7f1218101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f121810bd80 0x7f1218101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 -- 172.21.15.116:0/2192449872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 msgr2=0x7f1218100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 0x7f1218100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 -- 172.21.15.116:0/2192449872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12181aa930 con 0x7f1218103360 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f12177fe640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 0x7f1218100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1216ffd640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f120c00e9f0 tx=0x7f120c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1214ff9640 1 -- 172.21.15.116:0/2192449872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c00cd80 con 0x7f1218103360 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12181aab60 con 0x7f1218103360 2024-03-19T02:43:29.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1214ff9640 1 -- 172.21.15.116:0/2192449872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120c0054d0 con 0x7f1218103360 2024-03-19T02:43:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f1214ff9640 1 -- 172.21.15.116:0/2192449872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c010490 con 0x7f1218103360 2024-03-19T02:43:29.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.721+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12181ab0b0 con 0x7f1218103360 2024-03-19T02:43:29.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f1214ff9640 1 -- 172.21.15.116:0/2192449872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f120c005020 con 0x7f1218103360 2024-03-19T02:43:29.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 Processor -- start 2024-03-19T02:43:29.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 -- start start 2024-03-19T02:43:29.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd336096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336127d0 0x55fd33614bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd3356c4b0 con 0x55fd336092d0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd33514d70 con 0x55fd336127d0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f122099f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd33568550 con 0x55fd33609bf0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f1214ff9640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.724+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336127d0 0x55fd33614bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f1217fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd336096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36992/0 (socket says 172.21.15.116:36992) 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 -- 172.21.15.116:0/3212148243 learned_addr learned my addr 172.21.15.116:0/3212148243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f1217fff640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f1214ff9640 1 -- 172.21.15.116:0/2192449872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f120c0cf4a0 con 0x7f1218103360 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 -- 172.21.15.116:0/3212148243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336127d0 msgr2=0x55fd33614bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336127d0 0x55fd33614bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 -- 172.21.15.116:0/3212148243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 msgr2=0x55fd336096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd336096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 -- 172.21.15.116:0/3212148243 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd33537210 con 0x55fd33609bf0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f1217fff640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 0x7f12000a4eb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1210009990 tx=0x7f12100096a0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f12177fe640 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1208009900 tx=0x7f120802c690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf50acb415b51b1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f11fdffb640 1 -- 172.21.15.116:0/3212148243 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208043070 con 0x55fd33609bf0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f11fdffb640 1 -- 172.21.15.116:0/3212148243 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120802c880 con 0x55fd33609bf0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.725+0000 7f11fdffb640 1 -- 172.21.15.116:0/3212148243 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208003a10 con 0x55fd33609bf0 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 -- 172.21.15.116:0/3212148243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 msgr2=0x55fd33612290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1208009900 tx=0x7f120802c690 comp rx=0 tx=0).stop 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 -- 172.21.15.116:0/3212148243 shutdown_connections 2024-03-19T02:43:29.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336127d0 0x55fd33614bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd33609bf0 0x55fd33612290 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 --2- 172.21.15.116:0/3212148243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd336096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 -- 172.21.15.116:0/3212148243 >> 172.21.15.116:0/3212148243 conn(0x55fd33603a00 msgr2=0x55fd33603de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 -- 172.21.15.116:0/3212148243 shutdown_connections 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.726+0000 7f122099f740 1 -- 172.21.15.116:0/3212148243 wait complete. 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 -- start start 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 Processor -- start 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 0x55fd3360f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd33615560 con 0x55fd336127d0 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd33528310 con 0x55fd33609bf0 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd336156e0 con 0x55fd336092d0 2024-03-19T02:43:29.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f122099f740 10 client.0 _ll_register_callbacks cb 0x55fd3351c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f1217fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 0x55fd3360f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f1217fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47158/0 (socket says 172.21.15.116:47158) 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f1217fff640 1 -- 172.21.15.116:0/2293090762 learned_addr learned my addr 172.21.15.116:0/2293090762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.727+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 -- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 msgr2=0x55fd3360f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 0x55fd3360f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 -- 172.21.15.116:0/2293090762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 msgr2=0x55fd3360fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 -- 172.21.15.116:0/2293090762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd332803f0 con 0x55fd336127d0 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f12177fe640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 0x55fd3360f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f1217fff640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f1210003cf0 tx=0x7f1210004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5449d5722648fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f11f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210003d20 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f1210004040 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f120802c880 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1210004240 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f122099f740 10 client.20085 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd3334a7d0 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.728+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120802c880 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f12100107c0 con 0x55fd336127d0 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 10 client.20085 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 10 client.20085 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 20 client.20085 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd33622c20 0x55fd33625070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 4 client.20085 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:29.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd335ca3e0 con 0x55fd33622c20 2024-03-19T02:43:29.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f12177fe640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd33622c20 0x55fd33625070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f12177fe640 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd33622c20 0x55fd33625070 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e52f1b495a98c05 server_cookie=495002b1a282ed9b in_seq=0 out_seq=0 2024-03-19T02:43:29.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.729+0000 7f11f6ffd640 10 client.20085 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f11f6ffd640 1 -- 172.21.15.116:0/2293090762 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd335ca3e0 con 0x55fd33622c20 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f11f6ffd640 10 client.20085 handle_command_reply: tid=0 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 client.20085 shutdown 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 msgr2=0x55fd336102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f1210003cf0 tx=0x7f1210004a40 comp rx=0 tx=0).stop 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 shutdown_connections 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd33622c20 0x55fd33625070 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336127d0 0x55fd336102b0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 --2- 172.21.15.116:0/2293090762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336092d0 0x55fd3360f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 >> 172.21.15.116:0/2293090762 conn(0x55fd33603a00 msgr2=0x55fd33605750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.730+0000 7f11f6ffd640 0 client.20085 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:43:29.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.731+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 shutdown_connections 2024-03-19T02:43:29.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.731+0000 7f122099f740 1 -- 172.21.15.116:0/2293090762 wait complete. 2024-03-19T02:43:29.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.731+0000 7f122099f740 20 client.20085 trim_cache size 0 max 16384 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 Processor -- start 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 -- start start 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 0x55fd33611f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3361e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b5140 con 0x55fd33609bf0 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b6c80 con 0x55fd336092d0 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f122099f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b5810 con 0x55fd336127d0 2024-03-19T02:43:29.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47162/0 (socket says 172.21.15.116:47162) 2024-03-19T02:43:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f1216ffd640 1 -- 172.21.15.116:0/1734746561 learned_addr learned my addr 172.21.15.116:0/1734746561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f12177fe640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 0x55fd33611f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.749+0000 7f1217fff640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3361e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 -- 172.21.15.116:0/1734746561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 msgr2=0x55fd3361e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3361e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 -- 172.21.15.116:0/1734746561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 msgr2=0x55fd33611f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 0x55fd33611f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 -- 172.21.15.116:0/1734746561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd336045d0 con 0x55fd33609bf0 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f12177fe640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 0x55fd33611f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1217fff640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3361e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f120c097c20 tx=0x7f120c06e560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cc1fc36345a1ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f11fd7fa640 1 -- 172.21.15.116:0/1734746561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c01b160 con 0x55fd33609bf0 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f11fd7fa640 1 -- 172.21.15.116:0/1734746561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120c005310 con 0x55fd33609bf0 2024-03-19T02:43:29.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.750+0000 7f11fd7fa640 1 -- 172.21.15.116:0/1734746561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c09f910 con 0x55fd33609bf0 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- 172.21.15.116:0/1734746561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 msgr2=0x55fd335fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f120c097c20 tx=0x7f120c06e560 comp rx=0 tx=0).stop 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- 172.21.15.116:0/1734746561 shutdown_connections 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3361e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd33609bf0 0x55fd335fea30 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 --2- 172.21.15.116:0/1734746561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd336092d0 0x55fd33611f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- 172.21.15.116:0/1734746561 >> 172.21.15.116:0/1734746561 conn(0x55fd33603a00 msgr2=0x55fd3360ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- 172.21.15.116:0/1734746561 shutdown_connections 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- 172.21.15.116:0/1734746561 wait complete. 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.752+0000 7f122099f740 1 -- start start 2024-03-19T02:43:29.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 Processor -- start 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3360c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b65b0 con 0x55fd336092d0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b5ee0 con 0x55fd33609bf0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b3140 con 0x55fd336127d0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f122099f740 10 client.0 _ll_register_callbacks cb 0x7f1210003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47164/0 (socket says 172.21.15.116:47164) 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 -- 172.21.15.116:0/1850902440 learned_addr learned my addr 172.21.15.116:0/1850902440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f1217fff640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3360c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 -- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 msgr2=0x55fd3360c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3360c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 -- 172.21.15.116:0/1850902440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 msgr2=0x55fd3360bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.753+0000 7f12177fe640 1 -- 172.21.15.116:0/1850902440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3361b020 con 0x55fd336092d0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f12177fe640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f120802e010 tx=0x7f12080035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc01e09fee803000 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f11f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208043040 con 0x55fd336092d0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f120804fc30 con 0x55fd336092d0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fd335799f0 con 0x55fd336092d0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.754+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f120802c880 con 0x55fd336092d0 2024-03-19T02:43:29.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.755+0000 7f122099f740 10 client.20091 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.755+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd33688a70 con 0x55fd336092d0 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.756+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fd335799f0 con 0x55fd336092d0 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1208003940 con 0x55fd336092d0 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 10 client.20091 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 10 client.20091 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:43:29.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 20 client.20091 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd336890f0 0x55fd3368b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 4 client.20091 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd33573d40 con 0x55fd336890f0 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.757+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd336890f0 0x55fd3368b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.759+0000 7f1216ffd640 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd336890f0 0x55fd3368b510 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbfcbb426ae49add server_cookie=dc58998b88323755 in_seq=0 out_seq=0 2024-03-19T02:43:29.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.759+0000 7f11f6ffd640 10 client.20091 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.760+0000 7f11f6ffd640 1 -- 172.21.15.116:0/1850902440 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd33573d40 con 0x55fd336890f0 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.760+0000 7f11f6ffd640 10 client.20091 handle_command_reply: tid=0 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.760+0000 7f122099f740 1 client.20091 shutdown 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 msgr2=0x55fd3361a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f120802e010 tx=0x7f12080035f0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 shutdown_connections 2024-03-19T02:43:29.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55fd336890f0 0x55fd3368b510 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd336127d0 0x55fd3360c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f11f6ffd640 0 client.20091 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd33609bf0 0x55fd3360bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 --2- 172.21.15.116:0/1850902440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd336092d0 0x55fd3361a930 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 >> 172.21.15.116:0/1850902440 conn(0x55fd33603a00 msgr2=0x55fd33602310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 shutdown_connections 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 1 -- 172.21.15.116:0/1850902440 wait complete. 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f122099f740 20 client.20091 trim_cache size 0 max 16384 2024-03-19T02:43:29.763 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:43:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 msgr2=0x7f12000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 0x7f12000a4eb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1210009990 tx=0x7f12100096a0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 msgr2=0x7f1218100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.761+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f120c00e9f0 tx=0x7f120c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 shutdown_connections 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f121810bd80 0x7f1218101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218103360 0x7f1218100bd0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 --2- 172.21.15.116:0/2192449872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218102a40 0x7f1218100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 >> 172.21.15.116:0/2192449872 conn(0x7f12180778d0 msgr2=0x7f12180fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 shutdown_connections 2024-03-19T02:43:29.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.762+0000 7f121ddb2640 1 -- 172.21.15.116:0/2192449872 wait complete. 2024-03-19T02:43:29.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 -- 172.21.15.116:0/1015623740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281087a0 msgr2=0x7fae28114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 --2- 172.21.15.116:0/1015623740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281087a0 0x7fae28114c60 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fae24009900 tx=0x7fae2402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 -- 172.21.15.116:0/1015623740 shutdown_connections 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 --2- 172.21.15.116:0/1015623740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae281087a0 0x7fae28114c60 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 --2- 172.21.15.116:0/1015623740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107e00 0x7fae28108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 --2- 172.21.15.116:0/1015623740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae2810dec0 0x7fae2810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 -- 172.21.15.116:0/1015623740 >> 172.21.15.116:0/1015623740 conn(0x7fae280775f0 msgr2=0x7fae28106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 -- 172.21.15.116:0/1015623740 shutdown_connections 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.909+0000 7fae306c6640 1 -- 172.21.15.116:0/1015623740 wait complete. 2024-03-19T02:43:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 Processor -- start 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 -- start start 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae28107e00 0x7fae2819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae2810dec0 0x7fae281a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae28118050 con 0x7fae28107e00 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae28117ed0 con 0x7fae281087a0 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae306c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae281181d0 con 0x7fae2810dec0 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57272/0 (socket says 172.21.15.116:57272) 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae2dc61640 1 -- 172.21.15.116:0/1006644706 learned_addr learned my addr 172.21.15.116:0/1006644706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae2ec63640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae2810dec0 0x7fae281a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.910+0000 7fae2e462640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae28107e00 0x7fae2819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 -- 172.21.15.116:0/1006644706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae2810dec0 msgr2=0x7fae281a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae2810dec0 0x7fae281a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 -- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae28107e00 msgr2=0x7fae2819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae28107e00 0x7fae2819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 -- 172.21.15.116:0/1006644706 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae281a34f0 con 0x7fae281087a0 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae2dc61640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fae1800e9f0 tx=0x7fae1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae1f7fe640 1 -- 172.21.15.116:0/1006644706 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800cd80 con 0x7fae281087a0 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae281a37e0 con 0x7fae281087a0 2024-03-19T02:43:29.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.911+0000 7fae1f7fe640 1 -- 172.21.15.116:0/1006644706 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae180054d0 con 0x7fae281087a0 2024-03-19T02:43:29.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.912+0000 7fae1f7fe640 1 -- 172.21.15.116:0/1006644706 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18010490 con 0x7fae281087a0 2024-03-19T02:43:29.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.912+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae281ab070 con 0x7fae281087a0 2024-03-19T02:43:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.914+0000 7fae1f7fe640 1 -- 172.21.15.116:0/1006644706 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fae18005020 con 0x7fae281087a0 2024-03-19T02:43:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 Processor -- start 2024-03-19T02:43:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 -- start start 2024-03-19T02:43:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407d2d0 0x559b5407d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54088bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b53fe04b0 con 0x559b5407d2d0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559b53f88d70 con 0x559b540867d0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.915+0000 7fae3338e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559b53fdc550 con 0x559b5407dbf0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2ec63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407d2d0 0x559b5407d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37052/0 (socket says 172.21.15.116:37052) 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 -- 172.21.15.116:0/908261353 learned_addr learned my addr 172.21.15.116:0/908261353 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2dc61640 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54088bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 -- 172.21.15.116:0/908261353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 msgr2=0x559b54088bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54088bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 -- 172.21.15.116:0/908261353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407d2d0 msgr2=0x559b5407d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407d2d0 0x559b5407d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 -- 172.21.15.116:0/908261353 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b53fab210 con 0x559b5407dbf0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2dc61640 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54088bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae1f7fe640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 0x7fae140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2ec63640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 0x7fae140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.916+0000 7fae2e462640 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae200114f0 tx=0x7fae200119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93fdf45fbb782719 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae1f7fe640 1 -- 172.21.15.116:0/1006644706 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fae180cf4f0 con 0x7fae281087a0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae1cff9640 1 -- 172.21.15.116:0/908261353 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2001e070 con 0x559b5407dbf0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae1cff9640 1 -- 172.21.15.116:0/908261353 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae2000fad0 con 0x559b5407dbf0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae1cff9640 1 -- 172.21.15.116:0/908261353 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20019ab0 con 0x559b5407dbf0 2024-03-19T02:43:29.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae3338e740 1 -- 172.21.15.116:0/908261353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 msgr2=0x559b54086290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae2ec63640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 0x7fae140a4e30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fae24005ab0 tx=0x7fae240058d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.917+0000 7fae3338e740 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae200114f0 tx=0x7fae200119b0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- 172.21.15.116:0/908261353 shutdown_connections 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54088bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407dbf0 0x559b54086290 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- 172.21.15.116:0/908261353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407d2d0 0x559b5407d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- 172.21.15.116:0/908261353 >> 172.21.15.116:0/908261353 conn(0x559b54077a00 msgr2=0x559b54077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- 172.21.15.116:0/908261353 shutdown_connections 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- 172.21.15.116:0/908261353 wait complete. 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- start start 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 Processor -- start 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b540837b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54083cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b54084230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b540895c0 con 0x559b540867d0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559b53f9c310 con 0x559b5407dbf0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.918+0000 7fae3338e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559b54089740 con 0x559b5407d2d0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae3338e740 10 client.0 _ll_register_callbacks cb 0x559b53f902d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54083cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54083cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57290/0 (socket says 172.21.15.116:57290) 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b540837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2ec63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b54084230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2dc61640 1 -- 172.21.15.116:0/248485947 learned_addr learned my addr 172.21.15.116:0/248485947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 -- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 msgr2=0x559b54083cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54083cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 -- 172.21.15.116:0/248485947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 msgr2=0x559b54084230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b54084230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 -- 172.21.15.116:0/248485947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b53cf43f0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2ec63640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b54084230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae2e462640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b540837b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fae200099d0 tx=0x7fae20009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b208ea4c067b61c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20026020 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fae2000f6a0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.919+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae2000fad0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fae2001e040 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 10 client.29815 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b53dbe7d0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae2000fad0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fae200197f0 con 0x559b5407d2d0 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 10 client.29815 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 10 client.29815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 20 client.29815 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b54096a40 0x559b54098e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 4 client.29815 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:29.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b5403e3e0 con 0x559b54096a40 2024-03-19T02:43:29.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.920+0000 7fae2dc61640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b54096a40 0x559b54098e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.921+0000 7fae11ffb640 10 client.29815 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:29.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.921+0000 7fae2dc61640 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b54096a40 0x559b54098e30 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0e824044d8b2765 server_cookie=87a325c0317c802c in_seq=0 out_seq=0 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.922+0000 7fae11ffb640 1 -- 172.21.15.116:0/248485947 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559b5403e3e0 con 0x559b54096a40 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.922+0000 7fae11ffb640 10 client.29815 handle_command_reply: tid=0 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.922+0000 7fae3338e740 1 client.29815 shutdown 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 msgr2=0x559b540837b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b540837b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fae200099d0 tx=0x7fae20009ca0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 shutdown_connections 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b54096a40 0x559b54098e30 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b54084230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54083cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 --2- 172.21.15.116:0/248485947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b540837b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 >> 172.21.15.116:0/248485947 conn(0x559b54077a00 msgr2=0x559b54079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae11ffb640 0 client.29815 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:29.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 shutdown_connections 2024-03-19T02:43:29.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 1 -- 172.21.15.116:0/248485947 wait complete. 2024-03-19T02:43:29.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.923+0000 7fae3338e740 20 client.29815 trim_cache size 0 max 16384 2024-03-19T02:43:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 Processor -- start 2024-03-19T02:43:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 -- start start 2024-03-19T02:43:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 0x559b54092390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54084050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b51f0 con 0x559b5407dbf0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b6d30 con 0x559b540867d0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.940+0000 7fae3338e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b58c0 con 0x559b5407d2d0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 0x559b54092390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37084/0 (socket says 172.21.15.116:37084) 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2dc61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 0x559b54092390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47190/0 (socket says 172.21.15.116:47190) 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 -- 172.21.15.116:0/2622085626 learned_addr learned my addr 172.21.15.116:0/2622085626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2ec63640 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54084050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 -- 172.21.15.116:0/2622085626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 msgr2=0x559b54084050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54084050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 -- 172.21.15.116:0/2622085626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 msgr2=0x559b54092390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 0x559b54092390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 -- 172.21.15.116:0/2622085626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b54077830 con 0x559b5407d2d0 2024-03-19T02:43:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae2e462640 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fae20000f80 tx=0x7fae20009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135aaa797d71c920 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae13fff640 1 -- 172.21.15.116:0/2622085626 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20026070 con 0x559b5407d2d0 2024-03-19T02:43:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae13fff640 1 -- 172.21.15.116:0/2622085626 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae20019900 con 0x559b5407d2d0 2024-03-19T02:43:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.941+0000 7fae13fff640 1 -- 172.21.15.116:0/2622085626 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000f7c0 con 0x559b5407d2d0 2024-03-19T02:43:29.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 -- 172.21.15.116:0/2622085626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 msgr2=0x559b54085ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fae20000f80 tx=0x7fae20009500 comp rx=0 tx=0).stop 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 -- 172.21.15.116:0/2622085626 shutdown_connections 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b540867d0 0x559b54084050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b5407dbf0 0x559b54092390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 --2- 172.21.15.116:0/2622085626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54085ee0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 -- 172.21.15.116:0/2622085626 >> 172.21.15.116:0/2622085626 conn(0x559b54077a00 msgr2=0x559b54077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 -- 172.21.15.116:0/2622085626 shutdown_connections 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.942+0000 7fae3338e740 1 -- 172.21.15.116:0/2622085626 wait complete. 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 -- start start 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 Processor -- start 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54089e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b5408a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b6660 con 0x559b540867d0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b5f90 con 0x559b5407dbf0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae281b31f0 con 0x559b5407d2d0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae3338e740 10 client.0 _ll_register_callbacks cb 0x7fae2001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae2e462640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37100/0 (socket says 172.21.15.116:37100) 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae2e462640 1 -- 172.21.15.116:0/4128785480 learned_addr learned my addr 172.21.15.116:0/4128785480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae2dc61640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54089e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.943+0000 7fae2ec63640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b5408a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 -- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 msgr2=0x559b54089e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54089e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 -- 172.21.15.116:0/4128785480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 msgr2=0x559b5408a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b5408a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 -- 172.21.15.116:0/4128785480 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b540fac10 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae2e462640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fae20009700 tx=0x7fae200045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c38fb4a71117a4a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20026050 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fae2000f970 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae20019900 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.944+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fae20004850 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 10 client.29821 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b540fbdf0 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae20019900 con 0x559b5407d2d0 2024-03-19T02:43:29.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fae20031030 con 0x559b5407d2d0 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 10 client.29821 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 10 client.29821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 20 client.29821 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b540fc180 0x559b540fe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 4 client.29821 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b53fed9f0 con 0x559b540fc180 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.945+0000 7fae2dc61640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b540fc180 0x559b540fe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.946+0000 7fae11ffb640 10 client.29821 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:29.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.946+0000 7fae2dc61640 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b540fc180 0x559b540fe5a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc449265a2bfa3db server_cookie=be7db739651d7c0 in_seq=0 out_seq=0 2024-03-19T02:43:29.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae11ffb640 1 -- 172.21.15.116:0/4128785480 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b53fed9f0 con 0x559b540fc180 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae11ffb640 10 client.29821 handle_command_reply: tid=0 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 client.29821 shutdown 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 msgr2=0x559b54089950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fae20009700 tx=0x7fae200045f0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 shutdown_connections 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559b540fc180 0x559b540fe5a0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b540867d0 0x559b5408a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b5407dbf0 0x559b54089e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 --2- 172.21.15.116:0/4128785480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559b5407d2d0 0x559b54089950 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 >> 172.21.15.116:0/4128785480 conn(0x559b54077a00 msgr2=0x559b54076310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae11ffb640 0 client.29821 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.948+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 shutdown_connections 2024-03-19T02:43:29.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae3338e740 1 -- 172.21.15.116:0/4128785480 wait complete. 2024-03-19T02:43:29.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae3338e740 20 client.29821 trim_cache size 0 max 16384 2024-03-19T02:43:29.954 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 msgr2=0x7fae140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 0x7fae140a4e30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fae24005ab0 tx=0x7fae240058d0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 msgr2=0x7fae2819ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fae1800e9f0 tx=0x7fae1800edf0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 shutdown_connections 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fae140a2970 0x7fae140a4e30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae2810dec0 0x7fae281a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae281087a0 0x7fae2819ea90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 --2- 172.21.15.116:0/1006644706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae28107e00 0x7fae2819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 >> 172.21.15.116:0/1006644706 conn(0x7fae280775f0 msgr2=0x7fae28078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 shutdown_connections 2024-03-19T02:43:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:29.949+0000 7fae306c6640 1 -- 172.21.15.116:0/1006644706 wait complete. 2024-03-19T02:43:30.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 19996 5.98 MB/sec execute 52 sec latency 129.088 ms 2024-03-19T02:43:30.964 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:43:30.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:43:30.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:43:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 -- 172.21.15.116:0/120115063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd8101040 msgr2=0x7fafd8101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 --2- 172.21.15.116:0/120115063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd8101040 0x7fafd8101420 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009930 tx=0x7fafd402bc80 comp rx=0 tx=0).stop 2024-03-19T02:43:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 -- 172.21.15.116:0/120115063 shutdown_connections 2024-03-19T02:43:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 --2- 172.21.15.116:0/120115063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd81059f0 0x7fafd810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 --2- 172.21.15.116:0/120115063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81019f0 0x7fafd81052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 --2- 172.21.15.116:0/120115063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd8101040 0x7fafd8101420 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 -- 172.21.15.116:0/120115063 >> 172.21.15.116:0/120115063 conn(0x7fafd8076c40 msgr2=0x7fafd80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 -- 172.21.15.116:0/120115063 shutdown_connections 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.117+0000 7fafe0036640 1 -- 172.21.15.116:0/120115063 wait complete. 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 Processor -- start 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 -- start start 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 0x7fafd8196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81059f0 0x7fafd8197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8110960 con 0x7fafd819b7a0 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81107e0 con 0x7fafd8101040 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafe0036640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8110ae0 con 0x7fafd81059f0 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47230/0 (socket says 172.21.15.116:47230) 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 -- 172.21.15.116:0/215961916 learned_addr learned my addr 172.21.15.116:0/215961916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafdddd2640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 0x7fafd8196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81059f0 0x7fafd8197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 -- 172.21.15.116:0/215961916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81059f0 msgr2=0x7fafd8197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81059f0 0x7fafd8197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 -- 172.21.15.116:0/215961916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 msgr2=0x7fafd8196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 0x7fafd8196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafde5d3640 1 -- 172.21.15.116:0/215961916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd819c290 con 0x7fafd819b7a0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.118+0000 7fafdddd2640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 0x7fafd8196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafde5d3640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fafd000efd0 tx=0x7fafd000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafceffd640 1 -- 172.21.15.116:0/215961916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0003b70 con 0x7fafd819b7a0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafceffd640 1 -- 172.21.15.116:0/215961916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafd0003d10 con 0x7fafd819b7a0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd819c580 con 0x7fafd819b7a0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafceffd640 1 -- 172.21.15.116:0/215961916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0010980 con 0x7fafd819b7a0 2024-03-19T02:43:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.119+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a3d20 con 0x7fafd819b7a0 2024-03-19T02:43:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.121+0000 7fafceffd640 1 -- 172.21.15.116:0/215961916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fafd0004020 con 0x7fafd819b7a0 2024-03-19T02:43:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.122+0000 7fafceffd640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 0x7fafc40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 Processor -- start 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafceffd640 1 -- 172.21.15.116:0/215961916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fafd00cf6b0 con 0x7fafd819b7a0 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdddd2640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 0x7fafc40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 -- start start 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 0x5600a7e71e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 0x5600a7e798f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5600a7dc9160 con 0x5600a7e64ed0 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600a7d71e10 con 0x5600a7e71a20 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafe2c27740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600a7dc5200 con 0x5600a7e64530 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdddd2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 0x5600a7e71e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37136/0 (socket says 172.21.15.116:37136) 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdddd2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 0x5600a7e71e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57320/0 (socket says 172.21.15.116:57320) 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.123+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2060030465 learned_addr learned my addr 172.21.15.116:0/2060030465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdddd2640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 0x7fafc40a4e30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005ab0 tx=0x7fafd4005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2060030465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 msgr2=0x5600a7e71e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 0x5600a7e71e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2060030465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 msgr2=0x5600a7e798f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafde5d3640 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 0x5600a7e798f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 0x5600a7e798f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2060030465 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a7ba57e0 con 0x5600a7e64530 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafde5d3640 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 0x5600a7e798f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:43:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009cc0 tx=0x7fafc8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad8a881c0db46b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.124+0000 7fafc3fff640 1 -- 172.21.15.116:0/2060030465 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004250 con 0x5600a7e64530 2024-03-19T02:43:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.125+0000 7fafc3fff640 1 -- 172.21.15.116:0/2060030465 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc80043f0 con 0x5600a7e64530 2024-03-19T02:43:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.125+0000 7fafc3fff640 1 -- 172.21.15.116:0/2060030465 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004d80 con 0x5600a7e64530 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.125+0000 7fafe2c27740 1 -- 172.21.15.116:0/2060030465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 msgr2=0x5600a7e64990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.125+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009cc0 tx=0x7fafc8011810 comp rx=0 tx=0).stop 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 -- 172.21.15.116:0/2060030465 shutdown_connections 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64ed0 0x5600a7e798f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e64990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2060030465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e71a20 0x5600a7e71e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 -- 172.21.15.116:0/2060030465 >> 172.21.15.116:0/2060030465 conn(0x5600a7e5f520 msgr2=0x5600a7e5f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 -- 172.21.15.116:0/2060030465 shutdown_connections 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 -- 172.21.15.116:0/2060030465 wait complete. 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 -- start start 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.126+0000 7fafe2c27740 1 Processor -- start 2024-03-19T02:43:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e76e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e71a20 0x5600a7e778d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5600a7e8af00 con 0x5600a7e71a20 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600a7d851a0 con 0x5600a7e64ed0 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600a7e8b080 con 0x5600a7e64530 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafe2c27740 10 client.0 _ll_register_callbacks cb 0x5600a7adcc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdddd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e76e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57324/0 (socket says 172.21.15.116:57324) 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2085406668 learned_addr learned my addr 172.21.15.116:0/2085406668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafde5d3640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e71a20 0x5600a7e778d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2085406668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 msgr2=0x5600a7e76e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e76e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2085406668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e71a20 msgr2=0x5600a7e778d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e71a20 0x5600a7e778d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdd5d1640 1 -- 172.21.15.116:0/2085406668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a7c05f70 con 0x5600a7e64ed0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.127+0000 7fafdddd2640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e76e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fe0 tx=0x7fafc80098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62e531e1dec87d5e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8026020 con 0x5600a7e64ed0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fafc8003900 con 0x5600a7e64ed0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc80043f0 con 0x5600a7e64ed0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fafc8031050 con 0x5600a7e64ed0 2024-03-19T02:43:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafe2c27740 10 client.29864 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5600a7a9d170 con 0x5600a7e64ed0 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.128+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc8022400 con 0x5600a7e64ed0 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fafc8018070 con 0x5600a7e64ed0 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 10 client.29864 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 10 client.29864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 20 client.29864 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e98100 0x5600a7e9a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 4 client.29864 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fafc80043f0 con 0x5600a7e98100 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafdddd2640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e98100 0x5600a7e9a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafc17fa640 10 client.29864 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.129+0000 7fafdddd2640 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e98100 0x5600a7e9a4f0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2fb97745b3cd7e6 server_cookie=c7e5ef73f202f111 in_seq=0 out_seq=0 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafc17fa640 1 -- 172.21.15.116:0/2085406668 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fafc80043f0 con 0x5600a7e98100 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafc17fa640 10 client.29864 handle_command_reply: tid=0 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 client.29864 shutdown 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 msgr2=0x5600a7e77390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fe0 tx=0x7fafc80098e0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 shutdown_connections 2024-03-19T02:43:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e98100 0x5600a7e9a4f0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e71a20 0x5600a7e778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e77390 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 --2- 172.21.15.116:0/2085406668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64530 0x5600a7e76e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafc17fa640 0 client.29864 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 >> 172.21.15.116:0/2085406668 conn(0x5600a7e5f520 msgr2=0x5600a7e730e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 shutdown_connections 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 1 -- 172.21.15.116:0/2085406668 wait complete. 2024-03-19T02:43:31.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.132+0000 7fafe2c27740 20 client.29864 trim_cache size 0 max 16384 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 Processor -- start 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 -- start start 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e5f520 0x5600a7e79580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64ed0 0x5600a7e93ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81a41c0 con 0x5600a7e64530 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81a5d00 con 0x5600a7e5f520 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafe2c27740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81a4890 con 0x5600a7e64ed0 2024-03-19T02:43:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47252/0 (socket says 172.21.15.116:47252) 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 -- 172.21.15.116:0/1654850995 learned_addr learned my addr 172.21.15.116:0/1654850995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdddd2640 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e5f520 0x5600a7e79580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafde5d3640 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64ed0 0x5600a7e93ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 -- 172.21.15.116:0/1654850995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64ed0 msgr2=0x5600a7e93ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64ed0 0x5600a7e93ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 -- 172.21.15.116:0/1654850995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e5f520 msgr2=0x5600a7e79580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e5f520 0x5600a7e79580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.156+0000 7fafdd5d1640 1 -- 172.21.15.116:0/1654850995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a7e74df0 con 0x5600a7e64530 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.157+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fafc8000c00 tx=0x7fafc8009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08a90cc5028f0d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.157+0000 7fafc37fe640 1 -- 172.21.15.116:0/1654850995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc80043d0 con 0x5600a7e64530 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.157+0000 7fafc37fe640 1 -- 172.21.15.116:0/1654850995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc8019600 con 0x5600a7e64530 2024-03-19T02:43:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.157+0000 7fafc37fe640 1 -- 172.21.15.116:0/1654850995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8003dc0 con 0x5600a7e64530 2024-03-19T02:43:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- 172.21.15.116:0/1654850995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 msgr2=0x5600a7e748b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fafc8000c00 tx=0x7fafc8009f30 comp rx=0 tx=0).stop 2024-03-19T02:43:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- 172.21.15.116:0/1654850995 shutdown_connections 2024-03-19T02:43:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e64ed0 0x5600a7e93ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e748b0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 --2- 172.21.15.116:0/1654850995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e5f520 0x5600a7e79580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- 172.21.15.116:0/1654850995 >> 172.21.15.116:0/1654850995 conn(0x5600a7e634b0 msgr2=0x5600a7e5e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- 172.21.15.116:0/1654850995 shutdown_connections 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- 172.21.15.116:0/1654850995 wait complete. 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 -- start start 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.158+0000 7fafe2c27740 1 Processor -- start 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e8b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e8bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81a5630 con 0x5600a7e64530 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafdd5d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e8b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafd81a4f60 con 0x5600a7e64ed0 2024-03-19T02:43:31.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8071c30 con 0x5600a7e5f520 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafdddd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafe2c27740 10 client.0 _ll_register_callbacks cb 0x7fafc8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafdddd2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37170/0 (socket says 172.21.15.116:37170) 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafdddd2640 1 -- 172.21.15.116:0/3989694479 learned_addr learned my addr 172.21.15.116:0/3989694479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.159+0000 7fafde5d3640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e8bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 -- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 msgr2=0x5600a7e8bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e8bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 -- 172.21.15.116:0/3989694479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 msgr2=0x5600a7e8b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e8b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 -- 172.21.15.116:0/3989694479 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a7efe460 con 0x5600a7e5f520 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e8b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafdddd2640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fafd402c7f0 tx=0x7fafd402bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e465bfbc168eb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003800 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fafd40039a0 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc8019600 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fafd4064050 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafe2c27740 10 client.29839 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5600a7eff760 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.160+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc8019600 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fafd403fc40 con 0x5600a7e5f520 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 10 client.29839 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 10 client.29839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 20 client.29839 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e71a20 0x5600a7f01ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 4 client.29839 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:43:31.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5600a7dd66a0 con 0x5600a7e71a20 2024-03-19T02:43:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e71a20 0x5600a7f01ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafc17fa640 10 client.29839 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.161+0000 7fafdd5d1640 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e71a20 0x5600a7f01ec0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3992b0dcb516176 server_cookie=955795d47cff71a8 in_seq=0 out_seq=0 2024-03-19T02:43:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.165+0000 7fafc17fa640 1 -- 172.21.15.116:0/3989694479 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5600a7dd66a0 con 0x5600a7e71a20 2024-03-19T02:43:31.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.165+0000 7fafc17fa640 10 client.29839 handle_command_reply: tid=0 2024-03-19T02:43:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.165+0000 7fafe2c27740 1 client.29839 shutdown 2024-03-19T02:43:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.165+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 msgr2=0x5600a7e8b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.165+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fafd402c7f0 tx=0x7fafd402bc80 comp rx=0 tx=0).stop 2024-03-19T02:43:31.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 shutdown_connections 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5600a7e71a20 0x5600a7f01ec0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600a7e64ed0 0x5600a7e8bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600a7e64530 0x5600a7e8b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 --2- 172.21.15.116:0/3989694479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600a7e5f520 0x5600a7e8b2d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafc17fa640 0 client.29839 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.166+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 >> 172.21.15.116:0/3989694479 conn(0x5600a7e634b0 msgr2=0x5600a7e74600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.167+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 shutdown_connections 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.167+0000 7fafe2c27740 1 -- 172.21.15.116:0/3989694479 wait complete. 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.167+0000 7fafe2c27740 20 client.29839 trim_cache size 0 max 16384 2024-03-19T02:43:31.169 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:43:31.170 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:43:31.170 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "c4c80883-0b16-4fdf-85a2-85cfd5a14eff", 2024-03-19T02:43:31.170 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:43:31.170 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 msgr2=0x7fafc40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 0x7fafc40a4e30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005ab0 tx=0x7fafd4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 msgr2=0x7fafd819bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fafd000efd0 tx=0x7fafd000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 shutdown_connections 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafc40a2970 0x7fafc40a4e30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafd819b7a0 0x7fafd819bc50 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafd81059f0 0x7fafd8197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 --2- 172.21.15.116:0/215961916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafd8101040 0x7fafd8196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 >> 172.21.15.116:0/215961916 conn(0x7fafd8076c40 msgr2=0x7fafd8103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 shutdown_connections 2024-03-19T02:43:31.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.168+0000 7fafe0036640 1 -- 172.21.15.116:0/215961916 wait complete. 2024-03-19T02:43:31.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:43:31.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:43:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:31 smithi116 ceph-mon[32263]: pgmap v494: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 264 op/s 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1709045596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac1066a0 msgr2=0x7f3dac106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1709045596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac1066a0 0x7f3dac106a80 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009930 tx=0x7f3d9c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1709045596 shutdown_connections 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1709045596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac101050 0x7f3dac10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1709045596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac100690 0x7f3dac100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1709045596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac1066a0 0x7f3dac106a80 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1709045596 >> 172.21.15.116:0/1709045596 conn(0x7f3dac077820 msgr2=0x7f3dac077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1709045596 shutdown_connections 2024-03-19T02:43:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.313+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1709045596 wait complete. 2024-03-19T02:43:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 Processor -- start 2024-03-19T02:43:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 -- start start 2024-03-19T02:43:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 0x7f3dac198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 0x7f3dac1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac1066a0 0x7f3dac1a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac110960 con 0x7f3dac100690 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1107e0 con 0x7f3dac1066a0 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3db0a8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac110ae0 con 0x7f3dac101050 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 0x7f3dac198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 0x7f3dac1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 0x7f3dac198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47278/0 (socket says 172.21.15.116:47278) 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3daa59c640 1 -- 172.21.15.116:0/1455887960 learned_addr learned my addr 172.21.15.116:0/1455887960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3daad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac1066a0 0x7f3dac1a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 -- 172.21.15.116:0/1455887960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac1066a0 msgr2=0x7f3dac1a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac1066a0 0x7f3dac1a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 -- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 msgr2=0x7f3dac198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 0x7f3dac198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 -- 172.21.15.116:0/1455887960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac1a4420 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.314+0000 7f3da3fff640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 0x7f3dac1993a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400e9f0 tx=0x7f3d9400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.315+0000 7f3da37fe640 1 -- 172.21.15.116:0/1455887960 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9400ccc0 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.315+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac1a45f0 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.315+0000 7f3da37fe640 1 -- 172.21.15.116:0/1455887960 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d94005020 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.315+0000 7f3da37fe640 1 -- 172.21.15.116:0/1455887960 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94004a00 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.315+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac1a4b40 con 0x7f3dac101050 2024-03-19T02:43:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.318+0000 7f3db3670740 1 Processor -- start 2024-03-19T02:43:31.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.318+0000 7f3db3670740 1 -- start start 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.318+0000 7f3db3670740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.318+0000 7f3db3670740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf263a80 0x5568bf24fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.318+0000 7f3da37fe640 1 -- 172.21.15.116:0/1455887960 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3d94004ba0 con 0x7f3dac101050 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3db3670740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf250230 0x5568bf250690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3db3670740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568bf1b9500 con 0x5568bf250230 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3db3670740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5568bf161be0 con 0x5568bf263a80 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3db3670740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5568bf1b55a0 con 0x5568bf2630d0 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf263a80 0x5568bf24fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37190/0 (socket says 172.21.15.116:37190) 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 -- 172.21.15.116:0/2602333866 learned_addr learned my addr 172.21.15.116:0/2602333866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3da3fff640 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf250230 0x5568bf250690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 -- 172.21.15.116:0/2602333866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf263a80 msgr2=0x5568bf24fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf263a80 0x5568bf24fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 -- 172.21.15.116:0/2602333866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf250230 msgr2=0x5568bf250690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf250230 0x5568bf250690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 -- 172.21.15.116:0/2602333866 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568bf183000 con 0x5568bf2630d0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daad9d640 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009f50 tx=0x7f3da40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d1b48c9c0248307 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3da37fe640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 0x7f3d980a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3da0ff9640 1 -- 172.21.15.116:0/2602333866 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4004d20 con 0x5568bf2630d0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3da0ff9640 1 -- 172.21.15.116:0/2602333866 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3da4004050 con 0x5568bf2630d0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3da0ff9640 1 -- 172.21.15.116:0/2602333866 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4005280 con 0x5568bf2630d0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.319+0000 7f3daa59c640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 0x7f3d980a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3da37fe640 1 -- 172.21.15.116:0/1455887960 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f3d940cf260 con 0x7f3dac101050 2024-03-19T02:43:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 -- 172.21.15.116:0/2602333866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 msgr2=0x5568bf2634b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3daa59c640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 0x7f3d980a4eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c000c00 tx=0x7f3d9c0058d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009f50 tx=0x7f3da40118d0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 -- 172.21.15.116:0/2602333866 shutdown_connections 2024-03-19T02:43:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf250230 0x5568bf250690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf263a80 0x5568bf24fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 --2- 172.21.15.116:0/2602333866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2634b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 -- 172.21.15.116:0/2602333866 >> 172.21.15.116:0/2602333866 conn(0x5568bf2526d0 msgr2=0x5568bf252ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 -- 172.21.15.116:0/2602333866 shutdown_connections 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.320+0000 7f3db3670740 1 -- 172.21.15.116:0/2602333866 wait complete. 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 -- start start 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 Processor -- start 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf267070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 0x5568bf267af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568bf26a510 con 0x5568bf263a80 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5568bf175160 con 0x5568bf250230 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5568bf26a690 con 0x5568bf2630d0 2024-03-19T02:43:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3db3670740 10 client.0 _ll_register_callbacks cb 0x5568bf1b1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf267070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf267070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57382/0 (socket says 172.21.15.116:57382) 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3daa59c640 1 -- 172.21.15.116:0/2123301203 learned_addr learned my addr 172.21.15.116:0/2123301203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3da3fff640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37198/0 (socket says 172.21.15.116:37198) 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.321+0000 7f3daad9d640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 0x5568bf267af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 -- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 msgr2=0x5568bf267070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf267070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 -- 172.21.15.116:0/2123301203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 msgr2=0x5568bf267af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 0x5568bf267af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 -- 172.21.15.116:0/2123301203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568bef977d0 con 0x5568bf2630d0 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3daad9d640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 0x5568bf267af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3da3fff640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cc570 tx=0x7f3d94004f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c10b34488a7836 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3d91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9401c070 con 0x5568bf2630d0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f3d94097af0 con 0x5568bf2630d0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da4004050 con 0x5568bf2630d0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3d9409ade0 con 0x5568bf2630d0 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3db3670740 10 client.29851 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.322+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5568beecd3f0 con 0x5568bf2630d0 2024-03-19T02:43:31.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20129 5.93 MB/sec execute 53 sec latency 185.848 ms 2024-03-19T02:43:31.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3da4004050 con 0x5568bf2630d0 2024-03-19T02:43:31.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3d940a3790 con 0x5568bf2630d0 2024-03-19T02:43:31.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 10 client.29851 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:31.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 10 client.29851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:31.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 20 client.29851 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf277b00 0x5568bf279f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 4 client.29851 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568bf1c6a40 con 0x5568bf277b00 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3daa59c640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf277b00 0x5568bf279f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3daa59c640 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf277b00 0x5568bf279f10 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22dd9ba57ab55567 server_cookie=3e2b215c0a8abd4f in_seq=0 out_seq=0 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.323+0000 7f3d91ffb640 10 client.29851 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3d91ffb640 1 -- 172.21.15.116:0/2123301203 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568bf1c6a40 con 0x5568bf277b00 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3d91ffb640 10 client.29851 handle_command_reply: tid=0 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 client.29851 shutdown 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 msgr2=0x5568bf2675b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cc570 tx=0x7f3d94004f50 comp rx=0 tx=0).stop 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 shutdown_connections 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf277b00 0x5568bf279f10 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf263a80 0x5568bf267af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3d91ffb640 0 client.29851 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf2630d0 0x5568bf2675b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 --2- 172.21.15.116:0/2123301203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf267070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.326+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 >> 172.21.15.116:0/2123301203 conn(0x5568bf2526d0 msgr2=0x5568bf251e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.327+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 shutdown_connections 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.327+0000 7f3db3670740 1 -- 172.21.15.116:0/2123301203 wait complete. 2024-03-19T02:43:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.327+0000 7f3db3670740 20 client.29851 trim_cache size 0 max 16384 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 Processor -- start 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 -- start start 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf2697a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf262780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1acaf0 con 0x5568bf2630d0 2024-03-19T02:43:31.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1ae630 con 0x5568bf250230 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.348+0000 7f3db3670740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1ad1c0 con 0x5568bf263a80 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf2697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37210/0 (socket says 172.21.15.116:37210) 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf262780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daad9d640 1 -- 172.21.15.116:0/688653761 learned_addr learned my addr 172.21.15.116:0/688653761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 -- 172.21.15.116:0/688653761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 msgr2=0x5568bf2732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 -- 172.21.15.116:0/688653761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 msgr2=0x5568bf262780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf262780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 -- 172.21.15.116:0/688653761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568bee8d0f0 con 0x5568bf250230 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3da3fff640 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf262780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daad9d640 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3daa59c640 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf2697a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5568bf27a720 tx=0x7f3d9c045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f5117716c79a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3d93fff640 1 -- 172.21.15.116:0/688653761 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c02c920 con 0x5568bf250230 2024-03-19T02:43:31.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3d93fff640 1 -- 172.21.15.116:0/688653761 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d9c05c070 con 0x5568bf250230 2024-03-19T02:43:31.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.349+0000 7f3d93fff640 1 -- 172.21.15.116:0/688653761 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0054c0 con 0x5568bf250230 2024-03-19T02:43:31.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 -- 172.21.15.116:0/688653761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 msgr2=0x5568bf2697a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf2697a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5568bf27a720 tx=0x7f3d9c045000 comp rx=0 tx=0).stop 2024-03-19T02:43:31.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 -- 172.21.15.116:0/688653761 shutdown_connections 2024-03-19T02:43:31.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf262780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 --2- 172.21.15.116:0/688653761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf2697a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 -- 172.21.15.116:0/688653761 >> 172.21.15.116:0/688653761 conn(0x5568bf2526d0 msgr2=0x5568bf24e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 -- 172.21.15.116:0/688653761 shutdown_connections 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.350+0000 7f3db3670740 1 -- 172.21.15.116:0/688653761 wait complete. 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 -- start start 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 Processor -- start 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf27f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2dce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1adf60 con 0x5568bf2630d0 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1ad890 con 0x5568bf250230 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac1aaaf0 con 0x5568bf263a80 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3db3670740 10 client.0 _ll_register_callbacks cb 0x7f3d94097060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:43:31.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3daa59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf27f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47330/0 (socket says 172.21.15.116:47330) 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 -- 172.21.15.116:0/441567923 learned_addr learned my addr 172.21.15.116:0/441567923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3daad9d640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 -- 172.21.15.116:0/441567923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 msgr2=0x5568bf2dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 -- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 msgr2=0x5568bf27f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf27f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3da3fff640 1 -- 172.21.15.116:0/441567923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568bf2dd400 con 0x5568bf2630d0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.351+0000 7f3daa59c640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf27f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3da3fff640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cc5a0 tx=0x7f3d94005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce32a0c7831efc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:43:31.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9401c040 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f3d9409f300 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d9c05c070 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3d9409b250 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 10 client.20142 fetch_fsmap learned FSMap version 38 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5568bf2de6f0 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d9c05c070 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3d940971d0 con 0x5568bf2630d0 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 10 client.20142 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 10 client.20142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 20 client.20142 populate_metadata read hostname 'smithi116' 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf2ded80 0x5568bf2e11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 4 client.20142 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:43:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.352+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568bf1c0d90 con 0x5568bf2ded80 2024-03-19T02:43:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.353+0000 7f3daa59c640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf2ded80 0x5568bf2e11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.353+0000 7f3d91ffb640 10 client.20142 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.353+0000 7f3daa59c640 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf2ded80 0x5568bf2e11a0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1233562007e99992 server_cookie=45ca28e29f5d0728 in_seq=0 out_seq=0 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3d91ffb640 1 -- 172.21.15.116:0/441567923 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5568bf1c0d90 con 0x5568bf2ded80 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3d91ffb640 10 client.20142 handle_command_reply: tid=0 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 client.20142 shutdown 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 msgr2=0x5568bf27f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cc5a0 tx=0x7f3d94005400 comp rx=0 tx=0).stop 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 shutdown_connections 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5568bf2ded80 0x5568bf2e11a0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5568bf263a80 0x5568bf2dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3d91ffb640 0 client.20142 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568bf2630d0 0x5568bf27f7e0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 --2- 172.21.15.116:0/441567923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568bf250230 0x5568bf27f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 >> 172.21.15.116:0/441567923 conn(0x5568bf2526d0 msgr2=0x5568bf24e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 shutdown_connections 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 1 -- 172.21.15.116:0/441567923 wait complete. 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.355+0000 7f3db3670740 20 client.20142 trim_cache size 0 max 16384 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (427 inodes in the stack)", 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:43:31.361 INFO:teuthology.orchestra.run.smithi116.stdout: "c4c80883-0b16-4fdf-85a2-85cfd5a14eff": { 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "c4c80883-0b16-4fdf-85a2-85cfd5a14eff", 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:43:31.362 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 msgr2=0x7f3d980a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 0x7f3d980a4eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c000c00 tx=0x7f3d9c0058d0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 msgr2=0x7f3dac1993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 0x7f3dac1993a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400e9f0 tx=0x7f3d9400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 shutdown_connections 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d980a29f0 0x7f3d980a4eb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac1066a0 0x7f3dac1a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3dac101050 0x7f3dac1993a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 --2- 172.21.15.116:0/1455887960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac100690 0x7f3dac198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 >> 172.21.15.116:0/1455887960 conn(0x7f3dac077820 msgr2=0x7f3dac103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 shutdown_connections 2024-03-19T02:43:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:43:31.356+0000 7f3db0a8a640 1 -- 172.21.15.116:0/1455887960 wait complete. 2024-03-19T02:43:31.369 INFO:tasks.cephfs.filesystem:scrub status for tag:c4c80883-0b16-4fdf-85a2-85cfd5a14eff - {'path': '/', 'tag': 'c4c80883-0b16-4fdf-85a2-85cfd5a14eff', 'options': 'recursive,force'} 2024-03-19T02:43:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:31 smithi162 ceph-mon[35761]: pgmap v494: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 264 op/s 2024-03-19T02:43:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:31 smithi154 ceph-mon[35693]: pgmap v494: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 264 op/s 2024-03-19T02:43:32.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20251 5.94 MB/sec execute 54 sec latency 210.731 ms 2024-03-19T02:43:32.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:32 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:43:32.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:32 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:32.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:32 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:32.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:32 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:32.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:32 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:32 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:43:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:32 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:32 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:32 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:32 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:32 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:43:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:32 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:32 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:32 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:32 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20362 5.95 MB/sec execute 55 sec latency 136.507 ms 2024-03-19T02:43:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:33 smithi162 ceph-mon[35761]: pgmap v495: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:33 smithi154 ceph-mon[35693]: pgmap v495: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-03-19T02:43:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:43:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:33 smithi116 ceph-mon[32263]: pgmap v495: 129 pgs: 129 active+clean; 810 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-03-19T02:43:34.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20492 5.98 MB/sec execute 56 sec latency 133.160 ms 2024-03-19T02:43:35.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20588 5.99 MB/sec execute 57 sec latency 154.586 ms 2024-03-19T02:43:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:35 smithi162 ceph-mon[35761]: pgmap v496: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:43:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:35 smithi154 ceph-mon[35693]: pgmap v496: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:43:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:35 smithi116 ceph-mon[32263]: pgmap v496: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:43:36.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20703 6.07 MB/sec execute 58 sec latency 172.237 ms 2024-03-19T02:43:37.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 20851 6.26 MB/sec execute 59 sec latency 171.648 ms 2024-03-19T02:43:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:37 smithi162 ceph-mon[35761]: pgmap v497: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:37 smithi154 ceph-mon[35693]: pgmap v497: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:37 smithi116 ceph-mon[32263]: pgmap v497: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:38.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21001 6.28 MB/sec execute 60 sec latency 173.376 ms 2024-03-19T02:43:39.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:39 smithi116 ceph-mon[32263]: pgmap v498: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:39.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21187 6.38 MB/sec execute 61 sec latency 129.367 ms 2024-03-19T02:43:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:39 smithi162 ceph-mon[35761]: pgmap v498: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:39 smithi154 ceph-mon[35693]: pgmap v498: 129 pgs: 129 active+clean; 821 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-19T02:43:40.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21342 6.47 MB/sec execute 62 sec latency 159.440 ms 2024-03-19T02:43:41.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21444 6.40 MB/sec execute 63 sec latency 127.771 ms 2024-03-19T02:43:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:41 smithi162 ceph-mon[35761]: pgmap v499: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-03-19T02:43:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:41 smithi154 ceph-mon[35693]: pgmap v499: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-03-19T02:43:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:41 smithi116 ceph-mon[32263]: pgmap v499: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 238 op/s 2024-03-19T02:43:42.323 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21531 6.36 MB/sec execute 64 sec latency 128.337 ms 2024-03-19T02:43:43.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21592 6.33 MB/sec execute 65 sec latency 123.164 ms 2024-03-19T02:43:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:43 smithi162 ceph-mon[35761]: pgmap v500: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-03-19T02:43:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:43 smithi154 ceph-mon[35693]: pgmap v500: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-03-19T02:43:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:43 smithi116 ceph-mon[32263]: pgmap v500: 129 pgs: 129 active+clean; 839 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 147 op/s 2024-03-19T02:43:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21686 6.33 MB/sec execute 66 sec latency 131.249 ms 2024-03-19T02:43:44.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:44 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:44 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:44 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:45.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21755 6.31 MB/sec execute 67 sec latency 174.301 ms 2024-03-19T02:43:45.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:45 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:43:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:45 smithi162 ceph-mon[35761]: pgmap v501: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:43:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:45 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:43:46.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:45 smithi154 ceph-mon[35693]: pgmap v501: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:43:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:45 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:43:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:45 smithi116 ceph-mon[32263]: pgmap v501: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:43:46.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21892 6.46 MB/sec execute 68 sec latency 178.470 ms 2024-03-19T02:43:47.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 21993 6.42 MB/sec execute 69 sec latency 153.017 ms 2024-03-19T02:43:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:47 smithi162 ceph-mon[35761]: pgmap v502: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:47 smithi154 ceph-mon[35693]: pgmap v502: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:47 smithi116 ceph-mon[32263]: pgmap v502: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:48.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22112 6.49 MB/sec execute 70 sec latency 178.237 ms 2024-03-19T02:43:49.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22217 6.43 MB/sec execute 71 sec latency 160.588 ms 2024-03-19T02:43:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:49 smithi162 ceph-mon[35761]: pgmap v503: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:49 smithi154 ceph-mon[35693]: pgmap v503: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:49.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:49 smithi116 ceph-mon[32263]: pgmap v503: 129 pgs: 129 active+clean; 845 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-03-19T02:43:50.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22306 6.40 MB/sec execute 72 sec latency 216.240 ms 2024-03-19T02:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22405 6.40 MB/sec execute 73 sec latency 198.740 ms 2024-03-19T02:43:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:51 smithi162 ceph-mon[35761]: pgmap v504: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-03-19T02:43:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:51 smithi154 ceph-mon[35693]: pgmap v504: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-03-19T02:43:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:51 smithi116 ceph-mon[32263]: pgmap v504: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-03-19T02:43:52.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22514 6.34 MB/sec execute 74 sec latency 168.045 ms 2024-03-19T02:43:53.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:53 smithi154 ceph-mon[35693]: pgmap v505: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-03-19T02:43:53.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22624 6.29 MB/sec execute 75 sec latency 165.933 ms 2024-03-19T02:43:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:53 smithi162 ceph-mon[35761]: pgmap v505: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-03-19T02:43:53.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:53 smithi116 ceph-mon[32263]: pgmap v505: 129 pgs: 129 active+clean; 857 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 119 op/s 2024-03-19T02:43:54.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22727 6.24 MB/sec execute 76 sec latency 162.192 ms 2024-03-19T02:43:55.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22855 6.26 MB/sec execute 77 sec latency 156.094 ms 2024-03-19T02:43:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:55 smithi162 ceph-mon[35761]: pgmap v506: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-03-19T02:43:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:55 smithi154 ceph-mon[35693]: pgmap v506: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-03-19T02:43:55.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:55 smithi116 ceph-mon[32263]: pgmap v506: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-03-19T02:43:56.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 22943 6.20 MB/sec execute 78 sec latency 177.372 ms 2024-03-19T02:43:57.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23048 6.20 MB/sec execute 79 sec latency 149.315 ms 2024-03-19T02:43:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:57 smithi162 ceph-mon[35761]: pgmap v507: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:43:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:57 smithi154 ceph-mon[35693]: pgmap v507: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:43:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:57 smithi116 ceph-mon[32263]: pgmap v507: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:43:58.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23150 6.18 MB/sec execute 80 sec latency 141.246 ms 2024-03-19T02:43:59.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23254 6.16 MB/sec execute 81 sec latency 142.863 ms 2024-03-19T02:43:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:43:59 smithi162 ceph-mon[35761]: pgmap v508: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:43:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:43:59 smithi154 ceph-mon[35693]: pgmap v508: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:43:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:43:59 smithi116 ceph-mon[32263]: pgmap v508: 129 pgs: 129 active+clean; 863 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 128 op/s 2024-03-19T02:44:00.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23322 6.12 MB/sec execute 82 sec latency 521.052 ms 2024-03-19T02:44:01.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23366 6.06 MB/sec execute 83 sec latency 1049.880 ms 2024-03-19T02:44:01.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:44:01.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:44:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:01 smithi162 ceph-mon[35761]: pgmap v509: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 227 op/s 2024-03-19T02:44:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:01 smithi154 ceph-mon[35693]: pgmap v509: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 227 op/s 2024-03-19T02:44:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:01 smithi116 ceph-mon[32263]: pgmap v509: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 227 op/s 2024-03-19T02:44:01.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 -- 172.21.15.116:0/39153963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36800744f0 msgr2=0x7f368010db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:01.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 --2- 172.21.15.116:0/39153963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36800744f0 0x7f368010db30 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3674009900 tx=0x7f367402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 -- 172.21.15.116:0/39153963 shutdown_connections 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 --2- 172.21.15.116:0/39153963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f368010e070 0x7f3680110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 --2- 172.21.15.116:0/39153963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36800744f0 0x7f368010db30 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 --2- 172.21.15.116:0/39153963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3680073bd0 0x7f3680073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 -- 172.21.15.116:0/39153963 >> 172.21.15.116:0/39153963 conn(0x7f3680078be0 msgr2=0x7f3680078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 -- 172.21.15.116:0/39153963 shutdown_connections 2024-03-19T02:44:01.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.984+0000 7f36860fe640 1 -- 172.21.15.116:0/39153963 wait complete. 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 Processor -- start 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 -- start start 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 0x7f368019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f368010e070 0x7f368019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113c40 con 0x7f3680073bd0 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113ac0 con 0x7f368010e070 2024-03-19T02:44:01.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f36860fe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113dc0 con 0x7f36800744f0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42748/0 (socket says 172.21.15.116:42748) 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f367f7fe640 1 -- 172.21.15.116:0/2509293963 learned_addr learned my addr 172.21.15.116:0/2509293963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f367effd640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 0x7f368019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.985+0000 7f367ffff640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f368010e070 0x7f368019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 -- 172.21.15.116:0/2509293963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 msgr2=0x7f368019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 0x7f368019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 -- 172.21.15.116:0/2509293963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f368010e070 msgr2=0x7f368019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f368010e070 0x7f368019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 -- 172.21.15.116:0/2509293963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368019f1e0 con 0x7f3680073bd0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367effd640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 0x7f368019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f367000e9f0 tx=0x7f367000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:01.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367cff9640 1 -- 172.21.15.116:0/2509293963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367000cd80 con 0x7f3680073bd0 2024-03-19T02:44:01.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368019f4d0 con 0x7f3680073bd0 2024-03-19T02:44:01.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367cff9640 1 -- 172.21.15.116:0/2509293963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36700108e0 con 0x7f3680073bd0 2024-03-19T02:44:01.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.986+0000 7f367cff9640 1 -- 172.21.15.116:0/2509293963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670018920 con 0x7f3680073bd0 2024-03-19T02:44:01.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.991+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36801a6d60 con 0x7f3680073bd0 2024-03-19T02:44:01.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.994+0000 7f367cff9640 1 -- 172.21.15.116:0/2509293963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3670010a90 con 0x7f3680073bd0 2024-03-19T02:44:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.994+0000 7f367cff9640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.995+0000 7f367cff9640 1 -- 172.21.15.116:0/2509293963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f36700d3060 con 0x7f3680073bd0 2024-03-19T02:44:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.995+0000 7f367effd640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:01.995+0000 7f367effd640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3674006010 tx=0x7f3674005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 Processor -- start 2024-03-19T02:44:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 -- start start 2024-03-19T02:44:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f61320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559287eb9500 con 0x559287f55a30 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559287e61be0 con 0x559287f5ef30 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f3688dbd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559287eb55a0 con 0x559287f56350 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42762/0 (socket says 172.21.15.116:42762) 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 -- 172.21.15.116:0/3730086775 learned_addr learned my addr 172.21.15.116:0/3730086775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367effd640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367ffff640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f61320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 -- 172.21.15.116:0/3730086775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 msgr2=0x559287f5e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 -- 172.21.15.116:0/3730086775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 msgr2=0x559287f61320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f61320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.027+0000 7f367f7fe640 1 -- 172.21.15.116:0/3730086775 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559287e83000 con 0x559287f55a30 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.028+0000 7f367effd640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.028+0000 7f367f7fe640 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f36700cc810 tx=0x7f3670010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1236d04b5098347 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.028+0000 7f3665ffb640 1 -- 172.21.15.116:0/3730086775 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367009b6c0 con 0x559287f55a30 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.028+0000 7f3665ffb640 1 -- 172.21.15.116:0/3730086775 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36700d2020 con 0x559287f55a30 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.028+0000 7f3665ffb640 1 -- 172.21.15.116:0/3730086775 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367009ab10 con 0x559287f55a30 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 -- 172.21.15.116:0/3730086775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 msgr2=0x559287f55e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f36700cc810 tx=0x7f3670010ea0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 -- 172.21.15.116:0/3730086775 shutdown_connections 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f61320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3730086775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f55e10 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 -- 172.21.15.116:0/3730086775 >> 172.21.15.116:0/3730086775 conn(0x559287f50160 msgr2=0x559287f50540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 -- 172.21.15.116:0/3730086775 shutdown_connections 2024-03-19T02:44:02.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.029+0000 7f3688dbd740 1 -- 172.21.15.116:0/3730086775 wait complete. 2024-03-19T02:44:02.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.040+0000 7f3688dbd740 1 -- start start 2024-03-19T02:44:02.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 Processor -- start 2024-03-19T02:44:02.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f5c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559287f61c60 con 0x559287f55a30 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559287e75160 con 0x559287f5ef30 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559287f61de0 con 0x559287f56350 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42778/0 (socket says 172.21.15.116:42778) 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367f7fe640 1 -- 172.21.15.116:0/2469756867 learned_addr learned my addr 172.21.15.116:0/2469756867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367ffff640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f5c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 -- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 msgr2=0x559287f5c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f5c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 -- 172.21.15.116:0/2469756867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 msgr2=0x559287f5be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367effd640 1 -- 172.21.15.116:0/2469756867 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559287c977d0 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.041+0000 7f3688dbd740 10 client.0 _ll_register_callbacks cb 0x559287eb1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.042+0000 7f367effd640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5c370 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3674000c00 tx=0x7f367402bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403ca3b99952d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.042+0000 7f365effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.042+0000 7f365effd640 1 -- 172.21.15.116:0/2469756867 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674050020 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.042+0000 7f365effd640 1 -- 172.21.15.116:0/2469756867 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f3674003800 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.042+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36700d2020 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f365effd640 1 -- 172.21.15.116:0/2469756867 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f367405b050 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 10 client.29869 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=38}) -- 0x559287bcd3f0 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f365effd640 1 -- 172.21.15.116:0/2469756867 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3674003a00 con 0x559287f56350 2024-03-19T02:44:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 10 client.29869 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 10 client.29869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 20 client.29869 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287f6ee30 0x559287f712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 4 client.29869 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559287ec6a40 con 0x559287f6ee30 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.043+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287f6ee30 0x559287f712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.044+0000 7f365effd640 10 client.29869 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.044+0000 7f367f7fe640 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287f6ee30 0x559287f712f0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e321dc497b8704dd server_cookie=1be973a007a6aa88 in_seq=0 out_seq=0 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.045+0000 7f365effd640 1 -- 172.21.15.116:0/2469756867 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559287ec6a40 con 0x559287f6ee30 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.045+0000 7f365effd640 10 client.29869 handle_command_reply: tid=0 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.045+0000 7f3688dbd740 1 client.29869 shutdown 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.045+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 msgr2=0x559287f5c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.045+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5c370 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3674000c00 tx=0x7f367402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 shutdown_connections 2024-03-19T02:44:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287f6ee30 0x559287f712f0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f5ef30 0x559287f5c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f5c370 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 --2- 172.21.15.116:0/2469756867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f55a30 0x559287f5be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f365effd640 0 client.29869 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 >> 172.21.15.116:0/2469756867 conn(0x559287f50160 msgr2=0x559287f51eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 shutdown_connections 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 1 -- 172.21.15.116:0/2469756867 wait complete. 2024-03-19T02:44:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.046+0000 7f3688dbd740 20 client.29869 trim_cache size 0 max 16384 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f3688dbd740 1 Processor -- start 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f3688dbd740 1 -- start start 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f367ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34476/0 (socket says 172.21.15.116:34476) 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.073+0000 7f367effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41502/0 (socket says 172.21.15.116:41502) 2024-03-19T02:44:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42790/0 (socket says 172.21.15.116:42790) 2024-03-19T02:44:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f3688dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b0f50 con 0x559287f5ef30 2024-03-19T02:44:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b2a90 con 0x559287f56350 2024-03-19T02:44:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 -- 172.21.15.116:0/1401411160 learned_addr learned my addr 172.21.15.116:0/1401411160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b1620 con 0x559287f55a30 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 -- 172.21.15.116:0/1401411160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 msgr2=0x559287f6aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 -- 172.21.15.116:0/1401411160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 msgr2=0x559287f76bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367f7fe640 1 -- 172.21.15.116:0/1401411160 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559287b8d0f0 con 0x559287f55a30 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367ffff640 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.074+0000 7f367effd640 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.075+0000 7f367f7fe640 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f367009b3c0 tx=0x7f367000cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=833389b34c5eb8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.075+0000 7f36657fa640 1 -- 172.21.15.116:0/1401411160 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700ae070 con 0x559287f55a30 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.075+0000 7f36657fa640 1 -- 172.21.15.116:0/1401411160 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f367009b6c0 con 0x559287f55a30 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.075+0000 7f36657fa640 1 -- 172.21.15.116:0/1401411160 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700b4490 con 0x559287f55a30 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 msgr2=0x559287f5e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f367009b3c0 tx=0x7f367000cf20 comp rx=0 tx=0).stop 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 shutdown_connections 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f76bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f56350 0x559287f6aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 --2- 172.21.15.116:0/1401411160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f55a30 0x559287f5e560 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 >> 172.21.15.116:0/1401411160 conn(0x559287f50160 msgr2=0x559287f4e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 shutdown_connections 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- 172.21.15.116:0/1401411160 wait complete. 2024-03-19T02:44:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.076+0000 7f3688dbd740 1 -- start start 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 Processor -- start 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 0x559287f67200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f62570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f62ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b23c0 con 0x559287f5ef30 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b1cf0 con 0x559287f55a30 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36801aef50 con 0x559287f56350 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f3688dbd740 10 client.0 _ll_register_callbacks cb 0x7f3674009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 0x559287f67200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f62ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f62570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 0x559287f67200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41508/0 (socket says 172.21.15.116:41508) 2024-03-19T02:44:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367f7fe640 1 -- 172.21.15.116:0/3748294055 learned_addr learned my addr 172.21.15.116:0/3748294055 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 -- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 msgr2=0x559287f67200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 0x559287f67200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 -- 172.21.15.116:0/3748294055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 msgr2=0x559287f62ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f62ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.077+0000 7f367effd640 1 -- 172.21.15.116:0/3748294055 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559287f631f0 con 0x559287f56350 2024-03-19T02:44:02.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f367effd640 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f62570 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3674036040 tx=0x7f3674041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb76222003a000e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:02.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674064020 con 0x559287f56350 2024-03-19T02:44:02.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f367009b6c0 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f367403f490 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f367406f050 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f3688dbd740 10 client.29875 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559287fd5130 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f367009b6c0 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.078+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f36740604e0 con 0x559287f56350 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 10 client.29875 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 10 client.29875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 20 client.29875 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287fd5930 0x559287fd7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 4 client.29875 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559287ec0d90 con 0x559287fd5930 2024-03-19T02:44:02.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f367f7fe640 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287fd5930 0x559287fd7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f365effd640 10 client.29875 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.079+0000 7f367f7fe640 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287fd5930 0x559287fd7d50 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=708a4c5bcd0ee247 server_cookie=af657d34226a8e22 in_seq=0 out_seq=0 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f365effd640 1 -- 172.21.15.116:0/3748294055 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559287ec0d90 con 0x559287fd5930 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f365effd640 10 client.29875 handle_command_reply: tid=0 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 client.29875 shutdown 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 msgr2=0x559287f62570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f62570 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3674036040 tx=0x7f3674041000 comp rx=0 tx=0).stop 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 shutdown_connections 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559287fd5930 0x559287fd7d50 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559287f5ef30 0x559287f62ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559287f56350 0x559287f62570 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 --2- 172.21.15.116:0/3748294055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559287f55a30 0x559287f67200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f365effd640 0 client.29875 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 >> 172.21.15.116:0/3748294055 conn(0x559287f50160 msgr2=0x559287f4f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 shutdown_connections 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 1 -- 172.21.15.116:0/3748294055 wait complete. 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.081+0000 7f3688dbd740 20 client.29875 trim_cache size 0 max 16384 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:44:02.086 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 msgr2=0x7f36680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3674006010 tx=0x7f3674005e40 comp rx=0 tx=0).stop 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 msgr2=0x7f368019a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f367000e9f0 tx=0x7f367000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 shutdown_connections 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f36680a2970 0x7f36680a4e30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f368010e070 0x7f368019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36800744f0 0x7f368019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 --2- 172.21.15.116:0/2509293963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3680073bd0 0x7f368019a1f0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 >> 172.21.15.116:0/2509293963 conn(0x7f3680078be0 msgr2=0x7f36800fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 shutdown_connections 2024-03-19T02:44:02.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.082+0000 7f36860fe640 1 -- 172.21.15.116:0/2509293963 wait complete. 2024-03-19T02:44:02.107 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:44:02.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 -- 172.21.15.116:0/1937860095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e41017b0 msgr2=0x7f24e4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 --2- 172.21.15.116:0/1937860095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e41017b0 0x7f24e4101c30 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009890 tx=0x7f24d802bb70 comp rx=0 tx=0).stop 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 -- 172.21.15.116:0/1937860095 shutdown_connections 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 --2- 172.21.15.116:0/1937860095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e4102170 0x7f24e410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 --2- 172.21.15.116:0/1937860095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e41017b0 0x7f24e4101c30 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 --2- 172.21.15.116:0/1937860095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41077c0 0x7f24e4107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 -- 172.21.15.116:0/1937860095 >> 172.21.15.116:0/1937860095 conn(0x7f24e40fd2c0 msgr2=0x7f24e40ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 -- 172.21.15.116:0/1937860095 shutdown_connections 2024-03-19T02:44:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.239+0000 7f24e9953640 1 -- 172.21.15.116:0/1937860095 wait complete. 2024-03-19T02:44:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 Processor -- start 2024-03-19T02:44:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 -- start start 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e41077c0 0x7f24e419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24e4111a80 con 0x7f24e4102170 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24e4111900 con 0x7f24e41017b0 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.240+0000 7f24e9953640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f24e4111c00 con 0x7f24e41077c0 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42812/0 (socket says 172.21.15.116:42812) 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41526/0 (socket says 172.21.15.116:41526) 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 -- 172.21.15.116:0/300033875 learned_addr learned my addr 172.21.15.116:0/300033875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e37fe640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e41077c0 0x7f24e419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 -- 172.21.15.116:0/300033875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e41077c0 msgr2=0x7f24e419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e41077c0 0x7f24e419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 -- 172.21.15.116:0/300033875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 msgr2=0x7f24e4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 -- 172.21.15.116:0/300033875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e419cfd0 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e2ffd640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e27fc640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009860 tx=0x7f24d8035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8041020 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e419d260 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24d802c9c0 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8039040 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.241+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e4073a50 con 0x7f24e4102170 2024-03-19T02:44:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.243+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e4068610 con 0x7f24e4102170 2024-03-19T02:44:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.244+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f24d8044020 con 0x7f24e4102170 2024-03-19T02:44:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.245+0000 7f24e8951640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 0x7f24cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.245+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f24d80f08f0 con 0x7f24e4102170 2024-03-19T02:44:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.246+0000 7f24e2ffd640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 0x7f24cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.246+0000 7f24e2ffd640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 0x7f24cc0a4eb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f24d4005fd0 tx=0x7f24d4012040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.253+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24d804b020 con 0x7f24e4102170 2024-03-19T02:44:02.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23494 6.10 MB/sec execute 84 sec latency 151.155 ms 2024-03-19T02:44:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.542+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24e4109d40 con 0x7f24e4102170 2024-03-19T02:44:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.543+0000 7f24e8951640 1 -- 172.21.15.116:0/300033875 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f24d80bd380 con 0x7f24e4102170 2024-03-19T02:44:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:44:02.544 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:44:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:44:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 msgr2=0x7f24cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 0x7f24cc0a4eb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f24d4005fd0 tx=0x7f24d4012040 comp rx=0 tx=0).stop 2024-03-19T02:44:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 msgr2=0x7f24e4198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009860 tx=0x7f24d8035990 comp rx=0 tx=0).stop 2024-03-19T02:44:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 shutdown_connections 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f24cc0a29f0 0x7f24cc0a4eb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e41077c0 0x7f24e419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24e4102170 0x7f24e4198520 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 --2- 172.21.15.116:0/300033875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e41017b0 0x7f24e4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 >> 172.21.15.116:0/300033875 conn(0x7f24e40fd2c0 msgr2=0x7f24e40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 shutdown_connections 2024-03-19T02:44:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.545+0000 7f24e9953640 1 -- 172.21.15.116:0/300033875 wait complete. 2024-03-19T02:44:02.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 -- 172.21.15.116:0/1113448410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1077d0 msgr2=0x7f4bfc107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 --2- 172.21.15.116:0/1113448410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc107bb0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009930 tx=0x7f4bec02bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 -- 172.21.15.116:0/1113448410 shutdown_connections 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 --2- 172.21.15.116:0/1113448410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 0x7f4bfc10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 --2- 172.21.15.116:0/1113448410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 --2- 172.21.15.116:0/1113448410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc107bb0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 -- 172.21.15.116:0/1113448410 >> 172.21.15.116:0/1113448410 conn(0x7f4bfc0fd2d0 msgr2=0x7f4bfc0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 -- 172.21.15.116:0/1113448410 shutdown_connections 2024-03-19T02:44:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 -- 172.21.15.116:0/1113448410 wait complete. 2024-03-19T02:44:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.686+0000 7f4c01f89640 1 Processor -- start 2024-03-19T02:44:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 -- start start 2024-03-19T02:44:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 0x7f4bfc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc111a50 con 0x7f4bfc1017c0 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1118d0 con 0x7f4bfc1077d0 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4c01f89640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc111bd0 con 0x7f4bfc102180 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42822/0 (socket says 172.21.15.116:42822) 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/15296411 learned_addr learned my addr 172.21.15.116:0/15296411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 0x7f4bfc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/15296411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 msgr2=0x7f4bfc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 0x7f4bfc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/15296411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1077d0 msgr2=0x7f4bfc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/15296411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc19cff0 con 0x7f4bfc1017c0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.687+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009230 tx=0x7f4bec02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.688+0000 7f4bf8ff9640 1 -- 172.21.15.116:0/15296411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec043020 con 0x7f4bfc1017c0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.688+0000 7f4bf8ff9640 1 -- 172.21.15.116:0/15296411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bec02ca70 con 0x7f4bfc1017c0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.688+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc19d280 con 0x7f4bfc1017c0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.688+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc1a4b70 con 0x7f4bfc1017c0 2024-03-19T02:44:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.688+0000 7f4bf8ff9640 1 -- 172.21.15.116:0/15296411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec003970 con 0x7f4bfc1017c0 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4bf8ff9640 1 -- 172.21.15.116:0/15296411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4bec046020 con 0x7f4bfc1017c0 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 Processor -- start 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 -- start start 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a30780a4b0 con 0x55a3078a7bf0 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a3077b2d70 con 0x55a3078b07d0 2024-03-19T02:44:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a307806550 con 0x55a3078a72d0 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.691+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078a76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34528/0 (socket says 172.21.15.116:34528) 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfbfff640 1 -- 172.21.15.116:0/1287746186 learned_addr learned my addr 172.21.15.116:0/1287746186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1287746186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 msgr2=0x55a3078a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078a76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1287746186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 msgr2=0x55a3078b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1287746186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3077d5210 con 0x55a3078b07d0 2024-03-19T02:44:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bf8ff9640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 0x7f4be40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b2bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0013ba0 tx=0x7f4bf0009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825b696ff562d9fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 0x7f4be40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.692+0000 7f4bf8ff9640 1 -- 172.21.15.116:0/15296411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f4bec0f10d0 con 0x7f4bfc1017c0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.693+0000 7f4be1ffb640 1 -- 172.21.15.116:0/1287746186 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0009e20 con 0x55a3078b07d0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.693+0000 7f4be1ffb640 1 -- 172.21.15.116:0/1287746186 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bf000e990 con 0x55a3078b07d0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.693+0000 7f4be1ffb640 1 -- 172.21.15.116:0/1287746186 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0003b40 con 0x55a3078b07d0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1287746186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 msgr2=0x55a3078b2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 0x7f4be40a4eb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40125f0 tx=0x7f4bf4009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b2bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0013ba0 tx=0x7f4bf0009b70 comp rx=0 tx=0).stop 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1287746186 shutdown_connections 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b2bc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1287746186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1287746186 >> 172.21.15.116:0/1287746186 conn(0x55a3078a1a00 msgr2=0x55a3078a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1287746186 shutdown_connections 2024-03-19T02:44:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1287746186 wait complete. 2024-03-19T02:44:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 -- start start 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.694+0000 7f4c04b7a740 1 Processor -- start 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3078b35c0 con 0x55a3078a7bf0 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a3077c6310 con 0x55a3078b07d0 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3078b3740 con 0x55a3078a72d0 2024-03-19T02:44:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4c04b7a740 10 client.0 _ll_register_callbacks cb 0x55a3077ba2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42836/0 (socket says 172.21.15.116:42836) 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1001364429 learned_addr learned my addr 172.21.15.116:0/1001364429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 msgr2=0x55a3078ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1001364429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 msgr2=0x55a3078ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfaffd640 1 -- 172.21.15.116:0/1001364429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30751e3f0 con 0x55a3078a7bf0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.695+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00037a0 tx=0x7f4bf00037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18734a0110324205 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0004050 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f4bf0019070 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bf000e990 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4bf0034050 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4c04b7a740 10 client.20178 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a3075e87d0 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.696+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bf002b070 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f4bf001d650 con 0x55a3078a7bf0 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 10 client.20178 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 10 client.20178 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 20 client.20178 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a3078c0a90 0x55a3078c2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 4 client.20178 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4bf000e990 con 0x55a3078c0a90 2024-03-19T02:44:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a3078c0a90 0x55a3078c2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4bdaffd640 10 client.20178 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.697+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a3078c0a90 0x55a3078c2e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=da4797ab3b520a77 server_cookie=ff9d86927f69b62e in_seq=0 out_seq=0 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.698+0000 7f4bdaffd640 1 -- 172.21.15.116:0/1001364429 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4bf000e990 con 0x55a3078c0a90 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.698+0000 7f4bdaffd640 10 client.20178 handle_command_reply: tid=0 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.698+0000 7f4c04b7a740 1 client.20178 shutdown 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 msgr2=0x55a3078adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00037a0 tx=0x7f4bf00037d0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 shutdown_connections 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a3078c0a90 0x55a3078c2e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4bdaffd640 0 client.20178 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078adce0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/1001364429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 >> 172.21.15.116:0/1001364429 conn(0x55a3078a1a00 msgr2=0x55a3078a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 shutdown_connections 2024-03-19T02:44:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 1 -- 172.21.15.116:0/1001364429 wait complete. 2024-03-19T02:44:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.699+0000 7f4c04b7a740 20 client.20178 trim_cache size 0 max 16384 2024-03-19T02:44:02.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 Processor -- start 2024-03-19T02:44:02.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 -- start start 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 0x55a3078ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1acb30 con 0x55a3078a72d0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ae670 con 0x55a3078b07d0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ad200 con 0x55a3078a7bf0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42842/0 (socket says 172.21.15.116:42842) 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/3597272724 learned_addr learned my addr 172.21.15.116:0/3597272724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 0x55a3078ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/3597272724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 msgr2=0x55a3078ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 0x55a3078ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/3597272724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 msgr2=0x55a3078a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/3597272724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3078a1830 con 0x55a3078a72d0 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 0x55a3078ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.716+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b98c0 tx=0x7f4bec0dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0928806061d0ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4be17fa640 1 -- 172.21.15.116:0/3597272724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec0b9700 con 0x55a3078a72d0 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4be17fa640 1 -- 172.21.15.116:0/3597272724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bec0bdb60 con 0x55a3078a72d0 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4be17fa640 1 -- 172.21.15.116:0/3597272724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec0bcdc0 con 0x55a3078a72d0 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 -- 172.21.15.116:0/3597272724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 msgr2=0x55a3078afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b98c0 tx=0x7f4bec0dd040 comp rx=0 tx=0).stop 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 -- 172.21.15.116:0/3597272724 shutdown_connections 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a7bf0 0x55a3078ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/3597272724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a72d0 0x55a3078afed0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.717+0000 7f4c04b7a740 1 -- 172.21.15.116:0/3597272724 >> 172.21.15.116:0/3597272724 conn(0x55a3078a1a00 msgr2=0x55a3078a0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- 172.21.15.116:0/3597272724 shutdown_connections 2024-03-19T02:44:02.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- 172.21.15.116:0/3597272724 wait complete. 2024-03-19T02:44:02.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- start start 2024-03-19T02:44:02.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 Processor -- start 2024-03-19T02:44:02.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1adfa0 con 0x55a3078a7bf0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1ad8d0 con 0x55a3078b07d0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc1aab30 con 0x55a3078a72d0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.718+0000 7f4c04b7a740 10 client.0 _ll_register_callbacks cb 0x7f4bec0f4af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34562/0 (socket says 172.21.15.116:34562) 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/2807502027 learned_addr learned my addr 172.21.15.116:0/2807502027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfbfff640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/2807502027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 msgr2=0x55a3078b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/2807502027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 msgr2=0x55a3078b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 -- 172.21.15.116:0/2807502027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3078b2bc0 con 0x55a3078a72d0 2024-03-19T02:44:02.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bfb7fe640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b98c0 tx=0x7f4bec0bcdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d31820e65ddb559 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec0bcfa0 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f4bec0bb760 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bec0bdb60 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4bec0bd230 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4c04b7a740 10 client.29899 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.719+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a307877880 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bec0bdb60 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f4bec0c1170 con 0x55a3078a72d0 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 10 client.29899 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 10 client.29899 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:44:02.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 20 client.29899 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a307927350 0x55a307929740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 4 client.29899 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3078683e0 con 0x55a307927350 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a307927350 0x55a307929740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4bfaffd640 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a307927350 0x55a307929740 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=61cb0e802d1f5d8e server_cookie=947bf9715c7d9329 in_seq=0 out_seq=0 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.720+0000 7f4bdaffd640 10 client.29899 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4bdaffd640 1 -- 172.21.15.116:0/2807502027 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3078683e0 con 0x55a307927350 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4bdaffd640 10 client.29899 handle_command_reply: tid=0 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 client.29899 shutdown 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 msgr2=0x55a3078b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b98c0 tx=0x7f4bec0bcdc0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 shutdown_connections 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55a307927350 0x55a307929740 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a3078b07d0 0x55a3078b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4bdaffd640 0 client.29899 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3078a7bf0 0x55a3078b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 --2- 172.21.15.116:0/2807502027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3078a72d0 0x55a3078b4510 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 >> 172.21.15.116:0/2807502027 conn(0x55a3078a1a00 msgr2=0x55a3078a0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 shutdown_connections 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 1 -- 172.21.15.116:0/2807502027 wait complete. 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.721+0000 7f4c04b7a740 20 client.29899 trim_cache size 0 max 16384 2024-03-19T02:44:02.728 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 msgr2=0x7f4be40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 0x7f4be40a4eb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40125f0 tx=0x7f4bf4009500 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 msgr2=0x7f4bfc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009230 tx=0x7f4bec02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 shutdown_connections 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4be40a29f0 0x7f4be40a4eb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc1077d0 0x7f4bfc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bfc102180 0x7f4bfc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 --2- 172.21.15.116:0/15296411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bfc1017c0 0x7f4bfc198000 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 >> 172.21.15.116:0/15296411 conn(0x7f4bfc0fd2d0 msgr2=0x7f4bfc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 shutdown_connections 2024-03-19T02:44:02.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.722+0000 7f4c01f89640 1 -- 172.21.15.116:0/15296411 wait complete. 2024-03-19T02:44:02.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.866+0000 7f8e3506c640 1 -- 172.21.15.116:0/2739361244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 msgr2=0x7f8e30100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.866+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2739361244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30100b20 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009930 tx=0x7f8e2402bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 -- 172.21.15.116:0/2739361244 shutdown_connections 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2739361244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e30101060 0x7f8e3010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2739361244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30100b20 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2739361244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e301066b0 0x7f8e30106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 -- 172.21.15.116:0/2739361244 >> 172.21.15.116:0/2739361244 conn(0x7f8e300778d0 msgr2=0x7f8e30077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 -- 172.21.15.116:0/2739361244 shutdown_connections 2024-03-19T02:44:02.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 -- 172.21.15.116:0/2739361244 wait complete. 2024-03-19T02:44:02.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.867+0000 7f8e3506c640 1 Processor -- start 2024-03-19T02:44:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 -- start start 2024-03-19T02:44:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e301066b0 0x7f8e3019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e30068d10 con 0x7f8e301006a0 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e30068b90 con 0x7f8e30101060 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e3506c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e30068e90 con 0x7f8e301066b0 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41580/0 (socket says 172.21.15.116:41580) 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 -- 172.21.15.116:0/2296837898 learned_addr learned my addr 172.21.15.116:0/2296837898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e301066b0 0x7f8e3019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 -- 172.21.15.116:0/2296837898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e301066b0 msgr2=0x7f8e3019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e301066b0 0x7f8e3019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 -- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 msgr2=0x7f8e30198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.868+0000 7f8e2e59c640 1 -- 172.21.15.116:0/2296837898 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3019cfb0 con 0x7f8e30101060 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009230 tx=0x7f8e2402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e17fff640 1 -- 172.21.15.116:0/2296837898 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24043020 con 0x7f8e30101060 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3019d240 con 0x7f8e30101060 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e17fff640 1 -- 172.21.15.116:0/2296837898 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2402ca70 con 0x7f8e30101060 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e17fff640 1 -- 172.21.15.116:0/2296837898 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24003a50 con 0x7f8e30101060 2024-03-19T02:44:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.869+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e301a4b30 con 0x7f8e30101060 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 Processor -- start 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e17fff640 1 -- 172.21.15.116:0/2296837898 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8e24046020 con 0x7f8e30101060 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 -- start start 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dcafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564569d1e4b0 con 0x564569dbba90 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564569cc6d70 con 0x564569dc8be0 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564569d1a550 con 0x564569dbb0e0 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2f59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42874/0 (socket says 172.21.15.116:42874) 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/3960305189 learned_addr learned my addr 172.21.15.116:0/3960305189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2f59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34584/0 (socket says 172.21.15.116:34584) 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.872+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dcafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e17fff640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 0x7f8e180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 -- 172.21.15.116:0/3960305189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 msgr2=0x564569dcafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dcafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 -- 172.21.15.116:0/3960305189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 msgr2=0x564569dc86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 -- 172.21.15.116:0/3960305189 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564569ce9210 con 0x564569dbb0e0 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dcafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 0x7f8e180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e17fff640 1 -- 172.21.15.116:0/2296837898 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f8e240f0f50 con 0x7f8e30101060 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8e28011490 tx=0x7f8e28011950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3f115064178309 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e157fa640 1 -- 172.21.15.116:0/3960305189 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28017070 con 0x564569dbb0e0 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 0x7f8e180a4e30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e28022000 tx=0x7f8e28023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e157fa640 1 -- 172.21.15.116:0/3960305189 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2800fcb0 con 0x564569dbb0e0 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.873+0000 7f8e157fa640 1 -- 172.21.15.116:0/3960305189 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2801bc70 con 0x564569dbb0e0 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- 172.21.15.116:0/3960305189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 msgr2=0x564569dbb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8e28011490 tx=0x7f8e28011950 comp rx=0 tx=0).stop 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- 172.21.15.116:0/3960305189 shutdown_connections 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dcafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/3960305189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dbb4c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- 172.21.15.116:0/3960305189 >> 172.21.15.116:0/3960305189 conn(0x564569db5a00 msgr2=0x564569db5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- 172.21.15.116:0/3960305189 shutdown_connections 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- 172.21.15.116:0/3960305189 wait complete. 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 -- start start 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.874+0000 7f8e37c4b740 1 Processor -- start 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dc6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564569dcb9a0 con 0x564569dbba90 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564569cda310 con 0x564569dc8be0 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564569dcbb20 con 0x564569dbb0e0 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e37c4b740 10 client.0 _ll_register_callbacks cb 0x564569cce2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34588/0 (socket says 172.21.15.116:34588) 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2176149154 learned_addr learned my addr 172.21.15.116:0/2176149154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dc6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2176149154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 msgr2=0x564569dc6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dc6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2176149154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 msgr2=0x564569dc6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2176149154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564569a323f0 con 0x564569dbb0e0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e20012710 tx=0x7f8e20012b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1238dc1ddd134f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.875+0000 7f8e12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20004050 con 0x564569dbb0e0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e2800fcb0 con 0x564569dbb0e0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f8e200041f0 con 0x564569dbb0e0 2024-03-19T02:44:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8e20025050 con 0x564569dbb0e0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 10 client.29911 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564569afc7d0 con 0x564569dbb0e0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2800fcb0 con 0x564569dbb0e0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8e20003a40 con 0x564569dbb0e0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 10 client.29911 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 10 client.29911 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 20 client.29911 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569dd8f60 0x564569ddb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 4 client.29911 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564569d7c3e0 con 0x564569dd8f60 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.876+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569dd8f60 0x564569ddb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.877+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569dd8f60 0x564569ddb400 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f594bc8c997e3427 server_cookie=b58f0be2ae71600b in_seq=0 out_seq=0 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.877+0000 7f8e12ff5640 10 client.29911 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.877+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2176149154 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564569d7c3e0 con 0x564569dd8f60 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.877+0000 7f8e12ff5640 10 client.29911 handle_command_reply: tid=0 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.877+0000 7f8e37c4b740 1 client.29911 shutdown 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 msgr2=0x564569dc5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8e20012710 tx=0x7f8e20012b10 comp rx=0 tx=0).stop 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 shutdown_connections 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dc8be0 0x564569dc6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569dd8f60 0x564569ddb400 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbba90 0x564569dc6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2176149154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dc5c10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e12ff5640 0 client.29911 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 >> 172.21.15.116:0/2176149154 conn(0x564569db5a00 msgr2=0x564569db7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 shutdown_connections 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2176149154 wait complete. 2024-03-19T02:44:02.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.878+0000 7f8e37c4b740 20 client.29911 trim_cache size 0 max 16384 2024-03-19T02:44:02.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 Processor -- start 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 -- start start 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dd4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569db7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 0x564569de0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c00ff10 con 0x564569dc8be0 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c011a50 con 0x564569dbba90 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c0105e0 con 0x564569dbb0e0 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dd4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 0x564569de0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569db7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dd4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34604/0 (socket says 172.21.15.116:34604) 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569db7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41604/0 (socket says 172.21.15.116:41604) 2024-03-19T02:44:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2975989317 learned_addr learned my addr 172.21.15.116:0/2975989317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 -- 172.21.15.116:0/2975989317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 msgr2=0x564569dd4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dd4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 -- 172.21.15.116:0/2975989317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 msgr2=0x564569db7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569db7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 -- 172.21.15.116:0/2975989317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564569dd7bb0 con 0x564569dc8be0 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.894+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 0x564569de0cd0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8e2801bfd0 tx=0x7f8e28009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23a2a0928557532 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.895+0000 7f8e14ff9640 1 -- 172.21.15.116:0/2975989317 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28035070 con 0x564569dc8be0 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.895+0000 7f8e14ff9640 1 -- 172.21.15.116:0/2975989317 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e28003bf0 con 0x564569dc8be0 2024-03-19T02:44:02.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.895+0000 7f8e14ff9640 1 -- 172.21.15.116:0/2975989317 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28017040 con 0x564569dc8be0 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.895+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2975989317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 msgr2=0x564569de0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.895+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 0x564569de0cd0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8e2801bfd0 tx=0x7f8e28009c30 comp rx=0 tx=0).stop 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2975989317 shutdown_connections 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dc8be0 0x564569de0cd0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569db7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2975989317 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dbb0e0 0x564569dd4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2975989317 >> 172.21.15.116:0/2975989317 conn(0x564569db5a00 msgr2=0x564569db6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2975989317 shutdown_connections 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2975989317 wait complete. 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- start start 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 Processor -- start 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569dcd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dc8be0 0x564569e3e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c011380 con 0x564569dbb0e0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c010cb0 con 0x564569dbba90 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c00df10 con 0x564569dc8be0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e37c4b740 10 client.0 _ll_register_callbacks cb 0x7f8e20009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42890/0 (socket says 172.21.15.116:42890) 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2699059809 learned_addr learned my addr 172.21.15.116:0/2699059809 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569dcd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.896+0000 7f8e2f59e640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dc8be0 0x564569e3e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dc8be0 msgr2=0x564569e3e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dc8be0 0x564569e3e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2699059809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 msgr2=0x564569dcd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:02.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569dcd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 -- 172.21.15.116:0/2699059809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564569e3e840 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e2ed9d640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000fc00 tx=0x7f8e2000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcb5be62cd1f188 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20022040 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f8e20003800 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e28003bf0 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8e20039050 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e37c4b740 10 client.20202 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564569e3f9e0 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.897+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e28003bf0 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8e20010430 con 0x564569dbb0e0 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 10 client.20202 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:02.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 10 client.20202 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 20 client.20202 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569e400a0 0x564569e424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 4 client.20202 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564569d2b9f0 con 0x564569e400a0 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569e400a0 0x564569e424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e2e59c640 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569e400a0 0x564569e424c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f4a9f3ebb1444b server_cookie=6d911b68ecfa5ffc in_seq=0 out_seq=0 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e12ff5640 10 client.20202 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e12ff5640 1 -- 172.21.15.116:0/2699059809 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564569d2b9f0 con 0x564569e400a0 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e12ff5640 10 client.20202 handle_command_reply: tid=0 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.898+0000 7f8e37c4b740 1 client.20202 shutdown 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 msgr2=0x564569dccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000fc00 tx=0x7f8e2000fc30 comp rx=0 tx=0).stop 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 shutdown_connections 2024-03-19T02:44:02.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564569dc8be0 0x564569e3e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564569dbba90 0x564569dcd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x564569e400a0 0x564569e424c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 --2- 172.21.15.116:0/2699059809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564569dbb0e0 0x564569dccae0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e12ff5640 0 client.20202 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 >> 172.21.15.116:0/2699059809 conn(0x564569db5a00 msgr2=0x564569dc6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 shutdown_connections 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 1 -- 172.21.15.116:0/2699059809 wait complete. 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e37c4b740 20 client.20202 trim_cache size 0 max 16384 2024-03-19T02:44:02.906 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 msgr2=0x7f8e180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 0x7f8e180a4e30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e28022000 tx=0x7f8e28023000 comp rx=0 tx=0).stop 2024-03-19T02:44:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 msgr2=0x7f8e30198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009230 tx=0x7f8e2402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 shutdown_connections 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e180a2970 0x7f8e180a4e30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e301066b0 0x7f8e3019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e30101060 0x7f8e30198550 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 --2- 172.21.15.116:0/2296837898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e301006a0 0x7f8e30198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.899+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 >> 172.21.15.116:0/2296837898 conn(0x7f8e300778d0 msgr2=0x7f8e30102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.900+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 shutdown_connections 2024-03-19T02:44:02.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:02.900+0000 7f8e3506c640 1 -- 172.21.15.116:0/2296837898 wait complete. 2024-03-19T02:44:02.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:02 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/300033875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:02.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:44:02.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:02 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/300033875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:03.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:02 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/300033875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.037+0000 7f6eab626640 1 -- 172.21.15.116:0/2645919169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41066b0 msgr2=0x7f6ea4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.037+0000 7f6eab626640 1 --2- 172.21.15.116:0/2645919169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41066b0 0x7f6ea4106a90 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009930 tx=0x7f6ea002bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.037+0000 7f6eab626640 1 -- 172.21.15.116:0/2645919169 shutdown_connections 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.037+0000 7f6eab626640 1 --2- 172.21.15.116:0/2645919169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 --2- 172.21.15.116:0/2645919169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41006a0 0x7f6ea4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 --2- 172.21.15.116:0/2645919169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41066b0 0x7f6ea4106a90 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 -- 172.21.15.116:0/2645919169 >> 172.21.15.116:0/2645919169 conn(0x7f6ea40778d0 msgr2=0x7f6ea4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 -- 172.21.15.116:0/2645919169 shutdown_connections 2024-03-19T02:44:03.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 -- 172.21.15.116:0/2645919169 wait complete. 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 Processor -- start 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 -- start start 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.038+0000 7f6eab626640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6eab626640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea4199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6eab626640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 0x7f6ea41a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6eab626640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41109c0 con 0x7f6ea41006a0 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6eab626640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4110840 con 0x7f6ea4101060 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6eab626640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4110b40 con 0x7f6ea41066b0 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42904/0 (socket says 172.21.15.116:42904) 2024-03-19T02:44:03.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea8bc1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea4199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 -- 172.21.15.116:0/1895371278 learned_addr learned my addr 172.21.15.116:0/1895371278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 0x7f6ea41a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 -- 172.21.15.116:0/1895371278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 msgr2=0x7f6ea41a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 0x7f6ea41a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 -- 172.21.15.116:0/1895371278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 msgr2=0x7f6ea4199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea4199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 -- 172.21.15.116:0/1895371278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea41a43a0 con 0x7f6ea41006a0 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea4199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 0x7f6ea41a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.039+0000 7f6e9a7fc640 1 -- 172.21.15.116:0/1895371278 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0043020 con 0x7f6ea41006a0 2024-03-19T02:44:03.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.040+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea41a4570 con 0x7f6ea41006a0 2024-03-19T02:44:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.040+0000 7f6e9a7fc640 1 -- 172.21.15.116:0/1895371278 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ea002caf0 con 0x7f6ea41006a0 2024-03-19T02:44:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.040+0000 7f6e9a7fc640 1 -- 172.21.15.116:0/1895371278 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003b40 con 0x7f6ea41006a0 2024-03-19T02:44:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.040+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a4ab0 con 0x7f6ea41006a0 2024-03-19T02:44:03.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.042+0000 7f6e9a7fc640 1 -- 172.21.15.116:0/1895371278 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6ea0046020 con 0x7f6ea41006a0 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 Processor -- start 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 -- start start 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6e9a7fc640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 0x7f6e900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952246290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952248bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629521a04b0 con 0x56295223dbf0 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562952148d70 con 0x5629522467d0 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6eae20c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56295219c550 con 0x56295223d2d0 2024-03-19T02:44:03.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.043+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 0x7f6e900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea9bc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952248bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34642/0 (socket says 172.21.15.116:34642) 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6e9a7fc640 1 -- 172.21.15.116:0/1895371278 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6ea00f1130 con 0x7f6ea41006a0 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2646065524 learned_addr learned my addr 172.21.15.116:0/2646065524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952246290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2646065524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 msgr2=0x562952248bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952248bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2646065524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 msgr2=0x562952246290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952246290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2646065524 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56295216b210 con 0x56295223d2d0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 0x7f6e900a4e30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0099e0 tx=0x7f6e9c009ef0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6ea002c210 tx=0x7f6ea00bd290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ffbdafc7fdee1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6e8f7fe640 1 -- 172.21.15.116:0/2646065524 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00a12c0 con 0x56295223d2d0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6e8f7fe640 1 -- 172.21.15.116:0/2646065524 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ea00bdbc0 con 0x56295223d2d0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.044+0000 7f6e8f7fe640 1 -- 172.21.15.116:0/2646065524 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003ce0 con 0x56295223d2d0 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 -- 172.21.15.116:0/2646065524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 msgr2=0x56295223d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6ea002c210 tx=0x7f6ea00bd290 comp rx=0 tx=0).stop 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 -- 172.21.15.116:0/2646065524 shutdown_connections 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952248bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952246290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2646065524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x56295223d6b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 -- 172.21.15.116:0/2646065524 >> 172.21.15.116:0/2646065524 conn(0x562952237a00 msgr2=0x562952237de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 -- 172.21.15.116:0/2646065524 shutdown_connections 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.045+0000 7f6eae20c740 1 -- 172.21.15.116:0/2646065524 wait complete. 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 -- start start 2024-03-19T02:44:03.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 Processor -- start 2024-03-19T02:44:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952243ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952244220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56295224d790 con 0x56295223dbf0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56295215c310 con 0x5629522467d0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56295224d910 con 0x56295223d2d0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6eae20c740 10 client.0 _ll_register_callbacks cb 0x5629521502d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6ea8bc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952243ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.046+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34650/0 (socket says 172.21.15.116:34650) 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea93c2640 1 -- 172.21.15.116:0/3268367414 learned_addr learned my addr 172.21.15.116:0/3268367414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952244220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 -- 172.21.15.116:0/3268367414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 msgr2=0x5629522437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 -- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 msgr2=0x562952244220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952244220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 -- 172.21.15.116:0/3268367414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562951eb43f0 con 0x56295223dbf0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952243ce0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f6e94012cf0 tx=0x7f6e9400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2f631ee36e44d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6e8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94012110 con 0x56295223dbf0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6e94004050 con 0x56295223dbf0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.047+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea00bdbc0 con 0x56295223dbf0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6e94025050 con 0x56295223dbf0 2024-03-19T02:44:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 10 client.20214 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562951f7e7d0 con 0x56295223dbf0 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ea00bdbc0 con 0x56295223dbf0 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6e94010430 con 0x56295223dbf0 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 10 client.20214 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 10 client.20214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 20 client.20214 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56295225ac10 0x56295225d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 4 client.20214 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629521fe3e0 con 0x56295225ac10 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.048+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56295225ac10 0x56295225d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.049+0000 7f6e8cff9640 10 client.20214 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.049+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56295225ac10 0x56295225d060 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6c7e5cb2062165b server_cookie=cebf5defb0661a4a in_seq=0 out_seq=0 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6e8cff9640 1 -- 172.21.15.116:0/3268367414 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629521fe3e0 con 0x56295225ac10 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6e8cff9640 10 client.20214 handle_command_reply: tid=0 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 client.20214 shutdown 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 msgr2=0x562952243ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952243ce0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f6e94012cf0 tx=0x7f6e9400f790 comp rx=0 tx=0).stop 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 shutdown_connections 2024-03-19T02:44:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56295225ac10 0x56295225d060 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x562952244220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223dbf0 0x562952243ce0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 --2- 172.21.15.116:0/3268367414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223d2d0 0x5629522437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 >> 172.21.15.116:0/3268367414 conn(0x562952237a00 msgr2=0x562952239750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.050+0000 7f6e8cff9640 0 client.20214 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.051+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 shutdown_connections 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.051+0000 7f6eae20c740 1 -- 172.21.15.116:0/3268367414 wait complete. 2024-03-19T02:44:03.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.051+0000 7f6eae20c740 20 client.20214 trim_cache size 0 max 16384 2024-03-19T02:44:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 Processor -- start 2024-03-19T02:44:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 -- start start 2024-03-19T02:44:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 0x562952245ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x562952242070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41acae0 con 0x5629522467d0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41ae620 con 0x56295223d2d0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6eae20c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41ad1b0 con 0x56295223dbf0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6ea8bc1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x562952242070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6ea9bc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6ea9bc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42938/0 (socket says 172.21.15.116:42938) 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.070+0000 7f6ea9bc3640 1 -- 172.21.15.116:0/1230803886 learned_addr learned my addr 172.21.15.116:0/1230803886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 0x562952245ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 -- 172.21.15.116:0/1230803886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 msgr2=0x562952242070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x562952242070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 -- 172.21.15.116:0/1230803886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 msgr2=0x562952245ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 0x562952245ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 -- 172.21.15.116:0/1230803886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562952242750 con 0x5629522467d0 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 0x562952245ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x562952242070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0130c0 tx=0x7f6e9c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb88ce67b740680 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6e8effd640 1 -- 172.21.15.116:0/1230803886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0130f0 con 0x5629522467d0 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6e8effd640 1 -- 172.21.15.116:0/1230803886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e9c013290 con 0x5629522467d0 2024-03-19T02:44:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.071+0000 7f6e8effd640 1 -- 172.21.15.116:0/1230803886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c02bbd0 con 0x5629522467d0 2024-03-19T02:44:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 -- 172.21.15.116:0/1230803886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 msgr2=0x562952256450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0130c0 tx=0x7f6e9c021000 comp rx=0 tx=0).stop 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 -- 172.21.15.116:0/1230803886 shutdown_connections 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629522467d0 0x562952256450 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x562952242070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 --2- 172.21.15.116:0/1230803886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56295223d2d0 0x562952245ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.072+0000 7f6eae20c740 1 -- 172.21.15.116:0/1230803886 >> 172.21.15.116:0/1230803886 conn(0x562952237a00 msgr2=0x562952236ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- 172.21.15.116:0/1230803886 shutdown_connections 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- 172.21.15.116:0/1230803886 wait complete. 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- start start 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 Processor -- start 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x56295224e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x56295224ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41adf50 con 0x56295223d2d0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41ad880 con 0x5629522467d0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6eae20c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41aaae0 con 0x56295223dbf0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.073+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42952/0 (socket says 172.21.15.116:42952) 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea8bc1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x56295224e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6eae20c740 10 client.0 _ll_register_callbacks cb 0x7f6e94012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2026750052 learned_addr learned my addr 172.21.15.116:0/2026750052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea9bc3640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x56295224ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2026750052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 msgr2=0x56295224e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x56295224e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 msgr2=0x56295224ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x56295224ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 -- 172.21.15.116:0/2026750052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629522bff50 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6ea93c2640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00bd450 tx=0x7f6ea00bd3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8fad3af12e9a36 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6e8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00db060 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f6ea00d6270 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e9c013290 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6ea00d5030 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6eae20c740 10 client.20220 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.074+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5629522c1190 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e9c013290 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6ea00ee720 con 0x56295223d2d0 2024-03-19T02:44:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 10 client.20220 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 10 client.20220 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 20 client.20220 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5629522c16d0 0x5629522c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 4 client.20220 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629521ad9f0 con 0x5629522c16d0 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5629522c16d0 0x5629522c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6e8cff9640 10 client.20220 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.075+0000 7f6ea8bc1640 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5629522c16d0 0x5629522c3ac0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a230b5b44f125596 server_cookie=c9569d7a85a293ec in_seq=0 out_seq=0 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6e8cff9640 1 -- 172.21.15.116:0/2026750052 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629521ad9f0 con 0x5629522c16d0 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6e8cff9640 10 client.20220 handle_command_reply: tid=0 2024-03-19T02:44:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 client.20220 shutdown 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 msgr2=0x56295224e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00bd450 tx=0x7f6ea00bd3c0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 shutdown_connections 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5629522c16d0 0x5629522c3ac0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629522467d0 0x56295224ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6e8cff9640 0 client.20220 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56295223dbf0 0x56295224e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 --2- 172.21.15.116:0/2026750052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56295223d2d0 0x56295224e2d0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 >> 172.21.15.116:0/2026750052 conn(0x562952237a00 msgr2=0x562952236ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 shutdown_connections 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 1 -- 172.21.15.116:0/2026750052 wait complete. 2024-03-19T02:44:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eae20c740 20 client.20220 trim_cache size 0 max 16384 2024-03-19T02:44:03.085 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 msgr2=0x7f6e900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 0x7f6e900a4e30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0099e0 tx=0x7f6e9c009ef0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 msgr2=0x7f6ea4198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea002c660 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 shutdown_connections 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6e900a2970 0x7f6e900a4e30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.076+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ea41066b0 0x7f6ea41a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.077+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ea4101060 0x7f6ea4199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.077+0000 7f6eab626640 1 --2- 172.21.15.116:0/1895371278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea41006a0 0x7f6ea4198de0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.077+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 >> 172.21.15.116:0/1895371278 conn(0x7f6ea40778d0 msgr2=0x7f6ea4102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.077+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 shutdown_connections 2024-03-19T02:44:03.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.077+0000 7f6eab626640 1 -- 172.21.15.116:0/1895371278 wait complete. 2024-03-19T02:44:03.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.221+0000 7fe164532640 1 -- 172.21.15.116:0/4109295459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c100dd0 msgr2=0x7fe15c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.221+0000 7fe164532640 1 --2- 172.21.15.116:0/4109295459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c100dd0 0x7fe15c101250 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fe14c009930 tx=0x7fe14c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 -- 172.21.15.116:0/4109295459 shutdown_connections 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 --2- 172.21.15.116:0/4109295459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c101790 0x7fe15c10d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 --2- 172.21.15.116:0/4109295459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c100dd0 0x7fe15c101250 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 --2- 172.21.15.116:0/4109295459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c1068c0 0x7fe15c106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 -- 172.21.15.116:0/4109295459 >> 172.21.15.116:0/4109295459 conn(0x7fe15c076c50 msgr2=0x7fe15c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 -- 172.21.15.116:0/4109295459 shutdown_connections 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 -- 172.21.15.116:0/4109295459 wait complete. 2024-03-19T02:44:03.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 Processor -- start 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.222+0000 7fe164532640 1 -- start start 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 0x7fe15c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 0x7fe15c1a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c110bd0 con 0x7fe15c101790 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c110a50 con 0x7fe15c100dd0 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe164532640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c110d50 con 0x7fe15c1068c0 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe161acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 0x7fe15c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41688/0 (socket says 172.21.15.116:41688) 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 -- 172.21.15.116:0/579521376 learned_addr learned my addr 172.21.15.116:0/579521376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe162acf640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 0x7fe15c1a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 -- 172.21.15.116:0/579521376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 msgr2=0x7fe15c1a4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 0x7fe15c1a4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 -- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 msgr2=0x7fe15c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 0x7fe15c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 -- 172.21.15.116:0/579521376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15c1a4660 con 0x7fe15c100dd0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe162acf640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 0x7fe15c1a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe161acd640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 0x7fe15c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.223+0000 7fe1622ce640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe15800e9f0 tx=0x7fe15800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.224+0000 7fe1537fe640 1 -- 172.21.15.116:0/579521376 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15800cd80 con 0x7fe15c100dd0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.224+0000 7fe1537fe640 1 -- 172.21.15.116:0/579521376 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1580108e0 con 0x7fe15c100dd0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.224+0000 7fe1537fe640 1 -- 172.21.15.116:0/579521376 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158018920 con 0x7fe15c100dd0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.224+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe15c1a4830 con 0x7fe15c100dd0 2024-03-19T02:44:03.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.224+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe15c1a4da0 con 0x7fe15c100dd0 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.226+0000 7fe1537fe640 1 -- 172.21.15.116:0/579521376 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe158010a90 con 0x7fe15c100dd0 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 Processor -- start 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 -- start start 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 0x55c1e08d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08e16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1e08374b0 con 0x55c1e08d48a0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1e07dfd70 con 0x55c1e08d3f80 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.227+0000 7fe1671f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1e0833550 con 0x55c1e08e1be0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe162acf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 0x55c1e08d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe161acd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1537fe640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 0x7fe1480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08e16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe161acd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34696/0 (socket says 172.21.15.116:34696) 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe161acd640 1 -- 172.21.15.116:0/1264212336 learned_addr learned my addr 172.21.15.116:0/1264212336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe162acf640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 0x7fe1480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1537fe640 1 -- 172.21.15.116:0/579521376 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fe1580cf450 con 0x7fe15c100dd0 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 -- 172.21.15.116:0/1264212336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 msgr2=0x55c1e08e3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 -- 172.21.15.116:0/1264212336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 msgr2=0x55c1e08d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 0x55c1e08d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 -- 172.21.15.116:0/1264212336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e0802210 con 0x55c1e08d48a0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe161acd640 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe162acf640 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 0x55c1e08d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.228+0000 7fe1622ce640 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08e16a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe1580cc7c0 tx=0x7fe15801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efec3693ba39095f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe162acf640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 0x7fe1480a4eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1540049d0 tx=0x7fe154004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe150bfe640 1 -- 172.21.15.116:0/1264212336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15809b780 con 0x55c1e08d48a0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe150bfe640 1 -- 172.21.15.116:0/1264212336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe15809f820 con 0x55c1e08d48a0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe150bfe640 1 -- 172.21.15.116:0/1264212336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1580cf280 con 0x55c1e08d48a0 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe1671f5740 1 -- 172.21.15.116:0/1264212336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 msgr2=0x55c1e08e16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.229+0000 7fe1671f5740 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08e16a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe1580cc7c0 tx=0x7fe15801f000 comp rx=0 tx=0).stop 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 -- 172.21.15.116:0/1264212336 shutdown_connections 2024-03-19T02:44:03.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08e16a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 --2- 172.21.15.116:0/1264212336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d3f80 0x55c1e08d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 -- 172.21.15.116:0/1264212336 >> 172.21.15.116:0/1264212336 conn(0x55c1e08cea00 msgr2=0x55c1e08cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 -- 172.21.15.116:0/1264212336 shutdown_connections 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 -- 172.21.15.116:0/1264212336 wait complete. 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 -- start start 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 Processor -- start 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.230+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 0x55c1e08dec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c1e08e49c0 con 0x55c1e08d48a0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1e07f3310 con 0x55c1e08e1be0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c1e08e4b40 con 0x55c1e08d3f80 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 0x55c1e08dec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe161acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1671f5740 10 client.0 _ll_register_callbacks cb 0x55c1e07e72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41710/0 (socket says 172.21.15.116:41710) 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe161acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42998/0 (socket says 172.21.15.116:42998) 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 -- 172.21.15.116:0/4198341188 learned_addr learned my addr 172.21.15.116:0/4198341188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 -- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 msgr2=0x55c1e08dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 0x55c1e08dec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 -- 172.21.15.116:0/4198341188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 msgr2=0x55c1e08df190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 -- 172.21.15.116:0/4198341188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e054b3f0 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe1622ce640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 0x55c1e08dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe161acd640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe162acf640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe154010b80 tx=0x7fe154004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0c2f35e3aef60a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.231+0000 7fe145ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1540207f0 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fe154020990 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe15809f820 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe154020b90 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe1671f5740 10 client.29978 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.232+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c1e06157d0 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe154020d30 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe15401f040 con 0x55c1e08e1be0 2024-03-19T02:44:03.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 10 client.29978 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 10 client.29978 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 20 client.29978 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08f2070 0x55c1e08f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 4 client.29978 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe15809f820 con 0x55c1e08f2070 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.233+0000 7fe1622ce640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08f2070 0x55c1e08f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.234+0000 7fe145ffb640 10 client.29978 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.234+0000 7fe1622ce640 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08f2070 0x55c1e08f4460 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa0690caa2018e48 server_cookie=2b1164e0d9a6fb84 in_seq=0 out_seq=0 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe145ffb640 1 -- 172.21.15.116:0/4198341188 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe15809f820 con 0x55c1e08f2070 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe145ffb640 10 client.29978 handle_command_reply: tid=0 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 client.29978 shutdown 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 msgr2=0x55c1e08df6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe154010b80 tx=0x7fe154004a40 comp rx=0 tx=0).stop 2024-03-19T02:44:03.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 shutdown_connections 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08f2070 0x55c1e08f4460 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e1be0 0x55c1e08df6d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d48a0 0x55c1e08df190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 --2- 172.21.15.116:0/4198341188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d3f80 0x55c1e08dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 >> 172.21.15.116:0/4198341188 conn(0x55c1e08cea00 msgr2=0x55c1e08d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe145ffb640 0 client.29978 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 shutdown_connections 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 1 -- 172.21.15.116:0/4198341188 wait complete. 2024-03-19T02:44:03.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.235+0000 7fe1671f5740 20 client.29978 trim_cache size 0 max 16384 2024-03-19T02:44:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 Processor -- start 2024-03-19T02:44:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 -- start start 2024-03-19T02:44:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d48a0 0x55c1e08ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1aca80 con 0x55c1e08d3f80 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1ae5c0 con 0x55c1e08d48a0 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1671f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1ad150 con 0x55c1e08e1be0 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.254+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43004/0 (socket says 172.21.15.116:43004) 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 -- 172.21.15.116:0/2144425618 learned_addr learned my addr 172.21.15.116:0/2144425618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe162acf640 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe161acd640 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d48a0 0x55c1e08ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 -- 172.21.15.116:0/2144425618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 msgr2=0x55c1e08cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 -- 172.21.15.116:0/2144425618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d48a0 msgr2=0x55c1e08ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d48a0 0x55c1e08ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 -- 172.21.15.116:0/2144425618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e08dd1a0 con 0x55c1e08d3f80 2024-03-19T02:44:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe1622ce640 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe15809b480 tx=0x7fe15809b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16148339e114f511 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe147fff640 1 -- 172.21.15.116:0/2144425618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15809b780 con 0x55c1e08d3f80 2024-03-19T02:44:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe147fff640 1 -- 172.21.15.116:0/2144425618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe15809f820 con 0x55c1e08d3f80 2024-03-19T02:44:03.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.255+0000 7fe147fff640 1 -- 172.21.15.116:0/2144425618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15809abd0 con 0x55c1e08d3f80 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 -- 172.21.15.116:0/2144425618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 msgr2=0x55c1e08e1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe15809b480 tx=0x7fe15809b370 comp rx=0 tx=0).stop 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 -- 172.21.15.116:0/2144425618 shutdown_connections 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08e1be0 0x55c1e08cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08d48a0 0x55c1e08ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 --2- 172.21.15.116:0/2144425618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08d3f80 0x55c1e08e1380 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 -- 172.21.15.116:0/2144425618 >> 172.21.15.116:0/2144425618 conn(0x55c1e08cea00 msgr2=0x55c1e08cd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 -- 172.21.15.116:0/2144425618 shutdown_connections 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.256+0000 7fe1671f5740 1 -- 172.21.15.116:0/2144425618 wait complete. 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 -- start start 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 Processor -- start 2024-03-19T02:44:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 0x55c1e08ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e5980 0x55c1e08e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1adef0 con 0x55c1e08e1be0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1ad820 con 0x55c1e08e5980 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c1aaa80 con 0x55c1e08d48a0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe161acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 0x55c1e08ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe161acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43006/0 (socket says 172.21.15.116:43006) 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe161acd640 1 -- 172.21.15.116:0/3651827526 learned_addr learned my addr 172.21.15.116:0/3651827526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe162acf640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e5980 0x55c1e08e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1671f5740 10 client.0 _ll_register_callbacks cb 0x7fe154013480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 -- 172.21.15.116:0/3651827526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e5980 msgr2=0x55c1e08e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e5980 0x55c1e08e5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 -- 172.21.15.116:0/3651827526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 msgr2=0x55c1e08e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe1622ce640 1 -- 172.21.15.116:0/3651827526 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e09574c0 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.257+0000 7fe161acd640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe1622ce640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 0x55c1e08ea120 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55c1e08e0220 tx=0x7fe1580cf750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75fb02190ca3277d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158099a80 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fe1580cf160 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe15809f820 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe1580a9310 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe1671f5740 10 client.29953 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c1e08d52f0 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe15809f820 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.258+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe1580b4230 con 0x55c1e08d48a0 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 10 client.29953 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 10 client.29953 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:44:03.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 20 client.29953 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08d55e0 0x55c1e08d7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 4 client.29953 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1e08953e0 con 0x55c1e08d55e0 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe161acd640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08d55e0 0x55c1e08d7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe145ffb640 10 client.29953 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe161acd640 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08d55e0 0x55c1e08d7a00 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9fdd30e1e5f636da server_cookie=43132a26321cec62 in_seq=0 out_seq=0 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe145ffb640 1 -- 172.21.15.116:0/3651827526 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1e08953e0 con 0x55c1e08d55e0 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe145ffb640 10 client.29953 handle_command_reply: tid=0 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.259+0000 7fe1671f5740 1 client.29953 shutdown 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 msgr2=0x55c1e08ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 0x55c1e08ea120 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55c1e08e0220 tx=0x7fe1580cf750 comp rx=0 tx=0).stop 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 shutdown_connections 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55c1e08d55e0 0x55c1e08d7a00 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1e08e5980 0x55c1e08e5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe145ffb640 0 client.29953 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c1e08e1be0 0x55c1e08e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 --2- 172.21.15.116:0/3651827526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c1e08d48a0 0x55c1e08ea120 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 >> 172.21.15.116:0/3651827526 conn(0x55c1e08cea00 msgr2=0x55c1e08df4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 shutdown_connections 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 1 -- 172.21.15.116:0/3651827526 wait complete. 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.260+0000 7fe1671f5740 20 client.29953 trim_cache size 0 max 16384 2024-03-19T02:44:03.267 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 msgr2=0x7fe1480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 0x7fe1480a4eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1540049d0 tx=0x7fe154004ab0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 msgr2=0x7fe15c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe15800e9f0 tx=0x7fe15800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 shutdown_connections 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1480a29f0 0x7fe1480a4eb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe15c1068c0 0x7fe15c1a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe15c101790 0x7fe15c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 --2- 172.21.15.116:0/579521376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe15c100dd0 0x7fe15c1990a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 >> 172.21.15.116:0/579521376 conn(0x7fe15c076c50 msgr2=0x7fe15c103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 shutdown_connections 2024-03-19T02:44:03.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.261+0000 7fe164532640 1 -- 172.21.15.116:0/579521376 wait complete. 2024-03-19T02:44:03.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:44:03.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23631 6.21 MB/sec execute 85 sec latency 143.702 ms 2024-03-19T02:44:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 -- 172.21.15.116:0/83293930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 msgr2=0x7f16a0100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 --2- 172.21.15.116:0/83293930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0100b30 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f1694009930 tx=0x7f169402bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 -- 172.21.15.116:0/83293930 shutdown_connections 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 --2- 172.21.15.116:0/83293930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 --2- 172.21.15.116:0/83293930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0100b30 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 --2- 172.21.15.116:0/83293930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 0x7f16a0106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 -- 172.21.15.116:0/83293930 >> 172.21.15.116:0/83293930 conn(0x7f16a0077810 msgr2=0x7f16a0077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 -- 172.21.15.116:0/83293930 shutdown_connections 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.407+0000 7f16a4881640 1 -- 172.21.15.116:0/83293930 wait complete. 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 Processor -- start 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 -- start start 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a01973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 0x7f16a019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0110980 con 0x7f16a01006b0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0110800 con 0x7f16a0101070 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.408+0000 7f16a4881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0110b00 con 0x7f16a01066c0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43016/0 (socket says 172.21.15.116:43016) 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169dd9b640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a01973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 -- 172.21.15.116:0/603246663 learned_addr learned my addr 172.21.15.116:0/603246663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169ed9d640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 0x7f16a019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 -- 172.21.15.116:0/603246663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 msgr2=0x7f16a019b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 0x7f16a019b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 -- 172.21.15.116:0/603246663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 msgr2=0x7f16a01973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a01973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 -- 172.21.15.116:0/603246663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a019be50 con 0x7f16a01006b0 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169dd9b640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a01973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f169e59c640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f169000e9f0 tx=0x7f169000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f168f7fe640 1 -- 172.21.15.116:0/603246663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169000cd80 con 0x7f16a01006b0 2024-03-19T02:44:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f168f7fe640 1 -- 172.21.15.116:0/603246663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16900054d0 con 0x7f16a01006b0 2024-03-19T02:44:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a019c140 con 0x7f16a01006b0 2024-03-19T02:44:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f168f7fe640 1 -- 172.21.15.116:0/603246663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690010490 con 0x7f16a01006b0 2024-03-19T02:44:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.409+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a01a39d0 con 0x7f16a01006b0 2024-03-19T02:44:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.412+0000 7f168f7fe640 1 -- 172.21.15.116:0/603246663 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1690005020 con 0x7f16a01006b0 2024-03-19T02:44:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.412+0000 7f16a746f740 1 Processor -- start 2024-03-19T02:44:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.412+0000 7f16a746f740 1 -- start start 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acadc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acae7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1aca3f4b0 con 0x55e1acadcbf0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e1ac9e7d70 con 0x55e1acadc2d0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f16a746f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1aca3b550 con 0x55e1acae57d0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acadc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43030/0 (socket says 172.21.15.116:43030) 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169e59c640 1 -- 172.21.15.116:0/2920079404 learned_addr learned my addr 172.21.15.116:0/2920079404 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169dd9b640 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acae7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f168f7fe640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 0x7f16880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 -- 172.21.15.116:0/2920079404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 msgr2=0x55e1acae7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acae7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 -- 172.21.15.116:0/2920079404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 msgr2=0x55e1acae5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169ed9d640 1 -- 172.21.15.116:0/2920079404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1aca0a210 con 0x55e1acadc2d0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f169e59c640 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.413+0000 7f168f7fe640 1 -- 172.21.15.116:0/603246663 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f16900cf4c0 con 0x7f16a01006b0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f169ed9d640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 0x7f16880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f169ed9d640 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acadc6b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1698009fd0 tx=0x7f1698011920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b569d69f196e2e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f169ed9d640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 0x7f16880a4eb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1698022000 tx=0x7f1698023000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f168cff9640 1 -- 172.21.15.116:0/2920079404 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698017070 con 0x55e1acadc2d0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f168cff9640 1 -- 172.21.15.116:0/2920079404 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f169800fbf0 con 0x55e1acadc2d0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.414+0000 7f168cff9640 1 -- 172.21.15.116:0/2920079404 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800ea10 con 0x55e1acadc2d0 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- 172.21.15.116:0/2920079404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 msgr2=0x55e1acadc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acadc6b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1698009fd0 tx=0x7f1698011920 comp rx=0 tx=0).stop 2024-03-19T02:44:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- 172.21.15.116:0/2920079404 shutdown_connections 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acae7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadcbf0 0x55e1acae5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 --2- 172.21.15.116:0/2920079404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acadc6b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- 172.21.15.116:0/2920079404 >> 172.21.15.116:0/2920079404 conn(0x55e1acad6a00 msgr2=0x55e1acad6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- 172.21.15.116:0/2920079404 shutdown_connections 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- 172.21.15.116:0/2920079404 wait complete. 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 -- start start 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.415+0000 7f16a746f740 1 Processor -- start 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acae2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1acae85c0 con 0x55e1acae57d0 2024-03-19T02:44:03.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e1ac9fb310 con 0x55e1acadc2d0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1acae8740 con 0x55e1acadcbf0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 10 client.0 _ll_register_callbacks cb 0x55e1ac9ef2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43040/0 (socket says 172.21.15.116:43040) 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 -- 172.21.15.116:0/1824626718 learned_addr learned my addr 172.21.15.116:0/1824626718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169dd9b640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169e59c640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acae2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 -- 172.21.15.116:0/1824626718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 msgr2=0x55e1acae2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 -- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 msgr2=0x55e1acae2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acae2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 -- 172.21.15.116:0/1824626718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ac7533f0 con 0x55e1acae57d0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f1698009700 tx=0x7f1698009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72cf6425505adbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f1685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:03.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698017070 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f169800e790 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.416+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f169800fbf0 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f169803a050 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 10 client.20250 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e1ac81d7d0 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f169800fbf0 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1698028790 con 0x55e1acae57d0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 10 client.20250 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 10 client.20250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 20 client.20250 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acaf5a00 0x55e1acaf7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 4 client.20250 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1aca9d3e0 con 0x55e1acaf5a00 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.417+0000 7f169e59c640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acaf5a00 0x55e1acaf7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.418+0000 7f1685ffb640 10 client.20250 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.418+0000 7f169e59c640 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acaf5a00 0x55e1acaf7e30 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c55f9dd97b6d4851 server_cookie=a4b89250903c32ec in_seq=0 out_seq=0 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f1685ffb640 1 -- 172.21.15.116:0/1824626718 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e1aca9d3e0 con 0x55e1acaf5a00 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f1685ffb640 10 client.20250 handle_command_reply: tid=0 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 client.20250 shutdown 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 msgr2=0x55e1acae3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f1698009700 tx=0x7f1698009c30 comp rx=0 tx=0).stop 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 shutdown_connections 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acaf5a00 0x55e1acaf7e30 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae3290 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 --2- 172.21.15.116:0/1824626718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acae2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 >> 172.21.15.116:0/1824626718 conn(0x55e1acad6a00 msgr2=0x55e1acad8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f1685ffb640 0 client.20250 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:03.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 shutdown_connections 2024-03-19T02:44:03.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.419+0000 7f16a746f740 1 -- 172.21.15.116:0/1824626718 wait complete. 2024-03-19T02:44:03.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.420+0000 7f16a746f740 20 client.20250 trim_cache size 0 max 16384 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 Processor -- start 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 -- start start 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 0x55e1acaf1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01aca60 con 0x55e1acadc2d0 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01ae5a0 con 0x55e1acadcbf0 2024-03-19T02:44:03.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f16a746f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01ad130 con 0x55e1acae57d0 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 0x55e1acaf1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43054/0 (socket says 172.21.15.116:43054) 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 0x55e1acaf1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41774/0 (socket says 172.21.15.116:41774) 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34766/0 (socket says 172.21.15.116:34766) 2024-03-19T02:44:03.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.438+0000 7f169e59c640 1 -- 172.21.15.116:0/1205987150 learned_addr learned my addr 172.21.15.116:0/1205987150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 -- 172.21.15.116:0/1205987150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 msgr2=0x55e1acaf1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 0x55e1acaf1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 -- 172.21.15.116:0/1205987150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 msgr2=0x55e1acae4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 -- 172.21.15.116:0/1205987150 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1acadca20 con 0x55e1acae57d0 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169e59c640 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f169ed9d640 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f169800ffd0 tx=0x7f1698011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b716cc639730c841 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f1687fff640 1 -- 172.21.15.116:0/1205987150 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698037020 con 0x55e1acae57d0 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f1687fff640 1 -- 172.21.15.116:0/1205987150 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1698009e50 con 0x55e1acae57d0 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.439+0000 7f1687fff640 1 -- 172.21.15.116:0/1205987150 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698017040 con 0x55e1acae57d0 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 -- 172.21.15.116:0/1205987150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 msgr2=0x55e1acafd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f169800ffd0 tx=0x7f1698011920 comp rx=0 tx=0).stop 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 -- 172.21.15.116:0/1205987150 shutdown_connections 2024-03-19T02:44:03.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acae57d0 0x55e1acafd700 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadcbf0 0x55e1acaf1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 --2- 172.21.15.116:0/1205987150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acadc2d0 0x55e1acae4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 -- 172.21.15.116:0/1205987150 >> 172.21.15.116:0/1205987150 conn(0x55e1acad6a00 msgr2=0x55e1acae1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 -- 172.21.15.116:0/1205987150 shutdown_connections 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.440+0000 7f16a746f740 1 -- 172.21.15.116:0/1205987150 wait complete. 2024-03-19T02:44:03.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f16a746f740 1 -- start start 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f16a746f740 1 Processor -- start 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f16a746f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acaeda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f16a746f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f16a746f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.441+0000 7f169dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43066/0 (socket says 172.21.15.116:43066) 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34780/0 (socket says 172.21.15.116:34780) 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169ed9d640 1 -- 172.21.15.116:0/4236399595 learned_addr learned my addr 172.21.15.116:0/4236399595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01aded0 con 0x55e1acae57d0 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01ad800 con 0x55e1acadc2d0 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16a01aaa60 con 0x55e1acadcbf0 2024-03-19T02:44:03.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f16a746f740 10 client.0 _ll_register_callbacks cb 0x7f16900d2ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 -- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 msgr2=0x55e1acaeda40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acaeda40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 -- 172.21.15.116:0/4236399595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 msgr2=0x55e1acae9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.442+0000 7f169dd9b640 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1acae9a20 con 0x55e1acadcbf0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f169dd9b640 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1694009230 tx=0x7f169402c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7453d9ad88a27438 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f1685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940035f0 con 0x55e1acadcbf0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7f1694003790 con 0x55e1acadcbf0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1698009e50 con 0x55e1acadcbf0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.443+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1694003990 con 0x55e1acadcbf0 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 10 client.29971 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:03.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e1acb5baf0 con 0x55e1acadcbf0 2024-03-19T02:44:03.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1698009e50 con 0x55e1acadcbf0 2024-03-19T02:44:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1694054480 con 0x55e1acadcbf0 2024-03-19T02:44:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 10 client.29971 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 10 client.29971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 20 client.29971 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:03.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acb5c2a0 0x55e1acb5e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 4 client.29971 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1aca4c9f0 con 0x55e1acb5c2a0 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.444+0000 7f169e59c640 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acb5c2a0 0x55e1acb5e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f1685ffb640 10 client.29971 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f169e59c640 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acb5c2a0 0x55e1acb5e6c0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53914b31017b2b7b server_cookie=69a31bae0051bee6 in_seq=0 out_seq=0 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f1685ffb640 1 -- 172.21.15.116:0/4236399595 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1aca4c9f0 con 0x55e1acb5c2a0 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f1685ffb640 10 client.29971 handle_command_reply: tid=0 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 client.29971 shutdown 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 msgr2=0x55e1acae8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1694009230 tx=0x7f169402c730 comp rx=0 tx=0).stop 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 shutdown_connections 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1acb5c2a0 0x55e1acb5e6c0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f1685ffb640 0 client.29971 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1acae57d0 0x55e1acae9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1acadcbf0 0x55e1acae8ef0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 --2- 172.21.15.116:0/4236399595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1acadc2d0 0x55e1acaeda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 >> 172.21.15.116:0/4236399595 conn(0x55e1acad6a00 msgr2=0x55e1acad5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.445+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 shutdown_connections 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a746f740 1 -- 172.21.15.116:0/4236399595 wait complete. 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a746f740 20 client.29971 trim_cache size 0 max 16384 2024-03-19T02:44:03.450 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 msgr2=0x7f16880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 0x7f16880a4eb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1698022000 tx=0x7f1698023000 comp rx=0 tx=0).stop 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 msgr2=0x7f16a0196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f169000e9f0 tx=0x7f169000edf0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 shutdown_connections 2024-03-19T02:44:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16880a29f0 0x7f16880a4eb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16a01066c0 0x7f16a019b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a0101070 0x7f16a01973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 --2- 172.21.15.116:0/603246663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16a01006b0 0x7f16a0196e60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 >> 172.21.15.116:0/603246663 conn(0x7f16a0077810 msgr2=0x7f16a01032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 shutdown_connections 2024-03-19T02:44:03.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:03.446+0000 7f16a4881640 1 -- 172.21.15.116:0/603246663 wait complete. 2024-03-19T02:44:03.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:03 smithi162 ceph-mon[35761]: pgmap v510: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-03-19T02:44:04.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:03 smithi154 ceph-mon[35693]: pgmap v510: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-03-19T02:44:04.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:03 smithi116 ceph-mon[32263]: pgmap v510: 129 pgs: 129 active+clean; 855 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-03-19T02:44:04.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23754 6.15 MB/sec execute 86 sec latency 144.853 ms 2024-03-19T02:44:04.462 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:44:04.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:44:04.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:44:04.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.591+0000 7fd88e64b640 1 -- 172.21.15.116:0/3116279176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101050 msgr2=0x7fd88810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.591+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3116279176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101050 0x7fd88810d560 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fd880009cd0 tx=0x7fd88002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 -- 172.21.15.116:0/3116279176 shutdown_connections 2024-03-19T02:44:04.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3116279176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101050 0x7fd88810d560 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3116279176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888100690 0x7fd888100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3116279176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8881066a0 0x7fd888106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 -- 172.21.15.116:0/3116279176 >> 172.21.15.116:0/3116279176 conn(0x7fd888077820 msgr2=0x7fd888077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 -- 172.21.15.116:0/3116279176 shutdown_connections 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 -- 172.21.15.116:0/3116279176 wait complete. 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 Processor -- start 2024-03-19T02:44:04.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.592+0000 7fd88e64b640 1 -- start start 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888101050 0x7fd8881985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 0x7fd88819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd888110960 con 0x7fd8881066a0 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881107e0 con 0x7fd888101050 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88e64b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd888110ae0 con 0x7fd888100690 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd887fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 0x7fd88819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd887fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34804/0 (socket says 172.21.15.116:34804) 2024-03-19T02:44:04.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd887fff640 1 -- 172.21.15.116:0/3214939414 learned_addr learned my addr 172.21.15.116:0/3214939414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888101050 0x7fd8881985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 -- 172.21.15.116:0/3214939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 msgr2=0x7fd888198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 -- 172.21.15.116:0/3214939414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888101050 msgr2=0x7fd8881985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888101050 0x7fd8881985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 -- 172.21.15.116:0/3214939414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88819d010 con 0x7fd8881066a0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd887fff640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.593+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 0x7fd88819c8e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fd88002c630 tx=0x7fd880035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.594+0000 7fd885ffb640 1 -- 172.21.15.116:0/3214939414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880039040 con 0x7fd8881066a0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.594+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88819d2a0 con 0x7fd8881066a0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.594+0000 7fd885ffb640 1 -- 172.21.15.116:0/3214939414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd880044070 con 0x7fd8881066a0 2024-03-19T02:44:04.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.594+0000 7fd885ffb640 1 -- 172.21.15.116:0/3214939414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880004330 con 0x7fd8881066a0 2024-03-19T02:44:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.594+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8881a4b90 con 0x7fd8881066a0 2024-03-19T02:44:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.597+0000 7fd885ffb640 1 -- 172.21.15.116:0/3214939414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd8800044d0 con 0x7fd8881066a0 2024-03-19T02:44:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.597+0000 7fd89123a740 1 Processor -- start 2024-03-19T02:44:04.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 -- start start 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895045340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a894fa0160 con 0x55a89503cca0 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a894f48e10 con 0x55a89503c380 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd89123a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a894f9c200 con 0x55a895045880 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34816/0 (socket says 172.21.15.116:34816) 2024-03-19T02:44:04.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd87f7fe640 1 -- 172.21.15.116:0/2586498341 learned_addr learned my addr 172.21.15.116:0/2586498341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.598+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd887fff640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895045340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 -- 172.21.15.116:0/2586498341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 msgr2=0x55a89503c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 -- 172.21.15.116:0/2586498341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 msgr2=0x55a895045340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895045340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 -- 172.21.15.116:0/2586498341 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a894d7c7e0 con 0x55a895045880 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd887fff640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895045340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd885ffb640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 0x7fd8740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd887fff640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 0x7fd8740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd885ffb640 1 -- 172.21.15.116:0/3214939414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9510+0+0 (secure 0 0 0) 0x7fd8800f15c0 con 0x7fd8881066a0 2024-03-19T02:44:04.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.599+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd8700058c0 tx=0x7fd870005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f28bbd739e7af89 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.600+0000 7fd87e7fc640 1 -- 172.21.15.116:0/2586498341 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870004030 con 0x55a895045880 2024-03-19T02:44:04.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.600+0000 7fd887fff640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 0x7fd8740a4e30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55a89503be50 tx=0x7fd87800b410 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.600+0000 7fd87e7fc640 1 -- 172.21.15.116:0/2586498341 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd870005020 con 0x55a895045880 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.600+0000 7fd87e7fc640 1 -- 172.21.15.116:0/2586498341 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870004c60 con 0x55a895045880 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 -- 172.21.15.116:0/2586498341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 msgr2=0x55a895047c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd8700058c0 tx=0x7fd870005850 comp rx=0 tx=0).stop 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 -- 172.21.15.116:0/2586498341 shutdown_connections 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895047c70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895045340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 --2- 172.21.15.116:0/2586498341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 -- 172.21.15.116:0/2586498341 >> 172.21.15.116:0/2586498341 conn(0x55a895036ab0 msgr2=0x55a895036e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 -- 172.21.15.116:0/2586498341 shutdown_connections 2024-03-19T02:44:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.601+0000 7fd89123a740 1 -- 172.21.15.116:0/2586498341 wait complete. 2024-03-19T02:44:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.602+0000 7fd89123a740 1 -- start start 2024-03-19T02:44:04.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 Processor -- start 2024-03-19T02:44:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a89503f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a895048480 con 0x55a89503cca0 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a894f5c1a0 con 0x55a89503c380 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a895048600 con 0x55a895045880 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.604+0000 7fd89123a740 10 client.0 _ll_register_callbacks cb 0x55a894cb3c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.605+0000 7fd88cbe8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.605+0000 7fd88cbe8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34828/0 (socket says 172.21.15.116:34828) 2024-03-19T02:44:04.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.605+0000 7fd88cbe8640 1 -- 172.21.15.116:0/2091958512 learned_addr learned my addr 172.21.15.116:0/2091958512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.606+0000 7fd887fff640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd88cbe8640 1 -- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 msgr2=0x55a89503eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd88cbe8640 1 -- 172.21.15.116:0/2091958512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 msgr2=0x55a89503f000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:44:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a89503f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a89503f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd88cbe8640 1 -- 172.21.15.116:0/2091958512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a894ddcf70 con 0x55a895045880 2024-03-19T02:44:04.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.607+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a89503f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8800ba470 tx=0x7fd8800bd5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f92d0eaa162739 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd8677fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800c60f0 con 0x55a895045880 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd870005020 con 0x55a895045880 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9598+0+0 (secure 0 0 0) 0x7fd8800ba110 con 0x55a895045880 2024-03-19T02:44:04.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.608+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd8800f1190 con 0x55a895045880 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.609+0000 7fd89123a740 10 client.29983 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.609+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a894c74170 con 0x55a895045880 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd870005020 con 0x55a895045880 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd88003c030 con 0x55a895045880 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 10 client.29983 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 10 client.29983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:04.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 20 client.29983 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:04.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a89506f3e0 0x55a895071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 4 client.29983 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:04.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.610+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a894fad6a0 con 0x55a89506f3e0 2024-03-19T02:44:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.613+0000 7fd887fff640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a89506f3e0 0x55a895071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.618+0000 7fd887fff640 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a89506f3e0 0x55a895071830 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbca645dd9e0f195 server_cookie=acda7934432f6386 in_seq=0 out_seq=0 2024-03-19T02:44:04.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.618+0000 7fd8677fe640 10 client.29983 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.623+0000 7fd8677fe640 1 -- 172.21.15.116:0/2091958512 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a894fad6a0 con 0x55a89506f3e0 2024-03-19T02:44:04.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.623+0000 7fd8677fe640 10 client.29983 handle_command_reply: tid=0 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.623+0000 7fd89123a740 1 client.29983 shutdown 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 msgr2=0x55a895043280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8800ba470 tx=0x7fd8800bd5e0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 shutdown_connections 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a89506f3e0 0x55a895071830 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a895043280 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a89503f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 --2- 172.21.15.116:0/2091958512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a89503eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 >> 172.21.15.116:0/2091958512 conn(0x55a895036ab0 msgr2=0x55a895038800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.624+0000 7fd8677fe640 0 client.29983 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.627+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 shutdown_connections 2024-03-19T02:44:04.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.628+0000 7fd89123a740 1 -- 172.21.15.116:0/2091958512 wait complete. 2024-03-19T02:44:04.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.630+0000 7fd89123a740 20 client.29983 trim_cache size 0 max 16384 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 Processor -- start 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 -- start start 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a8950430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a89504b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd888073040 con 0x55a89503cca0 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881a5e90 con 0x55a89503c380 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.672+0000 7fd89123a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd888073710 con 0x55a895045880 2024-03-19T02:44:04.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.673+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.673+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43114/0 (socket says 172.21.15.116:43114) 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.673+0000 7fd87f7fe640 1 -- 172.21.15.116:0/1826988128 learned_addr learned my addr 172.21.15.116:0/1826988128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.673+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a89504b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 -- 172.21.15.116:0/1826988128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 msgr2=0x55a89504b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a89504b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 -- 172.21.15.116:0/1826988128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 msgr2=0x55a8950430e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a8950430e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 -- 172.21.15.116:0/1826988128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8950372f0 con 0x55a89503cca0 2024-03-19T02:44:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.674+0000 7fd87f7fe640 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fd87001c000 tx=0x7fd8700051b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4426760961b097c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.675+0000 7fd87dffb640 1 -- 172.21.15.116:0/1826988128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700055d0 con 0x55a89503cca0 2024-03-19T02:44:04.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.675+0000 7fd87dffb640 1 -- 172.21.15.116:0/1826988128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8700114c0 con 0x55a89503cca0 2024-03-19T02:44:04.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.675+0000 7fd87dffb640 1 -- 172.21.15.116:0/1826988128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870003b90 con 0x55a89503cca0 2024-03-19T02:44:04.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.676+0000 7fd89123a740 1 -- 172.21.15.116:0/1826988128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 msgr2=0x55a895077100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.676+0000 7fd89123a740 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fd87001c000 tx=0x7fd8700051b0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.680+0000 7fd89123a740 1 -- 172.21.15.116:0/1826988128 shutdown_connections 2024-03-19T02:44:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.680+0000 7fd89123a740 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a895045880 0x55a89504b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.680+0000 7fd89123a740 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a89503cca0 0x55a895077100 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.680+0000 7fd89123a740 1 --2- 172.21.15.116:0/1826988128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a8950430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.680+0000 7fd89123a740 1 -- 172.21.15.116:0/1826988128 >> 172.21.15.116:0/1826988128 conn(0x55a895036ab0 msgr2=0x55a89503d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.682+0000 7fd89123a740 1 -- 172.21.15.116:0/1826988128 shutdown_connections 2024-03-19T02:44:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.682+0000 7fd89123a740 1 -- 172.21.15.116:0/1826988128 wait complete. 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 -- start start 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 Processor -- start 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a895052b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a89503cca0 0x55a895053080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881a57c0 con 0x55a895045880 2024-03-19T02:44:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881a50f0 con 0x55a89503c380 2024-03-19T02:44:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.683+0000 7fd89123a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd888071040 con 0x55a89503cca0 2024-03-19T02:44:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.684+0000 7fd89123a740 10 client.0 _ll_register_callbacks cb 0x7fd880009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.684+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a89503cca0 0x55a895053080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.684+0000 7fd88cbe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.684+0000 7fd88cbe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43118/0 (socket says 172.21.15.116:43118) 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.684+0000 7fd88cbe8640 1 -- 172.21.15.116:0/1176579201 learned_addr learned my addr 172.21.15.116:0/1176579201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 -- 172.21.15.116:0/1176579201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a89503cca0 msgr2=0x55a895053080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a89503cca0 0x55a895053080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 -- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 msgr2=0x55a895052b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a895052b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 -- 172.21.15.116:0/1176579201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a89504df40 con 0x55a895045880 2024-03-19T02:44:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd88cbe8640 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd8800bd7a0 tx=0x7fd8800bcfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eecccc35d3bfedb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.685+0000 7fd8677fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.687+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800c60f0 con 0x55a895045880 2024-03-19T02:44:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.688+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8700114c0 con 0x55a895045880 2024-03-19T02:44:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.688+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9598+0+0 (secure 0 0 0) 0x7fd8800ba110 con 0x55a895045880 2024-03-19T02:44:04.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.689+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd8800c5c60 con 0x55a895045880 2024-03-19T02:44:04.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.689+0000 7fd89123a740 10 client.20271 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:04.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.689+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a89504df40 con 0x55a895045880 2024-03-19T02:44:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.690+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8700114c0 con 0x55a895045880 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd8800b97f0 con 0x55a895045880 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 10 client.20271 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 10 client.20271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 20 client.20271 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a8950cdd90 0x55a8950d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 4 client.20271 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:44:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.691+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a894fa79f0 con 0x55a8950cdd90 2024-03-19T02:44:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.692+0000 7fd887fff640 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a8950cdd90 0x55a8950d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.693+0000 7fd887fff640 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a8950cdd90 0x55a8950d0180 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb638cfa1beec3e9 server_cookie=b846452139b2ea65 in_seq=0 out_seq=0 2024-03-19T02:44:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.693+0000 7fd8677fe640 10 client.20271 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd8677fe640 1 -- 172.21.15.116:0/1176579201 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a894fa79f0 con 0x55a8950cdd90 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd8677fe640 10 client.20271 handle_command_reply: tid=0 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 client.20271 shutdown 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 msgr2=0x55a89505a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd8800bd7a0 tx=0x7fd8800bcfe0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 shutdown_connections 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a8950cdd90 0x55a8950d0180 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a895045880 0x55a89505a870 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a89503cca0 0x55a895053080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 --2- 172.21.15.116:0/1176579201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a89503c380 0x55a895052b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 >> 172.21.15.116:0/1176579201 conn(0x55a895036ab0 msgr2=0x55a8950352a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.696+0000 7fd8677fe640 0 client.20271 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.697+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 shutdown_connections 2024-03-19T02:44:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.697+0000 7fd89123a740 1 -- 172.21.15.116:0/1176579201 wait complete. 2024-03-19T02:44:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.697+0000 7fd89123a740 20 client.20271 trim_cache size 0 max 16384 2024-03-19T02:44:04.699 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:04.699 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:44:04.699 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "68953cb6-eb40-4f84-8bff-a1f95510e936", 2024-03-19T02:44:04.699 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:44:04.699 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 msgr2=0x7fd8740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 0x7fd8740a4e30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55a89503be50 tx=0x7fd87800b410 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 msgr2=0x7fd88819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 0x7fd88819c8e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fd88002c630 tx=0x7fd880035d60 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 shutdown_connections 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd8740a2970 0x7fd8740a4e30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8881066a0 0x7fd88819c8e0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd888101050 0x7fd8881985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 --2- 172.21.15.116:0/3214939414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd888100690 0x7fd888198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 >> 172.21.15.116:0/3214939414 conn(0x7fd888077820 msgr2=0x7fd888068e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 shutdown_connections 2024-03-19T02:44:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.698+0000 7fd88e64b640 1 -- 172.21.15.116:0/3214939414 wait complete. 2024-03-19T02:44:04.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:44:04.710 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:44:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 -- 172.21.15.116:0/3129929962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc110102920 msgr2=0x7fc110102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 --2- 172.21.15.116:0/3129929962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc110102920 0x7fc110102d00 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fc100009900 tx=0x7fc10002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 -- 172.21.15.116:0/3129929962 shutdown_connections 2024-03-19T02:44:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 --2- 172.21.15.116:0/3129929962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc11010bcf0 0x7fc11010e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 --2- 172.21.15.116:0/3129929962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 0x7fc11010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 --2- 172.21.15.116:0/3129929962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc110102920 0x7fc110102d00 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 -- 172.21.15.116:0/3129929962 >> 172.21.15.116:0/3129929962 conn(0x7fc1100fc0a0 msgr2=0x7fc1100fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 -- 172.21.15.116:0/3129929962 shutdown_connections 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.840+0000 7fc1157ff640 1 -- 172.21.15.116:0/3129929962 wait complete. 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 Processor -- start 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 -- start start 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc110102920 0x7fc110197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 0x7fc110198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc110111940 con 0x7fc11010bcf0 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101117c0 con 0x7fc110102920 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc1157ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc110111ac0 con 0x7fc110103240 2024-03-19T02:44:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc10f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc10f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43136/0 (socket says 172.21.15.116:43136) 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc10effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc110102920 0x7fc110197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc10f7fe640 1 -- 172.21.15.116:0/2304764409 learned_addr learned my addr 172.21.15.116:0/2304764409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.841+0000 7fc107fff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 0x7fc110198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 -- 172.21.15.116:0/2304764409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 msgr2=0x7fc110198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 0x7fc110198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 -- 172.21.15.116:0/2304764409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc110102920 msgr2=0x7fc110197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc110102920 0x7fc110197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 -- 172.21.15.116:0/2304764409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11019ced0 con 0x7fc11010bcf0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fc10800eb10 tx=0x7fc10800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10cff9640 1 -- 172.21.15.116:0/2304764409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108004b20 con 0x7fc11010bcf0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10cff9640 1 -- 172.21.15.116:0/2304764409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc108004cc0 con 0x7fc11010bcf0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc11019d1c0 con 0x7fc11010bcf0 2024-03-19T02:44:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc10cff9640 1 -- 172.21.15.116:0/2304764409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108010690 con 0x7fc11010bcf0 2024-03-19T02:44:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.842+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1101a4a50 con 0x7fc11010bcf0 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc10cff9640 1 -- 172.21.15.116:0/2304764409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc108005020 con 0x7fc11010bcf0 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc1183eb740 1 Processor -- start 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc1183eb740 1 -- start start 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc10cff9640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 0x7fc0fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc107fff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 0x7fc0fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc10cff9640 1 -- 172.21.15.116:0/2304764409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9598+0+0 (secure 0 0 0) 0x7fc1080cf7e0 con 0x7fc11010bcf0 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.845+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28f40f0 0x5624c28f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1183eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624c284e500 con 0x5624c28f40f0 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1183eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5624c27f6be0 con 0x5624c28eb510 2024-03-19T02:44:04.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1183eb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5624c284a5a0 con 0x5624c28eabf0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28f40f0 0x5624c28f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc107fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34886/0 (socket says 172.21.15.116:34886) 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc107fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:41856/0 (socket says 172.21.15.116:41856) 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 -- 172.21.15.116:0/3831701803 learned_addr learned my addr 172.21.15.116:0/3831701803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 -- 172.21.15.116:0/3831701803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 msgr2=0x5624c28f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 -- 172.21.15.116:0/3831701803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28f40f0 msgr2=0x5624c28f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28f40f0 0x5624c28f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc107fff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 0x7fc0fc0a4e30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1100fe7a0 tx=0x7fc0f8011920 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 -- 172.21.15.116:0/3831701803 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c2818000 con 0x5624c28eabf0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc10effd640 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc100009900 tx=0x7fc1000095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c926e38b0bc6d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.846+0000 7fc1057fa640 1 -- 172.21.15.116:0/3831701803 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100004040 con 0x5624c28eabf0 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1057fa640 1 -- 172.21.15.116:0/3831701803 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1000054f0 con 0x5624c28eabf0 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1057fa640 1 -- 172.21.15.116:0/3831701803 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10002ce20 con 0x5624c28eabf0 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 -- 172.21.15.116:0/3831701803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 msgr2=0x5624c28eafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc100009900 tx=0x7fc1000095f0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 -- 172.21.15.116:0/3831701803 shutdown_connections 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28f40f0 0x5624c28f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3831701803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28eafd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 -- 172.21.15.116:0/3831701803 >> 172.21.15.116:0/3831701803 conn(0x5624c28e5320 msgr2=0x5624c28e5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 -- 172.21.15.116:0/3831701803 shutdown_connections 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.847+0000 7fc1183eb740 1 -- 172.21.15.116:0/3831701803 wait complete. 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 -- start start 2024-03-19T02:44:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 Processor -- start 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28f40f0 0x5624c28f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624c28f6e80 con 0x5624c28eb510 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5624c280a160 con 0x5624c28f40f0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5624c28f7000 con 0x5624c28eabf0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc1183eb740 10 client.0 _ll_register_callbacks cb 0x5624c2846a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc10effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43152/0 (socket says 172.21.15.116:43152) 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 -- 172.21.15.116:0/1836209143 learned_addr learned my addr 172.21.15.116:0/1836209143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28f40f0 0x5624c28f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 -- 172.21.15.116:0/1836209143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 msgr2=0x5624c28f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 -- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28f40f0 msgr2=0x5624c28f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28f40f0 0x5624c28f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc107fff640 1 -- 172.21.15.116:0/1836209143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c262c7d0 con 0x5624c28eb510 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.848+0000 7fc10effd640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc107fff640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009b10 tx=0x7fc0f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d1cbb953e6e44f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f801a040 con 0x5624c28eb510 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9598+0+0 (secure 0 0 0) 0x7fc0f8003ac0 con 0x5624c28eb510 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1000054f0 con 0x5624c28eb510 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc0f8003cc0 con 0x5624c28eb510 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 10 client.20283 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5624c25623f0 con 0x5624c28eb510 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1000054f0 con 0x5624c28eb510 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fc0f801d030 con 0x5624c28eb510 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 10 client.20283 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 10 client.20283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.849+0000 7fc1183eb740 20 client.20283 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c29067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc1183eb740 4 client.20283 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624c285ba40 con 0x5624c2904300 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc10effd640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c29067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc10effd640 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c29067c0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aac878c7f3c3fe2f server_cookie=b3116fef2f546cf4 in_seq=0 out_seq=0 2024-03-19T02:44:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.850+0000 7fc0f67fc640 10 client.20283 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc0f67fc640 1 -- 172.21.15.116:0/1836209143 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624c285ba40 con 0x5624c2904300 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc0f67fc640 10 client.20283 handle_command_reply: tid=0 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 client.20283 shutdown 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 msgr2=0x5624c28f1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009b10 tx=0x7fc0f8004730 comp rx=0 tx=0).stop 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 shutdown_connections 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c29067c0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc0f67fc640 0 client.20283 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28f40f0 0x5624c28f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28f1550 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 --2- 172.21.15.116:0/1836209143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28eabf0 0x5624c28f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 >> 172.21.15.116:0/1836209143 conn(0x5624c28e5320 msgr2=0x5624c28e7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 shutdown_connections 2024-03-19T02:44:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 1 -- 172.21.15.116:0/1836209143 wait complete. 2024-03-19T02:44:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.855+0000 7fc1183eb740 20 client.20283 trim_cache size 0 max 16384 2024-03-19T02:44:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.878+0000 7fc1183eb740 1 Processor -- start 2024-03-19T02:44:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.878+0000 7fc1183eb740 1 -- start start 2024-03-19T02:44:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eabf0 0x5624c28f3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28e4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101aeae0 con 0x5624c28eb510 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101b0620 con 0x5624c28eabf0 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc1183eb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101af1b0 con 0x5624c28f40f0 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc107fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc107fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43154/0 (socket says 172.21.15.116:43154) 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc107fff640 1 -- 172.21.15.116:0/603081329 learned_addr learned my addr 172.21.15.116:0/603081329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10effd640 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eabf0 0x5624c28f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 -- 172.21.15.116:0/603081329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eabf0 msgr2=0x5624c28f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eabf0 0x5624c28f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 -- 172.21.15.116:0/603081329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 msgr2=0x5624c28ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 -- 172.21.15.116:0/603081329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c25220f0 con 0x5624c28f40f0 2024-03-19T02:44:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc107fff640 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28e4320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc1080ccb00 tx=0x7fc108086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab1f508a7d687c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.879+0000 7fc104ff9640 1 -- 172.21.15.116:0/603081329 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10809fb60 con 0x5624c28f40f0 2024-03-19T02:44:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc104ff9640 1 -- 172.21.15.116:0/603081329 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1080a41b0 con 0x5624c28f40f0 2024-03-19T02:44:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc104ff9640 1 -- 172.21.15.116:0/603081329 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1080a3f80 con 0x5624c28f40f0 2024-03-19T02:44:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 -- 172.21.15.116:0/603081329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 msgr2=0x5624c28e4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28e4320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc1080ccb00 tx=0x7fc108086040 comp rx=0 tx=0).stop 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 -- 172.21.15.116:0/603081329 shutdown_connections 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28e4320 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eb510 0x5624c28ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 --2- 172.21.15.116:0/603081329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eabf0 0x5624c28f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 -- 172.21.15.116:0/603081329 >> 172.21.15.116:0/603081329 conn(0x5624c28e5320 msgr2=0x5624c28e3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 -- 172.21.15.116:0/603081329 shutdown_connections 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.880+0000 7fc1183eb740 1 -- 172.21.15.116:0/603081329 wait complete. 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 -- start start 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 Processor -- start 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101aff50 con 0x5624c28eabf0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101af880 con 0x5624c28eb510 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1101acae0 con 0x5624c28f40f0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc1183eb740 10 client.0 _ll_register_callbacks cb 0x7fc0f8009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43156/0 (socket says 172.21.15.116:43156) 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 -- 172.21.15.116:0/3478100131 learned_addr learned my addr 172.21.15.116:0/3478100131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10f7fe640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc107fff640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 -- 172.21.15.116:0/3478100131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 msgr2=0x5624c28f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 -- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 msgr2=0x5624c28f7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 -- 172.21.15.116:0/3478100131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c28f85c0 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.881+0000 7fc10effd640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fc10002c210 tx=0x7fc10002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264c3a3113d08fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100057060 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9598+0+0 (secure 0 0 0) 0x7fc100051480 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1080a41b0 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc10005e050 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 10 client.20289 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5624c296a6c0 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1080a41b0 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fc100004a00 con 0x5624c28eabf0 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 10 client.20289 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 10 client.20289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 20 client.20289 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:04.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c296ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 4 client.20289 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.882+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624c2855d90 con 0x5624c2904300 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.883+0000 7fc107fff640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c296ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.883+0000 7fc0f67fc640 10 client.20289 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.883+0000 7fc107fff640 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c296ca10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd286c66dd3f1c4 server_cookie=af0231aa360f19d8 in_seq=0 out_seq=0 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc0f67fc640 1 -- 172.21.15.116:0/3478100131 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5624c2855d90 con 0x5624c2904300 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc0f67fc640 10 client.20289 handle_command_reply: tid=0 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 client.20289 shutdown 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 msgr2=0x5624c28fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fc10002c210 tx=0x7fc10002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 shutdown_connections 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5624c2904300 0x5624c296ca10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.884+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624c28f40f0 0x5624c28f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624c28eb510 0x5624c28f7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc0f67fc640 0 client.20289 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 1 --2- 172.21.15.116:0/3478100131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624c28eabf0 0x5624c28fc4e0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 >> 172.21.15.116:0/3478100131 conn(0x5624c28e5320 msgr2=0x5624c28f1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 shutdown_connections 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 1 -- 172.21.15.116:0/3478100131 wait complete. 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1183eb740 20 client.20289 trim_cache size 0 max 16384 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (443 inodes in the stack)", 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stdout: "68953cb6-eb40-4f84-8bff-a1f95510e936": { 2024-03-19T02:44:04.891 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "68953cb6-eb40-4f84-8bff-a1f95510e936", 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 msgr2=0x7fc0fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 0x7fc0fc0a4e30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc1100fe7a0 tx=0x7fc0f8011920 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 msgr2=0x7fc11019c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fc10800eb10 tx=0x7fc10800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 shutdown_connections 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc0fc0a2970 0x7fc0fc0a4e30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc11010bcf0 0x7fc11019c7a0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc110103240 0x7fc110198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 --2- 172.21.15.116:0/2304764409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc110102920 0x7fc110197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 >> 172.21.15.116:0/2304764409 conn(0x7fc1100fc0a0 msgr2=0x7fc1100fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 shutdown_connections 2024-03-19T02:44:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:04.885+0000 7fc1157ff640 1 -- 172.21.15.116:0/2304764409 wait complete. 2024-03-19T02:44:04.898 INFO:tasks.cephfs.filesystem:scrub status for tag:68953cb6-eb40-4f84-8bff-a1f95510e936 - {'path': '/', 'tag': '68953cb6-eb40-4f84-8bff-a1f95510e936', 'options': 'recursive,force'} 2024-03-19T02:44:05.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23877 6.18 MB/sec execute 87 sec latency 221.193 ms 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: osdmap e116: 12 total, 12 up, 12 in 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: pgmap v512: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-03-19T02:44:05.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:05 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: osdmap e116: 12 total, 12 up, 12 in 2024-03-19T02:44:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:44:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:06.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: pgmap v512: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-03-19T02:44:06.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:05 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: osdmap e116: 12 total, 12 up, 12 in 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: pgmap v512: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 218 op/s 2024-03-19T02:44:06.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:05 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:06.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 23994 6.13 MB/sec execute 88 sec latency 161.725 ms 2024-03-19T02:44:07.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24168 6.13 MB/sec execute 89 sec latency 124.169 ms 2024-03-19T02:44:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:07 smithi162 ceph-mon[35761]: osdmap e117: 12 total, 12 up, 12 in 2024-03-19T02:44:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:07 smithi162 ceph-mon[35761]: pgmap v514: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-03-19T02:44:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:07 smithi154 ceph-mon[35693]: osdmap e117: 12 total, 12 up, 12 in 2024-03-19T02:44:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:07 smithi154 ceph-mon[35693]: pgmap v514: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-03-19T02:44:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:07 smithi116 ceph-mon[32263]: osdmap e117: 12 total, 12 up, 12 in 2024-03-19T02:44:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:07 smithi116 ceph-mon[32263]: pgmap v514: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 272 op/s 2024-03-19T02:44:08.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24266 6.09 MB/sec execute 90 sec latency 152.701 ms 2024-03-19T02:44:09.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24398 6.07 MB/sec execute 91 sec latency 156.525 ms 2024-03-19T02:44:09.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:09 smithi162 ceph-mon[35761]: pgmap v515: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T02:44:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:09 smithi154 ceph-mon[35693]: pgmap v515: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T02:44:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:09 smithi116 ceph-mon[32263]: pgmap v515: 129 pgs: 129 active+clean; 861 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T02:44:10.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24506 6.04 MB/sec execute 92 sec latency 176.118 ms 2024-03-19T02:44:11.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24645 6.02 MB/sec execute 93 sec latency 147.709 ms 2024-03-19T02:44:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:11 smithi162 ceph-mon[35761]: pgmap v516: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-03-19T02:44:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:11 smithi154 ceph-mon[35693]: pgmap v516: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-03-19T02:44:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:11 smithi116 ceph-mon[32263]: pgmap v516: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 251 op/s 2024-03-19T02:44:12.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24744 6.00 MB/sec execute 94 sec latency 134.167 ms 2024-03-19T02:44:13.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24853 5.97 MB/sec execute 95 sec latency 129.184 ms 2024-03-19T02:44:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:13 smithi162 ceph-mon[35761]: pgmap v517: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 240 op/s 2024-03-19T02:44:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:13 smithi154 ceph-mon[35693]: pgmap v517: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 240 op/s 2024-03-19T02:44:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:13 smithi116 ceph-mon[32263]: pgmap v517: 129 pgs: 129 active+clean; 870 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 240 op/s 2024-03-19T02:44:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 24982 6.03 MB/sec execute 96 sec latency 128.048 ms 2024-03-19T02:44:15.324 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25100 6.03 MB/sec execute 97 sec latency 176.309 ms 2024-03-19T02:44:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:15 smithi162 ceph-mon[35761]: pgmap v518: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 216 op/s 2024-03-19T02:44:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:15 smithi154 ceph-mon[35693]: pgmap v518: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 216 op/s 2024-03-19T02:44:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:15 smithi116 ceph-mon[32263]: pgmap v518: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 216 op/s 2024-03-19T02:44:16.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25204 6.03 MB/sec execute 98 sec latency 240.910 ms 2024-03-19T02:44:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:16 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:16.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:16 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:16.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:16 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:17.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25365 6.11 MB/sec execute 99 sec latency 166.961 ms 2024-03-19T02:44:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:17 smithi162 ceph-mon[35761]: pgmap v519: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 214 op/s 2024-03-19T02:44:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:17 smithi154 ceph-mon[35693]: pgmap v519: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 214 op/s 2024-03-19T02:44:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:17 smithi116 ceph-mon[32263]: pgmap v519: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 214 op/s 2024-03-19T02:44:18.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25540 6.19 MB/sec execute 100 sec latency 145.910 ms 2024-03-19T02:44:19.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25718 6.28 MB/sec execute 101 sec latency 139.814 ms 2024-03-19T02:44:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:19 smithi162 ceph-mon[35761]: pgmap v520: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-03-19T02:44:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:19 smithi154 ceph-mon[35693]: pgmap v520: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-03-19T02:44:19.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:19 smithi116 ceph-mon[32263]: pgmap v520: 129 pgs: 129 active+clean; 885 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 180 op/s 2024-03-19T02:44:20.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25814 6.31 MB/sec execute 102 sec latency 156.148 ms 2024-03-19T02:44:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:21.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25903 6.27 MB/sec execute 103 sec latency 151.398 ms 2024-03-19T02:44:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:21 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:21 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:21 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:21 smithi162 ceph-mon[35761]: pgmap v521: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-03-19T02:44:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:21 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:21 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:21 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:21 smithi154 ceph-mon[35693]: pgmap v521: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-03-19T02:44:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:21 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:21 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:21 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:21 smithi116 ceph-mon[32263]: pgmap v521: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 239 op/s 2024-03-19T02:44:22.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 25967 6.24 MB/sec execute 104 sec latency 131.966 ms 2024-03-19T02:44:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:22 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:22 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:22 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:44:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:22 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:22 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:22 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:44:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:22 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:22 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:22 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:44:23.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:23 smithi162 ceph-mon[35761]: pgmap v522: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 154 op/s 2024-03-19T02:44:23.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26032 6.19 MB/sec execute 105 sec latency 160.133 ms 2024-03-19T02:44:23.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:23 smithi154 ceph-mon[35693]: pgmap v522: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 154 op/s 2024-03-19T02:44:23.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:23 smithi116 ceph-mon[32263]: pgmap v522: 129 pgs: 129 active+clean; 904 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 154 op/s 2024-03-19T02:44:24.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26129 6.23 MB/sec execute 106 sec latency 125.653 ms 2024-03-19T02:44:25.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26215 6.18 MB/sec execute 107 sec latency 153.796 ms 2024-03-19T02:44:25.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:25 smithi162 ceph-mon[35761]: pgmap v523: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 216 op/s 2024-03-19T02:44:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:25 smithi154 ceph-mon[35693]: pgmap v523: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 216 op/s 2024-03-19T02:44:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:25 smithi116 ceph-mon[32263]: pgmap v523: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 216 op/s 2024-03-19T02:44:26.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26367 6.26 MB/sec execute 108 sec latency 149.020 ms 2024-03-19T02:44:27.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26466 6.23 MB/sec execute 109 sec latency 211.473 ms 2024-03-19T02:44:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:27 smithi162 ceph-mon[35761]: pgmap v524: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:27 smithi154 ceph-mon[35693]: pgmap v524: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:27 smithi116 ceph-mon[32263]: pgmap v524: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:28.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26593 6.29 MB/sec execute 110 sec latency 159.523 ms 2024-03-19T02:44:29.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26728 6.29 MB/sec execute 111 sec latency 146.038 ms 2024-03-19T02:44:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:29 smithi162 ceph-mon[35761]: pgmap v525: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:29 smithi154 ceph-mon[35693]: pgmap v525: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:29.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:29 smithi116 ceph-mon[32263]: pgmap v525: 129 pgs: 129 active+clean; 910 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 120 op/s 2024-03-19T02:44:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26797 6.27 MB/sec execute 112 sec latency 162.287 ms 2024-03-19T02:44:31.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26881 6.24 MB/sec execute 113 sec latency 176.472 ms 2024-03-19T02:44:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:31 smithi162 ceph-mon[35761]: pgmap v526: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:44:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:31 smithi154 ceph-mon[35693]: pgmap v526: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:44:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:31 smithi116 ceph-mon[32263]: pgmap v526: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.3 MiB/s wr, 182 op/s 2024-03-19T02:44:32.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 26967 6.21 MB/sec execute 114 sec latency 152.697 ms 2024-03-19T02:44:33.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27052 6.17 MB/sec execute 115 sec latency 201.308 ms 2024-03-19T02:44:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:33 smithi162 ceph-mon[35761]: pgmap v527: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 123 op/s 2024-03-19T02:44:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:33 smithi154 ceph-mon[35693]: pgmap v527: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 123 op/s 2024-03-19T02:44:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:33 smithi116 ceph-mon[32263]: pgmap v527: 129 pgs: 129 active+clean; 859 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 123 op/s 2024-03-19T02:44:34.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27153 6.15 MB/sec execute 116 sec latency 193.552 ms 2024-03-19T02:44:34.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:44:34.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.091+0000 7f46f8176640 1 -- 172.21.15.116:0/1414528585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f0107900 msgr2=0x7f46f0107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.091+0000 7f46f8176640 1 --2- 172.21.15.116:0/1414528585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f0107900 0x7f46f0107ce0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009900 tx=0x7f46ec02bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 -- 172.21.15.116:0/1414528585 shutdown_connections 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 --2- 172.21.15.116:0/1414528585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f01002b0 0x7f46f010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 --2- 172.21.15.116:0/1414528585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f00ff8f0 0x7f46f00ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 --2- 172.21.15.116:0/1414528585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f0107900 0x7f46f0107ce0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 -- 172.21.15.116:0/1414528585 >> 172.21.15.116:0/1414528585 conn(0x7f46f0076870 msgr2=0x7f46f00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 -- 172.21.15.116:0/1414528585 shutdown_connections 2024-03-19T02:44:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.092+0000 7f46f8176640 1 -- 172.21.15.116:0/1414528585 wait complete. 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 Processor -- start 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 -- start start 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f01002b0 0x7f46f0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f0107900 0x7f46f019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46f0111ab0 con 0x7f46f0107900 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46f0111930 con 0x7f46f00ff8f0 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f8176640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46f0111c30 con 0x7f46f01002b0 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f5f12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f5f12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49210/0 (socket says 172.21.15.116:49210) 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f6713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f0107900 0x7f46f019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f5f12640 1 -- 172.21.15.116:0/3676059939 learned_addr learned my addr 172.21.15.116:0/3676059939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.093+0000 7f46f5711640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f01002b0 0x7f46f0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 -- 172.21.15.116:0/3676059939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f01002b0 msgr2=0x7f46f0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f01002b0 0x7f46f0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 -- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f0107900 msgr2=0x7f46f019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f0107900 0x7f46f019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 -- 172.21.15.116:0/3676059939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f019cf40 con 0x7f46f00ff8f0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f5f12640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000c00 tx=0x7f46ec02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46e6ffd640 1 -- 172.21.15.116:0/3676059939 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec043020 con 0x7f46f00ff8f0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f019d1d0 con 0x7f46f00ff8f0 2024-03-19T02:44:35.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f01a4ac0 con 0x7f46f00ff8f0 2024-03-19T02:44:35.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46e6ffd640 1 -- 172.21.15.116:0/3676059939 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46ec02cbb0 con 0x7f46f00ff8f0 2024-03-19T02:44:35.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.094+0000 7f46e6ffd640 1 -- 172.21.15.116:0/3676059939 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec003b10 con 0x7f46f00ff8f0 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.097+0000 7f46e6ffd640 1 -- 172.21.15.116:0/3676059939 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f46ec046020 con 0x7f46f00ff8f0 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.097+0000 7f46e6ffd640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 0x7f46dc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46e6ffd640 1 -- 172.21.15.116:0/3676059939 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f46ec0f10f0 con 0x7f46f00ff8f0 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f5711640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 0x7f46dc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 Processor -- start 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 -- start start 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d54e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d60320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abb6cb8500 con 0x55abb6d55350 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55abb6c60be0 con 0x55abb6d54a30 2024-03-19T02:44:35.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46fae35740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55abb6cb45a0 con 0x55abb6d5df30 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f5711640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 0x7f46dc0a4eb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f46f00fd8b0 tx=0x7f46e000b410 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49610/0 (socket says 172.21.15.116:49610) 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f5711640 1 -- 172.21.15.116:0/3358048494 learned_addr learned my addr 172.21.15.116:0/3358048494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.098+0000 7f46f6713640 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d60320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5f12640 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d54e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 -- 172.21.15.116:0/3358048494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 msgr2=0x55abb6d60320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d60320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 -- 172.21.15.116:0/3358048494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 msgr2=0x55abb6d54e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d54e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 -- 172.21.15.116:0/3358048494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb6c82000 con 0x55abb6d55350 2024-03-19T02:44:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46f5711640 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c000 tx=0x7f46e0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31c42cc1c5c6778 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46dbfff640 1 -- 172.21.15.116:0/3358048494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e001e620 con 0x55abb6d55350 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46dbfff640 1 -- 172.21.15.116:0/3358048494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46e001ec70 con 0x55abb6d55350 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.099+0000 7f46dbfff640 1 -- 172.21.15.116:0/3358048494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e001dce0 con 0x55abb6d55350 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 -- 172.21.15.116:0/3358048494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 msgr2=0x55abb6d5d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c000 tx=0x7f46e0013000 comp rx=0 tx=0).stop 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 -- 172.21.15.116:0/3358048494 shutdown_connections 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d60320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d5d9f0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 --2- 172.21.15.116:0/3358048494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d54e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 -- 172.21.15.116:0/3358048494 >> 172.21.15.116:0/3358048494 conn(0x55abb6d4f160 msgr2=0x55abb6d4f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 -- 172.21.15.116:0/3358048494 shutdown_connections 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.100+0000 7f46fae35740 1 -- 172.21.15.116:0/3358048494 wait complete. 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.101+0000 7f46fae35740 1 -- start start 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.101+0000 7f46fae35740 1 Processor -- start 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d5af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 0x55abb6d5b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abb6d65130 con 0x55abb6d5df30 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55abb6c74160 con 0x55abb6d54a30 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55abb6d652b0 con 0x55abb6d55350 2024-03-19T02:44:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46fae35740 10 client.0 _ll_register_callbacks cb 0x55abb6cb0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46f6713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46f6713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49626/0 (socket says 172.21.15.116:49626) 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46f6713640 1 -- 172.21.15.116:0/452203204 learned_addr learned my addr 172.21.15.116:0/452203204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.102+0000 7f46f6713640 1 -- 172.21.15.116:0/452203204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 msgr2=0x55abb6d5b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f5f12640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d5af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f5711640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 0x55abb6d5b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f6713640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 0x55abb6d5b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f6713640 1 -- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 msgr2=0x55abb6d5af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f6713640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d5af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f6713640 1 -- 172.21.15.116:0/452203204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb6a967d0 con 0x55abb6d5df30 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f5f12640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d5af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f5711640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 0x55abb6d5b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:44:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46f6713640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f46e8011d40 tx=0x7f46e800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7ebcb5f5361dc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.103+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e800fe60 con 0x55abb6d5df30 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f46e8004b20 con 0x55abb6d5df30 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46e001ec70 con 0x55abb6d5df30 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f46e8024050 con 0x55abb6d5df30 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 10 client.20301 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55abb69cc3f0 con 0x55abb6d5df30 2024-03-19T02:44:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46e001ec70 con 0x55abb6d5df30 2024-03-19T02:44:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f46e801b470 con 0x55abb6d5df30 2024-03-19T02:44:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 10 client.20301 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 10 client.20301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 20 client.20301 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6d72cb0 0x55abb6d75110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 4 client.20301 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.104+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb6cc5a40 con 0x55abb6d72cb0 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.105+0000 7f46f5f12640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6d72cb0 0x55abb6d75110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.105+0000 7f46f5f12640 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6d72cb0 0x55abb6d75110 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a067538f3da3060f server_cookie=c82bc078c026aa25 in_seq=0 out_seq=0 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.106+0000 7f46d97fa640 10 client.20301 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.108+0000 7f46d97fa640 1 -- 172.21.15.116:0/452203204 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abb6cc5a40 con 0x55abb6d72cb0 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.108+0000 7f46d97fa640 10 client.20301 handle_command_reply: tid=0 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.108+0000 7f46fae35740 1 client.20301 shutdown 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 msgr2=0x55abb6d5b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f46e8011d40 tx=0x7f46e800f790 comp rx=0 tx=0).stop 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 shutdown_connections 2024-03-19T02:44:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6d72cb0 0x55abb6d75110 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d5df30 0x55abb6d5b9e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d55350 0x55abb6d5b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 --2- 172.21.15.116:0/452203204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d5af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 >> 172.21.15.116:0/452203204 conn(0x55abb6d4f160 msgr2=0x55abb6d50eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46d97fa640 0 client.20301 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.109+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 shutdown_connections 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.110+0000 7f46fae35740 1 -- 172.21.15.116:0/452203204 wait complete. 2024-03-19T02:44:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.110+0000 7f46fae35740 20 client.20301 trim_cache size 0 max 16384 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 Processor -- start 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 -- start start 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d54a30 0x55abb6d5d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d5df30 0x55abb6d7aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46d400ff60 con 0x55abb6d55350 2024-03-19T02:44:35.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46d4011aa0 con 0x55abb6d5df30 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46fae35740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46d4010630 con 0x55abb6d54a30 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49628/0 (socket says 172.21.15.116:49628) 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46f5711640 1 -- 172.21.15.116:0/2458515942 learned_addr learned my addr 172.21.15.116:0/2458515942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46f5f12640 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d54a30 0x55abb6d5d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.136+0000 7f46f6713640 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d5df30 0x55abb6d7aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 -- 172.21.15.116:0/2458515942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d54a30 msgr2=0x55abb6d5d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d54a30 0x55abb6d5d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 -- 172.21.15.116:0/2458515942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d5df30 msgr2=0x55abb6d7aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d5df30 0x55abb6d7aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 -- 172.21.15.116:0/2458515942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb698c0f0 con 0x55abb6d55350 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46f5711640 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f46e000bfd0 tx=0x7f46e0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3903d3437075aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46db7fe640 1 -- 172.21.15.116:0/2458515942 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0036070 con 0x55abb6d55350 2024-03-19T02:44:35.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46db7fe640 1 -- 172.21.15.116:0/2458515942 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46e000b910 con 0x55abb6d55350 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.137+0000 7f46db7fe640 1 -- 172.21.15.116:0/2458515942 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0031930 con 0x55abb6d55350 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- 172.21.15.116:0/2458515942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 msgr2=0x55abb6d6df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f46e000bfd0 tx=0x7f46e0010d50 comp rx=0 tx=0).stop 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- 172.21.15.116:0/2458515942 shutdown_connections 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d5df30 0x55abb6d7aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6df60 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 --2- 172.21.15.116:0/2458515942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d54a30 0x55abb6d5d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- 172.21.15.116:0/2458515942 >> 172.21.15.116:0/2458515942 conn(0x55abb6d4f160 msgr2=0x55abb6d59830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- 172.21.15.116:0/2458515942 shutdown_connections 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- 172.21.15.116:0/2458515942 wait complete. 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 -- start start 2024-03-19T02:44:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.138+0000 7f46fae35740 1 Processor -- start 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d6a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d65630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46d40113d0 con 0x55abb6d55350 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46d4010d00 con 0x55abb6d54a30 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f46d400df60 con 0x55abb6d5df30 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46fae35740 10 client.0 _ll_register_callbacks cb 0x7f46e8009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49640/0 (socket says 172.21.15.116:49640) 2024-03-19T02:44:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 -- 172.21.15.116:0/2501554490 learned_addr learned my addr 172.21.15.116:0/2501554490 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5f12640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d6a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f6713640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d65630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 -- 172.21.15.116:0/2501554490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 msgr2=0x55abb6d65630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d65630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 -- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 msgr2=0x55abb6d6a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d6a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.139+0000 7f46f5711640 1 -- 172.21.15.116:0/2501554490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb6d65e30 con 0x55abb6d55350 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46f5711640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f46e0005830 tx=0x7f46e00319e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d983d1bf54730f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0036070 con 0x55abb6d55350 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f46e0031de0 con 0x55abb6d55350 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abb6cbfd90 con 0x55abb6d55350 2024-03-19T02:44:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f46e002a020 con 0x55abb6d55350 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46fae35740 10 client.20307 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.140+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55abb6dd8b00 con 0x55abb6d55350 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55abb6cbfd90 con 0x55abb6d55350 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f46e003a760 con 0x55abb6d55350 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 10 client.20307 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 10 client.20307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 20 client.20307 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6dd9020 0x55abb6ddb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 4 client.20307 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55abb6c3d540 con 0x55abb6dd9020 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46f5f12640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6dd9020 0x55abb6ddb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.141+0000 7f46d97fa640 10 client.20307 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.142+0000 7f46f5f12640 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6dd9020 0x55abb6ddb440 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa75aa1b1c13117f server_cookie=57bf43783cd45d2 in_seq=0 out_seq=0 2024-03-19T02:44:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46d97fa640 1 -- 172.21.15.116:0/2501554490 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55abb6c3d540 con 0x55abb6dd9020 2024-03-19T02:44:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46d97fa640 10 client.20307 handle_command_reply: tid=0 2024-03-19T02:44:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 client.20307 shutdown 2024-03-19T02:44:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 msgr2=0x55abb6d6a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f46e0005830 tx=0x7f46e00319e0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 shutdown_connections 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55abb6dd9020 0x55abb6ddb440 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abb6d5df30 0x55abb6d65630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46d97fa640 0 client.20307 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abb6d55350 0x55abb6d6a770 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 --2- 172.21.15.116:0/2501554490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abb6d54a30 0x55abb6d6a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 >> 172.21.15.116:0/2501554490 conn(0x55abb6d4f160 msgr2=0x55abb6d50590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 shutdown_connections 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 1 -- 172.21.15.116:0/2501554490 wait complete. 2024-03-19T02:44:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.148+0000 7f46fae35740 20 client.20307 trim_cache size 0 max 16384 2024-03-19T02:44:35.151 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:44:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:44:35.151 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 msgr2=0x7f46dc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 0x7f46dc0a4eb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f46f00fd8b0 tx=0x7f46e000b410 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 msgr2=0x7f46f0197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000c00 tx=0x7f46ec02c6c0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 shutdown_connections 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f46dc0a29f0 0x7f46dc0a4eb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46f0107900 0x7f46f019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46f01002b0 0x7f46f0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 --2- 172.21.15.116:0/3676059939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f00ff8f0 0x7f46f0197f50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 >> 172.21.15.116:0/3676059939 conn(0x7f46f0076870 msgr2=0x7f46f00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 shutdown_connections 2024-03-19T02:44:35.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.149+0000 7f46f8176640 1 -- 172.21.15.116:0/3676059939 wait complete. 2024-03-19T02:44:35.162 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:44:35.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:44:35.191 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:35 smithi116 ceph-mon[32263]: pgmap v528: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 190 op/s 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.322+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1000207923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10e4b0 msgr2=0x7fa7ac11d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.322+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1000207923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac11d010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009860 tx=0x7fa7a402bb70 comp rx=0 tx=0).stop 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.323+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1000207923 shutdown_connections 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.323+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1000207923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac11d010 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.323+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1000207923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.323+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1000207923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10fdc0 0x7fa7ac1101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.323+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1000207923 >> 172.21.15.116:0/1000207923 conn(0x7fa7ac06c700 msgr2=0x7fa7ac06cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.324+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1000207923 shutdown_connections 2024-03-19T02:44:35.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27262 6.13 MB/sec execute 117 sec latency 158.242 ms 2024-03-19T02:44:35.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.327+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1000207923 wait complete. 2024-03-19T02:44:35.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.328+0000 7fa7b08d9640 1 Processor -- start 2024-03-19T02:44:35.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.329+0000 7fa7b08d9640 1 -- start start 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac1ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10fdc0 0x7fa7ac1a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac1206b0 con 0x7fa7ac10e4b0 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac120530 con 0x7fa7ac10d920 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7b08d9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac120830 con 0x7fa7ac10fdc0 2024-03-19T02:44:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10fdc0 0x7fa7ac1a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7ab7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7ab7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49274/0 (socket says 172.21.15.116:49274) 2024-03-19T02:44:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1388995218 learned_addr learned my addr 172.21.15.116:0/1388995218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.330+0000 7fa7aaffd640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac1ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1388995218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10fdc0 msgr2=0x7fa7ac1a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10fdc0 0x7fa7ac1a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10e4b0 msgr2=0x7fa7ac1ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac1ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1388995218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac1a7270 con 0x7fa7ac10d920 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.331+0000 7fa7ab7fe640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa79400e9f0 tx=0x7fa79400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.332+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79400cd80 con 0x7fa7ac10d920 2024-03-19T02:44:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.332+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ac1a7560 con 0x7fa7ac10d920 2024-03-19T02:44:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.332+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ac1b3800 con 0x7fa7ac10d920 2024-03-19T02:44:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.332+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7940108e0 con 0x7fa7ac10d920 2024-03-19T02:44:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.332+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794018880 con 0x7fa7ac10d920 2024-03-19T02:44:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.334+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c005290 con 0x7fa7ac10d920 2024-03-19T02:44:35.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.336+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa794010a90 con 0x7fa7ac10d920 2024-03-19T02:44:35.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.336+0000 7fa7a8ff9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 0x7fa7980a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.337+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7fa7940cf750 con 0x7fa7ac10d920 2024-03-19T02:44:35.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.337+0000 7fa7aaffd640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 0x7fa7980a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.338+0000 7fa7aaffd640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 0x7fa7980a5030 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0727c0 tx=0x7fa79c010040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.345+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa79409bac0 con 0x7fa7ac10d920 2024-03-19T02:44:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:35 smithi162 ceph-mon[35761]: pgmap v528: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 190 op/s 2024-03-19T02:44:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:35 smithi154 ceph-mon[35693]: pgmap v528: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 190 op/s 2024-03-19T02:44:35.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.685+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa78c005580 con 0x7fa7ac10d920 2024-03-19T02:44:35.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.686+0000 7fa7a8ff9640 1 -- 172.21.15.116:0/1388995218 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7fa7940ce090 con 0x7fa7ac10d920 2024-03-19T02:44:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:44:35.687 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:44:35.687 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:44:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 msgr2=0x7fa7980a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 0x7fa7980a5030 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0727c0 tx=0x7fa79c010040 comp rx=0 tx=0).stop 2024-03-19T02:44:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 msgr2=0x7fa7ac1ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa79400e9f0 tx=0x7fa79400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 shutdown_connections 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7980a2b70 0x7fa7980a5030 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7ac10fdc0 0x7fa7ac1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ac10e4b0 0x7fa7ac1ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 --2- 172.21.15.116:0/1388995218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7ac10d920 0x7fa7ac1ac940 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 >> 172.21.15.116:0/1388995218 conn(0x7fa7ac06c700 msgr2=0x7fa7ac070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 shutdown_connections 2024-03-19T02:44:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.688+0000 7fa7b08d9640 1 -- 172.21.15.116:0/1388995218 wait complete. 2024-03-19T02:44:35.703 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.883+0000 7f68a7953640 1 -- 172.21.15.116:0/1593042800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a0101790 msgr2=0x7f68a0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.883+0000 7f68a7953640 1 --2- 172.21.15.116:0/1593042800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a0101790 0x7f68a0101c10 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f6890009930 tx=0x7f689002bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.883+0000 7f68a7953640 1 -- 172.21.15.116:0/1593042800 shutdown_connections 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 --2- 172.21.15.116:0/1593042800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 0x7f68a010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 --2- 172.21.15.116:0/1593042800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a0101790 0x7f68a0101c10 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 --2- 172.21.15.116:0/1593042800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a01077a0 0x7f68a0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 -- 172.21.15.116:0/1593042800 >> 172.21.15.116:0/1593042800 conn(0x7f68a00fd2c0 msgr2=0x7f68a00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 -- 172.21.15.116:0/1593042800 shutdown_connections 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 -- 172.21.15.116:0/1593042800 wait complete. 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 Processor -- start 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.884+0000 7f68a7953640 1 -- start start 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 0x7f68a0074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0111a20 con 0x7f68a01077a0 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01118a0 con 0x7f68a0101790 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a7953640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0111ba0 con 0x7f68a0102150 2024-03-19T02:44:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49662/0 (socket says 172.21.15.116:49662) 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a5ef0640 1 -- 172.21.15.116:0/2227037397 learned_addr learned my addr 172.21.15.116:0/2227037397 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49288/0 (socket says 172.21.15.116:49288) 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a4eee640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 0x7f68a0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 -- 172.21.15.116:0/2227037397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 msgr2=0x7f68a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 0x7f68a0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 -- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 msgr2=0x7f68a00745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a56ef640 1 -- 172.21.15.116:0/2227037397 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a01aa970 con 0x7f68a0101790 2024-03-19T02:44:35.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.885+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f689c00e9f0 tx=0x7f689c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68967fc640 1 -- 172.21.15.116:0/2227037397 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c00ccc0 con 0x7f68a0101790 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a01aaba0 con 0x7f68a0101790 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68967fc640 1 -- 172.21.15.116:0/2227037397 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f689c0108e0 con 0x7f68a0101790 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68967fc640 1 -- 172.21.15.116:0/2227037397 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c018880 con 0x7f68a0101790 2024-03-19T02:44:35.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.886+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a01ab110 con 0x7f68a0101790 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 Processor -- start 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 -- start start 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344efc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f05290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e344e5f4b0 con 0x55e344efcbf0 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e344e07d70 con 0x55e344f057d0 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68aa61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e344e5b550 con 0x55e344efc2d0 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.889+0000 7f68967fc640 1 -- 172.21.15.116:0/2227037397 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f689c010a90 con 0x7f68a0101790 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a56ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f05290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a4eee640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a4eee640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49290/0 (socket says 172.21.15.116:49290) 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a4eee640 1 -- 172.21.15.116:0/2598487234 learned_addr learned my addr 172.21.15.116:0/2598487234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344efc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 -- 172.21.15.116:0/2598487234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 msgr2=0x55e344f07bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 -- 172.21.15.116:0/2598487234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 msgr2=0x55e344f05290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f05290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 -- 172.21.15.116:0/2598487234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e344e2a210 con 0x55e344efc2d0 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a4eee640 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f05290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344efc6b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f68980117e0 tx=0x7f6898011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85bf232c78424519 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f68967fc640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 0x7f688c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f688b7fe640 1 -- 172.21.15.116:0/2598487234 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898011250 con 0x55e344efc2d0 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f688b7fe640 1 -- 172.21.15.116:0/2598487234 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f689800f9a0 con 0x55e344efc2d0 2024-03-19T02:44:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.890+0000 7f688b7fe640 1 -- 172.21.15.116:0/2598487234 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898005530 con 0x55e344efc2d0 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68967fc640 1 -- 172.21.15.116:0/2227037397 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f689c0cf550 con 0x7f68a0101790 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 0x7f688c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 -- 172.21.15.116:0/2598487234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 msgr2=0x55e344efc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344efc6b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f68980117e0 tx=0x7f6898011ca0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 -- 172.21.15.116:0/2598487234 shutdown_connections 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f07bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f05290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 --2- 172.21.15.116:0/2598487234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344efc6b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 -- 172.21.15.116:0/2598487234 >> 172.21.15.116:0/2598487234 conn(0x55e344ef6a00 msgr2=0x55e344ef6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68a56ef640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 0x7f688c0a4eb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55e344efbda0 tx=0x7f689c009700 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 -- 172.21.15.116:0/2598487234 shutdown_connections 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.891+0000 7f68aa61b740 1 -- 172.21.15.116:0/2598487234 wait complete. 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68aa61b740 1 -- start start 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68aa61b740 1 Processor -- start 2024-03-19T02:44:35.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f02d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45444/0 (socket says 172.21.15.116:45444) 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68a5ef0640 1 -- 172.21.15.116:0/3433798090 learned_addr learned my addr 172.21.15.116:0/3433798090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.892+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f02d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e344f085c0 con 0x55e344efcbf0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e344e1b310 con 0x55e344f057d0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e344f08740 con 0x55e344efc2d0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 10 client.0 _ll_register_callbacks cb 0x55e344e0f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 -- 172.21.15.116:0/3433798090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 msgr2=0x55e344f03260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 -- 172.21.15.116:0/3433798090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 msgr2=0x55e344f02d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f02d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e344b733f0 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6898009700 tx=0x7f6898005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b356bfa33cceba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f6888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898040060 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f6898005230 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689800f9a0 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.893+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6898047050 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 10 client.30037 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e344c3d7d0 con 0x55e344efc2d0 2024-03-19T02:44:35.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f689800f9a0 con 0x55e344efc2d0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6898039890 con 0x55e344efc2d0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 10 client.30037 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 10 client.30037 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 20 client.30037 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f15af0 0x55e344f17ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 4 client.30037 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e344ebd3e0 con 0x55e344f15af0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f15af0 0x55e344f17ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f6888ff9640 10 client.30037 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.894+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f15af0 0x55e344f17ee0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9d91577f6bcbf686 server_cookie=b861d05f59ec335 in_seq=0 out_seq=0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f6888ff9640 1 -- 172.21.15.116:0/3433798090 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e344ebd3e0 con 0x55e344f15af0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f6888ff9640 10 client.30037 handle_command_reply: tid=0 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 client.30037 shutdown 2024-03-19T02:44:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 msgr2=0x55e344f027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6898009700 tx=0x7f6898005000 comp rx=0 tx=0).stop 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 shutdown_connections 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f15af0 0x55e344f17ee0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f6888ff9640 0 client.30037 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f02d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3433798090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f027e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 >> 172.21.15.116:0/3433798090 conn(0x55e344ef6a00 msgr2=0x55e344ef8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 shutdown_connections 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 1 -- 172.21.15.116:0/3433798090 wait complete. 2024-03-19T02:44:35.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.895+0000 7f68aa61b740 20 client.30037 trim_cache size 0 max 16384 2024-03-19T02:44:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 Processor -- start 2024-03-19T02:44:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 -- start start 2024-03-19T02:44:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efc2d0 0x55e344f04f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01acac0 con 0x55e344efc2d0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01ae600 con 0x55e344f057d0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.912+0000 7f68aa61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01ad190 con 0x55e344efcbf0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efc2d0 0x55e344f04f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45456/0 (socket says 172.21.15.116:45456) 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 -- 172.21.15.116:0/3893516433 learned_addr learned my addr 172.21.15.116:0/3893516433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 -- 172.21.15.116:0/3893516433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 msgr2=0x55e344f03080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 -- 172.21.15.116:0/3893516433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efc2d0 msgr2=0x55e344f04f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efc2d0 0x55e344f04f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 -- 172.21.15.116:0/3893516433 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e344efe3c0 con 0x55e344efcbf0 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f68a4eee640 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6890036040 tx=0x7f6890005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21a4279db9b9b1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f688affd640 1 -- 172.21.15.116:0/3893516433 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68900042a0 con 0x55e344efcbf0 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f688affd640 1 -- 172.21.15.116:0/3893516433 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f689002cc90 con 0x55e344efcbf0 2024-03-19T02:44:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.913+0000 7f688affd640 1 -- 172.21.15.116:0/3893516433 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68900046c0 con 0x55e344efcbf0 2024-03-19T02:44:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 -- 172.21.15.116:0/3893516433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 msgr2=0x55e344f11360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6890036040 tx=0x7f6890005d90 comp rx=0 tx=0).stop 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 -- 172.21.15.116:0/3893516433 shutdown_connections 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f03080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efcbf0 0x55e344f11360 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 --2- 172.21.15.116:0/3893516433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efc2d0 0x55e344f04f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 -- 172.21.15.116:0/3893516433 >> 172.21.15.116:0/3893516433 conn(0x55e344ef6a00 msgr2=0x55e344ef5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.914+0000 7f68aa61b740 1 -- 172.21.15.116:0/3893516433 shutdown_connections 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.915+0000 7f68aa61b740 1 -- 172.21.15.116:0/3893516433 wait complete. 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.915+0000 7f68aa61b740 1 -- start start 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.915+0000 7f68aa61b740 1 Processor -- start 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.915+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f0d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f0dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f08ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01adf30 con 0x55e344efcbf0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68a01ad860 con 0x55e344f057d0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68a0073220 con 0x55e344efc2d0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68aa61b740 10 client.0 _ll_register_callbacks cb 0x7f6898004b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f0d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a5ef0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f0d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45468/0 (socket says 172.21.15.116:45468) 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f0dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a5ef0640 1 -- 172.21.15.116:0/1110169511 learned_addr learned my addr 172.21.15.116:0/1110169511 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f08ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 -- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 msgr2=0x55e344f0d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f0d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 -- 172.21.15.116:0/1110169511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 msgr2=0x55e344f08ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f08ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 -- 172.21.15.116:0/1110169511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e344f096c0 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.916+0000 7f68a4eee640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f0dd80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f689002c170 tx=0x7f6890005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd10a2a9ea0f7663 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689002c7e0 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f6890047040 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689002cc90 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6890061050 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f68aa61b740 10 client.20331 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e344f7b660 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f689002cc90 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.917+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f689005c030 con 0x55e344efcbf0 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 10 client.20331 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 10 client.20331 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:44:35.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 20 client.20331 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f7be10 0x55e344f7e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 4 client.20331 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e344e6c9f0 con 0x55e344f7be10 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f7be10 0x55e344f7e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f6888ff9640 10 client.20331 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.918+0000 7f68a5ef0640 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f7be10 0x55e344f7e230 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de81a794aa4238a server_cookie=d889978e0056e7a6 in_seq=0 out_seq=0 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f6888ff9640 1 -- 172.21.15.116:0/1110169511 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e344e6c9f0 con 0x55e344f7be10 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f6888ff9640 10 client.20331 handle_command_reply: tid=0 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 client.20331 shutdown 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 msgr2=0x55e344f0dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f0dd80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f689002c170 tx=0x7f6890005780 comp rx=0 tx=0).stop 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 shutdown_connections 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e344f7be10 0x55e344f7e230 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e344f057d0 0x55e344f08ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e344efcbf0 0x55e344f0dd80 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 --2- 172.21.15.116:0/1110169511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e344efc2d0 0x55e344f0d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 >> 172.21.15.116:0/1110169511 conn(0x55e344ef6a00 msgr2=0x55e344f00f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f6888ff9640 0 client.20331 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 shutdown_connections 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 1 -- 172.21.15.116:0/1110169511 wait complete. 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.919+0000 7f68aa61b740 20 client.20331 trim_cache size 0 max 16384 2024-03-19T02:44:35.925 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 msgr2=0x7f688c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 0x7f688c0a4eb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55e344efbda0 tx=0x7f689c009700 comp rx=0 tx=0).stop 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 msgr2=0x7f68a0073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f689c00e9f0 tx=0x7f689c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 shutdown_connections 2024-03-19T02:44:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f688c0a29f0 0x7f688c0a4eb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68a01077a0 0x7f68a00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68a0102150 0x7f68a0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 --2- 172.21.15.116:0/2227037397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a0101790 0x7f68a0073b20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 >> 172.21.15.116:0/2227037397 conn(0x7f68a00fd2c0 msgr2=0x7f68a00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 shutdown_connections 2024-03-19T02:44:35.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:35.920+0000 7f68a7953640 1 -- 172.21.15.116:0/2227037397 wait complete. 2024-03-19T02:44:35.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:44:36.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 -- 172.21.15.116:0/3657095660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb481006a0 msgr2=0x7feb48100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 --2- 172.21.15.116:0/3657095660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb481006a0 0x7feb48100b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009890 tx=0x7feb3c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 -- 172.21.15.116:0/3657095660 shutdown_connections 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 --2- 172.21.15.116:0/3657095660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb48101060 0x7feb4810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 --2- 172.21.15.116:0/3657095660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb481006a0 0x7feb48100b20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 --2- 172.21.15.116:0/3657095660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481066b0 0x7feb48106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.063+0000 7feb4e20b640 1 -- 172.21.15.116:0/3657095660 >> 172.21.15.116:0/3657095660 conn(0x7feb480778d0 msgr2=0x7feb48077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.064+0000 7feb4e20b640 1 -- 172.21.15.116:0/3657095660 shutdown_connections 2024-03-19T02:44:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.064+0000 7feb4e20b640 1 -- 172.21.15.116:0/3657095660 wait complete. 2024-03-19T02:44:36.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.064+0000 7feb4e20b640 1 Processor -- start 2024-03-19T02:44:36.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.064+0000 7feb4e20b640 1 -- start start 2024-03-19T02:44:36.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481006a0 0x7feb48197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 0x7feb4819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb48110930 con 0x7feb481066b0 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb481107b0 con 0x7feb481006a0 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb4e20b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb48110ab0 con 0x7feb48101060 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 0x7feb4819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481006a0 0x7feb48197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45482/0 (socket says 172.21.15.116:45482) 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 -- 172.21.15.116:0/393248142 learned_addr learned my addr 172.21.15.116:0/393248142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 0x7feb4819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49716/0 (socket says 172.21.15.116:49716) 2024-03-19T02:44:36.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 -- 172.21.15.116:0/393248142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481006a0 msgr2=0x7feb48197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481006a0 0x7feb48197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 -- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 msgr2=0x7feb4819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 0x7feb4819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.065+0000 7feb46ffd640 1 -- 172.21.15.116:0/393248142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4819cf70 con 0x7feb48101060 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb46ffd640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c035bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb44ff9640 1 -- 172.21.15.116:0/393248142 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c041020 con 0x7feb48101060 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4819d200 con 0x7feb48101060 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb44ff9640 1 -- 172.21.15.116:0/393248142 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb3c0044d0 con 0x7feb48101060 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb44ff9640 1 -- 172.21.15.116:0/393248142 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c039040 con 0x7feb48101060 2024-03-19T02:44:36.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.066+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb481a4af0 con 0x7feb48101060 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 Processor -- start 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 -- start start 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d25e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d25ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56080d1c74b0 con 0x56080d271970 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56080d16fd70 con 0x56080d272320 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.069+0000 7feb50df8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56080d1c3550 con 0x56080d25ead0 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49724/0 (socket says 172.21.15.116:49724) 2024-03-19T02:44:36.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb477fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d25e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 -- 172.21.15.116:0/3606535011 learned_addr learned my addr 172.21.15.116:0/3606535011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb44ff9640 1 -- 172.21.15.116:0/393248142 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7feb3c044020 con 0x7feb48101060 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d25ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 -- 172.21.15.116:0/3606535011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 msgr2=0x56080d25ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d25ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 -- 172.21.15.116:0/3606535011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 msgr2=0x56080d25e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d25e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 -- 172.21.15.116:0/3606535011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56080d192210 con 0x56080d271970 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d25ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb47fff640 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7feb40009fd0 tx=0x7feb400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94219df8058ff52d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.070+0000 7feb2dffb640 1 -- 172.21.15.116:0/3606535011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40004d20 con 0x56080d271970 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb2dffb640 1 -- 172.21.15.116:0/3606535011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb40004050 con 0x56080d271970 2024-03-19T02:44:36.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb2dffb640 1 -- 172.21.15.116:0/3606535011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40005260 con 0x56080d271970 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb44ff9640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 0x7feb300a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb477fe640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 0x7feb300a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb44ff9640 1 -- 172.21.15.116:0/393248142 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7feb3c0f0940 con 0x7feb48101060 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb477fe640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 0x7feb300a4f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x56080d25f5f0 tx=0x7feb3800b390 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 -- 172.21.15.116:0/3606535011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 msgr2=0x56080d271d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7feb40009fd0 tx=0x7feb400118d0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 -- 172.21.15.116:0/3606535011 shutdown_connections 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d25ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d25e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 --2- 172.21.15.116:0/3606535011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d271d50 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.071+0000 7feb50df8740 1 -- 172.21.15.116:0/3606535011 >> 172.21.15.116:0/3606535011 conn(0x56080d260f70 msgr2=0x56080d261350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.072+0000 7feb50df8740 1 -- 172.21.15.116:0/3606535011 shutdown_connections 2024-03-19T02:44:36.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.072+0000 7feb50df8740 1 -- 172.21.15.116:0/3606535011 wait complete. 2024-03-19T02:44:36.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.072+0000 7feb50df8740 1 -- start start 2024-03-19T02:44:36.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.072+0000 7feb50df8740 1 Processor -- start 2024-03-19T02:44:36.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d275920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d275e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56080d278db0 con 0x56080d271970 2024-03-19T02:44:36.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56080d183310 con 0x56080d272320 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56080d278f30 con 0x56080d25ead0 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb50df8740 10 client.0 _ll_register_callbacks cb 0x56080d1772d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb477fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d275920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb477fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d275920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45498/0 (socket says 172.21.15.116:45498) 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49362/0 (socket says 172.21.15.116:49362) 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb477fe640 1 -- 172.21.15.116:0/3088660205 learned_addr learned my addr 172.21.15.116:0/3088660205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d275e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 -- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 msgr2=0x56080d275920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d275920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 -- 172.21.15.116:0/3088660205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 msgr2=0x56080d275e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d275e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.073+0000 7feb47fff640 1 -- 172.21.15.116:0/3088660205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56080cedb3f0 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.074+0000 7feb47fff640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb4001b040 tx=0x7feb40004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b11fe0ca03411b44 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.074+0000 7feb26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.074+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40028020 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.074+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb40004050 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.074+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7feb40003c40 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7feb40033050 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 10 client.30077 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56080cfa57d0 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb40004050 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7feb4000f4e0 con 0x56080d272320 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 10 client.30077 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 10 client.30077 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 20 client.30077 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d286260 0x56080d288650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 4 client.30077 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56080d2253e0 con 0x56080d286260 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.075+0000 7feb477fe640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d286260 0x56080d288650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.076+0000 7feb477fe640 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d286260 0x56080d288650 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e956797bb32d11ae server_cookie=c2e5d2e7f15ec5d2 in_seq=0 out_seq=0 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.076+0000 7feb26ffd640 10 client.30077 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb26ffd640 1 -- 172.21.15.116:0/3088660205 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56080d2253e0 con 0x56080d286260 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb26ffd640 10 client.30077 handle_command_reply: tid=0 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 client.30077 shutdown 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 msgr2=0x56080d2763a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb4001b040 tx=0x7feb40004c70 comp rx=0 tx=0).stop 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 shutdown_connections 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2763a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d286260 0x56080d288650 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d275e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb26ffd640 0 client.30077 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 --2- 172.21.15.116:0/3088660205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d275920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 >> 172.21.15.116:0/3088660205 conn(0x56080d260f70 msgr2=0x56080d25dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 shutdown_connections 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 1 -- 172.21.15.116:0/3088660205 wait complete. 2024-03-19T02:44:36.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.077+0000 7feb50df8740 20 client.30077 trim_cache size 0 max 16384 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 Processor -- start 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 -- start start 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d281b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2758e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb481aca20 con 0x56080d271970 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb481ae560 con 0x56080d272320 2024-03-19T02:44:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb50df8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb481ad0f0 con 0x56080d25ead0 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49738/0 (socket says 172.21.15.116:49738) 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 -- 172.21.15.116:0/2754659979 learned_addr learned my addr 172.21.15.116:0/2754659979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb477fe640 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d281b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb47fff640 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 -- 172.21.15.116:0/2754659979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 msgr2=0x56080d281b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d281b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 -- 172.21.15.116:0/2754659979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 msgr2=0x56080d2758e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2758e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.096+0000 7feb46ffd640 1 -- 172.21.15.116:0/2754659979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56080d272080 con 0x56080d271970 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.097+0000 7feb46ffd640 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0b9640 tx=0x7feb3c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e570993270f4dd95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.097+0000 7feb2d7fa640 1 -- 172.21.15.116:0/2754659979 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0bc5d0 con 0x56080d271970 2024-03-19T02:44:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.097+0000 7feb2d7fa640 1 -- 172.21.15.116:0/2754659979 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb3c0bd370 con 0x56080d271970 2024-03-19T02:44:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.097+0000 7feb2d7fa640 1 -- 172.21.15.116:0/2754659979 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0ee060 con 0x56080d271970 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- 172.21.15.116:0/2754659979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 msgr2=0x56080d28de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0b9640 tx=0x7feb3c005f90 comp rx=0 tx=0).stop 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- 172.21.15.116:0/2754659979 shutdown_connections 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d272320 0x56080d2758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d271970 0x56080d28de10 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 --2- 172.21.15.116:0/2754659979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d25ead0 0x56080d281b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- 172.21.15.116:0/2754659979 >> 172.21.15.116:0/2754659979 conn(0x56080d260f70 msgr2=0x56080d25dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- 172.21.15.116:0/2754659979 shutdown_connections 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- 172.21.15.116:0/2754659979 wait complete. 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.098+0000 7feb50df8740 1 -- start start 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 Processor -- start 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d271970 0x56080d27a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 0x56080d27a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb481ade90 con 0x56080d25ead0 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb481ad7c0 con 0x56080d271970 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb481aaa20 con 0x56080d272320 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb50df8740 10 client.0 _ll_register_callbacks cb 0x7feb38004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49742/0 (socket says 172.21.15.116:49742) 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 -- 172.21.15.116:0/3657685420 learned_addr learned my addr 172.21.15.116:0/3657685420 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d271970 0x56080d27a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb47fff640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 0x56080d27a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 -- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 msgr2=0x56080d27a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 0x56080d27a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 -- 172.21.15.116:0/3657685420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d271970 msgr2=0x56080d27a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d271970 0x56080d27a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb477fe640 1 -- 172.21.15.116:0/3657685420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56080d2eb710 con 0x56080d25ead0 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.099+0000 7feb47fff640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 0x56080d27a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb477fe640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7feb38004010 tx=0x7feb380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe4e5b1f1b84ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3802ab90 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7feb3802ad30 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb3c0bd370 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7feb3803c050 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 10 client.20349 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56080d2eca70 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb3c0bd370 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7feb3801e4b0 con 0x56080d25ead0 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 10 client.20349 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 10 client.20349 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 20 client.20349 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d2ed140 0x56080d2ef560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 4 client.20349 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.100+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56080d1d49f0 con 0x56080d2ed140 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.101+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d2ed140 0x56080d2ef560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.101+0000 7feb26ffd640 10 client.20349 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.101+0000 7feb46ffd640 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d2ed140 0x56080d2ef560 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=95f531d951651abc server_cookie=5b967709f869eb6f in_seq=0 out_seq=0 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.101+0000 7feb26ffd640 1 -- 172.21.15.116:0/3657685420 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56080d1d49f0 con 0x56080d2ed140 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb26ffd640 10 client.20349 handle_command_reply: tid=0 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 client.20349 shutdown 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 msgr2=0x56080d279d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7feb38004010 tx=0x7feb380046c0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 shutdown_connections 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56080d272320 0x56080d27a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56080d271970 0x56080d27a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56080d2ed140 0x56080d2ef560 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 --2- 172.21.15.116:0/3657685420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56080d25ead0 0x56080d279d90 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 >> 172.21.15.116:0/3657685420 conn(0x56080d260f70 msgr2=0x56080d275e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb26ffd640 0 client.20349 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:44:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 shutdown_connections 2024-03-19T02:44:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 1 -- 172.21.15.116:0/3657685420 wait complete. 2024-03-19T02:44:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.102+0000 7feb50df8740 20 client.20349 trim_cache size 0 max 16384 2024-03-19T02:44:36.109 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 msgr2=0x7feb300a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 0x7feb300a4f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x56080d25f5f0 tx=0x7feb3800b390 comp rx=0 tx=0).stop 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 msgr2=0x7feb481984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c035bc0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 shutdown_connections 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feb300a2a70 0x7feb300a4f30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb481066b0 0x7feb4819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb48101060 0x7feb481984c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 --2- 172.21.15.116:0/393248142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb481006a0 0x7feb48197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 >> 172.21.15.116:0/393248142 conn(0x7feb480778d0 msgr2=0x7feb48068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 shutdown_connections 2024-03-19T02:44:36.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.103+0000 7feb4e20b640 1 -- 172.21.15.116:0/393248142 wait complete. 2024-03-19T02:44:36.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:44:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1388995218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 -- 172.21.15.116:0/3511279993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 msgr2=0x7f9ef4113230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 --2- 172.21.15.116:0/3511279993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 0x7f9ef4113230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009900 tx=0x7f9ee002bcc0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 -- 172.21.15.116:0/3511279993 shutdown_connections 2024-03-19T02:44:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 --2- 172.21.15.116:0/3511279993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ef4113770 0x7f9ef4115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 --2- 172.21.15.116:0/3511279993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 0x7f9ef4113230 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 --2- 172.21.15.116:0/3511279993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef410a1c0 0x7f9ef410a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 -- 172.21.15.116:0/3511279993 >> 172.21.15.116:0/3511279993 conn(0x7f9ef406c910 msgr2=0x7f9ef406cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.259+0000 7f9efacac640 1 -- 172.21.15.116:0/3511279993 shutdown_connections 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.260+0000 7f9efacac640 1 -- 172.21.15.116:0/3511279993 wait complete. 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.260+0000 7f9efacac640 1 Processor -- start 2024-03-19T02:44:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.260+0000 7f9efacac640 1 -- start start 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ef410a1c0 0x7f9ef410d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 0x7f9ef410d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef4119150 con 0x7f9ef4113770 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef4118fd0 con 0x7f9ef410aae0 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9efacac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41192d0 con 0x7f9ef410a1c0 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49770/0 (socket says 172.21.15.116:49770) 2024-03-19T02:44:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 -- 172.21.15.116:0/201131571 learned_addr learned my addr 172.21.15.116:0/201131571 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 -- 172.21.15.116:0/201131571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ef410a1c0 msgr2=0x7f9ef410d460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ef410a1c0 0x7f9ef410d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 -- 172.21.15.116:0/201131571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 msgr2=0x7f9ef410d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 0x7f9ef410d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.261+0000 7f9ef9249640 1 -- 172.21.15.116:0/201131571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef41b3090 con 0x7f9ef4113770 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.262+0000 7f9ef9249640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00db20 tx=0x7f9eec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.263+0000 7f9ef1ffb640 1 -- 172.21.15.116:0/201131571 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec017070 con 0x7f9ef4113770 2024-03-19T02:44:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.263+0000 7f9ef1ffb640 1 -- 172.21.15.116:0/201131571 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9eec003e20 con 0x7f9ef4113770 2024-03-19T02:44:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.263+0000 7f9ef1ffb640 1 -- 172.21.15.116:0/201131571 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec010460 con 0x7f9ef4113770 2024-03-19T02:44:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.263+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef41b3380 con 0x7f9ef4113770 2024-03-19T02:44:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.264+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef41b37c0 con 0x7f9ef4113770 2024-03-19T02:44:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.268+0000 7f9efd971740 1 Processor -- start 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 -- start start 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dc76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dd2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b751d2a4b0 con 0x55b751dc7bf0 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b751cd2d70 con 0x55b751dc72d0 2024-03-19T02:44:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9efd971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b751d26550 con 0x55b751dd07d0 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.269+0000 7f9ef1ffb640 1 -- 172.21.15.116:0/201131571 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9eec003970 con 0x7f9ef4113770 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.270+0000 7f9ef1ffb640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 0x7f9ee40a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef9249640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 0x7f9ee40a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49778/0 (socket says 172.21.15.116:49778) 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3533209699 learned_addr learned my addr 172.21.15.116:0/3533209699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dd2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dc76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3533209699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 msgr2=0x55b751dd2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dd2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3533209699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 msgr2=0x55b751dc76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dc76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.271+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3533209699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b751cf5210 con 0x55b751dc7bf0 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.272+0000 7f9ef1ffb640 1 -- 172.21.15.116:0/201131571 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f9eec0cefc0 con 0x7f9ef4113770 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.272+0000 7f9ef9249640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 0x7f9ee40a5030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9eec003fd0 tx=0x7f9eec065f20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.272+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009bd0 tx=0x7f9ee800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663d51c06e90d443 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.272+0000 7f9edeffd640 1 -- 172.21.15.116:0/3533209699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8005020 con 0x55b751dc7bf0 2024-03-19T02:44:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.273+0000 7f9edeffd640 1 -- 172.21.15.116:0/3533209699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ee80047a0 con 0x55b751dc7bf0 2024-03-19T02:44:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.273+0000 7f9edeffd640 1 -- 172.21.15.116:0/3533209699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee80186d0 con 0x55b751dc7bf0 2024-03-19T02:44:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 -- 172.21.15.116:0/3533209699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 msgr2=0x55b751dd0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009bd0 tx=0x7f9ee800e720 comp rx=0 tx=0).stop 2024-03-19T02:44:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 -- 172.21.15.116:0/3533209699 shutdown_connections 2024-03-19T02:44:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dd2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc7bf0 0x55b751dd0290 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 --2- 172.21.15.116:0/3533209699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dc76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.274+0000 7f9efd971740 1 -- 172.21.15.116:0/3533209699 >> 172.21.15.116:0/3533209699 conn(0x55b751dc1a00 msgr2=0x55b751dc1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.275+0000 7f9efd971740 1 -- 172.21.15.116:0/3533209699 shutdown_connections 2024-03-19T02:44:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.275+0000 7f9efd971740 1 -- 172.21.15.116:0/3533209699 wait complete. 2024-03-19T02:44:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.276+0000 7f9efd971740 1 -- start start 2024-03-19T02:44:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.276+0000 7f9efd971740 1 Processor -- start 2024-03-19T02:44:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.276+0000 7f9efd971740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751dcddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b751de45d0 con 0x55b751dc72d0 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b751ce6310 con 0x55b751dc7bf0 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b751de4750 con 0x55b751dd07d0 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9efd971740 10 client.0 _ll_register_callbacks cb 0x55b751cda2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49786/0 (socket says 172.21.15.116:49786) 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3616046409 learned_addr learned my addr 172.21.15.116:0/3616046409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.277+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751dcddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.278+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3616046409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 msgr2=0x55b751dce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.278+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.278+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 msgr2=0x55b751dcddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.278+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751dcddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.278+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3616046409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b751a3e3f0 con 0x55b751dc72d0 2024-03-19T02:44:36.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.280+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009850 tx=0x7f9ee0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd5d18a560bd31c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.280+0000 7f9ed3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.280+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee002cd90 con 0x55b751dc72d0 2024-03-19T02:44:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.281+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee80047a0 con 0x55b751dc72d0 2024-03-19T02:44:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.281+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f9ee0005040 con 0x55b751dc72d0 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.281+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9ee0005240 con 0x55b751dc72d0 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.281+0000 7f9efd971740 10 client.20361 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.281+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b751b087d0 con 0x55b751dc72d0 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ee002e560 con 0x55b751dc72d0 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f9ee00047d0 con 0x55b751dc72d0 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 10 client.20361 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 10 client.20361 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 20 client.20361 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751df1fa0 0x55b751df43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 4 client.20361 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.282+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ee80047a0 con 0x55b751df1fa0 2024-03-19T02:44:36.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.283+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751df1fa0 0x55b751df43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.286+0000 7f9ed3fff640 10 client.20361 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:36.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.286+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751df1fa0 0x55b751df43b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db34a7a44f051464 server_cookie=9a838ae67a1c842a in_seq=0 out_seq=0 2024-03-19T02:44:36.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3616046409 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f9ee80047a0 con 0x55b751df1fa0 2024-03-19T02:44:36.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9ed3fff640 10 client.20361 handle_command_reply: tid=0 2024-03-19T02:44:36.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 client.20361 shutdown 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 msgr2=0x55b751dcd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009850 tx=0x7f9ee0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 shutdown_connections 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751df1fa0 0x55b751df43b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dd07d0 0x55b751dce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751dcddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 --2- 172.21.15.116:0/3616046409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dc72d0 0x55b751dcd870 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 >> 172.21.15.116:0/3616046409 conn(0x55b751dc1a00 msgr2=0x55b751dc3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.288+0000 7f9ed3fff640 0 client.20361 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.289+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 shutdown_connections 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.289+0000 7f9efd971740 1 -- 172.21.15.116:0/3616046409 wait complete. 2024-03-19T02:44:36.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.289+0000 7f9efd971740 20 client.20361 trim_cache size 0 max 16384 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.303+0000 7f9efd971740 1 Processor -- start 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 -- start start 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dcffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 0x55b751ded370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41b53a0 con 0x55b751dd07d0 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41b6ee0 con 0x55b751dc72d0 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9efd971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41b5a70 con 0x55b751dc7bf0 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 0x55b751ded370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49792/0 (socket says 172.21.15.116:49792) 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3030112244 learned_addr learned my addr 172.21.15.116:0/3030112244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dcffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3030112244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 msgr2=0x55b751dcffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dcffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3030112244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 msgr2=0x55b751dc3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 -- 172.21.15.116:0/3030112244 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b751dc25d0 con 0x55b751dc7bf0 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.304+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 0x55b751ded370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009640 tx=0x7f9ee8009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fe1953be6939c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.305+0000 7f9ede7fc640 1 -- 172.21.15.116:0/3030112244 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8009cc0 con 0x55b751dc7bf0 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.305+0000 7f9ede7fc640 1 -- 172.21.15.116:0/3030112244 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ee8009e60 con 0x55b751dc7bf0 2024-03-19T02:44:36.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.305+0000 7f9ede7fc640 1 -- 172.21.15.116:0/3030112244 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8005630 con 0x55b751dc7bf0 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.305+0000 7f9efd971740 1 -- 172.21.15.116:0/3030112244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 msgr2=0x55b751ded370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.305+0000 7f9efd971740 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 0x55b751ded370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009640 tx=0x7f9ee8009820 comp rx=0 tx=0).stop 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- 172.21.15.116:0/3030112244 shutdown_connections 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751dc3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc7bf0 0x55b751ded370 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- 172.21.15.116:0/3030112244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc72d0 0x55b751dcffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- 172.21.15.116:0/3030112244 >> 172.21.15.116:0/3030112244 conn(0x55b751dc1a00 msgr2=0x55b751dc2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- 172.21.15.116:0/3030112244 shutdown_connections 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- 172.21.15.116:0/3030112244 wait complete. 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- start start 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 Processor -- start 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc72d0 0x55b751de9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751de4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41b6810 con 0x55b751dd07d0 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef41b6140 con 0x55b751dc7bf0 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef410e990 con 0x55b751dc72d0 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.306+0000 7f9efd971740 10 client.0 _ll_register_callbacks cb 0x7f9ee0004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49806/0 (socket says 172.21.15.116:49806) 2024-03-19T02:44:36.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3339977392 learned_addr learned my addr 172.21.15.116:0/3339977392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751de4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc72d0 0x55b751de9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3339977392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc72d0 msgr2=0x55b751de9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc72d0 0x55b751de9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 msgr2=0x55b751de4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751de4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 -- 172.21.15.116:0/3339977392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b751de57f0 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751de4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ef3fff640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002f020 tx=0x7f9ee002ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914a6fbd16b46435 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ed3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0003830 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f9ee00039d0 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee8009e60 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9ee0003bd0 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.307+0000 7f9efd971740 10 client.20367 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b751e58000 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ee8009e60 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f9ee0058040 con 0x55b751dd07d0 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 10 client.20367 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 10 client.20367 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 20 client.20367 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751e58330 0x55b751e5a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 4 client.20367 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b751d883e0 con 0x55b751e58330 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751e58330 0x55b751e5a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9ed3fff640 10 client.20367 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.308+0000 7f9ef8a48640 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751e58330 0x55b751e5a750 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7abd4e0cd9aafce7 server_cookie=5cdc593e995e6f10 in_seq=0 out_seq=0 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9ed3fff640 1 -- 172.21.15.116:0/3339977392 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b751d883e0 con 0x55b751e58330 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9ed3fff640 10 client.20367 handle_command_reply: tid=0 2024-03-19T02:44:36.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 client.20367 shutdown 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 msgr2=0x55b751de52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002f020 tx=0x7f9ee002ef50 comp rx=0 tx=0).stop 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 shutdown_connections 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55b751e58330 0x55b751e5a750 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b751dd07d0 0x55b751de52b0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b751dc7bf0 0x55b751de4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9ed3fff640 0 client.20367 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 --2- 172.21.15.116:0/3339977392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b751dc72d0 0x55b751de9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 >> 172.21.15.116:0/3339977392 conn(0x55b751dc1a00 msgr2=0x55b751dc36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 shutdown_connections 2024-03-19T02:44:36.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 1 -- 172.21.15.116:0/3339977392 wait complete. 2024-03-19T02:44:36.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.309+0000 7f9efd971740 20 client.20367 trim_cache size 0 max 16384 2024-03-19T02:44:36.318 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 msgr2=0x7f9ee40a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 0x7f9ee40a5030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9eec003fd0 tx=0x7f9eec065f20 comp rx=0 tx=0).stop 2024-03-19T02:44:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 msgr2=0x7f9ef410dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00db20 tx=0x7f9eec00b260 comp rx=0 tx=0).stop 2024-03-19T02:44:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 shutdown_connections 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9ee40a2b70 0x7f9ee40a5030 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ef4113770 0x7f9ef410dee0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef410aae0 0x7f9ef410d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 --2- 172.21.15.116:0/201131571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9ef410a1c0 0x7f9ef410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 >> 172.21.15.116:0/201131571 conn(0x7f9ef406c910 msgr2=0x7f9ef41090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 shutdown_connections 2024-03-19T02:44:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.310+0000 7f9efacac640 1 -- 172.21.15.116:0/201131571 wait complete. 2024-03-19T02:44:36.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:44:36.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27375 6.12 MB/sec execute 118 sec latency 169.046 ms 2024-03-19T02:44:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1388995218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 -- 172.21.15.116:0/295175655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad81017b0 msgr2=0x7f1ad810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 --2- 172.21.15.116:0/295175655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad81017b0 0x7f1ad810d530 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009ca0 tx=0x7f1ad002c180 comp rx=0 tx=0).stop 2024-03-19T02:44:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 -- 172.21.15.116:0/295175655 shutdown_connections 2024-03-19T02:44:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 --2- 172.21.15.116:0/295175655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad81017b0 0x7f1ad810d530 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 --2- 172.21.15.116:0/295175655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8100df0 0x7f1ad8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 --2- 172.21.15.116:0/295175655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 0x7f1ad8106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 -- 172.21.15.116:0/295175655 >> 172.21.15.116:0/295175655 conn(0x7f1ad8076c50 msgr2=0x7f1ad80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 -- 172.21.15.116:0/295175655 shutdown_connections 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.463+0000 7f1adede8640 1 -- 172.21.15.116:0/295175655 wait complete. 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 Processor -- start 2024-03-19T02:44:36.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 -- start start 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad81017b0 0x7f1ad8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 0x7f1ad819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8110960 con 0x7f1ad8100df0 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81107e0 con 0x7f1ad81066e0 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adede8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8110ae0 con 0x7f1ad81017b0 2024-03-19T02:44:36.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49816/0 (socket says 172.21.15.116:49816) 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 -- 172.21.15.116:0/310954882 learned_addr learned my addr 172.21.15.116:0/310954882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad81017b0 0x7f1ad8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1add385640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 0x7f1ad819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 -- 172.21.15.116:0/310954882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad81017b0 msgr2=0x7f1ad8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad81017b0 0x7f1ad8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 -- 172.21.15.116:0/310954882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 msgr2=0x7f1ad819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 0x7f1ad819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.464+0000 7f1adcb84640 1 -- 172.21.15.116:0/310954882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad819cf00 con 0x7f1ad8100df0 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1adcb84640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e9f0 tx=0x7f1acc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1ad5ffb640 1 -- 172.21.15.116:0/310954882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc00cd80 con 0x7f1ad8100df0 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad819d1f0 con 0x7f1ad8100df0 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1ad5ffb640 1 -- 172.21.15.116:0/310954882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1acc0108e0 con 0x7f1ad8100df0 2024-03-19T02:44:36.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1ad5ffb640 1 -- 172.21.15.116:0/310954882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc018920 con 0x7f1ad8100df0 2024-03-19T02:44:36.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.465+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad81a4a80 con 0x7f1ad8100df0 2024-03-19T02:44:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ad5ffb640 1 -- 172.21.15.116:0/310954882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1acc010a90 con 0x7f1ad8100df0 2024-03-19T02:44:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 Processor -- start 2024-03-19T02:44:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 -- start start 2024-03-19T02:44:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ad5ffb640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 0x7f1ac80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1add385640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 0x7f1ac80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 0x5570de2776a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de277be0 0x5570de279fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570de1cd4b0 con 0x5570de269cb0 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ad5ffb640 1 -- 172.21.15.116:0/310954882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f1acc0cf4d0 con 0x7f1ad8100df0 2024-03-19T02:44:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570de175d70 con 0x5570de277be0 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.468+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5570de1c9550 con 0x5570de26a660 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1ad7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 0x5570de2776a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49820/0 (socket says 172.21.15.116:49820) 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 -- 172.21.15.116:0/2147619423 learned_addr learned my addr 172.21.15.116:0/2147619423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1add385640 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de277be0 0x5570de279fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 -- 172.21.15.116:0/2147619423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 msgr2=0x5570de2776a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 0x5570de2776a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 -- 172.21.15.116:0/2147619423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de277be0 msgr2=0x5570de279fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de277be0 0x5570de279fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 -- 172.21.15.116:0/2147619423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570de198210 con 0x5570de269cb0 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 0x5570de2776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1add385640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 0x7f1ac80a4e30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0004af0 tx=0x7f1ad0003bb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1adcb84640 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0981d0 tx=0x7f1acc010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d458529893f275c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1ac2ffd640 1 -- 172.21.15.116:0/2147619423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0a3ef0 con 0x5570de269cb0 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1ac2ffd640 1 -- 172.21.15.116:0/2147619423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1acc0860b0 con 0x5570de269cb0 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.469+0000 7f1ac2ffd640 1 -- 172.21.15.116:0/2147619423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc09ac50 con 0x5570de269cb0 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 -- 172.21.15.116:0/2147619423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 msgr2=0x5570de26a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0981d0 tx=0x7f1acc010ea0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 -- 172.21.15.116:0/2147619423 shutdown_connections 2024-03-19T02:44:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de277be0 0x5570de279fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de26a660 0x5570de2776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/2147619423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de26a090 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 -- 172.21.15.116:0/2147619423 >> 172.21.15.116:0/2147619423 conn(0x5570de264a00 msgr2=0x5570de264de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 -- 172.21.15.116:0/2147619423 shutdown_connections 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.470+0000 7f1ae19cc740 1 -- 172.21.15.116:0/2147619423 wait complete. 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 -- start start 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 Processor -- start 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 0x5570de275650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570de27a8e0 con 0x5570de277be0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570de189310 con 0x5570de26a660 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5570de27aa60 con 0x5570de269cb0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ad7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ad7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49456/0 (socket says 172.21.15.116:49456) 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45580/0 (socket says 172.21.15.116:45580) 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ad7fff640 1 -- 172.21.15.116:0/3343665778 learned_addr learned my addr 172.21.15.116:0/3343665778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.471+0000 7f1ae19cc740 10 client.0 _ll_register_callbacks cb 0x5570de17d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1add385640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 0x5570de275650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 -- 172.21.15.116:0/3343665778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 msgr2=0x5570de275110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 -- 172.21.15.116:0/3343665778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 msgr2=0x5570de275650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 0x5570de275650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 -- 172.21.15.116:0/3343665778 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570ddee13f0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1add385640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 0x5570de275650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1adcb84640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1acc097c60 tx=0x7f1acc0cf3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f35fe58e6c4db75 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0a3ef0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1acc0860b0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f1acc0ccac0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1acc010640 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ae19cc740 10 client.30085 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.472+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5570ddfab7d0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1acc0860b0 con 0x5570de269cb0 2024-03-19T02:44:36.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1acc018cc0 con 0x5570de269cb0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 10 client.30085 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 10 client.30085 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 20 client.30085 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de28a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 4 client.30085 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570de22b3e0 con 0x5570de2880e0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.473+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de28a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.474+0000 7f1ab7fff640 10 client.30085 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.474+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de28a580 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a97a71ffa5bd6d63 server_cookie=10d83b7f84f3c449 in_seq=0 out_seq=0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.475+0000 7f1ab7fff640 1 -- 172.21.15.116:0/3343665778 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570de22b3e0 con 0x5570de2880e0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.475+0000 7f1ab7fff640 10 client.30085 handle_command_reply: tid=0 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.475+0000 7f1ae19cc740 1 client.30085 shutdown 2024-03-19T02:44:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.475+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 msgr2=0x5570de274bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.475+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1acc097c60 tx=0x7f1acc0cf3e0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 shutdown_connections 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de28a580 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ab7fff640 0 client.30085 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de277be0 0x5570de275650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de275110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/3343665778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de269cb0 0x5570de274bd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 >> 172.21.15.116:0/3343665778 conn(0x5570de264a00 msgr2=0x5570de266750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 shutdown_connections 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 1 -- 172.21.15.116:0/3343665778 wait complete. 2024-03-19T02:44:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.476+0000 7f1ae19cc740 20 client.30085 trim_cache size 0 max 16384 2024-03-19T02:44:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.495+0000 7f1ae19cc740 1 Processor -- start 2024-03-19T02:44:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.495+0000 7f1ae19cc740 1 -- start start 2024-03-19T02:44:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de269cb0 0x5570de277300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de28fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8072fc0 con 0x5570de26a660 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81a5dd0 con 0x5570de269cb0 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8073690 con 0x5570de277be0 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49840/0 (socket says 172.21.15.116:49840) 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 -- 172.21.15.116:0/4218870171 learned_addr learned my addr 172.21.15.116:0/4218870171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1add385640 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de28fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1adcb84640 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de269cb0 0x5570de277300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 -- 172.21.15.116:0/4218870171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 msgr2=0x5570de28fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de28fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 -- 172.21.15.116:0/4218870171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de269cb0 msgr2=0x5570de277300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de269cb0 0x5570de277300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 -- 172.21.15.116:0/4218870171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570de26f9b0 con 0x5570de26a660 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40096f0 tx=0x7f1ac4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff0ea10836dd21d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.496+0000 7f1ac27fc640 1 -- 172.21.15.116:0/4218870171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4012cd0 con 0x5570de26a660 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ac27fc640 1 -- 172.21.15.116:0/4218870171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ac4036070 con 0x5570de26a660 2024-03-19T02:44:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ac27fc640 1 -- 172.21.15.116:0/4218870171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4031370 con 0x5570de26a660 2024-03-19T02:44:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 -- 172.21.15.116:0/4218870171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 msgr2=0x5570de2836f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40096f0 tx=0x7f1ac4012ba0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 -- 172.21.15.116:0/4218870171 shutdown_connections 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de28fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de26a660 0x5570de2836f0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/4218870171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de269cb0 0x5570de277300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 -- 172.21.15.116:0/4218870171 >> 172.21.15.116:0/4218870171 conn(0x5570de264a00 msgr2=0x5570de264de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 -- 172.21.15.116:0/4218870171 shutdown_connections 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.497+0000 7f1ae19cc740 1 -- 172.21.15.116:0/4218870171 wait complete. 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 -- start start 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 Processor -- start 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de27b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de2ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81a5700 con 0x5570de269cb0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81a5030 con 0x5570de26a660 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8070fa0 con 0x5570de277be0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ae19cc740 10 client.0 _ll_register_callbacks cb 0x7f1ac4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1adcb84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49848/0 (socket says 172.21.15.116:49848) 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1adcb84640 1 -- 172.21.15.116:0/1696954607 learned_addr learned my addr 172.21.15.116:0/1696954607 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1add385640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de2ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.498+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de27b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 -- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 msgr2=0x5570de2ec4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de2ec4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 -- 172.21.15.116:0/1696954607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 msgr2=0x5570de27b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de27b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 -- 172.21.15.116:0/1696954607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570de2ec9f0 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1adcb84640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0cce40 tx=0x7f1acc009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673b7a9f46f28118 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0a3ef0 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f1acc018ac0 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac4036070 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1acc018cc0 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ae19cc740 10 client.20385 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5570de2edcd0 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.499+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ac4036070 con 0x5570de269cb0 2024-03-19T02:44:36.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1acc097550 con 0x5570de269cb0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 10 client.20385 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 10 client.20385 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 20 client.20385 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de2f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 4 client.20385 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570de1da9f0 con 0x5570de2880e0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de2f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ad7fff640 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de2f02d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=711a4e7127c03bec server_cookie=647ec8016d00029d in_seq=0 out_seq=0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ab7fff640 10 client.20385 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ab7fff640 1 -- 172.21.15.116:0/1696954607 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570de1da9f0 con 0x5570de2880e0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.500+0000 7f1ab7fff640 10 client.20385 handle_command_reply: tid=0 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 client.20385 shutdown 2024-03-19T02:44:36.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 msgr2=0x5570de27aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0cce40 tx=0x7f1acc009770 comp rx=0 tx=0).stop 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 shutdown_connections 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5570de2880e0 0x5570de2f02d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570de277be0 0x5570de2ec4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570de26a660 0x5570de27b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ab7fff640 0 client.20385 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 --2- 172.21.15.116:0/1696954607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570de269cb0 0x5570de27aed0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 >> 172.21.15.116:0/1696954607 conn(0x5570de264a00 msgr2=0x5570de263310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 shutdown_connections 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 1 -- 172.21.15.116:0/1696954607 wait complete. 2024-03-19T02:44:36.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1ae19cc740 20 client.20385 trim_cache size 0 max 16384 2024-03-19T02:44:36.509 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 msgr2=0x7f1ac80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 0x7f1ac80a4e30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0004af0 tx=0x7f1ad0003bb0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 msgr2=0x7f1ad8197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.501+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00e9f0 tx=0x7f1acc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 shutdown_connections 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1ac80a2970 0x7f1ac80a4e30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ad81066e0 0x7f1ad819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad81017b0 0x7f1ad8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 --2- 172.21.15.116:0/310954882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8100df0 0x7f1ad8197f10 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 >> 172.21.15.116:0/310954882 conn(0x7f1ad8076c50 msgr2=0x7f1ad80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 shutdown_connections 2024-03-19T02:44:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.502+0000 7f1adede8640 1 -- 172.21.15.116:0/310954882 wait complete. 2024-03-19T02:44:36.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1388995218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:44:36.514 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:44:36.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 -- 172.21.15.116:0/2171273629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0103360 msgr2=0x7f7da010b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 --2- 172.21.15.116:0/2171273629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0103360 0x7f7da010b840 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009900 tx=0x7f7d9402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 -- 172.21.15.116:0/2171273629 shutdown_connections 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 --2- 172.21.15.116:0/2171273629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da010bd80 0x7f7da010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 --2- 172.21.15.116:0/2171273629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0103360 0x7f7da010b840 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.639+0000 7f7da5af8640 1 --2- 172.21.15.116:0/2171273629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da0102a40 0x7f7da0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.640+0000 7f7da5af8640 1 -- 172.21.15.116:0/2171273629 >> 172.21.15.116:0/2171273629 conn(0x7f7da00778d0 msgr2=0x7f7da0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.640+0000 7f7da5af8640 1 -- 172.21.15.116:0/2171273629 shutdown_connections 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.640+0000 7f7da5af8640 1 -- 172.21.15.116:0/2171273629 wait complete. 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.640+0000 7f7da5af8640 1 Processor -- start 2024-03-19T02:44:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.640+0000 7f7da5af8640 1 -- start start 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da0103360 0x7f7da0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da010bd80 0x7f7da019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0111a30 con 0x7f7da0102a40 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01118b0 con 0x7f7da0103360 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0111bb0 con 0x7f7da010bd80 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49868/0 (socket says 172.21.15.116:49868) 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1031848955 learned_addr learned my addr 172.21.15.116:0/1031848955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9effd640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da0103360 0x7f7da0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da010bd80 0x7f7da019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1031848955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da010bd80 msgr2=0x7f7da019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da010bd80 0x7f7da019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1031848955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da0103360 msgr2=0x7f7da0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da0103360 0x7f7da0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1031848955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da019cf70 con 0x7f7da0102a40 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9cff9640 1 -- 172.21.15.116:0/1031848955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000cd80 con 0x7f7da0102a40 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9cff9640 1 -- 172.21.15.116:0/1031848955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d900054d0 con 0x7f7da0102a40 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7d9cff9640 1 -- 172.21.15.116:0/1031848955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90010490 con 0x7f7da0102a40 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.641+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da019d260 con 0x7f7da0102a40 2024-03-19T02:44:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.642+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da01a4af0 con 0x7f7da0102a40 2024-03-19T02:44:36.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.644+0000 7f7d9cff9640 1 -- 172.21.15.116:0/1031848955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7d90005020 con 0x7f7da0102a40 2024-03-19T02:44:36.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.644+0000 7f7da87c0740 1 Processor -- start 2024-03-19T02:44:36.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 -- start start 2024-03-19T02:44:36.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9cff9640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 0x7f7d880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 0x7f7d880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f61cfa64b0 con 0x55f61d0432d0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f61cf4ed70 con 0x55f61d043bf0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7da87c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f61cfa2550 con 0x55f61d04c7d0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9cff9640 1 -- 172.21.15.116:0/1031848955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f7d900cf5b0 con 0x7f7da0102a40 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:49502/0 (socket says 172.21.15.116:49502) 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49880/0 (socket says 172.21.15.116:49880) 2024-03-19T02:44:36.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9effd640 1 -- 172.21.15.116:0/1693135563 learned_addr learned my addr 172.21.15.116:0/1693135563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1693135563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 msgr2=0x55f61d04ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1693135563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 msgr2=0x55f61d04c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/1693135563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f61cf71210 con 0x55f61d0432d0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9effd640 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 0x7f7d880a4e30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005fd0 tx=0x7f7d98011b40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.645+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7d900cc920 tx=0x7f7d9009b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80819896768d2cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7d85ffb640 1 -- 172.21.15.116:0/1693135563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900b4070 con 0x55f61d0432d0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7d85ffb640 1 -- 172.21.15.116:0/1693135563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d9009b830 con 0x55f61d0432d0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7d85ffb640 1 -- 172.21.15.116:0/1693135563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900861e0 con 0x55f61d0432d0 2024-03-19T02:44:36.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7da87c0740 1 -- 172.21.15.116:0/1693135563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 msgr2=0x55f61d0436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7da87c0740 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7d900cc920 tx=0x7f7d9009b250 comp rx=0 tx=0).stop 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7da87c0740 1 -- 172.21.15.116:0/1693135563 shutdown_connections 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.646+0000 7f7da87c0740 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d04c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 --2- 172.21.15.116:0/1693135563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d0436b0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- 172.21.15.116:0/1693135563 >> 172.21.15.116:0/1693135563 conn(0x55f61d03da00 msgr2=0x55f61d03dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- 172.21.15.116:0/1693135563 shutdown_connections 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- 172.21.15.116:0/1693135563 wait complete. 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- start start 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 Processor -- start 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d049da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f61d0539d0 con 0x55f61d0432d0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f61cf62310 con 0x55f61d043bf0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f61d053b50 con 0x55f61d04c7d0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.647+0000 7f7da87c0740 10 client.0 _ll_register_callbacks cb 0x55f61cf562d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49882/0 (socket says 172.21.15.116:49882) 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9effd640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d049da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/620247911 learned_addr learned my addr 172.21.15.116:0/620247911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/620247911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 msgr2=0x55f61d04a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 msgr2=0x55f61d049da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d049da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/620247911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f61ccba3f0 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f7d90099ae0 tx=0x7f7d900860b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baee2a00e8597439 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d7effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900b8070 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f7d90097ea0 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d9009b830 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7d900ae050 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7da87c0740 10 client.20397 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f61cd847d0 con 0x55f61d0432d0 2024-03-19T02:44:36.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.648+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d9009b830 con 0x55f61d0432d0 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f7d900a3b40 con 0x55f61d0432d0 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 10 client.20397 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 10 client.20397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 20 client.20397 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d060e50 0x55f61d063240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 4 client.20397 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f61d0043e0 con 0x55f61d060e50 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7d9effd640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d060e50 0x55f61d063240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7d7effd640 10 client.20397 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.649+0000 7f7d9effd640 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d060e50 0x55f61d063240 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e1317c636b035da server_cookie=c654793a8e775656 in_seq=0 out_seq=0 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.651+0000 7f7d7effd640 1 -- 172.21.15.116:0/620247911 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f61d0043e0 con 0x55f61d060e50 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.651+0000 7f7d7effd640 10 client.20397 handle_command_reply: tid=0 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.651+0000 7f7da87c0740 1 client.20397 shutdown 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 msgr2=0x55f61d049860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f7d90099ae0 tx=0x7f7d900860b0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 shutdown_connections 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d060e50 0x55f61d063240 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d04c7d0 0x55f61d04a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d043bf0 0x55f61d049da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7d7effd640 0 client.20397 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 --2- 172.21.15.116:0/620247911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d049860 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 >> 172.21.15.116:0/620247911 conn(0x55f61d03da00 msgr2=0x55f61d03f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 shutdown_connections 2024-03-19T02:44:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 1 -- 172.21.15.116:0/620247911 wait complete. 2024-03-19T02:44:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.652+0000 7f7da87c0740 20 client.20397 trim_cache size 0 max 16384 2024-03-19T02:44:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.671+0000 7f7da87c0740 1 Processor -- start 2024-03-19T02:44:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.671+0000 7f7da87c0740 1 -- start start 2024-03-19T02:44:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.671+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d043bf0 0x55f61d05c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d03ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7da87c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01aca10 con 0x55f61d0432d0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7da87c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ae550 con 0x55f61d04c7d0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7da87c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ad0e0 con 0x55f61d043bf0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49892/0 (socket says 172.21.15.116:49892) 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/3884974244 learned_addr learned my addr 172.21.15.116:0/3884974244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d03ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9effd640 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d043bf0 0x55f61d05c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/3884974244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d043bf0 msgr2=0x55f61d05c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d043bf0 0x55f61d05c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/3884974244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 msgr2=0x55f61d03ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d03ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 -- 172.21.15.116:0/3884974244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f61d03e5d0 con 0x55f61d0432d0 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9c0 tx=0x7f7d900057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b275ede7eeeabea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.672+0000 7f7d857fa640 1 -- 172.21.15.116:0/3884974244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9008b030 con 0x55f61d0432d0 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.673+0000 7f7d857fa640 1 -- 172.21.15.116:0/3884974244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d9009bf30 con 0x55f61d0432d0 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.673+0000 7f7d857fa640 1 -- 172.21.15.116:0/3884974244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900b8040 con 0x55f61d0432d0 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.673+0000 7f7da87c0740 1 -- 172.21.15.116:0/3884974244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 msgr2=0x55f61d04bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.673+0000 7f7da87c0740 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9c0 tx=0x7f7d900057a0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 -- 172.21.15.116:0/3884974244 shutdown_connections 2024-03-19T02:44:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d03ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d043bf0 0x55f61d05c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 --2- 172.21.15.116:0/3884974244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d0432d0 0x55f61d04bf20 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 -- 172.21.15.116:0/3884974244 >> 172.21.15.116:0/3884974244 conn(0x55f61d03da00 msgr2=0x55f61d03c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 -- 172.21.15.116:0/3884974244 shutdown_connections 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 -- 172.21.15.116:0/3884974244 wait complete. 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 -- start start 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.674+0000 7f7da87c0740 1 Processor -- start 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d0432d0 0x55f61d064470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d0bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ade80 con 0x55f61d043bf0 2024-03-19T02:44:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01ad7b0 con 0x55f61d04c7d0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7da01aaa10 con 0x55f61d0432d0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7da87c0740 10 client.0 _ll_register_callbacks cb 0x7f7d900b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49902/0 (socket says 172.21.15.116:49902) 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d0432d0 0x55f61d064470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 -- 172.21.15.116:0/2110613644 learned_addr learned my addr 172.21.15.116:0/2110613644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9ffff640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d0bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 -- 172.21.15.116:0/2110613644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d0432d0 msgr2=0x55f61d064470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d0432d0 0x55f61d064470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 -- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 msgr2=0x55f61d0bfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d0bfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.675+0000 7f7d9effd640 1 -- 172.21.15.116:0/2110613644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f61d0c04a0 con 0x55f61d043bf0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7d9effd640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7d9402c210 tx=0x7f7d94005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6ef70afe7fc4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7d7effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9402c880 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f7d9402ca20 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d9009bf30 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7d94057050 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7da87c0740 10 client.20403 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.676+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f61d0c78d0 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d9009bf30 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f7d940047d0 con 0x55f61d043bf0 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 10 client.20403 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 10 client.20403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 20 client.20403 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d0c8080 0x55f61d0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 4 client.20403 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f61cfb39f0 con 0x55f61d0c8080 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d0c8080 0x55f61d0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.677+0000 7f7d7effd640 10 client.20403 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.678+0000 7f7d9f7fe640 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d0c8080 0x55f61d0ca4a0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8c5cac45ff29925 server_cookie=3151e45a2986c5e7 in_seq=0 out_seq=0 2024-03-19T02:44:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7d7effd640 1 -- 172.21.15.116:0/2110613644 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f61cfb39f0 con 0x55f61d0c8080 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7d7effd640 10 client.20403 handle_command_reply: tid=0 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 client.20403 shutdown 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 msgr2=0x55f61d0649b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7d9402c210 tx=0x7f7d94005fd0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 shutdown_connections 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f61d0c8080 0x55f61d0ca4a0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f61d04c7d0 0x55f61d0bfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7d7effd640 0 client.20403 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61d043bf0 0x55f61d0649b0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 --2- 172.21.15.116:0/2110613644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f61d0432d0 0x55f61d064470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 >> 172.21.15.116:0/2110613644 conn(0x55f61d03da00 msgr2=0x55f61d05c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.680+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 shutdown_connections 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da87c0740 1 -- 172.21.15.116:0/2110613644 wait complete. 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da87c0740 20 client.20403 trim_cache size 0 max 16384 2024-03-19T02:44:36.686 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:44:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 msgr2=0x7f7d880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 0x7f7d880a4e30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005fd0 tx=0x7f7d98011b40 comp rx=0 tx=0).stop 2024-03-19T02:44:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 msgr2=0x7f7da0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000edf0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 shutdown_connections 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7d880a2970 0x7f7d880a4e30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7da010bd80 0x7f7da019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7da0103360 0x7f7da0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 --2- 172.21.15.116:0/1031848955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da0102a40 0x7f7da0197fd0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 >> 172.21.15.116:0/1031848955 conn(0x7f7da00778d0 msgr2=0x7f7da00fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 shutdown_connections 2024-03-19T02:44:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:36.681+0000 7f7da5af8640 1 -- 172.21.15.116:0/1031848955 wait complete. 2024-03-19T02:44:37.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27509 6.11 MB/sec execute 119 sec latency 187.626 ms 2024-03-19T02:44:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:37 smithi162 ceph-mon[35761]: pgmap v529: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:37 smithi154 ceph-mon[35693]: pgmap v529: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:37 smithi116 ceph-mon[32263]: pgmap v529: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:37.696 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:44:37.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:44:37.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 -- 172.21.15.116:0/1881078518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8654102170 msgr2=0x7f865410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 --2- 172.21.15.116:0/1881078518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8654102170 0x7f865410e680 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f864c009cd0 tx=0x7f864c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 -- 172.21.15.116:0/1881078518 shutdown_connections 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 --2- 172.21.15.116:0/1881078518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8654102170 0x7f865410e680 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 --2- 172.21.15.116:0/1881078518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541017b0 0x7f8654101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 --2- 172.21.15.116:0/1881078518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541077c0 0x7f8654107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 -- 172.21.15.116:0/1881078518 >> 172.21.15.116:0/1881078518 conn(0x7f8654078be0 msgr2=0x7f8654078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.834+0000 7f86596bf640 1 -- 172.21.15.116:0/1881078518 shutdown_connections 2024-03-19T02:44:37.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.835+0000 7f86596bf640 1 -- 172.21.15.116:0/1881078518 wait complete. 2024-03-19T02:44:37.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.835+0000 7f86596bf640 1 Processor -- start 2024-03-19T02:44:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.835+0000 7f86596bf640 1 -- start start 2024-03-19T02:44:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541017b0 0x7f8654197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86541077c0 0x7f865419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8654111a80 con 0x7f8654102170 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8654111900 con 0x7f86541017b0 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86596bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8654111c00 con 0x7f86541077c0 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46196/0 (socket says 172.21.15.116:46196) 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 -- 172.21.15.116:0/2435061089 learned_addr learned my addr 172.21.15.116:0/2435061089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86537fe640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86541077c0 0x7f865419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f8652ffd640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541017b0 0x7f8654197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 -- 172.21.15.116:0/2435061089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86541077c0 msgr2=0x7f865419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86541077c0 0x7f865419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 -- 172.21.15.116:0/2435061089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541017b0 msgr2=0x7f8654197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541017b0 0x7f8654197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 -- 172.21.15.116:0/2435061089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865419cf90 con 0x7f8654102170 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.836+0000 7f86527fc640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea60 tx=0x7f864800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.837+0000 7f863bfff640 1 -- 172.21.15.116:0/2435061089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800cde0 con 0x7f8654102170 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.837+0000 7f863bfff640 1 -- 172.21.15.116:0/2435061089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86480035f0 con 0x7f8654102170 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.837+0000 7f863bfff640 1 -- 172.21.15.116:0/2435061089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648010690 con 0x7f8654102170 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.837+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865419d280 con 0x7f8654102170 2024-03-19T02:44:37.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.837+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86541a4b10 con 0x7f8654102170 2024-03-19T02:44:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.839+0000 7f863bfff640 1 -- 172.21.15.116:0/2435061089 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f86480037a0 con 0x7f8654102170 2024-03-19T02:44:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 Processor -- start 2024-03-19T02:44:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 -- start start 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d435760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 0x556a2d440c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a2d399160 con 0x556a2d43e880 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a2d341e10 con 0x556a2d435380 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f865c381740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a2d395200 con 0x556a2d435ca0 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f863bfff640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 0x7f863c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f86537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d435760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55338/0 (socket says 172.21.15.116:55338) 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 -- 172.21.15.116:0/1755818424 learned_addr learned my addr 172.21.15.116:0/1755818424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f86527fc640 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 0x556a2d440c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f86537fe640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 0x7f863c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 -- 172.21.15.116:0/1755818424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 msgr2=0x556a2d435760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d435760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 -- 172.21.15.116:0/1755818424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 msgr2=0x556a2d440c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 0x556a2d440c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f8652ffd640 1 -- 172.21.15.116:0/1755818424 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a2d1757e0 con 0x556a2d435ca0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f86527fc640 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 0x556a2d440c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.840+0000 7f863bfff640 1 -- 172.21.15.116:0/2435061089 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f86480cf480 con 0x7f8654102170 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.841+0000 7f8652ffd640 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8644009c70 tx=0x7f8644011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca750758e89a0ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.841+0000 7f86537fe640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 0x7f863c0a4eb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f864c005cc0 tx=0x7f864c02c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.841+0000 7f86397fa640 1 -- 172.21.15.116:0/1755818424 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644004230 con 0x556a2d435ca0 2024-03-19T02:44:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.841+0000 7f86397fa640 1 -- 172.21.15.116:0/1755818424 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86440043d0 con 0x556a2d435ca0 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.841+0000 7f86397fa640 1 -- 172.21.15.116:0/1755818424 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86440196d0 con 0x556a2d435ca0 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- 172.21.15.116:0/1755818424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 msgr2=0x556a2d43e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8644009c70 tx=0x7f8644011810 comp rx=0 tx=0).stop 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- 172.21.15.116:0/1755818424 shutdown_connections 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d43e880 0x556a2d440c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d435ca0 0x556a2d43e340 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- 172.21.15.116:0/1755818424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d435760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- 172.21.15.116:0/1755818424 >> 172.21.15.116:0/1755818424 conn(0x556a2d42fab0 msgr2=0x556a2d42fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- 172.21.15.116:0/1755818424 shutdown_connections 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- 172.21.15.116:0/1755818424 wait complete. 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- start start 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 Processor -- start 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d43b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a2d4416c0 con 0x556a2d435380 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a2d3551a0 con 0x556a2d435ca0 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a2d441840 con 0x556a2d43e880 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.842+0000 7f865c381740 10 client.0 _ll_register_callbacks cb 0x556a2d0acc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55340/0 (socket says 172.21.15.116:55340) 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86527fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 -- 172.21.15.116:0/342794818 learned_addr learned my addr 172.21.15.116:0/342794818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f8652ffd640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d43b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 -- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 msgr2=0x556a2d43bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 -- 172.21.15.116:0/342794818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 msgr2=0x556a2d43b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d43b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 -- 172.21.15.116:0/342794818 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a2d1d5f70 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f8652ffd640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d43b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f86537fe640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f864c036000 tx=0x7f864c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e5c3cab99b70585 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f8636ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c053060 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86440043d0 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.843+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f864c004450 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f864c05a050 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 10 client.30121 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556a2d06d170 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86440043d0 con 0x556a2d43e880 2024-03-19T02:44:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f864c03f070 con 0x556a2d43e880 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 10 client.30121 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 10 client.30121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 20 client.30121 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d44ef60 0x556a2d4513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 4 client.30121 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a2d3a66a0 con 0x556a2d44ef60 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.844+0000 7f8652ffd640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d44ef60 0x556a2d4513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.845+0000 7f8652ffd640 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d44ef60 0x556a2d4513c0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1e6835fe1dedd6c server_cookie=5628e612d58f9978 in_seq=0 out_seq=0 2024-03-19T02:44:37.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.845+0000 7f8636ff5640 10 client.30121 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f8636ff5640 1 -- 172.21.15.116:0/342794818 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a2d3a66a0 con 0x556a2d44ef60 2024-03-19T02:44:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f8636ff5640 10 client.30121 handle_command_reply: tid=0 2024-03-19T02:44:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 client.30121 shutdown 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 msgr2=0x556a2d43c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f864c036000 tx=0x7f864c0095c0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 shutdown_connections 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d44ef60 0x556a2d4513c0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d43c210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 --2- 172.21.15.116:0/342794818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d43b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 >> 172.21.15.116:0/342794818 conn(0x556a2d42fab0 msgr2=0x556a2d431800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.846+0000 7f8636ff5640 0 client.30121 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.847+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 shutdown_connections 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.847+0000 7f865c381740 1 -- 172.21.15.116:0/342794818 wait complete. 2024-03-19T02:44:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.847+0000 7f865c381740 20 client.30121 trim_cache size 0 max 16384 2024-03-19T02:44:37.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.869+0000 7f865c381740 1 Processor -- start 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.869+0000 7f865c381740 1 -- start start 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.869+0000 7f865c381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.869+0000 7f865c381740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.869+0000 7f865c381740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d42e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f865c381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86541aed30 con 0x556a2d435380 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f865c381740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86541b0870 con 0x556a2d435ca0 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f865c381740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86541af400 con 0x556a2d43e880 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46222/0 (socket says 172.21.15.116:46222) 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 -- 172.21.15.116:0/2842865295 learned_addr learned my addr 172.21.15.116:0/2842865295 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f86527fc640 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f86537fe640 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d42e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 -- 172.21.15.116:0/2842865295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 msgr2=0x556a2d42e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d42e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 -- 172.21.15.116:0/2842865295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 msgr2=0x556a2d43c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 -- 172.21.15.116:0/2842865295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a2d42f8e0 con 0x556a2d435380 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8652ffd640 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x556a2d434e50 tx=0x7f8644009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68e15c2fa38e268 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8638ff9640 1 -- 172.21.15.116:0/2842865295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644031070 con 0x556a2d435380 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.870+0000 7f8638ff9640 1 -- 172.21.15.116:0/2842865295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8644003aa0 con 0x556a2d435380 2024-03-19T02:44:37.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f8638ff9640 1 -- 172.21.15.116:0/2842865295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864401a6a0 con 0x556a2d435380 2024-03-19T02:44:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 -- 172.21.15.116:0/2842865295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 msgr2=0x556a2d44a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x556a2d434e50 tx=0x7f8644009680 comp rx=0 tx=0).stop 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 -- 172.21.15.116:0/2842865295 shutdown_connections 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d42e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435ca0 0x556a2d43c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 --2- 172.21.15.116:0/2842865295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435380 0x556a2d44a460 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.871+0000 7f865c381740 1 -- 172.21.15.116:0/2842865295 >> 172.21.15.116:0/2842865295 conn(0x556a2d42fab0 msgr2=0x556a2d42e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- 172.21.15.116:0/2842865295 shutdown_connections 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- 172.21.15.116:0/2842865295 wait complete. 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- start start 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 Processor -- start 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 0x556a2d442030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d442570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86541b01a0 con 0x556a2d435ca0 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86541afad0 con 0x556a2d435380 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86541acd30 con 0x556a2d43e880 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f865c381740 10 client.0 _ll_register_callbacks cb 0x7f8644005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.872+0000 7f86527fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 0x556a2d442030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8652ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38254/0 (socket says 172.21.15.116:38254) 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8652ffd640 1 -- 172.21.15.116:0/3451175176 learned_addr learned my addr 172.21.15.116:0/3451175176 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86537fe640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d442570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 -- 172.21.15.116:0/3451175176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 msgr2=0x556a2d442570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d442570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 -- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 msgr2=0x556a2d446bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 -- 172.21.15.116:0/3451175176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a2d442c10 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8652ffd640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f86527fc640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 0x556a2d442030 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f86480cc7f0 tx=0x7f86480d1990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcdf95758229e8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8636ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864809b060 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8644003aa0 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7f8648097d70 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.873+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8648003b30 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 10 client.20421 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556a2d4b5220 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8644003aa0 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f864809ab50 con 0x556a2d435ca0 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 10 client.20421 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:37.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 10 client.20421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 20 client.20421 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d4b59a0 0x556a2d4b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 4 client.20421 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a2d3a09f0 con 0x556a2d4b59a0 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.874+0000 7f8652ffd640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d4b59a0 0x556a2d4b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.875+0000 7f8636ff5640 10 client.20421 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.875+0000 7f8652ffd640 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d4b59a0 0x556a2d4b7dc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e78cd84201696b6 server_cookie=5d5e732c523e76bc in_seq=0 out_seq=0 2024-03-19T02:44:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f8636ff5640 1 -- 172.21.15.116:0/3451175176 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a2d3a09f0 con 0x556a2d4b59a0 2024-03-19T02:44:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f8636ff5640 10 client.20421 handle_command_reply: tid=0 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 client.20421 shutdown 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 msgr2=0x556a2d442030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 0x556a2d442030 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f86480cc7f0 tx=0x7f86480d1990 comp rx=0 tx=0).stop 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 shutdown_connections 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556a2d4b59a0 0x556a2d4b7dc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f8636ff5640 0 client.20421 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2d43e880 0x556a2d442570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a2d435ca0 0x556a2d442030 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 --2- 172.21.15.116:0/3451175176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a2d435380 0x556a2d446bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 >> 172.21.15.116:0/3451175176 conn(0x556a2d42fab0 msgr2=0x556a2d430ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.879+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 shutdown_connections 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f865c381740 1 -- 172.21.15.116:0/3451175176 wait complete. 2024-03-19T02:44:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f865c381740 20 client.20421 trim_cache size 0 max 16384 2024-03-19T02:44:37.882 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:37.882 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:44:37.882 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "da0ea8b0-0b89-4950-94f0-f834d41c57f4", 2024-03-19T02:44:37.882 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:44:37.882 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 msgr2=0x7f863c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 0x7f863c0a4eb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f864c005cc0 tx=0x7f864c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 msgr2=0x7f86541984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.880+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea60 tx=0x7f864800ef20 comp rx=0 tx=0).stop 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 shutdown_connections 2024-03-19T02:44:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f863c0a29f0 0x7f863c0a4eb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86541077c0 0x7f865419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8654102170 0x7f86541984e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 --2- 172.21.15.116:0/2435061089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86541017b0 0x7f8654197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 >> 172.21.15.116:0/2435061089 conn(0x7f8654078be0 msgr2=0x7f86541041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 shutdown_connections 2024-03-19T02:44:37.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:37.881+0000 7f86596bf640 1 -- 172.21.15.116:0/2435061089 wait complete. 2024-03-19T02:44:37.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:44:37.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 -- 172.21.15.116:0/1497651702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c068a20 msgr2=0x7fa66c068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 --2- 172.21.15.116:0/1497651702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c068a20 0x7fa66c068ea0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fa660009930 tx=0x7fa66002bc30 comp rx=0 tx=0).stop 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 -- 172.21.15.116:0/1497651702 shutdown_connections 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 --2- 172.21.15.116:0/1497651702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c0693e0 0x7fa66c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 --2- 172.21.15.116:0/1497651702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c068a20 0x7fa66c068ea0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 --2- 172.21.15.116:0/1497651702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c107870 0x7fa66c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 -- 172.21.15.116:0/1497651702 >> 172.21.15.116:0/1497651702 conn(0x7fa66c077820 msgr2=0x7fa66c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 -- 172.21.15.116:0/1497651702 shutdown_connections 2024-03-19T02:44:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 -- 172.21.15.116:0/1497651702 wait complete. 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.121+0000 7fa673cc0640 1 Processor -- start 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 -- start start 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 0x7fa66c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c107870 0x7fa66c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c111a70 con 0x7fa66c0693e0 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1118f0 con 0x7fa66c068a20 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa673cc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c111bf0 con 0x7fa66c107870 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa671a5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 0x7fa66c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46260/0 (socket says 172.21.15.116:46260) 2024-03-19T02:44:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 -- 172.21.15.116:0/675253141 learned_addr learned my addr 172.21.15.116:0/675253141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67225d640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c107870 0x7fa66c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 -- 172.21.15.116:0/675253141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c107870 msgr2=0x7fa66c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c107870 0x7fa66c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 -- 172.21.15.116:0/675253141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 msgr2=0x7fa66c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 0x7fa66c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.122+0000 7fa67125b640 1 -- 172.21.15.116:0/675253141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c19cf90 con 0x7fa66c0693e0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa671a5c640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 0x7fa66c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa67125b640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fa660005940 tx=0x7fa66002cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa65effd640 1 -- 172.21.15.116:0/675253141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660043020 con 0x7fa66c0693e0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa65effd640 1 -- 172.21.15.116:0/675253141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6600043d0 con 0x7fa66c0693e0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c19d220 con 0x7fa66c0693e0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa65effd640 1 -- 172.21.15.116:0/675253141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660003ae0 con 0x7fa66c0693e0 2024-03-19T02:44:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.123+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c1a4b10 con 0x7fa66c0693e0 2024-03-19T02:44:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.126+0000 7fa65effd640 1 -- 172.21.15.116:0/675253141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa660046020 con 0x7fa66c0693e0 2024-03-19T02:44:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 Processor -- start 2024-03-19T02:44:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 -- start start 2024-03-19T02:44:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa65effd640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 0x7fa6580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d0359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d035f30 0x55d04d038320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d04cf90500 con 0x55d04d02d350 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d04cf38be0 con 0x55d04d035f30 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa676985740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d04cf8c5a0 con 0x55d04d02ca30 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa671a5c640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 0x7fa6580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa67225d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa67225d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55396/0 (socket says 172.21.15.116:55396) 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa65effd640 1 -- 172.21.15.116:0/675253141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7fa6600f0c00 con 0x7fa66c0693e0 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa67225d640 1 -- 172.21.15.116:0/835764037 learned_addr learned my addr 172.21.15.116:0/835764037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa67225d640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d035f30 0x55d04d038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.127+0000 7fa67125b640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d0359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 -- 172.21.15.116:0/835764037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d035f30 msgr2=0x55d04d038320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d035f30 0x55d04d038320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 -- 172.21.15.116:0/835764037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 msgr2=0x55d04d0359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d0359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 -- 172.21.15.116:0/835764037 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d04cf5a000 con 0x55d04d02ca30 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67125b640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d0359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa671a5c640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 0x7fa6580a4e30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa664005fd0 tx=0x7fa664009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa67225d640 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa668014970 tx=0x7fa66800c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420f1ebf551bea91 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa657fff640 1 -- 172.21.15.116:0/835764037 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6680142d0 con 0x55d04d02ca30 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa657fff640 1 -- 172.21.15.116:0/835764037 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6680054d0 con 0x55d04d02ca30 2024-03-19T02:44:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.128+0000 7fa657fff640 1 -- 172.21.15.116:0/835764037 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66800f760 con 0x55d04d02ca30 2024-03-19T02:44:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- 172.21.15.116:0/835764037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 msgr2=0x55d04d02ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa668014970 tx=0x7fa66800c4f0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- 172.21.15.116:0/835764037 shutdown_connections 2024-03-19T02:44:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d035f30 0x55d04d038320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d0359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 --2- 172.21.15.116:0/835764037 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d02ce10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- 172.21.15.116:0/835764037 >> 172.21.15.116:0/835764037 conn(0x55d04d027160 msgr2=0x55d04d027540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- 172.21.15.116:0/835764037 shutdown_connections 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- 172.21.15.116:0/835764037 wait complete. 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.129+0000 7fa676985740 1 -- start start 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 Processor -- start 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 0x55d04d0333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d033900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d04d038d20 con 0x55d04d035f30 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d04cf4c160 con 0x55d04d02ca30 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d04d038ea0 con 0x55d04d02d350 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa676985740 10 client.0 _ll_register_callbacks cb 0x55d04cf88a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa671a5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa671a5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38292/0 (socket says 172.21.15.116:38292) 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa671a5c640 1 -- 172.21.15.116:0/262694886 learned_addr learned my addr 172.21.15.116:0/262694886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa67125b640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 0x55d04d0333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.130+0000 7fa67225d640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d033900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 -- 172.21.15.116:0/262694886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 msgr2=0x55d04d0333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 0x55d04d0333c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 -- 172.21.15.116:0/262694886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 msgr2=0x55d04d033900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d033900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 -- 172.21.15.116:0/262694886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04cd6e7d0 con 0x55d04d02ca30 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa67125b640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 0x55d04d0333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa67225d640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d033900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa671a5c640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa664003dc0 tx=0x7fa664004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7687cd0bc1302b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa6557fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66401e830 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6680054d0 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7fa66401e9d0 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa66402f050 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa676985740 10 client.30164 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.131+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d04cca43f0 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6680054d0 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa664026c20 con 0x55d04d02ca30 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 10 client.30164 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 10 client.30164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 20 client.30164 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d046290 0x55d04d048700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 4 client.30164 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:44:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d04cf9da40 con 0x55d04d046290 2024-03-19T02:44:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa67125b640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d046290 0x55d04d048700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa67125b640 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d046290 0x55d04d048700 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425f21095d687247 server_cookie=273d31eaa83c18d5 in_seq=0 out_seq=0 2024-03-19T02:44:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.132+0000 7fa6557fa640 10 client.30164 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa6557fa640 1 -- 172.21.15.116:0/262694886 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d04cf9da40 con 0x55d04d046290 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa6557fa640 10 client.30164 handle_command_reply: tid=0 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 client.30164 shutdown 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 msgr2=0x55d04d032e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa664003dc0 tx=0x7fa664004ab0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 shutdown_connections 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d046290 0x55d04d048700 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d033900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.136+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02d350 0x55d04d0333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa676985740 1 --2- 172.21.15.116:0/262694886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d032e80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 >> 172.21.15.116:0/262694886 conn(0x55d04d027160 msgr2=0x55d04d028eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa6557fa640 0 client.30164 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 shutdown_connections 2024-03-19T02:44:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa676985740 1 -- 172.21.15.116:0/262694886 wait complete. 2024-03-19T02:44:38.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.137+0000 7fa676985740 20 client.30164 trim_cache size 0 max 16384 2024-03-19T02:44:38.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.158+0000 7fa676985740 1 Processor -- start 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.158+0000 7fa676985740 1 -- start start 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d0355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d035f30 0x55d04d041af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1aeb70 con 0x55d04d02d350 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1b06b0 con 0x55d04d02ca30 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa676985740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1af240 con 0x55d04d035f30 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46272/0 (socket says 172.21.15.116:46272) 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 -- 172.21.15.116:0/351666547 learned_addr learned my addr 172.21.15.116:0/351666547 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:38.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67225d640 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d035f30 0x55d04d041af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa671a5c640 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d0355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 -- 172.21.15.116:0/351666547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d035f30 msgr2=0x55d04d041af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d035f30 0x55d04d041af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 -- 172.21.15.116:0/351666547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 msgr2=0x55d04d0355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d0355b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 -- 172.21.15.116:0/351666547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d04cc640f0 con 0x55d04d02d350 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa67125b640 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b9960 tx=0x7fa6600dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653f8bb6c8de765a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.159+0000 7fa6577fe640 1 -- 172.21.15.116:0/351666547 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600c10e0 con 0x55d04d02d350 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa6577fe640 1 -- 172.21.15.116:0/351666547 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6600c5730 con 0x55d04d02d350 2024-03-19T02:44:38.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa6577fe640 1 -- 172.21.15.116:0/351666547 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600bbac0 con 0x55d04d02d350 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 -- 172.21.15.116:0/351666547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 msgr2=0x55d04d031eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b9960 tx=0x7fa6600dd040 comp rx=0 tx=0).stop 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 -- 172.21.15.116:0/351666547 shutdown_connections 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d035f30 0x55d04d041af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d02d350 0x55d04d031eb0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 --2- 172.21.15.116:0/351666547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02ca30 0x55d04d0355b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 -- 172.21.15.116:0/351666547 >> 172.21.15.116:0/351666547 conn(0x55d04d027160 msgr2=0x55d04d028500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 -- 172.21.15.116:0/351666547 shutdown_connections 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.160+0000 7fa676985740 1 -- 172.21.15.116:0/351666547 wait complete. 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 -- start start 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 Processor -- start 2024-03-19T02:44:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 0x55d04d0394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1affe0 con 0x55d04d035f30 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1af910 con 0x55d04d02d350 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1acb70 con 0x55d04d02ca30 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa676985740 10 client.0 _ll_register_callbacks cb 0x7fa66400fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa67225d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa671a5c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa67125b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 0x55d04d0394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa67225d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46278/0 (socket says 172.21.15.116:46278) 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa671a5c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55430/0 (socket says 172.21.15.116:55430) 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.161+0000 7fa67225d640 1 -- 172.21.15.116:0/1563970700 learned_addr learned my addr 172.21.15.116:0/1563970700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 -- 172.21.15.116:0/1563970700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 msgr2=0x55d04d03e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 -- 172.21.15.116:0/1563970700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 msgr2=0x55d04d0399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 -- 172.21.15.116:0/1563970700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d04d039f30 con 0x55d04d02d350 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa671a5c640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67225d640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa67125b640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 0x55d04d0394b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55d04d0324c0 tx=0x7fa6600bd110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf22051ac875afc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa6557fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660003c80 con 0x55d04d02d350 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9510+0+0 (secure 0 0 0) 0x7fa6600bb550 con 0x55d04d02d350 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6600c5730 con 0x55d04d02d350 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa660003e20 con 0x55d04d02d350 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.162+0000 7fa676985740 10 client.30170 fetch_fsmap learned FSMap version 38 2024-03-19T02:44:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d04d0ac2d0 con 0x55d04d02d350 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6600c5730 con 0x55d04d02d350 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa6600c52a0 con 0x55d04d02d350 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 10 client.30170 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 10 client.30170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 20 client.30170 populate_metadata read hostname 'smithi116' 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d0ac940 0x55d04d0aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 4 client.30170 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d04d032530 con 0x55d04d0ac940 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.163+0000 7fa671a5c640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d0ac940 0x55d04d0aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.164+0000 7fa6557fa640 10 client.30170 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.164+0000 7fa671a5c640 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d0ac940 0x55d04d0aed60 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b481dcdea1abdd6 server_cookie=3d683e889822f0e4 in_seq=0 out_seq=0 2024-03-19T02:44:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa6557fa640 1 -- 172.21.15.116:0/1563970700 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55d04d032530 con 0x55d04d0ac940 2024-03-19T02:44:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa6557fa640 10 client.30170 handle_command_reply: tid=0 2024-03-19T02:44:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 client.30170 shutdown 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 msgr2=0x55d04d0394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 0x55d04d0394b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55d04d0324c0 tx=0x7fa6600bd110 comp rx=0 tx=0).stop 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 shutdown_connections 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d04d0ac940 0x55d04d0aed60 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d04d035f30 0x55d04d0399f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04d02d350 0x55d04d0394b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 --2- 172.21.15.116:0/1563970700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d04d02ca30 0x55d04d03e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa6557fa640 0 client.30170 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 >> 172.21.15.116:0/1563970700 conn(0x55d04d027160 msgr2=0x55d04d031d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 shutdown_connections 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 1 -- 172.21.15.116:0/1563970700 wait complete. 2024-03-19T02:44:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.166+0000 7fa676985740 20 client.30170 trim_cache size 0 max 16384 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (356 inodes in the stack)", 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "da0ea8b0-0b89-4950-94f0-f834d41c57f4": { 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "da0ea8b0-0b89-4950-94f0-f834d41c57f4", 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:44:38.171 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:44:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 msgr2=0x7fa6580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 0x7fa6580a4e30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa664005fd0 tx=0x7fa664009500 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 msgr2=0x7fa66c198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fa660005940 tx=0x7fa66002cd20 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 shutdown_connections 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa6580a2970 0x7fa6580a4e30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa66c107870 0x7fa66c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa66c0693e0 0x7fa66c198530 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 --2- 172.21.15.116:0/675253141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa66c068a20 0x7fa66c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 >> 172.21.15.116:0/675253141 conn(0x7fa66c077820 msgr2=0x7fa66c0fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 shutdown_connections 2024-03-19T02:44:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:44:38.167+0000 7fa673cc0640 1 -- 172.21.15.116:0/675253141 wait complete. 2024-03-19T02:44:38.182 INFO:tasks.cephfs.filesystem:scrub status for tag:da0ea8b0-0b89-4950-94f0-f834d41c57f4 - {'path': '/', 'tag': 'da0ea8b0-0b89-4950-94f0-f834d41c57f4', 'options': 'recursive,force'} 2024-03-19T02:44:38.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27601 6.07 MB/sec execute 120 sec latency 232.039 ms 2024-03-19T02:44:38.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:38 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:38.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:38 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:38 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:38 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:38 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:38 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27715 6.08 MB/sec execute 121 sec latency 156.544 ms 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:39 smithi116 ceph-mon[32263]: pgmap v530: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:44:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:39 smithi162 ceph-mon[35761]: pgmap v530: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:44:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:44:39.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:39 smithi154 ceph-mon[35693]: pgmap v530: 129 pgs: 129 active+clean; 865 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 128 op/s 2024-03-19T02:44:40.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27843 6.10 MB/sec execute 122 sec latency 137.120 ms 2024-03-19T02:44:41.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 27964 6.11 MB/sec execute 123 sec latency 143.153 ms 2024-03-19T02:44:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:41 smithi162 ceph-mon[35761]: pgmap v531: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-03-19T02:44:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:41 smithi154 ceph-mon[35693]: pgmap v531: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-03-19T02:44:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:41 smithi116 ceph-mon[32263]: pgmap v531: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 227 op/s 2024-03-19T02:44:42.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28067 6.18 MB/sec execute 124 sec latency 131.592 ms 2024-03-19T02:44:43.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28163 6.15 MB/sec execute 125 sec latency 138.617 ms 2024-03-19T02:44:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:43 smithi162 ceph-mon[35761]: pgmap v532: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 165 op/s 2024-03-19T02:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:43 smithi154 ceph-mon[35693]: pgmap v532: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 165 op/s 2024-03-19T02:44:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:43 smithi116 ceph-mon[32263]: pgmap v532: 129 pgs: 129 active+clean; 859 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 165 op/s 2024-03-19T02:44:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28279 6.18 MB/sec execute 126 sec latency 142.217 ms 2024-03-19T02:44:45.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28388 6.14 MB/sec execute 127 sec latency 153.248 ms 2024-03-19T02:44:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:45 smithi162 ceph-mon[35761]: pgmap v533: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:44:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:45 smithi154 ceph-mon[35693]: pgmap v533: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:44:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:45 smithi116 ceph-mon[32263]: pgmap v533: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-19T02:44:46.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28491 6.11 MB/sec execute 128 sec latency 132.784 ms 2024-03-19T02:44:47.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28630 6.10 MB/sec execute 129 sec latency 125.969 ms 2024-03-19T02:44:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:47 smithi162 ceph-mon[35761]: pgmap v534: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:47 smithi154 ceph-mon[35693]: pgmap v534: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:47 smithi116 ceph-mon[32263]: pgmap v534: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:48.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28737 6.06 MB/sec execute 130 sec latency 127.429 ms 2024-03-19T02:44:49.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28880 6.03 MB/sec execute 131 sec latency 124.812 ms 2024-03-19T02:44:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:49 smithi162 ceph-mon[35761]: pgmap v535: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:49 smithi154 ceph-mon[35693]: pgmap v535: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:49.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:49 smithi116 ceph-mon[32263]: pgmap v535: 129 pgs: 129 active+clean; 867 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-19T02:44:50.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 28992 6.01 MB/sec execute 132 sec latency 118.278 ms 2024-03-19T02:44:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:50.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:50.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:51.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29142 6.03 MB/sec execute 133 sec latency 133.848 ms 2024-03-19T02:44:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:44:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:51 smithi162 ceph-mon[35761]: pgmap v536: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-03-19T02:44:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:44:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:51 smithi154 ceph-mon[35693]: pgmap v536: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-03-19T02:44:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:44:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:51 smithi116 ceph-mon[32263]: pgmap v536: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 253 op/s 2024-03-19T02:44:52.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29240 6.02 MB/sec execute 134 sec latency 175.838 ms 2024-03-19T02:44:53.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:53 smithi154 ceph-mon[35693]: pgmap v537: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-03-19T02:44:53.325 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29329 6.01 MB/sec execute 135 sec latency 131.188 ms 2024-03-19T02:44:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:53 smithi162 ceph-mon[35761]: pgmap v537: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-03-19T02:44:53.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:53 smithi116 ceph-mon[32263]: pgmap v537: 129 pgs: 129 active+clean; 870 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-03-19T02:44:54.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29447 6.04 MB/sec execute 136 sec latency 167.472 ms 2024-03-19T02:44:55.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29559 6.04 MB/sec execute 137 sec latency 162.459 ms 2024-03-19T02:44:55.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:55 smithi162 ceph-mon[35761]: pgmap v538: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 231 op/s 2024-03-19T02:44:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:55 smithi154 ceph-mon[35693]: pgmap v538: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 231 op/s 2024-03-19T02:44:55.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:55 smithi116 ceph-mon[32263]: pgmap v538: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 231 op/s 2024-03-19T02:44:56.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29679 6.06 MB/sec execute 138 sec latency 168.699 ms 2024-03-19T02:44:57.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29857 6.12 MB/sec execute 139 sec latency 160.697 ms 2024-03-19T02:44:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:57 smithi162 ceph-mon[35761]: pgmap v539: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:44:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:57 smithi154 ceph-mon[35693]: pgmap v539: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:44:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:57 smithi116 ceph-mon[32263]: pgmap v539: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:44:58.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 29995 6.14 MB/sec execute 140 sec latency 136.838 ms 2024-03-19T02:44:59.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30181 6.19 MB/sec execute 141 sec latency 140.985 ms 2024-03-19T02:44:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:44:59 smithi162 ceph-mon[35761]: pgmap v540: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:44:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:44:59 smithi154 ceph-mon[35693]: pgmap v540: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:44:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:44:59 smithi116 ceph-mon[32263]: pgmap v540: 129 pgs: 129 active+clean; 881 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-03-19T02:45:00.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30298 6.21 MB/sec execute 142 sec latency 138.909 ms 2024-03-19T02:45:01.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30379 6.18 MB/sec execute 143 sec latency 231.693 ms 2024-03-19T02:45:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:01 smithi162 ceph-mon[35761]: pgmap v541: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-03-19T02:45:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:01 smithi154 ceph-mon[35693]: pgmap v541: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-03-19T02:45:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:01 smithi116 ceph-mon[32263]: pgmap v541: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 227 op/s 2024-03-19T02:45:02.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30482 6.20 MB/sec execute 144 sec latency 160.470 ms 2024-03-19T02:45:03.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30510 6.16 MB/sec execute 145 sec latency 148.990 ms 2024-03-19T02:45:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:03 smithi162 ceph-mon[35761]: pgmap v542: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-03-19T02:45:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:03 smithi154 ceph-mon[35693]: pgmap v542: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-03-19T02:45:03.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:03 smithi116 ceph-mon[32263]: pgmap v542: 129 pgs: 129 active+clean; 898 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 144 op/s 2024-03-19T02:45:04.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30601 6.18 MB/sec execute 146 sec latency 159.165 ms 2024-03-19T02:45:05.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30685 6.16 MB/sec execute 147 sec latency 159.675 ms 2024-03-19T02:45:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:05 smithi162 ceph-mon[35761]: osdmap e118: 12 total, 12 up, 12 in 2024-03-19T02:45:05.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:05 smithi162 ceph-mon[35761]: pgmap v544: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 164 op/s 2024-03-19T02:45:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:05 smithi154 ceph-mon[35693]: osdmap e118: 12 total, 12 up, 12 in 2024-03-19T02:45:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:05 smithi154 ceph-mon[35693]: pgmap v544: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 164 op/s 2024-03-19T02:45:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:05 smithi116 ceph-mon[32263]: osdmap e118: 12 total, 12 up, 12 in 2024-03-19T02:45:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:05 smithi116 ceph-mon[32263]: pgmap v544: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 164 op/s 2024-03-19T02:45:06.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30807 6.21 MB/sec execute 148 sec latency 170.263 ms 2024-03-19T02:45:07.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 30907 6.18 MB/sec execute 149 sec latency 202.525 ms 2024-03-19T02:45:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:07 smithi162 ceph-mon[35761]: osdmap e119: 12 total, 12 up, 12 in 2024-03-19T02:45:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:07 smithi162 ceph-mon[35761]: pgmap v546: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 206 op/s 2024-03-19T02:45:08.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:45:08.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:45:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:07 smithi154 ceph-mon[35693]: osdmap e119: 12 total, 12 up, 12 in 2024-03-19T02:45:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:07 smithi154 ceph-mon[35693]: pgmap v546: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 206 op/s 2024-03-19T02:45:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:07 smithi116 ceph-mon[32263]: osdmap e119: 12 total, 12 up, 12 in 2024-03-19T02:45:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:07 smithi116 ceph-mon[32263]: pgmap v546: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 206 op/s 2024-03-19T02:45:08.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31038 6.22 MB/sec execute 150 sec latency 145.515 ms 2024-03-19T02:45:08.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 -- 172.21.15.116:0/3457790175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c1077d0 msgr2=0x7fdc5c107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/3457790175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c107bb0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009930 tx=0x7fdc5802bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 -- 172.21.15.116:0/3457790175 shutdown_connections 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/3457790175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c102180 0x7fdc5c10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/3457790175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/3457790175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c107bb0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.410+0000 7fdc63a0b640 1 -- 172.21.15.116:0/3457790175 >> 172.21.15.116:0/3457790175 conn(0x7fdc5c0fd250 msgr2=0x7fdc5c0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- 172.21.15.116:0/3457790175 shutdown_connections 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- 172.21.15.116:0/3457790175 wait complete. 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 Processor -- start 2024-03-19T02:45:08.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- start start 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c111a90 con 0x7fdc5c102180 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c111910 con 0x7fdc5c1077d0 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.411+0000 7fdc63a0b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c111c10 con 0x7fdc5c1017c0 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44542/0 (socket says 172.21.15.116:44542) 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54048/0 (socket says 172.21.15.116:54048) 2024-03-19T02:45:08.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 -- 172.21.15.116:0/202255597 learned_addr learned my addr 172.21.15.116:0/202255597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 -- 172.21.15.116:0/202255597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c1077d0 msgr2=0x7fdc5c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 -- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 msgr2=0x7fdc5c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 -- 172.21.15.116:0/202255597 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5c19d000 con 0x7fdc5c1017c0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc617a7640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009230 tx=0x7fdc5802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc527fc640 1 -- 172.21.15.116:0/202255597 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58043020 con 0x7fdc5c1017c0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc5c19d290 con 0x7fdc5c1017c0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc527fc640 1 -- 172.21.15.116:0/202255597 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5802ca70 con 0x7fdc5c1017c0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc5c1a4b80 con 0x7fdc5c1017c0 2024-03-19T02:45:08.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.412+0000 7fdc527fc640 1 -- 172.21.15.116:0/202255597 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58003a50 con 0x7fdc5c1017c0 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.415+0000 7fdc527fc640 1 -- 172.21.15.116:0/202255597 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdc58046020 con 0x7fdc5c1017c0 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.415+0000 7fdc665f4740 1 Processor -- start 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.415+0000 7fdc665f4740 1 -- start start 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e5b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc527fc640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 0x7fdc480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e5bf30 0x55c154e5e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c154db6500 con 0x55c154e53350 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c154d5ebe0 con 0x55c154e52a30 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc665f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c154db25a0 con 0x55c154e5bf30 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc617a7640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 0x7fdc480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34232/0 (socket says 172.21.15.116:34232) 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 -- 172.21.15.116:0/694283488 learned_addr learned my addr 172.21.15.116:0/694283488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e5b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e5bf30 0x55c154e5e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 -- 172.21.15.116:0/694283488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e5bf30 msgr2=0x55c154e5e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e5bf30 0x55c154e5e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 -- 172.21.15.116:0/694283488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 msgr2=0x55c154e5b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e5b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 -- 172.21.15.116:0/694283488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c154d80000 con 0x55c154e52a30 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc527fc640 1 -- 172.21.15.116:0/202255597 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fdc580f10f0 con 0x7fdc5c1017c0 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc54005f40 tx=0x7fdc54005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c7dc011b5a43116 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc617a7640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 0x7fdc480a4e30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdc580b9e50 tx=0x7fdc580dd040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.416+0000 7fdc477fe640 1 -- 172.21.15.116:0/694283488 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54004b20 con 0x55c154e52a30 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.417+0000 7fdc477fe640 1 -- 172.21.15.116:0/694283488 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5400fd30 con 0x55c154e52a30 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.417+0000 7fdc477fe640 1 -- 172.21.15.116:0/694283488 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc540054b0 con 0x55c154e52a30 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- 172.21.15.116:0/694283488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 msgr2=0x55c154e52e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc54005f40 tx=0x7fdc54005bd0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- 172.21.15.116:0/694283488 shutdown_connections 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e5bf30 0x55c154e5e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e5b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 --2- 172.21.15.116:0/694283488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e52e10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- 172.21.15.116:0/694283488 >> 172.21.15.116:0/694283488 conn(0x55c154e4d160 msgr2=0x55c154e4d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- 172.21.15.116:0/694283488 shutdown_connections 2024-03-19T02:45:08.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- 172.21.15.116:0/694283488 wait complete. 2024-03-19T02:45:08.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 -- start start 2024-03-19T02:45:08.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.418+0000 7fdc665f4740 1 Processor -- start 2024-03-19T02:45:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e66180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c154e5ed20 con 0x55c154e5bf30 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c154d72160 con 0x55c154e52a30 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c154e5eea0 con 0x55c154e53350 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc665f4740 10 client.0 _ll_register_callbacks cb 0x55c154daea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54074/0 (socket says 172.21.15.116:54074) 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc61fa8640 1 -- 172.21.15.116:0/4057297042 learned_addr learned my addr 172.21.15.116:0/4057297042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e66180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 -- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 msgr2=0x55c154e66180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e66180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 -- 172.21.15.116:0/4057297042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 msgr2=0x55c154e60590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 -- 172.21.15.116:0/4057297042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c154b947d0 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e666c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0ff9c0 tx=0x7fdc4c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d89685fda21abb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.419+0000 7fdc44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00fe60 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fdc4c012110 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5400fd30 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdc4c025050 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 10 client.30157 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c154aca3f0 con 0x55c154e53350 2024-03-19T02:45:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc5400fd30 con 0x55c154e53350 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdc4c003cb0 con 0x55c154e53350 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 10 client.30157 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 10 client.30157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 20 client.30157 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154e70780 0x55c154e72ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 4 client.30157 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.420+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c154dc3a40 con 0x55c154e70780 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.421+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154e70780 0x55c154e72ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.421+0000 7fdc44ff9640 10 client.30157 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.421+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154e70780 0x55c154e72ba0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b179109643dbf4 server_cookie=ba6e07d1b1d18520 in_seq=0 out_seq=0 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc44ff9640 1 -- 172.21.15.116:0/4057297042 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c154dc3a40 con 0x55c154e70780 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc44ff9640 10 client.30157 handle_command_reply: tid=0 2024-03-19T02:45:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 client.30157 shutdown 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 msgr2=0x55c154e666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e666c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0ff9c0 tx=0x7fdc4c00f790 comp rx=0 tx=0).stop 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 shutdown_connections 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154e70780 0x55c154e72ba0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc44ff9640 0 client.30157 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e666c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 --2- 172.21.15.116:0/4057297042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e66180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.423+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 >> 172.21.15.116:0/4057297042 conn(0x55c154e4d160 msgr2=0x55c154e4eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.424+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 shutdown_connections 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.424+0000 7fdc665f4740 1 -- 172.21.15.116:0/4057297042 wait complete. 2024-03-19T02:45:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.424+0000 7fdc665f4740 20 client.30157 trim_cache size 0 max 16384 2024-03-19T02:45:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 Processor -- start 2024-03-19T02:45:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 -- start start 2024-03-19T02:45:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e5f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1acae0 con 0x55c154e5bf30 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1ae620 con 0x55c154e52a30 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc665f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1ad1b0 con 0x55c154e53350 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44564/0 (socket says 172.21.15.116:44564) 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2994771805 learned_addr learned my addr 172.21.15.116:0/2994771805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54088/0 (socket says 172.21.15.116:54088) 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.445+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e5f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2994771805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 msgr2=0x55c154e5f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e5f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2994771805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 msgr2=0x55c154e60490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2994771805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c154a8a0f0 con 0x55c154e53350 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009710 tx=0x7fdc4c009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c060a07c8e116f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc46ffd640 1 -- 172.21.15.116:0/2994771805 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c02e070 con 0x55c154e53350 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc46ffd640 1 -- 172.21.15.116:0/2994771805 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc4c012110 con 0x55c154e53350 2024-03-19T02:45:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.446+0000 7fdc46ffd640 1 -- 172.21.15.116:0/2994771805 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0055c0 con 0x55c154e53350 2024-03-19T02:45:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- 172.21.15.116:0/2994771805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 msgr2=0x55c154e66560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c009710 tx=0x7fdc4c009f80 comp rx=0 tx=0).stop 2024-03-19T02:45:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- 172.21.15.116:0/2994771805 shutdown_connections 2024-03-19T02:45:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e5bf30 0x55c154e60490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e53350 0x55c154e66560 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2994771805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e52a30 0x55c154e5f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- 172.21.15.116:0/2994771805 >> 172.21.15.116:0/2994771805 conn(0x55c154e4d160 msgr2=0x55c154e4b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- 172.21.15.116:0/2994771805 shutdown_connections 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- 172.21.15.116:0/2994771805 wait complete. 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 -- start start 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.447+0000 7fdc665f4740 1 Processor -- start 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e52a30 0x55c154e6a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e5bf30 0x55c154ed5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1adf50 con 0x55c154e53350 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1ad880 con 0x55c154e5bf30 2024-03-19T02:45:08.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5c1aaae0 con 0x55c154e52a30 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc665f4740 10 client.0 _ll_register_callbacks cb 0x7fdc4c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc61fa8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e52a30 0x55c154e6a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54092/0 (socket says 172.21.15.116:54092) 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2900842936 learned_addr learned my addr 172.21.15.116:0/2900842936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e5bf30 0x55c154ed5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2900842936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e52a30 msgr2=0x55c154e6a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e52a30 0x55c154e6a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e5bf30 msgr2=0x55c154ed5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e5bf30 0x55c154ed5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 -- 172.21.15.116:0/2900842936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c154ed5fb0 con 0x55c154e53350 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc60fa6640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00a010 tx=0x7fdc4c0108f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b541ca010f6d5fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.448+0000 7fdc44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c02e040 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc4c012110 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fdc4c010ae0 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdc4c03d050 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 10 client.20457 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c154ed72c0 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc4c012110 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdc4c003760 con 0x55c154e53350 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 10 client.20457 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 10 client.20457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 20 client.20457 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154ed79a0 0x55c154ed9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 4 client.20457 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:45:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c154dbdd90 con 0x55c154ed79a0 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.449+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154ed79a0 0x55c154ed9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.450+0000 7fdc44ff9640 10 client.20457 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.450+0000 7fdc61fa8640 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154ed79a0 0x55c154ed9dc0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e54f35cbd1706cac server_cookie=8ec2d23775e58886 in_seq=0 out_seq=0 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.451+0000 7fdc44ff9640 1 -- 172.21.15.116:0/2900842936 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c154dbdd90 con 0x55c154ed79a0 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.451+0000 7fdc44ff9640 10 client.20457 handle_command_reply: tid=0 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.451+0000 7fdc665f4740 1 client.20457 shutdown 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 msgr2=0x55c154e6adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00a010 tx=0x7fdc4c0108f0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 shutdown_connections 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c154ed79a0 0x55c154ed9dc0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c154e5bf30 0x55c154ed5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc44ff9640 0 client.20457 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c154e53350 0x55c154e6adb0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 --2- 172.21.15.116:0/2900842936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c154e52a30 0x55c154e6a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 >> 172.21.15.116:0/2900842936 conn(0x55c154e4d160 msgr2=0x55c154e4b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 shutdown_connections 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 1 -- 172.21.15.116:0/2900842936 wait complete. 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc665f4740 20 client.20457 trim_cache size 0 max 16384 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:45:08.458 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.452+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 msgr2=0x7fdc480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 0x7fdc480a4e30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdc580b9e50 tx=0x7fdc580dd040 comp rx=0 tx=0).stop 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 msgr2=0x7fdc5c198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009230 tx=0x7fdc5802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 shutdown_connections 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc480a2970 0x7fdc480a4e30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc5c1077d0 0x7fdc5c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc5c102180 0x7fdc5c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 --2- 172.21.15.116:0/202255597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc5c1017c0 0x7fdc5c198010 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 >> 172.21.15.116:0/202255597 conn(0x7fdc5c0fd250 msgr2=0x7fdc5c0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 shutdown_connections 2024-03-19T02:45:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.453+0000 7fdc63a0b640 1 -- 172.21.15.116:0/202255597 wait complete. 2024-03-19T02:45:08.470 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:45:08.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 -- 172.21.15.116:0/3503413270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc568106730 msgr2=0x7fc568106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 --2- 172.21.15.116:0/3503413270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc568106730 0x7fc568106b10 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fc558009930 tx=0x7fc55802bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 -- 172.21.15.116:0/3503413270 shutdown_connections 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 --2- 172.21.15.116:0/3503413270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5681010e0 0x7fc56810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 --2- 172.21.15.116:0/3503413270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568100720 0x7fc568100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 --2- 172.21.15.116:0/3503413270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc568106730 0x7fc568106b10 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 -- 172.21.15.116:0/3503413270 >> 172.21.15.116:0/3503413270 conn(0x7fc568077820 msgr2=0x7fc568077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 -- 172.21.15.116:0/3503413270 shutdown_connections 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.595+0000 7fc56d485640 1 -- 172.21.15.116:0/3503413270 wait complete. 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 Processor -- start 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 -- start start 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc568100720 0x7fc568196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 0x7fc56819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5681109b0 con 0x7fc5681010e0 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc568110830 con 0x7fc568100720 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc56d485640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc568110b30 con 0x7fc568106730 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc5667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc5667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54118/0 (socket says 172.21.15.116:54118) 2024-03-19T02:45:08.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc5667fc640 1 -- 172.21.15.116:0/3421888644 learned_addr learned my addr 172.21.15.116:0/3421888644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc5677fe640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 0x7fc56819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.596+0000 7fc566ffd640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc568100720 0x7fc568196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 -- 172.21.15.116:0/3421888644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 msgr2=0x7fc56819b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 0x7fc56819b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 -- 172.21.15.116:0/3421888644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc568100720 msgr2=0x7fc568196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc568100720 0x7fc568196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 -- 172.21.15.116:0/3421888644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56819bf60 con 0x7fc5681010e0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5677fe640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 0x7fc56819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc5667fc640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00eb00 tx=0x7fc55c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c01c070 con 0x7fc5681010e0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56819c250 con 0x7fc5681010e0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc55c00bc30 con 0x7fc5681010e0 2024-03-19T02:45:08.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c016610 con 0x7fc5681010e0 2024-03-19T02:45:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.597+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc568068980 con 0x7fc5681010e0 2024-03-19T02:45:08.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.598+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc568068610 con 0x7fc5681010e0 2024-03-19T02:45:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.599+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc55c01f070 con 0x7fc5681010e0 2024-03-19T02:45:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.600+0000 7fc54ffff640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 0x7fc5500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.600+0000 7fc566ffd640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 0x7fc5500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.600+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fc55c0d0820 con 0x7fc5681010e0 2024-03-19T02:45:08.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.601+0000 7fc566ffd640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 0x7fc5500a4e30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc558005ab0 tx=0x7fc558005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:08.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.606+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc55c00bde0 con 0x7fc5681010e0 2024-03-19T02:45:08.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.907+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc568069400 con 0x7fc5681010e0 2024-03-19T02:45:08.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.908+0000 7fc54ffff640 1 -- 172.21.15.116:0/3421888644 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7fc55c09cb50 con 0x7fc5681010e0 2024-03-19T02:45:08.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:45:08.909 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:45:08.909 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.910+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 msgr2=0x7fc5500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.910+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 0x7fc5500a4e30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc558005ab0 tx=0x7fc558005fd0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 msgr2=0x7fc5681974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00eb00 tx=0x7fc55c009f20 comp rx=0 tx=0).stop 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 shutdown_connections 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc5500a2970 0x7fc5500a4e30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc568106730 0x7fc56819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5681010e0 0x7fc5681974b0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 --2- 172.21.15.116:0/3421888644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc568100720 0x7fc568196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:08.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 >> 172.21.15.116:0/3421888644 conn(0x7fc568077820 msgr2=0x7fc568103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:08.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 shutdown_connections 2024-03-19T02:45:08.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:08.911+0000 7fc56d485640 1 -- 172.21.15.116:0/3421888644 wait complete. 2024-03-19T02:45:08.923 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 -- 172.21.15.116:0/963872806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 msgr2=0x7f719c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 --2- 172.21.15.116:0/963872806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c100b20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f718c009930 tx=0x7f718c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 -- 172.21.15.116:0/963872806 shutdown_connections 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 --2- 172.21.15.116:0/963872806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 0x7f719c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 --2- 172.21.15.116:0/963872806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c100b20 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 --2- 172.21.15.116:0/963872806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 0x7f719c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 -- 172.21.15.116:0/963872806 >> 172.21.15.116:0/963872806 conn(0x7f719c0778d0 msgr2=0x7f719c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 -- 172.21.15.116:0/963872806 shutdown_connections 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.046+0000 7f71a42cc640 1 -- 172.21.15.116:0/963872806 wait complete. 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 Processor -- start 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 -- start start 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 0x7f719c199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 0x7f719c1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f719c110970 con 0x7f719c1006a0 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1107f0 con 0x7f719c1066b0 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a42cc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f719c110af0 con 0x7f719c101060 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54136/0 (socket says 172.21.15.116:54136) 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 -- 172.21.15.116:0/1859387529 learned_addr learned my addr 172.21.15.116:0/1859387529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a1867640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 0x7f719c199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2869640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 0x7f719c1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 -- 172.21.15.116:0/1859387529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 msgr2=0x7f719c199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 0x7f719c199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 -- 172.21.15.116:0/1859387529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 msgr2=0x7f719c1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 0x7f719c1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 -- 172.21.15.116:0/1859387529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c1a43b0 con 0x7f719c1006a0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.047+0000 7f71a2068640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f719800e9f0 tx=0x7f719800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.048+0000 7f71937fe640 1 -- 172.21.15.116:0/1859387529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719800cd80 con 0x7f719c1006a0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.048+0000 7f71937fe640 1 -- 172.21.15.116:0/1859387529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71980108e0 con 0x7f719c1006a0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.048+0000 7f71937fe640 1 -- 172.21.15.116:0/1859387529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198018920 con 0x7f719c1006a0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.048+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c1a4580 con 0x7f719c1006a0 2024-03-19T02:45:09.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.048+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c1a4aa0 con 0x7f719c1006a0 2024-03-19T02:45:09.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.050+0000 7f71937fe640 1 -- 172.21.15.116:0/1859387529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7198010a90 con 0x7f719c1006a0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71937fe640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 0x7f71880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 Processor -- start 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a1867640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 0x7f71880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71937fe640 1 -- 172.21.15.116:0/1859387529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f71980cf4d0 con 0x7f719c1006a0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 -- start start 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6253290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d6255bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558d61ad4b0 con 0x5558d624a2d0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5558d6155d70 con 0x5558d62537d0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.051+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5558d61a9550 con 0x5558d624abf0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54152/0 (socket says 172.21.15.116:54152) 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 -- 172.21.15.116:0/291707900 learned_addr learned my addr 172.21.15.116:0/291707900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a1867640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 0x7f71880a4e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f718c005ab0 tx=0x7f718c005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a1867640 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6253290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2869640 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d6255bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 -- 172.21.15.116:0/291707900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 msgr2=0x5558d6253290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6253290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 -- 172.21.15.116:0/291707900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 msgr2=0x5558d6255bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d6255bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 -- 172.21.15.116:0/291707900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558d6178210 con 0x5558d624a2d0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f71a2068640 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f71980981d0 tx=0x7f7198010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f567c5c10a40c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f7190bfe640 1 -- 172.21.15.116:0/291707900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71980a3ef0 con 0x5558d624a2d0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f7190bfe640 1 -- 172.21.15.116:0/291707900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71980860b0 con 0x5558d624a2d0 2024-03-19T02:45:09.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.052+0000 7f7190bfe640 1 -- 172.21.15.116:0/291707900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719809ac50 con 0x5558d624a2d0 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 -- 172.21.15.116:0/291707900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 msgr2=0x5558d624a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f71980981d0 tx=0x7f7198010ea0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 -- 172.21.15.116:0/291707900 shutdown_connections 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d6255bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6253290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/291707900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d624a6b0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 -- 172.21.15.116:0/291707900 >> 172.21.15.116:0/291707900 conn(0x5558d6244a00 msgr2=0x5558d6244de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 -- 172.21.15.116:0/291707900 shutdown_connections 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.053+0000 7f71a6f8b740 1 -- 172.21.15.116:0/291707900 wait complete. 2024-03-19T02:45:09.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 -- start start 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 Processor -- start 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624abf0 0x5558d6250d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 0x5558d62512d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558d6256500 con 0x5558d624a2d0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5558d6169310 con 0x5558d624abf0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5558d6256680 con 0x5558d62537d0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a6f8b740 10 client.0 _ll_register_callbacks cb 0x5558d615d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54156/0 (socket says 172.21.15.116:54156) 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 -- 172.21.15.116:0/2074230051 learned_addr learned my addr 172.21.15.116:0/2074230051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2869640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 0x5558d62512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a1867640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624abf0 0x5558d6250d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 -- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 msgr2=0x5558d62512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 0x5558d62512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 -- 172.21.15.116:0/2074230051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624abf0 msgr2=0x5558d6250d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624abf0 0x5558d6250d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2068640 1 -- 172.21.15.116:0/2074230051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558d5ec13f0 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.054+0000 7f71a2869640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 0x5558d62512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a2068640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f71980cf7f0 tx=0x7f719800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1388178fc965d550 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71980a3ef0 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f7198010430 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71980860b0 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7198010650 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 10 client.20475 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5558d5f8b7d0 con 0x5558d624a2d0 2024-03-19T02:45:09.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71980860b0 con 0x5558d624a2d0 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f71980975c0 con 0x5558d624a2d0 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 10 client.20475 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 10 client.20475 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.055+0000 7f71a6f8b740 20 client.20475 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d6263970 0x5558d6265d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f71a6f8b740 4 client.20475 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558d620b3e0 con 0x5558d6263970 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f71a1867640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d6263970 0x5558d6265d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f7185ffb640 10 client.20475 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.056+0000 7f71a1867640 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d6263970 0x5558d6265d60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6cccfad0d79b2b43 server_cookie=1c2b2a683b2e1a41 in_seq=0 out_seq=0 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f7185ffb640 1 -- 172.21.15.116:0/2074230051 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558d620b3e0 con 0x5558d6263970 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f7185ffb640 10 client.20475 handle_command_reply: tid=0 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 client.20475 shutdown 2024-03-19T02:45:09.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 msgr2=0x5558d6250850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f71980cf7f0 tx=0x7f719800cf90 comp rx=0 tx=0).stop 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 shutdown_connections 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d6263970 0x5558d6265d60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d62537d0 0x5558d62512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624abf0 0x5558d6250d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f7185ffb640 0 client.20475 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/2074230051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624a2d0 0x5558d6250850 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 >> 172.21.15.116:0/2074230051 conn(0x5558d6244a00 msgr2=0x5558d6246750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 shutdown_connections 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.057+0000 7f71a6f8b740 1 -- 172.21.15.116:0/2074230051 wait complete. 2024-03-19T02:45:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.058+0000 7f71a6f8b740 20 client.20475 trim_cache size 0 max 16384 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 Processor -- start 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 -- start start 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624a2d0 0x5558d6252f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d62510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1aed70 con 0x5558d624abf0 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1b08b0 con 0x5558d62537d0 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1af440 con 0x5558d624a2d0 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a2068640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624a2d0 0x5558d6252f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a1867640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a1867640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54172/0 (socket says 172.21.15.116:54172) 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a1867640 1 -- 172.21.15.116:0/985358432 learned_addr learned my addr 172.21.15.116:0/985358432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.076+0000 7f71a2869640 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d62510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 -- 172.21.15.116:0/985358432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624a2d0 msgr2=0x5558d6252f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624a2d0 0x5558d6252f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 -- 172.21.15.116:0/985358432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 msgr2=0x5558d62510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d62510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 -- 172.21.15.116:0/985358432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558d624f800 con 0x5558d624abf0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a2869640 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d62510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f71a1867640 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f718c009900 tx=0x7f718c042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e883baa61e9aa69f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f7187fff640 1 -- 172.21.15.116:0/985358432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c02c9e0 con 0x5558d624abf0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f7187fff640 1 -- 172.21.15.116:0/985358432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f718c05b070 con 0x5558d624abf0 2024-03-19T02:45:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.077+0000 7f7187fff640 1 -- 172.21.15.116:0/985358432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c005340 con 0x5558d624abf0 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 -- 172.21.15.116:0/985358432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 msgr2=0x5558d623fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f718c009900 tx=0x7f718c042000 comp rx=0 tx=0).stop 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 -- 172.21.15.116:0/985358432 shutdown_connections 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d62537d0 0x5558d62510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d624abf0 0x5558d623fa30 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/985358432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624a2d0 0x5558d6252f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 -- 172.21.15.116:0/985358432 >> 172.21.15.116:0/985358432 conn(0x5558d6244a00 msgr2=0x5558d6243ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 -- 172.21.15.116:0/985358432 shutdown_connections 2024-03-19T02:45:09.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.078+0000 7f71a6f8b740 1 -- 172.21.15.116:0/985358432 wait complete. 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 -- start start 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 Processor -- start 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624a2d0 0x5558d6268fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 0x5558d62668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1b01e0 con 0x5558d62537d0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1afb10 con 0x5558d624a2d0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f719c1acd70 con 0x5558d624abf0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a6f8b740 10 client.0 _ll_register_callbacks cb 0x7f7194009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a2068640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624a2d0 0x5558d6268fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44664/0 (socket says 172.21.15.116:44664) 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a2869640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 0x5558d62668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 -- 172.21.15.116:0/1955237113 learned_addr learned my addr 172.21.15.116:0/1955237113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 -- 172.21.15.116:0/1955237113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624a2d0 msgr2=0x5558d6268fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624a2d0 0x5558d6268fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 -- 172.21.15.116:0/1955237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 msgr2=0x5558d62668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 0x5558d62668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a1867640 1 -- 172.21.15.116:0/1955237113 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558d6267070 con 0x5558d624abf0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.079+0000 7f71a2869640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 0x5558d62668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f71a1867640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f718c02c170 tx=0x7f718c02cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ce256948bc3881 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c064060 con 0x5558d624abf0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f718c05b070 con 0x5558d624abf0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f718c005180 con 0x5558d624abf0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f718c06b050 con 0x5558d624abf0 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f71a6f8b740 10 client.30187 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5558d62c98b0 con 0x5558d624abf0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f718c05b070 con 0x5558d624abf0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.080+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f71980cf1b0 con 0x5558d624abf0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 10 client.30187 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 10 client.30187 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 20 client.30187 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d62ca000 0x5558d62cc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 4 client.30187 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558d61ba9f0 con 0x5558d62ca000 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a2068640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d62ca000 0x5558d62cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a2068640 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d62ca000 0x5558d62cc420 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d34557b38e04e67 server_cookie=676d4a8d52bf6617 in_seq=0 out_seq=0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f7185ffb640 10 client.30187 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f7185ffb640 1 -- 172.21.15.116:0/1955237113 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558d61ba9f0 con 0x5558d62ca000 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f7185ffb640 10 client.30187 handle_command_reply: tid=0 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.081+0000 7f71a6f8b740 1 client.30187 shutdown 2024-03-19T02:45:09.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 msgr2=0x5558d6266360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f718c02c170 tx=0x7f718c02cdd0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 shutdown_connections 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5558d62ca000 0x5558d62cc420 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558d62537d0 0x5558d62668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558d624abf0 0x5558d6266360 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 --2- 172.21.15.116:0/1955237113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558d624a2d0 0x5558d6268fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f7185ffb640 0 client.30187 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 >> 172.21.15.116:0/1955237113 conn(0x5558d6244a00 msgr2=0x5558d625e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 shutdown_connections 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 1 -- 172.21.15.116:0/1955237113 wait complete. 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.082+0000 7f71a6f8b740 20 client.30187 trim_cache size 0 max 16384 2024-03-19T02:45:09.088 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 msgr2=0x7f71880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 0x7f71880a4e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f718c005ab0 tx=0x7f718c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 msgr2=0x7f719c198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f719800e9f0 tx=0x7f719800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 shutdown_connections 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f71880a2970 0x7f71880a4e30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f719c1066b0 0x7f719c1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f719c101060 0x7f719c199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 --2- 172.21.15.116:0/1859387529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f719c1006a0 0x7f719c198df0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 >> 172.21.15.116:0/1859387529 conn(0x7f719c0778d0 msgr2=0x7f719c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 shutdown_connections 2024-03-19T02:45:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.083+0000 7f71a42cc640 1 -- 172.21.15.116:0/1859387529 wait complete. 2024-03-19T02:45:09.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.220+0000 7ff291d37640 1 -- 172.21.15.116:0/1829548216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c1066b0 msgr2=0x7ff28c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 --2- 172.21.15.116:0/1829548216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c1066b0 0x7ff28c106a90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00b9f0 tx=0x7ff27c02d8e0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 -- 172.21.15.116:0/1829548216 shutdown_connections 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 --2- 172.21.15.116:0/1829548216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c101060 0x7ff28c10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 --2- 172.21.15.116:0/1829548216 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 --2- 172.21.15.116:0/1829548216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c1066b0 0x7ff28c106a90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 -- 172.21.15.116:0/1829548216 >> 172.21.15.116:0/1829548216 conn(0x7ff28c0778d0 msgr2=0x7ff28c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 -- 172.21.15.116:0/1829548216 shutdown_connections 2024-03-19T02:45:09.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 -- 172.21.15.116:0/1829548216 wait complete. 2024-03-19T02:45:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.221+0000 7ff291d37640 1 Processor -- start 2024-03-19T02:45:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 -- start start 2024-03-19T02:45:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 0x7ff28c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 0x7ff28c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c113c80 con 0x7ff28c1066b0 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c113b00 con 0x7ff28c101060 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff291d37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c113e00 con 0x7ff28c1006a0 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 0x7ff28c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44672/0 (socket says 172.21.15.116:44672) 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff28b7fe640 1 -- 172.21.15.116:0/2051302762 learned_addr learned my addr 172.21.15.116:0/2051302762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.222+0000 7ff28affd640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 0x7ff28c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 -- 172.21.15.116:0/2051302762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 msgr2=0x7ff28c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 0x7ff28c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 -- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 msgr2=0x7ff28c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 0x7ff28c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 -- 172.21.15.116:0/2051302762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c19f140 con 0x7ff28c1006a0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28bfff640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 0x7ff28c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28affd640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 0x7ff28c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff27c000c00 tx=0x7ff27c009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff288ff9640 1 -- 172.21.15.116:0/2051302762 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c037040 con 0x7ff28c1006a0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c19f3d0 con 0x7ff28c1006a0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff288ff9640 1 -- 172.21.15.116:0/2051302762 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff27c005040 con 0x7ff28c1006a0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff288ff9640 1 -- 172.21.15.116:0/2051302762 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c004b80 con 0x7ff28c1006a0 2024-03-19T02:45:09.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.223+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1a6cc0 con 0x7ff28c1006a0 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff288ff9640 1 -- 172.21.15.116:0/2051302762 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff27c03a020 con 0x7ff28c1006a0 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 Processor -- start 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 -- start start 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b946b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff288ff9640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 0x7ff2740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2af74b0 con 0x55c5a2b942d0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2a9fd70 con 0x55c5a2b94bf0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff2949f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2af3550 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.226+0000 7ff28affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44678/0 (socket says 172.21.15.116:44678) 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 -- 172.21.15.116:0/2953347568 learned_addr learned my addr 172.21.15.116:0/2953347568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28bfff640 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b946b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff288ff9640 1 -- 172.21.15.116:0/2051302762 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7ff27c0f1040 con 0x7ff28c1006a0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28bfff640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 0x7ff2740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 -- 172.21.15.116:0/2953347568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b94bf0 msgr2=0x55c5a2b9d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 -- 172.21.15.116:0/2953347568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 msgr2=0x55c5a2b946b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b946b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 -- 172.21.15.116:0/2953347568 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5a2ac2210 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28bfff640 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b946b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:09.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff28affd640 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2800114f0 tx=0x7ff2800119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f48df4b7555e8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff271ffb640 1 -- 172.21.15.116:0/2953347568 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280005020 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff271ffb640 1 -- 172.21.15.116:0/2953347568 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2800047a0 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.227+0000 7ff271ffb640 1 -- 172.21.15.116:0/2953347568 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28000f8c0 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- 172.21.15.116:0/2953347568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 msgr2=0x55c5a2b9fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff28bfff640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 0x7ff2740a4e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff284005c40 tx=0x7ff284009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2800114f0 tx=0x7ff2800119b0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- 172.21.15.116:0/2953347568 shutdown_connections 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9fbc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 --2- 172.21.15.116:0/2953347568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b946b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- 172.21.15.116:0/2953347568 >> 172.21.15.116:0/2953347568 conn(0x55c5a2b8ea00 msgr2=0x55c5a2b8ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- 172.21.15.116:0/2953347568 shutdown_connections 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- 172.21.15.116:0/2953347568 wait complete. 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.228+0000 7ff2949f9740 1 -- start start 2024-03-19T02:45:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 Processor -- start 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2ba05c0 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2ab3310 con 0x55c5a2b942d0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c5a2ba0740 con 0x55c5a2b94bf0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff2949f9740 10 client.0 _ll_register_callbacks cb 0x55c5a2aa72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff28affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34384/0 (socket says 172.21.15.116:34384) 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff28b7fe640 1 -- 172.21.15.116:0/1299858838 learned_addr learned my addr 172.21.15.116:0/1299858838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.229+0000 7ff28bfff640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 -- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 msgr2=0x55c5a2b9ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 -- 172.21.15.116:0/1299858838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b9d7d0 msgr2=0x55c5a2b9b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 -- 172.21.15.116:0/1299858838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5a280b3f0 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28affd640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0b9c90 tx=0x7ff27c00e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9e8a1d5c83d2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0bd2c0 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7ff27c005560 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2800047a0 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff27c0d6150 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff2949f9740 10 client.30224 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c5a28d57d0 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2800047a0 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.230+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7ff27c04c030 con 0x55c5a2b942d0 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 10 client.30224 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 10 client.30224 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 20 client.30224 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2bada90 0x55c5a2baff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 4 client.30224 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5a2b553e0 con 0x55c5a2bada90 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff28affd640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2bada90 0x55c5a2baff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff28affd640 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2bada90 0x55c5a2baff00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b48c768a114bcfe server_cookie=994f3f3c5e46a5ca in_seq=0 out_seq=0 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.231+0000 7ff26affd640 10 client.30224 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff26affd640 1 -- 172.21.15.116:0/1299858838 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c5a2b553e0 con 0x55c5a2bada90 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff26affd640 10 client.30224 handle_command_reply: tid=0 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 client.30224 shutdown 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 msgr2=0x55c5a2b9a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0b9c90 tx=0x7ff27c00e000 comp rx=0 tx=0).stop 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 shutdown_connections 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2bada90 0x55c5a2baff00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2b9b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b9ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff26affd640 0 client.30224 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 --2- 172.21.15.116:0/1299858838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9a7a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 >> 172.21.15.116:0/1299858838 conn(0x55c5a2b8ea00 msgr2=0x55c5a2b90750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 shutdown_connections 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 1 -- 172.21.15.116:0/1299858838 wait complete. 2024-03-19T02:45:09.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.232+0000 7ff2949f9740 20 client.30224 trim_cache size 0 max 16384 2024-03-19T02:45:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:08 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3421888644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.255+0000 7ff2949f9740 1 Processor -- start 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.255+0000 7ff2949f9740 1 -- start start 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2bb57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b0c20 con 0x55c5a2b942d0 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b2760 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff2949f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b12f0 con 0x55c5a2b94bf0 2024-03-19T02:45:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.256+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54226/0 (socket says 172.21.15.116:54226) 2024-03-19T02:45:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 -- 172.21.15.116:0/3877956567 learned_addr learned my addr 172.21.15.116:0/3877956567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28affd640 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28bfff640 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2bb57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 -- 172.21.15.116:0/3877956567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 msgr2=0x55c5a2b904f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b904f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 -- 172.21.15.116:0/3877956567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b9d7d0 msgr2=0x55c5a2bb57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2bb57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.257+0000 7ff28b7fe640 1 -- 172.21.15.116:0/3877956567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5a2b64880 con 0x55c5a2b942d0 2024-03-19T02:45:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:08 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3421888644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:09.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.258+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0bd250 tx=0x7ff27c0d11b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7df7d47e3bad57 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.258+0000 7ff2717fa640 1 -- 172.21.15.116:0/3877956567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0051f0 con 0x55c5a2b942d0 2024-03-19T02:45:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.258+0000 7ff2717fa640 1 -- 172.21.15.116:0/3877956567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff27c0dd100 con 0x55c5a2b942d0 2024-03-19T02:45:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.258+0000 7ff2717fa640 1 -- 172.21.15.116:0/3877956567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c005560 con 0x55c5a2b942d0 2024-03-19T02:45:09.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 -- 172.21.15.116:0/3877956567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 msgr2=0x55c5a2b9ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0bd250 tx=0x7ff27c0d11b0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 -- 172.21.15.116:0/3877956567 shutdown_connections 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2bb57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b94bf0 0x55c5a2b904f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 --2- 172.21.15.116:0/3877956567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b942d0 0x55c5a2b9ced0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.260+0000 7ff2949f9740 1 -- 172.21.15.116:0/3877956567 >> 172.21.15.116:0/3877956567 conn(0x55c5a2b8ea00 msgr2=0x55c5a2b8fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.261+0000 7ff2949f9740 1 -- 172.21.15.116:0/3877956567 shutdown_connections 2024-03-19T02:45:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.261+0000 7ff2949f9740 1 -- 172.21.15.116:0/3877956567 wait complete. 2024-03-19T02:45:09.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.261+0000 7ff2949f9740 1 -- start start 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 Processor -- start 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2ba5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 0x55c5a2ba0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2ba14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b2090 con 0x55c5a2b94bf0 2024-03-19T02:45:09.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b19c0 con 0x55c5a2b942d0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1aec20 con 0x55c5a2b9d7d0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff2949f9740 10 client.0 _ll_register_callbacks cb 0x7ff27c0bce00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2ba5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff28b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2ba5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34408/0 (socket says 172.21.15.116:34408) 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff28b7fe640 1 -- 172.21.15.116:0/4245881629 learned_addr learned my addr 172.21.15.116:0/4245881629 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff28affd640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 0x55c5a2ba0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.262+0000 7ff28bfff640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2ba14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 -- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 msgr2=0x55c5a2ba14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2ba14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 -- 172.21.15.116:0/4245881629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 msgr2=0x55c5a2ba5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2ba5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 -- 172.21.15.116:0/4245881629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5a2ba1d20 con 0x55c5a2b94bf0 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff28affd640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 0x55c5a2ba0fb0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff280011eb0 tx=0x7ff280004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61dc87d823474b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.263+0000 7ff26affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.264+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280004d60 con 0x55c5a2b94bf0 2024-03-19T02:45:09.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.264+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7ff28002ccc0 con 0x55c5a2b94bf0 2024-03-19T02:45:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.265+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff27c0dd100 con 0x55c5a2b94bf0 2024-03-19T02:45:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.265+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff28002c330 con 0x55c5a2b94bf0 2024-03-19T02:45:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff2949f9740 10 client.20499 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c5a2c1c660 con 0x55c5a2b94bf0 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff27c0dd100 con 0x55c5a2b94bf0 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7ff280004040 con 0x55c5a2b94bf0 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff2949f9740 10 client.20499 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff2949f9740 10 client.20499 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.266+0000 7ff2949f9740 20 client.20499 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.267+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2c1cbb0 0x55c5a2c1efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.267+0000 7ff2949f9740 4 client.20499 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.267+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5a2b049f0 con 0x55c5a2c1cbb0 2024-03-19T02:45:09.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.267+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2c1cbb0 0x55c5a2c1efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.268+0000 7ff26affd640 10 client.20499 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.268+0000 7ff28b7fe640 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2c1cbb0 0x55c5a2c1efd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d4bf1e6e4bd4c228 server_cookie=ad6284368824da4b in_seq=0 out_seq=0 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.268+0000 7ff26affd640 1 -- 172.21.15.116:0/4245881629 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5a2b049f0 con 0x55c5a2c1cbb0 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.268+0000 7ff26affd640 10 client.20499 handle_command_reply: tid=0 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.268+0000 7ff2949f9740 1 client.20499 shutdown 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 msgr2=0x55c5a2ba0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 0x55c5a2ba0fb0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff280011eb0 tx=0x7ff280004f60 comp rx=0 tx=0).stop 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 shutdown_connections 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c5a2b9d7d0 0x55c5a2ba14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55c5a2c1cbb0 0x55c5a2c1efd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c5a2b94bf0 0x55c5a2ba0fb0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff26affd640 0 client.20499 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 --2- 172.21.15.116:0/4245881629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5a2b942d0 0x55c5a2ba5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 >> 172.21.15.116:0/4245881629 conn(0x55c5a2b8ea00 msgr2=0x55c5a2b8fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 shutdown_connections 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 1 -- 172.21.15.116:0/4245881629 wait complete. 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.269+0000 7ff2949f9740 20 client.20499 trim_cache size 0 max 16384 2024-03-19T02:45:09.271 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 msgr2=0x7ff2740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 0x7ff2740a4e30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff284005c40 tx=0x7ff284009500 comp rx=0 tx=0).stop 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 msgr2=0x7ff28c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff27c000c00 tx=0x7ff27c009610 comp rx=0 tx=0).stop 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 shutdown_connections 2024-03-19T02:45:09.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2740a2970 0x7ff2740a4e30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff28c1066b0 0x7ff28c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff28c101060 0x7ff28c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 --2- 172.21.15.116:0/2051302762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff28c1006a0 0x7ff28c19a1a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 >> 172.21.15.116:0/2051302762 conn(0x7ff28c0778d0 msgr2=0x7ff28c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 shutdown_connections 2024-03-19T02:45:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.270+0000 7ff291d37640 1 -- 172.21.15.116:0/2051302762 wait complete. 2024-03-19T02:45:09.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:45:09.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31132 6.20 MB/sec execute 151 sec latency 172.163 ms 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- 172.21.15.116:0/3560793088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55141006a0 msgr2=0x7f5514100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 --2- 172.21.15.116:0/3560793088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55141006a0 0x7f5514100b20 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5504009930 tx=0x7f550402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- 172.21.15.116:0/3560793088 shutdown_connections 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 --2- 172.21.15.116:0/3560793088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5514101060 0x7f551410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 --2- 172.21.15.116:0/3560793088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55141006a0 0x7f5514100b20 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 --2- 172.21.15.116:0/3560793088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 0x7f5514106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- 172.21.15.116:0/3560793088 >> 172.21.15.116:0/3560793088 conn(0x7f55140778d0 msgr2=0x7f5514077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- 172.21.15.116:0/3560793088 shutdown_connections 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- 172.21.15.116:0/3560793088 wait complete. 2024-03-19T02:45:09.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 Processor -- start 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.412+0000 7f551c675640 1 -- start start 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 0x7f5514199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 0x7f55141a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5514110930 con 0x7f5514101060 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55141107b0 con 0x7f55141006a0 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551c675640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5514110ab0 con 0x7f55141066b0 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34426/0 (socket says 172.21.15.116:34426) 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 -- 172.21.15.116:0/464777223 learned_addr learned my addr 172.21.15.116:0/464777223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551ac12640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 0x7f55141a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f5519c10640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 0x7f5514199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 -- 172.21.15.116:0/464777223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 msgr2=0x7f55141a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 0x7f55141a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 -- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 msgr2=0x7f5514199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 0x7f5514199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 -- 172.21.15.116:0/464777223 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55141a43b0 con 0x7f55141006a0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f5519c10640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 0x7f5514199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.413+0000 7f551a411640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f551000e9f0 tx=0x7f551000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.414+0000 7f550b7fe640 1 -- 172.21.15.116:0/464777223 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551000cd80 con 0x7f55141006a0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.414+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55141a4580 con 0x7f55141006a0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.414+0000 7f550b7fe640 1 -- 172.21.15.116:0/464777223 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55100108e0 con 0x7f55141006a0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.414+0000 7f550b7fe640 1 -- 172.21.15.116:0/464777223 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510018920 con 0x7f55141006a0 2024-03-19T02:45:09.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.414+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55141a4af0 con 0x7f55141006a0 2024-03-19T02:45:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:08 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3421888644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.416+0000 7f550b7fe640 1 -- 172.21.15.116:0/464777223 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5510010a90 con 0x7f55141006a0 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.416+0000 7f551f25c740 1 Processor -- start 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 -- start start 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266519290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 0x56126651bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612664734b0 con 0x5612665197d0 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56126641bd70 con 0x561266510bf0 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551f25c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56126646f550 con 0x5612665102d0 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f550b7fe640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 0x7f55000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551a411640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266519290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f5519c10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 0x56126651bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551ac12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f5519c10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 0x56126651bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54270/0 (socket says 172.21.15.116:54270) 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551ac12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44736/0 (socket says 172.21.15.116:44736) 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551a411640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266519290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34442/0 (socket says 172.21.15.116:34442) 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f5519c10640 1 -- 172.21.15.116:0/1798218431 learned_addr learned my addr 172.21.15.116:0/1798218431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551ac12640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 0x7f55000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f550b7fe640 1 -- 172.21.15.116:0/464777223 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f55100cf4d0 con 0x7f55141006a0 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551ac12640 1 -- 172.21.15.116:0/1798218431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 msgr2=0x561266519290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.417+0000 7f551ac12640 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266519290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f551ac12640 1 -- 172.21.15.116:0/1798218431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 msgr2=0x56126651bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f551ac12640 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 0x56126651bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f551ac12640 1 -- 172.21.15.116:0/1798218431 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56126643e210 con 0x5612665102d0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f551ac12640 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f550c005fd0 tx=0x7f550c010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17982fc3b26f7483 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f551ac12640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 0x7f55000a4e30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f550c010c40 tx=0x7f550c00c000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f5508ff9640 1 -- 172.21.15.116:0/1798218431 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c023070 con 0x5612665102d0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f5508ff9640 1 -- 172.21.15.116:0/1798218431 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f550c012e70 con 0x5612665102d0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.418+0000 7f5508ff9640 1 -- 172.21.15.116:0/1798218431 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c00ce00 con 0x5612665102d0 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- 172.21.15.116:0/1798218431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 msgr2=0x5612665106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f550c005fd0 tx=0x7f550c010740 comp rx=0 tx=0).stop 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- 172.21.15.116:0/1798218431 shutdown_connections 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665197d0 0x56126651bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266519290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 --2- 172.21.15.116:0/1798218431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665102d0 0x5612665106b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- 172.21.15.116:0/1798218431 >> 172.21.15.116:0/1798218431 conn(0x56126650aa00 msgr2=0x56126650ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- 172.21.15.116:0/1798218431 shutdown_connections 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- 172.21.15.116:0/1798218431 wait complete. 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 -- start start 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.419+0000 7f551f25c740 1 Processor -- start 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 0x561266516e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665197d0 0x561266517340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56126651c560 con 0x5612665102d0 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56126642f310 con 0x5612665197d0 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56126651c6e0 con 0x561266510bf0 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 10 client.0 _ll_register_callbacks cb 0x5612664232d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54282/0 (socket says 172.21.15.116:54282) 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 -- 172.21.15.116:0/3049089765 learned_addr learned my addr 172.21.15.116:0/3049089765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551ac12640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665197d0 0x561266517340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f5519c10640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 0x561266516e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 -- 172.21.15.116:0/3049089765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 msgr2=0x561266516e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 0x561266516e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 -- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665197d0 msgr2=0x561266517340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665197d0 0x561266517340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 -- 172.21.15.116:0/3049089765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612661873f0 con 0x5612665102d0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f5519c10640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 0x561266516e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551a411640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f55100981d0 tx=0x7f5510010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696203d11e4ac1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f54fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:09.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510026040 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f55100cf170 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.420+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f550c012e70 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f55100a3ef0 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 10 client.20511 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5612662517d0 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f550c012e70 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f551009ade0 con 0x5612665102d0 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 10 client.20511 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 10 client.20511 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 20 client.20511 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561266529b00 0x56126652bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 4 client.20511 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612664d13e0 con 0x561266529b00 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.421+0000 7f5519c10640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561266529b00 0x56126652bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.422+0000 7f54fdffb640 10 client.20511 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.422+0000 7f5519c10640 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561266529b00 0x56126652bf30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0113962355b84b6 server_cookie=7660d773c6406534 in_seq=0 out_seq=0 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f54fdffb640 1 -- 172.21.15.116:0/3049089765 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612664d13e0 con 0x561266529b00 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f54fdffb640 10 client.20511 handle_command_reply: tid=0 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 client.20511 shutdown 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 msgr2=0x5612665168c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f55100981d0 tx=0x7f5510010de0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 shutdown_connections 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x561266529b00 0x56126652bf30 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665197d0 0x561266517340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561266510bf0 0x561266516e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 --2- 172.21.15.116:0/3049089765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x5612665168c0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 >> 172.21.15.116:0/3049089765 conn(0x56126650aa00 msgr2=0x56126650c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f54fdffb640 0 client.20511 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 shutdown_connections 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 1 -- 172.21.15.116:0/3049089765 wait complete. 2024-03-19T02:45:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.423+0000 7f551f25c740 20 client.20511 trim_cache size 0 max 16384 2024-03-19T02:45:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 Processor -- start 2024-03-19T02:45:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 -- start start 2024-03-19T02:45:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266517160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x5612665317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55141aecd0 con 0x5612665102d0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55141b0810 con 0x561266510bf0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551f25c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55141af3a0 con 0x5612665197d0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551a411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551ac12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x5612665317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.443+0000 7f551a411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54288/0 (socket says 172.21.15.116:54288) 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551a411640 1 -- 172.21.15.116:0/2661936062 learned_addr learned my addr 172.21.15.116:0/2661936062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f5519c10640 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266517160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 -- 172.21.15.116:0/2661936062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 msgr2=0x561266517160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266517160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 -- 172.21.15.116:0/2661936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 msgr2=0x561266525300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 -- 172.21.15.116:0/2661936062 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561266525840 con 0x5612665197d0 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551a411640 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f551ac12640 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x5612665317d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f550c012360 tx=0x7f550c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482627a3e151b203 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f54fffff640 1 -- 172.21.15.116:0/2661936062 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c01b040 con 0x5612665197d0 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f54fffff640 1 -- 172.21.15.116:0/2661936062 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f550c012050 con 0x5612665197d0 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.444+0000 7f54fffff640 1 -- 172.21.15.116:0/2661936062 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c023040 con 0x5612665197d0 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 -- 172.21.15.116:0/2661936062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 msgr2=0x5612665317d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x5612665317d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f550c012360 tx=0x7f550c009800 comp rx=0 tx=0).stop 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 -- 172.21.15.116:0/2661936062 shutdown_connections 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x5612665317d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561266510bf0 0x561266517160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 --2- 172.21.15.116:0/2661936062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612665102d0 0x561266525300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 -- 172.21.15.116:0/2661936062 >> 172.21.15.116:0/2661936062 conn(0x56126650aa00 msgr2=0x56126650c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 -- 172.21.15.116:0/2661936062 shutdown_connections 2024-03-19T02:45:09.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.445+0000 7f551f25c740 1 -- 172.21.15.116:0/2661936062 wait complete. 2024-03-19T02:45:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 -- start start 2024-03-19T02:45:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 Processor -- start 2024-03-19T02:45:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 0x561266513350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x561266513dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55141b0140 con 0x561266510bf0 2024-03-19T02:45:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55141afa70 con 0x5612665102d0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55141accd0 con 0x5612665197d0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551f25c740 10 client.0 _ll_register_callbacks cb 0x7f55100973e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551a411640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 0x561266513350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54290/0 (socket says 172.21.15.116:54290) 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 -- 172.21.15.116:0/1271087911 learned_addr learned my addr 172.21.15.116:0/1271087911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551ac12640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x561266513dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 -- 172.21.15.116:0/1271087911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 msgr2=0x561266513dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x561266513dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 -- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 msgr2=0x561266513350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 0x561266513350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f5519c10640 1 -- 172.21.15.116:0/1271087911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126658ee00 con 0x561266510bf0 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.446+0000 7f551a411640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 0x561266513350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f551ac12640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x561266513dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f5519c10640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f5504000c00 tx=0x7f550404cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56f3e96df34f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f54fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550404c430 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f550404c5d0 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f550c012050 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f550405c050 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f551f25c740 10 client.20517 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561266590020 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.447+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f550c012050 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5504003aa0 con 0x561266510bf0 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 10 client.20517 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 10 client.20517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:45:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 20 client.20517 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5612665907d0 0x561266592bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 4 client.20517 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612664809f0 con 0x5612665907d0 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551a411640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5612665907d0 0x561266592bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f551a411640 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5612665907d0 0x561266592bf0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc8ef97fab036d9b server_cookie=8df30ce8e7df5380 in_seq=0 out_seq=0 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.448+0000 7f54fdffb640 10 client.20517 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f54fdffb640 1 -- 172.21.15.116:0/1271087911 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612664809f0 con 0x5612665907d0 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f54fdffb640 10 client.20517 handle_command_reply: tid=0 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 client.20517 shutdown 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 msgr2=0x561266513890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f5504000c00 tx=0x7f550404cf20 comp rx=0 tx=0).stop 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 shutdown_connections 2024-03-19T02:45:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5612665907d0 0x561266592bf0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612665197d0 0x561266513dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561266510bf0 0x561266513890 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 --2- 172.21.15.116:0/1271087911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612665102d0 0x561266513350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 >> 172.21.15.116:0/1271087911 conn(0x56126650aa00 msgr2=0x56126650c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.449+0000 7f54fdffb640 0 client.20517 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 shutdown_connections 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551f25c740 1 -- 172.21.15.116:0/1271087911 wait complete. 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551f25c740 20 client.20517 trim_cache size 0 max 16384 2024-03-19T02:45:09.456 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 msgr2=0x7f55000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 0x7f55000a4e30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f550c010c40 tx=0x7f550c00c000 comp rx=0 tx=0).stop 2024-03-19T02:45:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 msgr2=0x7f5514198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f551000e9f0 tx=0x7f551000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 shutdown_connections 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f55000a2970 0x7f55000a4e30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55141066b0 0x7f55141a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5514101060 0x7f5514199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 --2- 172.21.15.116:0/464777223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55141006a0 0x7f5514198df0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 >> 172.21.15.116:0/464777223 conn(0x7f55140778d0 msgr2=0x7f5514102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 shutdown_connections 2024-03-19T02:45:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.450+0000 7f551c675640 1 -- 172.21.15.116:0/464777223 wait complete. 2024-03-19T02:45:09.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:45:09.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 -- 172.21.15.116:0/1670565192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d01066b0 msgr2=0x7f85d0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 --2- 172.21.15.116:0/1670565192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d01066b0 0x7f85d0106a90 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d940 tx=0x7f85c002b9a0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 -- 172.21.15.116:0/1670565192 shutdown_connections 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 --2- 172.21.15.116:0/1670565192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d0101060 0x7f85d010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 --2- 172.21.15.116:0/1670565192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01006a0 0x7f85d0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 --2- 172.21.15.116:0/1670565192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d01066b0 0x7f85d0106a90 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 -- 172.21.15.116:0/1670565192 >> 172.21.15.116:0/1670565192 conn(0x7f85d00778d0 msgr2=0x7f85d0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 -- 172.21.15.116:0/1670565192 shutdown_connections 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.629+0000 7f85d644b640 1 -- 172.21.15.116:0/1670565192 wait complete. 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 Processor -- start 2024-03-19T02:45:09.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 -- start start 2024-03-19T02:45:09.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 0x7f85d019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85d010f860 con 0x7f85d0101060 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85d010f6e0 con 0x7f85d01006a0 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d644b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85d010f9e0 con 0x7f85d01066b0 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85cffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54312/0 (socket says 172.21.15.116:54312) 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85cffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34494/0 (socket says 172.21.15.116:34494) 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85cf7fe640 1 -- 172.21.15.116:0/3464428191 learned_addr learned my addr 172.21.15.116:0/3464428191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.630+0000 7f85d49e8640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 0x7f85d019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 -- 172.21.15.116:0/3464428191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 msgr2=0x7f85d019a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 0x7f85d019a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 -- 172.21.15.116:0/3464428191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 msgr2=0x7f85d0195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 -- 172.21.15.116:0/3464428191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d019ad50 con 0x7f85d0101060 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cffff640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85d49e8640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 0x7f85d019a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009be0 tx=0x7f85c40071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cd7fa640 1 -- 172.21.15.116:0/3464428191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4007e00 con 0x7f85d0101060 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d019b040 con 0x7f85d0101060 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cd7fa640 1 -- 172.21.15.116:0/3464428191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c401e070 con 0x7f85d0101060 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85cd7fa640 1 -- 172.21.15.116:0/3464428191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4018710 con 0x7f85d0101060 2024-03-19T02:45:09.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.631+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d01a28d0 con 0x7f85d0101060 2024-03-19T02:45:09.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85cd7fa640 1 -- 172.21.15.116:0/3464428191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f85c4019b30 con 0x7f85d0101060 2024-03-19T02:45:09.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 Processor -- start 2024-03-19T02:45:09.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 -- start start 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85cd7fa640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 0x7f85b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 0x55828063d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85cd7fa640 1 -- 172.21.15.116:0/3464428191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f85c40d2b50 con 0x7f85d0101060 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5582806467d0 0x558280648bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5582805a04b0 con 0x55828063dbf0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d49e8640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 0x7f85b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558280548d70 con 0x5582806467d0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.634+0000 7f85d903c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55828059c550 con 0x55828063d2d0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 0x55828063d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54316/0 (socket says 172.21.15.116:54316) 2024-03-19T02:45:09.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1771872783 learned_addr learned my addr 172.21.15.116:0/1771872783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85d49e8640 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5582806467d0 0x558280648bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1771872783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 msgr2=0x55828063d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 0x55828063d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1771872783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5582806467d0 msgr2=0x558280648bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5582806467d0 0x558280648bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1771872783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55828056b210 con 0x55828063dbf0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cffff640 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 0x55828063d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85d49e8640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 0x7f85b80a4e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f85c80099e0 tx=0x7f85c800fe70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f85c409b7a0 tx=0x7f85c409b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e1859e089235b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85b67fc640 1 -- 172.21.15.116:0/1771872783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c40a7570 con 0x55828063dbf0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85b67fc640 1 -- 172.21.15.116:0/1771872783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c40d0140 con 0x55828063dbf0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.635+0000 7f85b67fc640 1 -- 172.21.15.116:0/1771872783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c409dc90 con 0x55828063dbf0 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 -- 172.21.15.116:0/1771872783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 msgr2=0x558280646290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f85c409b7a0 tx=0x7f85c409b0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 -- 172.21.15.116:0/1771872783 shutdown_connections 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5582806467d0 0x558280648bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280646290 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 --2- 172.21.15.116:0/1771872783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063d2d0 0x55828063d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 -- 172.21.15.116:0/1771872783 >> 172.21.15.116:0/1771872783 conn(0x558280637a00 msgr2=0x558280637de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 -- 172.21.15.116:0/1771872783 shutdown_connections 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.636+0000 7f85d903c740 1 -- 172.21.15.116:0/1771872783 wait complete. 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 -- start start 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 Processor -- start 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806437f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x558280644270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558280649660 con 0x55828063dbf0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55828055c310 con 0x55828063d2d0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5582806497e0 con 0x5582806467d0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d903c740 10 client.0 _ll_register_callbacks cb 0x5582805502d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54324/0 (socket says 172.21.15.116:54324) 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1713880801 learned_addr learned my addr 172.21.15.116:0/1713880801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85d49e8640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x558280644270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.637+0000 7f85cffff640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 msgr2=0x558280644270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x558280644270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1713880801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 msgr2=0x5582806437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 -- 172.21.15.116:0/1713880801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582802b43f0 con 0x55828063dbf0 2024-03-19T02:45:09.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f85c40a7040 tx=0x7f85c409ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c0023fb2a7c291 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c40bb070 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f85c409e430 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c40d0140 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f85c40a7570 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85d903c740 10 client.20529 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55828037e7d0 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.638+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c40d0140 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f85c40a3fd0 con 0x55828063dbf0 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 10 client.20529 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 10 client.20529 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 20 client.20529 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x558280656a60 0x558280658e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 4 client.20529 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:09.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582805fe3e0 con 0x558280656a60 2024-03-19T02:45:09.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85cffff640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x558280656a60 0x558280658e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85af7fe640 10 client.20529 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:09.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.639+0000 7f85cffff640 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x558280656a60 0x558280658e50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78fa85a02acd863c server_cookie=7449afedd2556453 in_seq=0 out_seq=0 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.640+0000 7f85af7fe640 1 -- 172.21.15.116:0/1713880801 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5582805fe3e0 con 0x558280656a60 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.640+0000 7f85af7fe640 10 client.20529 handle_command_reply: tid=0 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.640+0000 7f85d903c740 1 client.20529 shutdown 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 msgr2=0x558280643d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f85c40a7040 tx=0x7f85c409ef50 comp rx=0 tx=0).stop 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 shutdown_connections 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x558280656a60 0x558280658e50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x558280644270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063dbf0 0x558280643d30 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 --2- 172.21.15.116:0/1713880801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 >> 172.21.15.116:0/1713880801 conn(0x558280637a00 msgr2=0x558280639750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85af7fe640 0 client.20529 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 shutdown_connections 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 1 -- 172.21.15.116:0/1713880801 wait complete. 2024-03-19T02:45:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.641+0000 7f85d903c740 20 client.20529 trim_cache size 0 max 16384 2024-03-19T02:45:09.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 Processor -- start 2024-03-19T02:45:09.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 -- start start 2024-03-19T02:45:09.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063dbf0 0x558280636ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x55828065be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ac9f0 con 0x55828063d2d0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ae530 con 0x55828063dbf0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d903c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ad0c0 con 0x5582806467d0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54330/0 (socket says 172.21.15.116:54330) 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85cffff640 1 -- 172.21.15.116:0/2812891534 learned_addr learned my addr 172.21.15.116:0/2812891534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.658+0000 7f85d49e8640 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x55828065be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063dbf0 0x558280636ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 -- 172.21.15.116:0/2812891534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 msgr2=0x55828065be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x55828065be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 -- 172.21.15.116:0/2812891534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063dbf0 msgr2=0x558280636ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063dbf0 0x558280636ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 -- 172.21.15.116:0/2812891534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582806427a0 con 0x55828063d2d0 2024-03-19T02:45:09.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85cffff640 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d910 tx=0x7f85c000d570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d47d14d37060d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85b5ffb640 1 -- 172.21.15.116:0/2812891534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0004220 con 0x55828063d2d0 2024-03-19T02:45:09.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85b5ffb640 1 -- 172.21.15.116:0/2812891534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c00043c0 con 0x55828063d2d0 2024-03-19T02:45:09.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.659+0000 7f85b5ffb640 1 -- 172.21.15.116:0/2812891534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0003db0 con 0x55828063d2d0 2024-03-19T02:45:09.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- 172.21.15.116:0/2812891534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 msgr2=0x558280645f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d910 tx=0x7f85c000d570 comp rx=0 tx=0).stop 2024-03-19T02:45:09.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- 172.21.15.116:0/2812891534 shutdown_connections 2024-03-19T02:45:09.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5582806467d0 0x55828065be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063dbf0 0x558280636ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 --2- 172.21.15.116:0/2812891534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55828063d2d0 0x558280645f20 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- 172.21.15.116:0/2812891534 >> 172.21.15.116:0/2812891534 conn(0x558280637a00 msgr2=0x558280637de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- 172.21.15.116:0/2812891534 shutdown_connections 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- 172.21.15.116:0/2812891534 wait complete. 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.660+0000 7f85d903c740 1 -- start start 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 Processor -- start 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 0x558280659e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 0x55828065a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ade60 con 0x5582806467d0 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ad790 con 0x55828063d2d0 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01aa9f0 con 0x55828063dbf0 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d49e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 0x55828065a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d903c740 10 client.0 _ll_register_callbacks cb 0x7f85c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34534/0 (socket says 172.21.15.116:34534) 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 0x558280659e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cffff640 1 -- 172.21.15.116:0/722715477 learned_addr learned my addr 172.21.15.116:0/722715477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 -- 172.21.15.116:0/722715477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 msgr2=0x5582806598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 -- 172.21.15.116:0/722715477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 msgr2=0x55828065a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 0x55828065a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85cf7fe640 1 -- 172.21.15.116:0/722715477 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55828063da20 con 0x55828063dbf0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.661+0000 7f85d49e8640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 0x55828065a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85cf7fe640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 0x558280659e00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f85c409b750 tx=0x7f85c40077f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d61dbea11d2f114 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c40bb040 con 0x55828063dbf0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f85c40a3e40 con 0x55828063dbf0 2024-03-19T02:45:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c00043c0 con 0x55828063dbf0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f85c40d0140 con 0x55828063dbf0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 10 client.30241 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55828063da20 con 0x55828063dbf0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c00043c0 con 0x55828063dbf0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f85c40bd410 con 0x55828063dbf0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 10 client.30241 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 10 client.30241 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.662+0000 7f85d903c740 20 client.30241 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5582806bcfb0 0x5582806bf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85d903c740 4 client.30241 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582805ad9f0 con 0x5582806bcfb0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85cffff640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5582806bcfb0 0x5582806bf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85cffff640 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5582806bcfb0 0x5582806bf3d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ccd2c9d3125985a server_cookie=61755b3039979816 in_seq=0 out_seq=0 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85af7fe640 10 client.30241 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85af7fe640 1 -- 172.21.15.116:0/722715477 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582805ad9f0 con 0x5582806bcfb0 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85af7fe640 10 client.30241 handle_command_reply: tid=0 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.663+0000 7f85d903c740 1 client.30241 shutdown 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 msgr2=0x558280659e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 0x558280659e00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f85c409b750 tx=0x7f85c40077f0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 shutdown_connections 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5582806bcfb0 0x5582806bf3d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582806467d0 0x55828065a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55828063dbf0 0x558280659e00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85af7fe640 0 client.30241 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 --2- 172.21.15.116:0/722715477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55828063d2d0 0x5582806598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 >> 172.21.15.116:0/722715477 conn(0x558280637a00 msgr2=0x558280644040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 shutdown_connections 2024-03-19T02:45:09.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 1 -- 172.21.15.116:0/722715477 wait complete. 2024-03-19T02:45:09.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d903c740 20 client.30241 trim_cache size 0 max 16384 2024-03-19T02:45:09.671 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 msgr2=0x7f85b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 0x7f85b80a4e30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f85c80099e0 tx=0x7f85c800fe70 comp rx=0 tx=0).stop 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 msgr2=0x7f85d01962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.664+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009be0 tx=0x7f85c40071c0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 shutdown_connections 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85b80a2970 0x7f85b80a4e30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85d01066b0 0x7f85d019a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d0101060 0x7f85d01962a0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 --2- 172.21.15.116:0/3464428191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85d01006a0 0x7f85d0195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 >> 172.21.15.116:0/3464428191 conn(0x7f85d00778d0 msgr2=0x7f85d0103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 shutdown_connections 2024-03-19T02:45:09.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.665+0000 7f85d644b640 1 -- 172.21.15.116:0/3464428191 wait complete. 2024-03-19T02:45:09.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:45:09.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 -- 172.21.15.116:0/3128018065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 msgr2=0x7fc8b810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 --2- 172.21.15.116:0/3128018065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b810d570 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009cd0 tx=0x7fc8b002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 -- 172.21.15.116:0/3128018065 shutdown_connections 2024-03-19T02:45:09.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 --2- 172.21.15.116:0/3128018065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b810d570 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 --2- 172.21.15.116:0/3128018065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81006a0 0x7fc8b8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 --2- 172.21.15.116:0/3128018065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81066b0 0x7fc8b8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 -- 172.21.15.116:0/3128018065 >> 172.21.15.116:0/3128018065 conn(0x7fc8b80778d0 msgr2=0x7fc8b8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 -- 172.21.15.116:0/3128018065 shutdown_connections 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 -- 172.21.15.116:0/3128018065 wait complete. 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.802+0000 7fc8bf012640 1 Processor -- start 2024-03-19T02:45:09.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 -- start start 2024-03-19T02:45:09.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81006a0 0x7fc8b819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 0x7fc8b819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8110970 con 0x7fc8b8101060 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81107f0 con 0x7fc8b81006a0 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bf012640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8110af0 con 0x7fc8b81066b0 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54378/0 (socket says 172.21.15.116:54378) 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 -- 172.21.15.116:0/1302809291 learned_addr learned my addr 172.21.15.116:0/1302809291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 0x7fc8b819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81006a0 0x7fc8b819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 -- 172.21.15.116:0/1302809291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 msgr2=0x7fc8b819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 0x7fc8b819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 -- 172.21.15.116:0/1302809291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81006a0 msgr2=0x7fc8b819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81006a0 0x7fc8b819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8b7fff640 1 -- 172.21.15.116:0/1302809291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b819f160 con 0x7fc8b8101060 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.803+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 0x7fc8b819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9f0 tx=0x7fc8a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8b5ffb640 1 -- 172.21.15.116:0/1302809291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800cd80 con 0x7fc8b8101060 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8b5ffb640 1 -- 172.21.15.116:0/1302809291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a80054d0 con 0x7fc8b8101060 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b819f450 con 0x7fc8b8101060 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8b5ffb640 1 -- 172.21.15.116:0/1302809291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8010490 con 0x7fc8b8101060 2024-03-19T02:45:09.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.804+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b81a6ce0 con 0x7fc8b8101060 2024-03-19T02:45:09.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8b5ffb640 1 -- 172.21.15.116:0/1302809291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc8a8005020 con 0x7fc8b8101060 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 Processor -- start 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 -- start start 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafdd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cacaf404b0 con 0x55cacafe67d0 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cacaee8d70 con 0x55cacafddbf0 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cacaf3c550 con 0x55cacafdd2d0 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8b5ffb640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 0x7fc8a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34560/0 (socket says 172.21.15.116:34560) 2024-03-19T02:45:09.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8bcdae640 1 -- 172.21.15.116:0/1035936511 learned_addr learned my addr 172.21.15.116:0/1035936511 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.807+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafdd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 0x7fc8a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 -- 172.21.15.116:0/1035936511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 msgr2=0x55cacafdd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafdd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 -- 172.21.15.116:0/1035936511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 msgr2=0x55cacafe8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8b5ffb640 1 -- 172.21.15.116:0/1302809291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fc8a80cf4f0 con 0x7fc8b8101060 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 -- 172.21.15.116:0/1035936511 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacaf0b210 con 0x55cacafddbf0 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c70 tx=0x7fc8a0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e2f8f0cd00054a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 0x7fc8a40a4e30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0005cc0 tx=0x7fc8b0005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8aeffd640 1 -- 172.21.15.116:0/1035936511 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0004230 con 0x55cacafddbf0 2024-03-19T02:45:09.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8aeffd640 1 -- 172.21.15.116:0/1035936511 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a00043d0 con 0x55cacafddbf0 2024-03-19T02:45:09.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.808+0000 7fc8aeffd640 1 -- 172.21.15.116:0/1035936511 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0019740 con 0x55cacafddbf0 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 -- 172.21.15.116:0/1035936511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 msgr2=0x55cacafe6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c70 tx=0x7fc8a0011810 comp rx=0 tx=0).stop 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 -- 172.21.15.116:0/1035936511 shutdown_connections 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafe6290 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/1035936511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafdd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 -- 172.21.15.116:0/1035936511 >> 172.21.15.116:0/1035936511 conn(0x55cacafd7a00 msgr2=0x55cacafd7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 -- 172.21.15.116:0/1035936511 shutdown_connections 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.809+0000 7fc8c1c01740 1 -- 172.21.15.116:0/1035936511 wait complete. 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 -- start start 2024-03-19T02:45:09.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 Processor -- start 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 0x55cacafe3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cacafe95c0 con 0x55cacafe67d0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cacaefc310 con 0x55cacafdd2d0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cacafe9740 con 0x55cacafddbf0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8c1c01740 10 client.0 _ll_register_callbacks cb 0x55cacaef02d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.810+0000 7fc8bd5af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bd5af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54392/0 (socket says 172.21.15.116:54392) 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34568/0 (socket says 172.21.15.116:34568) 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 0x55cacafe3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bd5af640 1 -- 172.21.15.116:0/3032565304 learned_addr learned my addr 172.21.15.116:0/3032565304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3032565304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 msgr2=0x55cacafe3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 0x55cacafe3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3032565304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 msgr2=0x55cacafe41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3032565304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacac543f0 con 0x55cacafdd2d0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 0x55cacafe3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c70 tx=0x7fc8a0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3935a80501db82 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc897fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0027070 con 0x55cacafdd2d0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fc8a0003800 con 0x55cacafdd2d0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.811+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8a00043d0 con 0x55cacafdd2d0 2024-03-19T02:45:09.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc8a0033050 con 0x55cacafdd2d0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 10 client.30278 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cacad1e7d0 con 0x55cacafdd2d0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a00043d0 con 0x55cacafdd2d0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fc8a0018070 con 0x55cacafdd2d0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 10 client.30278 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 10 client.30278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 20 client.30278 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacaff8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 4 client.30278 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cacaf9e3e0 con 0x55cacaff69d0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.812+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacaff8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.813+0000 7fc897fff640 10 client.30278 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:09.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.813+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacaff8dc0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6ff1d94d204d45 server_cookie=3c6fd0feb045a801 in_seq=0 out_seq=0 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc897fff640 1 -- 172.21.15.116:0/3032565304 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cacaf9e3e0 con 0x55cacaff69d0 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc897fff640 10 client.30278 handle_command_reply: tid=0 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 client.30278 shutdown 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 msgr2=0x55cacafe3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c70 tx=0x7fc8a0005dc0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 shutdown_connections 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacaff8dc0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacafe41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc897fff640 0 client.30278 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafddbf0 0x55cacafe3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3032565304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe3770 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 >> 172.21.15.116:0/3032565304 conn(0x55cacafd7a00 msgr2=0x55cacafd9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 shutdown_connections 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.814+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3032565304 wait complete. 2024-03-19T02:45:09.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.815+0000 7fc8c1c01740 20 client.30278 trim_cache size 0 max 16384 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 Processor -- start 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 -- start start 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafe67d0 0x55cacafe4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81b0e70 con 0x55cacafddbf0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81b29b0 con 0x55cacafdd2d0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81b1540 con 0x55cacafe67d0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54406/0 (socket says 172.21.15.116:54406) 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8b7fff640 1 -- 172.21.15.116:0/2828583073 learned_addr learned my addr 172.21.15.116:0/2828583073 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.831+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafe67d0 0x55cacafe4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 -- 172.21.15.116:0/2828583073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafe67d0 msgr2=0x55cacafe4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafe67d0 0x55cacafe4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 -- 172.21.15.116:0/2828583073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 msgr2=0x55cacafe5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 -- 172.21.15.116:0/2828583073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacafdda20 con 0x55cacafddbf0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80981e0 tx=0x7fc8a80bb900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cc26da44e66944 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8ae7fc640 1 -- 172.21.15.116:0/2828583073 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a809bf10 con 0x55cacafddbf0 2024-03-19T02:45:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8ae7fc640 1 -- 172.21.15.116:0/2828583073 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a809b810 con 0x55cacafddbf0 2024-03-19T02:45:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.832+0000 7fc8ae7fc640 1 -- 172.21.15.116:0/2828583073 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8097e80 con 0x55cacafddbf0 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- 172.21.15.116:0/2828583073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 msgr2=0x55cacaff2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80981e0 tx=0x7fc8a80bb900 comp rx=0 tx=0).stop 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- 172.21.15.116:0/2828583073 shutdown_connections 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafe67d0 0x55cacafe4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafddbf0 0x55cacaff2390 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/2828583073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafdd2d0 0x55cacafe5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- 172.21.15.116:0/2828583073 >> 172.21.15.116:0/2828583073 conn(0x55cacafd7a00 msgr2=0x55cacafe27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- 172.21.15.116:0/2828583073 shutdown_connections 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- 172.21.15.116:0/2828583073 wait complete. 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 -- start start 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.833+0000 7fc8c1c01740 1 Processor -- start 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacb05ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81b22e0 con 0x55cacafe67d0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81b1c10 con 0x55cacafddbf0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b81aee70 con 0x55cacafdd2d0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8c1c01740 10 client.0 _ll_register_callbacks cb 0x7fc8a809b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bd5af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacb05ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44842/0 (socket says 172.21.15.116:44842) 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3720718517 learned_addr learned my addr 172.21.15.116:0/3720718517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 msgr2=0x55cacafea0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafea0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3720718517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 msgr2=0x55cacb05ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacb05ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 -- 172.21.15.116:0/3720718517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacb05b220 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bd5af640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacb05ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc8bcdae640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0000c00 tx=0x7fc8a0005fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a45d5c254ac533 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.834+0000 7fc897fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0027070 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8a809b810 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fc8a0003d40 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc8a0040050 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 10 client.30259 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cacb05c4c0 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8a809b810 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fc8a0018050 con 0x55cacafdd2d0 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 10 client.30259 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 10 client.30259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 20 client.30259 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:09.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacb05eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 4 client.30259 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cacaf4d9f0 con 0x55cacaff69d0 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.835+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacb05eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.836+0000 7fc8b7fff640 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacb05eb90 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa8a61f43c24414e server_cookie=62027b4399ea99 in_seq=0 out_seq=0 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.836+0000 7fc897fff640 10 client.30259 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc897fff640 1 -- 172.21.15.116:0/3720718517 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cacaf4d9f0 con 0x55cacaff69d0 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc897fff640 10 client.30259 handle_command_reply: tid=0 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 client.30259 shutdown 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 msgr2=0x55cacafe9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0000c00 tx=0x7fc8a0005fc0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 shutdown_connections 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cacaff69d0 0x55cacb05eb90 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cacafe67d0 0x55cacb05ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc897fff640 0 client.30259 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacafddbf0 0x55cacafea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 --2- 172.21.15.116:0/3720718517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cacafdd2d0 0x55cacafe9b70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 >> 172.21.15.116:0/3720718517 conn(0x55cacafd7a00 msgr2=0x55cacafd6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 shutdown_connections 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 1 -- 172.21.15.116:0/3720718517 wait complete. 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.837+0000 7fc8c1c01740 20 client.30259 trim_cache size 0 max 16384 2024-03-19T02:45:09.843 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 msgr2=0x7fc8a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 0x7fc8a40a4e30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0005cc0 tx=0x7fc8b0005c50 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 msgr2=0x7fc8b819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9f0 tx=0x7fc8a800edf0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 shutdown_connections 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc8a40a2970 0x7fc8a40a4e30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b81066b0 0x7fc8b819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8101060 0x7fc8b819a700 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 --2- 172.21.15.116:0/1302809291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b81006a0 0x7fc8b819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 >> 172.21.15.116:0/1302809291 conn(0x7fc8b80778d0 msgr2=0x7fc8b81041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:09.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 shutdown_connections 2024-03-19T02:45:09.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:09.838+0000 7fc8bf012640 1 -- 172.21.15.116:0/1302809291 wait complete. 2024-03-19T02:45:10.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:09 smithi154 ceph-mon[35693]: pgmap v547: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.3 MiB/s wr, 103 op/s 2024-03-19T02:45:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:09 smithi116 ceph-mon[32263]: pgmap v547: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.3 MiB/s wr, 103 op/s 2024-03-19T02:45:10.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31236 6.17 MB/sec execute 152 sec latency 183.792 ms 2024-03-19T02:45:10.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:09 smithi162 ceph-mon[35761]: pgmap v547: 129 pgs: 129 active+clean; 906 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.3 MiB/s wr, 103 op/s 2024-03-19T02:45:10.851 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:45:10.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:45:10.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 -- 172.21.15.116:0/694412541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81077c0 msgr2=0x7f08c8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 --2- 172.21.15.116:0/694412541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81077c0 0x7f08c8107ba0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009930 tx=0x7f08c402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 -- 172.21.15.116:0/694412541 shutdown_connections 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 --2- 172.21.15.116:0/694412541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 --2- 172.21.15.116:0/694412541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81017b0 0x7f08c8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 --2- 172.21.15.116:0/694412541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81077c0 0x7f08c8107ba0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 -- 172.21.15.116:0/694412541 >> 172.21.15.116:0/694412541 conn(0x7f08c8078be0 msgr2=0x7f08c8078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.984+0000 7f08cf80b640 1 -- 172.21.15.116:0/694412541 shutdown_connections 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 -- 172.21.15.116:0/694412541 wait complete. 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 Processor -- start 2024-03-19T02:45:10.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 -- start start 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c819f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81077c0 0x7f08c819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c80746a0 con 0x7f08c81017b0 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8074520 con 0x7f08c8102170 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cf80b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8074820 con 0x7f08c81077c0 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54438/0 (socket says 172.21.15.116:54438) 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cd5a7640 1 -- 172.21.15.116:0/2196814360 learned_addr learned my addr 172.21.15.116:0/2196814360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08ccda6640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c819f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.985+0000 7f08cdda8640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81077c0 0x7f08c819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 -- 172.21.15.116:0/2196814360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81077c0 msgr2=0x7f08c819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:10.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81077c0 0x7f08c819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 -- 172.21.15.116:0/2196814360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 msgr2=0x7f08c819f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c819f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 -- 172.21.15.116:0/2196814360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c819a9b0 con 0x7f08c81017b0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08ccda6640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c819f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009230 tx=0x7f08c402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08be7fc640 1 -- 172.21.15.116:0/2196814360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4043020 con 0x7f08c81017b0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08be7fc640 1 -- 172.21.15.116:0/2196814360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c402ca70 con 0x7f08c81017b0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c819ac40 con 0x7f08c81017b0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08be7fc640 1 -- 172.21.15.116:0/2196814360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4003a50 con 0x7f08c81017b0 2024-03-19T02:45:10.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.986+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c81a6ec0 con 0x7f08c81017b0 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08be7fc640 1 -- 172.21.15.116:0/2196814360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f08c4046020 con 0x7f08c81017b0 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08d23f5740 1 Processor -- start 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08d23f5740 1 -- start start 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.989+0000 7f08be7fc640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 0x7f08b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08d23f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561016903160 con 0x56101699f380 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08d23f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5610168abe10 con 0x5610169a8880 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08d23f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5610168ff200 con 0x56101699fca0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54442/0 (socket says 172.21.15.116:54442) 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 -- 172.21.15.116:0/358976177 learned_addr learned my addr 172.21.15.116:0/358976177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08ccda6640 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 0x7f08b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08be7fc640 1 -- 172.21.15.116:0/2196814360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f08c40f1110 con 0x7f08c81017b0 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 -- 172.21.15.116:0/358976177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 msgr2=0x5610169a8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 -- 172.21.15.116:0/358976177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 msgr2=0x5610169aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 -- 172.21.15.116:0/358976177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610166df7e0 con 0x56101699f380 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f08c0011da0 tx=0x7f08c000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3684ff4fd14e529 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08cdda8640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 0x7f08b40a4e30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f08c000fc90 tx=0x7f08c001e000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08b37fe640 1 -- 172.21.15.116:0/358976177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0022070 con 0x56101699f380 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08b37fe640 1 -- 172.21.15.116:0/358976177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c0009dc0 con 0x56101699f380 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.990+0000 7f08b37fe640 1 -- 172.21.15.116:0/358976177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0004050 con 0x56101699f380 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 -- 172.21.15.116:0/358976177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 msgr2=0x56101699f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f08c0011da0 tx=0x7f08c000f790 comp rx=0 tx=0).stop 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 -- 172.21.15.116:0/358976177 shutdown_connections 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 --2- 172.21.15.116:0/358976177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x56101699f760 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 -- 172.21.15.116:0/358976177 >> 172.21.15.116:0/358976177 conn(0x561016999ab0 msgr2=0x561016999e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 -- 172.21.15.116:0/358976177 shutdown_connections 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.991+0000 7f08d23f5740 1 -- 172.21.15.116:0/358976177 wait complete. 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 -- start start 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 Processor -- start 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169a6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5610169ab610 con 0x56101699f380 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5610168bf1a0 con 0x5610169a8880 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5610169ab790 con 0x56101699fca0 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08d23f5740 10 client.0 _ll_register_callbacks cb 0x561016616c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:10.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08ccda6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54444/0 (socket says 172.21.15.116:54444) 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08ccda6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44880/0 (socket says 172.21.15.116:44880) 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08cd5a7640 1 -- 172.21.15.116:0/2944663153 learned_addr learned my addr 172.21.15.116:0/2944663153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.992+0000 7f08cdda8640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 -- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 msgr2=0x5610169a6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169a6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 -- 172.21.15.116:0/2944663153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 msgr2=0x5610169a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 -- 172.21.15.116:0/2944663153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56101673ff70 con 0x56101699fca0 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08ccda6640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f08c8104000 tx=0x7f08b800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79fc37c3539aef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8012110 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f08b8004050 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c0009dc0 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f08b8025050 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08d23f5740 10 client.30271 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.993+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5610165d7170 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c0009dc0 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f08b8010430 con 0x56101699fca0 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 10 client.30271 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 10 client.30271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 20 client.30271 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5610169b8b40 0x5610169baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 4 client.30271 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610169106a0 con 0x5610169b8b40 2024-03-19T02:45:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5610169b8b40 0x5610169baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08b0ff9640 10 client.30271 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.994+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5610169b8b40 0x5610169baf50 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5171ff08e2366c server_cookie=8351a32bbc1abf51 in_seq=0 out_seq=0 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08b0ff9640 1 -- 172.21.15.116:0/2944663153 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5610169106a0 con 0x5610169b8b40 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08b0ff9640 10 client.30271 handle_command_reply: tid=0 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 client.30271 shutdown 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 msgr2=0x5610169a5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f08c8104000 tx=0x7f08b800f790 comp rx=0 tx=0).stop 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 shutdown_connections 2024-03-19T02:45:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5610169b8b40 0x5610169baf50 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169a6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169a5d40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 --2- 172.21.15.116:0/2944663153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08b0ff9640 0 client.30271 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 >> 172.21.15.116:0/2944663153 conn(0x561016999ab0 msgr2=0x56101699b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 shutdown_connections 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 1 -- 172.21.15.116:0/2944663153 wait complete. 2024-03-19T02:45:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:10.996+0000 7f08d23f5740 20 client.30271 trim_cache size 0 max 16384 2024-03-19T02:45:11.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.022+0000 7f08d23f5740 1 Processor -- start 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.022+0000 7f08d23f5740 1 -- start start 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.022+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.022+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56101699fca0 0x5610169a4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.022+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610169a8880 0x5610169b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08d23f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b0e30 con 0x56101699f380 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08d23f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b2970 con 0x56101699fca0 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08d23f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b1500 con 0x5610169a8880 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54454/0 (socket says 172.21.15.116:54454) 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 -- 172.21.15.116:0/867255092 learned_addr learned my addr 172.21.15.116:0/867255092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cdda8640 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610169a8880 0x5610169b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08ccda6640 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56101699fca0 0x5610169a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 -- 172.21.15.116:0/867255092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610169a8880 msgr2=0x5610169b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610169a8880 0x5610169b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 -- 172.21.15.116:0/867255092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56101699fca0 msgr2=0x5610169a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56101699fca0 0x5610169a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 -- 172.21.15.116:0/867255092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56101699fa90 con 0x56101699f380 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f08c40eea80 tx=0x7f08c40b9020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b87b5fce75c769e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08b2ffd640 1 -- 172.21.15.116:0/867255092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40bdba0 con 0x56101699f380 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08b2ffd640 1 -- 172.21.15.116:0/867255092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c40bd4a0 con 0x56101699f380 2024-03-19T02:45:11.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.023+0000 7f08b2ffd640 1 -- 172.21.15.116:0/867255092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40053d0 con 0x56101699f380 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 -- 172.21.15.116:0/867255092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 msgr2=0x5610169a7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f08c40eea80 tx=0x7f08c40b9020 comp rx=0 tx=0).stop 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 -- 172.21.15.116:0/867255092 shutdown_connections 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5610169a8880 0x5610169b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56101699fca0 0x5610169a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 --2- 172.21.15.116:0/867255092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169a7f30 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 -- 172.21.15.116:0/867255092 >> 172.21.15.116:0/867255092 conn(0x561016999ab0 msgr2=0x56101699b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 -- 172.21.15.116:0/867255092 shutdown_connections 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.024+0000 7f08d23f5740 1 -- 172.21.15.116:0/867255092 wait complete. 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 -- start start 2024-03-19T02:45:11.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 Processor -- start 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b22a0 con 0x56101699f380 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b1bd0 con 0x5610169a8880 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81aee30 con 0x56101699fca0 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08d23f5740 10 client.0 _ll_register_callbacks cb 0x7f08b8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08cd5a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54462/0 (socket says 172.21.15.116:54462) 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08cd5a7640 1 -- 172.21.15.116:0/1893787127 learned_addr learned my addr 172.21.15.116:0/1893787127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08cdda8640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.025+0000 7f08ccda6640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 -- 172.21.15.116:0/1893787127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 msgr2=0x5610169ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 -- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 msgr2=0x5610169adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 -- 172.21.15.116:0/1893787127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610169b10f0 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08cd5a7640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006010 tx=0x7f08c40c15f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05ebd5df64feec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40bdba0 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f08c40b9b10 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c40bd4a0 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f08c40053d0 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08d23f5740 10 client.20571 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561016a1f970 con 0x56101699f380 2024-03-19T02:45:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c40bd4a0 con 0x56101699f380 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.026+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f08c40bd790 con 0x56101699f380 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 10 client.20571 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 10 client.20571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 20 client.20571 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561016a1ff80 0x561016a223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 4 client.20571 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56101690a9f0 con 0x561016a1ff80 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08ccda6640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561016a1ff80 0x561016a223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08ccda6640 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561016a1ff80 0x561016a223a0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3812790aa9485aff server_cookie=13a99145f45b260e in_seq=0 out_seq=0 2024-03-19T02:45:11.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.027+0000 7f08b0ff9640 10 client.20571 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.030+0000 7f08b0ff9640 1 -- 172.21.15.116:0/1893787127 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56101690a9f0 con 0x561016a1ff80 2024-03-19T02:45:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.030+0000 7f08b0ff9640 10 client.20571 handle_command_reply: tid=0 2024-03-19T02:45:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.030+0000 7f08d23f5740 1 client.20571 shutdown 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.030+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 msgr2=0x5610169b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.030+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006010 tx=0x7f08c40c15f0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 shutdown_connections 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561016a1ff80 0x561016a223a0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5610169a8880 0x5610169adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56101699fca0 0x5610169ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08b0ff9640 0 client.20571 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 --2- 172.21.15.116:0/1893787127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56101699f380 0x5610169b09e0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 >> 172.21.15.116:0/1893787127 conn(0x561016999ab0 msgr2=0x5610169a57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 shutdown_connections 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 1 -- 172.21.15.116:0/1893787127 wait complete. 2024-03-19T02:45:11.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08d23f5740 20 client.20571 trim_cache size 0 max 16384 2024-03-19T02:45:11.036 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:11.036 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:45:11.036 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "d9ff1cb7-6061-4142-bb8b-9f184fe4ec3f", 2024-03-19T02:45:11.036 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:45:11.036 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 msgr2=0x7f08b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 0x7f08b40a4e30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f08c000fc90 tx=0x7f08c001e000 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 msgr2=0x7f08c819ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009230 tx=0x7f08c402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 shutdown_connections 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b40a2970 0x7f08b40a4e30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c81077c0 0x7f08c819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102170 0x7f08c819f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.031+0000 7f08cf80b640 1 --2- 172.21.15.116:0/2196814360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c81017b0 0x7f08c819ede0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.032+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 >> 172.21.15.116:0/2196814360 conn(0x7f08c8078be0 msgr2=0x7f08c8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.032+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 shutdown_connections 2024-03-19T02:45:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.032+0000 7f08cf80b640 1 -- 172.21.15.116:0/2196814360 wait complete. 2024-03-19T02:45:11.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:45:11.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:45:11.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/2281642295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 msgr2=0x7fe1f810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/2281642295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f810e680 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009cd0 tx=0x7fe1f002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/2281642295 shutdown_connections 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/2281642295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f810e680 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/2281642295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/2281642295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.166+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/2281642295 >> 172.21.15.116:0/2281642295 conn(0x7fe1f80fd2c0 msgr2=0x7fe1f80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.167+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/2281642295 shutdown_connections 2024-03-19T02:45:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.167+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/2281642295 wait complete. 2024-03-19T02:45:11.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.167+0000 7fe1fcdc6640 1 Processor -- start 2024-03-19T02:45:11.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.167+0000 7fe1fcdc6640 1 -- start start 2024-03-19T02:45:11.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f8111a80 con 0x7fe1f8102170 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f8111900 con 0x7fe1f81077c0 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1fcdc6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f8111c00 con 0x7fe1f81017b0 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44918/0 (socket says 172.21.15.116:44918) 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54476/0 (socket says 172.21.15.116:54476) 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f659c640 1 -- 172.21.15.116:0/4061400168 learned_addr learned my addr 172.21.15.116:0/4061400168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/4061400168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 msgr2=0x7fe1f819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/4061400168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 msgr2=0x7fe1f819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/4061400168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f819f190 con 0x7fe1f8102170 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00e9f0 tx=0x7fe1ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.168+0000 7fe1e77fe640 1 -- 172.21.15.116:0/4061400168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec00ccc0 con 0x7fe1f8102170 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.169+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f819f480 con 0x7fe1f8102170 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.169+0000 7fe1e77fe640 1 -- 172.21.15.116:0/4061400168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1ec005020 con 0x7fe1f8102170 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.169+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f81a6d10 con 0x7fe1f8102170 2024-03-19T02:45:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.169+0000 7fe1e77fe640 1 -- 172.21.15.116:0/4061400168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec004a00 con 0x7fe1f8102170 2024-03-19T02:45:11.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.172+0000 7fe1e77fe640 1 -- 172.21.15.116:0/4061400168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe1ec004ba0 con 0x7fe1f8102170 2024-03-19T02:45:11.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.172+0000 7fe1ff9a8740 1 Processor -- start 2024-03-19T02:45:11.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.172+0000 7fe1ff9a8740 1 -- start start 2024-03-19T02:45:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25dfa30 0x5630e25dfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 0x5630e25eb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630e2543500 con 0x5630e25dfa30 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5630e24ebbe0 con 0x5630e25e8f30 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630e253f5a0 con 0x5630e25e0350 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1e77fe640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 0x7fe1e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44922/0 (socket says 172.21.15.116:44922) 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 0x5630e25eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 -- 172.21.15.116:0/1405432183 learned_addr learned my addr 172.21.15.116:0/1405432183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25dfa30 0x5630e25dfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 0x7fe1e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1e77fe640 1 -- 172.21.15.116:0/4061400168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fe1ec0cf200 con 0x7fe1f8102170 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 -- 172.21.15.116:0/1405432183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 msgr2=0x5630e25eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 0x5630e25eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 -- 172.21.15.116:0/1405432183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25dfa30 msgr2=0x5630e25dfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25dfa30 0x5630e25dfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 -- 172.21.15.116:0/1405432183 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630e250d000 con 0x5630e25e0350 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 0x5630e25eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 0x7fe1e00a4e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1f002c630 tx=0x7fe1f0005d80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.173+0000 7fe1f659c640 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009d40 tx=0x7fe1e8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12f314f7de03259 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.174+0000 7fe1e4ff9640 1 -- 172.21.15.116:0/1405432183 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8004220 con 0x5630e25e0350 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.174+0000 7fe1e4ff9640 1 -- 172.21.15.116:0/1405432183 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1e80043c0 con 0x5630e25e0350 2024-03-19T02:45:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.174+0000 7fe1e4ff9640 1 -- 172.21.15.116:0/1405432183 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8004da0 con 0x5630e25e0350 2024-03-19T02:45:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.174+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/1405432183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 msgr2=0x5630e25e89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.174+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009d40 tx=0x7fe1e8011810 comp rx=0 tx=0).stop 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/1405432183 shutdown_connections 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e8f30 0x5630e25eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e89f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/1405432183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25dfa30 0x5630e25dfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/1405432183 >> 172.21.15.116:0/1405432183 conn(0x5630e25da160 msgr2=0x5630e25da540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/1405432183 shutdown_connections 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/1405432183 wait complete. 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- start start 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 Processor -- start 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25dfa30 0x5630e25e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e0350 0x5630e25e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630e25ebd20 con 0x5630e25e8f30 2024-03-19T02:45:11.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5630e24ff160 con 0x5630e25e0350 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630e25ebea0 con 0x5630e25dfa30 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1ff9a8740 10 client.0 _ll_register_callbacks cb 0x5630e253ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.175+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25dfa30 0x5630e25e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e0350 0x5630e25e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54496/0 (socket says 172.21.15.116:54496) 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/2560161157 learned_addr learned my addr 172.21.15.116:0/2560161157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/2560161157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25dfa30 msgr2=0x5630e25e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25dfa30 0x5630e25e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e0350 msgr2=0x5630e25e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e0350 0x5630e25e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/2560161157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630e23217d0 con 0x5630e25e8f30 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0032ae0 tx=0x7fe1f00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784949a1c5def275 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0054060 con 0x5630e25e8f30 2024-03-19T02:45:11.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fe1f0004620 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1e80043c0 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe1f0039040 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ff9a8740 10 client.20583 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.176+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5630e22573f0 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1e80043c0 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe1f0034440 con 0x5630e25e8f30 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 10 client.20583 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 10 client.20583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 20 client.20583 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e25f91a0 0x5630e25fb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 4 client.20583 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630e2550a40 con 0x5630e25f91a0 2024-03-19T02:45:11.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1f659c640 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e25f91a0 0x5630e25fb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1ddffb640 10 client.20583 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.177+0000 7fe1f659c640 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e25f91a0 0x5630e25fb5b0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ff0985739857de server_cookie=309866b1089af74f in_seq=0 out_seq=0 2024-03-19T02:45:11.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ddffb640 1 -- 172.21.15.116:0/2560161157 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630e2550a40 con 0x5630e25f91a0 2024-03-19T02:45:11.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ddffb640 10 client.20583 handle_command_reply: tid=0 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 client.20583 shutdown 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 msgr2=0x5630e25e68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0032ae0 tx=0x7fe1f00095c0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 shutdown_connections 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e25f91a0 0x5630e25fb5b0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e25e68e0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ddffb640 0 client.20583 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25e0350 0x5630e25e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/2560161157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25dfa30 0x5630e25e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 >> 172.21.15.116:0/2560161157 conn(0x5630e25da160 msgr2=0x5630e25dbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 shutdown_connections 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/2560161157 wait complete. 2024-03-19T02:45:11.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.183+0000 7fe1ff9a8740 20 client.20583 trim_cache size 0 max 16384 2024-03-19T02:45:11.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.202+0000 7fe1ff9a8740 1 Processor -- start 2024-03-19T02:45:11.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.202+0000 7fe1ff9a8740 1 -- start start 2024-03-19T02:45:11.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.202+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81b0ec0 con 0x5630e25e8f30 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81b2a00 con 0x5630e25dfa30 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81b1590 con 0x5630e25e0350 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54510/0 (socket says 172.21.15.116:54510) 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/3376089577 learned_addr learned my addr 172.21.15.116:0/3376089577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f659c640 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/3376089577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 msgr2=0x5630e25e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/3376089577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 msgr2=0x5630e25f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 -- 172.21.15.116:0/3376089577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630e22170f0 con 0x5630e25e8f30 2024-03-19T02:45:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005e30 tx=0x7fe1f0032400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c08b421eeed02a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1dffff640 1 -- 172.21.15.116:0/3376089577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0054040 con 0x5630e25e8f30 2024-03-19T02:45:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1dffff640 1 -- 172.21.15.116:0/3376089577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1f0009b20 con 0x5630e25e8f30 2024-03-19T02:45:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.203+0000 7fe1dffff640 1 -- 172.21.15.116:0/3376089577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f004f470 con 0x5630e25e8f30 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3376089577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 msgr2=0x5630e2600e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005e30 tx=0x7fe1f0032400 comp rx=0 tx=0).stop 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3376089577 shutdown_connections 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e8f30 0x5630e2600e30 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e0350 0x5630e25e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3376089577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3376089577 >> 172.21.15.116:0/3376089577 conn(0x5630e25da160 msgr2=0x5630e25e7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3376089577 shutdown_connections 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3376089577 wait complete. 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 -- start start 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.204+0000 7fe1ff9a8740 1 Processor -- start 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e8f30 0x5630e25e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81b2330 con 0x5630e25e0350 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81b1c60 con 0x5630e25dfa30 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f81aeec0 con 0x5630e25e8f30 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 10 client.0 _ll_register_callbacks cb 0x7fe1ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54526/0 (socket says 172.21.15.116:54526) 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/3266568681 learned_addr learned my addr 172.21.15.116:0/3266568681 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f659c640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f6d9d640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e8f30 0x5630e25e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/3266568681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e8f30 msgr2=0x5630e25e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e8f30 0x5630e25e2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 msgr2=0x5630e25f14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 -- 172.21.15.116:0/3266568681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630e25e3660 con 0x5630e25e0350 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1f5d9b640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec09b180 tx=0x7fe1ec06e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce601291514936bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec01c040 con 0x5630e25e0350 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fe1ec0b41f0 con 0x5630e25e0350 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.205+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1f0009b20 con 0x5630e25e0350 2024-03-19T02:45:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe1ec09afc0 con 0x5630e25e0350 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 10 client.20589 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5630e265f570 con 0x5630e25e0350 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1f0009b20 con 0x5630e25e0350 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe1ec09f200 con 0x5630e25e0350 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 10 client.20589 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 10 client.20589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 20 client.20589 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e265fd40 0x5630e2662160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 4 client.20589 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5630e254ad90 con 0x5630e265fd40 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.206+0000 7fe1f659c640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e265fd40 0x5630e2662160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.207+0000 7fe1f659c640 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e265fd40 0x5630e2662160 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e83bafabac4ebb2 server_cookie=2424ad68b7574214 in_seq=0 out_seq=0 2024-03-19T02:45:11.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.207+0000 7fe1ddffb640 10 client.20589 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.214+0000 7fe1ddffb640 1 -- 172.21.15.116:0/3266568681 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5630e254ad90 con 0x5630e265fd40 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.214+0000 7fe1ddffb640 10 client.20589 handle_command_reply: tid=0 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 client.20589 shutdown 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 msgr2=0x5630e25e2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec09b180 tx=0x7fe1ec06e160 comp rx=0 tx=0).stop 2024-03-19T02:45:11.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 shutdown_connections 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5630e265fd40 0x5630e2662160 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630e25e8f30 0x5630e25e2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630e25e0350 0x5630e25e2850 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ddffb640 0 client.20589 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 --2- 172.21.15.116:0/3266568681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5630e25dfa30 0x5630e25f14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 >> 172.21.15.116:0/3266568681 conn(0x5630e25da160 msgr2=0x5630e25e66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 shutdown_connections 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 1 -- 172.21.15.116:0/3266568681 wait complete. 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1ff9a8740 20 client.20589 trim_cache size 0 max 16384 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (497 inodes in the stack)", 2024-03-19T02:45:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: "d9ff1cb7-6061-4142-bb8b-9f184fe4ec3f": { 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "d9ff1cb7-6061-4142-bb8b-9f184fe4ec3f", 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:45:11.218 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 msgr2=0x7fe1e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 0x7fe1e00a4e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1f002c630 tx=0x7fe1f0005d80 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 msgr2=0x7fe1f819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00e9f0 tx=0x7fe1ec00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 shutdown_connections 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe1e00a2970 0x7fe1e00a4e30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1f81077c0 0x7fe1f819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1f8102170 0x7fe1f819a730 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 --2- 172.21.15.116:0/4061400168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1f81017b0 0x7fe1f819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 >> 172.21.15.116:0/4061400168 conn(0x7fe1f80fd2c0 msgr2=0x7fe1f80fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 shutdown_connections 2024-03-19T02:45:11.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:11.215+0000 7fe1fcdc6640 1 -- 172.21.15.116:0/4061400168 wait complete. 2024-03-19T02:45:11.227 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:11 smithi116 ceph-mon[32263]: pgmap v548: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:45:11.229 INFO:tasks.cephfs.filesystem:scrub status for tag:d9ff1cb7-6061-4142-bb8b-9f184fe4ec3f - {'path': '/', 'tag': 'd9ff1cb7-6061-4142-bb8b-9f184fe4ec3f', 'options': 'recursive,force'} 2024-03-19T02:45:11.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31330 6.16 MB/sec execute 153 sec latency 292.997 ms 2024-03-19T02:45:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:11 smithi162 ceph-mon[35761]: pgmap v548: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:45:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:11 smithi154 ceph-mon[35693]: pgmap v548: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 200 op/s 2024-03-19T02:45:12.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31406 6.13 MB/sec execute 154 sec latency 211.226 ms 2024-03-19T02:45:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:12 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:45:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:12.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:12.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:12 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:45:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:12.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:12 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31515 6.13 MB/sec execute 155 sec latency 191.757 ms 2024-03-19T02:45:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:13 smithi162 ceph-mon[35761]: pgmap v549: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 190 op/s 2024-03-19T02:45:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:13 smithi154 ceph-mon[35693]: pgmap v549: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 190 op/s 2024-03-19T02:45:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:13.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:13 smithi116 ceph-mon[32263]: pgmap v549: 129 pgs: 129 active+clean; 924 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 190 op/s 2024-03-19T02:45:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31597 6.11 MB/sec execute 156 sec latency 208.856 ms 2024-03-19T02:45:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31684 6.09 MB/sec execute 157 sec latency 219.141 ms 2024-03-19T02:45:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:15 smithi162 ceph-mon[35761]: pgmap v550: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 147 op/s 2024-03-19T02:45:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:15 smithi154 ceph-mon[35693]: pgmap v550: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 147 op/s 2024-03-19T02:45:15.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:15 smithi116 ceph-mon[32263]: pgmap v550: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 147 op/s 2024-03-19T02:45:16.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31778 6.07 MB/sec execute 158 sec latency 195.678 ms 2024-03-19T02:45:17.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 31891 6.07 MB/sec execute 159 sec latency 245.897 ms 2024-03-19T02:45:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:17 smithi162 ceph-mon[35761]: pgmap v551: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 146 op/s 2024-03-19T02:45:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:17 smithi154 ceph-mon[35693]: pgmap v551: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 146 op/s 2024-03-19T02:45:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:17 smithi116 ceph-mon[32263]: pgmap v551: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 146 op/s 2024-03-19T02:45:18.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32011 6.06 MB/sec execute 160 sec latency 136.593 ms 2024-03-19T02:45:19.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32138 6.06 MB/sec execute 161 sec latency 157.755 ms 2024-03-19T02:45:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:19 smithi162 ceph-mon[35761]: pgmap v552: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 122 op/s 2024-03-19T02:45:19.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:19 smithi154 ceph-mon[35693]: pgmap v552: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 122 op/s 2024-03-19T02:45:19.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:19 smithi116 ceph-mon[32263]: pgmap v552: 129 pgs: 129 active+clean; 939 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 122 op/s 2024-03-19T02:45:20.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32260 6.08 MB/sec execute 162 sec latency 139.149 ms 2024-03-19T02:45:21.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32366 6.07 MB/sec execute 163 sec latency 150.242 ms 2024-03-19T02:45:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:21 smithi162 ceph-mon[35761]: pgmap v553: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.3 MiB/s wr, 225 op/s 2024-03-19T02:45:21.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:21 smithi154 ceph-mon[35693]: pgmap v553: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.3 MiB/s wr, 225 op/s 2024-03-19T02:45:21.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:21 smithi116 ceph-mon[32263]: pgmap v553: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.3 MiB/s wr, 225 op/s 2024-03-19T02:45:22.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32490 6.08 MB/sec execute 164 sec latency 165.601 ms 2024-03-19T02:45:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:22 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:22 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:22 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:23.321 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:23 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.321 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:23 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:23.321 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:23 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:45:23.321 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:23 smithi162 ceph-mon[35761]: pgmap v554: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 161 op/s 2024-03-19T02:45:23.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32592 6.10 MB/sec execute 165 sec latency 137.641 ms 2024-03-19T02:45:23.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:23 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:23 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:23.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:23 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:45:23.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:23 smithi154 ceph-mon[35693]: pgmap v554: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 161 op/s 2024-03-19T02:45:23.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:23 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:23 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:23.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:23 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:45:23.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:23 smithi116 ceph-mon[32263]: pgmap v554: 129 pgs: 129 active+clean; 947 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 161 op/s 2024-03-19T02:45:24.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32707 6.09 MB/sec execute 166 sec latency 130.745 ms 2024-03-19T02:45:25.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:24 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:24 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:24 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:25.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32807 6.10 MB/sec execute 167 sec latency 147.364 ms 2024-03-19T02:45:26.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:26.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:26.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:26.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:26.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:25 smithi162 ceph-mon[35761]: pgmap v555: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 228 op/s 2024-03-19T02:45:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:25 smithi154 ceph-mon[35693]: pgmap v555: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 228 op/s 2024-03-19T02:45:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:25 smithi116 ceph-mon[32263]: pgmap v555: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 228 op/s 2024-03-19T02:45:26.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 32925 6.11 MB/sec execute 168 sec latency 168.075 ms 2024-03-19T02:45:27.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33014 6.08 MB/sec execute 169 sec latency 201.829 ms 2024-03-19T02:45:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:27 smithi162 ceph-mon[35761]: pgmap v556: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:27 smithi154 ceph-mon[35693]: pgmap v556: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:27 smithi116 ceph-mon[32263]: pgmap v556: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:28.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33137 6.05 MB/sec execute 170 sec latency 119.606 ms 2024-03-19T02:45:29.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33242 6.03 MB/sec execute 171 sec latency 148.712 ms 2024-03-19T02:45:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:29 smithi162 ceph-mon[35761]: pgmap v557: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:29 smithi154 ceph-mon[35693]: pgmap v557: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:29.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:29 smithi116 ceph-mon[32263]: pgmap v557: 129 pgs: 129 active+clean; 955 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-19T02:45:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33382 6.01 MB/sec execute 172 sec latency 139.565 ms 2024-03-19T02:45:31.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33490 6.01 MB/sec execute 173 sec latency 157.353 ms 2024-03-19T02:45:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:31 smithi162 ceph-mon[35761]: pgmap v558: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-19T02:45:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:31 smithi154 ceph-mon[35693]: pgmap v558: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-19T02:45:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:31 smithi116 ceph-mon[32263]: pgmap v558: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-19T02:45:32.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33618 5.99 MB/sec execute 174 sec latency 129.099 ms 2024-03-19T02:45:33.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33718 5.98 MB/sec execute 175 sec latency 185.978 ms 2024-03-19T02:45:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:33 smithi162 ceph-mon[35761]: pgmap v559: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.3 MiB/s wr, 149 op/s 2024-03-19T02:45:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:33 smithi154 ceph-mon[35693]: pgmap v559: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.3 MiB/s wr, 149 op/s 2024-03-19T02:45:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:33 smithi116 ceph-mon[32263]: pgmap v559: 129 pgs: 129 active+clean; 963 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.3 MiB/s wr, 149 op/s 2024-03-19T02:45:34.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33819 5.96 MB/sec execute 176 sec latency 131.868 ms 2024-03-19T02:45:35.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 33949 6.00 MB/sec execute 177 sec latency 198.537 ms 2024-03-19T02:45:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:35 smithi162 ceph-mon[35761]: pgmap v560: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-03-19T02:45:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:35 smithi154 ceph-mon[35693]: pgmap v560: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-03-19T02:45:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:35 smithi116 ceph-mon[32263]: pgmap v560: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 224 op/s 2024-03-19T02:45:36.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34070 6.01 MB/sec execute 178 sec latency 159.225 ms 2024-03-19T02:45:37.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34170 6.02 MB/sec execute 179 sec latency 175.462 ms 2024-03-19T02:45:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:37 smithi162 ceph-mon[35761]: pgmap v561: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:37 smithi154 ceph-mon[35693]: pgmap v561: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:37 smithi116 ceph-mon[32263]: pgmap v561: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:38.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34334 6.08 MB/sec execute 180 sec latency 163.503 ms 2024-03-19T02:45:39.311 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:39 smithi116 ceph-mon[32263]: pgmap v562: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:39.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34471 6.09 MB/sec execute 181 sec latency 164.250 ms 2024-03-19T02:45:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:39 smithi162 ceph-mon[35761]: pgmap v562: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:39 smithi154 ceph-mon[35693]: pgmap v562: 129 pgs: 129 active+clean; 970 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-03-19T02:45:40.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34614 6.11 MB/sec execute 182 sec latency 143.727 ms 2024-03-19T02:45:41.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:45:41.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:45:41.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34756 6.13 MB/sec execute 183 sec latency 153.272 ms 2024-03-19T02:45:41.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.359+0000 7fa37e2ea640 1 -- 172.21.15.116:0/686990112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 msgr2=0x7fa378100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.359+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/686990112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378100b10 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009930 tx=0x7fa36c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:41.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 -- 172.21.15.116:0/686990112 shutdown_connections 2024-03-19T02:45:41.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/686990112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa37810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/686990112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378100b10 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/686990112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa378106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 -- 172.21.15.116:0/686990112 >> 172.21.15.116:0/686990112 conn(0x7fa378077820 msgr2=0x7fa378077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 -- 172.21.15.116:0/686990112 shutdown_connections 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 -- 172.21.15.116:0/686990112 wait complete. 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 Processor -- start 2024-03-19T02:45:41.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.360+0000 7fa37e2ea640 1 -- start start 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa378198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa37819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781109b0 con 0x7fa378100690 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa378110830 con 0x7fa378101050 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37e2ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa378110b30 con 0x7fa3781066a0 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51122/0 (socket says 172.21.15.116:51122) 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 -- 172.21.15.116:0/787071532 learned_addr learned my addr 172.21.15.116:0/787071532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa3777fe640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa378198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37c887640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa37819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 -- 172.21.15.116:0/787071532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 msgr2=0x7fa37819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa37819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 -- 172.21.15.116:0/787071532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 msgr2=0x7fa378198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa378198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 -- 172.21.15.116:0/787071532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37819d030 con 0x7fa378100690 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa37c887640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa37819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa3777fe640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa378198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.361+0000 7fa377fff640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e9f0 tx=0x7fa36800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.362+0000 7fa3757fa640 1 -- 172.21.15.116:0/787071532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36800cd80 con 0x7fa378100690 2024-03-19T02:45:41.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.362+0000 7fa3757fa640 1 -- 172.21.15.116:0/787071532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3680054d0 con 0x7fa378100690 2024-03-19T02:45:41.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.362+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37819d320 con 0x7fa378100690 2024-03-19T02:45:41.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.362+0000 7fa3757fa640 1 -- 172.21.15.116:0/787071532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368010490 con 0x7fa378100690 2024-03-19T02:45:41.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.362+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa378102c50 con 0x7fa378100690 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.364+0000 7fa3757fa640 1 -- 172.21.15.116:0/787071532 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa368005020 con 0x7fa378100690 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 Processor -- start 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 -- start start 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba6a30 0x55a57bba6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbaf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a57bb0a500 con 0x55a57bbaff30 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a57bab2be0 con 0x55a57bba7350 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa380fb1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a57bb065a0 con 0x55a57bba6a30 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51132/0 (socket says 172.21.15.116:51132) 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa3777fe640 1 -- 172.21.15.116:0/2978552875 learned_addr learned my addr 172.21.15.116:0/2978552875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa3757fa640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 0x7fa3600a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa377fff640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbaf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.365+0000 7fa37c887640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba6a30 0x55a57bba6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 -- 172.21.15.116:0/2978552875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba6a30 msgr2=0x55a57bba6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba6a30 0x55a57bba6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 -- 172.21.15.116:0/2978552875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 msgr2=0x55a57bbaf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbaf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa37c887640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 0x7fa3600a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 -- 172.21.15.116:0/2978552875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57bad4000 con 0x55a57bbaff30 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa377fff640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbaf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3757fa640 1 -- 172.21.15.116:0/787071532 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fa3680cf4e0 con 0x7fa378100690 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02c240 tx=0x7fa36c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9af4754de31c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa35e7fc640 1 -- 172.21.15.116:0/2978552875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c045070 con 0x55a57bbaff30 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa35e7fc640 1 -- 172.21.15.116:0/2978552875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa36c02c7e0 con 0x55a57bbaff30 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa35e7fc640 1 -- 172.21.15.116:0/2978552875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c003b90 con 0x55a57bbaff30 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.366+0000 7fa37c887640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 0x7fa3600a4eb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa370009370 tx=0x7fa370004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- 172.21.15.116:0/2978552875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 msgr2=0x55a57bbb2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02c240 tx=0x7fa36c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- 172.21.15.116:0/2978552875 shutdown_connections 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bbaff30 0x55a57bbb2320 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbaf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2978552875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba6a30 0x55a57bba6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- 172.21.15.116:0/2978552875 >> 172.21.15.116:0/2978552875 conn(0x55a57bba1160 msgr2=0x55a57bba1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- 172.21.15.116:0/2978552875 shutdown_connections 2024-03-19T02:45:41.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- 172.21.15.116:0/2978552875 wait complete. 2024-03-19T02:45:41.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.367+0000 7fa380fb1740 1 -- start start 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 Processor -- start 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 0x55a57bbad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 0x55a57bbc4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a57bbb2cc0 con 0x55a57bba6a30 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a57bac6160 con 0x55a57bbadb70 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a57bbb2e40 con 0x55a57bba7350 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa380fb1740 10 client.0 _ll_register_callbacks cb 0x55a57bb02a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51148/0 (socket says 172.21.15.116:51148) 2024-03-19T02:45:41.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa377fff640 1 -- 172.21.15.116:0/2744713641 learned_addr learned my addr 172.21.15.116:0/2744713641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 0x55a57bbad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.368+0000 7fa37c887640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 0x55a57bbc4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 -- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 msgr2=0x55a57bbc4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 0x55a57bbc4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 -- 172.21.15.116:0/2744713641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 msgr2=0x55a57bbad0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 -- 172.21.15.116:0/2744713641 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57b8e87d0 con 0x55a57bba7350 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa377fff640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa37c887640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 0x55a57bbc4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3777fe640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 0x55a57bbad630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x55a57bba5f90 tx=0x7fa36c005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b6a3395251a829e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3577fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c045040 con 0x55a57bba7350 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fa36c003ad0 con 0x55a57bba7350 2024-03-19T02:45:41.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa36c02c7e0 con 0x55a57bba7350 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa36c05b050 con 0x55a57bba7350 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa380fb1740 10 client.30307 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a57b81e3f0 con 0x55a57bba7350 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.369+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa36c004490 con 0x55a57bba7350 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa36c03f440 con 0x55a57bba7350 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 10 client.30307 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 10 client.30307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 20 client.30307 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bbc8f00 0x55a57bbcb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 4 client.30307 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa36c02c7e0 con 0x55a57bbc8f00 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa377fff640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bbc8f00 0x55a57bbcb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa377fff640 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bbc8f00 0x55a57bbcb2f0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d395c86f4ee06841 server_cookie=e953bfb44d524780 in_seq=0 out_seq=0 2024-03-19T02:45:41.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.370+0000 7fa3577fe640 10 client.30307 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.372+0000 7fa3577fe640 1 -- 172.21.15.116:0/2744713641 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa36c02c7e0 con 0x55a57bbc8f00 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.372+0000 7fa3577fe640 10 client.30307 handle_command_reply: tid=0 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 client.30307 shutdown 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 msgr2=0x55a57bbad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 0x55a57bbad630 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x55a57bba5f90 tx=0x7fa36c005e00 comp rx=0 tx=0).stop 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 shutdown_connections 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bbc8f00 0x55a57bbcb2f0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bbadb70 0x55a57bbc4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bba7350 0x55a57bbad630 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 --2- 172.21.15.116:0/2744713641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbad0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 >> 172.21.15.116:0/2744713641 conn(0x55a57bba1160 msgr2=0x55a57bba2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa3577fe640 0 client.30307 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:41.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 shutdown_connections 2024-03-19T02:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 1 -- 172.21.15.116:0/2744713641 wait complete. 2024-03-19T02:45:41.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.373+0000 7fa380fb1740 20 client.30307 trim_cache size 0 max 16384 2024-03-19T02:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 Processor -- start 2024-03-19T02:45:41.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 -- start start 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba6a30 0x55a57bba8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bba2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781ac9f0 con 0x55a57bba7350 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781ae530 con 0x55a57bba6a30 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.393+0000 7fa380fb1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781ad0c0 con 0x55a57bbadb70 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa37c887640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bba2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa377fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba6a30 0x55a57bba8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51162/0 (socket says 172.21.15.116:51162) 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 -- 172.21.15.116:0/3879349701 learned_addr learned my addr 172.21.15.116:0/3879349701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 -- 172.21.15.116:0/3879349701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 msgr2=0x55a57bba2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bba2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 -- 172.21.15.116:0/3879349701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba6a30 msgr2=0x55a57bba8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba6a30 0x55a57bba8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 -- 172.21.15.116:0/3879349701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57bbac0c0 con 0x55a57bba7350 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa3777fe640 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02c170 tx=0x7fa36c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d01deadd77fd4e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa35dffb640 1 -- 172.21.15.116:0/3879349701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c064070 con 0x55a57bba7350 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa35dffb640 1 -- 172.21.15.116:0/3879349701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa36c058030 con 0x55a57bba7350 2024-03-19T02:45:41.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.394+0000 7fa35dffb640 1 -- 172.21.15.116:0/3879349701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c045040 con 0x55a57bba7350 2024-03-19T02:45:41.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- 172.21.15.116:0/3879349701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 msgr2=0x55a57bbc4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02c170 tx=0x7fa36c009780 comp rx=0 tx=0).stop 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- 172.21.15.116:0/3879349701 shutdown_connections 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bba2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba7350 0x55a57bbc4930 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 --2- 172.21.15.116:0/3879349701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba6a30 0x55a57bba8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- 172.21.15.116:0/3879349701 >> 172.21.15.116:0/3879349701 conn(0x55a57bba1160 msgr2=0x55a57bba2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- 172.21.15.116:0/3879349701 shutdown_connections 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- 172.21.15.116:0/3879349701 wait complete. 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.395+0000 7fa380fb1740 1 -- start start 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 Processor -- start 2024-03-19T02:45:41.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbcc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bbcc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781ade60 con 0x55a57bba6a30 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781ad790 con 0x55a57bba7350 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa3781aa9f0 con 0x55a57bbadb70 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa380fb1740 10 client.0 _ll_register_callbacks cb 0x7fa3680972d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51172/0 (socket says 172.21.15.116:51172) 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 -- 172.21.15.116:0/1522328665 learned_addr learned my addr 172.21.15.116:0/1522328665 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa37c887640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bbcc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa3777fe640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbcc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 -- 172.21.15.116:0/1522328665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 msgr2=0x55a57bbcc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bbcc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 -- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 msgr2=0x55a57bbcc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbcc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa377fff640 1 -- 172.21.15.116:0/1522328665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57b7de0f0 con 0x55a57bba6a30 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa37c887640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bbcc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.396+0000 7fa3777fe640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbcc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa377fff640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa3680cce50 tx=0x7fa3680d19f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679be1e1b7a76234 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36809be60 con 0x55a57bba6a30 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7fa368099a60 con 0x55a57bba6a30 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa36c058030 con 0x55a57bba6a30 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa3680a3f00 con 0x55a57bba6a30 2024-03-19T02:45:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 10 client.20607 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a57bc2f290 con 0x55a57bba6a30 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa36c058030 con 0x55a57bba6a30 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa3680cf120 con 0x55a57bba6a30 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 10 client.20607 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 10 client.20607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 20 client.20607 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bc2f960 0x55a57bc31d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 4 client.20607 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a57bb17a40 con 0x55a57bc2f960 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.397+0000 7fa3777fe640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bc2f960 0x55a57bc31d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.398+0000 7fa3777fe640 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bc2f960 0x55a57bc31d50 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5a2f2cf835db9b2 server_cookie=9ab646d57ef22474 in_seq=0 out_seq=0 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.398+0000 7fa3577fe640 10 client.20607 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa3577fe640 1 -- 172.21.15.116:0/1522328665 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a57bb17a40 con 0x55a57bc2f960 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa3577fe640 10 client.20607 handle_command_reply: tid=0 2024-03-19T02:45:41.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 client.20607 shutdown 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 msgr2=0x55a57bbcbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa3680cce50 tx=0x7fa3680d19f0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 shutdown_connections 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a57bc2f960 0x55a57bc31d50 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa3577fe640 0 client.20607 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a57bbadb70 0x55a57bbcc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a57bba7350 0x55a57bbcc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 --2- 172.21.15.116:0/1522328665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a57bba6a30 0x55a57bbcbb10 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 >> 172.21.15.116:0/1522328665 conn(0x55a57bba1160 msgr2=0x55a57bba01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 shutdown_connections 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 1 -- 172.21.15.116:0/1522328665 wait complete. 2024-03-19T02:45:41.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.399+0000 7fa380fb1740 20 client.20607 trim_cache size 0 max 16384 2024-03-19T02:45:41.406 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:41.406 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:45:41.406 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:45:41.406 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 msgr2=0x7fa3600a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 0x7fa3600a4eb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa370009370 tx=0x7fa370004ab0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 msgr2=0x7fa378198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e9f0 tx=0x7fa36800edf0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 shutdown_connections 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa3600a29f0 0x7fa3600a4eb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3781066a0 0x7fa37819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa378101050 0x7fa378198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 --2- 172.21.15.116:0/787071532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378100690 0x7fa378198040 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 >> 172.21.15.116:0/787071532 conn(0x7fa378077820 msgr2=0x7fa378068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 shutdown_connections 2024-03-19T02:45:41.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.400+0000 7fa37e2ea640 1 -- 172.21.15.116:0/787071532 wait complete. 2024-03-19T02:45:41.411 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:41 smithi116 ceph-mon[32263]: pgmap v563: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-03-19T02:45:41.413 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:45:41.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:45:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:41 smithi162 ceph-mon[35761]: pgmap v563: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-03-19T02:45:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:41 smithi154 ceph-mon[35693]: pgmap v563: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-03-19T02:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- 172.21.15.116:0/1977547778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c1010e0 msgr2=0x7f572c10d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 --2- 172.21.15.116:0/1977547778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c1010e0 0x7f572c10d5f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f5724009cd0 tx=0x7f572402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- 172.21.15.116:0/1977547778 shutdown_connections 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 --2- 172.21.15.116:0/1977547778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c1010e0 0x7f572c10d5f0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 --2- 172.21.15.116:0/1977547778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c100720 0x7f572c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 --2- 172.21.15.116:0/1977547778 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c106730 0x7f572c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- 172.21.15.116:0/1977547778 >> 172.21.15.116:0/1977547778 conn(0x7f572c077820 msgr2=0x7f572c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- 172.21.15.116:0/1977547778 shutdown_connections 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- 172.21.15.116:0/1977547778 wait complete. 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 Processor -- start 2024-03-19T02:45:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.548+0000 7f5732a44640 1 -- start start 2024-03-19T02:45:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c1010e0 0x7f572c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 0x7f572c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f572c111af0 con 0x7f572c100720 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f572c111970 con 0x7f572c106730 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5732a44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f572c111c70 con 0x7f572c1010e0 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51194/0 (socket says 172.21.15.116:51194) 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 -- 172.21.15.116:0/1470910508 learned_addr learned my addr 172.21.15.116:0/1470910508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572b7fe640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c1010e0 0x7f572c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5730fe1640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 0x7f572c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 -- 172.21.15.116:0/1470910508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c1010e0 msgr2=0x7f572c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c1010e0 0x7f572c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 -- 172.21.15.116:0/1470910508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 msgr2=0x7f572c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 0x7f572c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f572bfff640 1 -- 172.21.15.116:0/1470910508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c19cfe0 con 0x7f572c100720 2024-03-19T02:45:41.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.549+0000 7f5730fe1640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 0x7f572c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f572bfff640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f571c00e9f0 tx=0x7f571c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c00cd80 con 0x7f572c100720 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c19d2d0 con 0x7f572c100720 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1a4b60 con 0x7f572c100720 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f571c0054d0 con 0x7f572c100720 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.550+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c010490 con 0x7f572c100720 2024-03-19T02:45:41.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.551+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c068610 con 0x7f572c100720 2024-03-19T02:45:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.552+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f571c005020 con 0x7f572c100720 2024-03-19T02:45:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.553+0000 7f57297fa640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 0x7f57140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.553+0000 7f572b7fe640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 0x7f57140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.553+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f571c0cf5c0 con 0x7f572c100720 2024-03-19T02:45:41.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.554+0000 7f572b7fe640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 0x7f57140a4e30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f57200095f0 tx=0x7f5720005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.560+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f571c09bf40 con 0x7f572c100720 2024-03-19T02:45:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.847+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f572c108c90 con 0x7f572c100720 2024-03-19T02:45:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.848+0000 7f57297fa640 1 -- 172.21.15.116:0/1470910508 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f571c09b840 con 0x7f572c100720 2024-03-19T02:45:41.848 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:45:41.848 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:45:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:45:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 msgr2=0x7f57140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 0x7f57140a4e30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f57200095f0 tx=0x7f5720005f90 comp rx=0 tx=0).stop 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 msgr2=0x7f572c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f571c00e9f0 tx=0x7f571c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 shutdown_connections 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57140a2970 0x7f57140a4e30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f572c106730 0x7f572c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f572c1010e0 0x7f572c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 --2- 172.21.15.116:0/1470910508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f572c100720 0x7f572c197ff0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 >> 172.21.15.116:0/1470910508 conn(0x7f572c077820 msgr2=0x7f572c103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 shutdown_connections 2024-03-19T02:45:41.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.850+0000 7f5732a44640 1 -- 172.21.15.116:0/1470910508 wait complete. 2024-03-19T02:45:41.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.981+0000 7f2b94910640 1 -- 172.21.15.116:0/2323141233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b90101060 msgr2=0x7f2b9010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.981+0000 7f2b94910640 1 --2- 172.21.15.116:0/2323141233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b90101060 0x7f2b9010d570 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009cd0 tx=0x7f2b8802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.981+0000 7f2b94910640 1 -- 172.21.15.116:0/2323141233 shutdown_connections 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.981+0000 7f2b94910640 1 --2- 172.21.15.116:0/2323141233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b90101060 0x7f2b9010d570 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.981+0000 7f2b94910640 1 --2- 172.21.15.116:0/2323141233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b901006a0 0x7f2b90100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 --2- 172.21.15.116:0/2323141233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 0x7f2b90106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 -- 172.21.15.116:0/2323141233 >> 172.21.15.116:0/2323141233 conn(0x7f2b900778d0 msgr2=0x7f2b90077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 -- 172.21.15.116:0/2323141233 shutdown_connections 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 -- 172.21.15.116:0/2323141233 wait complete. 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 Processor -- start 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 -- start start 2024-03-19T02:45:41.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.982+0000 7f2b94910640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 0x7f2b900fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 0x7f2b900fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90110970 con 0x7f2b901006a0 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901107f0 con 0x7f2b901066b0 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90110af0 con 0x7f2b90101060 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 0x7f2b900fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51212/0 (socket says 172.21.15.116:51212) 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8e59c640 1 -- 172.21.15.116:0/1237583035 learned_addr learned my addr 172.21.15.116:0/1237583035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 0x7f2b900fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/1237583035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 msgr2=0x7f2b900fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 0x7f2b900fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 msgr2=0x7f2b900fd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/1237583035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b901a64a0 con 0x7f2b90101060 2024-03-19T02:45:41.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 0x7f2b900fdbb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9f0 tx=0x7f2b8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b7f7fe640 1 -- 172.21.15.116:0/1237583035 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8400ccc0 con 0x7f2b90101060 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b7f7fe640 1 -- 172.21.15.116:0/1237583035 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b84005020 con 0x7f2b90101060 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b7f7fe640 1 -- 172.21.15.116:0/1237583035 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b84004a00 con 0x7f2b90101060 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901a6790 con 0x7f2b90101060 2024-03-19T02:45:41.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.983+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b901a6c90 con 0x7f2b90101060 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.986+0000 7f2b7f7fe640 1 -- 172.21.15.116:0/1237583035 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2b84004ba0 con 0x7f2b90101060 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.986+0000 7f2b975d8740 1 Processor -- start 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.986+0000 7f2b975d8740 1 -- start start 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.986+0000 7f2b7f7fe640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 0x7f2b780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 0x7f2b780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b8736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b7f7fe640 1 -- 172.21.15.116:0/1237583035 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f2b840cf1b0 con 0x7f2b90101060 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b32b7d64b0 con 0x55b32b873bf0 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b32b77ed70 con 0x55b32b87c7d0 2024-03-19T02:45:41.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b975d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b32b7d2550 con 0x55b32b8732d0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b8736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51218/0 (socket says 172.21.15.116:51218) 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/3346482739 learned_addr learned my addr 172.21.15.116:0/3346482739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b8736b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51594/0 (socket says 172.21.15.116:51594) 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/3346482739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 msgr2=0x55b32b8736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b8736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/3346482739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 msgr2=0x55b32b87ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8dd9b640 1 -- 172.21.15.116:0/3346482739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b32b7a1210 con 0x55b32b873bf0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.987+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 0x7f2b780a4e60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005cc0 tx=0x7f2b88032e10 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.988+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f2b840a36a0 tx=0x7f2b8409bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b007de04e900acf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.988+0000 7f2b7cff9640 1 -- 172.21.15.116:0/3346482739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8401c040 con 0x55b32b873bf0 2024-03-19T02:45:41.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.988+0000 7f2b7cff9640 1 -- 172.21.15.116:0/3346482739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b840a3bd0 con 0x55b32b873bf0 2024-03-19T02:45:41.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.988+0000 7f2b7cff9640 1 -- 172.21.15.116:0/3346482739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8409f120 con 0x55b32b873bf0 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- 172.21.15.116:0/3346482739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 msgr2=0x55b32b87c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f2b840a36a0 tx=0x7f2b8409bbf0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- 172.21.15.116:0/3346482739 shutdown_connections 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b873bf0 0x55b32b87c290 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3346482739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b8736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- 172.21.15.116:0/3346482739 >> 172.21.15.116:0/3346482739 conn(0x55b32b86da00 msgr2=0x55b32b86dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- 172.21.15.116:0/3346482739 shutdown_connections 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- 172.21.15.116:0/3346482739 wait complete. 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 -- start start 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 Processor -- start 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.989+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b879da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b32b87f560 con 0x55b32b8732d0 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b32b792310 con 0x55b32b87c7d0 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b32b87f6e0 con 0x55b32b873bf0 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51230/0 (socket says 172.21.15.116:51230) 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 -- 172.21.15.116:0/1379546566 learned_addr learned my addr 172.21.15.116:0/1379546566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:41.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b879da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 10 client.0 _ll_register_callbacks cb 0x55b32b7862d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 -- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 msgr2=0x55b32b879da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b879da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 -- 172.21.15.116:0/1379546566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 msgr2=0x55b32b87a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 -- 172.21.15.116:0/1379546566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b32b4ea3f0 con 0x55b32b8732d0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b840a3bd0 con 0x55b32b8732d0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b879da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2b80012d20 tx=0x7f2b8000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684c41be2bb0ee26 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b75ffb640 10 client.20625 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1379546566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80004050 con 0x55b32b8732d0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.990+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1379546566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f2b800041f0 con 0x55b32b8732d0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1379546566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2b80004430 con 0x55b32b8732d0 2024-03-19T02:45:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 10 client.20625 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=38}) -- 0x55b32b5b47d0 con 0x55b32b8732d0 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1379546566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f2b800035f0 con 0x55b32b8732d0 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 10 client.20625 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 10 client.20625 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 20 client.20625 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b88ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 4 client.20625 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b32b8343e0 con 0x55b32b88ca00 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.991+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b88ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b75ffb640 10 client.20625 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b88ee30 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=80794f4636a40b8f server_cookie=5e2d194c4d1641 in_seq=0 out_seq=0 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1379546566 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b32b8343e0 con 0x55b32b88ca00 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b75ffb640 10 client.20625 handle_command_reply: tid=0 2024-03-19T02:45:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 client.20625 shutdown 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 msgr2=0x55b32b879860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2b80012d20 tx=0x7f2b8000f790 comp rx=0 tx=0).stop 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 shutdown_connections 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b88ee30 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b87c7d0 0x55b32b87a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b75ffb640 0 client.20625 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b879da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1379546566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b8732d0 0x55b32b879860 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.992+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 >> 172.21.15.116:0/1379546566 conn(0x55b32b86da00 msgr2=0x55b32b86f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.993+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 shutdown_connections 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.993+0000 7f2b975d8740 1 -- 172.21.15.116:0/1379546566 wait complete. 2024-03-19T02:45:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:41.993+0000 7f2b975d8740 20 client.20625 trim_cache size 0 max 16384 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 Processor -- start 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 -- start start 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b86ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b8946d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901b0dd0 con 0x55b32b87c7d0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901b2910 con 0x55b32b8732d0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b975d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901b14a0 con 0x55b32b873bf0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b8946d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b8dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b86ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b8e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51212/0 (socket says 172.21.15.116:51212) 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.009+0000 7f2b8e59c640 1 -- 172.21.15.116:0/3618582483 learned_addr learned my addr 172.21.15.116:0/3618582483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/3618582483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 msgr2=0x55b32b86ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b86ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/3618582483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 msgr2=0x55b32b888300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/3618582483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b32b872ca0 con 0x55b32b87c7d0 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b8946d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2b88037040 tx=0x7f2b8802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc001c7a669e7df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b77fff640 1 -- 172.21.15.116:0/3618582483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88052070 con 0x55b32b87c7d0 2024-03-19T02:45:42.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b77fff640 1 -- 172.21.15.116:0/3618582483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b88003d70 con 0x55b32b87c7d0 2024-03-19T02:45:42.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.010+0000 7f2b77fff640 1 -- 172.21.15.116:0/3618582483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88039040 con 0x55b32b87c7d0 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- 172.21.15.116:0/3618582483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 msgr2=0x55b32b8946d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b8946d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f2b88037040 tx=0x7f2b8802e050 comp rx=0 tx=0).stop 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- 172.21.15.116:0/3618582483 shutdown_connections 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b8946d0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b873bf0 0x55b32b86ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 --2- 172.21.15.116:0/3618582483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b8732d0 0x55b32b888300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- 172.21.15.116:0/3618582483 >> 172.21.15.116:0/3618582483 conn(0x55b32b86da00 msgr2=0x55b32b86c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- 172.21.15.116:0/3618582483 shutdown_connections 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- 172.21.15.116:0/3618582483 wait complete. 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 -- start start 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.011+0000 7f2b975d8740 1 Processor -- start 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b884960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 0x55b32b881cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901b2240 con 0x55b32b87c7d0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901b1b70 con 0x55b32b873bf0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901aedd0 con 0x55b32b8732d0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 10 client.0 _ll_register_callbacks cb 0x7f2b80009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 0x55b32b881cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51250/0 (socket says 172.21.15.116:51250) 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/1770534194 learned_addr learned my addr 172.21.15.116:0/1770534194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b884960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 msgr2=0x55b32b884960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b884960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/1770534194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 msgr2=0x55b32b881cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 0x55b32b881cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 -- 172.21.15.116:0/1770534194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b32b8827a0 con 0x55b32b87c7d0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8dd9b640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 0x55b32b881cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b8ed9d640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009940 tx=0x7f2b88003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b873d40b16ffb692 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88052030 con 0x55b32b87c7d0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f2b8804ea40 con 0x55b32b87c7d0 2024-03-19T02:45:42.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b88003d70 con 0x55b32b87c7d0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.012+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2b88065050 con 0x55b32b87c7d0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 10 client.20631 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b32b8f2a90 con 0x55b32b87c7d0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b88003d70 con 0x55b32b87c7d0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f2b88057440 con 0x55b32b87c7d0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 10 client.20631 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 10 client.20631 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 20 client.20631 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b8f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 4 client.20631 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b32b7e39f0 con 0x55b32b88ca00 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b8f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b75ffb640 10 client.20631 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:42.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.013+0000 7f2b8e59c640 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b8f51c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e79a704f7a20e52f server_cookie=aa23cdf780c1ebc1 in_seq=0 out_seq=0 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b75ffb640 1 -- 172.21.15.116:0/1770534194 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b32b7e39f0 con 0x55b32b88ca00 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b75ffb640 10 client.20631 handle_command_reply: tid=0 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 client.20631 shutdown 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 msgr2=0x55b32b882230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009940 tx=0x7f2b88003b70 comp rx=0 tx=0).stop 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 shutdown_connections 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55b32b88ca00 0x55b32b8f51c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b32b87c7d0 0x55b32b882230 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b32b873bf0 0x55b32b881cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 --2- 172.21.15.116:0/1770534194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b32b8732d0 0x55b32b884960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 >> 172.21.15.116:0/1770534194 conn(0x55b32b86da00 msgr2=0x55b32b86cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b75ffb640 0 client.20631 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 shutdown_connections 2024-03-19T02:45:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 1 -- 172.21.15.116:0/1770534194 wait complete. 2024-03-19T02:45:42.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.014+0000 7f2b975d8740 20 client.20631 trim_cache size 0 max 16384 2024-03-19T02:45:42.021 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 msgr2=0x7f2b780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 0x7f2b780a4e60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005cc0 tx=0x7f2b88032e10 comp rx=0 tx=0).stop 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 msgr2=0x7f2b900fdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 0x7f2b900fdbb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9f0 tx=0x7f2b8400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 shutdown_connections 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2b780a29a0 0x7f2b780a4e60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b901066b0 0x7f2b900fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b90101060 0x7f2b900fdbb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 --2- 172.21.15.116:0/1237583035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b901006a0 0x7f2b900fd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 >> 172.21.15.116:0/1237583035 conn(0x7f2b900778d0 msgr2=0x7f2b900ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 shutdown_connections 2024-03-19T02:45:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.015+0000 7f2b94910640 1 -- 172.21.15.116:0/1237583035 wait complete. 2024-03-19T02:45:42.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 -- 172.21.15.116:0/2919327610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 msgr2=0x7f3284100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 --2- 172.21.15.116:0/2919327610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284100b20 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f3274009890 tx=0x7f327402bb70 comp rx=0 tx=0).stop 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 -- 172.21.15.116:0/2919327610 shutdown_connections 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 --2- 172.21.15.116:0/2919327610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 0x7f328410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 --2- 172.21.15.116:0/2919327610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284100b20 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 --2- 172.21.15.116:0/2919327610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f3284106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 -- 172.21.15.116:0/2919327610 >> 172.21.15.116:0/2919327610 conn(0x7f32840778d0 msgr2=0x7f3284077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 -- 172.21.15.116:0/2919327610 shutdown_connections 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.151+0000 7f328c332640 1 -- 172.21.15.116:0/2919327610 wait complete. 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 Processor -- start 2024-03-19T02:45:42.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 -- start start 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 0x7f32841992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f32841a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3284110930 con 0x7f32841006a0 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32841107b0 con 0x7f3284101060 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328c332640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3284110ab0 con 0x7f32841066b0 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51260/0 (socket says 172.21.15.116:51260) 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 -- 172.21.15.116:0/552378218 learned_addr learned my addr 172.21.15.116:0/552378218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a8cf640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f32841a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f32898cd640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 0x7f32841992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 -- 172.21.15.116:0/552378218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 msgr2=0x7f32841a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f32841a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 -- 172.21.15.116:0/552378218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 msgr2=0x7f32841992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 0x7f32841992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 -- 172.21.15.116:0/552378218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32841a4350 con 0x7f32841006a0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a8cf640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f32841a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.152+0000 7f328a0ce640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f328000e9c0 tx=0x7f328000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.153+0000 7f327b7fe640 1 -- 172.21.15.116:0/552378218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328000cde0 con 0x7f32841006a0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.153+0000 7f327b7fe640 1 -- 172.21.15.116:0/552378218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32800035f0 con 0x7f32841006a0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.153+0000 7f327b7fe640 1 -- 172.21.15.116:0/552378218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280010690 con 0x7f32841006a0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.153+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32841a4580 con 0x7f32841006a0 2024-03-19T02:45:42.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.153+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32841a4aa0 con 0x7f32841006a0 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.155+0000 7f327b7fe640 1 -- 172.21.15.116:0/552378218 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f32800037a0 con 0x7f32841006a0 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.155+0000 7f328eff3740 1 Processor -- start 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 -- start start 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2348a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 0x557a23495bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557a233ed4b0 con 0x557a234937d0 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a23395d70 con 0x557a2348abf0 2024-03-19T02:45:42.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328eff3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a233e9550 con 0x557a2348a2d0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 0x557a23495bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51244/0 (socket says 172.21.15.116:51244) 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328a0ce640 1 -- 172.21.15.116:0/2630509067 learned_addr learned my addr 172.21.15.116:0/2630509067 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328a8cf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2348a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f327b7fe640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 0x7f32700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.156+0000 7f328a0ce640 1 -- 172.21.15.116:0/2630509067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 msgr2=0x557a2348a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f327b7fe640 1 -- 172.21.15.116:0/552378218 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f32800cf500 con 0x7f32841006a0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a0ce640 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2348a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a0ce640 1 -- 172.21.15.116:0/2630509067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 msgr2=0x557a23495bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a0ce640 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 0x557a23495bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a0ce640 1 -- 172.21.15.116:0/2630509067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a233b8210 con 0x557a2348abf0 2024-03-19T02:45:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a8cf640 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2348a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f32898cd640 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 0x557a23495bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a8cf640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 0x7f32700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f328a0ce640 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f32800cc810 tx=0x7f3280003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24be6c289064a632 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f3278bfe640 1 -- 172.21.15.116:0/2630509067 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328009b2c0 con 0x557a2348abf0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f3278bfe640 1 -- 172.21.15.116:0/2630509067 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3280003b10 con 0x557a2348abf0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.157+0000 7f3278bfe640 1 -- 172.21.15.116:0/2630509067 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328009b0e0 con 0x557a2348abf0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 -- 172.21.15.116:0/2630509067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 msgr2=0x557a23493290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328a8cf640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 0x7f32700a4e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f327c009fd0 tx=0x7f327c004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f32800cc810 tx=0x7f3280003e10 comp rx=0 tx=0).stop 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 -- 172.21.15.116:0/2630509067 shutdown_connections 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a234937d0 0x557a23495bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a2348abf0 0x557a23493290 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 --2- 172.21.15.116:0/2630509067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2348a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 -- 172.21.15.116:0/2630509067 >> 172.21.15.116:0/2630509067 conn(0x557a23484a00 msgr2=0x557a23484de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 -- 172.21.15.116:0/2630509067 shutdown_connections 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.158+0000 7f328eff3740 1 -- 172.21.15.116:0/2630509067 wait complete. 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 -- start start 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 Processor -- start 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a23490d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a23491270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557a23496560 con 0x557a2348abf0 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a233a9310 con 0x557a234937d0 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a234966e0 con 0x557a2348a2d0 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328eff3740 10 client.0 _ll_register_callbacks cb 0x557a2339d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a23490d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51662/0 (socket says 172.21.15.116:51662) 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 -- 172.21.15.116:0/3952820074 learned_addr learned my addr 172.21.15.116:0/3952820074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a8cf640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a23491270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 -- 172.21.15.116:0/3952820074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 msgr2=0x557a23491270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a23491270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 -- 172.21.15.116:0/3952820074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 msgr2=0x557a23490d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a23490d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.159+0000 7f328a0ce640 1 -- 172.21.15.116:0/3952820074 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a231013f0 con 0x557a2348a2d0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f32800cc920 tx=0x7f328009b1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e476cc90d6bfd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328009b780 con 0x557a2348a2d0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557a2344b3e0 con 0x557a2348a2d0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f3280099a80 con 0x557a2348a2d0 2024-03-19T02:45:42.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f328009be80 con 0x557a2348a2d0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 10 client.30349 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557a231cb7d0 con 0x557a2348a2d0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557a2344b3e0 con 0x557a2348a2d0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3280097070 con 0x557a2348a2d0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 10 client.30349 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 10 client.30349 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 20 client.30349 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a234a3960 0x557a234a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 4 client.30349 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.160+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a233fa9f0 con 0x557a234a3960 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.161+0000 7f32898cd640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a234a3960 0x557a234a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.161+0000 7f326dffb640 10 client.30349 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.161+0000 7f32898cd640 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a234a3960 0x557a234a5d50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f27277fea8a4962 server_cookie=622d93bbb3c753a6 in_seq=0 out_seq=0 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f326dffb640 1 -- 172.21.15.116:0/3952820074 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a233fa9f0 con 0x557a234a3960 2024-03-19T02:45:42.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f326dffb640 10 client.30349 handle_command_reply: tid=0 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 client.30349 shutdown 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 msgr2=0x557a234907f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f32800cc920 tx=0x7f328009b1f0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 shutdown_connections 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a23491270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a234a3960 0x557a234a5d50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a23490d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f326dffb640 0 client.30349 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 --2- 172.21.15.116:0/3952820074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a234907f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 >> 172.21.15.116:0/3952820074 conn(0x557a23484a00 msgr2=0x557a23486750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 shutdown_connections 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 1 -- 172.21.15.116:0/3952820074 wait complete. 2024-03-19T02:45:42.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.162+0000 7f328eff3740 20 client.30349 trim_cache size 0 max 16384 2024-03-19T02:45:42.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 Processor -- start 2024-03-19T02:45:42.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 -- start start 2024-03-19T02:45:42.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a23483de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2349f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32841aece0 con 0x557a2348abf0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32841b0820 con 0x557a234937d0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328eff3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32841af3b0 con 0x557a2348a2d0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f328a0ce640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a23483de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51282/0 (socket says 172.21.15.116:51282) 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.177+0000 7f32898cd640 1 -- 172.21.15.116:0/2753302834 learned_addr learned my addr 172.21.15.116:0/2753302834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f328a8cf640 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2349f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 -- 172.21.15.116:0/2753302834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 msgr2=0x557a23483de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a23483de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 -- 172.21.15.116:0/2753302834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 msgr2=0x557a2349f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2349f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 -- 172.21.15.116:0/2753302834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a23495bc0 con 0x557a2348abf0 2024-03-19T02:45:42.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f32898cd640 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f327402f020 tx=0x7f3274005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900cc9bb2eef605b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f326ffff640 1 -- 172.21.15.116:0/2753302834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327404d070 con 0x557a2348abf0 2024-03-19T02:45:42.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f326ffff640 1 -- 172.21.15.116:0/2753302834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3274009450 con 0x557a2348abf0 2024-03-19T02:45:42.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.178+0000 7f326ffff640 1 -- 172.21.15.116:0/2753302834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274039040 con 0x557a2348abf0 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 -- 172.21.15.116:0/2753302834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 msgr2=0x557a234900d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f327402f020 tx=0x7f3274005890 comp rx=0 tx=0).stop 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 -- 172.21.15.116:0/2753302834 shutdown_connections 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2349f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a234900d0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 --2- 172.21.15.116:0/2753302834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a23483de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 -- 172.21.15.116:0/2753302834 >> 172.21.15.116:0/2753302834 conn(0x557a23484a00 msgr2=0x557a234865c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 -- 172.21.15.116:0/2753302834 shutdown_connections 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.179+0000 7f328eff3740 1 -- 172.21.15.116:0/2753302834 wait complete. 2024-03-19T02:45:42.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 -- start start 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 Processor -- start 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2349bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f32898cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51284/0 (socket says 172.21.15.116:51284) 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f32898cd640 1 -- 172.21.15.116:0/3641768979 learned_addr learned my addr 172.21.15.116:0/3641768979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2348d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32841b0150 con 0x557a2348abf0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32841afa80 con 0x557a234937d0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32841acce0 con 0x557a2348a2d0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328eff3740 10 client.0 _ll_register_callbacks cb 0x7f327c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.180+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2349bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 -- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 msgr2=0x557a2349bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2349bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 -- 172.21.15.116:0/3641768979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 msgr2=0x557a2348d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2348d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a2348dcc0 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f32898cd640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f3274005940 tx=0x7f327402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcb58c32b3519d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f326dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274057060 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f3274034c40 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.181+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3274009450 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3274039050 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 10 client.20649 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557a235097f0 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3274009450 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f32740037b0 con 0x557a2348abf0 2024-03-19T02:45:42.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 10 client.20649 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 10 client.20649 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 20 client.20649 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a23509ea0 0x557a2350c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 4 client.20649 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.182+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a233f4d40 con 0x557a23509ea0 2024-03-19T02:45:42.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.183+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a23509ea0 0x557a2350c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328a0ce640 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a23509ea0 0x557a2350c2c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fec37d4fada60096 server_cookie=fee6f4925ded8173 in_seq=0 out_seq=0 2024-03-19T02:45:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f326dffb640 10 client.20649 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f326dffb640 1 -- 172.21.15.116:0/3641768979 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a233f4d40 con 0x557a23509ea0 2024-03-19T02:45:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f326dffb640 10 client.20649 handle_command_reply: tid=0 2024-03-19T02:45:42.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 client.20649 shutdown 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 msgr2=0x557a2348cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f3274005940 tx=0x7f327402bb70 comp rx=0 tx=0).stop 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 shutdown_connections 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a234937d0 0x557a2348d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x557a23509ea0 0x557a2350c2c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a2348abf0 0x557a2348cf80 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 --2- 172.21.15.116:0/3641768979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a2348a2d0 0x557a2349bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 >> 172.21.15.116:0/3641768979 conn(0x557a23484a00 msgr2=0x557a23490790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.184+0000 7f326dffb640 0 client.20649 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.185+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 shutdown_connections 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.185+0000 7f328eff3740 1 -- 172.21.15.116:0/3641768979 wait complete. 2024-03-19T02:45:42.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.185+0000 7f328eff3740 20 client.20649 trim_cache size 0 max 16384 2024-03-19T02:45:42.189 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:42.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 msgr2=0x7f32700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 0x7f32700a4e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f327c009fd0 tx=0x7f327c004a40 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 msgr2=0x7f3284198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f328000e9c0 tx=0x7f328000ee80 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 shutdown_connections 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32700a2970 0x7f32700a4e30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32841066b0 0x7f32841a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3284101060 0x7f32841992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 --2- 172.21.15.116:0/552378218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32841006a0 0x7f3284198d90 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 >> 172.21.15.116:0/552378218 conn(0x7f32840778d0 msgr2=0x7f3284102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 shutdown_connections 2024-03-19T02:45:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.186+0000 7f328c332640 1 -- 172.21.15.116:0/552378218 wait complete. 2024-03-19T02:45:42.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:42 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1470910508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:42.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:45:42.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34854 6.12 MB/sec execute 184 sec latency 130.997 ms 2024-03-19T02:45:42.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.330+0000 7f7f78a7e640 1 -- 172.21.15.116:0/1713392798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f741066b0 msgr2=0x7f7f74106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.330+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/1713392798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f741066b0 0x7f7f74106a90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009930 tx=0x7f7f6402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:42.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 -- 172.21.15.116:0/1713392798 shutdown_connections 2024-03-19T02:45:42.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/1713392798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f74101060 0x7f7f7410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/1713392798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741006a0 0x7f7f74100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/1713392798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f741066b0 0x7f7f74106a90 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 -- 172.21.15.116:0/1713392798 >> 172.21.15.116:0/1713392798 conn(0x7f7f740778d0 msgr2=0x7f7f74077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 -- 172.21.15.116:0/1713392798 shutdown_connections 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 -- 172.21.15.116:0/1713392798 wait complete. 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 Processor -- start 2024-03-19T02:45:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.331+0000 7f7f78a7e640 1 -- start start 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f74101060 0x7f7f74198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741066b0 0x7f7f7419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f74110970 con 0x7f7f74101060 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741107f0 con 0x7f7f741066b0 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f78a7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f74110af0 con 0x7f7f741006a0 2024-03-19T02:45:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51690/0 (socket says 172.21.15.116:51690) 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 -- 172.21.15.116:0/3534344814 learned_addr learned my addr 172.21.15.116:0/3534344814 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f74101060 0x7f7f74198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741066b0 0x7f7f7419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 -- 172.21.15.116:0/3534344814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741066b0 msgr2=0x7f7f7419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741066b0 0x7f7f7419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 -- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f74101060 msgr2=0x7f7f74198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f74101060 0x7f7f74198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.332+0000 7f7f7259c640 1 -- 172.21.15.116:0/3534344814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7419cfb0 con 0x7f7f741006a0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.333+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f64000c00 tx=0x7f7f640045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.334+0000 7f7f6b7fe640 1 -- 172.21.15.116:0/3534344814 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64043020 con 0x7f7f741006a0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.334+0000 7f7f6b7fe640 1 -- 172.21.15.116:0/3534344814 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f6402ca70 con 0x7f7f741006a0 2024-03-19T02:45:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.334+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7419d240 con 0x7f7f741006a0 2024-03-19T02:45:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.334+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f741a4b30 con 0x7f7f741006a0 2024-03-19T02:45:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.334+0000 7f7f6b7fe640 1 -- 172.21.15.116:0/3534344814 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64003ae0 con 0x7f7f741006a0 2024-03-19T02:45:42.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.337+0000 7f7f6b7fe640 1 -- 172.21.15.116:0/3534344814 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7f64046020 con 0x7f7f741006a0 2024-03-19T02:45:42.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.337+0000 7f7f7b665740 1 Processor -- start 2024-03-19T02:45:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 -- start start 2024-03-19T02:45:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6b7fe640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 0x7f7f600a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de9146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de91d7d0 0x5578de91fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578de8774b0 con 0x5578de914bf0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5578de81fd70 con 0x5578de9142d0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7b665740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5578de873550 con 0x5578de91d7d0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 0x7f7f600a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51322/0 (socket says 172.21.15.116:51322) 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 -- 172.21.15.116:0/2868889075 learned_addr learned my addr 172.21.15.116:0/2868889075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7259c640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de9146b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de91d7d0 0x5578de91fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6b7fe640 1 -- 172.21.15.116:0/3534344814 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f7f640f1130 con 0x7f7f741006a0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 -- 172.21.15.116:0/2868889075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de91d7d0 msgr2=0x5578de91fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de91d7d0 0x5578de91fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 -- 172.21.15.116:0/2868889075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 msgr2=0x5578de9146b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de9146b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f6bfff640 1 -- 172.21.15.116:0/2868889075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578de842210 con 0x5578de914bf0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.338+0000 7f7f7259c640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de9146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.339+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 0x7f7f600a4f00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009fd0 tx=0x7f7f6c009de0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.339+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00bc90 tx=0x7f7f5c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0be41f1b2c10aea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.339+0000 7f7f68ff9640 1 -- 172.21.15.116:0/2868889075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c005020 con 0x5578de914bf0 2024-03-19T02:45:42.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.339+0000 7f7f68ff9640 1 -- 172.21.15.116:0/2868889075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c0047a0 con 0x5578de914bf0 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.339+0000 7f7f68ff9640 1 -- 172.21.15.116:0/2868889075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c0116a0 con 0x5578de914bf0 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- 172.21.15.116:0/2868889075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 msgr2=0x5578de91d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00bc90 tx=0x7f7f5c0107a0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- 172.21.15.116:0/2868889075 shutdown_connections 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de91d7d0 0x5578de91fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de91d290 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 --2- 172.21.15.116:0/2868889075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de9146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- 172.21.15.116:0/2868889075 >> 172.21.15.116:0/2868889075 conn(0x5578de90ea00 msgr2=0x5578de90ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- 172.21.15.116:0/2868889075 shutdown_connections 2024-03-19T02:45:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- 172.21.15.116:0/2868889075 wait complete. 2024-03-19T02:45:42.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 -- start start 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.340+0000 7f7f7b665740 1 Processor -- start 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de921e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f72d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de921e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51310/0 (socket says 172.21.15.116:51310) 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f72d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de921e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51334/0 (socket says 172.21.15.116:51334) 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 -- 172.21.15.116:0/3440335765 learned_addr learned my addr 172.21.15.116:0/3440335765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578de9205c0 con 0x5578de91d7d0 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5578de833310 con 0x5578de9142d0 2024-03-19T02:45:42.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5578de920740 con 0x5578de914bf0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7b665740 10 client.0 _ll_register_callbacks cb 0x5578de8272d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 -- 172.21.15.116:0/3440335765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 msgr2=0x5578de927fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 -- 172.21.15.116:0/3440335765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 msgr2=0x5578de921e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de921e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578de58b3f0 con 0x5578de9142d0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f7259c640 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7f640c5730 tx=0x7f7f640bd0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfe43778807b22c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.341+0000 7f7f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64085870 con 0x5578de9142d0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f5c0047a0 con 0x5578de9142d0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f7f6408fa10 con 0x5578de9142d0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7f6408fe10 con 0x5578de9142d0 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 10 client.30386 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5578de6557d0 con 0x5578de9142d0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c0047a0 con 0x5578de9142d0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f7f64090000 con 0x5578de9142d0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 10 client.30386 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 10 client.30386 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 20 client.30386 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de932070 0x5578de9344e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 4 client.30386 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.342+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578de8d53e0 con 0x5578de932070 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.343+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de932070 0x5578de9344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.343+0000 7f7f59ffb640 10 client.30386 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.343+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de932070 0x5578de9344e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=84b3db6e654c60f server_cookie=b8065550ef7fe65e in_seq=0 out_seq=0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f59ffb640 1 -- 172.21.15.116:0/3440335765 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578de8d53e0 con 0x5578de932070 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f59ffb640 10 client.30386 handle_command_reply: tid=0 2024-03-19T02:45:42.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 client.30386 shutdown 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 msgr2=0x5578de927a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7f640c5730 tx=0x7f7f640bd0c0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 shutdown_connections 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de932070 0x5578de9344e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de921e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f59ffb640 0 client.30386 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 --2- 172.21.15.116:0/3440335765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de927a80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 >> 172.21.15.116:0/3440335765 conn(0x5578de90ea00 msgr2=0x5578de910750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 shutdown_connections 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.344+0000 7f7f7b665740 1 -- 172.21.15.116:0/3440335765 wait complete. 2024-03-19T02:45:42.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.345+0000 7f7f7b665740 20 client.30386 trim_cache size 0 max 16384 2024-03-19T02:45:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 Processor -- start 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 -- start start 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de921fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741ac9b0 con 0x5578de91d7d0 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741ae4f0 con 0x5578de9142d0 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7b665740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741ad080 con 0x5578de914bf0 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f72d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de921fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f72d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51348/0 (socket says 172.21.15.116:51348) 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de921fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51326/0 (socket says 172.21.15.116:51326) 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f72d9d640 1 -- 172.21.15.116:0/639570572 learned_addr learned my addr 172.21.15.116:0/639570572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.361+0000 7f7f6bfff640 1 -- 172.21.15.116:0/639570572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 msgr2=0x5578de921fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de921fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f6bfff640 1 -- 172.21.15.116:0/639570572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 msgr2=0x5578de939dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f6bfff640 1 -- 172.21.15.116:0/639570572 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578de9204a0 con 0x5578de914bf0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927ce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c000c00 tx=0x7f7f5c010fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5374618dad90f79a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f5bfff640 1 -- 172.21.15.116:0/639570572 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c004940 con 0x5578de914bf0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f5bfff640 1 -- 172.21.15.116:0/639570572 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c02b680 con 0x5578de914bf0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.362+0000 7f7f5bfff640 1 -- 172.21.15.116:0/639570572 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c005170 con 0x5578de914bf0 2024-03-19T02:45:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- 172.21.15.116:0/639570572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 msgr2=0x5578de927ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927ce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c000c00 tx=0x7f7f5c010fd0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- 172.21.15.116:0/639570572 shutdown_connections 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de91d7d0 0x5578de939dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de914bf0 0x5578de927ce0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- 172.21.15.116:0/639570572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de9142d0 0x5578de921fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- 172.21.15.116:0/639570572 >> 172.21.15.116:0/639570572 conn(0x5578de90ea00 msgr2=0x5578de9275d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- 172.21.15.116:0/639570572 shutdown_connections 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- 172.21.15.116:0/639570572 wait complete. 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- start start 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 Processor -- start 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de91d7d0 0x5578de92c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741ade20 con 0x5578de914bf0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741ad750 con 0x5578de91d7d0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741aa9b0 con 0x5578de9142d0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.363+0000 7f7f7b665740 10 client.0 _ll_register_callbacks cb 0x7f7f640bd290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51354/0 (socket says 172.21.15.116:51354) 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51740/0 (socket says 172.21.15.116:51740) 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f6bfff640 1 -- 172.21.15.116:0/1837614076 learned_addr learned my addr 172.21.15.116:0/1837614076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f72d9d640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de91d7d0 0x5578de92c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 -- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de91d7d0 msgr2=0x5578de92c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de91d7d0 0x5578de92c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 -- 172.21.15.116:0/1837614076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 msgr2=0x5578de92c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 -- 172.21.15.116:0/1837614076 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578de9976f0 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7259c640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f640bd1c0 tx=0x7f7f640bd130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=366160770c83827a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6408fc10 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f5c02b680 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f7f640bd720 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7f6402cc20 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7b665740 10 client.30370 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.364+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5578de9989f0 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c02b680 con 0x5578de9142d0 2024-03-19T02:45:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f7f640d6100 con 0x5578de9142d0 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 10 client.30370 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 10 client.30370 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 20 client.30370 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de998fb0 0x5578de99b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 4 client.30370 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578de8849f0 con 0x5578de998fb0 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de998fb0 0x5578de99b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f59ffb640 10 client.30370 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.365+0000 7f7f6bfff640 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de998fb0 0x5578de99b3d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17ebde279cdcc81b server_cookie=e98342952b073fbb in_seq=0 out_seq=0 2024-03-19T02:45:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f59ffb640 1 -- 172.21.15.116:0/1837614076 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578de8849f0 con 0x5578de998fb0 2024-03-19T02:45:42.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f59ffb640 10 client.30370 handle_command_reply: tid=0 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 client.30370 shutdown 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 msgr2=0x5578de92bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f640bd1c0 tx=0x7f7f640bd130 comp rx=0 tx=0).stop 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 shutdown_connections 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5578de998fb0 0x5578de99b3d0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578de91d7d0 0x5578de92c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f59ffb640 0 client.30370 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578de914bf0 0x5578de92c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 --2- 172.21.15.116:0/1837614076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5578de9142d0 0x5578de92bd20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 >> 172.21.15.116:0/1837614076 conn(0x5578de90ea00 msgr2=0x5578de90d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 shutdown_connections 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.366+0000 7f7f7b665740 1 -- 172.21.15.116:0/1837614076 wait complete. 2024-03-19T02:45:42.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f7b665740 20 client.30370 trim_cache size 0 max 16384 2024-03-19T02:45:42.374 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 msgr2=0x7f7f600a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 0x7f7f600a4f00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009fd0 tx=0x7f7f6c009de0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 msgr2=0x7f7f74197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f64000c00 tx=0x7f7f640045e0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 shutdown_connections 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f600a2a40 0x7f7f600a4f00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f741066b0 0x7f7f7419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f74101060 0x7f7f74198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 --2- 172.21.15.116:0/3534344814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f741006a0 0x7f7f74197fc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 >> 172.21.15.116:0/3534344814 conn(0x7f7f740778d0 msgr2=0x7f7f74068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 shutdown_connections 2024-03-19T02:45:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.367+0000 7f7f78a7e640 1 -- 172.21.15.116:0/3534344814 wait complete. 2024-03-19T02:45:42.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:45:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:42 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1470910508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:42 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1470910508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 -- 172.21.15.116:0/2236310202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201006a0 msgr2=0x7f8320100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 --2- 172.21.15.116:0/2236310202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201006a0 0x7f8320100b20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8314009930 tx=0x7f831402bc20 comp rx=0 tx=0).stop 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 -- 172.21.15.116:0/2236310202 shutdown_connections 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 --2- 172.21.15.116:0/2236310202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f832010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 --2- 172.21.15.116:0/2236310202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201006a0 0x7f8320100b20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 --2- 172.21.15.116:0/2236310202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201066b0 0x7f8320106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 -- 172.21.15.116:0/2236310202 >> 172.21.15.116:0/2236310202 conn(0x7f83200778d0 msgr2=0x7f8320077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 -- 172.21.15.116:0/2236310202 shutdown_connections 2024-03-19T02:45:42.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.513+0000 7f832624b640 1 -- 172.21.15.116:0/2236310202 wait complete. 2024-03-19T02:45:42.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.514+0000 7f832624b640 1 Processor -- start 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.514+0000 7f832624b640 1 -- start start 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201006a0 0x7f8320195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 0x7f832019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f832010f860 con 0x7f8320101060 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f832010f6e0 con 0x7f83201006a0 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f832624b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f832010f9e0 con 0x7f83201066b0 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51374/0 (socket says 172.21.15.116:51374) 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201006a0 0x7f8320195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 -- 172.21.15.116:0/904117650 learned_addr learned my addr 172.21.15.116:0/904117650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831ffff640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 0x7f832019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 -- 172.21.15.116:0/904117650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 msgr2=0x7f832019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 0x7f832019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 -- 172.21.15.116:0/904117650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201006a0 msgr2=0x7f8320195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201006a0 0x7f8320195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 -- 172.21.15.116:0/904117650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832019ae10 con 0x7f8320101060 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831effd640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8314009230 tx=0x7f831402c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.515+0000 7f831ffff640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 0x7f832019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.516+0000 7f831cff9640 1 -- 172.21.15.116:0/904117650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314043020 con 0x7f8320101060 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.516+0000 7f831cff9640 1 -- 172.21.15.116:0/904117650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831402ca60 con 0x7f8320101060 2024-03-19T02:45:42.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.516+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832019b0a0 con 0x7f8320101060 2024-03-19T02:45:42.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.516+0000 7f831cff9640 1 -- 172.21.15.116:0/904117650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140038e0 con 0x7f8320101060 2024-03-19T02:45:42.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.516+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83201a2990 con 0x7f8320101060 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.518+0000 7f831cff9640 1 -- 172.21.15.116:0/904117650 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8314046020 con 0x7f8320101060 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.518+0000 7f8328e39740 1 Processor -- start 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.518+0000 7f8328e39740 1 -- start start 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c7d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 0x557193c88bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557193be04b0 con 0x557193c867d0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557193b88d70 con 0x557193c7dbf0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f8328e39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557193bdc550 con 0x557193c7d2d0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c7d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 0x557193c88bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c7d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51766/0 (socket says 172.21.15.116:51766) 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51362/0 (socket says 172.21.15.116:51362) 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831ffff640 1 -- 172.21.15.116:0/4074969717 learned_addr learned my addr 172.21.15.116:0/4074969717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831cff9640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 0x7f83080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831ffff640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 0x7f83080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831cff9640 1 -- 172.21.15.116:0/904117650 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f83140f0fc0 con 0x7f8320101060 2024-03-19T02:45:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 -- 172.21.15.116:0/4074969717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 msgr2=0x557193c7d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c7d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 -- 172.21.15.116:0/4074969717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 msgr2=0x557193c88bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 0x557193c88bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831f7fe640 1 -- 172.21.15.116:0/4074969717 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557193bab210 con 0x557193c7dbf0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.519+0000 7f831effd640 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 0x557193c88bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.520+0000 7f831ffff640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 0x7f83080a4e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f831800eee0 tx=0x7f8318004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.520+0000 7f831f7fe640 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f831000bb60 tx=0x7f8310005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e704bd36366b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.520+0000 7f8305ffb640 1 -- 172.21.15.116:0/4074969717 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831001a070 con 0x557193c7dbf0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.520+0000 7f8305ffb640 1 -- 172.21.15.116:0/4074969717 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831000be50 con 0x557193c7dbf0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.520+0000 7f8305ffb640 1 -- 172.21.15.116:0/4074969717 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310004a00 con 0x557193c7dbf0 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- 172.21.15.116:0/4074969717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 msgr2=0x557193c86290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f831000bb60 tx=0x7f8310005f90 comp rx=0 tx=0).stop 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- 172.21.15.116:0/4074969717 shutdown_connections 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c867d0 0x557193c88bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c7dbf0 0x557193c86290 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 --2- 172.21.15.116:0/4074969717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c7d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- 172.21.15.116:0/4074969717 >> 172.21.15.116:0/4074969717 conn(0x557193c77a00 msgr2=0x557193c77de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- 172.21.15.116:0/4074969717 shutdown_connections 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- 172.21.15.116:0/4074969717 wait complete. 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 -- start start 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.521+0000 7f8328e39740 1 Processor -- start 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c83cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c841f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557193c895c0 con 0x557193c7d2d0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557193b9c310 con 0x557193c867d0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557193c89740 con 0x557193c7dbf0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f8328e39740 10 client.0 _ll_register_callbacks cb 0x557193b902d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51406/0 (socket says 172.21.15.116:51406) 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 -- 172.21.15.116:0/2729735560 learned_addr learned my addr 172.21.15.116:0/2729735560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831ffff640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c841f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831effd640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c83cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 -- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 msgr2=0x557193c83cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c83cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 -- 172.21.15.116:0/2729735560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 msgr2=0x557193c841f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c841f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 -- 172.21.15.116:0/2729735560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571938f43f0 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831effd640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c83cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f83100055c0 tx=0x7f8310004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f28968386c2d8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.522+0000 7f82feffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831002c060 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f831000bc50 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f831000be50 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8310033050 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 10 client.20679 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5571939be7d0 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831000be50 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f83100167a0 con 0x557193c7d2d0 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 10 client.20679 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 10 client.20679 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 20 client.20679 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193c96ea0 0x557193c99330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 4 client.20679 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.523+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557193c3e3e0 con 0x557193c96ea0 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.524+0000 7f831effd640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193c96ea0 0x557193c99330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.524+0000 7f82feffd640 10 client.20679 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.524+0000 7f831effd640 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193c96ea0 0x557193c99330 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f29047c872742467 server_cookie=207d999729e74f46 in_seq=0 out_seq=0 2024-03-19T02:45:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f82feffd640 1 -- 172.21.15.116:0/2729735560 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557193c3e3e0 con 0x557193c96ea0 2024-03-19T02:45:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f82feffd640 10 client.20679 handle_command_reply: tid=0 2024-03-19T02:45:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 client.20679 shutdown 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 msgr2=0x557193c83770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f83100055c0 tx=0x7f8310004280 comp rx=0 tx=0).stop 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 shutdown_connections 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193c96ea0 0x557193c99330 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c841f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c83cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 --2- 172.21.15.116:0/2729735560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c83770 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f82feffd640 0 client.20679 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 >> 172.21.15.116:0/2729735560 conn(0x557193c77a00 msgr2=0x557193c79750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 shutdown_connections 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.525+0000 7f8328e39740 1 -- 172.21.15.116:0/2729735560 wait complete. 2024-03-19T02:45:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.526+0000 7f8328e39740 20 client.20679 trim_cache size 0 max 16384 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 Processor -- start 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 -- start start 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c85ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c84010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83201acaa0 con 0x557193c7dbf0 2024-03-19T02:45:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83201ae5e0 con 0x557193c867d0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f8328e39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83201ad170 con 0x557193c7d2d0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f831effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f831effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51416/0 (socket says 172.21.15.116:51416) 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f831effd640 1 -- 172.21.15.116:0/68294715 learned_addr learned my addr 172.21.15.116:0/68294715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f831f7fe640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c85ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.541+0000 7f831ffff640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c84010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 -- 172.21.15.116:0/68294715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 msgr2=0x557193c85ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c85ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 -- 172.21.15.116:0/68294715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 msgr2=0x557193c84010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c84010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 -- 172.21.15.116:0/68294715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557193c77830 con 0x557193c7dbf0 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831f7fe640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c85ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f831effd640 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f83140b9d20 tx=0x7f83140dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce120148c802ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f83057fa640 1 -- 172.21.15.116:0/68294715 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140a29c0 con 0x557193c7dbf0 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f83057fa640 1 -- 172.21.15.116:0/68294715 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83140dbe50 con 0x557193c7dbf0 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.542+0000 7f83057fa640 1 -- 172.21.15.116:0/68294715 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140b99c0 con 0x557193c7dbf0 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- 172.21.15.116:0/68294715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 msgr2=0x557193c92390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f83140b9d20 tx=0x7f83140dd040 comp rx=0 tx=0).stop 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- 172.21.15.116:0/68294715 shutdown_connections 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c84010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7dbf0 0x557193c92390 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 --2- 172.21.15.116:0/68294715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7d2d0 0x557193c85ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- 172.21.15.116:0/68294715 >> 172.21.15.116:0/68294715 conn(0x557193c77a00 msgr2=0x557193c77de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- 172.21.15.116:0/68294715 shutdown_connections 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- 172.21.15.116:0/68294715 wait complete. 2024-03-19T02:45:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 -- start start 2024-03-19T02:45:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.543+0000 7f8328e39740 1 Processor -- start 2024-03-19T02:45:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c89db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c8a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83201adf10 con 0x557193c7d2d0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83201ad840 con 0x557193c867d0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f83201aaaa0 con 0x557193c7dbf0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f8328e39740 10 client.0 _ll_register_callbacks cb 0x7f83140bd120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51424/0 (socket says 172.21.15.116:51424) 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831f7fe640 1 -- 172.21.15.116:0/2819064206 learned_addr learned my addr 172.21.15.116:0/2819064206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c89db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831ffff640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c8a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 -- 172.21.15.116:0/2819064206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 msgr2=0x557193c8a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c8a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 -- 172.21.15.116:0/2819064206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 msgr2=0x557193c8e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831effd640 1 -- 172.21.15.116:0/2819064206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557193c8a830 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.544+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f831effd640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c89db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x557193bdc620 tx=0x7f831402c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc81f27ffcab6e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140c14a0 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83140dbe50 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f83140ee5b0 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f83140c5af0 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 10 client.30388 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557193cfcf80 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83140dbe50 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f83140c5660 con 0x557193c7dbf0 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 10 client.30388 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 10 client.30388 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 20 client.30388 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193cfd700 0x557193cffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 4 client.30388 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.545+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557193bed9f0 con 0x557193cfd700 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193cfd700 0x557193cffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f82feffd640 10 client.30388 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f831f7fe640 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193cfd700 0x557193cffb20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=384c380b46b6d2df server_cookie=5ffaff6f2f454dc8 in_seq=0 out_seq=0 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f82feffd640 1 -- 172.21.15.116:0/2819064206 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557193bed9f0 con 0x557193cfd700 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f82feffd640 10 client.30388 handle_command_reply: tid=0 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 client.30388 shutdown 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 msgr2=0x557193c89db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c89db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x557193bdc620 tx=0x7f831402c440 comp rx=0 tx=0).stop 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 shutdown_connections 2024-03-19T02:45:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557193cfd700 0x557193cffb20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f82feffd640 0 client.30388 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.546+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557193c867d0 0x557193c8a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557193c7dbf0 0x557193c89db0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 1 --2- 172.21.15.116:0/2819064206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557193c7d2d0 0x557193c8e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 >> 172.21.15.116:0/2819064206 conn(0x557193c77a00 msgr2=0x557193c76310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 shutdown_connections 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 1 -- 172.21.15.116:0/2819064206 wait complete. 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f8328e39740 20 client.30388 trim_cache size 0 max 16384 2024-03-19T02:45:42.553 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 msgr2=0x7f83080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 0x7f83080a4e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f831800eee0 tx=0x7f8318004ab0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 msgr2=0x7f8320196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.547+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8314009230 tx=0x7f831402c5c0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 shutdown_connections 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f83080a2970 0x7f83080a4e30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83201066b0 0x7f832019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8320101060 0x7f8320196360 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 --2- 172.21.15.116:0/904117650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83201006a0 0x7f8320195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 >> 172.21.15.116:0/904117650 conn(0x7f83200778d0 msgr2=0x7f8320102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 shutdown_connections 2024-03-19T02:45:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.548+0000 7f832624b640 1 -- 172.21.15.116:0/904117650 wait complete. 2024-03-19T02:45:42.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:45:42.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 -- 172.21.15.116:0/207627884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430100970 msgr2=0x7f5430100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 --2- 172.21.15.116:0/207627884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430100970 0x7f5430100df0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f5424009900 tx=0x7f542402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 -- 172.21.15.116:0/207627884 shutdown_connections 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 --2- 172.21.15.116:0/207627884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430101330 0x7f543010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 --2- 172.21.15.116:0/207627884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430100970 0x7f5430100df0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 --2- 172.21.15.116:0/207627884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430105660 0x7f5430105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 -- 172.21.15.116:0/207627884 >> 172.21.15.116:0/207627884 conn(0x7f54300774c0 msgr2=0x7f54300778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 -- 172.21.15.116:0/207627884 shutdown_connections 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.685+0000 7f5436337640 1 -- 172.21.15.116:0/207627884 wait complete. 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 Processor -- start 2024-03-19T02:45:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 -- start start 2024-03-19T02:45:42.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 0x7f5430195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f543010f830 con 0x7f5430105660 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f543010f6b0 con 0x7f5430100970 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f5436337640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f543010f9b0 con 0x7f5430101330 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f54348d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f54348d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51432/0 (socket says 172.21.15.116:51432) 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51800/0 (socket says 172.21.15.116:51800) 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f542ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 0x7f5430195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.686+0000 7f54348d4640 1 -- 172.21.15.116:0/2976509161 learned_addr learned my addr 172.21.15.116:0/2976509161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 -- 172.21.15.116:0/2976509161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 msgr2=0x7f54301962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 -- 172.21.15.116:0/2976509161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 msgr2=0x7f5430195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 0x7f5430195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 -- 172.21.15.116:0/2976509161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543019ada0 con 0x7f5430105660 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f542ffff640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 0x7f5430195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f54348d4640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5428010f90 tx=0x7f542800e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f542d7fa640 1 -- 172.21.15.116:0/2976509161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542801a030 con 0x7f5430105660 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543019b090 con 0x7f5430105660 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f542d7fa640 1 -- 172.21.15.116:0/2976509161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f542800ed30 con 0x7f5430105660 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f542d7fa640 1 -- 172.21.15.116:0/2976509161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428005980 con 0x7f5430105660 2024-03-19T02:45:42.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.687+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54301a2920 con 0x7f5430105660 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f542d7fa640 1 -- 172.21.15.116:0/2976509161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f542801d070 con 0x7f5430105660 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 Processor -- start 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 -- start start 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d033e70 0x55bb3d04d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb3cf9c4b0 con 0x55bb3d0334b0 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb3cf44d70 con 0x55bb3d045960 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.690+0000 7f5438ffb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb3cf98550 con 0x55bb3d033e70 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51444/0 (socket says 172.21.15.116:51444) 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542ffff640 1 -- 172.21.15.116:0/2829163228 learned_addr learned my addr 172.21.15.116:0/2829163228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51420/0 (socket says 172.21.15.116:51420) 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542d7fa640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 0x7f54180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d033e70 0x55bb3d04d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 -- 172.21.15.116:0/2829163228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d033e70 msgr2=0x55bb3d04d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d033e70 0x55bb3d04d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 -- 172.21.15.116:0/2829163228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 msgr2=0x55bb3d033930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542d7fa640 1 -- 172.21.15.116:0/2976509161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f54280d15e0 con 0x7f5430105660 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f54348d4640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 0x7f54180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 -- 172.21.15.116:0/2829163228 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3cf67210 con 0x55bb3d045960 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542ffff640 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5424000c00 tx=0x7f5424005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a86622c8f15d548 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f54348d4640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 0x7f54180a4e30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f54280ce9b0 tx=0x7f5428005b20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.691+0000 7f54167fc640 1 -- 172.21.15.116:0/2829163228 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542402c7e0 con 0x55bb3d045960 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f54167fc640 1 -- 172.21.15.116:0/2829163228 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5424041470 con 0x55bb3d045960 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f54167fc640 1 -- 172.21.15.116:0/2829163228 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424003850 con 0x55bb3d045960 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 -- 172.21.15.116:0/2829163228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 msgr2=0x55bb3d045d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5424000c00 tx=0x7f5424005b70 comp rx=0 tx=0).stop 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 -- 172.21.15.116:0/2829163228 shutdown_connections 2024-03-19T02:45:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d033e70 0x55bb3d04d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d0334b0 0x55bb3d033930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2829163228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d045d40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.692+0000 7f5438ffb740 1 -- 172.21.15.116:0/2829163228 >> 172.21.15.116:0/2829163228 conn(0x55bb3d035f70 msgr2=0x55bb3d036350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.693+0000 7f5438ffb740 1 -- 172.21.15.116:0/2829163228 shutdown_connections 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.693+0000 7f5438ffb740 1 -- 172.21.15.116:0/2829163228 wait complete. 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.693+0000 7f5438ffb740 1 -- start start 2024-03-19T02:45:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.693+0000 7f5438ffb740 1 Processor -- start 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d04b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d04b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb3d04de30 con 0x55bb3d033e70 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb3cf58310 con 0x55bb3d045960 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb3d04dfb0 con 0x55bb3d0334b0 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f5438ffb740 10 client.0 _ll_register_callbacks cb 0x55bb3cf4c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51816/0 (socket says 172.21.15.116:51816) 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 -- 172.21.15.116:0/2412959254 learned_addr learned my addr 172.21.15.116:0/2412959254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d04b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542ffff640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d04b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 -- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 msgr2=0x55bb3d04b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d04b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 -- 172.21.15.116:0/2412959254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 msgr2=0x55bb3d04b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d04b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.694+0000 7f542f7fe640 1 -- 172.21.15.116:0/2412959254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3ccb03f0 con 0x55bb3d0334b0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5424036040 tx=0x7f542402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a445a12c1d9cd069 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f540f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f542ffff640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d04b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424041820 con 0x55bb3d0334b0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5424041470 con 0x55bb3d0334b0 2024-03-19T02:45:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f54240419c0 con 0x55bb3d0334b0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5424041bc0 con 0x55bb3d0334b0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f5438ffb740 10 client.30400 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.695+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55bb3cd7a7d0 con 0x55bb3d0334b0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5424041470 con 0x55bb3d0334b0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5424034600 con 0x55bb3d0334b0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 10 client.30400 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 10 client.30400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 20 client.30400 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d05b290 0x55bb3d05d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 4 client.30400 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb3cffa3e0 con 0x55bb3d05b290 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.696+0000 7f542ffff640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d05b290 0x55bb3d05d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.697+0000 7f540f7fe640 10 client.30400 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.697+0000 7f542ffff640 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d05b290 0x55bb3d05d680 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928b438a9fd3c765 server_cookie=5621909692b22df3 in_seq=0 out_seq=0 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f540f7fe640 1 -- 172.21.15.116:0/2412959254 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb3cffa3e0 con 0x55bb3d05b290 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f540f7fe640 10 client.30400 handle_command_reply: tid=0 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f5438ffb740 1 client.30400 shutdown 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 msgr2=0x55bb3d04ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5424036040 tx=0x7f542402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 shutdown_connections 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.700+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d05b290 0x55bb3d05d680 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d04b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d04b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2412959254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04ada0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 >> 172.21.15.116:0/2412959254 conn(0x55bb3d035f70 msgr2=0x55bb3d032ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f540f7fe640 0 client.30400 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 shutdown_connections 2024-03-19T02:45:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 1 -- 172.21.15.116:0/2412959254 wait complete. 2024-03-19T02:45:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.701+0000 7f5438ffb740 20 client.30400 trim_cache size 0 max 16384 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 Processor -- start 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 -- start start 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d043440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54301acb40 con 0x55bb3d033e70 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ae680 con 0x55bb3d045960 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f5438ffb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ad210 con 0x55bb3d0334b0 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51454/0 (socket says 172.21.15.116:51454) 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542ffff640 1 -- 172.21.15.116:0/2755151537 learned_addr learned my addr 172.21.15.116:0/2755151537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d043440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542ffff640 1 -- 172.21.15.116:0/2755151537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 msgr2=0x55bb3d04d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.718+0000 7f542ffff640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f542ffff640 1 -- 172.21.15.116:0/2755151537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 msgr2=0x55bb3d043440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f542ffff640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d043440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f542ffff640 1 -- 172.21.15.116:0/2755151537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb3d049d50 con 0x55bb3d033e70 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d043440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f542ffff640 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5420011a40 tx=0x7f5420005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1be02b1dc514516 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f5415ffb640 1 -- 172.21.15.116:0/2755151537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420026070 con 0x55bb3d033e70 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f5415ffb640 1 -- 172.21.15.116:0/2755151537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f542000fdd0 con 0x55bb3d033e70 2024-03-19T02:45:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.719+0000 7f5415ffb640 1 -- 172.21.15.116:0/2755151537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200206d0 con 0x55bb3d033e70 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- 172.21.15.116:0/2755151537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 msgr2=0x55bb3d049670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5420011a40 tx=0x7f5420005f90 comp rx=0 tx=0).stop 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- 172.21.15.116:0/2755151537 shutdown_connections 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d045960 0x55bb3d043440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d033e70 0x55bb3d049670 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2755151537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d04d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- 172.21.15.116:0/2755151537 >> 172.21.15.116:0/2755151537 conn(0x55bb3d035f70 msgr2=0x55bb3d032310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- 172.21.15.116:0/2755151537 shutdown_connections 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- 172.21.15.116:0/2755151537 wait complete. 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 -- start start 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 Processor -- start 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.720+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d05dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d033e70 0x55bb3d05e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54301adfb0 con 0x55bb3d045960 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54301ad8e0 con 0x55bb3d033e70 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54301aab40 con 0x55bb3d0334b0 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 10 client.0 _ll_register_callbacks cb 0x7f5420009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51470/0 (socket says 172.21.15.116:51470) 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 -- 172.21.15.116:0/2743182893 learned_addr learned my addr 172.21.15.116:0/2743182893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d033e70 0x55bb3d05e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d05dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 -- 172.21.15.116:0/2743182893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 msgr2=0x55bb3d05dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d05dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 -- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d033e70 msgr2=0x55bb3d05e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:42.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d033e70 0x55bb3d05e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 -- 172.21.15.116:0/2743182893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb3d0c0330 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f542f7fe640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f5424009900 tx=0x7f542402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4a3e53ab5e72bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f540f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424041470 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f5424041610 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.721+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542000fdd0 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5424041810 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 10 client.20703 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55bb3d0c1630 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f542000fdd0 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f542406f030 con 0x55bb3d045960 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 10 client.20703 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 10 client.20703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 20 client.20703 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:42.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d0c1a40 0x55bb3d0c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 4 client.20703 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:45:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb3cfa99f0 con 0x55bb3d0c1a40 2024-03-19T02:45:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.722+0000 7f542ffff640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d0c1a40 0x55bb3d0c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.723+0000 7f540f7fe640 10 client.20703 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:42.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.723+0000 7f542ffff640 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d0c1a40 0x55bb3d0c3e60 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecf8c0f0caa69c3a server_cookie=9db24bfe526fa05e in_seq=0 out_seq=0 2024-03-19T02:45:42.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f540f7fe640 1 -- 172.21.15.116:0/2743182893 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb3cfa99f0 con 0x55bb3d0c1a40 2024-03-19T02:45:42.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f540f7fe640 10 client.20703 handle_command_reply: tid=0 2024-03-19T02:45:42.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 client.20703 shutdown 2024-03-19T02:45:42.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 msgr2=0x55bb3d05e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f5424009900 tx=0x7f542402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 shutdown_connections 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55bb3d0c1a40 0x55bb3d0c3e60 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb3d045960 0x55bb3d05e780 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb3d033e70 0x55bb3d05e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f540f7fe640 0 client.20703 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 --2- 172.21.15.116:0/2743182893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb3d0334b0 0x55bb3d05dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 >> 172.21.15.116:0/2743182893 conn(0x55bb3d035f70 msgr2=0x55bb3d032310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 shutdown_connections 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 1 -- 172.21.15.116:0/2743182893 wait complete. 2024-03-19T02:45:42.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.724+0000 7f5438ffb740 20 client.20703 trim_cache size 0 max 16384 2024-03-19T02:45:42.731 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:45:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 msgr2=0x7f54180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 0x7f54180a4e30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f54280ce9b0 tx=0x7f5428005b20 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 msgr2=0x7f543019a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5428010f90 tx=0x7f542800e490 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 shutdown_connections 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f54180a2970 0x7f54180a4e30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430105660 0x7f543019a670 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5430101330 0x7f54301962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 --2- 172.21.15.116:0/2976509161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5430100970 0x7f5430195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 >> 172.21.15.116:0/2976509161 conn(0x7f54300774c0 msgr2=0x7f54300fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 shutdown_connections 2024-03-19T02:45:42.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:42.725+0000 7f5436337640 1 -- 172.21.15.116:0/2976509161 wait complete. 2024-03-19T02:45:43.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34937 6.12 MB/sec execute 185 sec latency 134.990 ms 2024-03-19T02:45:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:43 smithi162 ceph-mon[35761]: pgmap v564: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 142 op/s 2024-03-19T02:45:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:43 smithi154 ceph-mon[35693]: pgmap v564: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 142 op/s 2024-03-19T02:45:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:43 smithi116 ceph-mon[32263]: pgmap v564: 129 pgs: 129 active+clean; 905 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 142 op/s 2024-03-19T02:45:43.738 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:45:43.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:45:43.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 -- 172.21.15.116:0/2666758192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c068a20 msgr2=0x7f144c068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 --2- 172.21.15.116:0/2666758192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c068a20 0x7f144c068ea0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f1434009930 tx=0x7f143402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 -- 172.21.15.116:0/2666758192 shutdown_connections 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 --2- 172.21.15.116:0/2666758192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c0693e0 0x7f144c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 --2- 172.21.15.116:0/2666758192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c068a20 0x7f144c068ea0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 --2- 172.21.15.116:0/2666758192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 0x7f144c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 -- 172.21.15.116:0/2666758192 >> 172.21.15.116:0/2666758192 conn(0x7f144c077820 msgr2=0x7f144c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 -- 172.21.15.116:0/2666758192 shutdown_connections 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.859+0000 7f1450c12640 1 -- 172.21.15.116:0/2666758192 wait complete. 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 Processor -- start 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 -- start start 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 0x7f144c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 0x7f144c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f144c111a30 con 0x7f144c0693e0 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1118b0 con 0x7f144c107870 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1450c12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f144c111bb0 con 0x7f144c068a20 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51490/0 (socket says 172.21.15.116:51490) 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 -- 172.21.15.116:0/3994283070 learned_addr learned my addr 172.21.15.116:0/3994283070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f144a59c640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 0x7f144c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f144ad9d640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 0x7f144c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 -- 172.21.15.116:0/3994283070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 msgr2=0x7f144c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 0x7f144c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 -- 172.21.15.116:0/3994283070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 msgr2=0x7f144c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 0x7f144c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 -- 172.21.15.116:0/3994283070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c19cfb0 con 0x7f144c0693e0 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f144a59c640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 0x7f144c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.860+0000 7f1443fff640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f1434005940 tx=0x7f1434005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.861+0000 7f14437fe640 1 -- 172.21.15.116:0/3994283070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434043020 con 0x7f144c0693e0 2024-03-19T02:45:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.861+0000 7f14437fe640 1 -- 172.21.15.116:0/3994283070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1434004350 con 0x7f144c0693e0 2024-03-19T02:45:43.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.861+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c19d240 con 0x7f144c0693e0 2024-03-19T02:45:43.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.861+0000 7f14437fe640 1 -- 172.21.15.116:0/3994283070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434003a80 con 0x7f144c0693e0 2024-03-19T02:45:43.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.861+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a4b30 con 0x7f144c0693e0 2024-03-19T02:45:43.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.863+0000 7f14437fe640 1 -- 172.21.15.116:0/3994283070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1434046020 con 0x7f144c0693e0 2024-03-19T02:45:43.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.863+0000 7f1453800740 1 Processor -- start 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.863+0000 7f1453800740 1 -- start start 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f14437fe640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 0x7f14380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a2f880 0x564664a31c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56466498a160 con 0x564664a26ca0 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564664932e10 con 0x564664a2f880 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1453800740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564664986200 con 0x564664a26380 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144ad9d640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 0x7f14380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f14437fe640 1 -- 172.21.15.116:0/3994283070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f14340f0a50 con 0x7f144c0693e0 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51504/0 (socket says 172.21.15.116:51504) 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51870/0 (socket says 172.21.15.116:51870) 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 -- 172.21.15.116:0/2737165502 learned_addr learned my addr 172.21.15.116:0/2737165502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144ad9d640 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a2f880 0x564664a31c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 -- 172.21.15.116:0/2737165502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 msgr2=0x564664a26760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 -- 172.21.15.116:0/2737165502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a2f880 msgr2=0x564664a31c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a2f880 0x564664a31c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 -- 172.21.15.116:0/2737165502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646647667e0 con 0x564664a26ca0 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144a59c640 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f144ad9d640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 0x7f14380a4e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1444005fd0 tx=0x7f1444011bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.864+0000 7f1443fff640 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f14340c5050 tx=0x7f14340b9140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491381af1b015b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.865+0000 7f1440ff9640 1 -- 172.21.15.116:0/2737165502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14340019a0 con 0x564664a26ca0 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.865+0000 7f1440ff9640 1 -- 172.21.15.116:0/2737165502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f143400b0a0 con 0x564664a26ca0 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.865+0000 7f1440ff9640 1 -- 172.21.15.116:0/2737165502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143403f400 con 0x564664a26ca0 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.865+0000 7f1453800740 1 -- 172.21.15.116:0/2737165502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 msgr2=0x564664a2f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.865+0000 7f1453800740 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f14340c5050 tx=0x7f14340b9140 comp rx=0 tx=0).stop 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- 172.21.15.116:0/2737165502 shutdown_connections 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a2f880 0x564664a31c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2f340 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- 172.21.15.116:0/2737165502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a26760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- 172.21.15.116:0/2737165502 >> 172.21.15.116:0/2737165502 conn(0x564664a20ab0 msgr2=0x564664a20e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- 172.21.15.116:0/2737165502 shutdown_connections 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- 172.21.15.116:0/2737165502 wait complete. 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- start start 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 Processor -- start 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a2c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 0x564664a2d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56466498a160 con 0x564664a26ca0 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5646649461a0 con 0x564664a26380 2024-03-19T02:45:43.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.866+0000 7f1453800740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564664a36840 con 0x564664a2f880 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1453800740 10 client.0 _ll_register_callbacks cb 0x56466469dc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f144a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a2c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f144a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a2c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51478/0 (socket says 172.21.15.116:51478) 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51512/0 (socket says 172.21.15.116:51512) 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f144a59c640 1 -- 172.21.15.116:0/1087073270 learned_addr learned my addr 172.21.15.116:0/1087073270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f144ad9d640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 0x564664a2d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 -- 172.21.15.116:0/1087073270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 msgr2=0x564664a2d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 0x564664a2d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 -- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 msgr2=0x564664a2c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a2c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 -- 172.21.15.116:0/1087073270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646647c6f70 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f144ad9d640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 0x564664a2d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1443fff640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f14340bcd70 tx=0x7f1434005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdac454ffb7b193 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1431ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14340d9020 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f14340c50f0 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5646649976a0 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f14340017c0 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1453800740 10 client.20715 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.867+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56466465e170 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5646649976a0 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f14340b8b30 con 0x564664a26ca0 2024-03-19T02:45:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 10 client.20715 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 10 client.20715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 20 client.20715 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664a43a10 0x564664a45e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 4 client.20715 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646649919f0 con 0x564664a43a10 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f144a59c640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664a43a10 0x564664a45e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f144a59c640 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664a43a10 0x564664a45e00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=909eac3c80031ee0 server_cookie=43c9c88dd9f5ad88 in_seq=0 out_seq=0 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.868+0000 7f1431ffb640 10 client.20715 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.870+0000 7f1431ffb640 1 -- 172.21.15.116:0/1087073270 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5646649919f0 con 0x564664a43a10 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.870+0000 7f1431ffb640 10 client.20715 handle_command_reply: tid=0 2024-03-19T02:45:43.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.870+0000 7f1453800740 1 client.20715 shutdown 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 msgr2=0x564664a2ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f14340bcd70 tx=0x7f1434005fd0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 shutdown_connections 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664a43a10 0x564664a45e00 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1431ffb640 0 client.20715 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a2f880 0x564664a2d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a26ca0 0x564664a2ccf0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 --2- 172.21.15.116:0/1087073270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a2c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 >> 172.21.15.116:0/1087073270 conn(0x564664a20ab0 msgr2=0x564664a22800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 shutdown_connections 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 1 -- 172.21.15.116:0/1087073270 wait complete. 2024-03-19T02:45:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.871+0000 7f1453800740 20 client.20715 trim_cache size 0 max 16384 2024-03-19T02:45:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 Processor -- start 2024-03-19T02:45:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 -- start start 2024-03-19T02:45:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a3f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a1f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ac990 con 0x564664a2f880 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ae4d0 con 0x564664a26ca0 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1453800740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ad060 con 0x564664a26380 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1443fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f144ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a1f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f144a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a3f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1443fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51494/0 (socket says 172.21.15.116:51494) 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.893+0000 7f1443fff640 1 -- 172.21.15.116:0/3971736308 learned_addr learned my addr 172.21.15.116:0/3971736308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 -- 172.21.15.116:0/3971736308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 msgr2=0x564664a3f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a3f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 -- 172.21.15.116:0/3971736308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 msgr2=0x564664a1f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a1f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 -- 172.21.15.116:0/3971736308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564664a26ad0 con 0x564664a26ca0 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f144a59c640 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a3f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1443fff640 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f14340bc8b0 tx=0x7f143402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b85dc42747b8df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1433fff640 1 -- 172.21.15.116:0/3971736308 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14340d9040 con 0x564664a26ca0 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1433fff640 1 -- 172.21.15.116:0/3971736308 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14340a60d0 con 0x564664a26ca0 2024-03-19T02:45:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.894+0000 7f1433fff640 1 -- 172.21.15.116:0/3971736308 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14340c0f30 con 0x564664a26ca0 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- 172.21.15.116:0/3971736308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 msgr2=0x564664a2d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f14340bc8b0 tx=0x7f143402bc30 comp rx=0 tx=0).stop 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- 172.21.15.116:0/3971736308 shutdown_connections 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a1f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26ca0 0x564664a2d050 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- 172.21.15.116:0/3971736308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26380 0x564664a3f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- 172.21.15.116:0/3971736308 >> 172.21.15.116:0/3971736308 conn(0x564664a20ab0 msgr2=0x564664a1f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- 172.21.15.116:0/3971736308 shutdown_connections 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- 172.21.15.116:0/3971736308 wait complete. 2024-03-19T02:45:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- start start 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 Processor -- start 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 0x564664a36dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ade00 con 0x564664a2f880 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ad730 con 0x564664a26380 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1aa990 con 0x564664a26ca0 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.895+0000 7f1453800740 10 client.0 _ll_register_callbacks cb 0x7f14340bc800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51508/0 (socket says 172.21.15.116:51508) 2024-03-19T02:45:43.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1443fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 0x564664a36dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51530/0 (socket says 172.21.15.116:51530) 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 -- 172.21.15.116:0/1315951914 learned_addr learned my addr 172.21.15.116:0/1315951914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 -- 172.21.15.116:0/1315951914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 msgr2=0x564664a36dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 0x564664a36dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 -- 172.21.15.116:0/1315951914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 msgr2=0x564664a37310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 -- 172.21.15.116:0/1315951914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564664a37850 con 0x564664a26380 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1443fff640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 0x564664a36dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144ad9d640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f144a59c640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f143c005e00 tx=0x7f143c010ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e24331bbc9e7aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:43.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1431ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c01ec40 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14340a60d0 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.896+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f143c01ede0 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f143c031050 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 10 client.30446 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564664aa99f0 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14340a60d0 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f143c004c40 con 0x564664a26380 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 10 client.30446 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 10 client.30446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 20 client.30446 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:43.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664aaa190 0x564664aac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 4 client.30446 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:45:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56466490fbe0 con 0x564664aaa190 2024-03-19T02:45:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.897+0000 7f1443fff640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664aaa190 0x564664aac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.898+0000 7f1443fff640 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664aaa190 0x564664aac5b0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afab2a695e310061 server_cookie=ea98880eddbac41c in_seq=0 out_seq=0 2024-03-19T02:45:43.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.898+0000 7f1431ffb640 10 client.30446 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1431ffb640 1 -- 172.21.15.116:0/1315951914 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56466490fbe0 con 0x564664aaa190 2024-03-19T02:45:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1431ffb640 10 client.30446 handle_command_reply: tid=0 2024-03-19T02:45:43.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 client.30446 shutdown 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 msgr2=0x564664a3b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f143c005e00 tx=0x7f143c010ed0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 shutdown_connections 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564664aaa190 0x564664aac5b0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564664a2f880 0x564664a37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1431ffb640 0 client.30446 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564664a26ca0 0x564664a36dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 --2- 172.21.15.116:0/1315951914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564664a26380 0x564664a3b950 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.900+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 >> 172.21.15.116:0/1315951914 conn(0x564664a20ab0 msgr2=0x564664a21e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 shutdown_connections 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1453800740 1 -- 172.21.15.116:0/1315951914 wait complete. 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1453800740 20 client.30446 trim_cache size 0 max 16384 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "52a42203-c8fc-425d-a2d3-47dc97fd2471", 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:45:43.907 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:43.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 msgr2=0x7f14380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 0x7f14380a4e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1444005fd0 tx=0x7f1444011bc0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 msgr2=0x7f144c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f1434005940 tx=0x7f1434005dc0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 shutdown_connections 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14380a2970 0x7f14380a4e30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c107870 0x7f144c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f144c0693e0 0x7f144c198550 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 --2- 172.21.15.116:0/3994283070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f144c068a20 0x7f144c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.901+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 >> 172.21.15.116:0/3994283070 conn(0x7f144c077820 msgr2=0x7f144c0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.902+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 shutdown_connections 2024-03-19T02:45:43.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:43.902+0000 7f1450c12640 1 -- 172.21.15.116:0/3994283070 wait complete. 2024-03-19T02:45:43.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:45:43.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:45:44.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 -- 172.21.15.116:0/2552327947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f24101050 msgr2=0x7f5f2410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 --2- 172.21.15.116:0/2552327947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f24101050 0x7f5f2410d560 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009cd0 tx=0x7f5f1c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 -- 172.21.15.116:0/2552327947 shutdown_connections 2024-03-19T02:45:44.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 --2- 172.21.15.116:0/2552327947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f24101050 0x7f5f2410d560 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 --2- 172.21.15.116:0/2552327947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 0x7f5f24100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 --2- 172.21.15.116:0/2552327947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f241066a0 0x7f5f24106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 -- 172.21.15.116:0/2552327947 >> 172.21.15.116:0/2552327947 conn(0x7f5f24077820 msgr2=0x7f5f24077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 -- 172.21.15.116:0/2552327947 shutdown_connections 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.030+0000 7f5f291f4640 1 -- 172.21.15.116:0/2552327947 wait complete. 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 Processor -- start 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 -- start start 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 0x7f5f24196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f24101050 0x7f5f241973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f24110920 con 0x7f5f241066a0 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241107a0 con 0x7f5f24100690 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f291f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f24110aa0 con 0x7f5f24101050 2024-03-19T02:45:44.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f2359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f22d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 0x7f5f24196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f2359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51534/0 (socket says 172.21.15.116:51534) 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f2359e640 1 -- 172.21.15.116:0/1493532180 learned_addr learned my addr 172.21.15.116:0/1493532180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.031+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f24101050 0x7f5f241973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 -- 172.21.15.116:0/1493532180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f24101050 msgr2=0x7f5f241973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f24101050 0x7f5f241973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 -- 172.21.15.116:0/1493532180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 msgr2=0x7f5f24196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 0x7f5f24196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 -- 172.21.15.116:0/1493532180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2419be90 con 0x7f5f241066a0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f2359e640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c004af0 tx=0x7f5f1c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f20d99640 1 -- 172.21.15.116:0/1493532180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c039040 con 0x7f5f241066a0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2419c120 con 0x7f5f241066a0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f20d99640 1 -- 172.21.15.116:0/1493532180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f1c044070 con 0x7f5f241066a0 2024-03-19T02:45:44.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f20d99640 1 -- 172.21.15.116:0/1493532180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0042a0 con 0x7f5f241066a0 2024-03-19T02:45:44.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.032+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f241a3a10 con 0x7f5f241066a0 2024-03-19T02:45:44.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f20d99640 1 -- 172.21.15.116:0/1493532180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5f1c003970 con 0x7f5f241066a0 2024-03-19T02:45:44.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 Processor -- start 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 -- start start 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd7197ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd7197f140 0x55fd71981530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd718d7500 con 0x55fd71975c40 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd7187fbe0 con 0x55fd7197f140 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd718d35a0 con 0x55fd71976560 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.035+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51540/0 (socket says 172.21.15.116:51540) 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f1bfff640 1 -- 172.21.15.116:0/307703172 learned_addr learned my addr 172.21.15.116:0/307703172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd7197ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f20d99640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 0x7f5f100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd7197f140 0x55fd71981530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f2359e640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 0x7f5f100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 -- 172.21.15.116:0/307703172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd7197f140 msgr2=0x55fd71981530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd7197f140 0x55fd71981530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 -- 172.21.15.116:0/307703172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 msgr2=0x55fd71976020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 -- 172.21.15.116:0/307703172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd718a1000 con 0x55fd71976560 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f20d99640 1 -- 172.21.15.116:0/1493532180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f5f1c0f14a0 con 0x7f5f241066a0 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd7197ec00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009c20 tx=0x7f5f14011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e5afc5cd571f144 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f2359e640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 0x7f5f100a4e30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0eea70 tx=0x7f5f1c0095c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f197fa640 1 -- 172.21.15.116:0/307703172 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14004050 con 0x55fd71976560 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f197fa640 1 -- 172.21.15.116:0/307703172 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f140054d0 con 0x55fd71976560 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.036+0000 7f5f197fa640 1 -- 172.21.15.116:0/307703172 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f14004bc0 con 0x55fd71976560 2024-03-19T02:45:44.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 -- 172.21.15.116:0/307703172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 msgr2=0x55fd7197ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd7197ec00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009c20 tx=0x7f5f14011810 comp rx=0 tx=0).stop 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 -- 172.21.15.116:0/307703172 shutdown_connections 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd7197f140 0x55fd71981530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd7197ec00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/307703172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd71976020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 -- 172.21.15.116:0/307703172 >> 172.21.15.116:0/307703172 conn(0x55fd71970370 msgr2=0x55fd71970750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 -- 172.21.15.116:0/307703172 shutdown_connections 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.037+0000 7f5f2bddd740 1 -- 172.21.15.116:0/307703172 wait complete. 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 -- start start 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 Processor -- start 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7197c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd7197cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd71981f30 con 0x55fd7197f140 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd71893160 con 0x55fd71976560 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd719820b0 con 0x55fd71975c40 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f2bddd740 10 client.0 _ll_register_callbacks cb 0x55fd718cfa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd7197cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51940/0 (socket says 172.21.15.116:51940) 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3525023798 learned_addr learned my addr 172.21.15.116:0/3525023798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7197c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 msgr2=0x55fd7197c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7197c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3525023798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 msgr2=0x55fd7197cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd7197cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.038+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3525023798 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd716b57d0 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00f700 tx=0x7f5f0c009950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e8f11dfcb95d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c00fe90 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f140054d0 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f5f0c013120 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5f0c02b050 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 10 client.30436 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd715eb3f0 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f140054d0 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5f0c0266d0 con 0x55fd71975c40 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 10 client.30436 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 10 client.30436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 20 client.30436 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd7198f410 0x55fd71991800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 4 client.30436 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:45:44.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.039+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd718e4a40 con 0x55fd7198f410 2024-03-19T02:45:44.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.040+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd7198f410 0x55fd71991800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.040+0000 7f5f0a7fc640 10 client.30436 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:44.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.040+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd7198f410 0x55fd71991800 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e104f02e95e9df server_cookie=7ea3a84ada6ed0d3 in_seq=0 out_seq=0 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3525023798 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd718e4a40 con 0x55fd7198f410 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f0a7fc640 10 client.30436 handle_command_reply: tid=0 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 client.30436 shutdown 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 msgr2=0x55fd7197c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00f700 tx=0x7f5f0c009950 comp rx=0 tx=0).stop 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 shutdown_connections 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd7198f410 0x55fd71991800 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd7197cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7197c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3525023798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71975c40 0x55fd7197c0f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f0a7fc640 0 client.30436 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 >> 172.21.15.116:0/3525023798 conn(0x55fd71970370 msgr2=0x55fd719720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 shutdown_connections 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3525023798 wait complete. 2024-03-19T02:45:44.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.045+0000 7f5f2bddd740 20 client.30436 trim_cache size 0 max 16384 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 Processor -- start 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 -- start start 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7198ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd7197f140 0x55fd7196d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241aca70 con 0x55fd71975c40 2024-03-19T02:45:44.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241ae5b0 con 0x55fd71976560 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241ad140 con 0x55fd7197f140 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51544/0 (socket says 172.21.15.116:51544) 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 -- 172.21.15.116:0/165133988 learned_addr learned my addr 172.21.15.116:0/165133988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7198ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd7197f140 0x55fd7196d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 -- 172.21.15.116:0/165133988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd7197f140 msgr2=0x55fd7196d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd7197f140 0x55fd7196d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 -- 172.21.15.116:0/165133988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 msgr2=0x55fd7198ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7198ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.066+0000 7f5f1bfff640 1 -- 172.21.15.116:0/165133988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd71991b70 con 0x55fd71975c40 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.067+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c006010 tx=0x7f5f0c0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8751c219f4306c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.067+0000 7f5f18ff9640 1 -- 172.21.15.116:0/165133988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c013120 con 0x55fd71975c40 2024-03-19T02:45:44.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.067+0000 7f5f18ff9640 1 -- 172.21.15.116:0/165133988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f0c0132c0 con 0x55fd71975c40 2024-03-19T02:45:44.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.067+0000 7f5f18ff9640 1 -- 172.21.15.116:0/165133988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c0268d0 con 0x55fd71975c40 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- 172.21.15.116:0/165133988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 msgr2=0x55fd7197e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c006010 tx=0x7f5f0c0092a0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- 172.21.15.116:0/165133988 shutdown_connections 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd7197f140 0x55fd7196d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71976560 0x55fd7198ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/165133988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd71975c40 0x55fd7197e820 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- 172.21.15.116:0/165133988 >> 172.21.15.116:0/165133988 conn(0x55fd71970370 msgr2=0x55fd7196c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- 172.21.15.116:0/165133988 shutdown_connections 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- 172.21.15.116:0/165133988 wait complete. 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- start start 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 Processor -- start 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71975c40 0x55fd71987520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd71978960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241adee0 con 0x55fd7197f140 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241ad810 con 0x55fd71975c40 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.068+0000 7f5f2bddd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241aaa70 con 0x55fd71976560 2024-03-19T02:45:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f2bddd740 10 client.0 _ll_register_callbacks cb 0x7f5f0c013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51558/0 (socket says 172.21.15.116:51558) 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3022890009 learned_addr learned my addr 172.21.15.116:0/3022890009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71975c40 0x55fd71987520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd71978960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3022890009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 msgr2=0x55fd71978960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd71978960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71975c40 msgr2=0x55fd71987520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71975c40 0x55fd71987520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 -- 172.21.15.116:0/3022890009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd71976390 con 0x55fd7197f140 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd71978960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:45:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f1bfff640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c013e90 tx=0x7f5f0c0264d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfbd2eef3c083d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c026840 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f0c0132c0 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9510+0+0 (secure 0 0 0) 0x7f5f0c005070 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5f0c005270 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f2bddd740 10 client.20739 fetch_fsmap learned FSMap version 38 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.069+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd715ab0f0 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f0c0132c0 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5f0c03d040 con 0x55fd7197f140 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 10 client.20739 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 10 client.20739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 20 client.20739 populate_metadata read hostname 'smithi116' 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd719f5820 0x55fd719f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 4 client.20739 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:45:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fd718ded90 con 0x55fd719f5820 2024-03-19T02:45:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd719f5820 0x55fd719f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f22d9d640 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd719f5820 0x55fd719f7c40 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=227c9bb3bbb2830f server_cookie=8ce4b2e541a94142 in_seq=0 out_seq=0 2024-03-19T02:45:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.070+0000 7f5f0a7fc640 10 client.20739 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f0a7fc640 1 -- 172.21.15.116:0/3022890009 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55fd718ded90 con 0x55fd719f5820 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f0a7fc640 10 client.20739 handle_command_reply: tid=0 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 client.20739 shutdown 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 msgr2=0x55fd71978ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c013e90 tx=0x7f5f0c0264d0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 shutdown_connections 2024-03-19T02:45:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd719f5820 0x55fd719f7c40 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd7197f140 0x55fd71978ea0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd71976560 0x55fd71978960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f0a7fc640 0 client.20739 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 --2- 172.21.15.116:0/3022890009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd71975c40 0x55fd71987520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 >> 172.21.15.116:0/3022890009 conn(0x55fd71970370 msgr2=0x55fd7197c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 shutdown_connections 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 1 -- 172.21.15.116:0/3022890009 wait complete. 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.073+0000 7f5f2bddd740 20 client.20739 trim_cache size 0 max 16384 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:45:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (539 inodes in the stack)", 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "52a42203-c8fc-425d-a2d3-47dc97fd2471": { 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "52a42203-c8fc-425d-a2d3-47dc97fd2471", 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:45:44.079 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 msgr2=0x7f5f100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 0x7f5f100a4e30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0eea70 tx=0x7f5f1c0095c0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 msgr2=0x7f5f2419b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c004af0 tx=0x7f5f1c035ce0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 shutdown_connections 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5f100a2970 0x7f5f100a4e30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f241066a0 0x7f5f2419b760 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5f24101050 0x7f5f241973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 --2- 172.21.15.116:0/1493532180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f24100690 0x7f5f24196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:44.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 >> 172.21.15.116:0/1493532180 conn(0x7f5f24077820 msgr2=0x7f5f24102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:44.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 shutdown_connections 2024-03-19T02:45:44.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:45:44.074+0000 7f5f291f4640 1 -- 172.21.15.116:0/1493532180 wait complete. 2024-03-19T02:45:44.085 INFO:tasks.cephfs.filesystem:scrub status for tag:52a42203-c8fc-425d-a2d3-47dc97fd2471 - {'path': '/', 'tag': '52a42203-c8fc-425d-a2d3-47dc97fd2471', 'options': 'recursive,force'} 2024-03-19T02:45:44.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 34989 6.09 MB/sec execute 186 sec latency 213.176 ms 2024-03-19T02:45:44.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:44 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:44.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:44 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:44 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35081 6.10 MB/sec execute 187 sec latency 181.234 ms 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:45 smithi162 ceph-mon[35761]: pgmap v565: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-03-19T02:45:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:45:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:45 smithi154 ceph-mon[35693]: pgmap v565: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-03-19T02:45:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:45:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:45:45.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:45 smithi116 ceph-mon[32263]: pgmap v565: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-03-19T02:45:46.328 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35196 6.11 MB/sec execute 188 sec latency 152.449 ms 2024-03-19T02:45:47.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35296 6.13 MB/sec execute 189 sec latency 166.025 ms 2024-03-19T02:45:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:47 smithi162 ceph-mon[35761]: pgmap v566: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:47 smithi154 ceph-mon[35693]: pgmap v566: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:47 smithi116 ceph-mon[32263]: pgmap v566: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:48.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35409 6.14 MB/sec execute 190 sec latency 161.019 ms 2024-03-19T02:45:49.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35530 6.17 MB/sec execute 191 sec latency 173.489 ms 2024-03-19T02:45:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:49 smithi162 ceph-mon[35761]: pgmap v567: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:49 smithi154 ceph-mon[35693]: pgmap v567: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:49.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:49 smithi116 ceph-mon[32263]: pgmap v567: 129 pgs: 129 active+clean; 912 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 128 op/s 2024-03-19T02:45:50.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35627 6.15 MB/sec execute 192 sec latency 190.938 ms 2024-03-19T02:45:51.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35727 6.13 MB/sec execute 193 sec latency 169.259 ms 2024-03-19T02:45:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:51 smithi162 ceph-mon[35761]: pgmap v568: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.8 MiB/s wr, 191 op/s 2024-03-19T02:45:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:51 smithi154 ceph-mon[35693]: pgmap v568: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.8 MiB/s wr, 191 op/s 2024-03-19T02:45:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:51 smithi116 ceph-mon[32263]: pgmap v568: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.8 MiB/s wr, 191 op/s 2024-03-19T02:45:52.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35811 6.11 MB/sec execute 194 sec latency 188.610 ms 2024-03-19T02:45:53.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:53 smithi154 ceph-mon[35693]: pgmap v569: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 124 op/s 2024-03-19T02:45:53.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 35904 6.10 MB/sec execute 195 sec latency 250.688 ms 2024-03-19T02:45:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:53 smithi162 ceph-mon[35761]: pgmap v569: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 124 op/s 2024-03-19T02:45:53.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:53 smithi116 ceph-mon[32263]: pgmap v569: 129 pgs: 129 active+clean; 925 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 124 op/s 2024-03-19T02:45:54.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36006 6.08 MB/sec execute 196 sec latency 162.955 ms 2024-03-19T02:45:54.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:54 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:55.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:54 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:54 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:55.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36105 6.05 MB/sec execute 197 sec latency 234.218 ms 2024-03-19T02:45:55.917 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:55.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:55.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:55.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:45:55.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:55 smithi162 ceph-mon[35761]: pgmap v570: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-03-19T02:45:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:45:56.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:55 smithi154 ceph-mon[35693]: pgmap v570: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-03-19T02:45:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:56.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:56.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:56.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:45:56.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:55 smithi116 ceph-mon[32263]: pgmap v570: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 187 op/s 2024-03-19T02:45:56.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36205 6.04 MB/sec execute 198 sec latency 152.294 ms 2024-03-19T02:45:57.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36335 6.05 MB/sec execute 199 sec latency 177.439 ms 2024-03-19T02:45:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:57 smithi162 ceph-mon[35761]: pgmap v571: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:45:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:57 smithi154 ceph-mon[35693]: pgmap v571: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:45:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:57 smithi116 ceph-mon[32263]: pgmap v571: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:45:58.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36445 6.04 MB/sec execute 200 sec latency 164.151 ms 2024-03-19T02:45:59.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36551 6.05 MB/sec execute 201 sec latency 163.400 ms 2024-03-19T02:45:59.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:45:59 smithi162 ceph-mon[35761]: pgmap v572: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:45:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:45:59 smithi154 ceph-mon[35693]: pgmap v572: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:45:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:45:59 smithi116 ceph-mon[32263]: pgmap v572: 129 pgs: 129 active+clean; 930 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 126 op/s 2024-03-19T02:46:00.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36658 6.04 MB/sec execute 202 sec latency 169.762 ms 2024-03-19T02:46:01.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36766 6.04 MB/sec execute 203 sec latency 204.397 ms 2024-03-19T02:46:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:01 smithi162 ceph-mon[35761]: pgmap v573: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-03-19T02:46:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:01 smithi154 ceph-mon[35693]: pgmap v573: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-03-19T02:46:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:01 smithi116 ceph-mon[32263]: pgmap v573: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 217 op/s 2024-03-19T02:46:02.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 36893 6.05 MB/sec execute 204 sec latency 161.578 ms 2024-03-19T02:46:03.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37014 6.08 MB/sec execute 205 sec latency 178.693 ms 2024-03-19T02:46:03.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:03 smithi162 ceph-mon[35761]: pgmap v574: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 154 op/s 2024-03-19T02:46:03.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:03 smithi154 ceph-mon[35693]: pgmap v574: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 154 op/s 2024-03-19T02:46:03.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:03 smithi116 ceph-mon[32263]: pgmap v574: 129 pgs: 129 active+clean; 927 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 154 op/s 2024-03-19T02:46:04.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37116 6.08 MB/sec execute 206 sec latency 139.064 ms 2024-03-19T02:46:05.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37209 6.05 MB/sec execute 207 sec latency 224.866 ms 2024-03-19T02:46:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:05 smithi162 ceph-mon[35761]: osdmap e120: 12 total, 12 up, 12 in 2024-03-19T02:46:05.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:05 smithi162 ceph-mon[35761]: pgmap v576: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 220 op/s 2024-03-19T02:46:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:05 smithi154 ceph-mon[35693]: osdmap e120: 12 total, 12 up, 12 in 2024-03-19T02:46:06.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:05 smithi154 ceph-mon[35693]: pgmap v576: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 220 op/s 2024-03-19T02:46:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:05 smithi116 ceph-mon[32263]: osdmap e120: 12 total, 12 up, 12 in 2024-03-19T02:46:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:05 smithi116 ceph-mon[32263]: pgmap v576: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 220 op/s 2024-03-19T02:46:06.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37324 6.05 MB/sec execute 208 sec latency 156.359 ms 2024-03-19T02:46:07.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37421 6.03 MB/sec execute 209 sec latency 146.088 ms 2024-03-19T02:46:08.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:07 smithi162 ceph-mon[35761]: osdmap e121: 12 total, 12 up, 12 in 2024-03-19T02:46:08.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:07 smithi162 ceph-mon[35761]: pgmap v578: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 275 op/s 2024-03-19T02:46:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:07 smithi154 ceph-mon[35693]: osdmap e121: 12 total, 12 up, 12 in 2024-03-19T02:46:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:07 smithi154 ceph-mon[35693]: pgmap v578: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 275 op/s 2024-03-19T02:46:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:07 smithi116 ceph-mon[32263]: osdmap e121: 12 total, 12 up, 12 in 2024-03-19T02:46:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:07 smithi116 ceph-mon[32263]: pgmap v578: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.1 MiB/s wr, 275 op/s 2024-03-19T02:46:08.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37563 6.04 MB/sec execute 210 sec latency 171.617 ms 2024-03-19T02:46:09.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37685 6.03 MB/sec execute 211 sec latency 126.689 ms 2024-03-19T02:46:09.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:09 smithi162 ceph-mon[35761]: pgmap v579: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:46:09.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:09 smithi154 ceph-mon[35693]: pgmap v579: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:46:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:09 smithi116 ceph-mon[32263]: pgmap v579: 129 pgs: 129 active+clean; 919 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-03-19T02:46:10.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37792 6.02 MB/sec execute 212 sec latency 181.839 ms 2024-03-19T02:46:11.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 37906 6.01 MB/sec execute 213 sec latency 270.952 ms 2024-03-19T02:46:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:11 smithi162 ceph-mon[35761]: pgmap v580: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-19T02:46:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:11 smithi154 ceph-mon[35693]: pgmap v580: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-19T02:46:11.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:11 smithi116 ceph-mon[32263]: pgmap v580: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-19T02:46:12.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38020 6.00 MB/sec execute 214 sec latency 124.936 ms 2024-03-19T02:46:13.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38135 5.99 MB/sec execute 215 sec latency 130.825 ms 2024-03-19T02:46:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:13 smithi162 ceph-mon[35761]: pgmap v581: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-03-19T02:46:13.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:13 smithi154 ceph-mon[35693]: pgmap v581: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-03-19T02:46:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:13 smithi116 ceph-mon[32263]: pgmap v581: 129 pgs: 129 active+clean; 929 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 239 op/s 2024-03-19T02:46:14.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:46:14.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:46:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 -- 172.21.15.116:0/1268636458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0100690 msgr2=0x7f98e0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 --2- 172.21.15.116:0/1268636458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0100690 0x7f98e0100b10 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009930 tx=0x7f98d002bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 -- 172.21.15.116:0/1268636458 shutdown_connections 2024-03-19T02:46:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 --2- 172.21.15.116:0/1268636458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0101050 0x7f98e010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 --2- 172.21.15.116:0/1268636458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0100690 0x7f98e0100b10 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 --2- 172.21.15.116:0/1268636458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e01066a0 0x7f98e0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.221+0000 7f98e7be6640 1 -- 172.21.15.116:0/1268636458 >> 172.21.15.116:0/1268636458 conn(0x7f98e0077820 msgr2=0x7f98e0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.222+0000 7f98e7be6640 1 -- 172.21.15.116:0/1268636458 shutdown_connections 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.222+0000 7f98e7be6640 1 -- 172.21.15.116:0/1268636458 wait complete. 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.222+0000 7f98e7be6640 1 Processor -- start 2024-03-19T02:46:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.222+0000 7f98e7be6640 1 -- start start 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0101050 0x7f98e0197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 0x7f98e019b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0110960 con 0x7f98e01066a0 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01107e0 con 0x7f98e0101050 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e7be6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0110ae0 con 0x7f98e0100690 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43586/0 (socket says 172.21.15.116:43586) 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 -- 172.21.15.116:0/36466922 learned_addr learned my addr 172.21.15.116:0/36466922 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5181640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0101050 0x7f98e0197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e6183640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 0x7f98e019b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 -- 172.21.15.116:0/36466922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0101050 msgr2=0x7f98e0197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0101050 0x7f98e0197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 -- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 msgr2=0x7f98e019b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 0x7f98e019b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 -- 172.21.15.116:0/36466922 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e019bed0 con 0x7f98e0100690 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e6183640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 0x7f98e019b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.223+0000 7f98e5982640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00eb00 tx=0x7f98dc009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.224+0000 7f98d6ffd640 1 -- 172.21.15.116:0/36466922 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc01c070 con 0x7f98e0100690 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.224+0000 7f98d6ffd640 1 -- 172.21.15.116:0/36466922 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98dc00bc30 con 0x7f98e0100690 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.224+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e019c1c0 con 0x7f98e0100690 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.224+0000 7f98d6ffd640 1 -- 172.21.15.116:0/36466922 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc016610 con 0x7f98e0100690 2024-03-19T02:46:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.224+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e0068980 con 0x7f98e0100690 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.226+0000 7f98d6ffd640 1 -- 172.21.15.116:0/36466922 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f98dc01f070 con 0x7f98e0100690 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98d6ffd640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 0x7f98cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 Processor -- start 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98d6ffd640 1 -- 172.21.15.116:0/36466922 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f98dc0d0720 con 0x7f98e0100690 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98e5181640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 0x7f98cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 -- start start 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 0x5639059d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059e2f30 0x5639059e5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56390593d500 con 0x5639059d9a30 2024-03-19T02:46:14.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639058e5be0 con 0x5639059e2f30 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.227+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5639059395a0 con 0x5639059da350 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 0x7f98cc0a4eb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f98d0005ab0 tx=0x7f98d0005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e6183640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059e2f30 0x5639059e5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5982640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 0x5639059d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43592/0 (socket says 172.21.15.116:43592) 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 -- 172.21.15.116:0/4019374006 learned_addr learned my addr 172.21.15.116:0/4019374006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 -- 172.21.15.116:0/4019374006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059e2f30 msgr2=0x5639059e5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059e2f30 0x5639059e5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 -- 172.21.15.116:0/4019374006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 msgr2=0x5639059d9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 0x5639059d9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 -- 172.21.15.116:0/4019374006 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563905907000 con 0x5639059da350 2024-03-19T02:46:14.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5982640 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 0x5639059d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.228+0000 7f98e5181640 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f98d00096c0 tx=0x7f98d003e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53292285168f6558 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98cbfff640 1 -- 172.21.15.116:0/4019374006 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d004f070 con 0x5639059da350 2024-03-19T02:46:14.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98cbfff640 1 -- 172.21.15.116:0/4019374006 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98d0045070 con 0x5639059da350 2024-03-19T02:46:14.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98cbfff640 1 -- 172.21.15.116:0/4019374006 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0004a60 con 0x5639059da350 2024-03-19T02:46:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98ea7d5740 1 -- 172.21.15.116:0/4019374006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 msgr2=0x5639059e29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f98d00096c0 tx=0x7f98d003e000 comp rx=0 tx=0).stop 2024-03-19T02:46:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.229+0000 7f98ea7d5740 1 -- 172.21.15.116:0/4019374006 shutdown_connections 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059e2f30 0x5639059e5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e29f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/4019374006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059d9a30 0x5639059d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 -- 172.21.15.116:0/4019374006 >> 172.21.15.116:0/4019374006 conn(0x5639059d4160 msgr2=0x5639059d4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 -- 172.21.15.116:0/4019374006 shutdown_connections 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 -- 172.21.15.116:0/4019374006 wait complete. 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 -- start start 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.230+0000 7f98ea7d5740 1 Processor -- start 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059e09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5639059ea0d0 con 0x5639059e2f30 2024-03-19T02:46:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639058f9160 con 0x5639059d9a30 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5639059ea250 con 0x5639059da350 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98ea7d5740 10 client.0 _ll_register_callbacks cb 0x563905935a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38296/0 (socket says 172.21.15.116:38296) 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 -- 172.21.15.116:0/3873094412 learned_addr learned my addr 172.21.15.116:0/3873094412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e6183640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059e09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5181640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 -- 172.21.15.116:0/3873094412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 msgr2=0x5639059e0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 -- 172.21.15.116:0/3873094412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 msgr2=0x5639059e09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059e09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 -- 172.21.15.116:0/3873094412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56390571b7d0 con 0x5639059d9a30 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98e5982640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00ead0 tx=0x7f98dc09c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c92fc53199c1a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.231+0000 7f98c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc0b5060 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98d0045070 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f98dc0d02c0 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f98dc0a0af0 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 10 client.30476 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5639056513f0 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98d0045070 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f98dc0988c0 con 0x5639059d9a30 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 10 client.30476 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 10 client.30476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 20 client.30476 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5639059f7670 0x5639059f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 4 client.30476 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:14.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.232+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56390594aa40 con 0x5639059f7670 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.233+0000 7f98e5181640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5639059f7670 0x5639059f9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.233+0000 7f98c97fa640 10 client.30476 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.233+0000 7f98e5181640 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5639059f7670 0x5639059f9a60 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5204c2e2254fa66 server_cookie=94b91078b0ef1213 in_seq=0 out_seq=0 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98c97fa640 1 -- 172.21.15.116:0/3873094412 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56390594aa40 con 0x5639059f7670 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98c97fa640 10 client.30476 handle_command_reply: tid=0 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 client.30476 shutdown 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 msgr2=0x5639059dff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00ead0 tx=0x7f98dc09c470 comp rx=0 tx=0).stop 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 shutdown_connections 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5639059f7670 0x5639059f9a60 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059e09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059e0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/3873094412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059dff40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 >> 172.21.15.116:0/3873094412 conn(0x5639059d4160 msgr2=0x5639059d5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98c97fa640 0 client.30476 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 shutdown_connections 2024-03-19T02:46:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 1 -- 172.21.15.116:0/3873094412 wait complete. 2024-03-19T02:46:14.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.235+0000 7f98ea7d5740 20 client.30476 trim_cache size 0 max 16384 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 Processor -- start 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 -- start start 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01aca40 con 0x5639059e2f30 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01ae580 con 0x5639059d9a30 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01ad110 con 0x5639059da350 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e5982640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e5982640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38310/0 (socket says 172.21.15.116:38310) 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43602/0 (socket says 172.21.15.116:43602) 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e5982640 1 -- 172.21.15.116:0/2203748134 learned_addr learned my addr 172.21.15.116:0/2203748134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.259+0000 7f98e6183640 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 -- 172.21.15.116:0/2203748134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 msgr2=0x5639059e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 -- 172.21.15.116:0/2203748134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 msgr2=0x5639059f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 -- 172.21.15.116:0/2203748134 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639059da180 con 0x5639059da350 2024-03-19T02:46:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5982640 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98e5181640 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f98d002c170 tx=0x7f98d00045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52fcc3f5d558e2d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98cb7fe640 1 -- 172.21.15.116:0/2203748134 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d004f040 con 0x5639059da350 2024-03-19T02:46:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98cb7fe640 1 -- 172.21.15.116:0/2203748134 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98d0045040 con 0x5639059da350 2024-03-19T02:46:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.260+0000 7f98cb7fe640 1 -- 172.21.15.116:0/2203748134 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d00046c0 con 0x5639059da350 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- 172.21.15.116:0/2203748134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 msgr2=0x5639059cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f98d002c170 tx=0x7f98d00045d0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- 172.21.15.116:0/2203748134 shutdown_connections 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059da350 0x5639059cef80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/2203748134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059d9a30 0x5639059e2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- 172.21.15.116:0/2203748134 >> 172.21.15.116:0/2203748134 conn(0x5639059d4160 msgr2=0x5639059dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- 172.21.15.116:0/2203748134 shutdown_connections 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- 172.21.15.116:0/2203748134 wait complete. 2024-03-19T02:46:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 -- start start 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.261+0000 7f98ea7d5740 1 Processor -- start 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 0x5639059dcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 0x5639059dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01adeb0 con 0x5639059e2f30 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01ad7e0 con 0x5639059da350 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01aaa40 con 0x5639059d9a30 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98ea7d5740 10 client.0 _ll_register_callbacks cb 0x7f98dc00e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e5982640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 0x5639059dcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e5181640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 0x5639059dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37244/0 (socket says 172.21.15.116:37244) 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e5982640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 0x5639059dcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43604/0 (socket says 172.21.15.116:43604) 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e5181640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 0x5639059dd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38326/0 (socket says 172.21.15.116:38326) 2024-03-19T02:46:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 -- 172.21.15.116:0/583514345 learned_addr learned my addr 172.21.15.116:0/583514345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 -- 172.21.15.116:0/583514345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 msgr2=0x5639059dcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 0x5639059dcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 -- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 msgr2=0x5639059dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 0x5639059dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 -- 172.21.15.116:0/583514345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639056110f0 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98e6183640 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009320 tx=0x7f98d80128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d3e246fc0b7ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.262+0000 7f98c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8004d20 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f98d8003970 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98d0045040 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f98d8029050 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 10 client.20757 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563905a5dda0 con 0x5639059e2f30 2024-03-19T02:46:14.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98d0045040 con 0x5639059e2f30 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f98d801e480 con 0x5639059e2f30 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 10 client.20757 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 10 client.20757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 20 client.20757 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563905a5e5a0 0x563905a609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 4 client.20757 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563905944d90 con 0x563905a5e5a0 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.263+0000 7f98e5982640 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563905a5e5a0 0x563905a609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.264+0000 7f98e5982640 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563905a5e5a0 0x563905a609c0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88aa0b81050cae04 server_cookie=5152554ea3752e34 in_seq=0 out_seq=0 2024-03-19T02:46:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.264+0000 7f98c97fa640 10 client.20757 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98c97fa640 1 -- 172.21.15.116:0/583514345 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563905944d90 con 0x563905a5e5a0 2024-03-19T02:46:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98c97fa640 10 client.20757 handle_command_reply: tid=0 2024-03-19T02:46:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 client.20757 shutdown 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 msgr2=0x5639059dd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009320 tx=0x7f98d80128e0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 shutdown_connections 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563905a5e5a0 0x563905a609c0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639059e2f30 0x5639059dd890 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639059da350 0x5639059dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98c97fa640 0 client.20757 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 --2- 172.21.15.116:0/583514345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639059d9a30 0x5639059dcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 >> 172.21.15.116:0/583514345 conn(0x5639059d4160 msgr2=0x5639059d5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 shutdown_connections 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 1 -- 172.21.15.116:0/583514345 wait complete. 2024-03-19T02:46:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.266+0000 7f98ea7d5740 20 client.20757 trim_cache size 0 max 16384 2024-03-19T02:46:14.272 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:14.272 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:46:14.272 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:46:14.272 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 msgr2=0x7f98cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 0x7f98cc0a4eb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f98d0005ab0 tx=0x7f98d0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 msgr2=0x7f98e0196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00eb00 tx=0x7f98dc009f20 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 shutdown_connections 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98cc0a29f0 0x7f98cc0a4eb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e01066a0 0x7f98e019b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0101050 0x7f98e0197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 --2- 172.21.15.116:0/36466922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98e0100690 0x7f98e0196ee0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 >> 172.21.15.116:0/36466922 conn(0x7f98e0077820 msgr2=0x7f98e0102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 shutdown_connections 2024-03-19T02:46:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.267+0000 7f98e7be6640 1 -- 172.21.15.116:0/36466922 wait complete. 2024-03-19T02:46:14.279 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:46:14.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:46:14.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38240 5.98 MB/sec execute 216 sec latency 124.536 ms 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 -- 172.21.15.116:0/2755263342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 msgr2=0x7f945010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 --2- 172.21.15.116:0/2755263342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945010e300 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f9448009ca0 tx=0x7f944802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 -- 172.21.15.116:0/2755263342 shutdown_connections 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 --2- 172.21.15.116:0/2755263342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945010e300 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 --2- 172.21.15.116:0/2755263342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f945010b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.402+0000 7f94579b9640 1 --2- 172.21.15.116:0/2755263342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 0x7f9450102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.403+0000 7f94579b9640 1 -- 172.21.15.116:0/2755263342 >> 172.21.15.116:0/2755263342 conn(0x7f9450077820 msgr2=0x7f9450077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.403+0000 7f94579b9640 1 -- 172.21.15.116:0/2755263342 shutdown_connections 2024-03-19T02:46:14.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.403+0000 7f94579b9640 1 -- 172.21.15.116:0/2755263342 wait complete. 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 Processor -- start 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 -- start start 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 0x7f9450197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9450111b10 con 0x7f945010be60 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9450111990 con 0x7f9450102a90 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.404+0000 7f94579b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9450111c90 con 0x7f94501033b0 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43608/0 (socket says 172.21.15.116:43608) 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 -- 172.21.15.116:0/4144630971 learned_addr learned my addr 172.21.15.116:0/4144630971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9455755640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 0x7f9450197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9455f56640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 -- 172.21.15.116:0/4144630971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 msgr2=0x7f9450197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 0x7f9450197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 -- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 msgr2=0x7f945019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 -- 172.21.15.116:0/4144630971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945019cf90 con 0x7f94501033b0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f9454f54640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f944000e9f0 tx=0x7f944000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944000cd80 con 0x7f94501033b0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945019d280 con 0x7f94501033b0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94400054d0 con 0x7f94501033b0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.405+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440010490 con 0x7f94501033b0 2024-03-19T02:46:14.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.406+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94501a4b10 con 0x7f94501033b0 2024-03-19T02:46:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.407+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9450068610 con 0x7f94501033b0 2024-03-19T02:46:14.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.408+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9440005020 con 0x7f94501033b0 2024-03-19T02:46:14.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.409+0000 7f94467fc640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 0x7f943c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.409+0000 7f9455755640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 0x7f943c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.409+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f94400cf5b0 con 0x7f94501033b0 2024-03-19T02:46:14.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.410+0000 7f9455755640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 0x7f943c0a4e30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f944c009bd0 tx=0x7f944c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.418+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f944009bf30 con 0x7f94501033b0 2024-03-19T02:46:14.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.737+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9450106f50 con 0x7f94501033b0 2024-03-19T02:46:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.739+0000 7f94467fc640 1 -- 172.21.15.116:0/4144630971 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f944009b830 con 0x7f94501033b0 2024-03-19T02:46:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:46:14.740 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:46:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 msgr2=0x7f943c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 0x7f943c0a4e30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f944c009bd0 tx=0x7f944c005f90 comp rx=0 tx=0).stop 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 msgr2=0x7f9450198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f944000e9f0 tx=0x7f944000edf0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 shutdown_connections 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f943c0a2970 0x7f943c0a4e30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f945010be60 0x7f945019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94501033b0 0x7f9450198530 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 --2- 172.21.15.116:0/4144630971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9450102a90 0x7f9450197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 >> 172.21.15.116:0/4144630971 conn(0x7f9450077820 msgr2=0x7f94500fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 shutdown_connections 2024-03-19T02:46:14.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.741+0000 7f94579b9640 1 -- 172.21.15.116:0/4144630971 wait complete. 2024-03-19T02:46:14.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- 172.21.15.116:0/1948232807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69981077a0 msgr2=0x7f6998107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 --2- 172.21.15.116:0/1948232807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69981077a0 0x7f6998107b80 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f698c009930 tx=0x7f698c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- 172.21.15.116:0/1948232807 shutdown_connections 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 --2- 172.21.15.116:0/1948232807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998102150 0x7f699810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 --2- 172.21.15.116:0/1948232807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6998101790 0x7f6998101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 --2- 172.21.15.116:0/1948232807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69981077a0 0x7f6998107b80 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- 172.21.15.116:0/1948232807 >> 172.21.15.116:0/1948232807 conn(0x7f69980fd2c0 msgr2=0x7f69980ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- 172.21.15.116:0/1948232807 shutdown_connections 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- 172.21.15.116:0/1948232807 wait complete. 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 Processor -- start 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.892+0000 7f699f0f1640 1 -- start start 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 0x7f6998198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69981077a0 0x7f699819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6998111a20 con 0x7f6998102150 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69981118a0 con 0x7f69981077a0 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6998111ba0 con 0x7f6998101790 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37270/0 (socket says 172.21.15.116:37270) 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 -- 172.21.15.116:0/922639077 learned_addr learned my addr 172.21.15.116:0/922639077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 0x7f6998198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699d68e640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69981077a0 0x7f699819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 -- 172.21.15.116:0/922639077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 msgr2=0x7f6998198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 0x7f6998198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 -- 172.21.15.116:0/922639077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69981077a0 msgr2=0x7f699819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69981077a0 0x7f699819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 -- 172.21.15.116:0/922639077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699819d000 con 0x7f6998102150 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699ce8d640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 0x7f6998198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6997fff640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f698400e9f0 tx=0x7f698400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6995ffb640 1 -- 172.21.15.116:0/922639077 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698400cd80 con 0x7f6998102150 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699819d2f0 con 0x7f6998102150 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6995ffb640 1 -- 172.21.15.116:0/922639077 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69840108e0 con 0x7f6998102150 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.893+0000 7f6995ffb640 1 -- 172.21.15.116:0/922639077 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984018920 con 0x7f6998102150 2024-03-19T02:46:14.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.894+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69981a4b80 con 0x7f6998102150 2024-03-19T02:46:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.896+0000 7f6995ffb640 1 -- 172.21.15.116:0/922639077 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6984010a90 con 0x7f6998102150 2024-03-19T02:46:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 Processor -- start 2024-03-19T02:46:14.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 -- start start 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebf2d0 0x55c583ebf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ecabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c583e224b0 con 0x55c583ec87d0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c583dcad70 con 0x55c583ebfbf0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f69a1db9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c583e1e550 con 0x55c583ebf2d0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f6995ffb640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 0x7f69880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699d68e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebf2d0 0x55c583ebf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38388/0 (socket says 172.21.15.116:38388) 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 -- 172.21.15.116:0/1744557883 learned_addr learned my addr 172.21.15.116:0/1744557883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f6997fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ecabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699d68e640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 0x7f69880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f6995ffb640 1 -- 172.21.15.116:0/922639077 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f69840cf4a0 con 0x7f6998102150 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 -- 172.21.15.116:0/1744557883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebf2d0 msgr2=0x55c583ebf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebf2d0 0x55c583ebf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 -- 172.21.15.116:0/1744557883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 msgr2=0x55c583ecabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ecabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.897+0000 7f699ce8d640 1 -- 172.21.15.116:0/1744557883 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c583ded210 con 0x55c583ebfbf0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.898+0000 7f699ce8d640 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f698c036040 tx=0x7f698c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=331954233b61347a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.898+0000 7f699d68e640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 0x7f69880a4e30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6990005fd0 tx=0x7f6990009f30 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.898+0000 7f6982ffd640 1 -- 172.21.15.116:0/1744557883 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c045070 con 0x55c583ebfbf0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.898+0000 7f6982ffd640 1 -- 172.21.15.116:0/1744557883 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698c02c7e0 con 0x55c583ebfbf0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.898+0000 7f6982ffd640 1 -- 172.21.15.116:0/1744557883 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c003950 con 0x55c583ebfbf0 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- 172.21.15.116:0/1744557883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 msgr2=0x55c583ec8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f698c036040 tx=0x7f698c0095f0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- 172.21.15.116:0/1744557883 shutdown_connections 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ecabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ec8290 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 --2- 172.21.15.116:0/1744557883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebf2d0 0x55c583ebf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- 172.21.15.116:0/1744557883 >> 172.21.15.116:0/1744557883 conn(0x55c583eb9a00 msgr2=0x55c583eb9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- 172.21.15.116:0/1744557883 shutdown_connections 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- 172.21.15.116:0/1744557883 wait complete. 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 -- start start 2024-03-19T02:46:14.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.899+0000 7f69a1db9740 1 Processor -- start 2024-03-19T02:46:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ec57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 0x55c583ec6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c583ecb560 con 0x55c583ebf2d0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c583dde310 con 0x55c583ec87d0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c583ecb6e0 con 0x55c583ebfbf0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f69a1db9740 10 client.0 _ll_register_callbacks cb 0x55c583dd22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f6997fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f6997fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43644/0 (socket says 172.21.15.116:43644) 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f6997fff640 1 -- 172.21.15.116:0/3685182575 learned_addr learned my addr 172.21.15.116:0/3685182575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f699d68e640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 0x55c583ec6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ec57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.900+0000 7f6997fff640 1 -- 172.21.15.116:0/3685182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 msgr2=0x55c583ec6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6997fff640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 0x55c583ec6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6997fff640 1 -- 172.21.15.116:0/3685182575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 msgr2=0x55c583ec57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6997fff640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ec57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6997fff640 1 -- 172.21.15.116:0/3685182575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c583b363f0 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f699d68e640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 0x55c583ec6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ec57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6997fff640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6984098240 tx=0x7f698409b430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fac2978b0fcfc288 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6977fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984026040 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f698c02c7e0 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f69840860b0 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f698409f910 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f69a1db9740 10 client.30478 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.901+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c583c007d0 con 0x55c583ebfbf0 2024-03-19T02:46:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698c02c7e0 con 0x55c583ebfbf0 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f69840c8ce0 con 0x55c583ebfbf0 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 10 client.30478 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 10 client.30478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 20 client.30478 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583edce00 0x55c583edf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 4 client.30478 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c583e803e0 con 0x55c583edce00 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583edce00 0x55c583edf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f6977fff640 10 client.30478 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.902+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583edce00 0x55c583edf270 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6389fa9a615b231 server_cookie=1616139d2e6b9c56 in_seq=0 out_seq=0 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f6977fff640 1 -- 172.21.15.116:0/3685182575 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c583e803e0 con 0x55c583edce00 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f6977fff640 10 client.30478 handle_command_reply: tid=0 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 client.30478 shutdown 2024-03-19T02:46:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 msgr2=0x55c583ec5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6984098240 tx=0x7f698409b430 comp rx=0 tx=0).stop 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 shutdown_connections 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583edce00 0x55c583edf270 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ec87d0 0x55c583ec6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ec5d00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3685182575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ec57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 >> 172.21.15.116:0/3685182575 conn(0x55c583eb9a00 msgr2=0x55c583ebb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f6977fff640 0 client.30478 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 shutdown_connections 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.903+0000 7f69a1db9740 1 -- 172.21.15.116:0/3685182575 wait complete. 2024-03-19T02:46:14.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.904+0000 7f69a1db9740 20 client.30478 trim_cache size 0 max 16384 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.924+0000 7f69a1db9740 1 Processor -- start 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 -- start start 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 0x55c583ec75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ee4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ed0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69981ac9a0 con 0x55c583ec87d0 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69981ae4e0 con 0x55c583ebf2d0 2024-03-19T02:46:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f69a1db9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69981ad070 con 0x55c583ebfbf0 2024-03-19T02:46:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f6997fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ee4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 0x55c583ec75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f6997fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ee4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43652/0 (socket says 172.21.15.116:43652) 2024-03-19T02:46:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f6997fff640 1 -- 172.21.15.116:0/803897574 learned_addr learned my addr 172.21.15.116:0/803897574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699d68e640 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ed0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699ce8d640 1 -- 172.21.15.116:0/803897574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 msgr2=0x55c583ee4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699ce8d640 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ee4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699ce8d640 1 -- 172.21.15.116:0/803897574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 msgr2=0x55c583ed0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.925+0000 7f699ce8d640 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ed0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f699ce8d640 1 -- 172.21.15.116:0/803897574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c583ec05a0 con 0x55c583ebf2d0 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f699d68e640 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ed0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f699ce8d640 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 0x55c583ec75f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f698c02c170 tx=0x7f698c02c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3b592427c6e7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f69827fc640 1 -- 172.21.15.116:0/803897574 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c045040 con 0x55c583ebf2d0 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f69827fc640 1 -- 172.21.15.116:0/803897574 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698c0035f0 con 0x55c583ebf2d0 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.926+0000 7f69827fc640 1 -- 172.21.15.116:0/803897574 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c02cb30 con 0x55c583ebf2d0 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 -- 172.21.15.116:0/803897574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 msgr2=0x55c583ec75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 0x55c583ec75f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f698c02c170 tx=0x7f698c02c3b0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 -- 172.21.15.116:0/803897574 shutdown_connections 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ec87d0 0x55c583ed0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ebfbf0 0x55c583ee4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 --2- 172.21.15.116:0/803897574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebf2d0 0x55c583ec75f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 -- 172.21.15.116:0/803897574 >> 172.21.15.116:0/803897574 conn(0x55c583eb9a00 msgr2=0x55c583ec6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.927+0000 7f69a1db9740 1 -- 172.21.15.116:0/803897574 shutdown_connections 2024-03-19T02:46:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.928+0000 7f69a1db9740 1 -- 172.21.15.116:0/803897574 wait complete. 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.928+0000 7f69a1db9740 1 -- start start 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.928+0000 7f69a1db9740 1 Processor -- start 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ecd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ec87d0 0x55c583ecd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69981ade10 con 0x55c583ebf2d0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69981ad740 con 0x55c583ebfbf0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69981aa9a0 con 0x55c583ec87d0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f69a1db9740 10 client.0 _ll_register_callbacks cb 0x7f698400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37312/0 (socket says 172.21.15.116:37312) 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 -- 172.21.15.116:0/3429081277 learned_addr learned my addr 172.21.15.116:0/3429081277 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699d68e640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ec87d0 0x55c583ecd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f6997fff640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ecd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 -- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ec87d0 msgr2=0x55c583ecd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ec87d0 0x55c583ecd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 -- 172.21.15.116:0/3429081277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 msgr2=0x55c583ecd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ecd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 -- 172.21.15.116:0/3429081277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c583ed7320 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f699ce8d640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f698c005a20 tx=0x7f698c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce236535b34442b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.929+0000 7f6977fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c045040 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f698c0035f0 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f698c0041b0 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f698c0044e0 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f69a1db9740 10 client.20781 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c583ed8620 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.930+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698c054440 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f698c03f440 con 0x55c583ebf2d0 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 10 client.20781 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 10 client.20781 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 20 client.20781 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583ed8900 0x55c583f49970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 4 client.20781 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f698c0035f0 con 0x55c583ed8900 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f6997fff640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583ed8900 0x55c583f49970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f6977fff640 10 client.20781 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.931+0000 7f6997fff640 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583ed8900 0x55c583f49970 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ee41da2311acbda5 server_cookie=64870904a8a79829 in_seq=0 out_seq=0 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f6977fff640 1 -- 172.21.15.116:0/3429081277 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f698c0035f0 con 0x55c583ed8900 2024-03-19T02:46:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f6977fff640 10 client.20781 handle_command_reply: tid=0 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 client.20781 shutdown 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 msgr2=0x55c583ed0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f698c005a20 tx=0x7f698c005a50 comp rx=0 tx=0).stop 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 shutdown_connections 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55c583ed8900 0x55c583f49970 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f6977fff640 0 client.20781 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c583ec87d0 0x55c583ecd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c583ebfbf0 0x55c583ecd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 --2- 172.21.15.116:0/3429081277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c583ebf2d0 0x55c583ed0d70 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 >> 172.21.15.116:0/3429081277 conn(0x55c583eb9a00 msgr2=0x55c583eb8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 shutdown_connections 2024-03-19T02:46:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 1 -- 172.21.15.116:0/3429081277 wait complete. 2024-03-19T02:46:14.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.932+0000 7f69a1db9740 20 client.20781 trim_cache size 0 max 16384 2024-03-19T02:46:14.938 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 msgr2=0x7f69880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 0x7f69880a4e30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6990005fd0 tx=0x7f6990009f30 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 msgr2=0x7f6998198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f698400e9f0 tx=0x7f698400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 shutdown_connections 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f69880a2970 0x7f69880a4e30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69981077a0 0x7f699819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6998102150 0x7f6998198550 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 --2- 172.21.15.116:0/922639077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6998101790 0x7f6998198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 >> 172.21.15.116:0/922639077 conn(0x7f69980fd2c0 msgr2=0x7f69980fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 shutdown_connections 2024-03-19T02:46:14.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:14.933+0000 7f699f0f1640 1 -- 172.21.15.116:0/922639077 wait complete. 2024-03-19T02:46:14.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:46:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:14 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4144630971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:15.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/2352281120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1017c0 msgr2=0x7f0ebc101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/2352281120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc101c40 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009930 tx=0x7f0eb002bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:15.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/2352281120 shutdown_connections 2024-03-19T02:46:15.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/2352281120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc10e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/2352281120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc101c40 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/2352281120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/2352281120 >> 172.21.15.116:0/2352281120 conn(0x7f0ebc0fd2d0 msgr2=0x7f0ebc0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/2352281120 shutdown_connections 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.079+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/2352281120 wait complete. 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.080+0000 7f0ec0b5b640 1 Processor -- start 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.080+0000 7f0ec0b5b640 1 -- start start 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.080+0000 7f0ec0b5b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ec0b5b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ec0b5b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ec0b5b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc111a90 con 0x7f0ebc1077d0 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ec0b5b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc111910 con 0x7f0ebc1017c0 2024-03-19T02:46:15.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ec0b5b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc111c10 con 0x7f0ebc102180 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38424/0 (socket says 172.21.15.116:38424) 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37336/0 (socket says 172.21.15.116:37336) 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eba59c640 1 -- 172.21.15.116:0/689130951 learned_addr learned my addr 172.21.15.116:0/689130951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/689130951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 msgr2=0x7f0ebc073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 msgr2=0x7f0ebc0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/689130951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc1a64f0 con 0x7f0ebc102180 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc074060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009230 tx=0x7f0eb00045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.081+0000 7f0eab7fe640 1 -- 172.21.15.116:0/689130951 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0043020 con 0x7f0ebc102180 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.082+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc1a6780 con 0x7f0ebc102180 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.082+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc1a6cf0 con 0x7f0ebc102180 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.082+0000 7f0eab7fe640 1 -- 172.21.15.116:0/689130951 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb002ca70 con 0x7f0ebc102180 2024-03-19T02:46:15.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.082+0000 7f0eab7fe640 1 -- 172.21.15.116:0/689130951 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00039b0 con 0x7f0ebc102180 2024-03-19T02:46:15.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.084+0000 7f0eab7fe640 1 -- 172.21.15.116:0/689130951 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0eb0046020 con 0x7f0ebc102180 2024-03-19T02:46:15.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0eab7fe640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 0x7f0ea40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 Processor -- start 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0eba59c640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 0x7f0ea40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0eab7fe640 1 -- 172.21.15.116:0/689130951 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f0eb00f0fb0 con 0x7f0ebc102180 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 -- start start 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fa56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fae290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558931f084b0 con 0x558931fae7d0 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558931eb0d70 con 0x558931fa52d0 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0ec3822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558931f04550 con 0x558931fa5bf0 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.085+0000 7f0eba59c640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 0x7f0ea40a4eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c10 tx=0x7f0eac005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fa56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fa56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38426/0 (socket says 172.21.15.116:38426) 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eba59c640 1 -- 172.21.15.116:0/4025422217 learned_addr learned my addr 172.21.15.116:0/4025422217 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/4025422217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 msgr2=0x558931fa56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fa56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/4025422217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 msgr2=0x558931fb0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/4025422217 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558931ed3210 con 0x558931fa5bf0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fae290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00ee430 tx=0x7f0eb002c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5ea76517055a608 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0ea8ff9640 1 -- 172.21.15.116:0/4025422217 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00c1490 con 0x558931fa5bf0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0ea8ff9640 1 -- 172.21.15.116:0/4025422217 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb00c5ae0 con 0x558931fa5bf0 2024-03-19T02:46:15.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.086+0000 7f0ea8ff9640 1 -- 172.21.15.116:0/4025422217 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00bd5a0 con 0x558931fa5bf0 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 -- 172.21.15.116:0/4025422217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 msgr2=0x558931fae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fae290 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00ee430 tx=0x7f0eb002c740 comp rx=0 tx=0).stop 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 -- 172.21.15.116:0/4025422217 shutdown_connections 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fae290 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 --2- 172.21.15.116:0/4025422217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fa56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 -- 172.21.15.116:0/4025422217 >> 172.21.15.116:0/4025422217 conn(0x558931f9fa00 msgr2=0x558931f9fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 -- 172.21.15.116:0/4025422217 shutdown_connections 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.087+0000 7f0ec3822740 1 -- 172.21.15.116:0/4025422217 wait complete. 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 -- start start 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 Processor -- start 2024-03-19T02:46:15.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558931fb1380 con 0x558931fa5bf0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558931ec4310 con 0x558931fa52d0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558931fb1500 con 0x558931fae7d0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ec3822740 10 client.0 _ll_register_callbacks cb 0x558931eb82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37366/0 (socket says 172.21.15.116:37366) 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38440/0 (socket says 172.21.15.116:38440) 2024-03-19T02:46:15.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/1961189724 learned_addr learned my addr 172.21.15.116:0/1961189724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 msgr2=0x558931fac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/1961189724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 msgr2=0x558931fab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/1961189724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558931c1c3f0 con 0x558931fa5bf0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00ee4e0 tx=0x7f0eb0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445423d14ea79a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.088+0000 7f0ea1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00c1170 con 0x558931fa5bf0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f0eb00bd5a0 con 0x558931fa5bf0 2024-03-19T02:46:15.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb00c5ae0 con 0x558931fa5bf0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0eb00bd7a0 con 0x558931fa5bf0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 10 client.20793 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558931ce67d0 con 0x558931fa5bf0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb00c5ae0 con 0x558931fa5bf0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f0eb00b9090 con 0x558931fa5bf0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 10 client.20793 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 10 client.20793 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 20 client.20793 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x558931fbe7d0 0x558931fc0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 4 client.20793 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558931f663e0 con 0x558931fbe7d0 2024-03-19T02:46:15.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.089+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x558931fbe7d0 0x558931fc0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.090+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x558931fbe7d0 0x558931fc0bc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dc2eee4ba9853850 server_cookie=975e72af15ff1a65 in_seq=0 out_seq=0 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.090+0000 7f0ea1ffb640 10 client.20793 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1961189724 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558931f663e0 con 0x558931fbe7d0 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ea1ffb640 10 client.20793 handle_command_reply: tid=0 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 client.20793 shutdown 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 msgr2=0x558931fabdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00ee4e0 tx=0x7f0eb0009260 comp rx=0 tx=0).stop 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 shutdown_connections 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x558931fbe7d0 0x558931fc0bc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fabdf0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ea1ffb640 0 client.20793 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1961189724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 >> 172.21.15.116:0/1961189724 conn(0x558931f9fa00 msgr2=0x558931fa1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 shutdown_connections 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 1 -- 172.21.15.116:0/1961189724 wait complete. 2024-03-19T02:46:15.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.091+0000 7f0ec3822740 20 client.20793 trim_cache size 0 max 16384 2024-03-19T02:46:15.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 Processor -- start 2024-03-19T02:46:15.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 -- start start 2024-03-19T02:46:15.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fadfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fb9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fc62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b0ef0 con 0x558931fa5bf0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b2a30 con 0x558931fa52d0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ec3822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b15c0 con 0x558931fae7d0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fb9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0eba59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fadfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fc62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0eb9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fb9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37380/0 (socket says 172.21.15.116:37380) 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0eb9d9b640 1 -- 172.21.15.116:0/2097983603 learned_addr learned my addr 172.21.15.116:0/2097983603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 -- 172.21.15.116:0/2097983603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 msgr2=0x558931fadfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fadfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 -- 172.21.15.116:0/2097983603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 msgr2=0x558931fb9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fb9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.105+0000 7f0ebad9d640 1 -- 172.21.15.116:0/2097983603 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558931f75880 con 0x558931fae7d0 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.106+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fc62b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x558931fa0b90 tx=0x7f0eb4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35419136f243d8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.106+0000 7f0ea3fff640 1 -- 172.21.15.116:0/2097983603 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb400fd20 con 0x558931fae7d0 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.106+0000 7f0ea3fff640 1 -- 172.21.15.116:0/2097983603 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb40120d0 con 0x558931fae7d0 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.106+0000 7f0ea3fff640 1 -- 172.21.15.116:0/2097983603 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4005700 con 0x558931fae7d0 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.106+0000 7f0ec3822740 1 -- 172.21.15.116:0/2097983603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 msgr2=0x558931fc62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fc62b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x558931fa0b90 tx=0x7f0eb4009500 comp rx=0 tx=0).stop 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- 172.21.15.116:0/2097983603 shutdown_connections 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fae7d0 0x558931fc62b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fa5bf0 0x558931fb9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- 172.21.15.116:0/2097983603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fadfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- 172.21.15.116:0/2097983603 >> 172.21.15.116:0/2097983603 conn(0x558931f9fa00 msgr2=0x558931f9eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- 172.21.15.116:0/2097983603 shutdown_connections 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- 172.21.15.116:0/2097983603 wait complete. 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- start start 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 Processor -- start 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fb2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fb2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b2360 con 0x558931fae7d0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1b1c90 con 0x558931fa52d0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1aeef0 con 0x558931fa5bf0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ec3822740 10 client.0 _ll_register_callbacks cb 0x7f0eb4009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37388/0 (socket says 172.21.15.116:37388) 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0ebad9d640 1 -- 172.21.15.116:0/1704030505 learned_addr learned my addr 172.21.15.116:0/1704030505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fb2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.107+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fb2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 -- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 msgr2=0x558931fb2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fb2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 -- 172.21.15.116:0/1704030505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 msgr2=0x558931fb2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fb2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 -- 172.21.15.116:0/1704030505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558932023990 con 0x558931fae7d0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fb2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0eb9d9b640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fb2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ebad9d640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400a010 tx=0x7f0eb400ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3dc5597dd809daf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ea1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4004c50 con 0x558931fae7d0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f0eb4004df0 con 0x558931fae7d0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb40120d0 con 0x558931fae7d0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0eb4005480 con 0x558931fae7d0 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ec3822740 10 client.20799 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558932024c50 con 0x558931fae7d0 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.108+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0eb40120d0 con 0x558931fae7d0 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f0eb4010cc0 con 0x558931fae7d0 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 10 client.20799 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 10 client.20799 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 20 client.20799 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5589320252f0 0x558932027710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 4 client.20799 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558931f159f0 con 0x5589320252f0 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5589320252f0 0x558932027710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0eba59c640 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5589320252f0 0x558932027710 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=407e18b52b190c12 server_cookie=2418b2c3929e5f56 in_seq=0 out_seq=0 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.109+0000 7f0ea1ffb640 10 client.20799 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ea1ffb640 1 -- 172.21.15.116:0/1704030505 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558931f159f0 con 0x5589320252f0 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ea1ffb640 10 client.20799 handle_command_reply: tid=0 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 client.20799 shutdown 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 msgr2=0x558931fb2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400a010 tx=0x7f0eb400ff50 comp rx=0 tx=0).stop 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 shutdown_connections 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5589320252f0 0x558932027710 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558931fae7d0 0x558931fb2a80 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558931fa5bf0 0x558931fb2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 --2- 172.21.15.116:0/1704030505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558931fa52d0 0x558931fb2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ea1ffb640 0 client.20799 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 >> 172.21.15.116:0/1704030505 conn(0x558931f9fa00 msgr2=0x558931fab870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 shutdown_connections 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 1 -- 172.21.15.116:0/1704030505 wait complete. 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec3822740 20 client.20799 trim_cache size 0 max 16384 2024-03-19T02:46:15.120 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 msgr2=0x7f0ea40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 0x7f0ea40a4eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c10 tx=0x7f0eac005f90 comp rx=0 tx=0).stop 2024-03-19T02:46:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 msgr2=0x7f0ebc074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc074060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009230 tx=0x7f0eb00045e0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 shutdown_connections 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0ea40a29f0 0x7f0ea40a4eb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ebc1077d0 0x7f0ebc0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ebc102180 0x7f0ebc074060 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 --2- 172.21.15.116:0/689130951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ebc1017c0 0x7f0ebc073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.110+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 >> 172.21.15.116:0/689130951 conn(0x7f0ebc0fd2d0 msgr2=0x7f0ebc0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.111+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 shutdown_connections 2024-03-19T02:46:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.111+0000 7f0ec0b5b640 1 -- 172.21.15.116:0/689130951 wait complete. 2024-03-19T02:46:15.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:46:15.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:14 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4144630971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:14 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4144630971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 -- 172.21.15.116:0/26897473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570102a40 msgr2=0x7f6570102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 --2- 172.21.15.116:0/26897473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570102a40 0x7f6570102e20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6560009900 tx=0x7f656002bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 -- 172.21.15.116:0/26897473 shutdown_connections 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 --2- 172.21.15.116:0/26897473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f657010e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 --2- 172.21.15.116:0/26897473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570103360 0x7f657010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 --2- 172.21.15.116:0/26897473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570102a40 0x7f6570102e20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 -- 172.21.15.116:0/26897473 >> 172.21.15.116:0/26897473 conn(0x7f65700778b0 msgr2=0x7f6570077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 -- 172.21.15.116:0/26897473 shutdown_connections 2024-03-19T02:46:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.264+0000 7f6575f77640 1 -- 172.21.15.116:0/26897473 wait complete. 2024-03-19T02:46:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.265+0000 7f6575f77640 1 Processor -- start 2024-03-19T02:46:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 -- start start 2024-03-19T02:46:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 0x7f6570100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570103360 0x7f6570100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6570111ac0 con 0x7f657010be10 2024-03-19T02:46:15.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6570111940 con 0x7f6570102a40 2024-03-19T02:46:15.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.266+0000 7f6575f77640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6570111c40 con 0x7f6570103360 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37414/0 (socket says 172.21.15.116:37414) 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 -- 172.21.15.116:0/517944200 learned_addr learned my addr 172.21.15.116:0/517944200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656f7fe640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 0x7f6570100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656effd640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570103360 0x7f6570100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 -- 172.21.15.116:0/517944200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570103360 msgr2=0x7f6570100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570103360 0x7f6570100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 -- 172.21.15.116:0/517944200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 msgr2=0x7f6570100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 0x7f6570100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656ffff640 1 -- 172.21.15.116:0/517944200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65701a6480 con 0x7f657010be10 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.267+0000 7f656f7fe640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 0x7f6570100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.268+0000 7f656ffff640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6568009f20 tx=0x7f656800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.268+0000 7f656cff9640 1 -- 172.21.15.116:0/517944200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568003b70 con 0x7f657010be10 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.268+0000 7f656cff9640 1 -- 172.21.15.116:0/517944200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6568003d10 con 0x7f657010be10 2024-03-19T02:46:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.268+0000 7f656cff9640 1 -- 172.21.15.116:0/517944200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568010a30 con 0x7f657010be10 2024-03-19T02:46:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.268+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65701a6770 con 0x7f657010be10 2024-03-19T02:46:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.269+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65701a6c70 con 0x7f657010be10 2024-03-19T02:46:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f656cff9640 1 -- 172.21.15.116:0/517944200 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6568004000 con 0x7f657010be10 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 Processor -- start 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 -- start start 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98706b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9879290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fef97d34b0 con 0x55fef98797d0 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fef977bd70 con 0x55fef9870bf0 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f65787fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fef97cf550 con 0x55fef98702d0 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.272+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656cff9640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 0x7f65580a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37426/0 (socket says 172.21.15.116:37426) 2024-03-19T02:46:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 -- 172.21.15.116:0/355264518 learned_addr learned my addr 172.21.15.116:0/355264518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656f7fe640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9879290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656ffff640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 0x7f65580a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 -- 172.21.15.116:0/355264518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 msgr2=0x55fef98706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 -- 172.21.15.116:0/355264518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 msgr2=0x55fef9879290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9879290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 -- 172.21.15.116:0/355264518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fef979e210 con 0x55fef98797d0 2024-03-19T02:46:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656f7fe640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9879290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656cff9640 1 -- 172.21.15.116:0/517944200 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f65680cf610 con 0x7f657010be10 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656effd640 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6564011490 tx=0x7f65640097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f903349a9f8763 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f656ffff640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 0x7f65580a4f80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f656809b820 tx=0x7f6568097490 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.273+0000 7f6555ffb640 1 -- 172.21.15.116:0/355264518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564004030 con 0x55fef98797d0 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.274+0000 7f6555ffb640 1 -- 172.21.15.116:0/355264518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65640035f0 con 0x55fef98797d0 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.274+0000 7f6555ffb640 1 -- 172.21.15.116:0/355264518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656401e710 con 0x55fef98797d0 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.274+0000 7f65787fd740 1 -- 172.21.15.116:0/355264518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 msgr2=0x55fef987bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.274+0000 7f65787fd740 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f6564011490 tx=0x7f65640097d0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 -- 172.21.15.116:0/355264518 shutdown_connections 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef987bbc0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9879290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 --2- 172.21.15.116:0/355264518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 -- 172.21.15.116:0/355264518 >> 172.21.15.116:0/355264518 conn(0x55fef986aa00 msgr2=0x55fef986ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 -- 172.21.15.116:0/355264518 shutdown_connections 2024-03-19T02:46:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 -- 172.21.15.116:0/355264518 wait complete. 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 -- start start 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.275+0000 7f65787fd740 1 Processor -- start 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9876de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98797d0 0x55fef9877320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fef987c5c0 con 0x55fef98702d0 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fef978f310 con 0x55fef9870bf0 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fef987c740 con 0x55fef98797d0 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f65787fd740 10 client.0 _ll_register_callbacks cb 0x55fef97832d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37432/0 (socket says 172.21.15.116:37432) 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 -- 172.21.15.116:0/2674282380 learned_addr learned my addr 172.21.15.116:0/2674282380 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656f7fe640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9876de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98797d0 0x55fef9877320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 -- 172.21.15.116:0/2674282380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98797d0 msgr2=0x55fef9877320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98797d0 0x55fef9877320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 -- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 msgr2=0x55fef9876de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9876de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 -- 172.21.15.116:0/2674282380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fef94e73f0 con 0x55fef98702d0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f656effd640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f656400a010 tx=0x7f6564005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c888d357fdd5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f654effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.276+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564004390 con 0x55fef98702d0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f6564005040 con 0x55fef98702d0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65640035f0 con 0x55fef98702d0 2024-03-19T02:46:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6564005240 con 0x55fef98702d0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 10 client.20811 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fef95b17d0 con 0x55fef98702d0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6564033070 con 0x55fef98702d0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f656402d4b0 con 0x55fef98702d0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 10 client.20811 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 10 client.20811 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 20 client.20811 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef9889a70 0x55fef988be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 4 client.20811 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.277+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fef98313e0 con 0x55fef9889a70 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.278+0000 7f656f7fe640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef9889a70 0x55fef988be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.278+0000 7f654effd640 10 client.20811 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.278+0000 7f656f7fe640 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef9889a70 0x55fef988be60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c5ad0deab23e2cb server_cookie=1de649e87c63b707 in_seq=0 out_seq=0 2024-03-19T02:46:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f654effd640 1 -- 172.21.15.116:0/2674282380 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fef98313e0 con 0x55fef9889a70 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f654effd640 10 client.20811 handle_command_reply: tid=0 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 client.20811 shutdown 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 msgr2=0x55fef98768a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f656400a010 tx=0x7f6564005f90 comp rx=0 tx=0).stop 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 shutdown_connections 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef9889a70 0x55fef988be60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98797d0 0x55fef9877320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef9870bf0 0x55fef9876de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f654effd640 0 client.20811 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 --2- 172.21.15.116:0/2674282380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98702d0 0x55fef98768a0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.279+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 >> 172.21.15.116:0/2674282380 conn(0x55fef986aa00 msgr2=0x55fef986c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.280+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 shutdown_connections 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.280+0000 7f65787fd740 1 -- 172.21.15.116:0/2674282380 wait complete. 2024-03-19T02:46:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.280+0000 7f65787fd740 20 client.20811 trim_cache size 0 max 16384 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.299+0000 7f65787fd740 1 Processor -- start 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.299+0000 7f65787fd740 1 -- start start 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.299+0000 7f65787fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f65787fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 0x55fef9885360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f65787fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98797d0 0x55fef9891550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f65787fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b0e30 con 0x55fef9870bf0 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f65787fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b2970 con 0x55fef98797d0 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f65787fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b1500 con 0x55fef98702d0 2024-03-19T02:46:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43780/0 (socket says 172.21.15.116:43780) 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 -- 172.21.15.116:0/147621762 learned_addr learned my addr 172.21.15.116:0/147621762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98797d0 0x55fef9891550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656f7fe640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 0x55fef9885360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 -- 172.21.15.116:0/147621762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98797d0 msgr2=0x55fef9891550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98797d0 0x55fef9891550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 -- 172.21.15.116:0/147621762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 msgr2=0x55fef9885360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 0x55fef9885360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 -- 172.21.15.116:0/147621762 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fef9874e10 con 0x55fef98702d0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656f7fe640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 0x55fef9885360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.300+0000 7f656effd640 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f65640059f0 tx=0x7f6564009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fb219d0acb386b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.301+0000 7f65557fa640 1 -- 172.21.15.116:0/147621762 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564005a70 con 0x55fef98702d0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.301+0000 7f65557fa640 1 -- 172.21.15.116:0/147621762 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6564033070 con 0x55fef98702d0 2024-03-19T02:46:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.301+0000 7f65557fa640 1 -- 172.21.15.116:0/147621762 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564005620 con 0x55fef98702d0 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- 172.21.15.116:0/147621762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 msgr2=0x55fef98761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f65640059f0 tx=0x7f6564009620 comp rx=0 tx=0).stop 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- 172.21.15.116:0/147621762 shutdown_connections 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98797d0 0x55fef9891550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef9870bf0 0x55fef9885360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 --2- 172.21.15.116:0/147621762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef98702d0 0x55fef98761a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- 172.21.15.116:0/147621762 >> 172.21.15.116:0/147621762 conn(0x55fef986aa00 msgr2=0x55fef9876860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- 172.21.15.116:0/147621762 shutdown_connections 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- 172.21.15.116:0/147621762 wait complete. 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 -- start start 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.302+0000 7f65787fd740 1 Processor -- start 2024-03-19T02:46:15.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 0x55fef9881d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 0x55fef98732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef98739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b22a0 con 0x55fef98797d0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f65701b1bd0 con 0x55fef98702d0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65701aee30 con 0x55fef9870bf0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f65787fd740 10 client.0 _ll_register_callbacks cb 0x7f6564009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 0x55fef9881d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef98739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 0x55fef98732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 0x55fef9881d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38512/0 (socket says 172.21.15.116:38512) 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656effd640 1 -- 172.21.15.116:0/1133437304 learned_addr learned my addr 172.21.15.116:0/1133437304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 -- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 msgr2=0x55fef9881d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 0x55fef9881d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 -- 172.21.15.116:0/1133437304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 msgr2=0x55fef98739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef98739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 -- 172.21.15.116:0/1133437304 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fef98eeb80 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f656f7fe640 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 0x55fef98732b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6560006010 tx=0x7f6560004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00b1d4d9338e943 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f654effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.303+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656004e060 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f656003a040 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6564033070 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6560055050 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 10 client.30520 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fef98efc80 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6564033070 con 0x55fef9870bf0 2024-03-19T02:46:15.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f65600042b0 con 0x55fef9870bf0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 10 client.30520 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 10 client.30520 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 20 client.30520 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.304+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef98f03e0 0x55fef98f2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f65787fd740 4 client.30520 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fef97e09f0 con 0x55fef98f03e0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f656effd640 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef98f03e0 0x55fef98f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f656effd640 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef98f03e0 0x55fef98f2800 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c3359018d642f92 server_cookie=de03d186a1561a3f in_seq=0 out_seq=0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f654effd640 10 client.30520 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f654effd640 1 -- 172.21.15.116:0/1133437304 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fef97e09f0 con 0x55fef98f03e0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f654effd640 10 client.30520 handle_command_reply: tid=0 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.305+0000 7f65787fd740 1 client.30520 shutdown 2024-03-19T02:46:15.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 msgr2=0x55fef98732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 0x55fef98732b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6560006010 tx=0x7f6560004d50 comp rx=0 tx=0).stop 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 shutdown_connections 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55fef98f03e0 0x55fef98f2800 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef98797d0 0x55fef98739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fef9870bf0 0x55fef98732b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 --2- 172.21.15.116:0/1133437304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fef98702d0 0x55fef9881d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 >> 172.21.15.116:0/1133437304 conn(0x55fef986aa00 msgr2=0x55fef98770d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f654effd640 0 client.30520 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 shutdown_connections 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 1 -- 172.21.15.116:0/1133437304 wait complete. 2024-03-19T02:46:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.306+0000 7f65787fd740 20 client.30520 trim_cache size 0 max 16384 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 msgr2=0x7f65580a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 0x7f65580a4f80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f656809b820 tx=0x7f6568097490 comp rx=0 tx=0).stop 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 msgr2=0x7f6570101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6568009f20 tx=0x7f656800c760 comp rx=0 tx=0).stop 2024-03-19T02:46:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 shutdown_connections 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f65580a2ac0 0x7f65580a4f80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f657010be10 0x7f6570101110 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6570103360 0x7f6570100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 --2- 172.21.15.116:0/517944200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6570102a40 0x7f6570100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 >> 172.21.15.116:0/517944200 conn(0x7f65700778b0 msgr2=0x7f65700fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.307+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 shutdown_connections 2024-03-19T02:46:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.308+0000 7f6575f77640 1 -- 172.21.15.116:0/517944200 wait complete. 2024-03-19T02:46:15.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:46:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38344 5.97 MB/sec execute 217 sec latency 159.859 ms 2024-03-19T02:46:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 -- 172.21.15.116:0/1483902731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 msgr2=0x7f8ce0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 --2- 172.21.15.116:0/1483902731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0100b20 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009930 tx=0x7f8cc802bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 -- 172.21.15.116:0/1483902731 shutdown_connections 2024-03-19T02:46:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 --2- 172.21.15.116:0/1483902731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 0x7f8ce010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 --2- 172.21.15.116:0/1483902731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0100b20 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 --2- 172.21.15.116:0/1483902731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 0x7f8ce0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 -- 172.21.15.116:0/1483902731 >> 172.21.15.116:0/1483902731 conn(0x7f8ce00778d0 msgr2=0x7f8ce0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 -- 172.21.15.116:0/1483902731 shutdown_connections 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.458+0000 7f8ce555b640 1 -- 172.21.15.116:0/1483902731 wait complete. 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 Processor -- start 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 -- start start 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 0x7f8ce0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 0x7f8ce019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce0111a70 con 0x7f8ce01006a0 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01118f0 con 0x7f8ce0101060 2024-03-19T02:46:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8ce555b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce0111bf0 con 0x7f8ce01066b0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37482/0 (socket says 172.21.15.116:37482) 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 -- 172.21.15.116:0/827464341 learned_addr learned my addr 172.21.15.116:0/827464341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdf7fe640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 0x7f8ce019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 0x7f8ce0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 -- 172.21.15.116:0/827464341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 msgr2=0x7f8ce019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 0x7f8ce019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 -- 172.21.15.116:0/827464341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 msgr2=0x7f8ce0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 0x7f8ce0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 -- 172.21.15.116:0/827464341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce019d030 con 0x7f8ce01006a0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.459+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000e9f0 tx=0x7f8cd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.460+0000 7f8cdcff9640 1 -- 172.21.15.116:0/827464341 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd000cd80 con 0x7f8ce01006a0 2024-03-19T02:46:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.460+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce019d320 con 0x7f8ce01006a0 2024-03-19T02:46:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.460+0000 7f8cdcff9640 1 -- 172.21.15.116:0/827464341 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cd00054d0 con 0x7f8ce01006a0 2024-03-19T02:46:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.460+0000 7f8cdcff9640 1 -- 172.21.15.116:0/827464341 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0010490 con 0x7f8ce01006a0 2024-03-19T02:46:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.460+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce01a4bb0 con 0x7f8ce01006a0 2024-03-19T02:46:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.462+0000 7f8cdcff9640 1 -- 172.21.15.116:0/827464341 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8cd0010750 con 0x7f8ce01006a0 2024-03-19T02:46:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8cdcff9640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 0x7f8ccc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 0x7f8ccc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8ce8224740 1 Processor -- start 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8cdcff9640 1 -- 172.21.15.116:0/827464341 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f8cd00cf5b0 con 0x7f8ce01006a0 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8ce8224740 1 -- start start 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.463+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 0x561301183120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301194f00 0x5613011972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8ce8224740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5613010e64b0 con 0x561301194f00 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8ce8224740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56130108ed70 con 0x561301182d40 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8ce8224740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613010e2550 con 0x5613011836f0 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 0x7f8ccc0a4e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005ab0 tx=0x7f8cc8005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43834/0 (socket says 172.21.15.116:43834) 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4151883095 learned_addr learned my addr 172.21.15.116:0/4151883095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cdf7fe640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301194f00 0x5613011972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 0x561301183120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4151883095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 msgr2=0x561301183120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 0x561301183120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4151883095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301194f00 msgr2=0x5613011972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301194f00 0x5613011972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4151883095 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613010b1210 con 0x5613011836f0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 0x561301183120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.464+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009930 tx=0x7f8cc803e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5914bd7d5cf760c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.465+0000 7f8cd57fa640 1 -- 172.21.15.116:0/4151883095 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc804f070 con 0x5613011836f0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.465+0000 7f8cd57fa640 1 -- 172.21.15.116:0/4151883095 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc8045070 con 0x5613011836f0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.465+0000 7f8cd57fa640 1 -- 172.21.15.116:0/4151883095 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc80049d0 con 0x5613011836f0 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.465+0000 7f8ce8224740 1 -- 172.21.15.116:0/4151883095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 msgr2=0x561301183b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.465+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009930 tx=0x7f8cc803e000 comp rx=0 tx=0).stop 2024-03-19T02:46:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- 172.21.15.116:0/4151883095 shutdown_connections 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301194f00 0x5613011972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x561301183b70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4151883095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301182d40 0x561301183120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- 172.21.15.116:0/4151883095 >> 172.21.15.116:0/4151883095 conn(0x56130117da00 msgr2=0x56130117dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- 172.21.15.116:0/4151883095 shutdown_connections 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- 172.21.15.116:0/4151883095 wait complete. 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- start start 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 Processor -- start 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301185360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x561301185de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561301197d50 con 0x561301182d40 2024-03-19T02:46:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613010a2310 con 0x561301194f00 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561301197ed0 con 0x5613011836f0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.466+0000 7f8ce8224740 10 client.0 _ll_register_callbacks cb 0x5613010962d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43842/0 (socket says 172.21.15.116:43842) 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 -- 172.21.15.116:0/2962468557 learned_addr learned my addr 172.21.15.116:0/2962468557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301185360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cdf7fe640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x561301185de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 -- 172.21.15.116:0/2962468557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 msgr2=0x561301185de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x561301185de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 -- 172.21.15.116:0/2962468557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 msgr2=0x561301185360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301185360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 -- 172.21.15.116:0/2962468557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561300dfa3f0 con 0x5613011836f0 2024-03-19T02:46:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036040 tx=0x7f8cc80052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1c4555cda72076b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.467+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc805b020 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f8cc8004930 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cc8045070 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8cc8066050 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 10 client.30532 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561300ec47d0 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc8045070 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8cc804a070 con 0x5613011836f0 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 10 client.30532 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 10 client.30532 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 20 client.30532 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613011a5210 0x5613011a7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 4 client.30532 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.468+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613011443e0 con 0x5613011a5210 2024-03-19T02:46:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.469+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613011a5210 0x5613011a7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.469+0000 7f8cc67fc640 10 client.30532 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:15.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.469+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613011a5210 0x5613011a7600 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d2146645aa3e8891 server_cookie=f20b4338baab63e4 in_seq=0 out_seq=0 2024-03-19T02:46:15.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8cc67fc640 1 -- 172.21.15.116:0/2962468557 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613011443e0 con 0x5613011a5210 2024-03-19T02:46:15.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8cc67fc640 10 client.30532 handle_command_reply: tid=0 2024-03-19T02:46:15.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 client.30532 shutdown 2024-03-19T02:46:15.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 msgr2=0x5613011858a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036040 tx=0x7f8cc80052f0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 shutdown_connections 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613011a5210 0x5613011a7600 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x561301185de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8cc67fc640 0 client.30532 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011858a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 --2- 172.21.15.116:0/2962468557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301185360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.470+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 >> 172.21.15.116:0/2962468557 conn(0x56130117da00 msgr2=0x56130117f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.471+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 shutdown_connections 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.471+0000 7f8ce8224740 1 -- 172.21.15.116:0/2962468557 wait complete. 2024-03-19T02:46:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.471+0000 7f8ce8224740 20 client.30532 trim_cache size 0 max 16384 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 Processor -- start 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 -- start start 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 0x561301197160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 0x5613011962f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01aca30 con 0x561301182d40 2024-03-19T02:46:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01ae570 con 0x5613011836f0 2024-03-19T02:46:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8ce8224740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01ad100 con 0x561301194f00 2024-03-19T02:46:15.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.486+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cdf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 0x5613011962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 0x561301197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37508/0 (socket says 172.21.15.116:37508) 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cdf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 0x5613011962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43852/0 (socket says 172.21.15.116:43852) 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cdeffd640 1 -- 172.21.15.116:0/4136409231 learned_addr learned my addr 172.21.15.116:0/4136409231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4136409231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 msgr2=0x5613011962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 0x5613011962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4136409231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 msgr2=0x561301187a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 -- 172.21.15.116:0/4136409231 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613011ad320 con 0x5613011836f0 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 0x561301197160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8000c00 tx=0x7f8cc802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5549c57ee76471cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd4ff9640 1 -- 172.21.15.116:0/4136409231 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc805b040 con 0x5613011836f0 2024-03-19T02:46:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd4ff9640 1 -- 172.21.15.116:0/4136409231 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc8045070 con 0x5613011836f0 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.487+0000 7f8cd4ff9640 1 -- 172.21.15.116:0/4136409231 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc804f040 con 0x5613011836f0 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 -- 172.21.15.116:0/4136409231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 msgr2=0x561301197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 0x561301197160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8000c00 tx=0x7f8cc802bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 -- 172.21.15.116:0/4136409231 shutdown_connections 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561301194f00 0x5613011962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613011836f0 0x561301197160 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 --2- 172.21.15.116:0/4136409231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x561301187a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 -- 172.21.15.116:0/4136409231 >> 172.21.15.116:0/4136409231 conn(0x56130117da00 msgr2=0x56130117cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 -- 172.21.15.116:0/4136409231 shutdown_connections 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.488+0000 7f8ce8224740 1 -- 172.21.15.116:0/4136409231 wait complete. 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 -- start start 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 Processor -- start 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x56130120a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01adea0 con 0x561301182d40 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01ad7d0 con 0x561301194f00 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01aaa30 con 0x5613011836f0 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8ce8224740 10 client.0 _ll_register_callbacks cb 0x7f8cd8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cdeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37522/0 (socket says 172.21.15.116:37522) 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43860/0 (socket says 172.21.15.116:43860) 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cdeffd640 1 -- 172.21.15.116:0/3646090812 learned_addr learned my addr 172.21.15.116:0/3646090812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.489+0000 7f8cdf7fe640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x56130120a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 -- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 msgr2=0x5613011a6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 -- 172.21.15.116:0/3646090812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 msgr2=0x56130120a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x56130120a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 -- 172.21.15.116:0/3646090812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130120a760 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdf7fe640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x56130120a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cdeffd640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f8cd009c450 tx=0x7f8cd0010630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f650dd5114a43b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd009c750 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f8cd00046c0 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cc8045070 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8cd00a08a0 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 10 client.20832 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56130120baa0 con 0x561301182d40 2024-03-19T02:46:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc8045070 con 0x561301182d40 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8cd0098dc0 con 0x561301182d40 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 10 client.20832 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 10 client.20832 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 20 client.20832 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56130120c250 0x56130120e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 4 client.20832 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.490+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613010f39f0 con 0x56130120c250 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56130120c250 0x56130120e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cc67fc640 10 client.20832 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cd7fff640 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56130120c250 0x56130120e670 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dba24ac7893125f2 server_cookie=f54b4356eb3ac9e5 in_seq=0 out_seq=0 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cc67fc640 1 -- 172.21.15.116:0/3646090812 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613010f39f0 con 0x56130120c250 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cc67fc640 10 client.20832 handle_command_reply: tid=0 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 client.20832 shutdown 2024-03-19T02:46:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 msgr2=0x5613011a66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f8cd009c450 tx=0x7f8cd0010630 comp rx=0 tx=0).stop 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 shutdown_connections 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56130120c250 0x56130120e670 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561301194f00 0x56130120a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613011836f0 0x5613011a6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8cc67fc640 0 client.20832 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 --2- 172.21.15.116:0/3646090812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561301182d40 0x5613011a66d0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 >> 172.21.15.116:0/3646090812 conn(0x56130117da00 msgr2=0x56130117ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 shutdown_connections 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 1 -- 172.21.15.116:0/3646090812 wait complete. 2024-03-19T02:46:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.491+0000 7f8ce8224740 20 client.20832 trim_cache size 0 max 16384 2024-03-19T02:46:15.500 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 msgr2=0x7f8ccc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 0x7f8ccc0a4e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005ab0 tx=0x7f8cc8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 msgr2=0x7f8ce0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000e9f0 tx=0x7f8cd000edf0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 shutdown_connections 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8ccc0a2970 0x7f8ccc0a4e30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce01066b0 0x7f8ce019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce0101060 0x7f8ce0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 --2- 172.21.15.116:0/827464341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ce01006a0 0x7f8ce0198040 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 >> 172.21.15.116:0/827464341 conn(0x7f8ce00778d0 msgr2=0x7f8ce0102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 shutdown_connections 2024-03-19T02:46:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.492+0000 7f8ce555b640 1 -- 172.21.15.116:0/827464341 wait complete. 2024-03-19T02:46:15.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:46:15.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 -- 172.21.15.116:0/3790369004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12181077a0 msgr2=0x7f1218107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 --2- 172.21.15.116:0/3790369004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12181077a0 0x7f1218107b80 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1208009930 tx=0x7f120802bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 -- 172.21.15.116:0/3790369004 shutdown_connections 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 --2- 172.21.15.116:0/3790369004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1218102150 0x7f121810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 --2- 172.21.15.116:0/3790369004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 --2- 172.21.15.116:0/3790369004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12181077a0 0x7f1218107b80 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.671+0000 7f121d5b3640 1 -- 172.21.15.116:0/3790369004 >> 172.21.15.116:0/3790369004 conn(0x7f12180fd2c0 msgr2=0x7f12180ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.672+0000 7f121d5b3640 1 -- 172.21.15.116:0/3790369004 shutdown_connections 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.672+0000 7f121d5b3640 1 -- 172.21.15.116:0/3790369004 wait complete. 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.672+0000 7f121d5b3640 1 Processor -- start 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.672+0000 7f121d5b3640 1 -- start start 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12181077a0 0x7f121819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1218111a20 con 0x7f1218102150 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181118a0 con 0x7f1218101790 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f121d5b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1218111ba0 con 0x7f12181077a0 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38590/0 (socket says 172.21.15.116:38590) 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 -- 172.21.15.116:0/989226772 learned_addr learned my addr 172.21.15.116:0/989226772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37536/0 (socket says 172.21.15.116:37536) 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f12177fe640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12181077a0 0x7f121819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 -- 172.21.15.116:0/989226772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12181077a0 msgr2=0x7f121819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12181077a0 0x7f121819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 -- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 msgr2=0x7f1218198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f1216ffd640 1 -- 172.21.15.116:0/989226772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121819d000 con 0x7f1218101790 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.673+0000 7f12167fc640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f1216ffd640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1208009230 tx=0x7f120802c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f11fffff640 1 -- 172.21.15.116:0/989226772 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208043020 con 0x7f1218101790 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121819d290 con 0x7f1218101790 2024-03-19T02:46:15.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f11fffff640 1 -- 172.21.15.116:0/989226772 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120802ca70 con 0x7f1218101790 2024-03-19T02:46:15.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f11fffff640 1 -- 172.21.15.116:0/989226772 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208003a50 con 0x7f1218101790 2024-03-19T02:46:15.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.674+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12181a4b80 con 0x7f1218101790 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 Processor -- start 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 -- start start 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f11fffff640 1 -- 172.21.15.116:0/989226772 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1208046020 con 0x7f1218101790 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195ba290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd195144b0 con 0x55dd195ba7d0 2024-03-19T02:46:15.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd194bcd70 con 0x55dd195b1bf0 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1220197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd19510550 con 0x55dd195b12d0 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37548/0 (socket says 172.21.15.116:37548) 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12167fc640 1 -- 172.21.15.116:0/4142574052 learned_addr learned my addr 172.21.15.116:0/4142574052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43902/0 (socket says 172.21.15.116:43902) 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f1216ffd640 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 -- 172.21.15.116:0/4142574052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 msgr2=0x55dd195ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195ba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 -- 172.21.15.116:0/4142574052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 msgr2=0x55dd195bcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195bcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f12177fe640 1 -- 172.21.15.116:0/4142574052 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd194df210 con 0x55dd195b12d0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.677+0000 7f11fffff640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 0x7f12000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f1216ffd640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 0x7f12000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f12177fe640 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1210009f00 tx=0x7f1210011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6767b94277ca4709 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f11fffff640 1 -- 172.21.15.116:0/989226772 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f12080f1110 con 0x7f1218101790 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f11fd7fa640 1 -- 172.21.15.116:0/4142574052 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210004d20 con 0x55dd195b12d0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f11fd7fa640 1 -- 172.21.15.116:0/4142574052 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1210004050 con 0x55dd195b12d0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.678+0000 7f11fd7fa640 1 -- 172.21.15.116:0/4142574052 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210005280 con 0x55dd195b12d0 2024-03-19T02:46:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- 172.21.15.116:0/4142574052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 msgr2=0x55dd195b16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1210009f00 tx=0x7f1210011810 comp rx=0 tx=0).stop 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1216ffd640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 0x7f12000a4e30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f12080ee640 tx=0x7f12080f3040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- 172.21.15.116:0/4142574052 shutdown_connections 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- 172.21.15.116:0/4142574052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b12d0 0x55dd195b16b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- 172.21.15.116:0/4142574052 >> 172.21.15.116:0/4142574052 conn(0x55dd195aba00 msgr2=0x55dd195abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- 172.21.15.116:0/4142574052 shutdown_connections 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- 172.21.15.116:0/4142574052 wait complete. 2024-03-19T02:46:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- start start 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 Processor -- start 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.679+0000 7f1220197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd195bd5c0 con 0x55dd195b12d0 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f1220197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd194d0310 con 0x55dd195b1bf0 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f1220197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd195bd740 con 0x55dd195ba7d0 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f1220197740 10 client.0 _ll_register_callbacks cb 0x55dd194c42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37564/0 (socket says 172.21.15.116:37564) 2024-03-19T02:46:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 -- 172.21.15.116:0/4223210831 learned_addr learned my addr 172.21.15.116:0/4223210831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12167fc640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 -- 172.21.15.116:0/4223210831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 msgr2=0x55dd195b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 -- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 msgr2=0x55dd195b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 -- 172.21.15.116:0/4223210831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd192283f0 con 0x55dd195b12d0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12167fc640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f12177fe640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1210004830 tx=0x7f1210004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70fba0336befc9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f11faff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210005040 con 0x55dd195b12d0 2024-03-19T02:46:15.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1210004050 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.680+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f12100051e0 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1210035050 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 10 client.20844 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55dd192f27d0 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1210004050 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f1210019880 con 0x55dd195b12d0 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 10 client.20844 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 10 client.20844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 20 client.20844 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd195caab0 0x55dd195ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 4 client.20844 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd195723e0 con 0x55dd195caab0 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.681+0000 7f12167fc640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd195caab0 0x55dd195ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.682+0000 7f12167fc640 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd195caab0 0x55dd195ccea0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac03fb57c2d9e3d server_cookie=aa1a58baa707d115 in_seq=0 out_seq=0 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.682+0000 7f11faff5640 10 client.20844 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f11faff5640 1 -- 172.21.15.116:0/4223210831 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd195723e0 con 0x55dd195caab0 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f11faff5640 10 client.20844 handle_command_reply: tid=0 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 client.20844 shutdown 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 msgr2=0x55dd195b7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f1210004830 tx=0x7f1210004890 comp rx=0 tx=0).stop 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 shutdown_connections 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd195caab0 0x55dd195ccea0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b1bf0 0x55dd195b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f11faff5640 0 client.20844 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 --2- 172.21.15.116:0/4223210831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b12d0 0x55dd195b7820 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 >> 172.21.15.116:0/4223210831 conn(0x55dd195aba00 msgr2=0x55dd195ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 shutdown_connections 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.683+0000 7f1220197740 1 -- 172.21.15.116:0/4223210831 wait complete. 2024-03-19T02:46:15.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.684+0000 7f1220197740 20 client.20844 trim_cache size 0 max 16384 2024-03-19T02:46:15.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 Processor -- start 2024-03-19T02:46:15.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 -- start start 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195b9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b0f50 con 0x55dd195b1bf0 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b2a90 con 0x55dd195b12d0 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f1220197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b1620 con 0x55dd195ba7d0 2024-03-19T02:46:15.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37580/0 (socket says 172.21.15.116:37580) 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 -- 172.21.15.116:0/3112805266 learned_addr learned my addr 172.21.15.116:0/3112805266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12177fe640 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195b9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 -- 172.21.15.116:0/3112805266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 msgr2=0x55dd195b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 -- 172.21.15.116:0/3112805266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 msgr2=0x55dd195b9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195b9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12167fc640 1 -- 172.21.15.116:0/3112805266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd195bcbc0 con 0x55dd195b1bf0 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.699+0000 7f12177fe640 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.700+0000 7f12167fc640 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x55dd195cd650 tx=0x7f120c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e118284e6db931 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.700+0000 7f11fcff9640 1 -- 172.21.15.116:0/3112805266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c004310 con 0x55dd195b1bf0 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.700+0000 7f11fcff9640 1 -- 172.21.15.116:0/3112805266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120c0044b0 con 0x55dd195b1bf0 2024-03-19T02:46:15.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.700+0000 7f11fcff9640 1 -- 172.21.15.116:0/3112805266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c0047a0 con 0x55dd195b1bf0 2024-03-19T02:46:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 -- 172.21.15.116:0/3112805266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 msgr2=0x55dd195c6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x55dd195cd650 tx=0x7f120c005f90 comp rx=0 tx=0).stop 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 -- 172.21.15.116:0/3112805266 shutdown_connections 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195ba7d0 0x55dd195b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195b1bf0 0x55dd195c6360 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 --2- 172.21.15.116:0/3112805266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195b9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 -- 172.21.15.116:0/3112805266 >> 172.21.15.116:0/3112805266 conn(0x55dd195aba00 msgr2=0x55dd195ad6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 -- 172.21.15.116:0/3112805266 shutdown_connections 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.701+0000 7f1220197740 1 -- 172.21.15.116:0/3112805266 wait complete. 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 -- start start 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 Processor -- start 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b23c0 con 0x55dd195ba7d0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12181b1cf0 con 0x55dd195b12d0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12181aef50 con 0x55dd195b1bf0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f1220197740 10 client.0 _ll_register_callbacks cb 0x7f120c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43912/0 (socket says 172.21.15.116:43912) 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 -- 172.21.15.116:0/1887771792 learned_addr learned my addr 172.21.15.116:0/1887771792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:15.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 -- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 msgr2=0x55dd195c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 -- 172.21.15.116:0/1887771792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 msgr2=0x55dd195b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.702+0000 7f12167fc640 1 -- 172.21.15.116:0/1887771792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd195b2640 con 0x55dd195b1bf0 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f12167fc640 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f120c023040 tx=0x7f120c005690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3639cf19c1b0e670 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c011230 con 0x55dd195b1bf0 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f120c0044b0 con 0x55dd195b1bf0 2024-03-19T02:46:15.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f120c0113d0 con 0x55dd195b1bf0 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f120c035050 con 0x55dd195b1bf0 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 10 client.30556 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55dd19630940 con 0x55dd195b1bf0 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120c0044b0 con 0x55dd195b1bf0 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f120c020670 con 0x55dd195b1bf0 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 10 client.30556 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 10 client.30556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 20 client.30556 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:15.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd19631090 0x55dd19633480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 4 client.30556 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.703+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd195219f0 con 0x55dd19631090 2024-03-19T02:46:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.705+0000 7f12177fe640 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd19631090 0x55dd19633480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.705+0000 7f12177fe640 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd19631090 0x55dd19633480 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd6039349b19c9d4 server_cookie=4aa9168a1b6c825 in_seq=0 out_seq=0 2024-03-19T02:46:15.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.705+0000 7f11faff5640 10 client.30556 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:15.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f11faff5640 1 -- 172.21.15.116:0/1887771792 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd195219f0 con 0x55dd19631090 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f11faff5640 10 client.30556 handle_command_reply: tid=0 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 client.30556 shutdown 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 msgr2=0x55dd195b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f120c023040 tx=0x7f120c005690 comp rx=0 tx=0).stop 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 shutdown_connections 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dd19631090 0x55dd19633480 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd195ba7d0 0x55dd195b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd195b1bf0 0x55dd195b3e80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f11faff5640 0 client.30556 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 --2- 172.21.15.116:0/1887771792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd195b12d0 0x55dd195c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.709+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 >> 172.21.15.116:0/1887771792 conn(0x55dd195aba00 msgr2=0x55dd195b5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 shutdown_connections 2024-03-19T02:46:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f1220197740 1 -- 172.21.15.116:0/1887771792 wait complete. 2024-03-19T02:46:15.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f1220197740 20 client.30556 trim_cache size 0 max 16384 2024-03-19T02:46:15.715 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 msgr2=0x7f12000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 0x7f12000a4e30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f12080ee640 tx=0x7f12080f3040 comp rx=0 tx=0).stop 2024-03-19T02:46:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 msgr2=0x7f1218198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1208009230 tx=0x7f120802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 shutdown_connections 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f12000a2970 0x7f12000a4e30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12181077a0 0x7f121819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1218102150 0x7f1218198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 --2- 172.21.15.116:0/989226772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1218101790 0x7f1218198010 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 >> 172.21.15.116:0/989226772 conn(0x7f12180fd2c0 msgr2=0x7f12180fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 shutdown_connections 2024-03-19T02:46:15.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:15.710+0000 7f121d5b3640 1 -- 172.21.15.116:0/989226772 wait complete. 2024-03-19T02:46:16.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:15 smithi162 ceph-mon[35761]: pgmap v582: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 185 op/s 2024-03-19T02:46:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:15 smithi154 ceph-mon[35693]: pgmap v582: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 185 op/s 2024-03-19T02:46:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:15 smithi116 ceph-mon[32263]: pgmap v582: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 185 op/s 2024-03-19T02:46:16.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38466 6.00 MB/sec execute 218 sec latency 141.419 ms 2024-03-19T02:46:16.726 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:46:16.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:46:16.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:46:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 -- 172.21.15.116:0/4131909304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 msgr2=0x7f604810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 --2- 172.21.15.116:0/4131909304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f604810e680 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f6040009cd0 tx=0x7f604002c0e0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 -- 172.21.15.116:0/4131909304 shutdown_connections 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 --2- 172.21.15.116:0/4131909304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f604810e680 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 --2- 172.21.15.116:0/4131909304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048068a20 0x7f6048068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 --2- 172.21.15.116:0/4131909304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048107870 0x7f6048107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 -- 172.21.15.116:0/4131909304 >> 172.21.15.116:0/4131909304 conn(0x7f6048077820 msgr2=0x7f6048077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 -- 172.21.15.116:0/4131909304 shutdown_connections 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.856+0000 7f604dc81640 1 -- 172.21.15.116:0/4131909304 wait complete. 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 Processor -- start 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 -- start start 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 0x7f6048197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 0x7f604819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6048111a70 con 0x7f60480693e0 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60481118f0 con 0x7f6048107870 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f604dc81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6048111bf0 con 0x7f6048068a20 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f60477fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 0x7f6048197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37606/0 (socket says 172.21.15.116:37606) 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f6046ffd640 1 -- 172.21.15.116:0/3426300857 learned_addr learned my addr 172.21.15.116:0/3426300857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.857+0000 7f6047fff640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 0x7f604819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 -- 172.21.15.116:0/3426300857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 msgr2=0x7f604819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 0x7f604819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 -- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 msgr2=0x7f6048198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 -- 172.21.15.116:0/3426300857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604819cf40 con 0x7f6048068a20 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f6047fff640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 0x7f604819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f6046ffd640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f60477fe640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 0x7f6048197f50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f603800e9f0 tx=0x7f603800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f6044ff9640 1 -- 172.21.15.116:0/3426300857 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603800ccc0 con 0x7f6048068a20 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604819d230 con 0x7f6048068a20 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f6044ff9640 1 -- 172.21.15.116:0/3426300857 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6038005020 con 0x7f6048068a20 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f6044ff9640 1 -- 172.21.15.116:0/3426300857 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6038004a00 con 0x7f6048068a20 2024-03-19T02:46:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.858+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a4ac0 con 0x7f6048068a20 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f6044ff9640 1 -- 172.21.15.116:0/3426300857 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6038004ba0 con 0x7f6048068a20 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f605086d740 1 Processor -- start 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f605086d740 1 -- start start 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f605086d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f605086d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.861+0000 7f605086d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dcac90 0x55fb44dcd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f605086d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb44d21160 con 0x55fb44dc1790 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f605086d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fb44cc9e10 con 0x55fb44dc20b0 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f605086d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fb44d1d200 con 0x55fb44dcac90 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37622/0 (socket says 172.21.15.116:37622) 2024-03-19T02:46:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 -- 172.21.15.116:0/1916070098 learned_addr learned my addr 172.21.15.116:0/1916070098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f60477fe640 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6046ffd640 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dcac90 0x55fb44dcd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6044ff9640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 0x7f60300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 -- 172.21.15.116:0/1916070098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dcac90 msgr2=0x55fb44dcd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dcac90 0x55fb44dcd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 -- 172.21.15.116:0/1916070098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 msgr2=0x55fb44dca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 -- 172.21.15.116:0/1916070098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb44afd7e0 con 0x55fb44dc1790 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6044ff9640 1 -- 172.21.15.116:0/3426300857 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f60380cf270 con 0x7f6048068a20 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 0x7f60300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.862+0000 7f6047fff640 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6040009ca0 tx=0x7f6040009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e470676f8d713b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f602dffb640 1 -- 172.21.15.116:0/1916070098 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040039040 con 0x55fb44dc1790 2024-03-19T02:46:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f602dffb640 1 -- 172.21.15.116:0/1916070098 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6040003b70 con 0x55fb44dc1790 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f602dffb640 1 -- 172.21.15.116:0/1916070098 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040034660 con 0x55fb44dc1790 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f6047fff640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 0x7f60300a4e30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f604004b000 tx=0x7f6040044000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 -- 172.21.15.116:0/1916070098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 msgr2=0x55fb44dc1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6040009ca0 tx=0x7f6040009800 comp rx=0 tx=0).stop 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 -- 172.21.15.116:0/1916070098 shutdown_connections 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dcac90 0x55fb44dcd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 --2- 172.21.15.116:0/1916070098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dc1b70 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 -- 172.21.15.116:0/1916070098 >> 172.21.15.116:0/1916070098 conn(0x55fb44dbbec0 msgr2=0x55fb44dbc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.863+0000 7f605086d740 1 -- 172.21.15.116:0/1916070098 shutdown_connections 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 -- 172.21.15.116:0/1916070098 wait complete. 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 -- start start 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 Processor -- start 2024-03-19T02:46:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 0x55fb44dc7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44dc86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb44dcda80 con 0x55fb44dcac90 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fb44cdd1a0 con 0x55fb44dc20b0 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fb44dcdc00 con 0x55fb44dc1790 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f605086d740 10 client.0 _ll_register_callbacks cb 0x55fb44a34c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f60477fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 0x55fb44dc7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38648/0 (socket says 172.21.15.116:38648) 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f6046ffd640 1 -- 172.21.15.116:0/2879309623 learned_addr learned my addr 172.21.15.116:0/2879309623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.864+0000 7f6047fff640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44dc86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 -- 172.21.15.116:0/2879309623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 msgr2=0x55fb44dc7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 0x55fb44dc7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 -- 172.21.15.116:0/2879309623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 msgr2=0x55fb44dc86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44dc86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 -- 172.21.15.116:0/2879309623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb44b5df70 con 0x55fb44dc20b0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f60477fe640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 0x55fb44dc7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6046ffd640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f603c011980 tx=0x7f603c011e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a828594741da8c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6026ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c010d20 con 0x55fb44dc20b0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6040003b70 con 0x55fb44dc20b0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f603c004050 con 0x55fb44dc20b0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.865+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f603c004250 con 0x55fb44dc20b0 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.866+0000 7f605086d740 10 client.30590 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:16.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.866+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fb449f5170 con 0x55fb44dc20b0 2024-03-19T02:46:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6040003b70 con 0x55fb44dc20b0 2024-03-19T02:46:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f603c01b420 con 0x55fb44dc20b0 2024-03-19T02:46:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 10 client.30590 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:16.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 10 client.30590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 20 client.30590 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44ddd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 4 client.30590 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.867+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb44d2e6a0 con 0x55fb44ddaf50 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.868+0000 7f60477fe640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44ddd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.868+0000 7f60477fe640 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44ddd3a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3e7d9d625b4d19 server_cookie=e3dd6d31ace5a0ad in_seq=0 out_seq=0 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.868+0000 7f6026ffd640 10 client.30590 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f6026ffd640 1 -- 172.21.15.116:0/2879309623 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb44d2e6a0 con 0x55fb44ddaf50 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f6026ffd640 10 client.30590 handle_command_reply: tid=0 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 client.30590 shutdown 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 msgr2=0x55fb44dc8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f603c011980 tx=0x7f603c011e40 comp rx=0 tx=0).stop 2024-03-19T02:46:16.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 shutdown_connections 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44ddd3a0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f6026ffd640 0 client.30590 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44dc86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc20b0 0x55fb44dc8190 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 --2- 172.21.15.116:0/2879309623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc1790 0x55fb44dc7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 >> 172.21.15.116:0/2879309623 conn(0x55fb44dbbec0 msgr2=0x55fb44dbdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 shutdown_connections 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 1 -- 172.21.15.116:0/2879309623 wait complete. 2024-03-19T02:46:16.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.870+0000 7f605086d740 20 client.30590 trim_cache size 0 max 16384 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 Processor -- start 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 -- start start 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc1790 0x55fb44dca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dd6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60481aca60 con 0x55fb44dcac90 2024-03-19T02:46:16.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60481ae5a0 con 0x55fb44dc1790 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f605086d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60481ad130 con 0x55fb44dc20b0 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37646/0 (socket says 172.21.15.116:37646) 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 -- 172.21.15.116:0/2307505780 learned_addr learned my addr 172.21.15.116:0/2307505780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6046ffd640 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dd6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f60477fe640 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc1790 0x55fb44dca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 -- 172.21.15.116:0/2307505780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 msgr2=0x55fb44dd6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dd6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 -- 172.21.15.116:0/2307505780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc1790 msgr2=0x55fb44dca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.893+0000 7f6047fff640 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc1790 0x55fb44dca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.894+0000 7f6047fff640 1 -- 172.21.15.116:0/2307505780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb44dbbcf0 con 0x55fb44dcac90 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.894+0000 7f6047fff640 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f6040009a60 tx=0x7f6040004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e7fa35e81a5dff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.894+0000 7f602d7fa640 1 -- 172.21.15.116:0/2307505780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040059070 con 0x55fb44dcac90 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.894+0000 7f602d7fa640 1 -- 172.21.15.116:0/2307505780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6040034640 con 0x55fb44dcac90 2024-03-19T02:46:16.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.894+0000 7f602d7fa640 1 -- 172.21.15.116:0/2307505780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040039070 con 0x55fb44dcac90 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- 172.21.15.116:0/2307505780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 msgr2=0x55fb44de2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f6040009a60 tx=0x7f6040004a40 comp rx=0 tx=0).stop 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- 172.21.15.116:0/2307505780 shutdown_connections 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dcac90 0x55fb44de2c70 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dd6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- 172.21.15.116:0/2307505780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dc1790 0x55fb44dca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- 172.21.15.116:0/2307505780 >> 172.21.15.116:0/2307505780 conn(0x55fb44dbbec0 msgr2=0x55fb44db6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- 172.21.15.116:0/2307505780 shutdown_connections 2024-03-19T02:46:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- 172.21.15.116:0/2307505780 wait complete. 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- start start 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 Processor -- start 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dd3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dcac90 0x55fb44dd0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60481aded0 con 0x55fb44dc1790 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60481ad800 con 0x55fb44dcac90 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.895+0000 7f605086d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f60481aaa60 con 0x55fb44dc20b0 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f60477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dd3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43964/0 (socket says 172.21.15.116:43964) 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6047fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dcac90 0x55fb44dd0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 -- 172.21.15.116:0/3930259140 learned_addr learned my addr 172.21.15.116:0/3930259140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f605086d740 10 client.0 _ll_register_callbacks cb 0x7f603c00be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 -- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dcac90 msgr2=0x55fb44dd0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dcac90 0x55fb44dd0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 -- 172.21.15.116:0/3930259140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 msgr2=0x55fb44dd3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dd3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 -- 172.21.15.116:0/3930259140 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb44dd0b90 con 0x55fb44dc20b0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f60477fe640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dd3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6046ffd640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f603c000c00 tx=0x7f603c00bc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5271b5e684f4e34b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6026ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c01bce0 con 0x55fb44dc20b0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6040034640 con 0x55fb44dc20b0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f603c005040 con 0x55fb44dc20b0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f603c005240 con 0x55fb44dc20b0 2024-03-19T02:46:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.896+0000 7f605086d740 10 client.30574 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fb44e42330 con 0x55fb44dc20b0 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6040034640 con 0x55fb44dc20b0 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f603c0349c0 con 0x55fb44dc20b0 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 10 client.30574 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 10 client.30574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 20 client.30574 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44e44b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 4 client.30574 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb44d289f0 con 0x55fb44ddaf50 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.897+0000 7f60477fe640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44e44b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.898+0000 7f6026ffd640 10 client.30574 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.898+0000 7f60477fe640 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44e44b70 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b72f096c058350d server_cookie=a9093cf096df353 in_seq=0 out_seq=0 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f6026ffd640 1 -- 172.21.15.116:0/3930259140 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb44d289f0 con 0x55fb44ddaf50 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f6026ffd640 10 client.30574 handle_command_reply: tid=0 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 client.30574 shutdown 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 msgr2=0x55fb44dcfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f603c000c00 tx=0x7f603c00bc00 comp rx=0 tx=0).stop 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 shutdown_connections 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fb44ddaf50 0x55fb44e44b70 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fb44dcac90 0x55fb44dd0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fb44dc20b0 0x55fb44dcfe20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 --2- 172.21.15.116:0/3930259140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb44dc1790 0x55fb44dd3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f6026ffd640 0 client.30574 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:16.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 >> 172.21.15.116:0/3930259140 conn(0x55fb44dbbec0 msgr2=0x55fb44dc7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 shutdown_connections 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 1 -- 172.21.15.116:0/3930259140 wait complete. 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.900+0000 7f605086d740 20 client.30574 trim_cache size 0 max 16384 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "8437d0c6-1d0d-4813-a2e1-5727ba796394", 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:46:16.906 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 msgr2=0x7f60300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 0x7f60300a4e30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f604004b000 tx=0x7f6040044000 comp rx=0 tx=0).stop 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 msgr2=0x7f6048197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 0x7f6048197f50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f603800e9f0 tx=0x7f603800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 shutdown_connections 2024-03-19T02:46:16.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f60300a2970 0x7f60300a4e30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6048107870 0x7f604819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60480693e0 0x7f6048198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 --2- 172.21.15.116:0/3426300857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6048068a20 0x7f6048197f50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 >> 172.21.15.116:0/3426300857 conn(0x7f6048077820 msgr2=0x7f60480fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 shutdown_connections 2024-03-19T02:46:16.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:16.901+0000 7f604dc81640 1 -- 172.21.15.116:0/3426300857 wait complete. 2024-03-19T02:46:16.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:46:16.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:46:17.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:16 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- 172.21.15.116:0/2619147761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 msgr2=0x7fdc20100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/2619147761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20100b10 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009930 tx=0x7fdc1002bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- 172.21.15.116:0/2619147761 shutdown_connections 2024-03-19T02:46:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/2619147761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc2010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/2619147761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20100b10 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/2619147761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 0x7fdc20106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- 172.21.15.116:0/2619147761 >> 172.21.15.116:0/2619147761 conn(0x7fdc20077820 msgr2=0x7fdc20077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- 172.21.15.116:0/2619147761 shutdown_connections 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- 172.21.15.116:0/2619147761 wait complete. 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 Processor -- start 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- start start 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc201993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 0x7fdc201a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.035+0000 7fdc27ca1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc20110960 con 0x7fdc20100690 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc27ca1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201107e0 con 0x7fdc20101050 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc27ca1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc20110ae0 con 0x7fdc201066a0 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37672/0 (socket says 172.21.15.116:37672) 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1202567040 learned_addr learned my addr 172.21.15.116:0/1202567040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc2623e640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 0x7fdc201a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc201993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1202567040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 msgr2=0x7fdc201a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 0x7fdc201a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1202567040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 msgr2=0x7fdc201993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc201993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1202567040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc201a4440 con 0x7fdc20100690 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc201993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00e9f0 tx=0x7fdc1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc16ffd640 1 -- 172.21.15.116:0/1202567040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c00cd80 con 0x7fdc20100690 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc201a4610 con 0x7fdc20100690 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc16ffd640 1 -- 172.21.15.116:0/1202567040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc1c0054d0 con 0x7fdc20100690 2024-03-19T02:46:17.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc16ffd640 1 -- 172.21.15.116:0/1202567040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c010490 con 0x7fdc20100690 2024-03-19T02:46:17.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.036+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc201a4b30 con 0x7fdc20100690 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.039+0000 7fdc16ffd640 1 -- 172.21.15.116:0/1202567040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdc1c005020 con 0x7fdc20100690 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.039+0000 7fdc2a96b740 1 Processor -- start 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.039+0000 7fdc2a96b740 1 -- start start 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b303628ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303635e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b30358c500 con 0x55b303636340 2024-03-19T02:46:17.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b303534be0 con 0x55b303629000 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3035885a0 con 0x55b3036286e0 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc16ffd640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 0x7fdc0c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2623e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b303628ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303635e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2623e640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 0x7fdc0c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37686/0 (socket says 172.21.15.116:37686) 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 -- 172.21.15.116:0/967520381 learned_addr learned my addr 172.21.15.116:0/967520381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc16ffd640 1 -- 172.21.15.116:0/1202567040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fdc1c0cf5d0 con 0x7fdc20100690 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 -- 172.21.15.116:0/967520381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 msgr2=0x55b303628ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b303628ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 -- 172.21.15.116:0/967520381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 msgr2=0x55b303635e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303635e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 -- 172.21.15.116:0/967520381 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b303556000 con 0x55b303636340 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2623e640 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b303628ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2523c640 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036040 tx=0x7fdc10005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703159f1af4c4fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc2623e640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 0x7fdc0c0a4e30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdc18005fd0 tx=0x7fdc18009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.040+0000 7fdc0bfff640 1 -- 172.21.15.116:0/967520381 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10044020 con 0x55b303636340 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc0bfff640 1 -- 172.21.15.116:0/967520381 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc1002cc90 con 0x55b303636340 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc0bfff640 1 -- 172.21.15.116:0/967520381 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100046c0 con 0x55b303636340 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 -- 172.21.15.116:0/967520381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 msgr2=0x55b303638730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fdc10036040 tx=0x7fdc10005e50 comp rx=0 tx=0).stop 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 -- 172.21.15.116:0/967520381 shutdown_connections 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b303638730 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303635e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/967520381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b303628ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 -- 172.21.15.116:0/967520381 >> 172.21.15.116:0/967520381 conn(0x55b303623160 msgr2=0x55b303623540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 -- 172.21.15.116:0/967520381 shutdown_connections 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.041+0000 7fdc2a96b740 1 -- 172.21.15.116:0/967520381 wait complete. 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 -- start start 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 Processor -- start 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 0x55b303633730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303636340 0x55b303633c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b303639120 con 0x55b3036286e0 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b303548160 con 0x55b303636340 2024-03-19T02:46:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3036392a0 con 0x55b303629000 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2a96b740 10 client.0 _ll_register_callbacks cb 0x55b303584a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37690/0 (socket says 172.21.15.116:37690) 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1513021761 learned_addr learned my addr 172.21.15.116:0/1513021761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 0x55b303633730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2623e640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303636340 0x55b303633c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1513021761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 msgr2=0x55b303633730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 0x55b303633730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303636340 msgr2=0x55b303633c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303636340 0x55b303633c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc25a3d640 1 -- 172.21.15.116:0/1513021761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b30336a7d0 con 0x55b3036286e0 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.042+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 0x55b303633730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c098220 tx=0x7fdc1c09b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2116a93ea9288cce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c005680 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fdc1c086070 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1002cc90 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c09bf50 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 10 client.20880 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b3032a03f0 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc1002cc90 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdc1c06a120 con 0x55b3036286e0 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 10 client.20880 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 10 client.20880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 20 client.20880 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b303646630 0x55b303648aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 4 client.20880 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b303599a40 con 0x55b303646630 2024-03-19T02:46:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.043+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b303646630 0x55b303648aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.044+0000 7fdc097fa640 10 client.20880 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.044+0000 7fdc2523c640 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b303646630 0x55b303648aa0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1170dc604b6568f0 server_cookie=256ccce1fe4b8904 in_seq=0 out_seq=0 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc097fa640 1 -- 172.21.15.116:0/1513021761 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b303599a40 con 0x55b303646630 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc097fa640 10 client.20880 handle_command_reply: tid=0 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 client.20880 shutdown 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 msgr2=0x55b3036331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c098220 tx=0x7fdc1c09b480 comp rx=0 tx=0).stop 2024-03-19T02:46:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 shutdown_connections 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b303646630 0x55b303648aa0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303636340 0x55b303633c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303629000 0x55b303633730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/1513021761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036331f0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc097fa640 0 client.20880 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.047+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 >> 172.21.15.116:0/1513021761 conn(0x55b303623160 msgr2=0x55b303624eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.048+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 shutdown_connections 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.048+0000 7fdc2a96b740 1 -- 172.21.15.116:0/1513021761 wait complete. 2024-03-19T02:46:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.048+0000 7fdc2a96b740 20 client.20880 trim_cache size 0 max 16384 2024-03-19T02:46:17.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 Processor -- start 2024-03-19T02:46:17.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 -- start start 2024-03-19T02:46:17.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303632ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303636340 0x55b303641f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201acaa0 con 0x55b3036286e0 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201ae5e0 con 0x55b303629000 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201ad170 con 0x55b303636340 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37704/0 (socket says 172.21.15.116:37704) 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 -- 172.21.15.116:0/2022254420 learned_addr learned my addr 172.21.15.116:0/2022254420 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2623e640 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303636340 0x55b303641f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc2523c640 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303632ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 -- 172.21.15.116:0/2022254420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303636340 msgr2=0x55b303641f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303636340 0x55b303641f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 -- 172.21.15.116:0/2022254420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 msgr2=0x55b303632ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303632ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 -- 172.21.15.116:0/2022254420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b303625460 con 0x55b3036286e0 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.068+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00cfd0 tx=0x7fdc1c0a3b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5690073d3957efab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc0b7fe640 1 -- 172.21.15.116:0/2022254420 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c06e020 con 0x55b3036286e0 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc0b7fe640 1 -- 172.21.15.116:0/2022254420 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc1c005680 con 0x55b3036286e0 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc0b7fe640 1 -- 172.21.15.116:0/2022254420 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c09f9a0 con 0x55b3036286e0 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 -- 172.21.15.116:0/2022254420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 msgr2=0x55b3036245b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00cfd0 tx=0x7fdc1c0a3b60 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 -- 172.21.15.116:0/2022254420 shutdown_connections 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b303636340 0x55b303641f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303632ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/2022254420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b3036286e0 0x55b3036245b0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 -- 172.21.15.116:0/2022254420 >> 172.21.15.116:0/2022254420 conn(0x55b303623160 msgr2=0x55b3036221e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 -- 172.21.15.116:0/2022254420 shutdown_connections 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.069+0000 7fdc2a96b740 1 -- 172.21.15.116:0/2022254420 wait complete. 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 -- start start 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 Processor -- start 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b30362fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303630280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201adf10 con 0x55b303636340 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201ad840 con 0x55b303629000 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdc201aaaa0 con 0x55b3036286e0 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2a96b740 10 client.0 _ll_register_callbacks cb 0x7fdc1c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2623e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b30362fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2623e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37710/0 (socket says 172.21.15.116:37710) 2024-03-19T02:46:17.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2623e640 1 -- 172.21.15.116:0/60938774 learned_addr learned my addr 172.21.15.116:0/60938774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2523c640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303630280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 -- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 msgr2=0x55b303630280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303630280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 -- 172.21.15.116:0/60938774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 msgr2=0x55b3036307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc25a3d640 1 -- 172.21.15.116:0/60938774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b303628e30 con 0x55b3036286e0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.070+0000 7fdc2623e640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc25a3d640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b30362fd40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c098220 tx=0x7fdc1c004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7722e0e3492b2eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c06e070 con 0x55b3036286e0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1c005680 con 0x55b3036286e0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fdc1c0cf3d0 con 0x55b3036286e0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c06a330 con 0x55b3036286e0 2024-03-19T02:46:17.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 10 client.30592 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b3032600f0 con 0x55b3036286e0 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc1c005680 con 0x55b3036286e0 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdc1c0975a0 con 0x55b3036286e0 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 10 client.30592 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 10 client.30592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 20 client.30592 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b3036acca0 0x55b3036af090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 4 client.30592 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.071+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b303593d90 con 0x55b3036acca0 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.072+0000 7fdc2523c640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b3036acca0 0x55b3036af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.072+0000 7fdc097fa640 10 client.30592 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:17.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.072+0000 7fdc2523c640 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b3036acca0 0x55b3036af090 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aeb732643fc59e5 server_cookie=856e7e279c810254 in_seq=0 out_seq=0 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.077+0000 7fdc097fa640 1 -- 172.21.15.116:0/60938774 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55b303593d90 con 0x55b3036acca0 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.077+0000 7fdc097fa640 10 client.30592 handle_command_reply: tid=0 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.077+0000 7fdc2a96b740 1 client.30592 shutdown 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.077+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 msgr2=0x55b30362fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.077+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b30362fd40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c098220 tx=0x7fdc1c004a00 comp rx=0 tx=0).stop 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 shutdown_connections 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b3036acca0 0x55b3036af090 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b303636340 0x55b3036307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b303629000 0x55b303630280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc097fa640 0 client.30592 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 --2- 172.21.15.116:0/60938774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3036286e0 0x55b30362fd40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 >> 172.21.15.116:0/60938774 conn(0x55b303623160 msgr2=0x55b303621950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 shutdown_connections 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 1 -- 172.21.15.116:0/60938774 wait complete. 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc2a96b740 20 client.30592 trim_cache size 0 max 16384 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (502 inodes in the stack)", 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:46:17.081 INFO:teuthology.orchestra.run.smithi116.stdout: "8437d0c6-1d0d-4813-a2e1-5727ba796394": { 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "8437d0c6-1d0d-4813-a2e1-5727ba796394", 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:46:17.082 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 msgr2=0x7fdc0c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 0x7fdc0c0a4e30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdc18005fd0 tx=0x7fdc18009500 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 msgr2=0x7fdc20198e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00e9f0 tx=0x7fdc1c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 shutdown_connections 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdc0c0a2970 0x7fdc0c0a4e30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc201066a0 0x7fdc201a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc20101050 0x7fdc201993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 --2- 172.21.15.116:0/1202567040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc20100690 0x7fdc20198e80 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 >> 172.21.15.116:0/1202567040 conn(0x7fdc20077820 msgr2=0x7fdc201030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:17.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 shutdown_connections 2024-03-19T02:46:17.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:17.078+0000 7fdc27ca1640 1 -- 172.21.15.116:0/1202567040 wait complete. 2024-03-19T02:46:17.091 INFO:tasks.cephfs.filesystem:scrub status for tag:8437d0c6-1d0d-4813-a2e1-5727ba796394 - {'path': '/', 'tag': '8437d0c6-1d0d-4813-a2e1-5727ba796394', 'options': 'recursive,force'} 2024-03-19T02:46:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:16 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:17.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38559 5.99 MB/sec execute 219 sec latency 244.284 ms 2024-03-19T02:46:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:16 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: pgmap v583: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:18.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:17 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: pgmap v583: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:18.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:17 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38676 6.00 MB/sec execute 220 sec latency 263.577 ms 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: pgmap v583: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 184 op/s 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:18.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:18.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:17 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:18 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:18 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:19.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 38850 6.04 MB/sec execute 221 sec latency 183.685 ms 2024-03-19T02:46:19.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:18 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:19 smithi154 ceph-mon[35693]: pgmap v584: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:46:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:19 smithi116 ceph-mon[32263]: pgmap v584: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:46:20.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39010 6.06 MB/sec execute 222 sec latency 143.074 ms 2024-03-19T02:46:20.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:19 smithi162 ceph-mon[35761]: pgmap v584: 129 pgs: 129 active+clean; 943 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:46:21.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39164 6.09 MB/sec execute 223 sec latency 144.004 ms 2024-03-19T02:46:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:21 smithi162 ceph-mon[35761]: pgmap v585: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 226 op/s 2024-03-19T02:46:21.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:21 smithi154 ceph-mon[35693]: pgmap v585: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 226 op/s 2024-03-19T02:46:21.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:21 smithi116 ceph-mon[32263]: pgmap v585: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 226 op/s 2024-03-19T02:46:22.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39258 6.10 MB/sec execute 224 sec latency 137.947 ms 2024-03-19T02:46:23.308 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:23 smithi154 ceph-mon[35693]: pgmap v586: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 151 op/s 2024-03-19T02:46:23.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39356 6.10 MB/sec execute 225 sec latency 138.751 ms 2024-03-19T02:46:23.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:23 smithi162 ceph-mon[35761]: pgmap v586: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 151 op/s 2024-03-19T02:46:23.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:23 smithi116 ceph-mon[32263]: pgmap v586: 129 pgs: 129 active+clean; 961 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 151 op/s 2024-03-19T02:46:24.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39403 6.07 MB/sec execute 226 sec latency 144.410 ms 2024-03-19T02:46:24.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:24 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:24 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:24 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:25.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39477 6.06 MB/sec execute 227 sec latency 204.299 ms 2024-03-19T02:46:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:46:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:25 smithi162 ceph-mon[35761]: pgmap v587: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 205 op/s 2024-03-19T02:46:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:46:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:25 smithi154 ceph-mon[35693]: pgmap v587: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 205 op/s 2024-03-19T02:46:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:46:25.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:25 smithi116 ceph-mon[32263]: pgmap v587: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 205 op/s 2024-03-19T02:46:26.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39569 6.08 MB/sec execute 228 sec latency 144.173 ms 2024-03-19T02:46:27.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39665 6.09 MB/sec execute 229 sec latency 237.112 ms 2024-03-19T02:46:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:27 smithi162 ceph-mon[35761]: pgmap v588: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:27 smithi154 ceph-mon[35693]: pgmap v588: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:27.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:27 smithi116 ceph-mon[32263]: pgmap v588: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:28.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39775 6.10 MB/sec execute 230 sec latency 231.295 ms 2024-03-19T02:46:29.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 39884 6.09 MB/sec execute 231 sec latency 159.143 ms 2024-03-19T02:46:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:29 smithi162 ceph-mon[35761]: pgmap v589: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:29.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:29 smithi154 ceph-mon[35693]: pgmap v589: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:29.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:29 smithi116 ceph-mon[32263]: pgmap v589: 129 pgs: 129 active+clean; 973 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 125 op/s 2024-03-19T02:46:30.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40003 6.11 MB/sec execute 232 sec latency 151.878 ms 2024-03-19T02:46:30.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:46:30.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:46:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:46:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:46:30.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:46:30.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:46:31.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40137 6.11 MB/sec execute 233 sec latency 185.901 ms 2024-03-19T02:46:31.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:46:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:46:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:46:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:31 smithi162 ceph-mon[35761]: pgmap v590: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 190 op/s 2024-03-19T02:46:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:46:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:46:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:46:31.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:31 smithi154 ceph-mon[35693]: pgmap v590: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 190 op/s 2024-03-19T02:46:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:46:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:46:31.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:46:31.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:31 smithi116 ceph-mon[32263]: pgmap v590: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 190 op/s 2024-03-19T02:46:32.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40211 6.10 MB/sec execute 234 sec latency 163.963 ms 2024-03-19T02:46:33.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40294 6.09 MB/sec execute 235 sec latency 212.143 ms 2024-03-19T02:46:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:33 smithi162 ceph-mon[35761]: pgmap v591: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-03-19T02:46:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:33 smithi154 ceph-mon[35693]: pgmap v591: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-03-19T02:46:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:33 smithi116 ceph-mon[32263]: pgmap v591: 129 pgs: 129 active+clean; 980 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 118 op/s 2024-03-19T02:46:34.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40384 6.07 MB/sec execute 236 sec latency 200.827 ms 2024-03-19T02:46:35.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40464 6.05 MB/sec execute 237 sec latency 231.540 ms 2024-03-19T02:46:35.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:35 smithi162 ceph-mon[35761]: pgmap v592: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 173 op/s 2024-03-19T02:46:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:35 smithi154 ceph-mon[35693]: pgmap v592: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 173 op/s 2024-03-19T02:46:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:35 smithi116 ceph-mon[32263]: pgmap v592: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 173 op/s 2024-03-19T02:46:36.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40562 6.03 MB/sec execute 238 sec latency 194.252 ms 2024-03-19T02:46:37.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40668 6.02 MB/sec execute 239 sec latency 208.520 ms 2024-03-19T02:46:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:37 smithi162 ceph-mon[35761]: pgmap v593: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:37 smithi154 ceph-mon[35693]: pgmap v593: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:37.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:37 smithi116 ceph-mon[32263]: pgmap v593: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:38.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40780 6.01 MB/sec execute 240 sec latency 210.772 ms 2024-03-19T02:46:39.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:39 smithi116 ceph-mon[32263]: pgmap v594: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:39.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40881 6.00 MB/sec execute 241 sec latency 193.691 ms 2024-03-19T02:46:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:39 smithi162 ceph-mon[35761]: pgmap v594: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:39 smithi154 ceph-mon[35693]: pgmap v594: 129 pgs: 129 active+clean; 995 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:46:40.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 40997 5.99 MB/sec execute 242 sec latency 170.375 ms 2024-03-19T02:46:41.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41111 5.99 MB/sec execute 243 sec latency 148.838 ms 2024-03-19T02:46:42.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41243 6.01 MB/sec execute 244 sec latency 147.891 ms 2024-03-19T02:46:42.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:42 smithi162 ceph-mon[35761]: pgmap v595: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-03-19T02:46:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:42 smithi154 ceph-mon[35693]: pgmap v595: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-03-19T02:46:42.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:42 smithi116 ceph-mon[32263]: pgmap v595: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 216 op/s 2024-03-19T02:46:43.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41374 6.02 MB/sec execute 245 sec latency 158.857 ms 2024-03-19T02:46:44.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41477 6.04 MB/sec execute 246 sec latency 178.069 ms 2024-03-19T02:46:44.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:44 smithi162 ceph-mon[35761]: pgmap v596: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-19T02:46:44.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:44 smithi154 ceph-mon[35693]: pgmap v596: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-19T02:46:44.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:44 smithi116 ceph-mon[32263]: pgmap v596: 129 pgs: 129 active+clean; 984 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-19T02:46:45.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41604 6.05 MB/sec execute 247 sec latency 150.273 ms 2024-03-19T02:46:46.330 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41710 6.05 MB/sec execute 248 sec latency 140.438 ms 2024-03-19T02:46:46.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:46 smithi162 ceph-mon[35761]: pgmap v597: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-03-19T02:46:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:46 smithi154 ceph-mon[35693]: pgmap v597: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-03-19T02:46:46.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:46 smithi116 ceph-mon[32263]: pgmap v597: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 222 op/s 2024-03-19T02:46:47.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:46:47.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:46:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 -- 172.21.15.116:0/3817104994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 msgr2=0x7fe30c10d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 --2- 172.21.15.116:0/3817104994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c10d5b0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fe304009cd0 tx=0x7fe30402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 -- 172.21.15.116:0/3817104994 shutdown_connections 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 --2- 172.21.15.116:0/3817104994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c10d5b0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 --2- 172.21.15.116:0/3817104994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c101290 0x7fe30c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 --2- 172.21.15.116:0/3817104994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c105720 0x7fe30c105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 -- 172.21.15.116:0/3817104994 >> 172.21.15.116:0/3817104994 conn(0x7fe30c076870 msgr2=0x7fe30c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 -- 172.21.15.116:0/3817104994 shutdown_connections 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.224+0000 7fe314709640 1 -- 172.21.15.116:0/3817104994 wait complete. 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 Processor -- start 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 -- start start 2024-03-19T02:46:47.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 0x7fe30c0fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c105720 0x7fe30c0fc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c110a00 con 0x7fe30c101c50 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c110880 con 0x7fe30c105720 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.225+0000 7fe314709640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c110b80 con 0x7fe30c101290 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57378/0 (socket says 172.21.15.116:57378) 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe311ca4640 1 -- 172.21.15.116:0/1190571632 learned_addr learned my addr 172.21.15.116:0/1190571632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 0x7fe30c0fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe312ca6640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c105720 0x7fe30c0fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 -- 172.21.15.116:0/1190571632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c105720 msgr2=0x7fe30c0fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c105720 0x7fe30c0fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 -- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 msgr2=0x7fe30c0fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 -- 172.21.15.116:0/1190571632 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c1a64d0 con 0x7fe30c101290 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 0x7fe30c0fbcc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9f0 tx=0x7fe30800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.226+0000 7fe3037fe640 1 -- 172.21.15.116:0/1190571632 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30800ccc0 con 0x7fe30c101290 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.227+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c1a67c0 con 0x7fe30c101290 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.227+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c1a6d10 con 0x7fe30c101290 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.227+0000 7fe3037fe640 1 -- 172.21.15.116:0/1190571632 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe308005020 con 0x7fe30c101290 2024-03-19T02:46:47.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.227+0000 7fe3037fe640 1 -- 172.21.15.116:0/1190571632 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308004a00 con 0x7fe30c101290 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.229+0000 7fe3037fe640 1 -- 172.21.15.116:0/1190571632 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe308004ba0 con 0x7fe30c101290 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.229+0000 7fe3173d4740 1 Processor -- start 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.229+0000 7fe3173d4740 1 -- start start 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a78e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e0c19dc500 con 0x55e0c1a78a30 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e0c1984be0 con 0x55e0c1a81f30 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3173d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0c19d85a0 con 0x55e0c1a79350 2024-03-19T02:46:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3037fe640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 0x7fe2f80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39868/0 (socket says 172.21.15.116:39868) 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 -- 172.21.15.116:0/1646373210 learned_addr learned my addr 172.21.15.116:0/1646373210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 -- 172.21.15.116:0/1646373210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 msgr2=0x55e0c1a819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a819f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 -- 172.21.15.116:0/1646373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 msgr2=0x55e0c1a78e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a78e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 -- 172.21.15.116:0/1646373210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c19a6000 con 0x55e0c1a81f30 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe312ca6640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 0x7fe2f80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0115f0 tx=0x7fe2fc011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f124a9c97b1171 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.230+0000 7fe3037fe640 1 -- 172.21.15.116:0/1190571632 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fe3080cef40 con 0x7fe30c101290 2024-03-19T02:46:47.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.231+0000 7fe300ff9640 1 -- 172.21.15.116:0/1646373210 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc01e070 con 0x55e0c1a81f30 2024-03-19T02:46:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.231+0000 7fe300ff9640 1 -- 172.21.15.116:0/1646373210 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2fc00f660 con 0x55e0c1a81f30 2024-03-19T02:46:47.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.231+0000 7fe300ff9640 1 -- 172.21.15.116:0/1646373210 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc003850 con 0x55e0c1a81f30 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe312ca6640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 0x7fe2f80a4eb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe30402c630 tx=0x7fe304005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 -- 172.21.15.116:0/1646373210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 msgr2=0x55e0c1a84320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0115f0 tx=0x7fe2fc011ab0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 -- 172.21.15.116:0/1646373210 shutdown_connections 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a84320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1646373210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a78e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 -- 172.21.15.116:0/1646373210 >> 172.21.15.116:0/1646373210 conn(0x55e0c1a73160 msgr2=0x55e0c1a73540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.232+0000 7fe3173d4740 1 -- 172.21.15.116:0/1646373210 shutdown_connections 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 -- 172.21.15.116:0/1646373210 wait complete. 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 -- start start 2024-03-19T02:46:47.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 Processor -- start 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 0x55e0c1a7f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e0c1a84d20 con 0x55e0c1a79350 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e0c1998160 con 0x55e0c1a81f30 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0c1a84ea0 con 0x55e0c1a78a30 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3124a5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 0x55e0c1a7f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3124a5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48082/0 (socket says 172.21.15.116:48082) 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3124a5640 1 -- 172.21.15.116:0/1134956142 learned_addr learned my addr 172.21.15.116:0/1134956142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe312ca6640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.233+0000 7fe3173d4740 10 client.0 _ll_register_callbacks cb 0x55e0c19d4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 -- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 msgr2=0x55e0c1a7f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 -- 172.21.15.116:0/1134956142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 msgr2=0x55e0c1a7f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 0x55e0c1a7f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 -- 172.21.15.116:0/1134956142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c17ba7d0 con 0x55e0c1a78a30 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe312ca6640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 0x55e0c1a7f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3124a5640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe3080cc310 tx=0x7fe30809adc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=940870139bccc386 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30801c040 con 0x55e0c1a78a30 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fe308097830 con 0x55e0c1a78a30 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2fc00f660 con 0x55e0c1a78a30 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe30809b420 con 0x55e0c1a78a30 2024-03-19T02:46:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3173d4740 10 client.30604 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.234+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e0c16f03f0 con 0x55e0c1a78a30 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2fc00f660 con 0x55e0c1a78a30 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe308096f10 con 0x55e0c1a78a30 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 10 client.30604 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 10 client.30604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 20 client.30604 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1a92160 0x55e0c1a945c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 4 client.30604 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0c19e9a40 con 0x55e0c1a92160 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.235+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1a92160 0x55e0c1a945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.236+0000 7fe311ca4640 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1a92160 0x55e0c1a945c0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fa4a3a26555e3c5 server_cookie=8cfd2d364f54516b in_seq=0 out_seq=0 2024-03-19T02:46:47.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.236+0000 7fe2f5ffb640 10 client.30604 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.237+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/1134956142 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e0c19e9a40 con 0x55e0c1a92160 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.237+0000 7fe2f5ffb640 10 client.30604 handle_command_reply: tid=0 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.237+0000 7fe3173d4740 1 client.30604 shutdown 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.237+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 msgr2=0x55e0c1a7eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.237+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe3080cc310 tx=0x7fe30809adc0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 shutdown_connections 2024-03-19T02:46:47.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1a92160 0x55e0c1a945c0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a79350 0x55e0c1a7f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe2f5ffb640 0 client.30604 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 --2- 172.21.15.116:0/1134956142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7eee0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 >> 172.21.15.116:0/1134956142 conn(0x55e0c1a73160 msgr2=0x55e0c1a74eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 shutdown_connections 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 1 -- 172.21.15.116:0/1134956142 wait complete. 2024-03-19T02:46:47.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.238+0000 7fe3173d4740 20 client.30604 trim_cache size 0 max 16384 2024-03-19T02:46:47.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 Processor -- start 2024-03-19T02:46:47.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 -- start start 2024-03-19T02:46:47.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 0x55e0c1a81610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 0x55e0c1a74ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f000ff10 con 0x55e0c1a81f30 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0011a50 con 0x55e0c1a78a30 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3173d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00105e0 con 0x55e0c1a79350 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe3124a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 0x55e0c1a81610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe312ca6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 0x55e0c1a74ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48090/0 (socket says 172.21.15.116:48090) 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.267+0000 7fe311ca4640 1 -- 172.21.15.116:0/3859218528 learned_addr learned my addr 172.21.15.116:0/3859218528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 -- 172.21.15.116:0/3859218528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 msgr2=0x55e0c1a8db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 -- 172.21.15.116:0/3859218528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 msgr2=0x55e0c1a74ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 0x55e0c1a74ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 -- 172.21.15.116:0/3859218528 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c16b00f0 con 0x55e0c1a78a30 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe311ca4640 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe312ca6640 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 0x55e0c1a74ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe3124a5640 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 0x55e0c1a81610 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe3080a3430 tx=0x7fe3080bb050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6193c660d4a2e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe2f7fff640 1 -- 172.21.15.116:0/3859218528 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30801c070 con 0x55e0c1a78a30 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe2f7fff640 1 -- 172.21.15.116:0/3859218528 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe30807d020 con 0x55e0c1a78a30 2024-03-19T02:46:47.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.268+0000 7fe2f7fff640 1 -- 172.21.15.116:0/3859218528 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3080a3960 con 0x55e0c1a78a30 2024-03-19T02:46:47.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- 172.21.15.116:0/3859218528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 msgr2=0x55e0c1a81610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 0x55e0c1a81610 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe3080a3430 tx=0x7fe3080bb050 comp rx=0 tx=0).stop 2024-03-19T02:46:47.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- 172.21.15.116:0/3859218528 shutdown_connections 2024-03-19T02:46:47.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a81f30 0x55e0c1a74ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a8db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 --2- 172.21.15.116:0/3859218528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a78a30 0x55e0c1a81610 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- 172.21.15.116:0/3859218528 >> 172.21.15.116:0/3859218528 conn(0x55e0c1a73160 msgr2=0x55e0c1a74590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- 172.21.15.116:0/3859218528 shutdown_connections 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- 172.21.15.116:0/3859218528 wait complete. 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.269+0000 7fe3173d4740 1 -- start start 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 Processor -- start 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0011380 con 0x55e0c1a78a30 2024-03-19T02:46:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0010cb0 con 0x55e0c1a81f30 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f000df10 con 0x55e0c1a79350 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3173d4740 10 client.0 _ll_register_callbacks cb 0x7fe308004f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48104/0 (socket says 172.21.15.116:48104) 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3124a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 -- 172.21.15.116:0/2535741621 learned_addr learned my addr 172.21.15.116:0/2535741621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe312ca6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 -- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 msgr2=0x55e0c1a7c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 -- 172.21.15.116:0/2535741621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 msgr2=0x55e0c1a7bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe311ca4640 1 -- 172.21.15.116:0/2535741621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c1af77a0 con 0x55e0c1a79350 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.270+0000 7fe3124a5640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe311ca4640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc01c040 tx=0x7fe2fc00fbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92639e6b4adfb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc01e040 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fe2fc0052c0 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30807d020 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.271+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe2fc0054c0 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe3173d4740 10 client.30610 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e0c1af8ab0 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2fc02fd00 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fe2fc0035f0 con 0x55e0c1a79350 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe3173d4740 10 client.30610 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe3173d4740 10 client.30610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.272+0000 7fe3173d4740 20 client.30610 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.273+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1af9040 0x55e0c1afb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.273+0000 7fe3173d4740 4 client.30610 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:46:47.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.273+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe30807d020 con 0x55e0c1af9040 2024-03-19T02:46:47.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.273+0000 7fe3124a5640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1af9040 0x55e0c1afb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.273+0000 7fe2f5ffb640 10 client.30610 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:47.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.274+0000 7fe3124a5640 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1af9040 0x55e0c1afb460 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94f927728f6fcda server_cookie=5bfc806d4cb68d1b in_seq=0 out_seq=0 2024-03-19T02:46:47.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe2f5ffb640 1 -- 172.21.15.116:0/2535741621 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe30807d020 con 0x55e0c1af9040 2024-03-19T02:46:47.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe2f5ffb640 10 client.30610 handle_command_reply: tid=0 2024-03-19T02:46:47.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 client.30610 shutdown 2024-03-19T02:46:47.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 msgr2=0x55e0c1a7c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc01c040 tx=0x7fe2fc00fbb0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 shutdown_connections 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e0c1af9040 0x55e0c1afb460 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e0c1a81f30 0x55e0c1a7c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0c1a79350 0x55e0c1a7c260 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 --2- 172.21.15.116:0/2535741621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0c1a78a30 0x55e0c1a7bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 >> 172.21.15.116:0/2535741621 conn(0x55e0c1a73160 msgr2=0x55e0c1a74e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe2f5ffb640 0 client.30610 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 shutdown_connections 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 1 -- 172.21.15.116:0/2535741621 wait complete. 2024-03-19T02:46:47.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.276+0000 7fe3173d4740 20 client.30610 trim_cache size 0 max 16384 2024-03-19T02:46:47.281 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:47.281 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:46:47.281 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:46:47.281 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:47.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 msgr2=0x7fe2f80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 0x7fe2f80a4eb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe30402c630 tx=0x7fe304005fd0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 msgr2=0x7fe30c0fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 0x7fe30c0fbcc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9f0 tx=0x7fe30800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 shutdown_connections 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe2f80a29f0 0x7fe2f80a4eb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe30c105720 0x7fe30c0fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe30c101c50 0x7fe30c0fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 --2- 172.21.15.116:0/1190571632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe30c101290 0x7fe30c0fbcc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 >> 172.21.15.116:0/1190571632 conn(0x7fe30c076870 msgr2=0x7fe30c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 shutdown_connections 2024-03-19T02:46:47.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.277+0000 7fe314709640 1 -- 172.21.15.116:0/1190571632 wait complete. 2024-03-19T02:46:47.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:46:47.292 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:46:47.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41801 6.03 MB/sec execute 249 sec latency 141.734 ms 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 -- 172.21.15.116:0/1133202921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 msgr2=0x7f624010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 --2- 172.21.15.116:0/1133202921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f624010d5f0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f6238009cf0 tx=0x7f623802e300 comp rx=0 tx=0).stop 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 -- 172.21.15.116:0/1133202921 shutdown_connections 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 --2- 172.21.15.116:0/1133202921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f624010d5f0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 --2- 172.21.15.116:0/1133202921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 --2- 172.21.15.116:0/1133202921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 0x7f6240106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 -- 172.21.15.116:0/1133202921 >> 172.21.15.116:0/1133202921 conn(0x7f6240077820 msgr2=0x7f6240077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 -- 172.21.15.116:0/1133202921 shutdown_connections 2024-03-19T02:46:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.428+0000 7f62472f1640 1 -- 172.21.15.116:0/1133202921 wait complete. 2024-03-19T02:46:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.429+0000 7f62472f1640 1 Processor -- start 2024-03-19T02:46:47.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.429+0000 7f62472f1640 1 -- start start 2024-03-19T02:46:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 0x7f624019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f624010f8e0 con 0x7f62401010e0 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f624010f760 con 0x7f6240100720 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f62472f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f624010fa60 con 0x7f6240106730 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624488c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624588e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 0x7f624019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624488c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57422/0 (socket says 172.21.15.116:57422) 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39910/0 (socket says 172.21.15.116:39910) 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624488c640 1 -- 172.21.15.116:0/1457077463 learned_addr learned my addr 172.21.15.116:0/1457077463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 -- 172.21.15.116:0/1457077463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 msgr2=0x7f624019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 0x7f624019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 -- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 msgr2=0x7f6240196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 -- 172.21.15.116:0/1457077463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624019adb0 con 0x7f6240100720 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624488c640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.430+0000 7f624508d640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f623c00e9f0 tx=0x7f623c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.431+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c00ccc0 con 0x7f6240100720 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.431+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f623c0108e0 con 0x7f6240100720 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.431+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c018880 con 0x7f6240100720 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.431+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624019b0a0 con 0x7f6240100720 2024-03-19T02:46:47.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.431+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62401a6d40 con 0x7f6240100720 2024-03-19T02:46:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.432+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6240068610 con 0x7f6240100720 2024-03-19T02:46:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.433+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f623c010a90 con 0x7f6240100720 2024-03-19T02:46:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.434+0000 7f62367fc640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 0x7f622c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.434+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f623c0cf4f0 con 0x7f6240100720 2024-03-19T02:46:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.434+0000 7f624488c640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 0x7f622c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.435+0000 7f624488c640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 0x7f622c0a4e30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6240102870 tx=0x7f6230005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.439+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f623c09b820 con 0x7f6240100720 2024-03-19T02:46:47.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.764+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6240108c90 con 0x7f6240100720 2024-03-19T02:46:47.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.765+0000 7f62367fc640 1 -- 172.21.15.116:0/1457077463 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f623c09f8c0 con 0x7f6240100720 2024-03-19T02:46:47.766 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:46:47.767 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:46:47.767 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:46:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 msgr2=0x7f622c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 0x7f622c0a4e30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6240102870 tx=0x7f6230005f90 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 msgr2=0x7f6240195dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f623c00e9f0 tx=0x7f623c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 shutdown_connections 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f622c0a2970 0x7f622c0a4e30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6240106730 0x7f624019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62401010e0 0x7f6240196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 --2- 172.21.15.116:0/1457077463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6240100720 0x7f6240195dc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 >> 172.21.15.116:0/1457077463 conn(0x7f6240077820 msgr2=0x7f6240103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 shutdown_connections 2024-03-19T02:46:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.769+0000 7f62472f1640 1 -- 172.21.15.116:0/1457077463 wait complete. 2024-03-19T02:46:47.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:46:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 -- 172.21.15.116:0/1784221576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101080 msgr2=0x7f69b0101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 --2- 172.21.15.116:0/1784221576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101080 0x7f69b0101460 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009cd0 tx=0x7f69a802c140 comp rx=0 tx=0).stop 2024-03-19T02:46:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 -- 172.21.15.116:0/1784221576 shutdown_connections 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 --2- 172.21.15.116:0/1784221576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01059d0 0x7f69b010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 --2- 172.21.15.116:0/1784221576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101a30 0x7f69b01052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 --2- 172.21.15.116:0/1784221576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b0101080 0x7f69b0101460 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 -- 172.21.15.116:0/1784221576 >> 172.21.15.116:0/1784221576 conn(0x7f69b0076c50 msgr2=0x7f69b00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 -- 172.21.15.116:0/1784221576 shutdown_connections 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.954+0000 7f69b8537640 1 -- 172.21.15.116:0/1784221576 wait complete. 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 Processor -- start 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 -- start start 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 0x7f69b01a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69b0110940 con 0x7f69b01059d0 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01107c0 con 0x7f69b01998b0 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.955+0000 7f69b8537640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69b0110ac0 con 0x7f69b0101080 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 0x7f69b01a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 0x7f69b01a3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39754/0 (socket says 172.21.15.116:39754) 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47848/0 (socket says 172.21.15.116:47848) 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b6ad4640 1 -- 172.21.15.116:0/477461756 learned_addr learned my addr 172.21.15.116:0/477461756 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45152/0 (socket says 172.21.15.116:45152) 2024-03-19T02:46:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 -- 172.21.15.116:0/477461756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 msgr2=0x7f69b01a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 0x7f69b01a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 -- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 msgr2=0x7f69b0199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 -- 172.21.15.116:0/477461756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b01a4660 con 0x7f69b0101080 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b62d3640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009bf0 tx=0x7f69a8035ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69a37fe640 1 -- 172.21.15.116:0/477461756 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8039040 con 0x7f69b0101080 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69a37fe640 1 -- 172.21.15.116:0/477461756 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69a8044070 con 0x7f69b0101080 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b01a48f0 con 0x7f69b0101080 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.956+0000 7f69a37fe640 1 -- 172.21.15.116:0/477461756 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a80042a0 con 0x7f69b0101080 2024-03-19T02:46:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.957+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b01a4db0 con 0x7f69b0101080 2024-03-19T02:46:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.959+0000 7f69a37fe640 1 -- 172.21.15.116:0/477461756 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f69a8004440 con 0x7f69b0101080 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 Processor -- start 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 -- start start 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f686b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f71290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f73bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560012ecb4b0 con 0x560012f682d0 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560012e73d70 con 0x560012f68bf0 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.960+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560012ec7550 con 0x560012f717d0 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f686b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69a37fe640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 0x7f699c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f686b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45164/0 (socket says 172.21.15.116:45164) 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b6ad4640 1 -- 172.21.15.116:0/3430496791 learned_addr learned my addr 172.21.15.116:0/3430496791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f73bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b62d3640 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f71290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 0x7f699c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69a37fe640 1 -- 172.21.15.116:0/477461756 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f69a80f14c0 con 0x7f69b0101080 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3430496791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 msgr2=0x560012f71290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f71290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3430496791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 msgr2=0x560012f686b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f686b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3430496791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560012e96210 con 0x560012f717d0 2024-03-19T02:46:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f73bc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69b0068950 tx=0x7f69ac0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7170b0d675129e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 0x7f699c0a4e30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f69a4012a10 tx=0x7f69a4004730 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.961+0000 7f69a0bfe640 1 -- 172.21.15.116:0/3430496791 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac004310 con 0x560012f717d0 2024-03-19T02:46:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69a0bfe640 1 -- 172.21.15.116:0/3430496791 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69ac0044b0 con 0x560012f717d0 2024-03-19T02:46:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69a0bfe640 1 -- 172.21.15.116:0/3430496791 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac004e20 con 0x560012f717d0 2024-03-19T02:46:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3430496791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 msgr2=0x560012f73bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f73bc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69b0068950 tx=0x7f69ac0118d0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3430496791 shutdown_connections 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f73bc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f71290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3430496791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f686b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.962+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3430496791 >> 172.21.15.116:0/3430496791 conn(0x560012f62a00 msgr2=0x560012f62de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3430496791 shutdown_connections 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3430496791 wait complete. 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 -- start start 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 Processor -- start 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.963+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f6f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560012f789d0 con 0x560012f682d0 2024-03-19T02:46:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560012e87310 con 0x560012f68bf0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560012f78b50 con 0x560012f717d0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b62d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b62d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45180/0 (socket says 172.21.15.116:45180) 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b62d3640 1 -- 172.21.15.116:0/4215290098 learned_addr learned my addr 172.21.15.116:0/4215290098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39774/0 (socket says 172.21.15.116:39774) 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69bb1fc740 10 client.0 _ll_register_callbacks cb 0x560012e7b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f6f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 -- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 msgr2=0x560012f6f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f6f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 -- 172.21.15.116:0/4215290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 msgr2=0x560012f6e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b5ad2640 1 -- 172.21.15.116:0/4215290098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560012bdf3f0 con 0x560012f68bf0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b62d3640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.964+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x560012f63b90 tx=0x7f69ac0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=908fad604f42df39 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f6999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac029070 con 0x560012f68bf0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69ac0044b0 con 0x560012f68bf0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f69ac00f480 con 0x560012f68bf0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f69ac035050 con 0x560012f68bf0 2024-03-19T02:46:47.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f69bb1fc740 10 client.30650 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.965+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560012ca97d0 con 0x560012f68bf0 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69ac0044b0 con 0x560012f68bf0 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f69ac0235f0 con 0x560012f68bf0 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 10 client.30650 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 10 client.30650 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 20 client.30650 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012f88000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 4 client.30650 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560012f293e0 con 0x560012f85c10 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69b62d3640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012f88000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f6999ffb640 10 client.30650 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:47.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.966+0000 7f69b62d3640 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012f88000 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=581f8bb19d9b96cc server_cookie=107de2832f4cd56f in_seq=0 out_seq=0 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.967+0000 7f6999ffb640 1 -- 172.21.15.116:0/4215290098 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560012f293e0 con 0x560012f85c10 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.967+0000 7f6999ffb640 10 client.30650 handle_command_reply: tid=0 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 client.30650 shutdown 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 msgr2=0x560012f6ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x560012f63b90 tx=0x7f69ac0038a0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 shutdown_connections 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012f88000 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f717d0 0x560012f6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f68bf0 0x560012f6ece0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f6999ffb640 0 client.30650 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/4215290098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f682d0 0x560012f6e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 >> 172.21.15.116:0/4215290098 conn(0x560012f62a00 msgr2=0x560012f64750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 shutdown_connections 2024-03-19T02:46:47.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 1 -- 172.21.15.116:0/4215290098 wait complete. 2024-03-19T02:46:47.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.968+0000 7f69bb1fc740 20 client.30650 trim_cache size 0 max 16384 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 Processor -- start 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 -- start start 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f70ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f8d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01ac8e0 con 0x560012f717d0 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01ae420 con 0x560012f682d0 2024-03-19T02:46:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01acfb0 con 0x560012f68bf0 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f8d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b5ad2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47872/0 (socket says 172.21.15.116:47872) 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f8d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45190/0 (socket says 172.21.15.116:45190) 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3339633505 learned_addr learned my addr 172.21.15.116:0/3339633505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.987+0000 7f69b62d3640 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f70ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3339633505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 msgr2=0x560012f70ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f70ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3339633505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 msgr2=0x560012f8d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f8d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 -- 172.21.15.116:0/3339633505 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560012f38880 con 0x560012f68bf0 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011910 tx=0x7f69ac011f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9e5c9d9b2ba676d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f699bfff640 1 -- 172.21.15.116:0/3339633505 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac029070 con 0x560012f68bf0 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f699bfff640 1 -- 172.21.15.116:0/3339633505 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69ac023c30 con 0x560012f68bf0 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.988+0000 7f699bfff640 1 -- 172.21.15.116:0/3339633505 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac02e460 con 0x560012f68bf0 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3339633505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 msgr2=0x560012f81770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f69ac011910 tx=0x7f69ac011f00 comp rx=0 tx=0).stop 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3339633505 shutdown_connections 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f8d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f81770 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3339633505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f70ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3339633505 >> 172.21.15.116:0/3339633505 conn(0x560012f62a00 msgr2=0x560012f646e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3339633505 shutdown_connections 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3339633505 wait complete. 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.989+0000 7f69bb1fc740 1 -- start start 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 Processor -- start 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f7ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01add50 con 0x560012f717d0 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01ad680 con 0x560012f682d0 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f69b01aa8e0 con 0x560012f68bf0 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69bb1fc740 10 client.0 _ll_register_callbacks cb 0x7f69a400bcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45194/0 (socket says 172.21.15.116:45194) 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 -- 172.21.15.116:0/3203163521 learned_addr learned my addr 172.21.15.116:0/3203163521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b62d3640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f7ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b5ad2640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 -- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 msgr2=0x560012f792a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f792a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 -- 172.21.15.116:0/3203163521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 msgr2=0x560012f7ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f7ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 -- 172.21.15.116:0/3203163521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560012f79d50 con 0x560012f717d0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.990+0000 7f69b6ad4640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f69a40123c0 tx=0x7f69a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4983cac9b119c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a4021950 con 0x560012f717d0 2024-03-19T02:46:47.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69ac023c30 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f69a401b070 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f69a4021af0 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 10 client.20925 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560012febc00 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69ac023c30 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f69a4020c90 con 0x560012f717d0 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 10 client.20925 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 10 client.20925 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 20 client.20925 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012fee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 4 client.20925 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.991+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560012ed89f0 con 0x560012f85c10 2024-03-19T02:46:47.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69b62d3640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012fee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f6999ffb640 10 client.20925 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69b62d3640 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012fee440 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f26a2c286759972 server_cookie=975bef944c2ea53a in_seq=0 out_seq=0 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f6999ffb640 1 -- 172.21.15.116:0/3203163521 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560012ed89f0 con 0x560012f85c10 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f6999ffb640 10 client.20925 handle_command_reply: tid=0 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69bb1fc740 1 client.20925 shutdown 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 msgr2=0x560012f797e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f69a40123c0 tx=0x7f69a40046c0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 shutdown_connections 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.992+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x560012f85c10 0x560012fee440 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560012f717d0 0x560012f797e0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560012f68bf0 0x560012f792a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:47.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 --2- 172.21.15.116:0/3203163521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560012f682d0 0x560012f7ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 >> 172.21.15.116:0/3203163521 conn(0x560012f62a00 msgr2=0x560012f6e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f6999ffb640 0 client.20925 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 shutdown_connections 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 1 -- 172.21.15.116:0/3203163521 wait complete. 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69bb1fc740 20 client.20925 trim_cache size 0 max 16384 2024-03-19T02:46:48.000 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 msgr2=0x7f699c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 0x7f699c0a4e30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f69a4012a10 tx=0x7f69a4004730 comp rx=0 tx=0).stop 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 msgr2=0x7f69b0198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009bf0 tx=0x7f69a8035ce0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 shutdown_connections 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f699c0a2970 0x7f699c0a4e30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69b01998b0 0x7f69b01a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b01059d0 0x7f69b0199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 --2- 172.21.15.116:0/477461756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69b0101080 0x7f69b0198de0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.993+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 >> 172.21.15.116:0/477461756 conn(0x7f69b0076c50 msgr2=0x7f69b0103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.994+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 shutdown_connections 2024-03-19T02:46:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:47.994+0000 7f69b8537640 1 -- 172.21.15.116:0/477461756 wait complete. 2024-03-19T02:46:48.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:46:48.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- 172.21.15.116:0/3431590333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1066b0 msgr2=0x7fa7cc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3431590333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc106a90 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009930 tx=0x7fa7bc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- 172.21.15.116:0/3431590333 shutdown_connections 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3431590333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 0x7fa7cc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3431590333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3431590333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc106a90 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- 172.21.15.116:0/3431590333 >> 172.21.15.116:0/3431590333 conn(0x7fa7cc0778d0 msgr2=0x7fa7cc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- 172.21.15.116:0/3431590333 shutdown_connections 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- 172.21.15.116:0/3431590333 wait complete. 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 Processor -- start 2024-03-19T02:46:48.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.138+0000 7fa7d28de640 1 -- start start 2024-03-19T02:46:48.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 0x7fa7cc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc110970 con 0x7fa7cc1006a0 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1107f0 con 0x7fa7cc1066b0 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d28de640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc110af0 con 0x7fa7cc101060 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45212/0 (socket says 172.21.15.116:45212) 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 -- 172.21.15.116:0/3595305171 learned_addr learned my addr 172.21.15.116:0/3595305171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 0x7fa7cc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 -- 172.21.15.116:0/3595305171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 msgr2=0x7fa7cc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 0x7fa7cc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 -- 172.21.15.116:0/3595305171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1066b0 msgr2=0x7fa7cc19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 -- 172.21.15.116:0/3595305171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc19cfb0 con 0x7fa7cc1006a0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.139+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009230 tx=0x7fa7bc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.140+0000 7fa7c97fa640 1 -- 172.21.15.116:0/3595305171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc043020 con 0x7fa7cc1006a0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.140+0000 7fa7c97fa640 1 -- 172.21.15.116:0/3595305171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7bc02ca70 con 0x7fa7cc1006a0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.140+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7cc19d240 con 0x7fa7cc1006a0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.140+0000 7fa7c97fa640 1 -- 172.21.15.116:0/3595305171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc003a50 con 0x7fa7cc1006a0 2024-03-19T02:46:48.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.140+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7cc1a4b30 con 0x7fa7cc1006a0 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.142+0000 7fa7c97fa640 1 -- 172.21.15.116:0/3595305171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa7bc046020 con 0x7fa7cc1006a0 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 Processor -- start 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 -- start start 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abdbf0 0x55b019ac6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ac8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b019a204b0 con 0x55b019abd2d0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0199c8d70 con 0x55b019abdbf0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d559b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b019a1c550 con 0x55b019ac67d0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d0e7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7c97fa640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 0x7fa7b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d0e7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45214/0 (socket says 172.21.15.116:45214) 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7d0e7b640 1 -- 172.21.15.116:0/4128609463 learned_addr learned my addr 172.21.15.116:0/4128609463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.143+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abdbf0 0x55b019ac6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7c97fa640 1 -- 172.21.15.116:0/3595305171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fa7bc0f1110 con 0x7fa7cc1006a0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 -- 172.21.15.116:0/4128609463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 msgr2=0x55b019ac8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ac8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 -- 172.21.15.116:0/4128609463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abdbf0 msgr2=0x55b019ac6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abdbf0 0x55b019ac6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 -- 172.21.15.116:0/4128609463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0199eb210 con 0x55b019abd2d0 2024-03-19T02:46:48.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 0x7fa7b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4005c40 tx=0x7fa7c4013930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99f846cc8e2da1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7b27fc640 1 -- 172.21.15.116:0/4128609463 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4010430 con 0x55b019abd2d0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 0x7fa7b40a4e30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4023000 tx=0x7fa7c4020000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7b27fc640 1 -- 172.21.15.116:0/4128609463 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7c4013190 con 0x55b019abd2d0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.144+0000 7fa7b27fc640 1 -- 172.21.15.116:0/4128609463 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c401b620 con 0x55b019abd2d0 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 -- 172.21.15.116:0/4128609463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 msgr2=0x55b019abd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4005c40 tx=0x7fa7c4013930 comp rx=0 tx=0).stop 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 -- 172.21.15.116:0/4128609463 shutdown_connections 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ac8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abdbf0 0x55b019ac6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 --2- 172.21.15.116:0/4128609463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019abd6b0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 -- 172.21.15.116:0/4128609463 >> 172.21.15.116:0/4128609463 conn(0x55b019ab7a00 msgr2=0x55b019ab7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 -- 172.21.15.116:0/4128609463 shutdown_connections 2024-03-19T02:46:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.145+0000 7fa7d559b740 1 -- 172.21.15.116:0/4128609463 wait complete. 2024-03-19T02:46:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 -- start start 2024-03-19T02:46:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 Processor -- start 2024-03-19T02:46:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abdbf0 0x55b019ac3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b019ac95c0 con 0x55b019abd2d0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0199dc310 con 0x55b019ac67d0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.146+0000 7fa7d559b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b019ac9740 con 0x55b019abdbf0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7d559b740 10 client.0 _ll_register_callbacks cb 0x55b0199d02d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abdbf0 0x55b019ac3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45218/0 (socket says 172.21.15.116:45218) 2024-03-19T02:46:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 -- 172.21.15.116:0/372584734 learned_addr learned my addr 172.21.15.116:0/372584734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 -- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abdbf0 msgr2=0x55b019ac3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abdbf0 0x55b019ac3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 -- 172.21.15.116:0/372584734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 msgr2=0x55b019ac42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 -- 172.21.15.116:0/372584734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0197343f0 con 0x55b019abd2d0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000c00 tx=0x7fa7bc02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8182ad594e7cf71d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0bd4a0 con 0x55b019abd2d0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.147+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fa7bc0b9b10 con 0x55b019abd2d0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7c4013190 con 0x55b019abd2d0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa7bc0c15f0 con 0x55b019abd2d0 2024-03-19T02:46:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 10 client.20937 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b0197fe7d0 con 0x55b019abd2d0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7c4013190 con 0x55b019abd2d0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa7bc0bd700 con 0x55b019abd2d0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 10 client.20937 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 10 client.20937 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 20 client.20937 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019adb020 0x55b019add410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 4 client.20937 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.148+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b019a7e3e0 con 0x55b019adb020 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.149+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019adb020 0x55b019add410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.149+0000 7fa7ab7fe640 10 client.20937 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.149+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019adb020 0x55b019add410 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8630aef5b39c88a7 server_cookie=46c960af82b11070 in_seq=0 out_seq=0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.149+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/372584734 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b019a7e3e0 con 0x55b019adb020 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7ab7fe640 10 client.20937 handle_command_reply: tid=0 2024-03-19T02:46:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 client.20937 shutdown 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 msgr2=0x55b019ac3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000c00 tx=0x7fa7bc02c740 comp rx=0 tx=0).stop 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 shutdown_connections 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abdbf0 0x55b019ac3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019adb020 0x55b019add410 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 --2- 172.21.15.116:0/372584734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abd2d0 0x55b019ac3850 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7ab7fe640 0 client.20937 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 >> 172.21.15.116:0/372584734 conn(0x55b019ab7a00 msgr2=0x55b019ab9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 shutdown_connections 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 1 -- 172.21.15.116:0/372584734 wait complete. 2024-03-19T02:46:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.150+0000 7fa7d559b740 20 client.20937 trim_cache size 0 max 16384 2024-03-19T02:46:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.166+0000 7fa7d559b740 1 Processor -- start 2024-03-19T02:46:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.166+0000 7fa7d559b740 1 -- start start 2024-03-19T02:46:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.166+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abd2d0 0x55b019ac5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.166+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ae2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7d559b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1aca80 con 0x55b019abdbf0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7d559b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1ae5c0 con 0x55b019abd2d0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7d559b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1ad150 con 0x55b019ac67d0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45228/0 (socket says 172.21.15.116:45228) 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/3007650150 learned_addr learned my addr 172.21.15.116:0/3007650150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ae2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abd2d0 0x55b019ac5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/3007650150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 msgr2=0x55b019ae2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ae2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/3007650150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abd2d0 msgr2=0x55b019ac5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abd2d0 0x55b019ac5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/3007650150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b019ac1da0 con 0x55b019abdbf0 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x55b019ac30b0 tx=0x7fa7c0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be40926d899b6e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7b1ffb640 1 -- 172.21.15.116:0/3007650150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0020020 con 0x55b019abdbf0 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7b1ffb640 1 -- 172.21.15.116:0/3007650150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7c001cd20 con 0x55b019abdbf0 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.167+0000 7fa7b1ffb640 1 -- 172.21.15.116:0/3007650150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c000fb50 con 0x55b019abdbf0 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- 172.21.15.116:0/3007650150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 msgr2=0x55b019ad67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x55b019ac30b0 tx=0x7fa7c0010800 comp rx=0 tx=0).stop 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- 172.21.15.116:0/3007650150 shutdown_connections 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019ac67d0 0x55b019ae2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ad67f0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 --2- 172.21.15.116:0/3007650150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019abd2d0 0x55b019ac5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- 172.21.15.116:0/3007650150 >> 172.21.15.116:0/3007650150 conn(0x55b019ab7a00 msgr2=0x55b019ab7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- 172.21.15.116:0/3007650150 shutdown_connections 2024-03-19T02:46:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- 172.21.15.116:0/3007650150 wait complete. 2024-03-19T02:46:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.168+0000 7fa7d559b740 1 -- start start 2024-03-19T02:46:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 Processor -- start 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 0x55b019ac0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1adef0 con 0x55b019abdbf0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1ad820 con 0x55b019ac67d0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc1aaa80 con 0x55b019abd2d0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d559b740 10 client.0 _ll_register_callbacks cb 0x7fa7bc0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 0x55b019ac0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45230/0 (socket says 172.21.15.116:45230) 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/1975904857 learned_addr learned my addr 172.21.15.116:0/1975904857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7d0e7b640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 msgr2=0x55b019ac0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 0x55b019ac0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/1975904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 msgr2=0x55b019ac0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cb7fe640 1 -- 172.21.15.116:0/1975904857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b019b40200 con 0x55b019abdbf0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.169+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 0x55b019ac0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7cb7fe640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000f7f0 tx=0x7fa7c000f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6591fdb1113b2a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0020030 con 0x55b019abdbf0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fa7c001c5c0 con 0x55b019abdbf0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7c001cd20 con 0x55b019abdbf0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa7c0037050 con 0x55b019abdbf0 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 10 client.20943 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b019b41400 con 0x55b019abdbf0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7c001cd20 con 0x55b019abdbf0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fa7c00256d0 con 0x55b019abdbf0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 10 client.20943 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 10 client.20943 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 20 client.20943 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019b41a60 0x55b019b43e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 4 client.20943 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.170+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b019a2d9f0 con 0x55b019b41a60 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019b41a60 0x55b019b43e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7cbfff640 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019b41a60 0x55b019b43e80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8a3be2449a1aceb server_cookie=3411f692be5f3834 in_seq=0 out_seq=0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7ab7fe640 10 client.20943 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7ab7fe640 1 -- 172.21.15.116:0/1975904857 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b019a2d9f0 con 0x55b019b41a60 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7ab7fe640 10 client.20943 handle_command_reply: tid=0 2024-03-19T02:46:48.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 client.20943 shutdown 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 msgr2=0x55b019ac0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000f7f0 tx=0x7fa7c000f5f0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 shutdown_connections 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b019ac67d0 0x55b019ac0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b019b41a60 0x55b019b43e80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b019abdbf0 0x55b019ac0860 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 --2- 172.21.15.116:0/1975904857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b019abd2d0 0x55b019ac0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 >> 172.21.15.116:0/1975904857 conn(0x55b019ab7a00 msgr2=0x55b019ac3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7ab7fe640 0 client.20943 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 shutdown_connections 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.171+0000 7fa7d559b740 1 -- 172.21.15.116:0/1975904857 wait complete. 2024-03-19T02:46:48.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d559b740 20 client.20943 trim_cache size 0 max 16384 2024-03-19T02:46:48.180 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:48.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 msgr2=0x7fa7b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 0x7fa7b40a4e30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4023000 tx=0x7fa7c4020000 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 msgr2=0x7fa7cc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009230 tx=0x7fa7bc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 shutdown_connections 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa7b40a2970 0x7fa7b40a4e30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7cc1066b0 0x7fa7cc19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7cc101060 0x7fa7cc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 --2- 172.21.15.116:0/3595305171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7cc1006a0 0x7fa7cc198010 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 >> 172.21.15.116:0/3595305171 conn(0x7fa7cc0778d0 msgr2=0x7fa7cc103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 shutdown_connections 2024-03-19T02:46:48.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.172+0000 7fa7d28de640 1 -- 172.21.15.116:0/3595305171 wait complete. 2024-03-19T02:46:48.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:46:48.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 -- 172.21.15.116:0/947421791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc102a40 msgr2=0x7f99fc102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 --2- 172.21.15.116:0/947421791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc102a40 0x7f99fc102e20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009860 tx=0x7f99f002bc00 comp rx=0 tx=0).stop 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 -- 172.21.15.116:0/947421791 shutdown_connections 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 --2- 172.21.15.116:0/947421791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc10bd80 0x7f99fc10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 --2- 172.21.15.116:0/947421791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc103360 0x7f99fc10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 --2- 172.21.15.116:0/947421791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc102a40 0x7f99fc102e20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 -- 172.21.15.116:0/947421791 >> 172.21.15.116:0/947421791 conn(0x7f99fc0778d0 msgr2=0x7f99fc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 -- 172.21.15.116:0/947421791 shutdown_connections 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.317+0000 7f9a031e6640 1 -- 172.21.15.116:0/947421791 wait complete. 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 Processor -- start 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 -- start start 2024-03-19T02:46:48.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc10bd80 0x7f99fc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc111a30 con 0x7f99fc102a40 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1118b0 con 0x7f99fc10bd80 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a031e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc111bb0 con 0x7f99fc103360 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45246/0 (socket says 172.21.15.116:45246) 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47944/0 (socket says 172.21.15.116:47944) 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a00f82640 1 -- 172.21.15.116:0/1761540562 learned_addr learned my addr 172.21.15.116:0/1761540562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.318+0000 7f9a01783640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc10bd80 0x7f99fc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 -- 172.21.15.116:0/1761540562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 msgr2=0x7f99fc1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 -- 172.21.15.116:0/1761540562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc10bd80 msgr2=0x7f99fc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc10bd80 0x7f99fc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 -- 172.21.15.116:0/1761540562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc19cf60 con 0x7f99fc102a40 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f99f0037040 tx=0x7f99f0035f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f99f9ffb640 1 -- 172.21.15.116:0/1761540562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0041020 con 0x7f99fc102a40 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc19d1f0 con 0x7f99fc102a40 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f99f9ffb640 1 -- 172.21.15.116:0/1761540562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99f002c5a0 con 0x7f99fc102a40 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f99f9ffb640 1 -- 172.21.15.116:0/1761540562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0039040 con 0x7f99fc102a40 2024-03-19T02:46:48.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.319+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc1a4ae0 con 0x7f99fc102a40 2024-03-19T02:46:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.322+0000 7f99f9ffb640 1 -- 172.21.15.116:0/1761540562 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f99f0044020 con 0x7f99fc102a40 2024-03-19T02:46:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.322+0000 7f9a05dcf740 1 Processor -- start 2024-03-19T02:46:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.322+0000 7f9a05dcf740 1 -- start start 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b496a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b49be0 0x55c369b4bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c369a9f4b0 con 0x55c369b49be0 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c369a47d70 con 0x55c369b3c8a0 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c369a9b550 con 0x55c369b3bf80 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b49be0 0x55c369b4bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47958/0 (socket says 172.21.15.116:47958) 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 -- 172.21.15.116:0/1996229559 learned_addr learned my addr 172.21.15.116:0/1996229559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 -- 172.21.15.116:0/1996229559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 msgr2=0x55c369b496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 -- 172.21.15.116:0/1996229559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b49be0 msgr2=0x55c369b4bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b49be0 0x55c369b4bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f99f9ffb640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 0x7f99ec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 -- 172.21.15.116:0/1996229559 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c369a6a210 con 0x55c369b3bf80 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 0x7f99ec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f9a01783640 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009f50 tx=0x7f99f40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d63c1e1330c81861 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.323+0000 7f99f9ffb640 1 -- 172.21.15.116:0/1761540562 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f99f00f0710 con 0x7f99fc102a40 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.324+0000 7f99e6ffd640 1 -- 172.21.15.116:0/1996229559 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4004250 con 0x55c369b3bf80 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.324+0000 7f9a01783640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 0x7f99ec0a4eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f99f4012fd0 tx=0x7f99f4021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.324+0000 7f99e6ffd640 1 -- 172.21.15.116:0/1996229559 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99f400fe50 con 0x55c369b3bf80 2024-03-19T02:46:48.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.324+0000 7f99e6ffd640 1 -- 172.21.15.116:0/1996229559 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f401bb60 con 0x55c369b3bf80 2024-03-19T02:46:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1996229559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 msgr2=0x55c369b3c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009f50 tx=0x7f99f40118d0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1996229559 shutdown_connections 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b49be0 0x55c369b4bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1996229559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b3bf80 0x55c369b3c360 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1996229559 >> 172.21.15.116:0/1996229559 conn(0x55c369b36a00 msgr2=0x55c369b36de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1996229559 shutdown_connections 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1996229559 wait complete. 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 -- start start 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.325+0000 7f9a05dcf740 1 Processor -- start 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b470d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b47610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c369b4ca20 con 0x55c369b3bf80 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c369a5b310 con 0x55c369b3c8a0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c369b4cba0 con 0x55c369b49be0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a05dcf740 10 client.0 _ll_register_callbacks cb 0x55c369a4f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45256/0 (socket says 172.21.15.116:45256) 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39898/0 (socket says 172.21.15.116:39898) 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a00f82640 1 -- 172.21.15.116:0/3274847248 learned_addr learned my addr 172.21.15.116:0/3274847248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a01783640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b47610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.326+0000 7f9a00f82640 1 -- 172.21.15.116:0/3274847248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 msgr2=0x55c369b47610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a00f82640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b47610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a00f82640 1 -- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 msgr2=0x55c369b470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a00f82640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a00f82640 1 -- 172.21.15.116:0/3274847248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3697b33f0 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a00f82640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f99f003d040 tx=0x7f99f0009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ba792281e56759 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f99dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00bc870 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f99f00edc70 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99f400fe50 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f99f00bc400 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a05dcf740 10 client.20955 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c36987d7d0 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.327+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99f00ede70 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f99f003c030 con 0x55c369b3bf80 2024-03-19T02:46:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 10 client.20955 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 10 client.20955 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 20 client.20955 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369b5a040 0x55c369b5c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 4 client.20955 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f99f400fe50 con 0x55c369b5a040 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f99fbfff640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369b5a040 0x55c369b5c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f99dbfff640 10 client.20955 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.328+0000 7f99fbfff640 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369b5a040 0x55c369b5c4c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1980538209d40563 server_cookie=f4dc2211ed14dc81 in_seq=0 out_seq=0 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f99dbfff640 1 -- 172.21.15.116:0/3274847248 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f99f400fe50 con 0x55c369b5a040 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f99dbfff640 10 client.20955 handle_command_reply: tid=0 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 client.20955 shutdown 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 msgr2=0x55c369b46b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f99f003d040 tx=0x7f99f0009790 comp rx=0 tx=0).stop 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 shutdown_connections 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369b5a040 0x55c369b5c4c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b47610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f99dbfff640 0 client.20955 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/3274847248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b46b90 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.329+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 >> 172.21.15.116:0/3274847248 conn(0x55c369b36a00 msgr2=0x55c369b38750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.330+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 shutdown_connections 2024-03-19T02:46:48.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.330+0000 7f9a05dcf740 1 -- 172.21.15.116:0/3274847248 wait complete. 2024-03-19T02:46:48.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 41901 6.01 MB/sec execute 250 sec latency 145.281 ms 2024-03-19T02:46:48.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.330+0000 7f9a05dcf740 20 client.20955 trim_cache size 0 max 16384 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.349+0000 7f9a05dcf740 1 Processor -- start 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 -- start start 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 0x55c369b49250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b35a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1ac9a0 con 0x55c369b3c8a0 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1ae4e0 con 0x55c369b3bf80 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1ad070 con 0x55c369b49be0 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45264/0 (socket says 172.21.15.116:45264) 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 0x55c369b49250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 -- 172.21.15.116:0/1926924234 learned_addr learned my addr 172.21.15.116:0/1926924234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a01783640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b35a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 -- 172.21.15.116:0/1926924234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 msgr2=0x55c369b35a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b35a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 -- 172.21.15.116:0/1926924234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 msgr2=0x55c369b49250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 0x55c369b49250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f99fbfff640 1 -- 172.21.15.116:0/1926924234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c369b5cb40 con 0x55c369b3c8a0 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.350+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 0x55c369b49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.351+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x55c369b3bc70 tx=0x7f99e8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b402cecd78a5b5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.351+0000 7f99e67fc640 1 -- 172.21.15.116:0/1926924234 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8010960 con 0x55c369b3c8a0 2024-03-19T02:46:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.351+0000 7f99e67fc640 1 -- 172.21.15.116:0/1926924234 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e8023030 con 0x55c369b3c8a0 2024-03-19T02:46:48.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.351+0000 7f99e67fc640 1 -- 172.21.15.116:0/1926924234 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8004a90 con 0x55c369b3c8a0 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1926924234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 msgr2=0x55c369b55890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x55c369b3bc70 tx=0x7f99e8012bc0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1926924234 shutdown_connections 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b35a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3c8a0 0x55c369b55890 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1926924234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3bf80 0x55c369b49250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1926924234 >> 172.21.15.116:0/1926924234 conn(0x55c369b36a00 msgr2=0x55c369b35310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1926924234 shutdown_connections 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.352+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1926924234 wait complete. 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 -- start start 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 Processor -- start 2024-03-19T02:46:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b4dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b4e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1ade10 con 0x55c369b3bf80 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1ad740 con 0x55c369b3c8a0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc1aa9a0 con 0x55c369b49be0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a05dcf740 10 client.0 _ll_register_callbacks cb 0x7f99f0005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a00f82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45266/0 (socket says 172.21.15.116:45266) 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a00f82640 1 -- 172.21.15.116:0/1676443776 learned_addr learned my addr 172.21.15.116:0/1676443776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b4dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.353+0000 7f9a01783640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b4e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 -- 172.21.15.116:0/1676443776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 msgr2=0x55c369b4e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b4e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 -- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 msgr2=0x55c369b4dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b4dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 -- 172.21.15.116:0/1676443776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c369b4bfd0 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b4dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a00f82640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ee080 tx=0x7f99f0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100f2ec7e9542ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f99dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00bc5e0 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f99f00edc40 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e8023030 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f99f00f0550 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a05dcf740 10 client.20961 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.354+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c369bc0510 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e8023030 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f99f00db120 con 0x55c369b3bf80 2024-03-19T02:46:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 10 client.20961 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 10 client.20961 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 20 client.20961 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369bc0b00 0x55c369bc2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 4 client.20961 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c369b42790 con 0x55c369bc0b00 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369bc0b00 0x55c369bc2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f99dbfff640 10 client.20961 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.355+0000 7f99fbfff640 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369bc0b00 0x55c369bc2f20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30a33bb896bf6205 server_cookie=bb1ba9d73deb2be9 in_seq=0 out_seq=0 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f99dbfff640 1 -- 172.21.15.116:0/1676443776 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c369b42790 con 0x55c369bc0b00 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f99dbfff640 10 client.20961 handle_command_reply: tid=0 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f9a05dcf740 1 client.20961 shutdown 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 msgr2=0x55c369b4d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ee080 tx=0x7f99f0009580 comp rx=0 tx=0).stop 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.356+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 shutdown_connections 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c369bc0b00 0x55c369bc2f20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c369b49be0 0x55c369b4e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f99dbfff640 0 client.20961 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c369b3c8a0 0x55c369b4dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 --2- 172.21.15.116:0/1676443776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c369b3bf80 0x55c369b4d7b0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 >> 172.21.15.116:0/1676443776 conn(0x55c369b36a00 msgr2=0x55c369b45060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 shutdown_connections 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 1 -- 172.21.15.116:0/1676443776 wait complete. 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.357+0000 7f9a05dcf740 20 client.20961 trim_cache size 0 max 16384 2024-03-19T02:46:48.363 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:48.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 msgr2=0x7f99ec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 0x7f99ec0a4eb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f99f4012fd0 tx=0x7f99f4021000 comp rx=0 tx=0).stop 2024-03-19T02:46:48.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 msgr2=0x7f99fc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f99f0037040 tx=0x7f99f0035f70 comp rx=0 tx=0).stop 2024-03-19T02:46:48.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 shutdown_connections 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99ec0a29f0 0x7f99ec0a4eb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99fc10bd80 0x7f99fc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99fc103360 0x7f99fc1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 --2- 172.21.15.116:0/1761540562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99fc102a40 0x7f99fc197f70 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 >> 172.21.15.116:0/1761540562 conn(0x7f99fc0778d0 msgr2=0x7f99fc0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 shutdown_connections 2024-03-19T02:46:48.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.358+0000 7f9a031e6640 1 -- 172.21.15.116:0/1761540562 wait complete. 2024-03-19T02:46:48.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:48 smithi116 ceph-mon[32263]: pgmap v598: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:48.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:48 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1457077463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:48.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:46:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:48 smithi162 ceph-mon[35761]: pgmap v598: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:48 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1457077463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:48 smithi154 ceph-mon[35693]: pgmap v598: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:48 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1457077463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 -- 172.21.15.116:0/2349421886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 msgr2=0x7f56fc10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 --2- 172.21.15.116:0/2349421886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc10e220 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009ca0 tx=0x7f56f402c180 comp rx=0 tx=0).stop 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 -- 172.21.15.116:0/2349421886 shutdown_connections 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 --2- 172.21.15.116:0/2349421886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc10e220 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 --2- 172.21.15.116:0/2349421886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc103360 0x7f56fc10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 --2- 172.21.15.116:0/2349421886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc102a40 0x7f56fc102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.505+0000 7f5702fd4640 1 -- 172.21.15.116:0/2349421886 >> 172.21.15.116:0/2349421886 conn(0x7f56fc0778d0 msgr2=0x7f56fc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- 172.21.15.116:0/2349421886 shutdown_connections 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- 172.21.15.116:0/2349421886 wait complete. 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 Processor -- start 2024-03-19T02:46:48.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- start start 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc102a40 0x7f56fc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc103360 0x7f56fc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc111a30 con 0x7f56fc10bd80 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1118b0 con 0x7f56fc102a40 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5702fd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc111bb0 con 0x7f56fc103360 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5701571640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5701571640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45286/0 (socket says 172.21.15.116:45286) 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc103360 0x7f56fc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5701571640 1 -- 172.21.15.116:0/3359991824 learned_addr learned my addr 172.21.15.116:0/3359991824 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.506+0000 7f5700d70640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc102a40 0x7f56fc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 -- 172.21.15.116:0/3359991824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc103360 msgr2=0x7f56fc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc103360 0x7f56fc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 -- 172.21.15.116:0/3359991824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc102a40 msgr2=0x7f56fc197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc102a40 0x7f56fc197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 -- 172.21.15.116:0/3359991824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc19cf00 con 0x7f56fc10bd80 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5701571640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f56f402c790 tx=0x7f56f4035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f56f1ffb640 1 -- 172.21.15.116:0/3359991824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4039040 con 0x7f56fc10bd80 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc19d190 con 0x7f56fc10bd80 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f56f1ffb640 1 -- 172.21.15.116:0/3359991824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56f4044070 con 0x7f56fc10bd80 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f56f1ffb640 1 -- 172.21.15.116:0/3359991824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4032850 con 0x7f56fc10bd80 2024-03-19T02:46:48.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.507+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc1a4a80 con 0x7f56fc10bd80 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f56f1ffb640 1 -- 172.21.15.116:0/3359991824 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f56f40329f0 con 0x7f56fc10bd80 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 Processor -- start 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 -- start start 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d67bf0 0x55f545d70290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d707d0 0x55f545d72bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f545cca4b0 con 0x55f545d67bf0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f545c72d70 con 0x55f545d707d0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f5705bc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f545cc6550 con 0x55f545d672d0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f56f1ffb640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 0x7f56e40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.510+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47996/0 (socket says 172.21.15.116:47996) 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f5700d70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d67bf0 0x55f545d70290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 -- 172.21.15.116:0/2554734312 learned_addr learned my addr 172.21.15.116:0/2554734312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d707d0 0x55f545d72bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f1ffb640 1 -- 172.21.15.116:0/3359991824 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f56f40f14b0 con 0x7f56fc10bd80 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 -- 172.21.15.116:0/2554734312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d707d0 msgr2=0x55f545d72bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d707d0 0x55f545d72bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 -- 172.21.15.116:0/2554734312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d67bf0 msgr2=0x55f545d70290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d67bf0 0x55f545d70290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f5701571640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 0x7f56e40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 -- 172.21.15.116:0/2554734312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f545c95210 con 0x55f545d672d0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56f3fff640 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f56ec011780 tx=0x7f56ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3eae2ec94075331 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f5701571640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 0x7f56e40a4e30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f56f40bd540 tx=0x7f56f40095c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56e2ffd640 1 -- 172.21.15.116:0/2554734312 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004050 con 0x55f545d672d0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56e2ffd640 1 -- 172.21.15.116:0/2554734312 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56ec0054d0 con 0x55f545d672d0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.511+0000 7f56e2ffd640 1 -- 172.21.15.116:0/2554734312 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004bf0 con 0x55f545d672d0 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- 172.21.15.116:0/2554734312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 msgr2=0x55f545d676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f56ec011780 tx=0x7f56ec004730 comp rx=0 tx=0).stop 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- 172.21.15.116:0/2554734312 shutdown_connections 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d707d0 0x55f545d72bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d67bf0 0x55f545d70290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 --2- 172.21.15.116:0/2554734312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d672d0 0x55f545d676b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- 172.21.15.116:0/2554734312 >> 172.21.15.116:0/2554734312 conn(0x55f545d61a00 msgr2=0x55f545d61de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- 172.21.15.116:0/2554734312 shutdown_connections 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- 172.21.15.116:0/2554734312 wait complete. 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.512+0000 7f5705bc3740 1 -- start start 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 Processor -- start 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 0x55f545d6dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d707d0 0x55f545d6e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f545d735c0 con 0x55f545d672d0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f545c86310 con 0x55f545d67bf0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f545d73740 con 0x55f545d707d0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5705bc3740 10 client.0 _ll_register_callbacks cb 0x55f545c7a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45304/0 (socket says 172.21.15.116:45304) 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f5700d70640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 0x55f545d6dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f56f3fff640 1 -- 172.21.15.116:0/3596284727 learned_addr learned my addr 172.21.15.116:0/3596284727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.513+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d707d0 0x55f545d6e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 -- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d707d0 msgr2=0x55f545d6e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d707d0 0x55f545d6e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 -- 172.21.15.116:0/3596284727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 msgr2=0x55f545d6dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 0x55f545d6dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 -- 172.21.15.116:0/3596284727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5459de3f0 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f5700d70640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 0x55f545d6dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009bf0 tx=0x7f56ec0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac64957d4dbb627 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004e20 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f56ec00f7c0 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56ec0054d0 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f56ec035050 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f5705bc3740 10 client.20973 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f545aa87d0 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.514+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56ec0054d0 con 0x55f545d672d0 2024-03-19T02:46:48.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f56ec01d020 con 0x55f545d672d0 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 10 client.20973 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 10 client.20973 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 20 client.20973 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545d84e20 0x55f545d87210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 4 client.20973 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f545d283e0 con 0x55f545d84e20 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5700d70640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545d84e20 0x55f545d87210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f56dbfff640 10 client.20973 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:48.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.515+0000 7f5700d70640 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545d84e20 0x55f545d87210 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b5bbf92db900465 server_cookie=ce862ebfdcb927d5 in_seq=0 out_seq=0 2024-03-19T02:46:48.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f56dbfff640 1 -- 172.21.15.116:0/3596284727 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f545d283e0 con 0x55f545d84e20 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f56dbfff640 10 client.20973 handle_command_reply: tid=0 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 client.20973 shutdown 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 msgr2=0x55f545d6d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009bf0 tx=0x7f56ec0048b0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 shutdown_connections 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545d84e20 0x55f545d87210 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d707d0 0x55f545d6e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d67bf0 0x55f545d6dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f56dbfff640 0 client.20973 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3596284727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d672d0 0x55f545d6d830 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 >> 172.21.15.116:0/3596284727 conn(0x55f545d61a00 msgr2=0x55f545d63750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 shutdown_connections 2024-03-19T02:46:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 1 -- 172.21.15.116:0/3596284727 wait complete. 2024-03-19T02:46:48.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.517+0000 7f5705bc3740 20 client.20973 trim_cache size 0 max 16384 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 Processor -- start 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 -- start start 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d7c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d6e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1aca30 con 0x55f545d707d0 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1ae570 con 0x55f545d672d0 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5705bc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1ad100 con 0x55f545d67bf0 2024-03-19T02:46:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45306/0 (socket says 172.21.15.116:45306) 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 -- 172.21.15.116:0/3399673095 learned_addr learned my addr 172.21.15.116:0/3399673095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5700d70640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d6e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d7c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 -- 172.21.15.116:0/3399673095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 msgr2=0x55f545d6e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d6e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 -- 172.21.15.116:0/3399673095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 msgr2=0x55f545d7c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d7c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f56f3fff640 1 -- 172.21.15.116:0/3399673095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f545d5bfb0 con 0x55f545d707d0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.536+0000 7f5700d70640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d6e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.537+0000 7f56f3fff640 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009af0 tx=0x7f56ec025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b424f00ef987d74f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.537+0000 7f56e27fc640 1 -- 172.21.15.116:0/3399673095 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec01fbb0 con 0x55f545d707d0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.537+0000 7f56e27fc640 1 -- 172.21.15.116:0/3399673095 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56ec004ca0 con 0x55f545d707d0 2024-03-19T02:46:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.537+0000 7f56e27fc640 1 -- 172.21.15.116:0/3399673095 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec0054d0 con 0x55f545d707d0 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- 172.21.15.116:0/3399673095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 msgr2=0x55f545d60a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009af0 tx=0x7f56ec025040 comp rx=0 tx=0).stop 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- 172.21.15.116:0/3399673095 shutdown_connections 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545d60a20 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d6e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 --2- 172.21.15.116:0/3399673095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d7c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- 172.21.15.116:0/3399673095 >> 172.21.15.116:0/3399673095 conn(0x55f545d61a00 msgr2=0x55f545d60310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- 172.21.15.116:0/3399673095 shutdown_connections 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- 172.21.15.116:0/3399673095 wait complete. 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 -- start start 2024-03-19T02:46:48.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.538+0000 7f5705bc3740 1 Processor -- start 2024-03-19T02:46:48.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d73b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545de9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1adea0 con 0x55f545d707d0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1ad7d0 con 0x55f545d672d0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1aaa50 con 0x55f545d67bf0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 10 client.0 _ll_register_callbacks cb 0x7f56f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545de9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d73b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48018/0 (socket says 172.21.15.116:48018) 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f56f3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d73b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39978/0 (socket says 172.21.15.116:39978) 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 -- 172.21.15.116:0/4233078919 learned_addr learned my addr 172.21.15.116:0/4233078919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 -- 172.21.15.116:0/4233078919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 msgr2=0x55f545d73b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d73b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 -- 172.21.15.116:0/4233078919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 msgr2=0x55f545de9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545de9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 -- 172.21.15.116:0/4233078919 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f545de95d0 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5700d70640 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005d00 tx=0x7f56f8012ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93ea9a69cf15113 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f56dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8020510 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.539+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56ec004ca0 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f56f80206b0 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f56f8031050 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 10 client.30682 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f545dea8c0 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56ec004ca0 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f56f8028420 con 0x55f545d67bf0 2024-03-19T02:46:48.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 10 client.30682 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 10 client.30682 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 20 client.30682 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545deb020 0x55f545ded440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 4 client.30682 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f545cd79f0 con 0x55f545deb020 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.540+0000 7f56f3fff640 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545deb020 0x55f545ded440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f56dbfff640 10 client.30682 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f56f3fff640 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545deb020 0x55f545ded440 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac2663e4e839ccf8 server_cookie=6209fec87445aa9e in_seq=0 out_seq=0 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f56dbfff640 1 -- 172.21.15.116:0/4233078919 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f545cd79f0 con 0x55f545deb020 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f56dbfff640 10 client.30682 handle_command_reply: tid=0 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 client.30682 shutdown 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 msgr2=0x55f545d74080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005d00 tx=0x7f56f8012ae0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 shutdown_connections 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55f545deb020 0x55f545ded440 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f56dbfff640 0 client.30682 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f545d707d0 0x55f545de9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f545d67bf0 0x55f545d74080 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 --2- 172.21.15.116:0/4233078919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f545d672d0 0x55f545d73b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 >> 172.21.15.116:0/4233078919 conn(0x55f545d61a00 msgr2=0x55f545d60ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 shutdown_connections 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 1 -- 172.21.15.116:0/4233078919 wait complete. 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.541+0000 7f5705bc3740 20 client.30682 trim_cache size 0 max 16384 2024-03-19T02:46:48.549 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 msgr2=0x7f56e40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 0x7f56e40a4e30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f56f40bd540 tx=0x7f56f40095c0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 msgr2=0x7f56fc19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f56f402c790 tx=0x7f56f4035db0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 shutdown_connections 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56e40a2970 0x7f56e40a4e30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56fc10bd80 0x7f56fc19c7d0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56fc103360 0x7f56fc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 --2- 172.21.15.116:0/3359991824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56fc102a40 0x7f56fc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 >> 172.21.15.116:0/3359991824 conn(0x7f56fc0778d0 msgr2=0x7f56fc0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 shutdown_connections 2024-03-19T02:46:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.542+0000 7f5702fd4640 1 -- 172.21.15.116:0/3359991824 wait complete. 2024-03-19T02:46:48.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:46:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 -- 172.21.15.116:0/413745707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd980100610 msgr2=0x7fd9801009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 --2- 172.21.15.116:0/413745707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd980100610 0x7fd9801009f0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009900 tx=0x7fd97c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 -- 172.21.15.116:0/413745707 shutdown_connections 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 --2- 172.21.15.116:0/413745707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd98010e1c0 0x7fd9801105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 --2- 172.21.15.116:0/413745707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 --2- 172.21.15.116:0/413745707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd980100610 0x7fd9801009f0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 -- 172.21.15.116:0/413745707 >> 172.21.15.116:0/413745707 conn(0x7fd980077830 msgr2=0x7fd980077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 -- 172.21.15.116:0/413745707 shutdown_connections 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.700+0000 7fd9882d3640 1 -- 172.21.15.116:0/413745707 wait complete. 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 Processor -- start 2024-03-19T02:46:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 -- start start 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd980100610 0x7fd98019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd980113cf0 con 0x7fd98010e1c0 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd980113b70 con 0x7fd980100610 2024-03-19T02:46:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd9882d3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd980113e70 con 0x7fd980100f30 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd986870640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd986870640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45326/0 (socket says 172.21.15.116:45326) 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd986870640 1 -- 172.21.15.116:0/3626012385 learned_addr learned my addr 172.21.15.116:0/3626012385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd98586e640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.701+0000 7fd98606f640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd980100610 0x7fd98019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 -- 172.21.15.116:0/3626012385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 msgr2=0x7fd98019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 -- 172.21.15.116:0/3626012385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd980100610 msgr2=0x7fd98019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd980100610 0x7fd98019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 -- 172.21.15.116:0/3626012385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98019f2a0 con 0x7fd98010e1c0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd98586e640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd986870640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fd97800e9d0 tx=0x7fd97800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd9777fe640 1 -- 172.21.15.116:0/3626012385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97800cd80 con 0x7fd98010e1c0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98019f590 con 0x7fd98010e1c0 2024-03-19T02:46:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd9777fe640 1 -- 172.21.15.116:0/3626012385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd978005020 con 0x7fd98010e1c0 2024-03-19T02:46:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd9777fe640 1 -- 172.21.15.116:0/3626012385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9780116d0 con 0x7fd98010e1c0 2024-03-19T02:46:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.702+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9801a6e20 con 0x7fd98010e1c0 2024-03-19T02:46:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.705+0000 7fd9777fe640 1 -- 172.21.15.116:0/3626012385 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd9780051d0 con 0x7fd98010e1c0 2024-03-19T02:46:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.705+0000 7fd9777fe640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 0x7fd96c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.706+0000 7fd9777fe640 1 -- 172.21.15.116:0/3626012385 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fd9780cf2d0 con 0x7fd98010e1c0 2024-03-19T02:46:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.706+0000 7fd98606f640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 0x7fd96c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.706+0000 7fd98af95740 1 Processor -- start 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.706+0000 7fd98af95740 1 -- start start 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98606f640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 0x7fd96c0a4eb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd97c005ab0 tx=0x7fd97c005a20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f08a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f095bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55659efed4b0 con 0x55659f08abf0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55659ef95d70 con 0x55659f0937d0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98af95740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55659efe9550 con 0x55659f08a2d0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98606f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f08a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45342/0 (socket says 172.21.15.116:45342) 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 -- 172.21.15.116:0/2763412794 learned_addr learned my addr 172.21.15.116:0/2763412794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd986870640 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f095bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 -- 172.21.15.116:0/2763412794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 msgr2=0x55659f08a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f08a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 -- 172.21.15.116:0/2763412794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 msgr2=0x55659f095bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f095bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 -- 172.21.15.116:0/2763412794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55659efb8210 con 0x55659f08abf0 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd986870640 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f095bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.707+0000 7fd98586e640 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd9700114f0 tx=0x7fd9700119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97b84887289881e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.708+0000 7fd974bfe640 1 -- 172.21.15.116:0/2763412794 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97001e070 con 0x55659f08abf0 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.708+0000 7fd974bfe640 1 -- 172.21.15.116:0/2763412794 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9700035f0 con 0x55659f08abf0 2024-03-19T02:46:48.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.708+0000 7fd974bfe640 1 -- 172.21.15.116:0/2763412794 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97000f820 con 0x55659f08abf0 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- 172.21.15.116:0/2763412794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 msgr2=0x55659f093290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd9700114f0 tx=0x7fd9700119b0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- 172.21.15.116:0/2763412794 shutdown_connections 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f095bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f093290 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- 172.21.15.116:0/2763412794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f08a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- 172.21.15.116:0/2763412794 >> 172.21.15.116:0/2763412794 conn(0x55659f084a00 msgr2=0x55659f084de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- 172.21.15.116:0/2763412794 shutdown_connections 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- 172.21.15.116:0/2763412794 wait complete. 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 -- start start 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 Processor -- start 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f090850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f090d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.709+0000 7fd98af95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 0x55659f0912d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98af95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55659f0965c0 con 0x55659f0937d0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98af95740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55659efa9310 con 0x55659f08abf0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98af95740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55659f096740 con 0x55659f08a2d0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98af95740 10 client.0 _ll_register_callbacks cb 0x55659ef9d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98586e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f090d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 0x55659f0912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98606f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f090850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98586e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f090d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40016/0 (socket says 172.21.15.116:40016) 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98586e640 1 -- 172.21.15.116:0/2746652486 learned_addr learned my addr 172.21.15.116:0/2746652486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 -- 172.21.15.116:0/2746652486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 msgr2=0x55659f090850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f090850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 -- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 msgr2=0x55659f090d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f090d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 -- 172.21.15.116:0/2746652486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55659ed013f0 con 0x55659f0937d0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd986870640 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 0x55659f0912d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd9780cc640 tx=0x7fd97809b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae51bc52b7016758 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd969ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9780b4060 con 0x55659f0937d0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fd9780b4200 con 0x55659f0937d0 2024-03-19T02:46:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.710+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9700035f0 con 0x55659f0937d0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd97801f040 con 0x55659f0937d0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 10 client.20991 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55659edcb7d0 con 0x55659f0937d0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9780b4400 con 0x55659f0937d0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd97800f040 con 0x55659f0937d0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 10 client.20991 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 10 client.20991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 20 client.20991 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f0ac360 0x55659f0ae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 4 client.20991 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd9700035f0 con 0x55659f0ac360 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.711+0000 7fd98606f640 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f0ac360 0x55659f0ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.712+0000 7fd969ffb640 10 client.20991 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.712+0000 7fd98606f640 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f0ac360 0x55659f0ae750 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15f0cc1110080812 server_cookie=c489a80840ca0466 in_seq=0 out_seq=0 2024-03-19T02:46:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.713+0000 7fd969ffb640 1 -- 172.21.15.116:0/2746652486 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd9700035f0 con 0x55659f0ac360 2024-03-19T02:46:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.713+0000 7fd969ffb640 10 client.20991 handle_command_reply: tid=0 2024-03-19T02:46:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 client.20991 shutdown 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 msgr2=0x55659f0912d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 0x55659f0912d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd9780cc640 tx=0x7fd97809b050 comp rx=0 tx=0).stop 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 shutdown_connections 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f0ac360 0x55659f0ae750 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f0937d0 0x55659f0912d0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f090d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 --2- 172.21.15.116:0/2746652486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f090850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 >> 172.21.15.116:0/2746652486 conn(0x55659f084a00 msgr2=0x55659f086750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd969ffb640 0 client.20991 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 shutdown_connections 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 1 -- 172.21.15.116:0/2746652486 wait complete. 2024-03-19T02:46:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.714+0000 7fd98af95740 20 client.20991 trim_cache size 0 max 16384 2024-03-19T02:46:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 Processor -- start 2024-03-19T02:46:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 -- start start 2024-03-19T02:46:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f092f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801aca90 con 0x55659f08abf0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801ae5d0 con 0x55659f0937d0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98af95740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801ad160 con 0x55659f08a2d0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45360/0 (socket says 172.21.15.116:45360) 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 -- 172.21.15.116:0/709014174 learned_addr learned my addr 172.21.15.116:0/709014174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98606f640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f092f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd986870640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 -- 172.21.15.116:0/709014174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 msgr2=0x55659f092f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f092f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 -- 172.21.15.116:0/709014174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 msgr2=0x55659f0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98586e640 1 -- 172.21.15.116:0/709014174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55659f084830 con 0x55659f08abf0 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.731+0000 7fd98606f640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f092f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.732+0000 7fd98586e640 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd970005b90 tx=0x7fd970004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b2bb125167a374 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.732+0000 7fd96bfff640 1 -- 172.21.15.116:0/709014174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970028070 con 0x55659f08abf0 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.732+0000 7fd96bfff640 1 -- 172.21.15.116:0/709014174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97001f040 con 0x55659f08abf0 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.732+0000 7fd96bfff640 1 -- 172.21.15.116:0/709014174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97001e040 con 0x55659f08abf0 2024-03-19T02:46:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- 172.21.15.116:0/709014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 msgr2=0x55659f0a7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd970005b90 tx=0x7fd970004f80 comp rx=0 tx=0).stop 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- 172.21.15.116:0/709014174 shutdown_connections 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f0937d0 0x55659f0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08abf0 0x55659f0a7bf0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- 172.21.15.116:0/709014174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f08a2d0 0x55659f092f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- 172.21.15.116:0/709014174 >> 172.21.15.116:0/709014174 conn(0x55659f084a00 msgr2=0x55659f083ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- 172.21.15.116:0/709014174 shutdown_connections 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- 172.21.15.116:0/709014174 wait complete. 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- start start 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 Processor -- start 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f0a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 0x55659f0a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801adf00 con 0x55659f08a2d0 2024-03-19T02:46:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801ad830 con 0x55659f08abf0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801aaa90 con 0x55659f0937d0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98af95740 10 client.0 _ll_register_callbacks cb 0x7fd9780860f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98606f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98606f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45362/0 (socket says 172.21.15.116:45362) 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd98606f640 1 -- 172.21.15.116:0/111745424 learned_addr learned my addr 172.21.15.116:0/111745424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.733+0000 7fd986870640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 0x55659f0a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98586e640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f0a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 -- 172.21.15.116:0/111745424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 msgr2=0x55659f0a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 0x55659f0a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 -- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 msgr2=0x55659f0a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f0a00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 -- 172.21.15.116:0/111745424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55659f095bc0 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd986870640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 0x55659f0a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98606f640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fd97c004190 tx=0x7fd97c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8e9c0a9114311b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd969ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c056de0 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fd97c0563f0 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd97001f040 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd97c065050 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98af95740 10 client.20997 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.734+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55659f1127c0 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97001f040 con 0x55659f08a2d0 2024-03-19T02:46:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fd97c004920 con 0x55659f08a2d0 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 10 client.20997 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 10 client.20997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 20 client.20997 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f112f10 0x55659f115330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 4 client.20997 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55659f04b3e0 con 0x55659f112f10 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98586e640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f112f10 0x55659f115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd969ffb640 10 client.20997 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:48.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.735+0000 7fd98586e640 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f112f10 0x55659f115330 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d047ca18f1400c8 server_cookie=ab64733ef2054a35 in_seq=0 out_seq=0 2024-03-19T02:46:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd969ffb640 1 -- 172.21.15.116:0/111745424 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55659f04b3e0 con 0x55659f112f10 2024-03-19T02:46:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd969ffb640 10 client.20997 handle_command_reply: tid=0 2024-03-19T02:46:48.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 client.20997 shutdown 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 msgr2=0x55659f09fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fd97c004190 tx=0x7fd97c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 shutdown_connections 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55659f112f10 0x55659f115330 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55659f0937d0 0x55659f0a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55659f08abf0 0x55659f0a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd969ffb640 0 client.20997 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.737+0000 7fd98af95740 1 --2- 172.21.15.116:0/111745424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55659f08a2d0 0x55659f09fb60 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 >> 172.21.15.116:0/111745424 conn(0x55659f084a00 msgr2=0x55659f083ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 shutdown_connections 2024-03-19T02:46:48.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd98af95740 1 -- 172.21.15.116:0/111745424 wait complete. 2024-03-19T02:46:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd98af95740 20 client.20997 trim_cache size 0 max 16384 2024-03-19T02:46:48.746 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:46:48.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 msgr2=0x7fd96c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 0x7fd96c0a4eb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd97c005ab0 tx=0x7fd97c005a20 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 msgr2=0x7fd98019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fd97800e9d0 tx=0x7fd97800ee90 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 shutdown_connections 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd96c0a29f0 0x7fd96c0a4eb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98010e1c0 0x7fd98019eb70 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd980100f30 0x7fd98019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 --2- 172.21.15.116:0/3626012385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd980100610 0x7fd98019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 >> 172.21.15.116:0/3626012385 conn(0x7fd980077830 msgr2=0x7fd98010f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 shutdown_connections 2024-03-19T02:46:48.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:48.738+0000 7fd9882d3640 1 -- 172.21.15.116:0/3626012385 wait complete. 2024-03-19T02:46:49.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42030 6.01 MB/sec execute 251 sec latency 145.115 ms 2024-03-19T02:46:49.753 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:46:49.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:46:49.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 -- 172.21.15.116:0/3376405073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109cd0 msgr2=0x7fcef810d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 --2- 172.21.15.116:0/3376405073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109cd0 0x7fcef810d270 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009900 tx=0x7fcee802bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 -- 172.21.15.116:0/3376405073 shutdown_connections 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 --2- 172.21.15.116:0/3376405073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef810d950 0x7fcef8115520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 --2- 172.21.15.116:0/3376405073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109cd0 0x7fcef810d270 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 --2- 172.21.15.116:0/3376405073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef8109320 0x7fcef8109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 -- 172.21.15.116:0/3376405073 >> 172.21.15.116:0/3376405073 conn(0x7fcef806cd60 msgr2=0x7fcef806d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 -- 172.21.15.116:0/3376405073 shutdown_connections 2024-03-19T02:46:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.893+0000 7fcf00768640 1 -- 172.21.15.116:0/3376405073 wait complete. 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 Processor -- start 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 -- start start 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109320 0x7fcef81a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef810d950 0x7fcef81a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8117fe0 con 0x7fcef81a28c0 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8117e60 con 0x7fcef810d950 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.894+0000 7fcf00768640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8118160 con 0x7fcef8109320 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45380/0 (socket says 172.21.15.116:45380) 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 -- 172.21.15.116:0/2148057263 learned_addr learned my addr 172.21.15.116:0/2148057263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefdd03640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef810d950 0x7fcef81a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefe504640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109320 0x7fcef81a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 -- 172.21.15.116:0/2148057263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109320 msgr2=0x7fcef81a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109320 0x7fcef81a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 -- 172.21.15.116:0/2148057263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef810d950 msgr2=0x7fcef81a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef810d950 0x7fcef81a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 -- 172.21.15.116:0/2148057263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef81a33e0 con 0x7fcef81a28c0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.895+0000 7fcefed05640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fcef000eb10 tx=0x7fcef000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.896+0000 7fceef7fe640 1 -- 172.21.15.116:0/2148057263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0004b20 con 0x7fcef81a28c0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.896+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef81a36d0 con 0x7fcef81a28c0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.896+0000 7fceef7fe640 1 -- 172.21.15.116:0/2148057263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcef0004cc0 con 0x7fcef81a28c0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.896+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef81af8d0 con 0x7fcef81a28c0 2024-03-19T02:46:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.896+0000 7fceef7fe640 1 -- 172.21.15.116:0/2148057263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0010690 con 0x7fcef81a28c0 2024-03-19T02:46:49.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.899+0000 7fceef7fe640 1 -- 172.21.15.116:0/2148057263 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcef0005020 con 0x7fcef81a28c0 2024-03-19T02:46:49.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.900+0000 7fceef7fe640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 0x7fcee40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.900+0000 7fcf0342a740 1 Processor -- start 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.900+0000 7fcefe504640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 0x7fcee40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.900+0000 7fceef7fe640 1 -- 172.21.15.116:0/2148057263 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fcef00cf700 con 0x7fcef81a28c0 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.900+0000 7fcf0342a740 1 -- start start 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e20340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e22c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dff0d7b160 con 0x55dff0e17380 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dff0d23e10 con 0x55dff0e20880 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcf0342a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dff0d77200 con 0x55dff0e17ca0 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45386/0 (socket says 172.21.15.116:45386) 2024-03-19T02:46:49.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 -- 172.21.15.116:0/1046558141 learned_addr learned my addr 172.21.15.116:0/1046558141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:49.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 -- 172.21.15.116:0/1046558141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 msgr2=0x55dff0e20340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e20340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 -- 172.21.15.116:0/1046558141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 msgr2=0x55dff0e22c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e22c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.901+0000 7fcefe504640 1 -- 172.21.15.116:0/1046558141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dff0b577e0 con 0x55dff0e17380 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.902+0000 7fcefe504640 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fcef40099f0 tx=0x7fcef4009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7516d92e8cb6e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.902+0000 7fcefe504640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 0x7fcee40a4eb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcef401b6c0 tx=0x7fcef4015000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.902+0000 7fceecff9640 1 -- 172.21.15.116:0/1046558141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4022070 con 0x55dff0e17380 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.902+0000 7fceecff9640 1 -- 172.21.15.116:0/1046558141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcef401e470 con 0x55dff0e17380 2024-03-19T02:46:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.902+0000 7fceecff9640 1 -- 172.21.15.116:0/1046558141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4026640 con 0x55dff0e17380 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- 172.21.15.116:0/1046558141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 msgr2=0x55dff0e17760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fcef40099f0 tx=0x7fcef4009eb0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- 172.21.15.116:0/1046558141 shutdown_connections 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e22c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e20340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1046558141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e17760 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- 172.21.15.116:0/1046558141 >> 172.21.15.116:0/1046558141 conn(0x55dff0e11ab0 msgr2=0x55dff0e11e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- 172.21.15.116:0/1046558141 shutdown_connections 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- 172.21.15.116:0/1046558141 wait complete. 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.904+0000 7fcf0342a740 1 -- start start 2024-03-19T02:46:49.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 Processor -- start 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e1deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e1e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45388/0 (socket says 172.21.15.116:45388) 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dff0e235b0 con 0x55dff0e17380 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcefe504640 1 -- 172.21.15.116:0/1225213931 learned_addr learned my addr 172.21.15.116:0/1225213931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.905+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dff0d371a0 con 0x55dff0e20880 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dff0e23730 con 0x55dff0e17ca0 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefed05640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e1e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e1deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcf0342a740 10 client.0 _ll_register_callbacks cb 0x55dff0a8ec60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 -- 172.21.15.116:0/1225213931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 msgr2=0x55dff0e1deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e1deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 -- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 msgr2=0x55dff0e1e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e1e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dff0bb7f70 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef401e470 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcefe504640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fcef4028040 tx=0x7fcef401d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259f3e5c5cf1c47b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.906+0000 7fcee1ffb640 10 client.21009 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1225213931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4022040 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1225213931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fcef400dc10 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1225213931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcef40095e0 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcf0342a740 10 client.21009 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=38}) -- 0x55dff0a4f170 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.907+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1225213931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fcef4019050 con 0x55dff0e17380 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 10 client.21009 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 10 client.21009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:49.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 20 client.21009 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:49.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e309f0 0x55dff0e32de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 4 client.21009 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:49.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dff0d886a0 con 0x55dff0e309f0 2024-03-19T02:46:49.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e309f0 0x55dff0e32de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcee1ffb640 10 client.21009 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.908+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e309f0 0x55dff0e32de0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15e42944151211c3 server_cookie=4e8a5c6b814b10bc in_seq=0 out_seq=0 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.910+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1225213931 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dff0d886a0 con 0x55dff0e309f0 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.910+0000 7fcee1ffb640 10 client.21009 handle_command_reply: tid=0 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 client.21009 shutdown 2024-03-19T02:46:49.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 msgr2=0x55dff0e1d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fcef4028040 tx=0x7fcef401d9d0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 shutdown_connections 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e309f0 0x55dff0e32de0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e1e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17ca0 0x55dff0e1deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1225213931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1d970 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 >> 172.21.15.116:0/1225213931 conn(0x55dff0e11ab0 msgr2=0x55dff0e13800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcee1ffb640 0 client.21009 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 shutdown_connections 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 1 -- 172.21.15.116:0/1225213931 wait complete. 2024-03-19T02:46:49.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.911+0000 7fcf0342a740 20 client.21009 trim_cache size 0 max 16384 2024-03-19T02:46:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 Processor -- start 2024-03-19T02:46:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 -- start start 2024-03-19T02:46:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17380 0x55dff0e2c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e38380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81b95e0 con 0x55dff0e17ca0 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81bb120 con 0x55dff0e20880 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcf0342a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81b9cb0 con 0x55dff0e17380 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45394/0 (socket says 172.21.15.116:45394) 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 -- 172.21.15.116:0/3000753017 learned_addr learned my addr 172.21.15.116:0/3000753017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefe504640 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17380 0x55dff0e2c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefed05640 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e38380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 -- 172.21.15.116:0/3000753017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17380 msgr2=0x55dff0e2c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17380 0x55dff0e2c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 -- 172.21.15.116:0/3000753017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 msgr2=0x55dff0e38380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e38380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.940+0000 7fcefdd03640 1 -- 172.21.15.116:0/3000753017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dff0e22fe0 con 0x55dff0e17ca0 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.941+0000 7fcefdd03640 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009230 tx=0x7fcee8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcaa7a2e45f4d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.941+0000 7fcee3fff640 1 -- 172.21.15.116:0/3000753017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8004160 con 0x55dff0e17ca0 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.941+0000 7fcee3fff640 1 -- 172.21.15.116:0/3000753017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcee802cc90 con 0x55dff0e17ca0 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.941+0000 7fcee3fff640 1 -- 172.21.15.116:0/3000753017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8005510 con 0x55dff0e17ca0 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- 172.21.15.116:0/3000753017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 msgr2=0x55dff0e1e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009230 tx=0x7fcee8005c00 comp rx=0 tx=0).stop 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- 172.21.15.116:0/3000753017 shutdown_connections 2024-03-19T02:46:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e20880 0x55dff0e38380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17ca0 0x55dff0e1e210 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- 172.21.15.116:0/3000753017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e17380 0x55dff0e2c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- 172.21.15.116:0/3000753017 >> 172.21.15.116:0/3000753017 conn(0x55dff0e11ab0 msgr2=0x55dff0e13790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- 172.21.15.116:0/3000753017 shutdown_connections 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- 172.21.15.116:0/3000753017 wait complete. 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- start start 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 Processor -- start 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e17ca0 0x55dff0e1ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e20880 0x55dff0e1b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81baa50 con 0x55dff0e17380 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81ba380 con 0x55dff0e17ca0 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcef81b75e0 con 0x55dff0e20880 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcf0342a740 10 client.0 _ll_register_callbacks cb 0x7fcef4009820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.942+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45402/0 (socket says 172.21.15.116:45402) 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 -- 172.21.15.116:0/1941160271 learned_addr learned my addr 172.21.15.116:0/1941160271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e17ca0 0x55dff0e1ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefed05640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e20880 0x55dff0e1b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 -- 172.21.15.116:0/1941160271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e20880 msgr2=0x55dff0e1b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e20880 0x55dff0e1b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 -- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e17ca0 msgr2=0x55dff0e1ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e17ca0 0x55dff0e1ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 -- 172.21.15.116:0/1941160271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dff0e0d160 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcefe504640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fcef400dfd0 tx=0x7fcef402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e84dfa5b0dbad7a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef403d060 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7fcef401dbb0 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcee802cc90 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcef400d7c0 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcf0342a740 10 client.21015 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55dff0e96b90 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.943+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcee802cc90 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fcef4019040 con 0x55dff0e17380 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 10 client.21015 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 10 client.21015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 20 client.21015 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e97280 0x55dff0e996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 4 client.21015 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dff0d829f0 con 0x55dff0e97280 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e97280 0x55dff0e996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcee1ffb640 10 client.21015 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.944+0000 7fcefdd03640 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e97280 0x55dff0e996a0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe61c759114a4371 server_cookie=e700e3fc3aef99ee in_seq=0 out_seq=0 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcee1ffb640 1 -- 172.21.15.116:0/1941160271 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dff0d829f0 con 0x55dff0e97280 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcee1ffb640 10 client.21015 handle_command_reply: tid=0 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 client.21015 shutdown 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 msgr2=0x55dff0e1a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fcef400dfd0 tx=0x7fcef402d000 comp rx=0 tx=0).stop 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 shutdown_connections 2024-03-19T02:46:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55dff0e97280 0x55dff0e996a0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dff0e20880 0x55dff0e1b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcee1ffb640 0 client.21015 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dff0e17ca0 0x55dff0e1ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 --2- 172.21.15.116:0/1941160271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dff0e17380 0x55dff0e1a600 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 >> 172.21.15.116:0/1941160271 conn(0x55dff0e11ab0 msgr2=0x55dff0e12f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 shutdown_connections 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 1 -- 172.21.15.116:0/1941160271 wait complete. 2024-03-19T02:46:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.947+0000 7fcf0342a740 20 client.21015 trim_cache size 0 max 16384 2024-03-19T02:46:49.955 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:49.955 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:46:49.955 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "e214d867-bfc7-4110-be10-a52728b9b175", 2024-03-19T02:46:49.955 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:46:49.955 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 msgr2=0x7fcee40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 0x7fcee40a4eb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcef401b6c0 tx=0x7fcef4015000 comp rx=0 tx=0).stop 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 msgr2=0x7fcef81a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fcef000eb10 tx=0x7fcef000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 shutdown_connections 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcee40a29f0 0x7fcee40a4eb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef81a28c0 0x7fcef81a2d70 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef810d950 0x7fcef81a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 --2- 172.21.15.116:0/2148057263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcef8109320 0x7fcef81a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 >> 172.21.15.116:0/2148057263 conn(0x7fcef806cd60 msgr2=0x7fcef8071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 shutdown_connections 2024-03-19T02:46:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:49.948+0000 7fcf00768640 1 -- 172.21.15.116:0/2148057263 wait complete. 2024-03-19T02:46:49.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:46:49.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:46:50.051 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:50 smithi116 ceph-mon[32263]: pgmap v599: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:50.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 -- 172.21.15.116:0/2914577243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17081066a0 msgr2=0x7f1708106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 --2- 172.21.15.116:0/2914577243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17081066a0 0x7f1708106a80 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f1704009930 tx=0x7f170402bc30 comp rx=0 tx=0).stop 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 -- 172.21.15.116:0/2914577243 shutdown_connections 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 --2- 172.21.15.116:0/2914577243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708101050 0x7f170810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 --2- 172.21.15.116:0/2914577243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1708100690 0x7f1708100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 --2- 172.21.15.116:0/2914577243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17081066a0 0x7f1708106a80 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 -- 172.21.15.116:0/2914577243 >> 172.21.15.116:0/2914577243 conn(0x7f1708077820 msgr2=0x7f1708077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 -- 172.21.15.116:0/2914577243 shutdown_connections 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.084+0000 7f1710659640 1 -- 172.21.15.116:0/2914577243 wait complete. 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 Processor -- start 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 -- start start 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708100690 0x7f1708198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17081066a0 0x7f17081a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1708110960 con 0x7f1708101050 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17081107e0 con 0x7f1708100690 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f1710659640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1708110ae0 con 0x7f17081066a0 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45422/0 (socket says 172.21.15.116:45422) 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f170dbf4640 1 -- 172.21.15.116:0/3614808161 learned_addr learned my addr 172.21.15.116:0/3614808161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708100690 0x7f1708198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.085+0000 7f170ebf6640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17081066a0 0x7f17081a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 -- 172.21.15.116:0/3614808161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17081066a0 msgr2=0x7f17081a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17081066a0 0x7f17081a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 -- 172.21.15.116:0/3614808161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708100690 msgr2=0x7f1708198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708100690 0x7f1708198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 -- 172.21.15.116:0/3614808161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17081a4440 con 0x7f1708101050 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f16f800e9f0 tx=0x7f16f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f16ff7fe640 1 -- 172.21.15.116:0/3614808161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f800cd80 con 0x7f1708101050 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f16ff7fe640 1 -- 172.21.15.116:0/3614808161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16f80054d0 con 0x7f1708101050 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17081a4610 con 0x7f1708101050 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f16ff7fe640 1 -- 172.21.15.116:0/3614808161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8010490 con 0x7f1708101050 2024-03-19T02:46:50.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.086+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17081a4ae0 con 0x7f1708101050 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.088+0000 7f16ff7fe640 1 -- 172.21.15.116:0/3614808161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f16f8005020 con 0x7f1708101050 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 Processor -- start 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 -- start start 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16dce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 0x560bd16e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560bd1640500 con 0x560bd16dd350 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bd15e8be0 con 0x560bd16dca30 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f1713242740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560bd163c5a0 con 0x560bd16e5f30 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f170e3f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 0x560bd16e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48164/0 (socket says 172.21.15.116:48164) 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f170dbf4640 1 -- 172.21.15.116:0/3395534704 learned_addr learned my addr 172.21.15.116:0/3395534704 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f170ebf6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.089+0000 7f16ff7fe640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 0x7f16f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 -- 172.21.15.116:0/3395534704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 msgr2=0x560bd16e8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 -- 172.21.15.116:0/3395534704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 msgr2=0x560bd16dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 -- 172.21.15.116:0/3395534704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd160a000 con 0x560bd16dd350 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170ebf6640 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170ebf6640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 0x7f16f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f16ff7fe640 1 -- 172.21.15.116:0/3614808161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f16f80cf4f0 con 0x7f1708101050 2024-03-19T02:46:50.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 0x560bd16e59f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f1704009230 tx=0x7f1704009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e88dbbddc03267d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f16fcbfe640 1 -- 172.21.15.116:0/3395534704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704045020 con 0x560bd16dd350 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f16fcbfe640 1 -- 172.21.15.116:0/3395534704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f170402c7e0 con 0x560bd16dd350 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.090+0000 7f16fcbfe640 1 -- 172.21.15.116:0/3395534704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17040053c0 con 0x560bd16dd350 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- 172.21.15.116:0/3395534704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 msgr2=0x560bd16e59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f170ebf6640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 0x7f16f40a4e30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1700005fd0 tx=0x7f1700014040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 0x560bd16e59f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f1704009230 tx=0x7f1704009560 comp rx=0 tx=0).stop 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- 172.21.15.116:0/3395534704 shutdown_connections 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16e5f30 0x560bd16e8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dd350 0x560bd16e59f0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- 172.21.15.116:0/3395534704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- 172.21.15.116:0/3395534704 >> 172.21.15.116:0/3395534704 conn(0x560bd16d7160 msgr2=0x560bd16d7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- 172.21.15.116:0/3395534704 shutdown_connections 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- 172.21.15.116:0/3395534704 wait complete. 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 -- start start 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 Processor -- start 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd16e3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.091+0000 7f1713242740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f1713242740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560bd16e8cc0 con 0x560bd16e5f30 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f1713242740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bd15fc160 con 0x560bd16dca30 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f1713242740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560bd16e8e40 con 0x560bd16dd350 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f1713242740 10 client.0 _ll_register_callbacks cb 0x560bd1638a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170ebf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170ebf6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45442/0 (socket says 172.21.15.116:45442) 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170ebf6640 1 -- 172.21.15.116:0/4137746001 learned_addr learned my addr 172.21.15.116:0/4137746001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:50.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd16e3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170e3f5640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 -- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 msgr2=0x560bd16e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 -- 172.21.15.116:0/4137746001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 msgr2=0x560bd16e39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 -- 172.21.15.116:0/4137746001 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd141e7d0 con 0x560bd16dd350 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170ebf6640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f170dbf4640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd16e3460 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f16f809b7a0 tx=0x7f16f8009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d5c031f9233ad9d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f16f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f809b5e0 con 0x560bd16dd350 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f16f8097e80 con 0x560bd16dd350 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.092+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f170402c7e0 con 0x560bd16dd350 2024-03-19T02:46:50.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f16f809b170 con 0x560bd16dd350 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 10 client.30727 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560bd13543f0 con 0x560bd16dd350 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f170402c7e0 con 0x560bd16dd350 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f16f80cf120 con 0x560bd16dd350 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 10 client.30727 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 10 client.30727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 20 client.30727 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16f62b0 0x560bd16f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 4 client.30727 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.093+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd164da40 con 0x560bd16f62b0 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.094+0000 7f170e3f5640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16f62b0 0x560bd16f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.094+0000 7f170e3f5640 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16f62b0 0x560bd16f86a0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d90c492c87025b7d server_cookie=f6cc8de283bfba43 in_seq=0 out_seq=0 2024-03-19T02:46:50.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.094+0000 7f16f1ffb640 10 client.30727 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.100+0000 7f16f1ffb640 1 -- 172.21.15.116:0/4137746001 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bd164da40 con 0x560bd16f62b0 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f16f1ffb640 10 client.30727 handle_command_reply: tid=0 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 client.30727 shutdown 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 msgr2=0x560bd16e3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd16e3460 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f16f809b7a0 tx=0x7f16f8009720 comp rx=0 tx=0).stop 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 shutdown_connections 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16f62b0 0x560bd16f86a0 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16e5f30 0x560bd16e39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd16e3460 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f16f1ffb640 0 client.30727 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 --2- 172.21.15.116:0/4137746001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 >> 172.21.15.116:0/4137746001 conn(0x560bd16d7160 msgr2=0x560bd16d8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 shutdown_connections 2024-03-19T02:46:50.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 1 -- 172.21.15.116:0/4137746001 wait complete. 2024-03-19T02:46:50.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.101+0000 7f1713242740 20 client.30727 trim_cache size 0 max 16384 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.121+0000 7f1713242740 1 Processor -- start 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.121+0000 7f1713242740 1 -- start start 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 0x560bd16e5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dd350 0x560bd16fde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17081aec00 con 0x560bd16dca30 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17081b0740 con 0x560bd16dd350 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f1713242740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17081af2d0 con 0x560bd16d7160 2024-03-19T02:46:50.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45444/0 (socket says 172.21.15.116:45444) 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 -- 172.21.15.116:0/2104532509 learned_addr learned my addr 172.21.15.116:0/2104532509 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170e3f5640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 0x560bd16e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170ebf6640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dd350 0x560bd16fde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 -- 172.21.15.116:0/2104532509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 msgr2=0x560bd16e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 0x560bd16e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 -- 172.21.15.116:0/2104532509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dd350 msgr2=0x560bd16fde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dd350 0x560bd16fde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 -- 172.21.15.116:0/2104532509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd13140f0 con 0x560bd16dca30 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170e3f5640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 0x560bd16e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f170dbf4640 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009380 tx=0x7f16f8005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ca872137b1a0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.122+0000 7f16f3fff640 1 -- 172.21.15.116:0/2104532509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f809b350 con 0x560bd16dca30 2024-03-19T02:46:50.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f16f3fff640 1 -- 172.21.15.116:0/2104532509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16f8005680 con 0x560bd16dca30 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f16f3fff640 1 -- 172.21.15.116:0/2104532509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80d5220 con 0x560bd16dca30 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 -- 172.21.15.116:0/2104532509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 msgr2=0x560bd16f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009380 tx=0x7f16f8005e70 comp rx=0 tx=0).stop 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 -- 172.21.15.116:0/2104532509 shutdown_connections 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dd350 0x560bd16fde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16dca30 0x560bd16f1a60 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 --2- 172.21.15.116:0/2104532509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16d7160 0x560bd16e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 -- 172.21.15.116:0/2104532509 >> 172.21.15.116:0/2104532509 conn(0x560bd16e1b60 msgr2=0x560bd16e3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 -- 172.21.15.116:0/2104532509 shutdown_connections 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.123+0000 7f1713242740 1 -- 172.21.15.116:0/2104532509 wait complete. 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 -- start start 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 Processor -- start 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd175b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17081b0070 con 0x560bd16d7160 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17081af9a0 con 0x560bd16dca30 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17081acc00 con 0x560bd16dd350 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f1713242740 10 client.0 _ll_register_callbacks cb 0x7f16f800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45454/0 (socket says 172.21.15.116:45454) 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 -- 172.21.15.116:0/3017836947 learned_addr learned my addr 172.21.15.116:0/3017836947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170ebf6640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd175b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 -- 172.21.15.116:0/3017836947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 msgr2=0x560bd175b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd175b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 -- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 msgr2=0x560bd16e07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 -- 172.21.15.116:0/3017836947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd175ba20 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f170e3f5640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f170402c780 tx=0x7f170402c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bede402cea55b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.124+0000 7f16f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704045040 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9510+0+0 (secure 0 0 0) 0x7f1704004450 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16f8005680 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1704005040 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 10 client.21033 fetch_fsmap learned FSMap version 38 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560bd175cd30 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16f8005680 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f17040035f0 con 0x560bd16d7160 2024-03-19T02:46:50.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 10 client.21033 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 10 client.21033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 20 client.21033 populate_metadata read hostname 'smithi116' 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16e5f30 0x560bd175f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 4 client.21033 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.125+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560bd1647d90 con 0x560bd16e5f30 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.126+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16e5f30 0x560bd175f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.126+0000 7f16f1ffb640 10 client.21033 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.126+0000 7f170dbf4640 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16e5f30 0x560bd175f140 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f7b201e5b07a8bb server_cookie=ce0fb9c91ca0f2f5 in_seq=0 out_seq=0 2024-03-19T02:46:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.128+0000 7f16f1ffb640 1 -- 172.21.15.116:0/3017836947 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560bd1647d90 con 0x560bd16e5f30 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.128+0000 7f16f1ffb640 10 client.21033 handle_command_reply: tid=0 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.128+0000 7f1713242740 1 client.21033 shutdown 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 msgr2=0x560bd16e0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f170402c780 tx=0x7f170402c3e0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 shutdown_connections 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560bd16e5f30 0x560bd175f140 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560bd16dd350 0x560bd175b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd16dca30 0x560bd16e07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 --2- 172.21.15.116:0/3017836947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560bd16d7160 0x560bd16e0270 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 >> 172.21.15.116:0/3017836947 conn(0x560bd16e1b60 msgr2=0x560bd16d5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f16f1ffb640 0 client.21033 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:46:50.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 shutdown_connections 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 1 -- 172.21.15.116:0/3017836947 wait complete. 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.129+0000 7f1713242740 20 client.21033 trim_cache size 0 max 16384 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (463 inodes in the stack)", 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "e214d867-bfc7-4110-be10-a52728b9b175": { 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "e214d867-bfc7-4110-be10-a52728b9b175", 2024-03-19T02:46:50.135 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:46:50.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:46:50.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:46:50.136 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:46:50.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 msgr2=0x7f16f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 0x7f16f40a4e30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1700005fd0 tx=0x7f1700014040 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 msgr2=0x7f17081993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f16f800e9f0 tx=0x7f16f800edf0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 shutdown_connections 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f16f40a2970 0x7f16f40a4e30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17081066a0 0x7f17081a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1708101050 0x7f17081993c0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 --2- 172.21.15.116:0/3614808161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1708100690 0x7f1708198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 >> 172.21.15.116:0/3614808161 conn(0x7f1708077820 msgr2=0x7f1708103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 shutdown_connections 2024-03-19T02:46:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:46:50.130+0000 7f1710659640 1 -- 172.21.15.116:0/3614808161 wait complete. 2024-03-19T02:46:50.143 INFO:tasks.cephfs.filesystem:scrub status for tag:e214d867-bfc7-4110-be10-a52728b9b175 - {'path': '/', 'tag': 'e214d867-bfc7-4110-be10-a52728b9b175', 'options': 'recursive,force'} 2024-03-19T02:46:50.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42119 5.99 MB/sec execute 252 sec latency 215.381 ms 2024-03-19T02:46:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:50 smithi162 ceph-mon[35761]: pgmap v599: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:50 smithi154 ceph-mon[35693]: pgmap v599: 129 pgs: 129 active+clean; 996 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-19T02:46:51.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42260 5.98 MB/sec execute 253 sec latency 139.281 ms 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:51 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:46:51.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:51 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:46:51.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:51.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:51 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:46:52.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42375 5.97 MB/sec execute 254 sec latency 138.915 ms 2024-03-19T02:46:52.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:52 smithi162 ceph-mon[35761]: pgmap v600: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-03-19T02:46:52.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:52 smithi154 ceph-mon[35693]: pgmap v600: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-03-19T02:46:52.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:52 smithi116 ceph-mon[32263]: pgmap v600: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 248 op/s 2024-03-19T02:46:53.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42496 5.97 MB/sec execute 255 sec latency 240.864 ms 2024-03-19T02:46:54.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42637 5.98 MB/sec execute 256 sec latency 148.078 ms 2024-03-19T02:46:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:54 smithi162 ceph-mon[35761]: pgmap v601: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-19T02:46:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:54 smithi154 ceph-mon[35693]: pgmap v601: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-19T02:46:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:54 smithi116 ceph-mon[32263]: pgmap v601: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 151 op/s 2024-03-19T02:46:55.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42712 5.97 MB/sec execute 257 sec latency 191.131 ms 2024-03-19T02:46:56.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42821 5.97 MB/sec execute 258 sec latency 158.980 ms 2024-03-19T02:46:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:56 smithi162 ceph-mon[35761]: pgmap v602: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 227 op/s 2024-03-19T02:46:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:56 smithi154 ceph-mon[35693]: pgmap v602: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 227 op/s 2024-03-19T02:46:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:56 smithi116 ceph-mon[32263]: pgmap v602: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 227 op/s 2024-03-19T02:46:57.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 42925 5.98 MB/sec execute 259 sec latency 160.820 ms 2024-03-19T02:46:58.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43038 5.98 MB/sec execute 260 sec latency 155.548 ms 2024-03-19T02:46:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:46:58 smithi162 ceph-mon[35761]: pgmap v603: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:46:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:46:58 smithi154 ceph-mon[35693]: pgmap v603: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:46:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:46:58 smithi116 ceph-mon[32263]: pgmap v603: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:46:59.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43158 5.99 MB/sec execute 261 sec latency 171.164 ms 2024-03-19T02:47:00.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43324 6.02 MB/sec execute 262 sec latency 180.560 ms 2024-03-19T02:47:00.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:00 smithi162 ceph-mon[35761]: pgmap v604: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:47:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:00 smithi154 ceph-mon[35693]: pgmap v604: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:47:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:00 smithi116 ceph-mon[32263]: pgmap v604: 129 pgs: 129 active+clean; 1012 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-03-19T02:47:01.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43468 6.03 MB/sec execute 263 sec latency 187.248 ms 2024-03-19T02:47:02.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43636 6.05 MB/sec execute 264 sec latency 149.337 ms 2024-03-19T02:47:02.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:02 smithi162 ceph-mon[35761]: pgmap v605: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-03-19T02:47:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:02 smithi154 ceph-mon[35693]: pgmap v605: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-03-19T02:47:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:02 smithi116 ceph-mon[32263]: pgmap v605: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 228 op/s 2024-03-19T02:47:03.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43729 6.05 MB/sec execute 265 sec latency 167.692 ms 2024-03-19T02:47:04.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43838 6.06 MB/sec execute 266 sec latency 157.170 ms 2024-03-19T02:47:04.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:04 smithi162 ceph-mon[35761]: pgmap v606: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 148 op/s 2024-03-19T02:47:04.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:04 smithi154 ceph-mon[35693]: pgmap v606: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 148 op/s 2024-03-19T02:47:04.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:04 smithi116 ceph-mon[32263]: pgmap v606: 129 pgs: 129 active+clean; 931 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 148 op/s 2024-03-19T02:47:05.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43911 6.05 MB/sec execute 267 sec latency 157.102 ms 2024-03-19T02:47:05.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:05.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:05 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:05.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:05.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:05 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:05.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:05.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:05 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:06.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 43958 6.04 MB/sec execute 268 sec latency 145.384 ms 2024-03-19T02:47:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:06 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:06 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:06 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:06 smithi162 ceph-mon[35761]: pgmap v607: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 226 op/s 2024-03-19T02:47:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:06 smithi162 ceph-mon[35761]: osdmap e122: 12 total, 12 up, 12 in 2024-03-19T02:47:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:06 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:06 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:06 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:06.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:06 smithi154 ceph-mon[35693]: pgmap v607: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 226 op/s 2024-03-19T02:47:06.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:06 smithi154 ceph-mon[35693]: osdmap e122: 12 total, 12 up, 12 in 2024-03-19T02:47:06.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:06 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:06 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:06 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:06 smithi116 ceph-mon[32263]: pgmap v607: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 226 op/s 2024-03-19T02:47:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:06 smithi116 ceph-mon[32263]: osdmap e122: 12 total, 12 up, 12 in 2024-03-19T02:47:07.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44040 6.04 MB/sec execute 269 sec latency 143.114 ms 2024-03-19T02:47:08.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44147 6.06 MB/sec execute 270 sec latency 150.657 ms 2024-03-19T02:47:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:08 smithi162 ceph-mon[35761]: pgmap v609: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:08 smithi154 ceph-mon[35693]: pgmap v609: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:08 smithi116 ceph-mon[32263]: pgmap v609: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:09.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44259 6.07 MB/sec execute 271 sec latency 167.454 ms 2024-03-19T02:47:10.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44365 6.06 MB/sec execute 272 sec latency 159.430 ms 2024-03-19T02:47:10.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:10 smithi162 ceph-mon[35761]: pgmap v610: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:10 smithi154 ceph-mon[35693]: pgmap v610: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:10 smithi116 ceph-mon[32263]: pgmap v610: 129 pgs: 129 active+clean; 953 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-19T02:47:11.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44488 6.09 MB/sec execute 273 sec latency 168.689 ms 2024-03-19T02:47:12.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44574 6.07 MB/sec execute 274 sec latency 187.406 ms 2024-03-19T02:47:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:12 smithi162 ceph-mon[35761]: pgmap v611: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 163 op/s 2024-03-19T02:47:13.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:12 smithi162 ceph-mon[35761]: osdmap e123: 12 total, 12 up, 12 in 2024-03-19T02:47:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:12 smithi154 ceph-mon[35693]: pgmap v611: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 163 op/s 2024-03-19T02:47:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:12 smithi154 ceph-mon[35693]: osdmap e123: 12 total, 12 up, 12 in 2024-03-19T02:47:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:12 smithi116 ceph-mon[32263]: pgmap v611: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 163 op/s 2024-03-19T02:47:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:12 smithi116 ceph-mon[32263]: osdmap e123: 12 total, 12 up, 12 in 2024-03-19T02:47:13.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44682 6.06 MB/sec execute 275 sec latency 158.625 ms 2024-03-19T02:47:14.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44777 6.04 MB/sec execute 276 sec latency 180.768 ms 2024-03-19T02:47:15.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:14 smithi162 ceph-mon[35761]: pgmap v613: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.1 MiB/s wr, 87 op/s 2024-03-19T02:47:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:14 smithi154 ceph-mon[35693]: pgmap v613: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.1 MiB/s wr, 87 op/s 2024-03-19T02:47:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:14 smithi116 ceph-mon[32263]: pgmap v613: 129 pgs: 129 active+clean; 947 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.1 MiB/s wr, 87 op/s 2024-03-19T02:47:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44869 6.04 MB/sec execute 277 sec latency 219.801 ms 2024-03-19T02:47:16.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 44978 6.03 MB/sec execute 278 sec latency 197.041 ms 2024-03-19T02:47:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:16 smithi154 ceph-mon[35693]: pgmap v614: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 141 op/s 2024-03-19T02:47:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:16 smithi116 ceph-mon[32263]: pgmap v614: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 141 op/s 2024-03-19T02:47:17.331 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45059 6.01 MB/sec execute 279 sec latency 189.204 ms 2024-03-19T02:47:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:16 smithi162 ceph-mon[35761]: pgmap v614: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 141 op/s 2024-03-19T02:47:18.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45154 6.01 MB/sec execute 280 sec latency 247.941 ms 2024-03-19T02:47:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:18 smithi154 ceph-mon[35693]: pgmap v615: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:18 smithi116 ceph-mon[32263]: pgmap v615: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:19.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45251 5.99 MB/sec execute 281 sec latency 175.274 ms 2024-03-19T02:47:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:18 smithi162 ceph-mon[35761]: pgmap v615: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:20.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:47:20.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:47:20.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 -- 172.21.15.116:0/2116710324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40107870 msgr2=0x7f6a40107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 --2- 172.21.15.116:0/2116710324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40107870 0x7f6a40107c50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009930 tx=0x7f6a3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:20.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 -- 172.21.15.116:0/2116710324 shutdown_connections 2024-03-19T02:47:20.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 --2- 172.21.15.116:0/2116710324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a400693e0 0x7f6a4010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 --2- 172.21.15.116:0/2116710324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40068a20 0x7f6a40068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 --2- 172.21.15.116:0/2116710324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40107870 0x7f6a40107c50 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.282+0000 7f6a44e20640 1 -- 172.21.15.116:0/2116710324 >> 172.21.15.116:0/2116710324 conn(0x7f6a40077820 msgr2=0x7f6a40077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.283+0000 7f6a44e20640 1 -- 172.21.15.116:0/2116710324 shutdown_connections 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.283+0000 7f6a44e20640 1 -- 172.21.15.116:0/2116710324 wait complete. 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.283+0000 7f6a44e20640 1 Processor -- start 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.283+0000 7f6a44e20640 1 -- start start 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 0x7f6a40197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40107870 0x7f6a4019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40111a70 con 0x7f6a400693e0 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401118f0 con 0x7f6a40107870 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a44e20640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40111bf0 con 0x7f6a40068a20 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 0x7f6a40197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34044/0 (socket says 172.21.15.116:34044) 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a37fff640 1 -- 172.21.15.116:0/3921178960 learned_addr learned my addr 172.21.15.116:0/3921178960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40107870 0x7f6a4019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 -- 172.21.15.116:0/3921178960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40107870 msgr2=0x7f6a4019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40107870 0x7f6a4019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 -- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 msgr2=0x7f6a40198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 -- 172.21.15.116:0/3921178960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4019cfc0 con 0x7f6a40068a20 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a37fff640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.284+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 0x7f6a40197fd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009230 tx=0x7f6a3002cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.285+0000 7f6a377fe640 1 -- 172.21.15.116:0/3921178960 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30043020 con 0x7f6a40068a20 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.285+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4019d250 con 0x7f6a40068a20 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.285+0000 7f6a377fe640 1 -- 172.21.15.116:0/3921178960 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a300042d0 con 0x7f6a40068a20 2024-03-19T02:47:20.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.285+0000 7f6a377fe640 1 -- 172.21.15.116:0/3921178960 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003a80 con 0x7f6a40068a20 2024-03-19T02:47:20.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.285+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401a4b40 con 0x7f6a40068a20 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.287+0000 7f6a377fe640 1 -- 172.21.15.116:0/3921178960 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6a30046020 con 0x7f6a40068a20 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.287+0000 7f6a47a0d740 1 Processor -- start 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 -- start start 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcc39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56143dc1e500 con 0x56143dcc3f30 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56143dbc6be0 con 0x56143dcbaa30 2024-03-19T02:47:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56143dc1a5a0 con 0x56143dcbb350 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34050/0 (socket says 172.21.15.116:34050) 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 -- 172.21.15.116:0/2006195746 learned_addr learned my addr 172.21.15.116:0/2006195746 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcc39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 -- 172.21.15.116:0/2006195746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 msgr2=0x56143dcc39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcc39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 -- 172.21.15.116:0/2006195746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 msgr2=0x56143dcbae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a37fff640 1 -- 172.21.15.116:0/2006195746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56143dbe8000 con 0x56143dcc3f30 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.288+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcc39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a377fe640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 0x7f6a2c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a37fff640 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f6a280115f0 tx=0x7f6a28011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa037ed66eed2916 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 0x7f6a2c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a377fe640 1 -- 172.21.15.116:0/3921178960 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6a300f0970 con 0x7f6a40068a20 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a34ff9640 1 -- 172.21.15.116:0/2006195746 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2801e070 con 0x56143dcc3f30 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a34ff9640 1 -- 172.21.15.116:0/2006195746 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a2800f660 con 0x56143dcc3f30 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.289+0000 7f6a34ff9640 1 -- 172.21.15.116:0/2006195746 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28003850 con 0x56143dcc3f30 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2006195746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 msgr2=0x56143dcc6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 0x7f6a2c0a4e30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005fd0 tx=0x7f6a38009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f6a280115f0 tx=0x7f6a28011ab0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2006195746 shutdown_connections 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc6320 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcc39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2006195746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2006195746 >> 172.21.15.116:0/2006195746 conn(0x56143dcb5160 msgr2=0x56143dcb5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2006195746 shutdown_connections 2024-03-19T02:47:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2006195746 wait complete. 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 -- start start 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.290+0000 7f6a47a0d740 1 Processor -- start 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbaa30 0x56143dcc0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbb350 0x56143dcc1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56143dccb180 con 0x56143dcc3f30 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56143dbda160 con 0x56143dcbb350 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56143dccb300 con 0x56143dcbaa30 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 10 client.0 _ll_register_callbacks cb 0x56143dc16a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34062/0 (socket says 172.21.15.116:34062) 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/1569760092 learned_addr learned my addr 172.21.15.116:0/1569760092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbaa30 0x56143dcc0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a37fff640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbb350 0x56143dcc1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/1569760092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbaa30 msgr2=0x56143dcc0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbaa30 0x56143dcc0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbb350 msgr2=0x56143dcc1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbb350 0x56143dcc1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/1569760092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56143d9fc7d0 con 0x56143dcc3f30 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f6a38003fd0 tx=0x7f6a38004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2120f74174fa28 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3800f8b0 con 0x56143dcc3f30 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6a3800fa50 con 0x56143dcc3f30 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.291+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2800f660 con 0x56143dcc3f30 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6a3800fc50 con 0x56143dcc3f30 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 10 client.21045 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56143d9323f0 con 0x56143dcc3f30 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a2800f660 con 0x56143dcc3f30 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6a38026070 con 0x56143dcc3f30 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 10 client.21045 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 10 client.21045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 20 client.21045 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dcd8710 0x56143dcdab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 4 client.21045 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56143dc2ba40 con 0x56143dcd8710 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.292+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dcd8710 0x56143dcdab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.293+0000 7f6a25ffb640 10 client.21045 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.293+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dcd8710 0x56143dcdab60 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dcf9dd9fbcbe7e6 server_cookie=aa2063b2c063e4cb in_seq=0 out_seq=0 2024-03-19T02:47:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a25ffb640 1 -- 172.21.15.116:0/1569760092 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56143dc2ba40 con 0x56143dcd8710 2024-03-19T02:47:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a25ffb640 10 client.21045 handle_command_reply: tid=0 2024-03-19T02:47:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 client.21045 shutdown 2024-03-19T02:47:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 msgr2=0x56143dcc19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f6a38003fd0 tx=0x7f6a38004ab0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 shutdown_connections 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dcd8710 0x56143dcdab60 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcc19d0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbb350 0x56143dcc1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a25ffb640 0 client.21045 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1569760092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbaa30 0x56143dcc0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 >> 172.21.15.116:0/1569760092 conn(0x56143dcb5160 msgr2=0x56143dcb6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 shutdown_connections 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.296+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1569760092 wait complete. 2024-03-19T02:47:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.297+0000 7f6a47a0d740 20 client.21045 trim_cache size 0 max 16384 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 Processor -- start 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 -- start start 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcb6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcc3f30 0x56143dce0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401aca30 con 0x56143dcbb350 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401ae570 con 0x56143dcbaa30 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.316+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401ad100 con 0x56143dcc3f30 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34076/0 (socket says 172.21.15.116:34076) 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 -- 172.21.15.116:0/1338320479 learned_addr learned my addr 172.21.15.116:0/1338320479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcb6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcc3f30 0x56143dce0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 -- 172.21.15.116:0/1338320479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcc3f30 msgr2=0x56143dce0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcc3f30 0x56143dce0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 -- 172.21.15.116:0/1338320479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 msgr2=0x56143dcb6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcb6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 -- 172.21.15.116:0/1338320479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56143dcbb180 con 0x56143dcbb350 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a37fff640 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f6a280115f0 tx=0x7f6a28011530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7192b4f2e72aa6d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a27fff640 1 -- 172.21.15.116:0/1338320479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28028070 con 0x56143dcbb350 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a27fff640 1 -- 172.21.15.116:0/1338320479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a2800fb10 con 0x56143dcbb350 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.317+0000 7f6a27fff640 1 -- 172.21.15.116:0/1338320479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2801e040 con 0x56143dcbb350 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1338320479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 msgr2=0x56143dcd3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f6a280115f0 tx=0x7f6a28011530 comp rx=0 tx=0).stop 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1338320479 shutdown_connections 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcc3f30 0x56143dce0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcbb350 0x56143dcd3f20 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/1338320479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1338320479 >> 172.21.15.116:0/1338320479 conn(0x56143dcb5160 msgr2=0x56143dcc0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1338320479 shutdown_connections 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- 172.21.15.116:0/1338320479 wait complete. 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 -- start start 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.318+0000 7f6a47a0d740 1 Processor -- start 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcbe320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401adea0 con 0x56143dcc3f30 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401ad7d0 con 0x56143dcbaa30 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401aaa30 con 0x56143dcbb350 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a47a0d740 10 client.0 _ll_register_callbacks cb 0x7f6a38009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcbe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34082/0 (socket says 172.21.15.116:34082) 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/2628080951 learned_addr learned my addr 172.21.15.116:0/2628080951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/2628080951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 msgr2=0x56143dcbe320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcbe320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 msgr2=0x56143dcbdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 -- 172.21.15.116:0/2628080951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56143d8f20f0 con 0x56143dcc3f30 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a37fff640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcbe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a3ed9d640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f6a38028550 tx=0x7f6a38009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1777fee4b01f36ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.319+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38003df0 con 0x56143dcc3f30 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2800fb10 con 0x56143dcc3f30 2024-03-19T02:47:20.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6a380289d0 con 0x56143dcc3f30 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6a38041050 con 0x56143dcc3f30 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 10 client.21051 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56143dd3ee40 con 0x56143dcc3f30 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a2800fb10 con 0x56143dcc3f30 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6a38026070 con 0x56143dcc3f30 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 10 client.21051 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 10 client.21051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 20 client.21051 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dd3f640 0x56143dd41a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 4 client.21051 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56143dcb7480 con 0x56143dd3f640 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.320+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dd3f640 0x56143dd41a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.321+0000 7f6a25ffb640 10 client.21051 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:20.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.321+0000 7f6a3e59c640 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dd3f640 0x56143dd41a60 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689083efad4b653b server_cookie=70750e836845381c in_seq=0 out_seq=0 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a25ffb640 1 -- 172.21.15.116:0/2628080951 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56143dcb7480 con 0x56143dd3f640 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a25ffb640 10 client.21051 handle_command_reply: tid=0 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 client.21051 shutdown 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 msgr2=0x56143dcbe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f6a38028550 tx=0x7f6a38009590 comp rx=0 tx=0).stop 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 shutdown_connections 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56143dd3f640 0x56143dd41a60 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56143dcc3f30 0x56143dcbe860 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56143dcbb350 0x56143dcbe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a25ffb640 0 client.21051 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 --2- 172.21.15.116:0/2628080951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56143dcbaa30 0x56143dcbdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 >> 172.21.15.116:0/2628080951 conn(0x56143dcb5160 msgr2=0x56143dcc0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 shutdown_connections 2024-03-19T02:47:20.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 1 -- 172.21.15.116:0/2628080951 wait complete. 2024-03-19T02:47:20.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.322+0000 7f6a47a0d740 20 client.21051 trim_cache size 0 max 16384 2024-03-19T02:47:20.330 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:20.330 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:47:20.330 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:47:20.330 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 msgr2=0x7f6a2c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 0x7f6a2c0a4e30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005fd0 tx=0x7f6a38009500 comp rx=0 tx=0).stop 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 msgr2=0x7f6a40197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 0x7f6a40197fd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009230 tx=0x7f6a3002cd20 comp rx=0 tx=0).stop 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 shutdown_connections 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a2970 0x7f6a2c0a4e30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a40107870 0x7f6a4019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a400693e0 0x7f6a40198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 --2- 172.21.15.116:0/3921178960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40068a20 0x7f6a40197fd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 >> 172.21.15.116:0/3921178960 conn(0x7f6a40077820 msgr2=0x7f6a400fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 shutdown_connections 2024-03-19T02:47:20.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.323+0000 7f6a44e20640 1 -- 172.21.15.116:0/3921178960 wait complete. 2024-03-19T02:47:20.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45351 5.99 MB/sec execute 282 sec latency 209.261 ms 2024-03-19T02:47:20.335 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:47:20.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:47:20.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 -- 172.21.15.116:0/2843818660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c4106730 msgr2=0x7f06c4106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/2843818660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c4106730 0x7f06c4106b10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009900 tx=0x7f06c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:20.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 -- 172.21.15.116:0/2843818660 shutdown_connections 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/2843818660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c41010e0 0x7f06c410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/2843818660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/2843818660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c4106730 0x7f06c4106b10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 -- 172.21.15.116:0/2843818660 >> 172.21.15.116:0/2843818660 conn(0x7f06c4077820 msgr2=0x7f06c4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 -- 172.21.15.116:0/2843818660 shutdown_connections 2024-03-19T02:47:20.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.461+0000 7f06cb4b2640 1 -- 172.21.15.116:0/2843818660 wait complete. 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 Processor -- start 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 -- start start 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c41010e0 0x7f06c41a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 0x7f06c419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06c41109f0 con 0x7f06c4106730 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06c4110870 con 0x7f06c41010e0 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06cb4b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06c4110b70 con 0x7f06c4100720 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c9a4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 0x7f06c419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51570/0 (socket says 172.21.15.116:51570) 2024-03-19T02:47:20.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 -- 172.21.15.116:0/3407480351 learned_addr learned my addr 172.21.15.116:0/3407480351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c8a4d640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c41010e0 0x7f06c41a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 -- 172.21.15.116:0/3407480351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c41010e0 msgr2=0x7f06c41a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c41010e0 0x7f06c41a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 -- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 msgr2=0x7f06c419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 0x7f06c419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c924e640 1 -- 172.21.15.116:0/3407480351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c419a9f0 con 0x7f06c4100720 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.462+0000 7f06c9a4f640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 0x7f06c419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06c924e640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006010 tx=0x7f06c0005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0043020 con 0x7f06c4100720 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c419ac80 con 0x7f06c4100720 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c41a6d80 con 0x7f06c4100720 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c0004350 con 0x7f06c4100720 2024-03-19T02:47:20.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.463+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0003a10 con 0x7f06c4100720 2024-03-19T02:47:20.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.464+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a8005290 con 0x7f06c4100720 2024-03-19T02:47:20.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.466+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f06c0046020 con 0x7f06c4100720 2024-03-19T02:47:20.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.466+0000 7f06ba7fc640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 0x7f06b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.467+0000 7f06c8a4d640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 0x7f06b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.467+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f06c00f0fe0 con 0x7f06c4100720 2024-03-19T02:47:20.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.467+0000 7f06c8a4d640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 0x7f06b00a4e30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f06c419b830 tx=0x7f06b4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.470+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f06c004d070 con 0x7f06c4100720 2024-03-19T02:47:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.796+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06a8005580 con 0x7f06c4100720 2024-03-19T02:47:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.797+0000 7f06ba7fc640 1 -- 172.21.15.116:0/3407480351 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7f06c00bda70 con 0x7f06c4100720 2024-03-19T02:47:20.798 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:47:20.798 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:47:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.799+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 msgr2=0x7f06b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.799+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 0x7f06b00a4e30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f06c419b830 tx=0x7f06b4005f90 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 msgr2=0x7f06c419ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006010 tx=0x7f06c0005d90 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 shutdown_connections 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f06b00a2970 0x7f06b00a4e30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c4106730 0x7f06c419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06c41010e0 0x7f06c41a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 --2- 172.21.15.116:0/3407480351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06c4100720 0x7f06c419ff30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 >> 172.21.15.116:0/3407480351 conn(0x7f06c4077820 msgr2=0x7f06c4102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 shutdown_connections 2024-03-19T02:47:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.800+0000 7f06cb4b2640 1 -- 172.21.15.116:0/3407480351 wait complete. 2024-03-19T02:47:20.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:47:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 -- 172.21.15.116:0/425713740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c101330 msgr2=0x7f268c10d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 --2- 172.21.15.116:0/425713740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c101330 0x7f268c10d520 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f2684009ca0 tx=0x7f268402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 -- 172.21.15.116:0/425713740 shutdown_connections 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 --2- 172.21.15.116:0/425713740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c101330 0x7f268c10d520 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 --2- 172.21.15.116:0/425713740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 0x7f268c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 --2- 172.21.15.116:0/425713740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c105690 0x7f268c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 -- 172.21.15.116:0/425713740 >> 172.21.15.116:0/425713740 conn(0x7f268c0774b0 msgr2=0x7f268c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.952+0000 7f2693cb5640 1 -- 172.21.15.116:0/425713740 shutdown_connections 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 -- 172.21.15.116:0/425713740 wait complete. 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 Processor -- start 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 -- start start 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 0x7f268c102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c101330 0x7f268c1028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f268c110a60 con 0x7f268c105690 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1108e0 con 0x7f268c101330 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2693cb5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f268c110be0 con 0x7f268c100970 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2692252640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2692252640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34120/0 (socket says 172.21.15.116:34120) 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.953+0000 7f2692252640 1 -- 172.21.15.116:0/4123055406 learned_addr learned my addr 172.21.15.116:0/4123055406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2691250640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c101330 0x7f268c1028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2691a51640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 0x7f268c102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 -- 172.21.15.116:0/4123055406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 msgr2=0x7f268c102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 0x7f268c102360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 -- 172.21.15.116:0/4123055406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c101330 msgr2=0x7f268c1028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c101330 0x7f268c1028a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 -- 172.21.15.116:0/4123055406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c1a64e0 con 0x7f268c105690 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2692252640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f268402c700 tx=0x7f2684035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2682ffd640 1 -- 172.21.15.116:0/4123055406 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684039040 con 0x7f268c105690 2024-03-19T02:47:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2682ffd640 1 -- 172.21.15.116:0/4123055406 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2684044070 con 0x7f268c105690 2024-03-19T02:47:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c1a6770 con 0x7f268c105690 2024-03-19T02:47:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2682ffd640 1 -- 172.21.15.116:0/4123055406 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840042a0 con 0x7f268c105690 2024-03-19T02:47:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.954+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c1a6c60 con 0x7f268c105690 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.956+0000 7f2682ffd640 1 -- 172.21.15.116:0/4123055406 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2684004440 con 0x7f268c105690 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 Processor -- start 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2682ffd640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 0x7f26780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 -- start start 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2691250640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 0x7f26780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6088bf0 0x55afb6091290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb6093bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55afb5feb4b0 con 0x55afb6088bf0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55afb5f93d70 con 0x55afb60917d0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2696976740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55afb5fe7550 con 0x55afb60882d0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.957+0000 7f2682ffd640 1 -- 172.21.15.116:0/4123055406 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f26840f15d0 con 0x7f268c105690 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2692252640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb6093bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51600/0 (socket says 172.21.15.116:51600) 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 -- 172.21.15.116:0/406426117 learned_addr learned my addr 172.21.15.116:0/406426117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691250640 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6088bf0 0x55afb6091290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 -- 172.21.15.116:0/406426117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 msgr2=0x55afb6093bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb6093bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 -- 172.21.15.116:0/406426117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6088bf0 msgr2=0x55afb6091290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6088bf0 0x55afb6091290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 -- 172.21.15.116:0/406426117 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb5fb6210 con 0x55afb60882d0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2692252640 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb6093bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691a51640 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2688009c70 tx=0x7f2688011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f8a54386715a3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2691250640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 0x7f26780a4e30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55afb6087da0 tx=0x7f267c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2677fff640 1 -- 172.21.15.116:0/406426117 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688004230 con 0x55afb60882d0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2677fff640 1 -- 172.21.15.116:0/406426117 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26880043d0 con 0x55afb60882d0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.958+0000 7f2677fff640 1 -- 172.21.15.116:0/406426117 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688019740 con 0x55afb60882d0 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 -- 172.21.15.116:0/406426117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 msgr2=0x55afb60886b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2688009c70 tx=0x7f2688011810 comp rx=0 tx=0).stop 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 -- 172.21.15.116:0/406426117 shutdown_connections 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb6093bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6088bf0 0x55afb6091290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 --2- 172.21.15.116:0/406426117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60882d0 0x55afb60886b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 -- 172.21.15.116:0/406426117 >> 172.21.15.116:0/406426117 conn(0x55afb6082a00 msgr2=0x55afb6082de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 -- 172.21.15.116:0/406426117 shutdown_connections 2024-03-19T02:47:20.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.959+0000 7f2696976740 1 -- 172.21.15.116:0/406426117 wait complete. 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 -- start start 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 Processor -- start 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb608ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb608f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2691a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2691a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34146/0 (socket says 172.21.15.116:34146) 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2691250640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb608ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2691a51640 1 -- 172.21.15.116:0/705783282 learned_addr learned my addr 172.21.15.116:0/705783282 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55afb60945c0 con 0x55afb60882d0 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55afb5fa7310 con 0x55afb6088bf0 2024-03-19T02:47:20.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55afb6094740 con 0x55afb60917d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2696976740 10 client.0 _ll_register_callbacks cb 0x55afb5f9b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.960+0000 7f2692252640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb608f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 -- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 msgr2=0x55afb608f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb608f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 -- 172.21.15.116:0/705783282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 msgr2=0x55afb608ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb608ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb5cff3f0 con 0x55afb60882d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691250640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb608ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2691a51640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2688000c00 tx=0x7f2688005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9bf211bdf4f537 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f26757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688027070 con 0x55afb60882d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f2688003800 con 0x55afb60882d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26880043d0 con 0x55afb60882d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2688033050 con 0x55afb60882d0 2024-03-19T02:47:20.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2696976740 10 client.21069 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:20.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.961+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55afb5dc97d0 con 0x55afb60882d0 2024-03-19T02:47:20.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.962+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26880043d0 con 0x55afb60882d0 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f2688018070 con 0x55afb60882d0 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 10 client.21069 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 10 client.21069 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 20 client.21069 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb60a1960 0x55afb60a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 4 client.21069 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afb60493e0 con 0x55afb60a1960 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.963+0000 7f2691250640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb60a1960 0x55afb60a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.964+0000 7f26757fa640 10 client.21069 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.964+0000 7f2691250640 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb60a1960 0x55afb60a3dd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18ed28041007f860 server_cookie=c9ee5d58d8aee659 in_seq=0 out_seq=0 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.965+0000 7f26757fa640 1 -- 172.21.15.116:0/705783282 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afb60493e0 con 0x55afb60a1960 2024-03-19T02:47:20.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.965+0000 7f26757fa640 10 client.21069 handle_command_reply: tid=0 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 client.21069 shutdown 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 msgr2=0x55afb608e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2688000c00 tx=0x7f2688005750 comp rx=0 tx=0).stop 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 shutdown_connections 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb60a1960 0x55afb60a3dd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb608f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb608ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 --2- 172.21.15.116:0/705783282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb608e790 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f26757fa640 0 client.21069 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 >> 172.21.15.116:0/705783282 conn(0x55afb6082a00 msgr2=0x55afb6084750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 shutdown_connections 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.966+0000 7f2696976740 1 -- 172.21.15.116:0/705783282 wait complete. 2024-03-19T02:47:20.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.967+0000 7f2696976740 20 client.21069 trim_cache size 0 max 16384 2024-03-19T02:47:20.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 Processor -- start 2024-03-19T02:47:20.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 -- start start 2024-03-19T02:47:20.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb609d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb60844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1b0c10 con 0x55afb60882d0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1b2750 con 0x55afb60917d0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.985+0000 7f2696976740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1b12e0 con 0x55afb6088bf0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691250640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb609d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691250640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51624/0 (socket says 172.21.15.116:51624) 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691250640 1 -- 172.21.15.116:0/1421303605 learned_addr learned my addr 172.21.15.116:0/1421303605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2692252640 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb60844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 -- 172.21.15.116:0/1421303605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 msgr2=0x55afb608f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 -- 172.21.15.116:0/1421303605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 msgr2=0x55afb60844f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb60844f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 -- 172.21.15.116:0/1421303605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb6082830 con 0x55afb60882d0 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691250640 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f2691a51640 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb609d360 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2688011850 tx=0x7f268801a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cf9407dcc45d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f26777fe640 1 -- 172.21.15.116:0/1421303605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688015020 con 0x55afb60882d0 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f26777fe640 1 -- 172.21.15.116:0/1421303605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2688009e20 con 0x55afb60882d0 2024-03-19T02:47:20.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.986+0000 7f26777fe640 1 -- 172.21.15.116:0/1421303605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688027070 con 0x55afb60882d0 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- 172.21.15.116:0/1421303605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 msgr2=0x55afb609d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb609d360 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2688011850 tx=0x7f268801a4c0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- 172.21.15.116:0/1421303605 shutdown_connections 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb60917d0 0x55afb60844f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb6088bf0 0x55afb608f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 --2- 172.21.15.116:0/1421303605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb60882d0 0x55afb609d360 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- 172.21.15.116:0/1421303605 >> 172.21.15.116:0/1421303605 conn(0x55afb6082a00 msgr2=0x55afb6083de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- 172.21.15.116:0/1421303605 shutdown_connections 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- 172.21.15.116:0/1421303605 wait complete. 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.987+0000 7f2696976740 1 -- start start 2024-03-19T02:47:20.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 Processor -- start 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb6099810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6094b80 0x55afb6094fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1b2080 con 0x55afb6094b80 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1b19b0 con 0x55afb6088bf0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1aeca0 con 0x55afb60917d0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2696976740 10 client.0 _ll_register_callbacks cb 0x7f267c009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2692252640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6094b80 0x55afb6094fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691a51640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb6099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51626/0 (socket says 172.21.15.116:51626) 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691a51640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb6099810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45570/0 (socket says 172.21.15.116:45570) 2024-03-19T02:47:20.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691a51640 1 -- 172.21.15.116:0/3944785777 learned_addr learned my addr 172.21.15.116:0/3944785777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 -- 172.21.15.116:0/3944785777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 msgr2=0x55afb6099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb6099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 -- 172.21.15.116:0/3944785777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6094b80 msgr2=0x55afb6094fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6094b80 0x55afb6094fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 -- 172.21.15.116:0/3944785777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb6093bc0 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f2691250640 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55afb609e220 tx=0x7f267c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e09e7ef788cd1295 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.988+0000 7f26757fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00ee30 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f267c01c070 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2688009e20 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f267c011a20 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 10 client.30772 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55afb6058880 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2688009e20 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f267c01f040 con 0x55afb60917d0 2024-03-19T02:47:20.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 10 client.30772 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 10 client.30772 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 20 client.30772 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.989+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb6108110 0x55afb610a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f2696976740 4 client.30772 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afb5ff89f0 con 0x55afb6108110 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f2691a51640 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb6108110 0x55afb610a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f2691a51640 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb6108110 0x55afb610a530 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5ca6655a6df8014b server_cookie=a88ed628d0e35b4b in_seq=0 out_seq=0 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f26757fa640 10 client.30772 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f26757fa640 1 -- 172.21.15.116:0/3944785777 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afb5ff89f0 con 0x55afb6108110 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f26757fa640 10 client.30772 handle_command_reply: tid=0 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.990+0000 7f2696976740 1 client.30772 shutdown 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 msgr2=0x55afb6099d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55afb609e220 tx=0x7f267c0046c0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 shutdown_connections 2024-03-19T02:47:20.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55afb6108110 0x55afb610a530 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55afb6094b80 0x55afb6094fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55afb60917d0 0x55afb6099d50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 --2- 172.21.15.116:0/3944785777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afb6088bf0 0x55afb6099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 >> 172.21.15.116:0/3944785777 conn(0x55afb6082a00 msgr2=0x55afb6083d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f26757fa640 0 client.30772 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 shutdown_connections 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 1 -- 172.21.15.116:0/3944785777 wait complete. 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.991+0000 7f2696976740 20 client.30772 trim_cache size 0 max 16384 2024-03-19T02:47:20.997 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:20.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 msgr2=0x7f26780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 0x7f26780a4e30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55afb6087da0 tx=0x7f267c005f90 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 msgr2=0x7f268c102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f268402c700 tx=0x7f2684035d60 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 shutdown_connections 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f26780a2970 0x7f26780a4e30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f268c105690 0x7f268c102de0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f268c101330 0x7f268c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 --2- 172.21.15.116:0/4123055406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f268c100970 0x7f268c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 >> 172.21.15.116:0/4123055406 conn(0x7f268c0774b0 msgr2=0x7f268c0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 shutdown_connections 2024-03-19T02:47:20.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:20.992+0000 7f2693cb5640 1 -- 172.21.15.116:0/4123055406 wait complete. 2024-03-19T02:47:21.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:47:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 -- 172.21.15.116:0/2051514913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8069c00 msgr2=0x7f6da806a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 --2- 172.21.15.116:0/2051514913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8069c00 0x7f6da806a060 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009900 tx=0x7f6d9c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:21.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 -- 172.21.15.116:0/2051514913 shutdown_connections 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 --2- 172.21.15.116:0/2051514913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 0x7f6da810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 --2- 172.21.15.116:0/2051514913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8069c00 0x7f6da806a060 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 --2- 172.21.15.116:0/2051514913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8107810 0x7f6da8107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 -- 172.21.15.116:0/2051514913 >> 172.21.15.116:0/2051514913 conn(0x7f6da80774b0 msgr2=0x7f6da80778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 -- 172.21.15.116:0/2051514913 shutdown_connections 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.144+0000 7f6dad67e640 1 -- 172.21.15.116:0/2051514913 wait complete. 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 Processor -- start 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 -- start start 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8069c00 0x7f6da819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 0x7f6da819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6da8111aa0 con 0x7f6da8107810 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6da8111920 con 0x7f6da8069c00 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6dad67e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6da8111c20 con 0x7f6da806a5a0 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34188/0 (socket says 172.21.15.116:34188) 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6da77fe640 1 -- 172.21.15.116:0/674460009 learned_addr learned my addr 172.21.15.116:0/674460009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.145+0000 7f6da67fc640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 0x7f6da819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8069c00 0x7f6da819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 -- 172.21.15.116:0/674460009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 msgr2=0x7f6da819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 0x7f6da819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 -- 172.21.15.116:0/674460009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8069c00 msgr2=0x7f6da819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8069c00 0x7f6da819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 -- 172.21.15.116:0/674460009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da819f230 con 0x7f6da8107810 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6da77fe640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009fd0 tx=0x7f6da000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6d8ffff640 1 -- 172.21.15.116:0/674460009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0017070 con 0x7f6da8107810 2024-03-19T02:47:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6d8ffff640 1 -- 172.21.15.116:0/674460009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6da0003e20 con 0x7f6da8107810 2024-03-19T02:47:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da819f520 con 0x7f6da8107810 2024-03-19T02:47:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6d8ffff640 1 -- 172.21.15.116:0/674460009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0010460 con 0x7f6da8107810 2024-03-19T02:47:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.146+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da81a6db0 con 0x7f6da8107810 2024-03-19T02:47:21.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.149+0000 7f6d8ffff640 1 -- 172.21.15.116:0/674460009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6da0003970 con 0x7f6da8107810 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 Processor -- start 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 -- start start 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6d8ffff640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 0x7f6d900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f39923a4b0 con 0x55f3992d70e0 2024-03-19T02:47:21.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3991e2d70 con 0x55f3992d7a90 2024-03-19T02:47:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6db0268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f399236550 con 0x55f3992e8fe0 2024-03-19T02:47:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6d8ffff640 1 -- 172.21.15.116:0/674460009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6da00cf130 con 0x7f6da8107810 2024-03-19T02:47:21.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6da67fc640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 0x7f6d900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34202/0 (socket says 172.21.15.116:34202) 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.150+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1672309425 learned_addr learned my addr 172.21.15.116:0/1672309425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1672309425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 msgr2=0x55f3992eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1672309425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 msgr2=0x55f3992e8aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e8aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1672309425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f399205210 con 0x55f3992d70e0 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6d9800b820 tx=0x7f6d9800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ad197f4e196a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6da67fc640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 0x7f6d900a4e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6da810eac0 tx=0x7f6d9c000bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.151+0000 7f6d8d7fa640 1 -- 172.21.15.116:0/1672309425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9800d950 con 0x55f3992d70e0 2024-03-19T02:47:21.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.152+0000 7f6d8d7fa640 1 -- 172.21.15.116:0/1672309425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d98019070 con 0x55f3992d70e0 2024-03-19T02:47:21.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.152+0000 7f6d8d7fa640 1 -- 172.21.15.116:0/1672309425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d980123e0 con 0x55f3992d70e0 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 -- 172.21.15.116:0/1672309425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 msgr2=0x55f3992d74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6d9800b820 tx=0x7f6d9800bce0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 -- 172.21.15.116:0/1672309425 shutdown_connections 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 --2- 172.21.15.116:0/1672309425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992d74c0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 -- 172.21.15.116:0/1672309425 >> 172.21.15.116:0/1672309425 conn(0x55f3992d1a00 msgr2=0x55f3992d1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 -- 172.21.15.116:0/1672309425 shutdown_connections 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.153+0000 7f6db0268740 1 -- 172.21.15.116:0/1672309425 wait complete. 2024-03-19T02:47:21.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 -- start start 2024-03-19T02:47:21.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 Processor -- start 2024-03-19T02:47:21.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3992ebda0 con 0x55f3992e8fe0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f3991f6310 con 0x55f3992d7a90 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f3992ebf20 con 0x55f3992d70e0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.154+0000 7f6db0268740 10 client.0 _ll_register_callbacks cb 0x55f3991ea2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34214/0 (socket says 172.21.15.116:34214) 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 -- 172.21.15.116:0/3334689750 learned_addr learned my addr 172.21.15.116:0/3334689750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da67fc640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 -- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 msgr2=0x55f3992e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 -- 172.21.15.116:0/3334689750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 msgr2=0x55f3992e65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 -- 172.21.15.116:0/3334689750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f398f4e3f0 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da67fc640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6da77fe640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6da009b340 tx=0x7f6da0003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657163085875689c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.155+0000 7f6d8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0020070 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6da0097a20 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d98019070 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6da00a3b50 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 10 client.21087 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f3990187d0 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d98019070 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6da0097100 con 0x55f3992e8fe0 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 10 client.21087 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 10 client.21087 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 20 client.21087 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3992f9830 0x55f3992fbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 4 client.21087 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.156+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3992983e0 con 0x55f3992f9830 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.157+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3992f9830 0x55f3992fbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.157+0000 7f6d8aff5640 10 client.21087 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.157+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3992f9830 0x55f3992fbc70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81b81ce1b6020e49 server_cookie=6674c7aadfefa81f in_seq=0 out_seq=0 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6d8aff5640 1 -- 172.21.15.116:0/3334689750 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3992983e0 con 0x55f3992f9830 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6d8aff5640 10 client.21087 handle_command_reply: tid=0 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 client.21087 shutdown 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 msgr2=0x55f3992e6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6da009b340 tx=0x7f6da0003da0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 shutdown_connections 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f3992f9830 0x55f3992fbc70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f3992e6ae0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 --2- 172.21.15.116:0/3334689750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6d8aff5640 0 client.21087 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.158+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 >> 172.21.15.116:0/3334689750 conn(0x55f3992d1a00 msgr2=0x55f3992d3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.159+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 shutdown_connections 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.159+0000 7f6db0268740 1 -- 172.21.15.116:0/3334689750 wait complete. 2024-03-19T02:47:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.159+0000 7f6db0268740 20 client.21087 trim_cache size 0 max 16384 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 Processor -- start 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 -- start start 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f399301580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81acb40 con 0x55f3992e8fe0 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81ae680 con 0x55f3992d7a90 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.184+0000 7f6db0268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81ad210 con 0x55f3992d70e0 2024-03-19T02:47:21.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f399301580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f399301580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34216/0 (socket says 172.21.15.116:34216) 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da77fe640 1 -- 172.21.15.116:0/1508411506 learned_addr learned my addr 172.21.15.116:0/1508411506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da67fc640 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1508411506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 msgr2=0x55f3992f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1508411506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 msgr2=0x55f399301580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f399301580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1508411506 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3992d7830 con 0x55f3992d70e0 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.185+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e8790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d98014b90 tx=0x7f6d9800f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b636cd8e370a1eca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.186+0000 7f6d8cff9640 1 -- 172.21.15.116:0/1508411506 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98026070 con 0x55f3992d70e0 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.186+0000 7f6d8cff9640 1 -- 172.21.15.116:0/1508411506 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d98019040 con 0x55f3992d70e0 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.186+0000 7f6d8cff9640 1 -- 172.21.15.116:0/1508411506 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98028800 con 0x55f3992d70e0 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- 172.21.15.116:0/1508411506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 msgr2=0x55f3992e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e8790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d98014b90 tx=0x7f6d9800f040 comp rx=0 tx=0).stop 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- 172.21.15.116:0/1508411506 shutdown_connections 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992e8fe0 0x55f399301580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 --2- 172.21.15.116:0/1508411506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992d70e0 0x55f3992e8790 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- 172.21.15.116:0/1508411506 >> 172.21.15.116:0/1508411506 conn(0x55f3992d1a00 msgr2=0x55f3992cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- 172.21.15.116:0/1508411506 shutdown_connections 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- 172.21.15.116:0/1508411506 wait complete. 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 -- start start 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.187+0000 7f6db0268740 1 Processor -- start 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992f1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992e2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81adfb0 con 0x55f3992d70e0 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81ad8e0 con 0x55f3992d7a90 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6da81aab80 con 0x55f3992e8fe0 2024-03-19T02:47:21.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992f1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6db0268740 10 client.0 _ll_register_callbacks cb 0x7f6da0009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992f1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34218/0 (socket says 172.21.15.116:34218) 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da6ffd640 1 -- 172.21.15.116:0/1416792595 learned_addr learned my addr 172.21.15.116:0/1416792595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da67fc640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992e2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 -- 172.21.15.116:0/1416792595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 msgr2=0x55f3992e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 -- 172.21.15.116:0/1416792595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 msgr2=0x55f3992f1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992f1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 -- 172.21.15.116:0/1416792595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3992e37b0 con 0x55f3992e8fe0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6da77fe640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992e2fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6da0097810 tx=0x7f6da00cc950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cbc4280c473abd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.188+0000 7f6d8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0020070 con 0x55f3992e8fe0 2024-03-19T02:47:21.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d98019040 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6da00bb0b0 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6da009b180 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 10 client.30790 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f39935fc80 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d98019040 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6da007d020 con 0x55f3992e8fe0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 10 client.30790 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 10 client.30790 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 20 client.30790 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f399360150 0x55f399362540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 4 client.30790 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3992479f0 con 0x55f399360150 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.189+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f399360150 0x55f399362540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6d8aff5640 10 client.30790 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6da6ffd640 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f399360150 0x55f399362540 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45341a6daa96f9bc server_cookie=f93b18afc8bb04a9 in_seq=0 out_seq=0 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6d8aff5640 1 -- 172.21.15.116:0/1416792595 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3992479f0 con 0x55f399360150 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6d8aff5640 10 client.30790 handle_command_reply: tid=0 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 client.30790 shutdown 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 msgr2=0x55f3992e2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992e2fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6da0097810 tx=0x7f6da00cc950 comp rx=0 tx=0).stop 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 shutdown_connections 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f3992e8fe0 0x55f3992e2fe0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f3992d7a90 0x55f3992e29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f399360150 0x55f399362540 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 --2- 172.21.15.116:0/1416792595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3992d70e0 0x55f3992f1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6d8aff5640 0 client.30790 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 >> 172.21.15.116:0/1416792595 conn(0x55f3992d1a00 msgr2=0x55f3992e6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 shutdown_connections 2024-03-19T02:47:21.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 1 -- 172.21.15.116:0/1416792595 wait complete. 2024-03-19T02:47:21.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.190+0000 7f6db0268740 20 client.30790 trim_cache size 0 max 16384 2024-03-19T02:47:21.197 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 msgr2=0x7f6d900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 0x7f6d900a4e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6da810eac0 tx=0x7f6d9c000bc0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 msgr2=0x7f6da819eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009fd0 tx=0x7f6da000b260 comp rx=0 tx=0).stop 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 shutdown_connections 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6d900a2970 0x7f6d900a4e30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da8107810 0x7f6da819eb00 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6da806a5a0 0x7f6da819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 --2- 172.21.15.116:0/674460009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da8069c00 0x7f6da819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 >> 172.21.15.116:0/674460009 conn(0x7f6da80774b0 msgr2=0x7f6da810cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 shutdown_connections 2024-03-19T02:47:21.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.191+0000 7f6dad67e640 1 -- 172.21.15.116:0/674460009 wait complete. 2024-03-19T02:47:21.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:20 smithi116 ceph-mon[32263]: pgmap v616: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:21.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:20 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3407480351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:21.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:47:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:20 smithi154 ceph-mon[35693]: pgmap v616: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:20 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3407480351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:21.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45454 5.98 MB/sec execute 283 sec latency 166.974 ms 2024-03-19T02:47:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 -- 172.21.15.116:0/2006136255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 msgr2=0x7f8a5410e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/2006136255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a5410e690 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009cd0 tx=0x7f8a4c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 -- 172.21.15.116:0/2006136255 shutdown_connections 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/2006136255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a5410e690 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/2006136255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/2006136255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 0x7f8a54107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 -- 172.21.15.116:0/2006136255 >> 172.21.15.116:0/2006136255 conn(0x7f8a540fd2f0 msgr2=0x7f8a540ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 -- 172.21.15.116:0/2006136255 shutdown_connections 2024-03-19T02:47:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.343+0000 7f8a5b91b640 1 -- 172.21.15.116:0/2006136255 wait complete. 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 Processor -- start 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 -- start start 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 0x7f8a5419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54111a90 con 0x7f8a54102180 2024-03-19T02:47:21.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54111910 con 0x7f8a541077d0 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a5b91b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54111c10 con 0x7f8a541017c0 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a58eb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a58eb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34238/0 (socket says 172.21.15.116:34238) 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a58eb6640 1 -- 172.21.15.116:0/1263376323 learned_addr learned my addr 172.21.15.116:0/1263376323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.344+0000 7f8a59eb8640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 0x7f8a5419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 -- 172.21.15.116:0/1263376323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 msgr2=0x7f8a54197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 -- 172.21.15.116:0/1263376323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 msgr2=0x7f8a5419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 0x7f8a5419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 -- 172.21.15.116:0/1263376323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5419cf60 con 0x7f8a54102180 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400ea60 tx=0x7f8a4400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a4a7fc640 1 -- 172.21.15.116:0/1263376323 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400cde0 con 0x7f8a54102180 2024-03-19T02:47:21.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a4a7fc640 1 -- 172.21.15.116:0/1263376323 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a440035f0 con 0x7f8a54102180 2024-03-19T02:47:21.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a4a7fc640 1 -- 172.21.15.116:0/1263376323 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44010690 con 0x7f8a54102180 2024-03-19T02:47:21.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5419d250 con 0x7f8a54102180 2024-03-19T02:47:21.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.345+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a541a4ae0 con 0x7f8a54102180 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a4a7fc640 1 -- 172.21.15.116:0/1263376323 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8a440037a0 con 0x7f8a54102180 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a4a7fc640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 0x7f8a400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a5e5e6740 1 Processor -- start 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 0x7f8a400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a5e5e6740 1 -- start start 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.348+0000 7f8a4a7fc640 1 -- 172.21.15.116:0/1263376323 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f8a440cf500 con 0x7f8a54102180 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bb62d0 0x5604b0bb66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5604b0b194b0 con 0x5604b0bb6bf0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5604b0ac1d70 con 0x5604b0bb62d0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604b0b15550 con 0x5604b0bbf7d0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 0x7f8a400a4e30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009f40 tx=0x7f8a50005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34244/0 (socket says 172.21.15.116:34244) 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 -- 172.21.15.116:0/478430229 learned_addr learned my addr 172.21.15.116:0/478430229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a59eb8640 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a596b7640 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bb62d0 0x5604b0bb66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 -- 172.21.15.116:0/478430229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bbf7d0 msgr2=0x5604b0bc1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 -- 172.21.15.116:0/478430229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bb62d0 msgr2=0x5604b0bb66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bb62d0 0x5604b0bb66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 -- 172.21.15.116:0/478430229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b0ae4210 con 0x5604b0bb6bf0 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.349+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8a44097be0 tx=0x7f8a4409b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306ce7dcba219035 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.350+0000 7f8a3f7fe640 1 -- 172.21.15.116:0/478430229 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4409be80 con 0x5604b0bb6bf0 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.350+0000 7f8a3f7fe640 1 -- 172.21.15.116:0/478430229 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a4409b780 con 0x5604b0bb6bf0 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.350+0000 7f8a3f7fe640 1 -- 172.21.15.116:0/478430229 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a440ccc80 con 0x5604b0bb6bf0 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.350+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/478430229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 msgr2=0x5604b0bbf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.350+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8a44097be0 tx=0x7f8a4409b2c0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/478430229 shutdown_connections 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbf290 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/478430229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bb62d0 0x5604b0bb66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/478430229 >> 172.21.15.116:0/478430229 conn(0x5604b0bb0a00 msgr2=0x5604b0bb0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/478430229 shutdown_connections 2024-03-19T02:47:21.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/478430229 wait complete. 2024-03-19T02:47:21.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- start start 2024-03-19T02:47:21.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 Processor -- start 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bbd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5604b0bc25c0 con 0x5604b0bb62d0 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5604b0ad5310 con 0x5604b0bbf7d0 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.351+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5604b0bc2740 con 0x5604b0bb6bf0 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a5e5e6740 10 client.0 _ll_register_callbacks cb 0x5604b0ac92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34256/0 (socket says 172.21.15.116:34256) 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 -- 172.21.15.116:0/3764354193 learned_addr learned my addr 172.21.15.116:0/3764354193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a59eb8640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bbd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 -- 172.21.15.116:0/3764354193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 msgr2=0x5604b0bbcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 -- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 msgr2=0x5604b0bbd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bbd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 -- 172.21.15.116:0/3764354193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b082d3f0 con 0x5604b0bb62d0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a596b7640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8a50003940 tx=0x7f8a50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbc597b1a3e8df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000fe10 con 0x5604b0bb62d0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f8a50028440 con 0x5604b0bb62d0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4409b780 con 0x5604b0bb62d0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.352+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8a50031050 con 0x5604b0bb62d0 2024-03-19T02:47:21.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 10 client.21105 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5604b08f77d0 con 0x5604b0bb62d0 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a4409b780 con 0x5604b0bb62d0 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8a50026020 con 0x5604b0bb62d0 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 10 client.21105 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 10 client.21105 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 20 client.21105 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0bd20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 4 client.21105 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604b0b773e0 con 0x5604b0bcfc40 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.353+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0bd20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.354+0000 7f8a3cff9640 10 client.21105 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.354+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0bd20b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=320a8057d6b6d811 server_cookie=ff2b090348113d63 in_seq=0 out_seq=0 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a3cff9640 1 -- 172.21.15.116:0/3764354193 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604b0b773e0 con 0x5604b0bcfc40 2024-03-19T02:47:21.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a3cff9640 10 client.21105 handle_command_reply: tid=0 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 client.21105 shutdown 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 msgr2=0x5604b0bbc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8a50003940 tx=0x7f8a50004730 comp rx=0 tx=0).stop 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 shutdown_connections 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0bd20b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a3cff9640 0 client.21105 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bbd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/3764354193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bbc8a0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 >> 172.21.15.116:0/3764354193 conn(0x5604b0bb0a00 msgr2=0x5604b0bb2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 shutdown_connections 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/3764354193 wait complete. 2024-03-19T02:47:21.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.355+0000 7f8a5e5e6740 20 client.21105 trim_cache size 0 max 16384 2024-03-19T02:47:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 Processor -- start 2024-03-19T02:47:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 -- start start 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bd7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541aed90 con 0x5604b0bb62d0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b08d0 con 0x5604b0bbf7d0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541af460 con 0x5604b0bb6bf0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34264/0 (socket says 172.21.15.116:34264) 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 -- 172.21.15.116:0/1269535052 learned_addr learned my addr 172.21.15.116:0/1269535052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a59eb8640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bd7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 -- 172.21.15.116:0/1269535052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 msgr2=0x5604b0bbd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 -- 172.21.15.116:0/1269535052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 msgr2=0x5604b0bd7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bd7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.373+0000 7f8a596b7640 1 -- 172.21.15.116:0/1269535052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b0bac2c0 con 0x5604b0bb62d0 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.374+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.374+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009730 tx=0x7f8a500092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426092921aec5ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.374+0000 7f8a3effd640 1 -- 172.21.15.116:0/1269535052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000f850 con 0x5604b0bb62d0 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.374+0000 7f8a3effd640 1 -- 172.21.15.116:0/1269535052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a5000e620 con 0x5604b0bb62d0 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.374+0000 7f8a3effd640 1 -- 172.21.15.116:0/1269535052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50028730 con 0x5604b0bb62d0 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1269535052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 msgr2=0x5604b0bcb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009730 tx=0x7f8a500092a0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1269535052 shutdown_connections 2024-03-19T02:47:21.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bd7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bbd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1269535052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bcb360 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1269535052 >> 172.21.15.116:0/1269535052 conn(0x5604b0bb0a00 msgr2=0x5604b0bafba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1269535052 shutdown_connections 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1269535052 wait complete. 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 -- start start 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 Processor -- start 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.375+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bc2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541b0200 con 0x5604b0bb62d0 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541afb30 con 0x5604b0bbf7d0 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541acd90 con 0x5604b0bb6bf0 2024-03-19T02:47:21.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34272/0 (socket says 172.21.15.116:34272) 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 -- 172.21.15.116:0/1059004972 learned_addr learned my addr 172.21.15.116:0/1059004972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 10 client.0 _ll_register_callbacks cb 0x7f8a50009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bc2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a59eb8640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 -- 172.21.15.116:0/1059004972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 msgr2=0x5604b0bc2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bc2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 -- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 msgr2=0x5604b0bc3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 -- 172.21.15.116:0/1059004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b0bc3a30 con 0x5604b0bb62d0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a596b7640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000e430 tx=0x7f8a50005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46c4b4231d3e10c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50005500 con 0x5604b0bb62d0 2024-03-19T02:47:21.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f8a5000e7d0 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.376+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5000e620 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8a50048050 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 10 client.21111 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5604b0c35da0 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a50004730 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f8a50034070 con 0x5604b0bb62d0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 10 client.21111 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 10 client.21111 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 20 client.21111 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0c383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 4 client.21111 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8a5000e620 con 0x5604b0bcfc40 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.377+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0c383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a3cff9640 10 client.21111 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a58eb6640 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0c383c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3eae87d5edf68974 server_cookie=3dfafeaa5adf9fa9 in_seq=0 out_seq=0 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a3cff9640 1 -- 172.21.15.116:0/1059004972 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8a5000e620 con 0x5604b0bcfc40 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a3cff9640 10 client.21111 handle_command_reply: tid=0 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 client.21111 shutdown 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 msgr2=0x5604b0bc79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000e430 tx=0x7f8a50005970 comp rx=0 tx=0).stop 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 shutdown_connections 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5604b0bcfc40 0x5604b0c383c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604b0bbf7d0 0x5604b0bc3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5604b0bb6bf0 0x5604b0bc2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a3cff9640 0 client.21111 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 --2- 172.21.15.116:0/1059004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604b0bb62d0 0x5604b0bc79c0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 >> 172.21.15.116:0/1059004972 conn(0x5604b0bb0a00 msgr2=0x5604b0bb1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 shutdown_connections 2024-03-19T02:47:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 1 -- 172.21.15.116:0/1059004972 wait complete. 2024-03-19T02:47:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.378+0000 7f8a5e5e6740 20 client.21111 trim_cache size 0 max 16384 2024-03-19T02:47:21.386 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 msgr2=0x7f8a400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 0x7f8a400a4e30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009f40 tx=0x7f8a50005f90 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 msgr2=0x7f8a541984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400ea60 tx=0x7f8a4400ef20 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 shutdown_connections 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8a400a2970 0x7f8a400a4e30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a541077d0 0x7f8a5419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a54102180 0x7f8a541984b0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 --2- 172.21.15.116:0/1263376323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a541017c0 0x7f8a54197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 >> 172.21.15.116:0/1263376323 conn(0x7f8a540fd2f0 msgr2=0x7f8a540fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 shutdown_connections 2024-03-19T02:47:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.379+0000 7f8a5b91b640 1 -- 172.21.15.116:0/1263376323 wait complete. 2024-03-19T02:47:21.394 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:47:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:20 smithi162 ceph-mon[35761]: pgmap v616: 129 pgs: 129 active+clean; 960 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 140 op/s 2024-03-19T02:47:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:20 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3407480351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:21.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 -- 172.21.15.116:0/3331407301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45741066b0 msgr2=0x7f4574106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 --2- 172.21.15.116:0/3331407301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45741066b0 0x7f4574106a90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f4570009930 tx=0x7f457002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 -- 172.21.15.116:0/3331407301 shutdown_connections 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 --2- 172.21.15.116:0/3331407301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4574101060 0x7f457410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 --2- 172.21.15.116:0/3331407301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741006a0 0x7f4574100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 --2- 172.21.15.116:0/3331407301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45741066b0 0x7f4574106a90 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 -- 172.21.15.116:0/3331407301 >> 172.21.15.116:0/3331407301 conn(0x7f45740778d0 msgr2=0x7f4574077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 -- 172.21.15.116:0/3331407301 shutdown_connections 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.534+0000 7f457b887640 1 -- 172.21.15.116:0/3331407301 wait complete. 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 Processor -- start 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 -- start start 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 0x7f4574197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 0x7f457419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4574110970 con 0x7f4574101060 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45741107f0 con 0x7f45741006a0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f457b887640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4574110af0 con 0x7f45741066b0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f4578e22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f4578e22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34286/0 (socket says 172.21.15.116:34286) 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f4578e22640 1 -- 172.21.15.116:0/3748977705 learned_addr learned my addr 172.21.15.116:0/3748977705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 0x7f457419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.535+0000 7f4579623640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 0x7f4574197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 -- 172.21.15.116:0/3748977705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 msgr2=0x7f4574197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 0x7f4574197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 -- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 msgr2=0x7f4574198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 -- 172.21.15.116:0/3748977705 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457419cfb0 con 0x7f45741066b0 2024-03-19T02:47:21.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579623640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 0x7f4574197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4578e22640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 0x7f457419c880 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f456a7fc640 1 -- 172.21.15.116:0/3748977705 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c00ce20 con 0x7f45741066b0 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457419d2a0 con 0x7f45741066b0 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f456a7fc640 1 -- 172.21.15.116:0/3748977705 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f456c005020 con 0x7f45741066b0 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45741a4b30 con 0x7f45741066b0 2024-03-19T02:47:21.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.536+0000 7f456a7fc640 1 -- 172.21.15.116:0/3748977705 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c010690 con 0x7f45741066b0 2024-03-19T02:47:21.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.539+0000 7f456a7fc640 1 -- 172.21.15.116:0/3748977705 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f456c004a40 con 0x7f45741066b0 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.539+0000 7f457e46a740 1 Processor -- start 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 -- start start 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 0x562a41b1b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b1dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a41a754b0 con 0x562a41b122d0 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a41a1dd70 con 0x562a41b12bf0 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f457e46a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a41a71550 con 0x562a41b1b7d0 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f456a7fc640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 0x7f45600a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f4578e22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f4578e22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34292/0 (socket says 172.21.15.116:34292) 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f4578e22640 1 -- 172.21.15.116:0/2093035685 learned_addr learned my addr 172.21.15.116:0/2093035685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f4579623640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 0x562a41b1b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.540+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 0x7f45600a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4578e22640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b1dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f456a7fc640 1 -- 172.21.15.116:0/3748977705 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f456c0cf880 con 0x7f45741066b0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 -- 172.21.15.116:0/2093035685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 msgr2=0x562a41b1dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b1dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 -- 172.21.15.116:0/2093035685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 msgr2=0x562a41b126b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 -- 172.21.15.116:0/2093035685 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a41a40210 con 0x562a41b12bf0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4578e22640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579e24640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 0x7f45600a4eb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f456c09b7a0 tx=0x7f456c090f30 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.541+0000 7f4579623640 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 0x562a41b1b290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4570009230 tx=0x7f4570005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3dce12ea51fb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.542+0000 7f455f7fe640 1 -- 172.21.15.116:0/2093035685 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570044020 con 0x562a41b12bf0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.542+0000 7f455f7fe640 1 -- 172.21.15.116:0/2093035685 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f457002cc90 con 0x562a41b12bf0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.542+0000 7f455f7fe640 1 -- 172.21.15.116:0/2093035685 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700046c0 con 0x562a41b12bf0 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- 172.21.15.116:0/2093035685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 msgr2=0x562a41b1b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 0x562a41b1b290 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4570009230 tx=0x7f4570005e50 comp rx=0 tx=0).stop 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- 172.21.15.116:0/2093035685 shutdown_connections 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b1dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b12bf0 0x562a41b1b290 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 --2- 172.21.15.116:0/2093035685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b122d0 0x562a41b126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- 172.21.15.116:0/2093035685 >> 172.21.15.116:0/2093035685 conn(0x562a41b0ca00 msgr2=0x562a41b0cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- 172.21.15.116:0/2093035685 shutdown_connections 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- 172.21.15.116:0/2093035685 wait complete. 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 -- start start 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 Processor -- start 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.543+0000 7f457e46a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b18d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a41b1e560 con 0x562a41b12bf0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562a41a31310 con 0x562a41b1b7d0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a41b1e6e0 con 0x562a41b122d0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4579623640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51770/0 (socket says 172.21.15.116:51770) 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 -- 172.21.15.116:0/1805797138 learned_addr learned my addr 172.21.15.116:0/1805797138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b18d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f457e46a740 10 client.0 _ll_register_callbacks cb 0x562a41a252d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 -- 172.21.15.116:0/1805797138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 msgr2=0x562a41b18d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b18d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 -- 172.21.15.116:0/1805797138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 msgr2=0x562a41b187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b187f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 -- 172.21.15.116:0/1805797138 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a417893f0 con 0x562a41b122d0 2024-03-19T02:47:21.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4579623640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f4578e22640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f456400f8f0 tx=0x7f4564004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efe8d1387c602c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.544+0000 7f455cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456401a040 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f4564005220 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457002cc90 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f456400fe70 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 10 client.30820 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562a418537d0 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f457002cc90 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f456400b6c0 con 0x562a41b122d0 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 10 client.30820 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 10 client.30820 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 20 client.30820 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.545+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b343d0 0x562a41b36840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.546+0000 7f457e46a740 4 client.30820 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.546+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a41ad33e0 con 0x562a41b343d0 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.546+0000 7f4579623640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b343d0 0x562a41b36840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.546+0000 7f455cff9640 10 client.30820 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.546+0000 7f4579623640 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b343d0 0x562a41b36840 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf7ddc4ec220ed73 server_cookie=2e7eb56b47772691 in_seq=0 out_seq=0 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f455cff9640 1 -- 172.21.15.116:0/1805797138 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a41ad33e0 con 0x562a41b343d0 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f455cff9640 10 client.30820 handle_command_reply: tid=0 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 client.30820 shutdown 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 msgr2=0x562a41b1b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f456400f8f0 tx=0x7f4564004730 comp rx=0 tx=0).stop 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 shutdown_connections 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b343d0 0x562a41b36840 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b18d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f455cff9640 0 client.30820 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:21.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 --2- 172.21.15.116:0/1805797138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b1b200 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 >> 172.21.15.116:0/1805797138 conn(0x562a41b0ca00 msgr2=0x562a41b0e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 shutdown_connections 2024-03-19T02:47:21.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 1 -- 172.21.15.116:0/1805797138 wait complete. 2024-03-19T02:47:21.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.547+0000 7f457e46a740 20 client.30820 trim_cache size 0 max 16384 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 Processor -- start 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 -- start start 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b122d0 0x562a41b17950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b1af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f455800ff60 con 0x562a41b12bf0 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4558011aa0 con 0x562a41b122d0 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f457e46a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4558010630 con 0x562a41b1b7d0 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4579623640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b1af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51782/0 (socket says 172.21.15.116:51782) 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 -- 172.21.15.116:0/2170124382 learned_addr learned my addr 172.21.15.116:0/2170124382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b122d0 0x562a41b17950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 -- 172.21.15.116:0/2170124382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b122d0 msgr2=0x562a41b17950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b122d0 0x562a41b17950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 -- 172.21.15.116:0/2170124382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 msgr2=0x562a41b1af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b1af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4578e22640 1 -- 172.21.15.116:0/2170124382 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a41b19090 con 0x562a41b1b7d0 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.565+0000 7f4579623640 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b1af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.566+0000 7f4578e22640 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f456400f440 tx=0x7f45640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d2b5e5e1fd60f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.566+0000 7f455effd640 1 -- 172.21.15.116:0/2170124382 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456400b690 con 0x562a41b1b7d0 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.566+0000 7f455effd640 1 -- 172.21.15.116:0/2170124382 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f456400b830 con 0x562a41b1b7d0 2024-03-19T02:47:21.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.566+0000 7f455effd640 1 -- 172.21.15.116:0/2170124382 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564004a90 con 0x562a41b1b7d0 2024-03-19T02:47:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- 172.21.15.116:0/2170124382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 msgr2=0x562a41b3c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f456400f440 tx=0x7f45640046c0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- 172.21.15.116:0/2170124382 shutdown_connections 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b1b7d0 0x562a41b3c150 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b1af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 --2- 172.21.15.116:0/2170124382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b122d0 0x562a41b17950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- 172.21.15.116:0/2170124382 >> 172.21.15.116:0/2170124382 conn(0x562a41b0ca00 msgr2=0x562a41b0cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- 172.21.15.116:0/2170124382 shutdown_connections 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- 172.21.15.116:0/2170124382 wait complete. 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 -- start start 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.567+0000 7f457e46a740 1 Processor -- start 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45580113d0 con 0x562a41b12bf0 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4558010d00 con 0x562a41b1b7d0 2024-03-19T02:47:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f455800df60 con 0x562a41b122d0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f457e46a740 10 client.0 _ll_register_callbacks cb 0x7f4564010510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4579623640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51790/0 (socket says 172.21.15.116:51790) 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4579623640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34314/0 (socket says 172.21.15.116:34314) 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 -- 172.21.15.116:0/758484197 learned_addr learned my addr 172.21.15.116:0/758484197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 -- 172.21.15.116:0/758484197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 msgr2=0x562a41b927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 -- 172.21.15.116:0/758484197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 msgr2=0x562a41b2c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4578e22640 1 -- 172.21.15.116:0/758484197 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a41b92d70 con 0x562a41b122d0 2024-03-19T02:47:21.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.568+0000 7f4579623640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f4578e22640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4564010bf0 tx=0x7f456400f4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c7c93231b4e220c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f455cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564035040 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f4564005410 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f456400b830 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f4564005610 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f457e46a740 10 client.30826 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562a41b9a100 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.569+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f456400b830 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f456403b070 con 0x562a41b122d0 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 10 client.30826 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 10 client.30826 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 20 client.30826 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b9a660 0x562a41b9ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 4 client.30826 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a41b22100 con 0x562a41b9a660 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f4579623640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b9a660 0x562a41b9ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f455cff9640 10 client.30826 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.570+0000 7f4579623640 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b9a660 0x562a41b9ca50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5998cb0896116c75 server_cookie=cefdeef3c6b15358 in_seq=0 out_seq=0 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f455cff9640 1 -- 172.21.15.116:0/758484197 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a41b22100 con 0x562a41b9a660 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f455cff9640 10 client.30826 handle_command_reply: tid=0 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 client.30826 shutdown 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 msgr2=0x562a41b2be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4564010bf0 tx=0x7f456400f4e0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 shutdown_connections 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562a41b9a660 0x562a41b9ca50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562a41b1b7d0 0x562a41b927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f455cff9640 0 client.30826 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a41b12bf0 0x562a41b2c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 --2- 172.21.15.116:0/758484197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a41b122d0 0x562a41b2be20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 >> 172.21.15.116:0/758484197 conn(0x562a41b0ca00 msgr2=0x562a41b0bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 shutdown_connections 2024-03-19T02:47:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 1 -- 172.21.15.116:0/758484197 wait complete. 2024-03-19T02:47:21.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457e46a740 20 client.30826 trim_cache size 0 max 16384 2024-03-19T02:47:21.577 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 msgr2=0x7f45600a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 0x7f45600a4eb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f456c09b7a0 tx=0x7f456c090f30 comp rx=0 tx=0).stop 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.571+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 msgr2=0x7f457419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 0x7f457419c880 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f456c009fd0 tx=0x7f456c00ef10 comp rx=0 tx=0).stop 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 shutdown_connections 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45600a29f0 0x7f45600a4eb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45741066b0 0x7f457419c880 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4574101060 0x7f4574198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 --2- 172.21.15.116:0/3748977705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45741006a0 0x7f4574197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 >> 172.21.15.116:0/3748977705 conn(0x7f45740778d0 msgr2=0x7f4574068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 shutdown_connections 2024-03-19T02:47:21.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.572+0000 7f457b887640 1 -- 172.21.15.116:0/3748977705 wait complete. 2024-03-19T02:47:21.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.725+0000 7f6f289b2640 1 -- 172.21.15.116:0/1329638429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24100a00 msgr2=0x7f6f24100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.725+0000 7f6f289b2640 1 --2- 172.21.15.116:0/1329638429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24100a00 0x7f6f24100e80 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009900 tx=0x7f6f1802bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.725+0000 7f6f289b2640 1 -- 172.21.15.116:0/1329638429 shutdown_connections 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 --2- 172.21.15.116:0/1329638429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f2410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 --2- 172.21.15.116:0/1329638429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24100a00 0x7f6f24100e80 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 --2- 172.21.15.116:0/1329638429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24105680 0x7f6f24105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 -- 172.21.15.116:0/1329638429 >> 172.21.15.116:0/1329638429 conn(0x7f6f24077700 msgr2=0x7f6f24077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 -- 172.21.15.116:0/1329638429 shutdown_connections 2024-03-19T02:47:21.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 -- 172.21.15.116:0/1329638429 wait complete. 2024-03-19T02:47:21.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 Processor -- start 2024-03-19T02:47:21.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.726+0000 7f6f289b2640 1 -- start start 2024-03-19T02:47:21.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f240fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24105680 0x7f6f240fef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f24110930 con 0x7f6f24105680 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f241107b0 con 0x7f6f24100a00 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f289b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f24110ab0 con 0x7f6f241013c0 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f21d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f240fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f22d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24105680 0x7f6f240fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45770/0 (socket says 172.21.15.116:45770) 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 -- 172.21.15.116:0/4276476591 learned_addr learned my addr 172.21.15.116:0/4276476591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 -- 172.21.15.116:0/4276476591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 msgr2=0x7f6f240fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f240fe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 -- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24105680 msgr2=0x7f6f240fef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24105680 0x7f6f240fef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 -- 172.21.15.116:0/4276476591 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f241a6440 con 0x7f6f24100a00 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f240fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.727+0000 7f6f2259c640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e9c0 tx=0x7f6f1400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.728+0000 7f6f137fe640 1 -- 172.21.15.116:0/4276476591 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400cd80 con 0x7f6f24100a00 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.728+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f241a6730 con 0x7f6f24100a00 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.728+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f241a6ca0 con 0x7f6f24100a00 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.728+0000 7f6f137fe640 1 -- 172.21.15.116:0/4276476591 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f140035f0 con 0x7f6f24100a00 2024-03-19T02:47:21.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.728+0000 7f6f137fe640 1 -- 172.21.15.116:0/4276476591 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14010690 con 0x7f6f24100a00 2024-03-19T02:47:21.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.731+0000 7f6f137fe640 1 -- 172.21.15.116:0/4276476591 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6f140037a0 con 0x7f6f24100a00 2024-03-19T02:47:21.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.731+0000 7f6f2b5a0740 1 Processor -- start 2024-03-19T02:47:21.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.731+0000 7f6f2b5a0740 1 -- start start 2024-03-19T02:47:21.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.731+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbacf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.731+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbadabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f22d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbacf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34342/0 (socket says 172.21.15.116:34342) 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 -- 172.21.15.116:0/2834329730 learned_addr learned my addr 172.21.15.116:0/2834329730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b9cba324b0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b9cb9dad70 con 0x55b9cbacf2d0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9cba2e550 con 0x55b9cbad87d0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbadabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f137fe640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 0x7f6f0c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 0x7f6f0c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 -- 172.21.15.116:0/2834329730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 msgr2=0x55b9cbadabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbadabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 -- 172.21.15.116:0/2834329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 msgr2=0x55b9cbacf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbacf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 -- 172.21.15.116:0/2834329730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9cb9fd210 con 0x55b9cbacfbf0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f137fe640 1 -- 172.21.15.116:0/4276476591 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6f140cf4f0 con 0x7f6f24100a00 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f2259c640 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f6f1409b410 tx=0x7f6f1401b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9352658fc001004 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f10ff9640 1 -- 172.21.15.116:0/2834329730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14003a90 con 0x55b9cbacfbf0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.732+0000 7f6f10ff9640 1 -- 172.21.15.116:0/2834329730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1409be10 con 0x55b9cbacfbf0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f10ff9640 1 -- 172.21.15.116:0/2834329730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f140a3eb0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 0x7f6f0c0a4eb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c005fd0 tx=0x7f6f1c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 msgr2=0x55b9cbad8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f6f1409b410 tx=0x7f6f1401b6d0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 shutdown_connections 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbadabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad8290 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/2834329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbacf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 >> 172.21.15.116:0/2834329730 conn(0x55b9cbac9a00 msgr2=0x55b9cbac9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 shutdown_connections 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.733+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/2834329730 wait complete. 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 -- start start 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 Processor -- start 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b9cbadb560 con 0x55b9cbacf2d0 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b9cb9ee310 con 0x55b9cbacfbf0 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9cbadb6e0 con 0x55b9cbad87d0 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2b5a0740 10 client.0 _ll_register_callbacks cb 0x55b9cb9e22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34344/0 (socket says 172.21.15.116:34344) 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f22d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 -- 172.21.15.116:0/218495750 learned_addr learned my addr 172.21.15.116:0/218495750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 -- 172.21.15.116:0/218495750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 msgr2=0x55b9cbad61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 -- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 msgr2=0x55b9cbad5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f2259c640 1 -- 172.21.15.116:0/218495750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9cb7463f0 con 0x55b9cbacf2d0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.734+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2259c640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f6f140980e0 tx=0x7f6f1400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f547923487c0de server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14003a90 con 0x55b9cbacf2d0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6f1409f5a0 con 0x55b9cbacf2d0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f1409be10 con 0x55b9cbacf2d0 2024-03-19T02:47:21.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6f140ccc90 con 0x55b9cbacf2d0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 10 client.21141 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b9cb8107d0 con 0x55b9cbacf2d0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1409be10 con 0x55b9cbacf2d0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6f1409ab60 con 0x55b9cbacf2d0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 10 client.21141 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 10 client.21141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 20 client.21141 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbae8900 0x55b9cbaeacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 4 client.21141 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.735+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9cba903e0 con 0x55b9cbae8900 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.736+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbae8900 0x55b9cbaeacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.736+0000 7f6f09ffb640 10 client.21141 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.736+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbae8900 0x55b9cbaeacf0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb47bad6775b2682 server_cookie=246307cbb19e0bf2 in_seq=0 out_seq=0 2024-03-19T02:47:21.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f09ffb640 1 -- 172.21.15.116:0/218495750 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b9cba903e0 con 0x55b9cbae8900 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f09ffb640 10 client.21141 handle_command_reply: tid=0 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f2b5a0740 1 client.21141 shutdown 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 msgr2=0x55b9cbad5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f6f140980e0 tx=0x7f6f1400cf60 comp rx=0 tx=0).stop 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.738+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 shutdown_connections 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbae8900 0x55b9cbaeacf0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f09ffb640 0 client.21141 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/218495750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad5720 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 >> 172.21.15.116:0/218495750 conn(0x55b9cbac9a00 msgr2=0x55b9cbacb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 shutdown_connections 2024-03-19T02:47:21.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/218495750 wait complete. 2024-03-19T02:47:21.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.739+0000 7f6f2b5a0740 20 client.21141 trim_cache size 0 max 16384 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 Processor -- start 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 -- start start 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0400fec0 con 0x55b9cbacf2d0 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04011a00 con 0x55b9cbacfbf0 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04010590 con 0x55b9cbad87d0 2024-03-19T02:47:21.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f22d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34354/0 (socket says 172.21.15.116:34354) 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f21d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f2259c640 1 -- 172.21.15.116:0/666105330 learned_addr learned my addr 172.21.15.116:0/666105330 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.759+0000 7f6f22d9d640 1 -- 172.21.15.116:0/666105330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 msgr2=0x55b9cbad5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f22d9d640 1 -- 172.21.15.116:0/666105330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 msgr2=0x55b9cbad7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f22d9d640 1 -- 172.21.15.116:0/666105330 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9cbac9830 con 0x55b9cbad87d0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f2259c640 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad45d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c013400 tx=0x7f6f1c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a5cdac302cb5ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f0bfff640 1 -- 172.21.15.116:0/666105330 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c004b20 con 0x55b9cbad87d0 2024-03-19T02:47:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f0bfff640 1 -- 172.21.15.116:0/666105330 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1c004cc0 con 0x55b9cbad87d0 2024-03-19T02:47:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.760+0000 7f6f0bfff640 1 -- 172.21.15.116:0/666105330 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c005310 con 0x55b9cbad87d0 2024-03-19T02:47:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/666105330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 msgr2=0x55b9cbad45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad45d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c013400 tx=0x7f6f1c019000 comp rx=0 tx=0).stop 2024-03-19T02:47:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/666105330 shutdown_connections 2024-03-19T02:47:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbad45d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacfbf0 0x55b9cbad5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/666105330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacf2d0 0x55b9cbad7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/666105330 >> 172.21.15.116:0/666105330 conn(0x55b9cbac9a00 msgr2=0x55b9cbad3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/666105330 shutdown_connections 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/666105330 wait complete. 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 -- start start 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.761+0000 7f6f2b5a0740 1 Processor -- start 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbaedf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbaeb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbaeb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04011330 con 0x55b9cbacfbf0 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04010c60 con 0x55b9cbacf2d0 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0400dec0 con 0x55b9cbad87d0 2024-03-19T02:47:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2b5a0740 10 client.0 _ll_register_callbacks cb 0x7f6f1c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbaeb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbaedf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbaedf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45794/0 (socket says 172.21.15.116:45794) 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f2259c640 1 -- 172.21.15.116:0/4250446293 learned_addr learned my addr 172.21.15.116:0/4250446293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f22d9d640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbaeb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 -- 172.21.15.116:0/4250446293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 msgr2=0x55b9cbaeb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbaeb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 -- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 msgr2=0x55b9cbaedf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbaedf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 -- 172.21.15.116:0/4250446293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9cbadabc0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f21d9b640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbaeb300 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f1802c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6406555827826248 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.762+0000 7f6f09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18049b10 con 0x55b9cbacfbf0 2024-03-19T02:47:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f6f18049cb0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f1c004cc0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6f1805a050 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 10 client.21147 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b9cbb4e6a0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1c004cc0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f6f180035f0 con 0x55b9cbacfbf0 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 10 client.21147 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 10 client.21147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 20 client.21147 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbb4edd0 0x55b9cbb511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 4 client.21147 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:21.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.763+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9cba3f9f0 con 0x55b9cbb4edd0 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.764+0000 7f6f2259c640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbb4edd0 0x55b9cbb511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.764+0000 7f6f2259c640 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbb4edd0 0x55b9cbb511f0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc7e69b5eca62345 server_cookie=3fdcfee80c536d84 in_seq=0 out_seq=0 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.764+0000 7f6f09ffb640 10 client.21147 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f09ffb640 1 -- 172.21.15.116:0/4250446293 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9cba3f9f0 con 0x55b9cbb4edd0 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f09ffb640 10 client.21147 handle_command_reply: tid=0 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f2b5a0740 1 client.21147 shutdown 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 msgr2=0x55b9cbaeb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbaeb300 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f6f18000c00 tx=0x7f6f1802c960 comp rx=0 tx=0).stop 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.765+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 shutdown_connections 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55b9cbb4edd0 0x55b9cbb511f0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9cbad87d0 0x55b9cbaeb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f09ffb640 0 client.21147 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9cbacfbf0 0x55b9cbaeb300 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 --2- 172.21.15.116:0/4250446293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b9cbacf2d0 0x55b9cbaedf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 >> 172.21.15.116:0/4250446293 conn(0x55b9cbac9a00 msgr2=0x55b9cbac8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 shutdown_connections 2024-03-19T02:47:21.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 1 -- 172.21.15.116:0/4250446293 wait complete. 2024-03-19T02:47:21.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.766+0000 7f6f2b5a0740 20 client.21147 trim_cache size 0 max 16384 2024-03-19T02:47:21.772 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 msgr2=0x7f6f0c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 0x7f6f0c0a4eb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c005fd0 tx=0x7f6f1c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 msgr2=0x7f6f240fe490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e9c0 tx=0x7f6f1400ee80 comp rx=0 tx=0).stop 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 shutdown_connections 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6f0c0a29f0 0x7f6f0c0a4eb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f24105680 0x7f6f240fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f241013c0 0x7f6f240fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 --2- 172.21.15.116:0/4276476591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f24100a00 0x7f6f240fe490 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.767+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 >> 172.21.15.116:0/4276476591 conn(0x7f6f24077700 msgr2=0x7f6f2406a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.768+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 shutdown_connections 2024-03-19T02:47:21.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:21.768+0000 7f6f289b2640 1 -- 172.21.15.116:0/4276476591 wait complete. 2024-03-19T02:47:22.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45561 5.97 MB/sec execute 284 sec latency 140.867 ms 2024-03-19T02:47:22.782 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:47:22.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:47:22.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 -- 172.21.15.116:0/1240982500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f301039c0 msgr2=0x7f3f30103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1240982500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f301039c0 0x7f3f30103e40 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009930 tx=0x7f3f2402bc20 comp rx=0 tx=0).stop 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 -- 172.21.15.116:0/1240982500 shutdown_connections 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1240982500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f30104380 0x7f3f30110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1240982500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f301039c0 0x7f3f30103e40 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1240982500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f30109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 -- 172.21.15.116:0/1240982500 >> 172.21.15.116:0/1240982500 conn(0x7f3f300ff470 msgr2=0x7f3f30101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 -- 172.21.15.116:0/1240982500 shutdown_connections 2024-03-19T02:47:22.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.924+0000 7f3f35aff640 1 -- 172.21.15.116:0/1240982500 wait complete. 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 Processor -- start 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 -- start start 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f301039c0 0x7f3f3019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f3019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30113c90 con 0x7f3f30104380 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30113b10 con 0x7f3f301039c0 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f35aff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30113e10 con 0x7f3f301099d0 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34388/0 (socket says 172.21.15.116:34388) 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f2effd640 1 -- 172.21.15.116:0/1039557878 learned_addr learned my addr 172.21.15.116:0/1039557878 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f3019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.925+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f301039c0 0x7f3f3019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 -- 172.21.15.116:0/1039557878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 msgr2=0x7f3f3019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f3019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 -- 172.21.15.116:0/1039557878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f301039c0 msgr2=0x7f3f3019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f301039c0 0x7f3f3019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 -- 172.21.15.116:0/1039557878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3019f290 con 0x7f3f30104380 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f3019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:22.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.926+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009230 tx=0x7f3f2402c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.927+0000 7f3f2cff9640 1 -- 172.21.15.116:0/1039557878 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24043020 con 0x7f3f30104380 2024-03-19T02:47:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.927+0000 7f3f2cff9640 1 -- 172.21.15.116:0/1039557878 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f2402cae0 con 0x7f3f30104380 2024-03-19T02:47:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.927+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3019f520 con 0x7f3f30104380 2024-03-19T02:47:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.927+0000 7f3f2cff9640 1 -- 172.21.15.116:0/1039557878 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24003ae0 con 0x7f3f30104380 2024-03-19T02:47:22.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.927+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f301a6e10 con 0x7f3f30104380 2024-03-19T02:47:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.929+0000 7f3f2cff9640 1 -- 172.21.15.116:0/1039557878 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3f24046020 con 0x7f3f30104380 2024-03-19T02:47:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 Processor -- start 2024-03-19T02:47:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 -- start start 2024-03-19T02:47:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c69340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c6bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a969bc4160 con 0x55a969c60380 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a969b6ce10 con 0x55a969c60ca0 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f387ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a969bc0200 con 0x55a969c69880 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2cff9640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 0x7f3f180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c69340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34400/0 (socket says 172.21.15.116:34400) 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 -- 172.21.15.116:0/1365233967 learned_addr learned my addr 172.21.15.116:0/1365233967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c6bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 0x7f3f180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2cff9640 1 -- 172.21.15.116:0/1039557878 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f3f240f1050 con 0x7f3f30104380 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 -- 172.21.15.116:0/1365233967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 msgr2=0x55a969c6bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c6bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 -- 172.21.15.116:0/1365233967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 msgr2=0x55a969c69340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c69340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.930+0000 7f3f2ffff640 1 -- 172.21.15.116:0/1365233967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9699a07e0 con 0x55a969c60380 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.931+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f3f280105a0 tx=0x7f3f28010a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598959aaaa97cedf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.931+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 0x7f3f180a4e30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3f28010f60 tx=0x7f3f28014000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.931+0000 7f3f15ffb640 1 -- 172.21.15.116:0/1365233967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28022070 con 0x55a969c60380 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.931+0000 7f3f15ffb640 1 -- 172.21.15.116:0/1365233967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f28012d10 con 0x55a969c60380 2024-03-19T02:47:22.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.931+0000 7f3f15ffb640 1 -- 172.21.15.116:0/1365233967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2801e3f0 con 0x55a969c60380 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- 172.21.15.116:0/1365233967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 msgr2=0x55a969c60760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f3f280105a0 tx=0x7f3f28010a60 comp rx=0 tx=0).stop 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- 172.21.15.116:0/1365233967 shutdown_connections 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c6bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c69340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1365233967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c60760 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- 172.21.15.116:0/1365233967 >> 172.21.15.116:0/1365233967 conn(0x55a969c5aab0 msgr2=0x55a969c5ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- 172.21.15.116:0/1365233967 shutdown_connections 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- 172.21.15.116:0/1365233967 wait complete. 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 -- start start 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.932+0000 7f3f387ca740 1 Processor -- start 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a969c6c5b0 con 0x55a969c60380 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a969b801a0 con 0x55a969c60ca0 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a969c6c730 con 0x55a969c69880 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 10 client.0 _ll_register_callbacks cb 0x55a9698d7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34414/0 (socket says 172.21.15.116:34414) 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/2137664749 learned_addr learned my addr 172.21.15.116:0/2137664749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:22.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2effd640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/2137664749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 msgr2=0x55a969c672d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c672d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 msgr2=0x55a969c66d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c66d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/2137664749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a969a00f70 con 0x55a969c60380 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f3f20012710 tx=0x7f3f20012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346e0a42770fb7cd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000f920 con 0x55a969c60380 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f3f2000fac0 con 0x55a969c60380 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.933+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f28012d10 con 0x55a969c60380 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3f2000fcc0 con 0x55a969c60380 2024-03-19T02:47:22.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 10 client.21159 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a969898170 con 0x55a969c60380 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f28012d10 con 0x55a969c60380 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3f2001c420 con 0x55a969c60380 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 10 client.21159 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 10 client.21159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 20 client.21159 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969c79ae0 0x55a969c7bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 4 client.21159 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a969bd16a0 con 0x55a969c79ae0 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.934+0000 7f3f2effd640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969c79ae0 0x55a969c7bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.935+0000 7f3f0effd640 10 client.21159 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:22.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.935+0000 7f3f2effd640 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969c79ae0 0x55a969c7bf50 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c85faa305fc1618b server_cookie=d6c4d947d01a139a in_seq=0 out_seq=0 2024-03-19T02:47:22.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f0effd640 1 -- 172.21.15.116:0/2137664749 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a969bd16a0 con 0x55a969c79ae0 2024-03-19T02:47:22.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f0effd640 10 client.21159 handle_command_reply: tid=0 2024-03-19T02:47:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 client.21159 shutdown 2024-03-19T02:47:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 msgr2=0x55a969c66850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f3f20012710 tx=0x7f3f20012bd0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 shutdown_connections 2024-03-19T02:47:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969c79ae0 0x55a969c7bf50 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c69880 0x55a969c672d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f0effd640 0 client.21159 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 --2- 172.21.15.116:0/2137664749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c66850 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 >> 172.21.15.116:0/2137664749 conn(0x55a969c5aab0 msgr2=0x55a969c5c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.937+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 shutdown_connections 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.938+0000 7f3f387ca740 1 -- 172.21.15.116:0/2137664749 wait complete. 2024-03-19T02:47:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.938+0000 7f3f387ca740 20 client.21159 trim_cache size 0 max 16384 2024-03-19T02:47:22.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 Processor -- start 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 -- start start 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c75350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b0e90 con 0x55a969c69880 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b29d0 con 0x55a969c60ca0 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.964+0000 7f3f387ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b1560 con 0x55a969c60380 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c75350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c75350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45840/0 (socket says 172.21.15.116:45840) 2024-03-19T02:47:22.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34424/0 (socket says 172.21.15.116:34424) 2024-03-19T02:47:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51888/0 (socket says 172.21.15.116:51888) 2024-03-19T02:47:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2ffff640 1 -- 172.21.15.116:0/266014340 learned_addr learned my addr 172.21.15.116:0/266014340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/266014340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 msgr2=0x55a969c75350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c75350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/266014340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 msgr2=0x55a969c81820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/266014340 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a969c75890 con 0x55a969c60380 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2ffff640 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f20012710 tx=0x7f3f20009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c560bfcb908f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.965+0000 7f3f157fa640 1 -- 172.21.15.116:0/266014340 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20009e80 con 0x55a969c60380 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f157fa640 1 -- 172.21.15.116:0/266014340 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f2000fe60 con 0x55a969c60380 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f157fa640 1 -- 172.21.15.116:0/266014340 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20005340 con 0x55a969c60380 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 -- 172.21.15.116:0/266014340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 msgr2=0x55a969c68f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f20012710 tx=0x7f3f20009d60 comp rx=0 tx=0).stop 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 -- 172.21.15.116:0/266014340 shutdown_connections 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c69880 0x55a969c81820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c60ca0 0x55a969c75350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 --2- 172.21.15.116:0/266014340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60380 0x55a969c68f80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 -- 172.21.15.116:0/266014340 >> 172.21.15.116:0/266014340 conn(0x55a969c5aab0 msgr2=0x55a969c59b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 -- 172.21.15.116:0/266014340 shutdown_connections 2024-03-19T02:47:22.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.966+0000 7f3f387ca740 1 -- 172.21.15.116:0/266014340 wait complete. 2024-03-19T02:47:22.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 -- start start 2024-03-19T02:47:22.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 Processor -- start 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 0x55a969c6e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c69880 0x55a969c6e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b2300 con 0x55a969c60380 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301b1c30 con 0x55a969c69880 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.967+0000 7f3f387ca740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301aee90 con 0x55a969c60ca0 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f387ca740 10 client.0 _ll_register_callbacks cb 0x7f3f20009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 0x55a969c6e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c69880 0x55a969c6e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34428/0 (socket says 172.21.15.116:34428) 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/1182334027 learned_addr learned my addr 172.21.15.116:0/1182334027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/1182334027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 msgr2=0x55a969c6e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 0x55a969c6e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c69880 msgr2=0x55a969c6e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c69880 0x55a969c6e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 -- 172.21.15.116:0/1182334027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a969cdfe60 con 0x55a969c60380 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 0x55a969c6e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f2f7fe640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3f2001c8c0 tx=0x7f3f2001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddedb014c7ecbe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:22.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20009e80 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.968+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f2000fe60 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f3f2001c420 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3f2003c050 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f387ca740 10 client.21165 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a969ce0fd0 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f2000fe60 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f3f20003bf0 con 0x55a969c60380 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.969+0000 7f3f387ca740 10 client.21165 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f387ca740 10 client.21165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f387ca740 20 client.21165 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969ce1630 0x55a969ce3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f387ca740 4 client.21165 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:47:22.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a969bcb9f0 con 0x55a969ce1630 2024-03-19T02:47:22.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969ce1630 0x55a969ce3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:22.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f0effd640 10 client.21165 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:22.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.970+0000 7f3f2effd640 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969ce1630 0x55a969ce3a20 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d32ed4700242217c server_cookie=a72ec0ae401f74f9 in_seq=0 out_seq=0 2024-03-19T02:47:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f0effd640 1 -- 172.21.15.116:0/1182334027 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a969bcb9f0 con 0x55a969ce1630 2024-03-19T02:47:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f0effd640 10 client.21165 handle_command_reply: tid=0 2024-03-19T02:47:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f387ca740 1 client.21165 shutdown 2024-03-19T02:47:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 msgr2=0x55a969c71d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f3f2001c8c0 tx=0x7f3f2001ccc0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.973+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 shutdown_connections 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55a969ce1630 0x55a969ce3a20 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a969c69880 0x55a969c6e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f0effd640 0 client.21165 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969c60ca0 0x55a969c6e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 --2- 172.21.15.116:0/1182334027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a969c60380 0x55a969c71d60 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 >> 172.21.15.116:0/1182334027 conn(0x55a969c5aab0 msgr2=0x55a969c667f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 shutdown_connections 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 1 -- 172.21.15.116:0/1182334027 wait complete. 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.974+0000 7f3f387ca740 20 client.21165 trim_cache size 0 max 16384 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "dd199cc0-fc81-4735-ae29-7b865b709623", 2024-03-19T02:47:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:47:22.981 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 msgr2=0x7f3f180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 0x7f3f180a4e30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3f28010f60 tx=0x7f3f28014000 comp rx=0 tx=0).stop 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 msgr2=0x7f3f3019a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009230 tx=0x7f3f2402c700 comp rx=0 tx=0).stop 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 shutdown_connections 2024-03-19T02:47:22.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.975+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3f180a2970 0x7f3f180a4e30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f301099d0 0x7f3f3019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f30104380 0x7f3f3019a7e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 --2- 172.21.15.116:0/1039557878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f301039c0 0x7f3f3019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 >> 172.21.15.116:0/1039557878 conn(0x7f3f300ff470 msgr2=0x7f3f300ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 shutdown_connections 2024-03-19T02:47:22.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:22.976+0000 7f3f35aff640 1 -- 172.21.15.116:0/1039557878 wait complete. 2024-03-19T02:47:22.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:47:22.992 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 -- 172.21.15.116:0/3387906596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 msgr2=0x7f571010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3387906596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f571010e110 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f5708009ca0 tx=0x7f570802c180 comp rx=0 tx=0).stop 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 -- 172.21.15.116:0/3387906596 shutdown_connections 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3387906596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f571010e110 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3387906596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3387906596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 0x7f5710102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 -- 172.21.15.116:0/3387906596 >> 172.21.15.116:0/3387906596 conn(0x7f57100fc070 msgr2=0x7f57100fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 -- 172.21.15.116:0/3387906596 shutdown_connections 2024-03-19T02:47:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.114+0000 7f5714b9d640 1 -- 172.21.15.116:0/3387906596 wait complete. 2024-03-19T02:47:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 Processor -- start 2024-03-19T02:47:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 -- start start 2024-03-19T02:47:23.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 0x7f571010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f57101a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f5714b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5710111920 con 0x7f571010bc70 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f570effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.115+0000 7f570effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51928/0 (socket says 172.21.15.116:51928) 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 -- 172.21.15.116:0/3180863627 learned_addr learned my addr 172.21.15.116:0/3180863627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57101117a0 con 0x7f5710102850 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5710111aa0 con 0x7f5710103170 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570f7fe640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 0x7f571010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570ffff640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f57101a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 -- 172.21.15.116:0/3180863627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 msgr2=0x7f571010a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 0x7f571010a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 -- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 msgr2=0x7f57101a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f57101a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 -- 172.21.15.116:0/3180863627 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57101a3430 con 0x7f5710103170 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570ffff640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f57101a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.116+0000 7f570effd640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f570400e9f0 tx=0x7f570400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.117+0000 7f570cff9640 1 -- 172.21.15.116:0/3180863627 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570400ccc0 con 0x7f5710103170 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.117+0000 7f570cff9640 1 -- 172.21.15.116:0/3180863627 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570400ce60 con 0x7f5710103170 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.117+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57101a3720 con 0x7f5710103170 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.117+0000 7f570cff9640 1 -- 172.21.15.116:0/3180863627 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704010430 con 0x7f5710103170 2024-03-19T02:47:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.117+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57101aafb0 con 0x7f5710103170 2024-03-19T02:47:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f570cff9640 1 -- 172.21.15.116:0/3180863627 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f57040046c0 con 0x7f5710103170 2024-03-19T02:47:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 Processor -- start 2024-03-19T02:47:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f570cff9640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 0x7f56f80a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 -- start start 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f570f7fe640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 0x7f56f80a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f570cff9640 1 -- 172.21.15.116:0/3180863627 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f57040cf500 con 0x7f5710103170 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fc9bf0 0x562048fc9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562048f2d500 con 0x562048fd30f0 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562048ed5be0 con 0x562048fca510 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.120+0000 7f57189e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562048f295a0 con 0x562048fc9bf0 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45868/0 (socket says 172.21.15.116:45868) 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34448/0 (socket says 172.21.15.116:34448) 2024-03-19T02:47:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570effd640 1 -- 172.21.15.116:0/1680360458 learned_addr learned my addr 172.21.15.116:0/1680360458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570f7fe640 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fc9bf0 0x562048fc9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570f7fe640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 0x7f56f80a4f80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5700009be0 tx=0x7f5700011440 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 -- 172.21.15.116:0/1680360458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fc9bf0 msgr2=0x562048fc9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fc9bf0 0x562048fc9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 -- 172.21.15.116:0/1680360458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 msgr2=0x562048fd2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 -- 172.21.15.116:0/1680360458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562048ef7000 con 0x562048fd30f0 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570effd640 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f570ffff640 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5708037040 tx=0x7f57080323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fcd4ecc80c40ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f56f5ffb640 1 -- 172.21.15.116:0/1680360458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708039050 con 0x562048fd30f0 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f56f5ffb640 1 -- 172.21.15.116:0/1680360458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5708045070 con 0x562048fd30f0 2024-03-19T02:47:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.121+0000 7f56f5ffb640 1 -- 172.21.15.116:0/1680360458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080325f0 con 0x562048fd30f0 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 -- 172.21.15.116:0/1680360458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 msgr2=0x562048fd54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f5708037040 tx=0x7f57080323c0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 -- 172.21.15.116:0/1680360458 shutdown_connections 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd54e0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fd2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 --2- 172.21.15.116:0/1680360458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fc9bf0 0x562048fc9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 -- 172.21.15.116:0/1680360458 >> 172.21.15.116:0/1680360458 conn(0x562048fc4320 msgr2=0x562048fc4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 -- 172.21.15.116:0/1680360458 shutdown_connections 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.122+0000 7f57189e1740 1 -- 172.21.15.116:0/1680360458 wait complete. 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 -- start start 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 Processor -- start 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fd0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562048fd5e80 con 0x562048fd30f0 2024-03-19T02:47:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562048ee9160 con 0x562048fc9bf0 2024-03-19T02:47:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562048fd6000 con 0x562048fca510 2024-03-19T02:47:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f570ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fd0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34458/0 (socket says 172.21.15.116:34458) 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570ffff640 1 -- 172.21.15.116:0/439950244 learned_addr learned my addr 172.21.15.116:0/439950244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570effd640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.123+0000 7f57189e1740 10 client.0 _ll_register_callbacks cb 0x562048f25a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:23.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 -- 172.21.15.116:0/439950244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 msgr2=0x562048fd0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 -- 172.21.15.116:0/439950244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 msgr2=0x562048fd0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 -- 172.21.15.116:0/439950244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562048d0b7d0 con 0x562048fc9bf0 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570ffff640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:23.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f570f7fe640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fd0010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5700011940 tx=0x7f5700004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b57c1d12fb06f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f56eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700017040 con 0x562048fc9bf0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5708045070 con 0x562048fc9bf0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.124+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f5700005220 con 0x562048fc9bf0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5700031050 con 0x562048fc9bf0 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 10 client.30899 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562048c413f0 con 0x562048fc9bf0 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5708045070 con 0x562048fc9bf0 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5700025030 con 0x562048fc9bf0 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 10 client.30899 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 10 client.30899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 20 client.30899 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x562048fe5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 4 client.30899 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562048f3aa40 con 0x562048fe33e0 2024-03-19T02:47:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f570effd640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x562048fe5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.125+0000 7f56eeffd640 10 client.30899 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.126+0000 7f570effd640 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x562048fe5880 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5beee35051614bab server_cookie=1da9e60183cfa103 in_seq=0 out_seq=0 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.135+0000 7f56eeffd640 1 -- 172.21.15.116:0/439950244 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562048f3aa40 con 0x562048fe33e0 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.135+0000 7f56eeffd640 10 client.30899 handle_command_reply: tid=0 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.135+0000 7f57189e1740 1 client.30899 shutdown 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 msgr2=0x562048fd0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fd0010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5700011940 tx=0x7f5700004730 comp rx=0 tx=0).stop 2024-03-19T02:47:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 shutdown_connections 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x562048fe5880 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f56eeffd640 0 client.30899 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 --2- 172.21.15.116:0/439950244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fd0010 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 >> 172.21.15.116:0/439950244 conn(0x562048fc4320 msgr2=0x562048fc6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 shutdown_connections 2024-03-19T02:47:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 1 -- 172.21.15.116:0/439950244 wait complete. 2024-03-19T02:47:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.136+0000 7f57189e1740 20 client.30899 trim_cache size 0 max 16384 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 Processor -- start 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 -- start start 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fc9bf0 0x562048fc5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fdec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc00ff60 con 0x562048fc9bf0 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc011aa0 con 0x562048fca510 2024-03-19T02:47:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.157+0000 7f57189e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc010630 con 0x562048fd30f0 2024-03-19T02:47:23.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.158+0000 7f570ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.158+0000 7f570ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51956/0 (socket says 172.21.15.116:51956) 2024-03-19T02:47:23.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.158+0000 7f570ffff640 1 -- 172.21.15.116:0/2339419022 learned_addr learned my addr 172.21.15.116:0/2339419022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.158+0000 7f570effd640 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fdec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.158+0000 7f570f7fe640 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fc9bf0 0x562048fc5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 -- 172.21.15.116:0/2339419022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 msgr2=0x562048fdec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fdec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 -- 172.21.15.116:0/2339419022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fc9bf0 msgr2=0x562048fc5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fc9bf0 0x562048fc5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 -- 172.21.15.116:0/2339419022 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562048fca340 con 0x562048fd30f0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f570ffff640 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f570802c6c0 tx=0x7f5708009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae6ec60e1b612a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f56f57fa640 1 -- 172.21.15.116:0/2339419022 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080046d0 con 0x562048fd30f0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f56f57fa640 1 -- 172.21.15.116:0/2339419022 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5708045040 con 0x562048fd30f0 2024-03-19T02:47:23.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.159+0000 7f56f57fa640 1 -- 172.21.15.116:0/2339419022 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708052ab0 con 0x562048fd30f0 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 -- 172.21.15.116:0/2339419022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 msgr2=0x562048feb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f570802c6c0 tx=0x7f5708009580 comp rx=0 tx=0).stop 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 -- 172.21.15.116:0/2339419022 shutdown_connections 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fd30f0 0x562048feb150 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fca510 0x562048fdec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 --2- 172.21.15.116:0/2339419022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fc9bf0 0x562048fc5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 -- 172.21.15.116:0/2339419022 >> 172.21.15.116:0/2339419022 conn(0x562048fc4320 msgr2=0x562048fc2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 -- 172.21.15.116:0/2339419022 shutdown_connections 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.160+0000 7f57189e1740 1 -- 172.21.15.116:0/2339419022 wait complete. 2024-03-19T02:47:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.161+0000 7f57189e1740 1 -- start start 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.161+0000 7f57189e1740 1 Processor -- start 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc0113d0 con 0x562048fd30f0 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc010d00 con 0x562048fc9bf0 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc00df60 con 0x562048fca510 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f57189e1740 10 client.0 _ll_register_callbacks cb 0x7f570000f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45890/0 (socket says 172.21.15.116:45890) 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570f7fe640 1 -- 172.21.15.116:0/2912608209 learned_addr learned my addr 172.21.15.116:0/2912608209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570effd640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570ffff640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570f7fe640 1 -- 172.21.15.116:0/2912608209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 msgr2=0x562048fd8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.162+0000 7f570f7fe640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570f7fe640 1 -- 172.21.15.116:0/2912608209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 msgr2=0x562048fd87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570f7fe640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570f7fe640 1 -- 172.21.15.116:0/2912608209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562048c010f0 con 0x562048fc9bf0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570ffff640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570effd640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f570f7fe640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5700011940 tx=0x7f570001e5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96916b72f0074e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f56eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700017040 con 0x562048fc9bf0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5708045040 con 0x562048fc9bf0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f57000049a0 con 0x562048fc9bf0 2024-03-19T02:47:23.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f570001ebe0 con 0x562048fc9bf0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f57189e1740 10 client.30905 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.163+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56204905af70 con 0x562048fc9bf0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5708045040 con 0x562048fc9bf0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f5700025040 con 0x562048fc9bf0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 10 client.30905 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 10 client.30905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 20 client.30905 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x56204905d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 4 client.30905 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562048f34d90 con 0x562048fe33e0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f570effd640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x56204905d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f56eeffd640 10 client.30905 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:23.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.164+0000 7f570effd640 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x56204905d720 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3aab513aa050dcd server_cookie=4d654d26b5c4569d in_seq=0 out_seq=0 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f56eeffd640 1 -- 172.21.15.116:0/2912608209 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x562048f34d90 con 0x562048fe33e0 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f56eeffd640 10 client.30905 handle_command_reply: tid=0 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 client.30905 shutdown 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 msgr2=0x562048fdb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5700011940 tx=0x7f570001e5d0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 shutdown_connections 2024-03-19T02:47:23.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562048fe33e0 0x56204905d720 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562048fd30f0 0x562048fd87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562048fca510 0x562048fd8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 --2- 172.21.15.116:0/2912608209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562048fc9bf0 0x562048fdb2b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f56eeffd640 0 client.30905 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 >> 172.21.15.116:0/2912608209 conn(0x562048fc4320 msgr2=0x562048fc5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 shutdown_connections 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 1 -- 172.21.15.116:0/2912608209 wait complete. 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.167+0000 7f57189e1740 20 client.30905 trim_cache size 0 max 16384 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:23.174 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (448 inodes in the stack)", 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: "dd199cc0-fc81-4735-ae29-7b865b709623": { 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "dd199cc0-fc81-4735-ae29-7b865b709623", 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:47:23.175 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 msgr2=0x7f56f80a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 0x7f56f80a4f80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5700009be0 tx=0x7f5700011440 comp rx=0 tx=0).stop 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 msgr2=0x7f571010af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f570400e9f0 tx=0x7f570400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 shutdown_connections 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56f80a2ac0 0x7f56f80a4f80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571010bc70 0x7f57101a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5710103170 0x7f571010af30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 --2- 172.21.15.116:0/3180863627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5710102850 0x7f571010a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 >> 172.21.15.116:0/3180863627 conn(0x7f57100fc070 msgr2=0x7f57100fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 shutdown_connections 2024-03-19T02:47:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:23.168+0000 7f5714b9d640 1 -- 172.21.15.116:0/3180863627 wait complete. 2024-03-19T02:47:23.179 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:22 smithi116 ceph-mon[32263]: pgmap v617: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 167 op/s 2024-03-19T02:47:23.180 INFO:tasks.cephfs.filesystem:scrub status for tag:dd199cc0-fc81-4735-ae29-7b865b709623 - {'path': '/', 'tag': 'dd199cc0-fc81-4735-ae29-7b865b709623', 'options': 'recursive,force'} 2024-03-19T02:47:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:22 smithi154 ceph-mon[35693]: pgmap v617: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 167 op/s 2024-03-19T02:47:23.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45667 5.98 MB/sec execute 285 sec latency 236.956 ms 2024-03-19T02:47:23.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:22 smithi162 ceph-mon[35761]: pgmap v617: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 167 op/s 2024-03-19T02:47:24.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:23 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:23 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:23 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45780 5.98 MB/sec execute 286 sec latency 157.144 ms 2024-03-19T02:47:24.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:24 smithi162 ceph-mon[35761]: pgmap v618: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-03-19T02:47:24.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:24 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:24.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:24 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:24.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:24 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:24 smithi154 ceph-mon[35693]: pgmap v618: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-03-19T02:47:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:24 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:24 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:24 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:24 smithi116 ceph-mon[32263]: pgmap v618: 129 pgs: 129 active+clean; 969 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-03-19T02:47:25.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:24 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:25.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:24 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:25.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:24 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:25.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 45886 5.99 MB/sec execute 287 sec latency 217.355 ms 2024-03-19T02:47:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:26.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:25 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:47:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:25 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:47:26.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46003 6.00 MB/sec execute 288 sec latency 160.493 ms 2024-03-19T02:47:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:25 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:47:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:26 smithi154 ceph-mon[35693]: pgmap v619: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.2 MiB/s wr, 229 op/s 2024-03-19T02:47:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:26 smithi116 ceph-mon[32263]: pgmap v619: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.2 MiB/s wr, 229 op/s 2024-03-19T02:47:27.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46122 6.00 MB/sec execute 289 sec latency 140.144 ms 2024-03-19T02:47:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:26 smithi162 ceph-mon[35761]: pgmap v619: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.2 MiB/s wr, 229 op/s 2024-03-19T02:47:28.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46251 6.01 MB/sec execute 290 sec latency 175.741 ms 2024-03-19T02:47:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:28 smithi154 ceph-mon[35693]: pgmap v620: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:28 smithi116 ceph-mon[32263]: pgmap v620: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:29.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46344 6.00 MB/sec execute 291 sec latency 145.773 ms 2024-03-19T02:47:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:28 smithi162 ceph-mon[35761]: pgmap v620: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:30.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46443 5.99 MB/sec execute 292 sec latency 214.671 ms 2024-03-19T02:47:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:30 smithi154 ceph-mon[35693]: pgmap v621: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:30 smithi116 ceph-mon[32263]: pgmap v621: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:31.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46559 5.98 MB/sec execute 293 sec latency 129.384 ms 2024-03-19T02:47:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:30 smithi162 ceph-mon[35761]: pgmap v621: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 171 op/s 2024-03-19T02:47:32.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46660 5.96 MB/sec execute 294 sec latency 139.508 ms 2024-03-19T02:47:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:32 smithi154 ceph-mon[35693]: pgmap v622: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-03-19T02:47:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:32 smithi116 ceph-mon[32263]: pgmap v622: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-03-19T02:47:33.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46769 5.95 MB/sec execute 295 sec latency 162.469 ms 2024-03-19T02:47:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:32 smithi162 ceph-mon[35761]: pgmap v622: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 237 op/s 2024-03-19T02:47:34.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 46884 5.94 MB/sec execute 296 sec latency 126.572 ms 2024-03-19T02:47:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:34 smithi154 ceph-mon[35693]: pgmap v623: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-19T02:47:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:34 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:34 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:34 smithi116 ceph-mon[32263]: pgmap v623: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-19T02:47:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:34 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:34 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:35.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47007 5.94 MB/sec execute 297 sec latency 158.508 ms 2024-03-19T02:47:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:34 smithi162 ceph-mon[35761]: pgmap v623: 129 pgs: 129 active+clean; 983 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-19T02:47:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:34 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:34 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:35 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:35 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:35 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:47:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:35 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:35 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:35 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:47:36.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47129 5.94 MB/sec execute 298 sec latency 124.283 ms 2024-03-19T02:47:36.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:35 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:35 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:35 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:47:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:36 smithi154 ceph-mon[35693]: pgmap v624: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-03-19T02:47:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:36 smithi116 ceph-mon[32263]: pgmap v624: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-03-19T02:47:37.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47256 5.94 MB/sec execute 299 sec latency 123.922 ms 2024-03-19T02:47:37.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:36 smithi162 ceph-mon[35761]: pgmap v624: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 236 op/s 2024-03-19T02:47:38.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47359 5.95 MB/sec execute 300 sec latency 138.757 ms 2024-03-19T02:47:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:38 smithi154 ceph-mon[35693]: pgmap v625: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:38 smithi116 ceph-mon[32263]: pgmap v625: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:39.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47511 5.97 MB/sec execute 301 sec latency 163.467 ms 2024-03-19T02:47:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:38 smithi162 ceph-mon[35761]: pgmap v625: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:40.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47609 5.98 MB/sec execute 302 sec latency 185.305 ms 2024-03-19T02:47:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:40 smithi154 ceph-mon[35693]: pgmap v626: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:40 smithi116 ceph-mon[32263]: pgmap v626: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:41.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47768 6.01 MB/sec execute 303 sec latency 176.158 ms 2024-03-19T02:47:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:40 smithi162 ceph-mon[35761]: pgmap v626: 129 pgs: 129 active+clean; 988 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 146 op/s 2024-03-19T02:47:42.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 47876 6.01 MB/sec execute 304 sec latency 166.103 ms 2024-03-19T02:47:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:42 smithi154 ceph-mon[35693]: pgmap v627: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-19T02:47:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:42 smithi116 ceph-mon[32263]: pgmap v627: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-19T02:47:43.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48012 6.03 MB/sec execute 305 sec latency 174.104 ms 2024-03-19T02:47:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:42 smithi162 ceph-mon[35761]: pgmap v627: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-19T02:47:44.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48152 6.03 MB/sec execute 306 sec latency 174.971 ms 2024-03-19T02:47:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:44 smithi154 ceph-mon[35693]: pgmap v628: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 153 op/s 2024-03-19T02:47:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:44 smithi116 ceph-mon[32263]: pgmap v628: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 153 op/s 2024-03-19T02:47:45.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48247 6.02 MB/sec execute 307 sec latency 175.950 ms 2024-03-19T02:47:45.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:44 smithi162 ceph-mon[35761]: pgmap v628: 129 pgs: 129 active+clean; 1007 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 153 op/s 2024-03-19T02:47:46.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48320 6.02 MB/sec execute 308 sec latency 160.359 ms 2024-03-19T02:47:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:46 smithi154 ceph-mon[35693]: pgmap v629: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 214 op/s 2024-03-19T02:47:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:46 smithi116 ceph-mon[32263]: pgmap v629: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 214 op/s 2024-03-19T02:47:47.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48388 6.00 MB/sec execute 309 sec latency 161.669 ms 2024-03-19T02:47:47.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:46 smithi162 ceph-mon[35761]: pgmap v629: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 214 op/s 2024-03-19T02:47:48.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48480 6.01 MB/sec execute 310 sec latency 135.646 ms 2024-03-19T02:47:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:48 smithi154 ceph-mon[35693]: pgmap v630: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:48 smithi116 ceph-mon[32263]: pgmap v630: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:49.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48591 6.03 MB/sec execute 311 sec latency 143.573 ms 2024-03-19T02:47:49.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:48 smithi162 ceph-mon[35761]: pgmap v630: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:50.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48714 6.04 MB/sec execute 312 sec latency 152.987 ms 2024-03-19T02:47:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:50 smithi154 ceph-mon[35693]: pgmap v631: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:50 smithi116 ceph-mon[32263]: pgmap v631: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:51.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48806 6.04 MB/sec execute 313 sec latency 170.338 ms 2024-03-19T02:47:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:50 smithi162 ceph-mon[35761]: pgmap v631: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 134 op/s 2024-03-19T02:47:52.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 48925 6.05 MB/sec execute 314 sec latency 166.688 ms 2024-03-19T02:47:53.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:47:53.181 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:47:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:52 smithi154 ceph-mon[35693]: pgmap v632: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 196 op/s 2024-03-19T02:47:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:52 smithi116 ceph-mon[32263]: pgmap v632: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 196 op/s 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.308+0000 7fdf34c9a640 1 -- 172.21.15.116:0/3675421286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101050 msgr2=0x7fdf3010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.308+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/3675421286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101050 0x7fdf3010d560 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009cd0 tx=0x7fdf2802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.308+0000 7fdf34c9a640 1 -- 172.21.15.116:0/3675421286 shutdown_connections 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/3675421286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30101050 0x7fdf3010d560 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/3675421286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf30100690 0x7fdf30100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/3675421286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf301066a0 0x7fdf30106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 -- 172.21.15.116:0/3675421286 >> 172.21.15.116:0/3675421286 conn(0x7fdf30077820 msgr2=0x7fdf30077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 -- 172.21.15.116:0/3675421286 shutdown_connections 2024-03-19T02:47:53.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 -- 172.21.15.116:0/3675421286 wait complete. 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 Processor -- start 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.309+0000 7fdf34c9a640 1 -- start start 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf30101050 0x7fdf30199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 0x7fdf301a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf30110960 con 0x7fdf30100690 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301107e0 con 0x7fdf30101050 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf34c9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf30110ae0 con 0x7fdf301066a0 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf30101050 0x7fdf30199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47274/0 (socket says 172.21.15.116:47274) 2024-03-19T02:47:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 -- 172.21.15.116:0/4124947180 learned_addr learned my addr 172.21.15.116:0/4124947180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 0x7fdf301a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 -- 172.21.15.116:0/4124947180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 msgr2=0x7fdf301a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 0x7fdf301a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 -- 172.21.15.116:0/4124947180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf30101050 msgr2=0x7fdf30199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf30101050 0x7fdf30199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 -- 172.21.15.116:0/4124947180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf301a43e0 con 0x7fdf30100690 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 0x7fdf301a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.310+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000e9f0 tx=0x7fdf2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.311+0000 7fdf277fe640 1 -- 172.21.15.116:0/4124947180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2000cd80 con 0x7fdf30100690 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.311+0000 7fdf277fe640 1 -- 172.21.15.116:0/4124947180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf200108e0 con 0x7fdf30100690 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.311+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf301a45b0 con 0x7fdf30100690 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.311+0000 7fdf277fe640 1 -- 172.21.15.116:0/4124947180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20018920 con 0x7fdf30100690 2024-03-19T02:47:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.311+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf301a4ad0 con 0x7fdf30100690 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 Processor -- start 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 -- start start 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf277fe640 1 -- 172.21.15.116:0/4124947180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdf20010a90 con 0x7fdf30100690 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c28d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c290320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55706c1e8500 con 0x55706c285350 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55706c190be0 con 0x55706c28df30 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.314+0000 7fdf37966740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55706c1e45a0 con 0x55706c284a30 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56178/0 (socket says 172.21.15.116:56178) 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1568196105 learned_addr learned my addr 172.21.15.116:0/1568196105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c28d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf27fff640 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c290320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1568196105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 msgr2=0x55706c290320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c290320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1568196105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 msgr2=0x55706c28d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c28d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1568196105 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706c1b2000 con 0x55706c284a30 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf277fe640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 0x7fdf1c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdf28004af0 tx=0x7fdf28005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=add75d8775201f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 0x7fdf1c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf277fe640 1 -- 172.21.15.116:0/4124947180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdf200cf4d0 con 0x7fdf30100690 2024-03-19T02:47:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.315+0000 7fdf24ff9640 1 -- 172.21.15.116:0/1568196105 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28043070 con 0x55706c284a30 2024-03-19T02:47:53.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf24ff9640 1 -- 172.21.15.116:0/1568196105 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf28003e20 con 0x55706c284a30 2024-03-19T02:47:53.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf24ff9640 1 -- 172.21.15.116:0/1568196105 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28039040 con 0x55706c284a30 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 -- 172.21.15.116:0/1568196105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 msgr2=0x55706c284e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf2e59c640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 0x7fdf1c0a4e30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf200cc8f0 tx=0x7fdf200d50e0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdf28004af0 tx=0x7fdf28005f70 comp rx=0 tx=0).stop 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 -- 172.21.15.116:0/1568196105 shutdown_connections 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c290320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c28d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 --2- 172.21.15.116:0/1568196105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c284a30 0x55706c284e10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.316+0000 7fdf37966740 1 -- 172.21.15.116:0/1568196105 >> 172.21.15.116:0/1568196105 conn(0x55706c27f160 msgr2=0x55706c27f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- 172.21.15.116:0/1568196105 shutdown_connections 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- 172.21.15.116:0/1568196105 wait complete. 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- start start 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 Processor -- start 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c28afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c28b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55706c2950d0 con 0x55706c28df30 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55706c1a4160 con 0x55706c284a30 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55706c295250 con 0x55706c285350 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf37966740 10 client.0 _ll_register_callbacks cb 0x55706c1e0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:53.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47298/0 (socket says 172.21.15.116:47298) 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1347477135 learned_addr learned my addr 172.21.15.116:0/1347477135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf27fff640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c28b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.317+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c28afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1347477135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 msgr2=0x55706c28b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c28b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 msgr2=0x55706c28afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c28afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/1347477135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706bfc67d0 con 0x55706c28df30 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009bf0 tx=0x7fdf2802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579280e3484f0cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28039030 con 0x55706c28df30 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf28003e20 con 0x55706c28df30 2024-03-19T02:47:53.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdf28003c10 con 0x55706c28df30 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdf28043040 con 0x55706c28df30 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf37966740 10 client.21192 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55706befc3f0 con 0x55706c28df30 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.318+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf28003e20 con 0x55706c28df30 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdf280489e0 con 0x55706c28df30 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 10 client.21192 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 10 client.21192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 20 client.21192 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c2a2530 0x55706c2a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 4 client.21192 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55706c1f5a40 con 0x55706c2a2530 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf27fff640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c2a2530 0x55706c2a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf27fff640 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c2a2530 0x55706c2a4920 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a883367b5ad16a98 server_cookie=bd2426852c2c424b in_seq=0 out_seq=0 2024-03-19T02:47:53.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.319+0000 7fdf15ffb640 10 client.21192 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf15ffb640 1 -- 172.21.15.116:0/1347477135 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55706c1f5a40 con 0x55706c2a2530 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf15ffb640 10 client.21192 handle_command_reply: tid=0 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 client.21192 shutdown 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 msgr2=0x55706c28ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009bf0 tx=0x7fdf2802e040 comp rx=0 tx=0).stop 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 shutdown_connections 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c2a2530 0x55706c2a4920 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c28df30 0x55706c28ba50 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c28b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 --2- 172.21.15.116:0/1347477135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c28afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf15ffb640 0 client.21192 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.321+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 >> 172.21.15.116:0/1347477135 conn(0x55706c27f160 msgr2=0x55706c280eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.322+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 shutdown_connections 2024-03-19T02:47:53.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.322+0000 7fdf37966740 1 -- 172.21.15.116:0/1347477135 wait complete. 2024-03-19T02:47:53.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.322+0000 7fdf37966740 20 client.21192 trim_cache size 0 max 16384 2024-03-19T02:47:53.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49033 6.05 MB/sec execute 315 sec latency 162.591 ms 2024-03-19T02:47:53.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 Processor -- start 2024-03-19T02:47:53.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 -- start start 2024-03-19T02:47:53.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c2a7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c28a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301acb20 con 0x55706c284a30 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301ae660 con 0x55706c28df30 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf37966740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301ad1f0 con 0x55706c285350 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47302/0 (socket says 172.21.15.116:47302) 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.343+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/4217848323 learned_addr learned my addr 172.21.15.116:0/4217848323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf27fff640 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c2a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c28a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/4217848323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 msgr2=0x55706c2a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c2a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/4217848323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 msgr2=0x55706c28a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c28a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 -- 172.21.15.116:0/4217848323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706bebc0f0 con 0x55706c284a30 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdf280045d0 tx=0x7fdf28005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec5df09a654f81c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf17fff640 1 -- 172.21.15.116:0/4217848323 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28039040 con 0x55706c284a30 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf17fff640 1 -- 172.21.15.116:0/4217848323 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf280324f0 con 0x55706c284a30 2024-03-19T02:47:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.344+0000 7fdf17fff640 1 -- 172.21.15.116:0/4217848323 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28048450 con 0x55706c284a30 2024-03-19T02:47:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- 172.21.15.116:0/4217848323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 msgr2=0x55706c28ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fdf280045d0 tx=0x7fdf28005f70 comp rx=0 tx=0).stop 2024-03-19T02:47:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- 172.21.15.116:0/4217848323 shutdown_connections 2024-03-19T02:47:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c28df30 0x55706c28a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c285350 0x55706c2a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 --2- 172.21.15.116:0/4217848323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c284a30 0x55706c28ce00 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- 172.21.15.116:0/4217848323 >> 172.21.15.116:0/4217848323 conn(0x55706c27f160 msgr2=0x55706c28c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- 172.21.15.116:0/4217848323 shutdown_connections 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- 172.21.15.116:0/4217848323 wait complete. 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.345+0000 7fdf37966740 1 -- start start 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 Processor -- start 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c2a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c28df30 0x55706c2a5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301adf90 con 0x55706c285350 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301ad8c0 con 0x55706c284a30 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf301aab20 con 0x55706c28df30 2024-03-19T02:47:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf37966740 10 client.0 _ll_register_callbacks cb 0x7fdf200d2ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47312/0 (socket says 172.21.15.116:47312) 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 -- 172.21.15.116:0/2105630097 learned_addr learned my addr 172.21.15.116:0/2105630097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c2a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c28df30 0x55706c2a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 -- 172.21.15.116:0/2105630097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c28df30 msgr2=0x55706c2a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c28df30 0x55706c2a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 -- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 msgr2=0x55706c2a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c2a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.346+0000 7fdf27fff640 1 -- 172.21.15.116:0/2105630097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706c3074d0 con 0x55706c285350 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf27fff640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x55706c1a4290 tx=0x7fdf180204d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85de42676e704a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf18011c80 con 0x55706c285350 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdf18005020 con 0x55706c285350 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf280324f0 con 0x55706c285350 2024-03-19T02:47:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdf18030050 con 0x55706c285350 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 10 client.21198 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55706c308660 con 0x55706c285350 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf280324f0 con 0x55706c285350 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdf18004c00 con 0x55706c285350 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 10 client.21198 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 10 client.21198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 20 client.21198 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.347+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c308e60 0x55706c30b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.348+0000 7fdf37966740 4 client.21198 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.348+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55706c1efd90 con 0x55706c308e60 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.348+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c308e60 0x55706c30b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.348+0000 7fdf15ffb640 10 client.21198 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.348+0000 7fdf2ed9d640 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c308e60 0x55706c30b280 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49b3617a18e6fbd server_cookie=c8be1287ca701011 in_seq=0 out_seq=0 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf15ffb640 1 -- 172.21.15.116:0/2105630097 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55706c1efd90 con 0x55706c308e60 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf15ffb640 10 client.21198 handle_command_reply: tid=0 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 client.21198 shutdown 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 msgr2=0x55706c2a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x55706c1a4290 tx=0x7fdf180204d0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 shutdown_connections 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55706c308e60 0x55706c30b280 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55706c28df30 0x55706c2a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55706c285350 0x55706c2a5250 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf15ffb640 0 client.21198 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 --2- 172.21.15.116:0/2105630097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55706c284a30 0x55706c2a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 >> 172.21.15.116:0/2105630097 conn(0x55706c27f160 msgr2=0x55706c28b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 shutdown_connections 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 1 -- 172.21.15.116:0/2105630097 wait complete. 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf37966740 20 client.21198 trim_cache size 0 max 16384 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:47:53.356 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 msgr2=0x7fdf1c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 0x7fdf1c0a4e30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf200cc8f0 tx=0x7fdf200d50e0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 msgr2=0x7fdf30198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000e9f0 tx=0x7fdf2000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.350+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 shutdown_connections 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf1c0a2970 0x7fdf1c0a4e30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf301066a0 0x7fdf301a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf30101050 0x7fdf30199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 --2- 172.21.15.116:0/4124947180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf30100690 0x7fdf30198e20 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 >> 172.21.15.116:0/4124947180 conn(0x7fdf30077820 msgr2=0x7fdf30102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 shutdown_connections 2024-03-19T02:47:53.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.351+0000 7fdf34c9a640 1 -- 172.21.15.116:0/4124947180 wait complete. 2024-03-19T02:47:53.364 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:47:53.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:47:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:52 smithi162 ceph-mon[35761]: pgmap v632: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 196 op/s 2024-03-19T02:47:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.482+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/1879712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0106730 msgr2=0x7fc9e0106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/1879712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0106730 0x7fc9e0106b10 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009930 tx=0x7fc9d002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/1879712445 shutdown_connections 2024-03-19T02:47:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/1879712445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 0x7fc9e010d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/1879712445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0100720 0x7fc9e0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/1879712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0106730 0x7fc9e0106b10 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/1879712445 >> 172.21.15.116:0/1879712445 conn(0x7fc9e0077820 msgr2=0x7fc9e0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/1879712445 shutdown_connections 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/1879712445 wait complete. 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.483+0000 7fc9e5cdc640 1 Processor -- start 2024-03-19T02:47:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 -- start start 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 0x7fc9e0103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0106730 0x7fc9e01038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e01109f0 con 0x7fc9e0100720 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0110870 con 0x7fc9e0106730 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9e5cdc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0110b70 con 0x7fc9e01010e0 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47334/0 (socket says 172.21.15.116:47334) 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 -- 172.21.15.116:0/701325499 learned_addr learned my addr 172.21.15.116:0/701325499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9dffff640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0106730 0x7fc9e01038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9deffd640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 0x7fc9e0103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 -- 172.21.15.116:0/701325499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 msgr2=0x7fc9e0103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 0x7fc9e0103370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 -- 172.21.15.116:0/701325499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0106730 msgr2=0x7fc9e01038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0106730 0x7fc9e01038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.484+0000 7fc9df7fe640 1 -- 172.21.15.116:0/701325499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e01a6500 con 0x7fc9e0100720 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009230 tx=0x7fc9d002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0043020 con 0x7fc9e0100720 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9d002ca70 con 0x7fc9e0100720 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e01a6790 con 0x7fc9e0100720 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0003a50 con 0x7fc9e0100720 2024-03-19T02:47:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.485+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e01a6d30 con 0x7fc9e0100720 2024-03-19T02:47:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.486+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e0068610 con 0x7fc9e0100720 2024-03-19T02:47:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.487+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc9d0046020 con 0x7fc9e0100720 2024-03-19T02:47:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.488+0000 7fc9dcff9640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 0x7fc9c80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.488+0000 7fc9deffd640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 0x7fc9c80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.488+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fc9d00f1070 con 0x7fc9e0100720 2024-03-19T02:47:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.489+0000 7fc9deffd640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 0x7fc9c80a4eb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40095f0 tx=0x7fc9d4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.494+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc9d004d020 con 0x7fc9e0100720 2024-03-19T02:47:53.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.803+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9e0109ca0 con 0x7fc9e0100720 2024-03-19T02:47:53.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.804+0000 7fc9dcff9640 1 -- 172.21.15.116:0/701325499 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11950 (secure 0 0 0) 0x7fc9d00bdb00 con 0x7fc9e0100720 2024-03-19T02:47:53.805 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:47:53.805 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:43:04.493715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:47:53.805 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 38 2024-03-19T02:47:53.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 msgr2=0x7fc9c80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 0x7fc9c80a4eb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40095f0 tx=0x7fc9d4005f90 comp rx=0 tx=0).stop 2024-03-19T02:47:53.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 msgr2=0x7fc9e0102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009230 tx=0x7fc9d002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 shutdown_connections 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc9c80a29f0 0x7fc9c80a4eb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0106730 0x7fc9e01038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9e01010e0 0x7fc9e0103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 --2- 172.21.15.116:0/701325499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9e0100720 0x7fc9e0102e30 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 >> 172.21.15.116:0/701325499 conn(0x7fc9e0077820 msgr2=0x7fc9e0068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 shutdown_connections 2024-03-19T02:47:53.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.806+0000 7fc9e5cdc640 1 -- 172.21.15.116:0/701325499 wait complete. 2024-03-19T02:47:53.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:47:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 -- 172.21.15.116:0/208709811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 msgr2=0x7faff0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 --2- 172.21.15.116:0/208709811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff0106a90 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009930 tx=0x7fafe002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 -- 172.21.15.116:0/208709811 shutdown_connections 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 --2- 172.21.15.116:0/208709811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 0x7faff010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 --2- 172.21.15.116:0/208709811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 --2- 172.21.15.116:0/208709811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff0106a90 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 -- 172.21.15.116:0/208709811 >> 172.21.15.116:0/208709811 conn(0x7faff00778d0 msgr2=0x7faff0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 -- 172.21.15.116:0/208709811 shutdown_connections 2024-03-19T02:47:53.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.950+0000 7faff48ac640 1 -- 172.21.15.116:0/208709811 wait complete. 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 Processor -- start 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 -- start start 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 0x7faff0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faff0110970 con 0x7faff01066b0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faff01107f0 con 0x7faff01006a0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7faff48ac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faff0110af0 con 0x7faff0101060 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7fafeed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7fafeed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47354/0 (socket says 172.21.15.116:47354) 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51212/0 (socket says 172.21.15.116:51212) 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.951+0000 7fafeed9d640 1 -- 172.21.15.116:0/398297210 learned_addr learned my addr 172.21.15.116:0/398297210 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafedd9b640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 0x7faff0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 -- 172.21.15.116:0/398297210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 msgr2=0x7faff0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 0x7faff0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 -- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 msgr2=0x7faff019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 -- 172.21.15.116:0/398297210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff019cfb0 con 0x7faff01006a0 2024-03-19T02:47:53.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafeed9d640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafee59c640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009230 tx=0x7fafe00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafdf7fe640 1 -- 172.21.15.116:0/398297210 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0043020 con 0x7faff01006a0 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff019d240 con 0x7faff01006a0 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafdf7fe640 1 -- 172.21.15.116:0/398297210 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafe002ca70 con 0x7faff01006a0 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7fafdf7fe640 1 -- 172.21.15.116:0/398297210 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00039b0 con 0x7faff01006a0 2024-03-19T02:47:53.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.952+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff01a4b30 con 0x7faff01006a0 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.955+0000 7faff7573740 1 Processor -- start 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 -- start start 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafdf7fe640 1 -- 172.21.15.116:0/398297210 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fafe0046020 con 0x7faff01006a0 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b517e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b5189bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644b50e14b0 con 0x5644b51877d0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644b5089d70 con 0x5644b517ebf0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7faff7573740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644b50dd550 con 0x5644b517e2d0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafeed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b517e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafedd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b5189bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafedd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b5189bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47360/0 (socket says 172.21.15.116:47360) 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafedd9b640 1 -- 172.21.15.116:0/855019312 learned_addr learned my addr 172.21.15.116:0/855019312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51220/0 (socket says 172.21.15.116:51220) 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 -- 172.21.15.116:0/855019312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 msgr2=0x5644b517e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b517e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 -- 172.21.15.116:0/855019312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 msgr2=0x5644b5189bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b5189bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafee59c640 1 -- 172.21.15.116:0/855019312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b50ac210 con 0x5644b517ebf0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafeed9d640 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b517e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.956+0000 7fafdf7fe640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 0x7fafd80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafeed9d640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 0x7fafd80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafee59c640 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fafe00ee4c0 tx=0x7fafe00f9160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4b5c87ec227b844 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafdf7fe640 1 -- 172.21.15.116:0/398297210 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fafe00f1040 con 0x7faff01006a0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafdcff9640 1 -- 172.21.15.116:0/855019312 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00bd1a0 con 0x5644b517ebf0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafdcff9640 1 -- 172.21.15.116:0/855019312 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafe00bdad0 con 0x5644b517ebf0 2024-03-19T02:47:53.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.957+0000 7fafdcff9640 1 -- 172.21.15.116:0/855019312 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00bcd30 con 0x5644b517ebf0 2024-03-19T02:47:53.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7fafeed9d640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 0x7fafd80a4e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fafe80049d0 tx=0x7fafe8009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 -- 172.21.15.116:0/855019312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 msgr2=0x5644b5187290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fafe00ee4c0 tx=0x7fafe00f9160 comp rx=0 tx=0).stop 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 -- 172.21.15.116:0/855019312 shutdown_connections 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b5189bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517ebf0 0x5644b5187290 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 --2- 172.21.15.116:0/855019312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b517e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 -- 172.21.15.116:0/855019312 >> 172.21.15.116:0/855019312 conn(0x5644b5178a00 msgr2=0x5644b5178de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 -- 172.21.15.116:0/855019312 shutdown_connections 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.958+0000 7faff7573740 1 -- 172.21.15.116:0/855019312 wait complete. 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 -- start start 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 Processor -- start 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b51847f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644b518a380 con 0x5644b517ebf0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5644b509d310 con 0x5644b51877d0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644b518a500 con 0x5644b517e2d0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7faff7573740 10 client.0 _ll_register_callbacks cb 0x5644b50912d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7fafedd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7fafedd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47376/0 (socket says 172.21.15.116:47376) 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7fafedd9b640 1 -- 172.21.15.116:0/1941038434 learned_addr learned my addr 172.21.15.116:0/1941038434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7fafee59c640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b51847f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.959+0000 7fafeed9d640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 -- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 msgr2=0x5644b51847f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b51847f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 -- 172.21.15.116:0/1941038434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 msgr2=0x5644b5185270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 -- 172.21.15.116:0/1941038434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b4df53f0 con 0x5644b517ebf0 2024-03-19T02:47:53.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafedd9b640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fafe40126e0 tx=0x7fafe4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340949544b36289a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4012110 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fafe4004050 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafe00bdad0 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fafe4025050 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7faff7573740 10 client.21216 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.960+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5644b4ebf7d0 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafe400fdd0 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fafe40039c0 con 0x5644b517ebf0 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 10 client.21216 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 10 client.21216 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 20 client.21216 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b5197920 0x5644b5199d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 4 client.21216 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fafe00bdad0 con 0x5644b5197920 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7fafee59c640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b5197920 0x5644b5199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7fafd5ffb640 10 client.21216 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.961+0000 7fafee59c640 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b5197920 0x5644b5199d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a460b04cfe3cf95 server_cookie=e225f227c9e5f006 in_seq=0 out_seq=0 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7fafd5ffb640 1 -- 172.21.15.116:0/1941038434 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fafe00bdad0 con 0x5644b5197920 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7fafd5ffb640 10 client.21216 handle_command_reply: tid=0 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 client.21216 shutdown 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 msgr2=0x5644b5184d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fafe40126e0 tx=0x7fafe4012ba0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 shutdown_connections 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b5197920 0x5644b5199d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517ebf0 0x5644b5184d30 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 --2- 172.21.15.116:0/1941038434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517e2d0 0x5644b51847f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 >> 172.21.15.116:0/1941038434 conn(0x5644b5178a00 msgr2=0x5644b517a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7fafd5ffb640 0 client.21216 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:53.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 shutdown_connections 2024-03-19T02:47:53.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 1 -- 172.21.15.116:0/1941038434 wait complete. 2024-03-19T02:47:53.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.962+0000 7faff7573740 20 client.21216 trim_cache size 0 max 16384 2024-03-19T02:47:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 Processor -- start 2024-03-19T02:47:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 -- start start 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5193050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faff01aebd0 con 0x5644b517e2d0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faff01b0710 con 0x5644b51877d0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7faff7573740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faff01af2a0 con 0x5644b517ebf0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47386/0 (socket says 172.21.15.116:47386) 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 -- 172.21.15.116:0/8276828 learned_addr learned my addr 172.21.15.116:0/8276828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafedd9b640 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5193050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafeed9d640 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 -- 172.21.15.116:0/8276828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 msgr2=0x5644b5193050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5193050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 -- 172.21.15.116:0/8276828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 msgr2=0x5644b5185090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.979+0000 7fafee59c640 1 -- 172.21.15.116:0/8276828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b51837a0 con 0x5644b517e2d0 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7fafee59c640 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fafe002c780 tx=0x7fafe002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61ea170b17afa33 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7fafd7fff640 1 -- 172.21.15.116:0/8276828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00bcf10 con 0x5644b517e2d0 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7fafd7fff640 1 -- 172.21.15.116:0/8276828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafe00bdad0 con 0x5644b517e2d0 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7fafd7fff640 1 -- 172.21.15.116:0/8276828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00f4880 con 0x5644b517e2d0 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7faff7573740 1 -- 172.21.15.116:0/8276828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 msgr2=0x5644b51830c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.980+0000 7faff7573740 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fafe002c780 tx=0x7fafe002bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- 172.21.15.116:0/8276828 shutdown_connections 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b51877d0 0x5644b5185090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5193050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- 172.21.15.116:0/8276828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b517e2d0 0x5644b51830c0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- 172.21.15.116:0/8276828 >> 172.21.15.116:0/8276828 conn(0x5644b5178a00 msgr2=0x5644b5177ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- 172.21.15.116:0/8276828 shutdown_connections 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- 172.21.15.116:0/8276828 wait complete. 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- start start 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 Processor -- start 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5182020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faff01b0040 con 0x5644b51877d0 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faff01af970 con 0x5644b517e2d0 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faff01acbd0 con 0x5644b517ebf0 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7faff7573740 10 client.0 _ll_register_callbacks cb 0x7fafe4009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafeed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafee59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51240/0 (socket says 172.21.15.116:51240) 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafee59c640 1 -- 172.21.15.116:0/3718667843 learned_addr learned my addr 172.21.15.116:0/3718667843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafeed9d640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47392/0 (socket says 172.21.15.116:47392) 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.981+0000 7fafedd9b640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5182020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 -- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 msgr2=0x5644b5182020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5182020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 -- 172.21.15.116:0/3718667843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 msgr2=0x5644b51fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 -- 172.21.15.116:0/3718667843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b51fd140 con 0x5644b517e2d0 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafeed9d640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafee59c640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fafe00ee570 tx=0x7fafe00c5c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d7b24f669a9db3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:47:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe00bcf10 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fafe00b9a40 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafe00bdad0 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fafe00bd3d0 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7faff7573740 10 client.30947 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.982+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5644b51fe3a0 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafe00bdad0 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fafe00ee5a0 con 0x5644b517e2d0 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 10 client.30947 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 10 client.30947 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 20 client.30947 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b51fe790 0x5644b5200b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 4 client.30947 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644b513f3e0 con 0x5644b51fe790 2024-03-19T02:47:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7fafedd9b640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b51fe790 0x5644b5200b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7fafedd9b640 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b51fe790 0x5644b5200b80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb667df7be19712b server_cookie=99ec7d660289daea in_seq=0 out_seq=0 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.983+0000 7fafd5ffb640 10 client.30947 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7fafd5ffb640 1 -- 172.21.15.116:0/3718667843 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644b513f3e0 con 0x5644b51fe790 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7fafd5ffb640 10 client.30947 handle_command_reply: tid=0 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 client.30947 shutdown 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 msgr2=0x5644b5181ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fafe00ee570 tx=0x7fafe00c5c70 comp rx=0 tx=0).stop 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 shutdown_connections 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x5644b51fe790 0x5644b5200b80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644b51877d0 0x5644b51fcc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7fafd5ffb640 0 client.30947 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:47:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b517ebf0 0x5644b5182020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 --2- 172.21.15.116:0/3718667843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5644b517e2d0 0x5644b5181ae0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 >> 172.21.15.116:0/3718667843 conn(0x5644b5178a00 msgr2=0x5644b5179ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 shutdown_connections 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 1 -- 172.21.15.116:0/3718667843 wait complete. 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.984+0000 7faff7573740 20 client.30947 trim_cache size 0 max 16384 2024-03-19T02:47:53.992 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 msgr2=0x7fafd80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 0x7fafd80a4e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fafe80049d0 tx=0x7fafe8009500 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 msgr2=0x7faff0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009230 tx=0x7fafe00045e0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 shutdown_connections 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fafd80a2970 0x7fafd80a4e30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faff01066b0 0x7faff019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faff0101060 0x7faff0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 --2- 172.21.15.116:0/398297210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faff01006a0 0x7faff0197fc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 >> 172.21.15.116:0/398297210 conn(0x7faff00778d0 msgr2=0x7faff0068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 shutdown_connections 2024-03-19T02:47:53.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:53.985+0000 7faff48ac640 1 -- 172.21.15.116:0/398297210 wait complete. 2024-03-19T02:47:54.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:47:54.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 -- 172.21.15.116:0/2145343695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038810bbd0 msgr2=0x7f038810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 --2- 172.21.15.116:0/2145343695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038810bbd0 0x7f038810e050 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0380009ca0 tx=0x7f038002c180 comp rx=0 tx=0).stop 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 -- 172.21.15.116:0/2145343695 shutdown_connections 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 --2- 172.21.15.116:0/2145343695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f038810bbd0 0x7f038810e050 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 --2- 172.21.15.116:0/2145343695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0388103120 0x7f038810b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 --2- 172.21.15.116:0/2145343695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388102800 0x7f0388102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 -- 172.21.15.116:0/2145343695 >> 172.21.15.116:0/2145343695 conn(0x7f03880772d0 msgr2=0x7f03880fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 -- 172.21.15.116:0/2145343695 shutdown_connections 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.124+0000 7f038d7ff640 1 -- 172.21.15.116:0/2145343695 wait complete. 2024-03-19T02:47:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 Processor -- start 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 -- start start 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 0x7f0388100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 0x7f0388100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0388111860 con 0x7f0388102800 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03881116e0 con 0x7f038810bbd0 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f038d7ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f03881119e0 con 0x7f0388103120 2024-03-19T02:47:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f03867fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 0x7f0388100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47422/0 (socket says 172.21.15.116:47422) 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f0386ffd640 1 -- 172.21.15.116:0/3318889304 learned_addr learned my addr 172.21.15.116:0/3318889304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.125+0000 7f03877fe640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 0x7f0388100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 -- 172.21.15.116:0/3318889304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 msgr2=0x7f0388100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 0x7f0388100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 -- 172.21.15.116:0/3318889304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 msgr2=0x7f0388100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 0x7f0388100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 -- 172.21.15.116:0/3318889304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03881a62c0 con 0x7f0388102800 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f03867fc640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 0x7f0388100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f03877fe640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 0x7f0388100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f037800e9f0 tx=0x7f037800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f036ffff640 1 -- 172.21.15.116:0/3318889304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037800cd80 con 0x7f0388102800 2024-03-19T02:47:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881a65b0 con 0x7f0388102800 2024-03-19T02:47:54.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881a6b00 con 0x7f0388102800 2024-03-19T02:47:54.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f036ffff640 1 -- 172.21.15.116:0/3318889304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03780054d0 con 0x7f0388102800 2024-03-19T02:47:54.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.126+0000 7f036ffff640 1 -- 172.21.15.116:0/3318889304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378010490 con 0x7f0388102800 2024-03-19T02:47:54.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 Processor -- start 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 -- start start 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f036ffff640 1 -- 172.21.15.116:0/3318889304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0378005020 con 0x7f0388102800 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603ea990 0x55b5603ecd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5603444b0 con 0x55b5603e1db0 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b5602ecd70 con 0x55b5603e1490 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03903ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b560340550 con 0x55b5603ea990 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47426/0 (socket says 172.21.15.116:47426) 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f0386ffd640 1 -- 172.21.15.116:0/3429966625 learned_addr learned my addr 172.21.15.116:0/3429966625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03877fe640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.129+0000 7f03867fc640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603ea990 0x55b5603ecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f036ffff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 0x7f03700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 -- 172.21.15.116:0/3429966625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603ea990 msgr2=0x55b5603ecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603ea990 0x55b5603ecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 -- 172.21.15.116:0/3429966625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 msgr2=0x55b5603e1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 -- 172.21.15.116:0/3429966625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b56030f210 con 0x55b5603e1db0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f03877fe640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f03877fe640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 0x7f03700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f0386ffd640 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc7c0 tx=0x7f0378097b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3391579347d48355 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f036ffff640 1 -- 172.21.15.116:0/3318889304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f03780cf4e0 con 0x7f0388102800 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f036d7fa640 1 -- 172.21.15.116:0/3429966625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037809b530 con 0x55b5603e1db0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f036d7fa640 1 -- 172.21.15.116:0/3429966625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0378005310 con 0x55b5603e1db0 2024-03-19T02:47:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.130+0000 7f036d7fa640 1 -- 172.21.15.116:0/3429966625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378099d20 con 0x55b5603e1db0 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- 172.21.15.116:0/3429966625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 msgr2=0x55b5603ea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03877fe640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 0x7f03700a4e30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0380004af0 tx=0x7f038002e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc7c0 tx=0x7f0378097b30 comp rx=0 tx=0).stop 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- 172.21.15.116:0/3429966625 shutdown_connections 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603ea990 0x55b5603ecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603e1db0 0x55b5603ea450 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 --2- 172.21.15.116:0/3429966625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- 172.21.15.116:0/3429966625 >> 172.21.15.116:0/3429966625 conn(0x55b5603dbbc0 msgr2=0x55b5603dbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- 172.21.15.116:0/3429966625 shutdown_connections 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- 172.21.15.116:0/3429966625 wait complete. 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 -- start start 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.131+0000 7f03903ee740 1 Processor -- start 2024-03-19T02:47:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603e7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5603ed780 con 0x55b5603ea990 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b560300310 con 0x55b5603e1490 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b5603ed900 con 0x55b5603e1db0 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03903ee740 10 client.0 _ll_register_callbacks cb 0x55b5602f42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47434/0 (socket says 172.21.15.116:47434) 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 -- 172.21.15.116:0/1051573719 learned_addr learned my addr 172.21.15.116:0/1051573719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03867fc640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603e7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 -- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 msgr2=0x55b5603e7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603e7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 -- 172.21.15.116:0/1051573719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 msgr2=0x55b5603e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.132+0000 7f03877fe640 1 -- 172.21.15.116:0/1051573719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5600583f0 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03877fe640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0380036000 tx=0x7f0380005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32598a5a5fe2900 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0380052060 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f0380032d70 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5603a23e0 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0380039040 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 10 client.21234 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b5601227d0 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b5603a23e0 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f038003f040 con 0x55b5603ea990 2024-03-19T02:47:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 10 client.21234 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 10 client.21234 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 20 client.21234 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.133+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b5603fae30 0x55b5603fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.134+0000 7f03903ee740 4 client.21234 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.134+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5603519f0 con 0x55b5603fae30 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.134+0000 7f0386ffd640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b5603fae30 0x55b5603fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.134+0000 7f036aff5640 10 client.21234 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.134+0000 7f0386ffd640 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b5603fae30 0x55b5603fd220 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64218ab91bb0736e server_cookie=adc259692df152c6 in_seq=0 out_seq=0 2024-03-19T02:47:54.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f036aff5640 1 -- 172.21.15.116:0/1051573719 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5603519f0 con 0x55b5603fae30 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f036aff5640 10 client.21234 handle_command_reply: tid=0 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 client.21234 shutdown 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 msgr2=0x55b5603e8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0380036000 tx=0x7f0380005c50 comp rx=0 tx=0).stop 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 shutdown_connections 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b5603fae30 0x55b5603fd220 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603e8410 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f036aff5640 0 client.21234 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 --2- 172.21.15.116:0/1051573719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 >> 172.21.15.116:0/1051573719 conn(0x55b5603dbbc0 msgr2=0x55b5603dd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 shutdown_connections 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 1 -- 172.21.15.116:0/1051573719 wait complete. 2024-03-19T02:47:54.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.135+0000 7f03903ee740 20 client.21234 trim_cache size 0 max 16384 2024-03-19T02:47:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.155+0000 7f03903ee740 1 Processor -- start 2024-03-19T02:47:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 -- start start 2024-03-19T02:47:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1db0 0x55b5603f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b560402b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f037400ff60 con 0x55b5603ea990 2024-03-19T02:47:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0374011aa0 con 0x55b5603e1db0 2024-03-19T02:47:54.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.156+0000 7f03903ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0374010630 con 0x55b5603e1490 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56324/0 (socket says 172.21.15.116:56324) 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 -- 172.21.15.116:0/4153137065 learned_addr learned my addr 172.21.15.116:0/4153137065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 -- 172.21.15.116:0/4153137065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1db0 msgr2=0x55b5603f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1db0 0x55b5603f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 -- 172.21.15.116:0/4153137065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 msgr2=0x55b560402b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b560402b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 -- 172.21.15.116:0/4153137065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5603e1be0 con 0x55b5603e1490 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.157+0000 7f0386ffd640 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc900 tx=0x7f03780d50e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a9c0886299f35a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.158+0000 7f036cff9640 1 -- 172.21.15.116:0/4153137065 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03780af070 con 0x55b5603e1490 2024-03-19T02:47:54.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.158+0000 7f036cff9640 1 -- 172.21.15.116:0/4153137065 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f037809be60 con 0x55b5603e1490 2024-03-19T02:47:54.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.158+0000 7f036cff9640 1 -- 172.21.15.116:0/4153137065 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378099f10 con 0x55b5603e1490 2024-03-19T02:47:54.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.159+0000 7f03903ee740 1 -- 172.21.15.116:0/4153137065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 msgr2=0x55b5603ea0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.159+0000 7f03903ee740 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03780cc900 tx=0x7f03780d50e0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.160+0000 7f03903ee740 1 -- 172.21.15.116:0/4153137065 shutdown_connections 2024-03-19T02:47:54.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.160+0000 7f03903ee740 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b560402b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.160+0000 7f03903ee740 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1db0 0x55b5603f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.160+0000 7f03903ee740 1 --2- 172.21.15.116:0/4153137065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1490 0x55b5603ea0c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.160+0000 7f03903ee740 1 -- 172.21.15.116:0/4153137065 >> 172.21.15.116:0/4153137065 conn(0x55b5603dbbc0 msgr2=0x55b5603e6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- 172.21.15.116:0/4153137065 shutdown_connections 2024-03-19T02:47:54.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- 172.21.15.116:0/4153137065 wait complete. 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- start start 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 Processor -- start 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603eef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03740113d0 con 0x55b5603ea990 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0374010d00 con 0x55b5603e1490 2024-03-19T02:47:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f037400df60 con 0x55b5603e1db0 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.161+0000 7f03903ee740 10 client.0 _ll_register_callbacks cb 0x7f0378005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f0386ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51292/0 (socket says 172.21.15.116:51292) 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f0386ffd640 1 -- 172.21.15.116:0/63940257 learned_addr learned my addr 172.21.15.116:0/63940257 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03867fc640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 -- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 msgr2=0x55b5603ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 -- 172.21.15.116:0/63940257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 msgr2=0x55b5603ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 -- 172.21.15.116:0/63940257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b560468950 con 0x55b5603ea990 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f0386ffd640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f03877fe640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603eef20 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0380037040 tx=0x7f0380032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1a22e4936bb2ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f036aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.162+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0380061020 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f0380041670 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037809be60 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f038006c050 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 10 client.21237 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b560469c00 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f037809be60 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f038005d4b0 con 0x55b5603ea990 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 10 client.21237 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 10 client.21237 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 20 client.21237 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b56046a370 0x55b56046c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 4 client.21237 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b56034bd40 con 0x55b56046a370 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.163+0000 7f0386ffd640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b56046a370 0x55b56046c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f036aff5640 10 client.21237 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f0386ffd640 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b56046a370 0x55b56046c790 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=597b7d788ee850b7 server_cookie=2e0922f5fb88afa1 in_seq=0 out_seq=0 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f036aff5640 1 -- 172.21.15.116:0/63940257 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b56034bd40 con 0x55b56046a370 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f036aff5640 10 client.21237 handle_command_reply: tid=0 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 client.21237 shutdown 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 msgr2=0x55b5603eef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603eef20 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0380037040 tx=0x7f0380032c70 comp rx=0 tx=0).stop 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 shutdown_connections 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b56046a370 0x55b56046c790 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5603ea990 0x55b5603eef20 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5603e1db0 0x55b5603ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 --2- 172.21.15.116:0/63940257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b5603e1490 0x55b5603ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 >> 172.21.15.116:0/63940257 conn(0x55b5603dbbc0 msgr2=0x55b5603da470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.164+0000 7f036aff5640 0 client.21237 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 shutdown_connections 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f03903ee740 1 -- 172.21.15.116:0/63940257 wait complete. 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f03903ee740 20 client.21237 trim_cache size 0 max 16384 2024-03-19T02:47:54.169 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 msgr2=0x7f03700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 0x7f03700a4e30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0380004af0 tx=0x7f038002e040 comp rx=0 tx=0).stop 2024-03-19T02:47:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 msgr2=0x7f0388100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f037800e9f0 tx=0x7f037800edf0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 shutdown_connections 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f03700a2970 0x7f03700a4e30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f038810bbd0 0x7f0388100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0388103120 0x7f0388100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 --2- 172.21.15.116:0/3318889304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0388102800 0x7f0388100450 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 >> 172.21.15.116:0/3318889304 conn(0x7f03880772d0 msgr2=0x7f03880fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 shutdown_connections 2024-03-19T02:47:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.165+0000 7f038d7ff640 1 -- 172.21.15.116:0/3318889304 wait complete. 2024-03-19T02:47:54.184 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:53 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/701325499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:54.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:47:54.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:53 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/701325499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 -- 172.21.15.116:0/2094378464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 msgr2=0x7f9be8107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 --2- 172.21.15.116:0/2094378464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be8107b80 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009cd0 tx=0x7f9be402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 -- 172.21.15.116:0/2094378464 shutdown_connections 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 --2- 172.21.15.116:0/2094378464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 --2- 172.21.15.116:0/2094378464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 0x7f9be8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 --2- 172.21.15.116:0/2094378464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be8107b80 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 -- 172.21.15.116:0/2094378464 >> 172.21.15.116:0/2094378464 conn(0x7f9be8076c50 msgr2=0x7f9be80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 -- 172.21.15.116:0/2094378464 shutdown_connections 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.303+0000 7f9bef60b640 1 -- 172.21.15.116:0/2094378464 wait complete. 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 Processor -- start 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 -- start start 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 0x7f9be8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9be8111a60 con 0x7f9be81077a0 2024-03-19T02:47:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81118e0 con 0x7f9be8100680 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bef60b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9be8111be0 con 0x7f9be80ffcc0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47472/0 (socket says 172.21.15.116:47472) 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9becba6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bedba8640 1 -- 172.21.15.116:0/3243955070 learned_addr learned my addr 172.21.15.116:0/3243955070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 0x7f9be8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bedba8640 1 -- 172.21.15.116:0/3243955070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 msgr2=0x7f9be8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.304+0000 7f9bedba8640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 0x7f9be8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bedba8640 1 -- 172.21.15.116:0/3243955070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 msgr2=0x7f9be81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bedba8640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bedba8640 1 -- 172.21.15.116:0/3243955070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be819d060 con 0x7f9be81077a0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9becba6640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bedba8640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9be000e970 tx=0x7f9be000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bde7fc640 1 -- 172.21.15.116:0/3243955070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be000ccc0 con 0x7f9be81077a0 2024-03-19T02:47:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be819d350 con 0x7f9be81077a0 2024-03-19T02:47:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bde7fc640 1 -- 172.21.15.116:0/3243955070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be000ce60 con 0x7f9be81077a0 2024-03-19T02:47:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bde7fc640 1 -- 172.21.15.116:0/3243955070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0010430 con 0x7f9be81077a0 2024-03-19T02:47:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.305+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be81a4be0 con 0x7f9be81077a0 2024-03-19T02:47:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.307+0000 7f9bde7fc640 1 -- 172.21.15.116:0/3243955070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9be0004ff0 con 0x7f9be81077a0 2024-03-19T02:47:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 Processor -- start 2024-03-19T02:47:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bde7fc640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 0x7f9bd40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 -- start start 2024-03-19T02:47:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bde7fc640 1 -- 172.21.15.116:0/3243955070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f9be00cf5e0 con 0x7f9be81077a0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9becba6640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 0x7f9bd40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e7c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559717dd64b0 con 0x559717e732d0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559717d7ed70 con 0x559717e73bf0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559717dd2550 con 0x559717e7c7d0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47480/0 (socket says 172.21.15.116:47480) 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 -- 172.21.15.116:0/3017438836 learned_addr learned my addr 172.21.15.116:0/3017438836 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bedba8640 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9becba6640 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e7c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9becba6640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 0x7f9bd40a4e30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9be80fdf60 tx=0x7f9bd8011860 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 -- 172.21.15.116:0/3017438836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 msgr2=0x559717e7ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 -- 172.21.15.116:0/3017438836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 msgr2=0x559717e7c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.308+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e7c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bed3a7640 1 -- 172.21.15.116:0/3017438836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559717da1210 con 0x559717e732d0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ca0 tx=0x7f9be4032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa227baed344c7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bd37fe640 1 -- 172.21.15.116:0/3017438836 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4043070 con 0x559717e732d0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bd37fe640 1 -- 172.21.15.116:0/3017438836 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be4032db0 con 0x559717e732d0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bd37fe640 1 -- 172.21.15.116:0/3017438836 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4039040 con 0x559717e732d0 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.309+0000 7f9bf21f9740 1 -- 172.21.15.116:0/3017438836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 msgr2=0x559717e736b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ca0 tx=0x7f9be4032340 comp rx=0 tx=0).stop 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- 172.21.15.116:0/3017438836 shutdown_connections 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e7c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/3017438836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e736b0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- 172.21.15.116:0/3017438836 >> 172.21.15.116:0/3017438836 conn(0x559717e6da00 msgr2=0x559717e6dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- 172.21.15.116:0/3017438836 shutdown_connections 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- 172.21.15.116:0/3017438836 wait complete. 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- start start 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 Processor -- start 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e797d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e79d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559717e7f560 con 0x559717e732d0 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559717d92310 con 0x559717e73bf0 2024-03-19T02:47:54.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.310+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559717e7f6e0 con 0x559717e7c7d0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bf21f9740 10 client.0 _ll_register_callbacks cb 0x559717d862d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56376/0 (socket says 172.21.15.116:56376) 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 -- 172.21.15.116:0/2890414410 learned_addr learned my addr 172.21.15.116:0/2890414410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e797d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9becba6640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e79d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 -- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 msgr2=0x559717e79d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e79d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 -- 172.21.15.116:0/2890414410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 msgr2=0x559717e797d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e797d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 -- 172.21.15.116:0/2890414410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559717aea3f0 con 0x559717e7c7d0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bedba8640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9be0098230 tx=0x7f9be009b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b4592677c07a60b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bd0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be00b3070 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9be4032db0 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.311+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f9be00052e0 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9be009f9b0 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 10 client.30952 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559717bb47d0 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9be4032db0 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f9be009cb90 con 0x559717e7c7d0 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 10 client.30952 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 10 client.30952 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 20 client.30952 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717e8cb10 0x559717e8ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 4 client.30952 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559717e343e0 con 0x559717e8cb10 2024-03-19T02:47:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.312+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717e8cb10 0x559717e8ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.313+0000 7f9bd0ff9640 10 client.30952 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.313+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717e8cb10 0x559717e8ef40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55bc1bae8a8baa7b server_cookie=b84ac8e58a6e4d11 in_seq=0 out_seq=0 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.313+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/2890414410 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559717e343e0 con 0x559717e8cb10 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.313+0000 7f9bd0ff9640 10 client.30952 handle_command_reply: tid=0 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.313+0000 7f9bf21f9740 1 client.30952 shutdown 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 msgr2=0x559717e7a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9be0098230 tx=0x7f9be009b460 comp rx=0 tx=0).stop 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 shutdown_connections 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717e8cb10 0x559717e8ef40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e7c7d0 0x559717e7a250 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e73bf0 0x559717e79d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2890414410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e732d0 0x559717e797d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bd0ff9640 0 client.30952 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:54.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 >> 172.21.15.116:0/2890414410 conn(0x559717e6da00 msgr2=0x559717e6f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 shutdown_connections 2024-03-19T02:47:54.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2890414410 wait complete. 2024-03-19T02:47:54.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.314+0000 7f9bf21f9740 20 client.30952 trim_cache size 0 max 16384 2024-03-19T02:47:54.322 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:53 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/701325499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 Processor -- start 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 -- start start 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e732d0 0x559717e7bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 0x559717e88360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81aebd0 con 0x559717e7c7d0 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81b0710 con 0x559717e732d0 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81af2a0 con 0x559717e73bf0 2024-03-19T02:47:54.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9becba6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 0x559717e88360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bedba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47510/0 (socket says 172.21.15.116:47510) 2024-03-19T02:47:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9bedba8640 1 -- 172.21.15.116:0/2172362116 learned_addr learned my addr 172.21.15.116:0/2172362116 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.331+0000 7f9bed3a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e732d0 0x559717e7bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49116 6.04 MB/sec execute 316 sec latency 151.786 ms 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 -- 172.21.15.116:0/2172362116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e732d0 msgr2=0x559717e7bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e732d0 0x559717e7bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 -- 172.21.15.116:0/2172362116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 msgr2=0x559717e6ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 -- 172.21.15.116:0/2172362116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559717e7ebc0 con 0x559717e73bf0 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9bedba8640 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9becba6640 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 0x559717e88360 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x559717e79030 tx=0x7f9bd801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0323bf8a340346 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9bd2ffd640 1 -- 172.21.15.116:0/2172362116 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8017040 con 0x559717e73bf0 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9bd2ffd640 1 -- 172.21.15.116:0/2172362116 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd8019050 con 0x559717e73bf0 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.332+0000 7f9bd2ffd640 1 -- 172.21.15.116:0/2172362116 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8024730 con 0x559717e73bf0 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2172362116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 msgr2=0x559717e88360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 0x559717e88360 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x559717e79030 tx=0x7f9bd801d000 comp rx=0 tx=0).stop 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2172362116 shutdown_connections 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e7c7d0 0x559717e6ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e73bf0 0x559717e88360 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/2172362116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e732d0 0x559717e7bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2172362116 >> 172.21.15.116:0/2172362116 conn(0x559717e6da00 msgr2=0x559717e6c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2172362116 shutdown_connections 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.333+0000 7f9bf21f9740 1 -- 172.21.15.116:0/2172362116 wait complete. 2024-03-19T02:47:54.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 -- start start 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 Processor -- start 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e73bf0 0x559717e80a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 0x559717e80f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81b0040 con 0x559717e73bf0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81af970 con 0x559717e7c7d0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9be81acbd0 con 0x559717e732d0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bed3a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9becba6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e73bf0 0x559717e80a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bed3a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56394/0 (socket says 172.21.15.116:56394) 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bed3a7640 1 -- 172.21.15.116:0/109161347 learned_addr learned my addr 172.21.15.116:0/109161347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bf21f9740 10 client.0 _ll_register_callbacks cb 0x7f9be00046e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.334+0000 7f9bedba8640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 0x559717e80f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 -- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 msgr2=0x559717e80f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 0x559717e80f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 -- 172.21.15.116:0/109161347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e73bf0 msgr2=0x559717e80a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e73bf0 0x559717e80a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 -- 172.21.15.116:0/109161347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559717ef2090 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bedba8640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 0x559717e80f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bed3a7640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9be4005cc0 tx=0x7f9be4009860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ad61ecbf3c0153 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4057060 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f9be40039f0 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd8019050 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9be405e050 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bf21f9740 10 client.30958 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559717ef33b0 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd8019050 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.335+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f9be40324f0 con 0x559717e732d0 2024-03-19T02:47:54.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 10 client.30958 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 10 client.30958 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 20 client.30958 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717ef36a0 0x559717ef5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 4 client.30958 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559717de39f0 con 0x559717ef36a0 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9becba6640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717ef36a0 0x559717ef5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9bd0ff9640 10 client.30958 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:54.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.336+0000 7f9becba6640 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717ef36a0 0x559717ef5a90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4d3d8d8ad3786ff server_cookie=708c3a624a6b03aa in_seq=0 out_seq=0 2024-03-19T02:47:54.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bd0ff9640 1 -- 172.21.15.116:0/109161347 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559717de39f0 con 0x559717ef36a0 2024-03-19T02:47:54.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bd0ff9640 10 client.30958 handle_command_reply: tid=0 2024-03-19T02:47:54.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 client.30958 shutdown 2024-03-19T02:47:54.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 msgr2=0x559717e804c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9be4005cc0 tx=0x7f9be4009860 comp rx=0 tx=0).stop 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 shutdown_connections 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559717ef36a0 0x559717ef5a90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559717e7c7d0 0x559717e80f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bd0ff9640 0 client.30958 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559717e73bf0 0x559717e80a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 --2- 172.21.15.116:0/109161347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559717e732d0 0x559717e804c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 >> 172.21.15.116:0/109161347 conn(0x559717e6da00 msgr2=0x559717e6cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 shutdown_connections 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 1 -- 172.21.15.116:0/109161347 wait complete. 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.338+0000 7f9bf21f9740 20 client.30958 trim_cache size 0 max 16384 2024-03-19T02:47:54.344 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 msgr2=0x7f9bd40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 0x7f9bd40a4e30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9be80fdf60 tx=0x7f9bd8011860 comp rx=0 tx=0).stop 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 msgr2=0x7f9be819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9be000e970 tx=0x7f9be000ed70 comp rx=0 tx=0).stop 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 shutdown_connections 2024-03-19T02:47:54.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f9bd40a2970 0x7f9bd40a4e30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9be81077a0 0x7f9be819c930 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8100680 0x7f9be81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 --2- 172.21.15.116:0/3243955070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9be80ffcc0 0x7f9be8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 >> 172.21.15.116:0/3243955070 conn(0x7f9be8076c50 msgr2=0x7f9be80fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 shutdown_connections 2024-03-19T02:47:54.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.339+0000 7f9bef60b640 1 -- 172.21.15.116:0/3243955070 wait complete. 2024-03-19T02:47:54.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:47:54.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 -- 172.21.15.116:0/1223967473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641017f0 msgr2=0x7f0764101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 --2- 172.21.15.116:0/1223967473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641017f0 0x7f0764101c70 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f0758009930 tx=0x7f075802bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:54.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 -- 172.21.15.116:0/1223967473 shutdown_connections 2024-03-19T02:47:54.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 --2- 172.21.15.116:0/1223967473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07641021b0 0x7f076410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 --2- 172.21.15.116:0/1223967473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641017f0 0x7f0764101c70 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 --2- 172.21.15.116:0/1223967473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0764107800 0x7f0764107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 -- 172.21.15.116:0/1223967473 >> 172.21.15.116:0/1223967473 conn(0x7f07640fd2c0 msgr2=0x7f07640ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 -- 172.21.15.116:0/1223967473 shutdown_connections 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.476+0000 7f07693b2640 1 -- 172.21.15.116:0/1223967473 wait complete. 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 Processor -- start 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 -- start start 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07641017f0 0x7f0764197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0764107800 0x7f076419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0764111ad0 con 0x7f07641021b0 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0764111950 con 0x7f07641017f0 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07693b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0764111c50 con 0x7f0764107800 2024-03-19T02:47:54.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47536/0 (socket says 172.21.15.116:47536) 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07627fc640 1 -- 172.21.15.116:0/2130213020 learned_addr learned my addr 172.21.15.116:0/2130213020 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f07637fe640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0764107800 0x7f076419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.477+0000 7f0762ffd640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07641017f0 0x7f0764197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 -- 172.21.15.116:0/2130213020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0764107800 msgr2=0x7f076419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0764107800 0x7f076419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 -- 172.21.15.116:0/2130213020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07641017f0 msgr2=0x7f0764197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07641017f0 0x7f0764197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 -- 172.21.15.116:0/2130213020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076419cfe0 con 0x7f07641021b0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07627fc640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f0758000c00 tx=0x7f075802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f074bfff640 1 -- 172.21.15.116:0/2130213020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758043020 con 0x7f07641021b0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076419d270 con 0x7f07641021b0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f074bfff640 1 -- 172.21.15.116:0/2130213020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f075802caf0 con 0x7f07641021b0 2024-03-19T02:47:54.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f074bfff640 1 -- 172.21.15.116:0/2130213020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758003a80 con 0x7f07641021b0 2024-03-19T02:47:54.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.478+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07641a4b60 con 0x7f07641021b0 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.480+0000 7f074bfff640 1 -- 172.21.15.116:0/2130213020 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0758046020 con 0x7f07641021b0 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.481+0000 7f074bfff640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 0x7f074c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.481+0000 7f076bf9e740 1 Processor -- start 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.481+0000 7f0762ffd640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 0x7f074c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.481+0000 7f074bfff640 1 -- 172.21.15.116:0/2130213020 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f07580f1130 con 0x7f07641021b0 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.481+0000 7f076bf9e740 1 -- start start 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744255f90 0x557744256370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x557744265fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577441b94b0 con 0x5577442568b0 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557744161d70 con 0x557744255f90 2024-03-19T02:47:54.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f076bf9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5577441b5550 con 0x557744263be0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f0762ffd640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 0x7f074c0a4e30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0754009b50 tx=0x7f0754005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07637fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x557744265fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47542/0 (socket says 172.21.15.116:47542) 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 -- 172.21.15.116:0/2115638591 learned_addr learned my addr 172.21.15.116:0/2115638591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f0762ffd640 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744255f90 0x557744256370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 -- 172.21.15.116:0/2115638591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 msgr2=0x557744265fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x557744265fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 -- 172.21.15.116:0/2115638591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744255f90 msgr2=0x557744256370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744255f90 0x557744256370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 -- 172.21.15.116:0/2115638591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557744184210 con 0x5577442568b0 2024-03-19T02:47:54.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07637fe640 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x557744265fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07627fc640 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f07580b9920 tx=0x7f0758009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a490bbeba35a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07497fa640 1 -- 172.21.15.116:0/2115638591 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07580bd4c0 con 0x5577442568b0 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07497fa640 1 -- 172.21.15.116:0/2115638591 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07580c1610 con 0x5577442568b0 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.482+0000 7f07497fa640 1 -- 172.21.15.116:0/2115638591 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07580bd980 con 0x5577442568b0 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 -- 172.21.15.116:0/2115638591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 msgr2=0x5577442636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f07580b9920 tx=0x7f0758009610 comp rx=0 tx=0).stop 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 -- 172.21.15.116:0/2115638591 shutdown_connections 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x557744265fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577442568b0 0x5577442636a0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 --2- 172.21.15.116:0/2115638591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744255f90 0x557744256370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 -- 172.21.15.116:0/2115638591 >> 172.21.15.116:0/2115638591 conn(0x557744250a00 msgr2=0x557744250de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 -- 172.21.15.116:0/2115638591 shutdown_connections 2024-03-19T02:47:54.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.483+0000 7f076bf9e740 1 -- 172.21.15.116:0/2115638591 wait complete. 2024-03-19T02:47:54.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 -- start start 2024-03-19T02:47:54.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 Processor -- start 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x557744261160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442616a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557744266790 con 0x557744255f90 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557744175310 con 0x5577442568b0 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557744266910 con 0x557744263be0 2024-03-19T02:47:54.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f076bf9e740 10 client.0 _ll_register_callbacks cb 0x5577441692d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47554/0 (socket says 172.21.15.116:47554) 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f0762ffd640 1 -- 172.21.15.116:0/1909419157 learned_addr learned my addr 172.21.15.116:0/1909419157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f07637fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.484+0000 7f07627fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x557744261160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 -- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 msgr2=0x5577442616a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442616a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 -- 172.21.15.116:0/1909419157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 msgr2=0x557744261160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x557744261160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 -- 172.21.15.116:0/1909419157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557743ecd3f0 con 0x557744255f90 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0762ffd640 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f0754003a70 tx=0x7f0754004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531eccb7c9aab10c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0746ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075401fe40 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f07540273e0 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07580c1610 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f07540275e0 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f076bf9e740 10 client.21267 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.485+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x557743f977d0 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07580c1610 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f0754026070 con 0x557744255f90 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 10 client.21267 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 10 client.21267 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 20 client.21267 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557744273d70 0x557744276190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 4 client.21267 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577442173e0 con 0x557744273d70 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.486+0000 7f07627fc640 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557744273d70 0x557744276190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f07627fc640 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557744273d70 0x557744276190 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57f6b641ac95d5e8 server_cookie=623556d9aede145a in_seq=0 out_seq=0 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f0746ff5640 10 client.21267 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f0746ff5640 1 -- 172.21.15.116:0/1909419157 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577442173e0 con 0x557744273d70 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f0746ff5640 10 client.21267 handle_command_reply: tid=0 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 client.21267 shutdown 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 msgr2=0x557744260c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f0754003a70 tx=0x7f0754004730 comp rx=0 tx=0).stop 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 shutdown_connections 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557744273d70 0x557744276190 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442616a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x557744261160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 --2- 172.21.15.116:0/1909419157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744260c20 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.487+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 >> 172.21.15.116:0/1909419157 conn(0x557744250a00 msgr2=0x557744252750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.488+0000 7f0746ff5640 0 client.21267 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.488+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 shutdown_connections 2024-03-19T02:47:54.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.488+0000 7f076bf9e740 1 -- 172.21.15.116:0/1909419157 wait complete. 2024-03-19T02:47:54.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.488+0000 7f076bf9e740 20 client.21267 trim_cache size 0 max 16384 2024-03-19T02:47:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 Processor -- start 2024-03-19T02:47:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 -- start start 2024-03-19T02:47:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 0x55774426f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 0x55774425fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07641acab0 con 0x557744255f90 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07641ae5f0 con 0x557744263be0 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f076bf9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07641ad180 con 0x5577442568b0 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47568/0 (socket says 172.21.15.116:47568) 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f0762ffd640 1 -- 172.21.15.116:0/88296979 learned_addr learned my addr 172.21.15.116:0/88296979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f07627fc640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 0x55774426f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.504+0000 7f07637fe640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 0x55774425fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 -- 172.21.15.116:0/88296979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 msgr2=0x55774426f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 0x55774426f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 -- 172.21.15.116:0/88296979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 msgr2=0x55774425fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 0x55774425fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 -- 172.21.15.116:0/88296979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55774425c7f0 con 0x557744255f90 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f07627fc640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 0x55774426f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f07637fe640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 0x55774425fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0762ffd640 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f075401f880 tx=0x7f0754005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649117add110500d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0748ff9640 1 -- 172.21.15.116:0/88296979 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540035f0 con 0x557744255f90 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0748ff9640 1 -- 172.21.15.116:0/88296979 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07540273e0 con 0x557744255f90 2024-03-19T02:47:54.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.505+0000 7f0748ff9640 1 -- 172.21.15.116:0/88296979 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754027d50 con 0x557744255f90 2024-03-19T02:47:54.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- 172.21.15.116:0/88296979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 msgr2=0x557744263350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f075401f880 tx=0x7f0754005830 comp rx=0 tx=0).stop 2024-03-19T02:47:54.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- 172.21.15.116:0/88296979 shutdown_connections 2024-03-19T02:47:54.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557744263be0 0x55774425fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577442568b0 0x55774426f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 --2- 172.21.15.116:0/88296979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744263350 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- 172.21.15.116:0/88296979 >> 172.21.15.116:0/88296979 conn(0x557744250a00 msgr2=0x55774425f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- 172.21.15.116:0/88296979 shutdown_connections 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- 172.21.15.116:0/88296979 wait complete. 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.506+0000 7f076bf9e740 1 -- start start 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 Processor -- start 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x5577442670d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442d2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07641adf20 con 0x557744255f90 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07641ad850 con 0x5577442568b0 2024-03-19T02:47:54.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07641aaab0 con 0x557744263be0 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f076bf9e740 10 client.0 _ll_register_callbacks cb 0x7f0754004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47572/0 (socket says 172.21.15.116:47572) 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 -- 172.21.15.116:0/3099407102 learned_addr learned my addr 172.21.15.116:0/3099407102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f07627fc640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x5577442670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f07637fe640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442d2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 -- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 msgr2=0x5577442d2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442d2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 -- 172.21.15.116:0/3099407102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 msgr2=0x5577442670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x5577442670d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f0762ffd640 1 -- 172.21.15.116:0/3099407102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577442d26c0 con 0x557744255f90 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f07637fe640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442d2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.507+0000 7f07627fc640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x5577442670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0762ffd640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f07540092e0 tx=0x7f07540286f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab815fd2c78985 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540035f0 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f0754027ca0 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07540273e0 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f075404c050 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f076bf9e740 10 client.21273 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5577442d9a60 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07540273e0 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f0754047030 con 0x557744255f90 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.508+0000 7f076bf9e740 10 client.21273 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 10 client.21273 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 20 client.21273 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5577442da120 0x5577442dc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 4 client.21273 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5577441c69f0 con 0x5577442da120 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f07627fc640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5577442da120 0x5577442dc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f0746ff5640 10 client.21273 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f07627fc640 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5577442da120 0x5577442dc540 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d8cefae29639dbda server_cookie=f70b5e7a3aa5ce74 in_seq=0 out_seq=0 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f0746ff5640 1 -- 172.21.15.116:0/3099407102 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5577441c69f0 con 0x5577442da120 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f0746ff5640 10 client.21273 handle_command_reply: tid=0 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.509+0000 7f076bf9e740 1 client.21273 shutdown 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 msgr2=0x557744266b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f07540092e0 tx=0x7f07540286f0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 shutdown_connections 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5577442da120 0x5577442dc540 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557744263be0 0x5577442d2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5577442568b0 0x5577442670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f0746ff5640 0 client.21273 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 --2- 172.21.15.116:0/3099407102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557744255f90 0x557744266b90 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 >> 172.21.15.116:0/3099407102 conn(0x557744250a00 msgr2=0x55774425fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 shutdown_connections 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 1 -- 172.21.15.116:0/3099407102 wait complete. 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f076bf9e740 20 client.21273 trim_cache size 0 max 16384 2024-03-19T02:47:54.517 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 msgr2=0x7f074c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 0x7f074c0a4e30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0754009b50 tx=0x7f0754005f90 comp rx=0 tx=0).stop 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 msgr2=0x7f0764198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.510+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f0758000c00 tx=0x7f075802c660 comp rx=0 tx=0).stop 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 shutdown_connections 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f074c0a2970 0x7f074c0a4e30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0764107800 0x7f076419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641021b0 0x7f0764198530 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 --2- 172.21.15.116:0/2130213020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07641017f0 0x7f0764197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 >> 172.21.15.116:0/2130213020 conn(0x7f07640fd2c0 msgr2=0x7f07640fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 shutdown_connections 2024-03-19T02:47:54.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.511+0000 7f07693b2640 1 -- 172.21.15.116:0/2130213020 wait complete. 2024-03-19T02:47:54.524 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:47:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 -- 172.21.15.116:0/3594779387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81006a0 msgr2=0x7fbbf8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3594779387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81006a0 0x7fbbf8100b20 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009930 tx=0x7fbbe802bc30 comp rx=0 tx=0).stop 2024-03-19T02:47:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 -- 172.21.15.116:0/3594779387 shutdown_connections 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3594779387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3594779387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81006a0 0x7fbbf8100b20 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3594779387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81066b0 0x7fbbf8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 -- 172.21.15.116:0/3594779387 >> 172.21.15.116:0/3594779387 conn(0x7fbbf80778d0 msgr2=0x7fbbf8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 -- 172.21.15.116:0/3594779387 shutdown_connections 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.641+0000 7fbc007e6640 1 -- 172.21.15.116:0/3594779387 wait complete. 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 Processor -- start 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 -- start start 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81006a0 0x7fbbf8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8110970 con 0x7fbbf81066b0 2024-03-19T02:47:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81107f0 con 0x7fbbf8101060 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbc007e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8110af0 con 0x7fbbf81006a0 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47592/0 (socket says 172.21.15.116:47592) 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 -- 172.21.15.116:0/3961063417 learned_addr learned my addr 172.21.15.116:0/3961063417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfe582640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81006a0 0x7fbbf8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 -- 172.21.15.116:0/3961063417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81006a0 msgr2=0x7fbbf8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81006a0 0x7fbbf8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 -- 172.21.15.116:0/3961063417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 msgr2=0x7fbbf81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfed83640 1 -- 172.21.15.116:0/3961063417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf819cf90 con 0x7fbbf81066b0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.642+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbbfed83640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000eb10 tx=0x7fbbf000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbbef7fe640 1 -- 172.21.15.116:0/3961063417 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0004b20 con 0x7fbbf81066b0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf819d280 con 0x7fbbf81066b0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbbef7fe640 1 -- 172.21.15.116:0/3961063417 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf0004cc0 con 0x7fbbf81066b0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbbef7fe640 1 -- 172.21.15.116:0/3961063417 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0010690 con 0x7fbbf81066b0 2024-03-19T02:47:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.643+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf81a4b10 con 0x7fbbf81066b0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.645+0000 7fbbef7fe640 1 -- 172.21.15.116:0/3961063417 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fbbf0005020 con 0x7fbbf81066b0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.645+0000 7fbc033d5740 1 Processor -- start 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 -- start start 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 0x5572016e2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55720163c4b0 con 0x5572016d92d0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572015e4d70 con 0x5572016e27d0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbc033d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557201638550 con 0x5572016d9bf0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47606/0 (socket says 172.21.15.116:47606) 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 -- 172.21.15.116:0/2046709564 learned_addr learned my addr 172.21.15.116:0/2046709564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfe582640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 0x5572016e2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbef7fe640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 0x7fbbe40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 -- 172.21.15.116:0/2046709564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 msgr2=0x5572016e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 0x5572016e2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 -- 172.21.15.116:0/2046709564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 msgr2=0x5572016e4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfdd81640 1 -- 172.21.15.116:0/2046709564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557201607210 con 0x5572016d92d0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfe582640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 0x5572016e2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.646+0000 7fbbfed83640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 0x7fbbe40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.647+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8005c40 tx=0x7fbbe8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7822eb3598762170 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.647+0000 7fbbef7fe640 1 -- 172.21.15.116:0/3961063417 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fbbf00cf860 con 0x7fbbf81066b0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.647+0000 7fbbecff9640 1 -- 172.21.15.116:0/2046709564 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe802c7e0 con 0x5572016d92d0 2024-03-19T02:47:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.647+0000 7fbbecff9640 1 -- 172.21.15.116:0/2046709564 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbe8048070 con 0x5572016d92d0 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.647+0000 7fbbecff9640 1 -- 172.21.15.116:0/2046709564 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8004920 con 0x5572016d92d0 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- 172.21.15.116:0/2046709564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 msgr2=0x5572016d96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbbfed83640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 0x7fbbe40a4e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00ccc80 tx=0x7fbbf0097e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8005c40 tx=0x7fbbe8005bd0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- 172.21.15.116:0/2046709564 shutdown_connections 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d9bf0 0x5572016e2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2046709564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d92d0 0x5572016d96b0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- 172.21.15.116:0/2046709564 >> 172.21.15.116:0/2046709564 conn(0x5572016d3a00 msgr2=0x5572016d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- 172.21.15.116:0/2046709564 shutdown_connections 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- 172.21.15.116:0/2046709564 wait complete. 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 -- start start 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.648+0000 7fbc033d5740 1 Processor -- start 2024-03-19T02:47:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016df7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572016e5560 con 0x5572016d9bf0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572015f8310 con 0x5572016e27d0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5572016e56e0 con 0x5572016d92d0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbc033d5740 10 client.0 _ll_register_callbacks cb 0x5572015ec2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47620/0 (socket says 172.21.15.116:47620) 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfdd81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016df7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 -- 172.21.15.116:0/599276817 learned_addr learned my addr 172.21.15.116:0/599276817 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 -- 172.21.15.116:0/599276817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 msgr2=0x5572016df7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016df7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 -- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 msgr2=0x5572016e0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfe582640 1 -- 172.21.15.116:0/599276817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572013503f0 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.649+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbfe582640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40126e0 tx=0x7fbbf4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf9ebce23ebf64a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40120d0 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fbbf400f920 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe8048070 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbbf4012270 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbc033d5740 10 client.21285 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55720141a7d0 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf4010430 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.650+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fbbf401c420 con 0x5572016d9bf0 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 10 client.21285 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 10 client.21285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:54.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 20 client.21285 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5572016f6fe0 0x5572016f93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 4 client.21285 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbbe8048070 con 0x5572016f6fe0 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5572016f6fe0 0x5572016f93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbbe1ffb640 10 client.21285 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:54.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.651+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5572016f6fe0 0x5572016f93d0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95338629fe8eaaba server_cookie=e7b23bce2e237358 in_seq=0 out_seq=0 2024-03-19T02:47:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/599276817 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fbbe8048070 con 0x5572016f6fe0 2024-03-19T02:47:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbbe1ffb640 10 client.21285 handle_command_reply: tid=0 2024-03-19T02:47:54.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 client.21285 shutdown 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 msgr2=0x5572016dfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40126e0 tx=0x7fbbf4012ba0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 shutdown_connections 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5572016f6fe0 0x5572016f93d0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016dfd20 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 --2- 172.21.15.116:0/599276817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016df7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 >> 172.21.15.116:0/599276817 conn(0x5572016d3a00 msgr2=0x5572016d5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbbe1ffb640 0 client.21285 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 shutdown_connections 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 1 -- 172.21.15.116:0/599276817 wait complete. 2024-03-19T02:47:54.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.654+0000 7fbc033d5740 20 client.21285 trim_cache size 0 max 16384 2024-03-19T02:47:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 Processor -- start 2024-03-19T02:47:54.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 -- start start 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016d92d0 0x5572016ee300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016e27d0 0x5572016d2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81aca20 con 0x5572016d9bf0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ae560 con 0x5572016d92d0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.670+0000 7fbc033d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ad0f0 con 0x5572016e27d0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfdd81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016e27d0 0x5572016d2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47626/0 (socket says 172.21.15.116:47626) 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 -- 172.21.15.116:0/2419103834 learned_addr learned my addr 172.21.15.116:0/2419103834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016d92d0 0x5572016ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 -- 172.21.15.116:0/2419103834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016e27d0 msgr2=0x5572016d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016e27d0 0x5572016d2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 -- 172.21.15.116:0/2419103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016d92d0 msgr2=0x5572016ee300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016d92d0 0x5572016ee300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 -- 172.21.15.116:0/2419103834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572016d9a20 con 0x5572016d9bf0 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbfe582640 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fbbf401ca80 tx=0x7fbbf4009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde59079dd4d013e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbe3fff640 1 -- 172.21.15.116:0/2419103834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf401cb00 con 0x5572016d9bf0 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbe3fff640 1 -- 172.21.15.116:0/2419103834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf401cca0 con 0x5572016d9bf0 2024-03-19T02:47:54.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.671+0000 7fbbe3fff640 1 -- 172.21.15.116:0/2419103834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40053f0 con 0x5572016d9bf0 2024-03-19T02:47:54.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- 172.21.15.116:0/2419103834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 msgr2=0x5572016e0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fbbf401ca80 tx=0x7fbbf4009870 comp rx=0 tx=0).stop 2024-03-19T02:47:54.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- 172.21.15.116:0/2419103834 shutdown_connections 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016e27d0 0x5572016d2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016e0080 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 --2- 172.21.15.116:0/2419103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016d92d0 0x5572016ee300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- 172.21.15.116:0/2419103834 >> 172.21.15.116:0/2419103834 conn(0x5572016d3a00 msgr2=0x5572016d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- 172.21.15.116:0/2419103834 shutdown_connections 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- 172.21.15.116:0/2419103834 wait complete. 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.672+0000 7fbc033d5740 1 -- start start 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 Processor -- start 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016ea750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ade90 con 0x5572016d9bf0 2024-03-19T02:47:54.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ad7c0 con 0x5572016e27d0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81aaa20 con 0x5572016d92d0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbc033d5740 10 client.0 _ll_register_callbacks cb 0x7fbbf4009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47632/0 (socket says 172.21.15.116:47632) 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 -- 172.21.15.116:0/3390297605 learned_addr learned my addr 172.21.15.116:0/3390297605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 -- 172.21.15.116:0/3390297605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 msgr2=0x5572016ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016ea750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 -- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 msgr2=0x5572016e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 -- 172.21.15.116:0/3390297605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572016e65f0 con 0x5572016d9bf0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbfe582640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4012be0 tx=0x7fbbf400fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cec915f0ac75a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:54.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.673+0000 7fbbe1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf401cb00 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fbbf4005170 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf401cca0 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbbf4005390 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 10 client.21291 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55720175ced0 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf401cca0 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fbbf4003cd0 con 0x5572016d9bf0 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 10 client.21291 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 10 client.21291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 20 client.21291 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55720175d5e0 0x55720175fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 4 client.21291 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:47:54.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55720169a3e0 con 0x55720175d5e0 2024-03-19T02:47:54.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.674+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55720175d5e0 0x55720175fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:54.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.675+0000 7fbbe1ffb640 10 client.21291 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:54.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.675+0000 7fbbfdd81640 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55720175d5e0 0x55720175fa00 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=219fe0a2082a7a9e server_cookie=a98976d4598d905f in_seq=0 out_seq=0 2024-03-19T02:47:54.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbbe1ffb640 1 -- 172.21.15.116:0/3390297605 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55720169a3e0 con 0x55720175d5e0 2024-03-19T02:47:54.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbbe1ffb640 10 client.21291 handle_command_reply: tid=0 2024-03-19T02:47:54.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 client.21291 shutdown 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 msgr2=0x5572016eac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4012be0 tx=0x7fbbf400fbd0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 shutdown_connections 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55720175d5e0 0x55720175fa00 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572016e27d0 0x5572016e5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbbe1ffb640 0 client.21291 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572016d9bf0 0x5572016eac90 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 --2- 172.21.15.116:0/3390297605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572016d92d0 0x5572016ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.677+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 >> 172.21.15.116:0/3390297605 conn(0x5572016d3a00 msgr2=0x5572016d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 shutdown_connections 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc033d5740 1 -- 172.21.15.116:0/3390297605 wait complete. 2024-03-19T02:47:54.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc033d5740 20 client.21291 trim_cache size 0 max 16384 2024-03-19T02:47:54.684 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:47:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 msgr2=0x7fbbe40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 0x7fbbe40a4e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00ccc80 tx=0x7fbbf0097e40 comp rx=0 tx=0).stop 2024-03-19T02:47:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 msgr2=0x7fbbf819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000eb10 tx=0x7fbbf000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 shutdown_connections 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fbbe40a2970 0x7fbbe40a4e30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbf81066b0 0x7fbbf819c860 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbf8101060 0x7fbbf81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 --2- 172.21.15.116:0/3961063417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbf81006a0 0x7fbbf8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 >> 172.21.15.116:0/3961063417 conn(0x7fbbf80778d0 msgr2=0x7fbbf8068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 shutdown_connections 2024-03-19T02:47:54.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:54.678+0000 7fbc007e6640 1 -- 172.21.15.116:0/3961063417 wait complete. 2024-03-19T02:47:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:54 smithi154 ceph-mon[35693]: pgmap v633: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 123 op/s 2024-03-19T02:47:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:54 smithi116 ceph-mon[32263]: pgmap v633: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 123 op/s 2024-03-19T02:47:55.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49205 6.03 MB/sec execute 317 sec latency 191.454 ms 2024-03-19T02:47:55.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:54 smithi162 ceph-mon[35761]: pgmap v633: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 123 op/s 2024-03-19T02:47:55.694 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:47:55.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:47:55.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.829+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2334295802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 msgr2=0x7fdb28107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.829+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2334295802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb28107bf0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009900 tx=0x7fdb1802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2334295802 shutdown_connections 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2334295802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb2810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2334295802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 0x7fdb2806a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2334295802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb28107bf0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2334295802 >> 172.21.15.116:0/2334295802 conn(0x7fdb280777b0 msgr2=0x7fdb28077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2334295802 shutdown_connections 2024-03-19T02:47:55.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2334295802 wait complete. 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 Processor -- start 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.830+0000 7fdb2e4ff640 1 -- start start 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 0x7fdb28197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb281984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28111a80 con 0x7fdb28107810 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28111900 con 0x7fdb28069c00 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28111c00 con 0x7fdb2806a5a0 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb27fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 0x7fdb28197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47644/0 (socket says 172.21.15.116:47644) 2024-03-19T02:47:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/2795825121 learned_addr learned my addr 172.21.15.116:0/2795825121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb281984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/2795825121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 msgr2=0x7fdb281984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb281984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/2795825121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 msgr2=0x7fdb28197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 0x7fdb28197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/2795825121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2819cf40 con 0x7fdb28107810 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb281984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000eb10 tx=0x7fdb2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb257fa640 1 -- 172.21.15.116:0/2795825121 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20004b20 con 0x7fdb28107810 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.831+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2819d230 con 0x7fdb28107810 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.832+0000 7fdb257fa640 1 -- 172.21.15.116:0/2795825121 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb20004cc0 con 0x7fdb28107810 2024-03-19T02:47:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.832+0000 7fdb257fa640 1 -- 172.21.15.116:0/2795825121 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20010690 con 0x7fdb28107810 2024-03-19T02:47:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.832+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb281a4ac0 con 0x7fdb28107810 2024-03-19T02:47:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.834+0000 7fdb257fa640 1 -- 172.21.15.116:0/2795825121 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdb20005020 con 0x7fdb28107810 2024-03-19T02:47:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.834+0000 7fdb311c8740 1 Processor -- start 2024-03-19T02:47:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.834+0000 7fdb311c8740 1 -- start start 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c0a3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb257fa640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d15bffe160 con 0x55d15c0a3880 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d15bfa6e10 con 0x55d15c09aca0 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb311c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d15bffa200 con 0x55d15c09a380 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56546/0 (socket says 172.21.15.116:56546) 2024-03-19T02:47:55.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb277fe640 1 -- 172.21.15.116:0/361592538 learned_addr learned my addr 172.21.15.116:0/361592538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb257fa640 1 -- 172.21.15.116:0/2795825121 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdb200cf8a0 con 0x7fdb28107810 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb27fff640 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c0a3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/361592538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 msgr2=0x55d15c09a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/361592538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 msgr2=0x55d15c0a3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c0a3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/361592538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15bdda7e0 con 0x55d15c0a3880 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.835+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 0x7fdb100a4e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c011de0 tx=0x7fdb1c009c70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.836+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0a5c70 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fdb200ccc10 tx=0x7fdb2000efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2279a09f345a719f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.836+0000 7fdb0e7fc640 1 -- 172.21.15.116:0/361592538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2009b660 con 0x55d15c0a3880 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.836+0000 7fdb0e7fc640 1 -- 172.21.15.116:0/361592538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb2009ce50 con 0x55d15c0a3880 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.836+0000 7fdb0e7fc640 1 -- 172.21.15.116:0/361592538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb200a8ec0 con 0x55d15c0a3880 2024-03-19T02:47:55.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- 172.21.15.116:0/361592538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 msgr2=0x55d15c0a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0a5c70 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fdb200ccc10 tx=0x7fdb2000efd0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- 172.21.15.116:0/361592538 shutdown_connections 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0a5c70 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c0a3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 --2- 172.21.15.116:0/361592538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- 172.21.15.116:0/361592538 >> 172.21.15.116:0/361592538 conn(0x55d15c094ab0 msgr2=0x55d15c094e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- 172.21.15.116:0/361592538 shutdown_connections 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- 172.21.15.116:0/361592538 wait complete. 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 -- start start 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.837+0000 7fdb311c8740 1 Processor -- start 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 0x55d15c0a1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d15c0a6610 con 0x55d15c09aca0 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d15bfba1a0 con 0x55d15c09a380 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d15c0a6790 con 0x55d15c0a3880 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb311c8740 10 client.0 _ll_register_callbacks cb 0x55d15bd11c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47654/0 (socket says 172.21.15.116:47654) 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 -- 172.21.15.116:0/812525989 learned_addr learned my addr 172.21.15.116:0/812525989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb27fff640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 0x55d15c0a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 -- 172.21.15.116:0/812525989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 msgr2=0x55d15c0a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 0x55d15c0a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 -- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 msgr2=0x55d15c0a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb277fe640 1 -- 172.21.15.116:0/812525989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15be3af70 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb27fff640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.838+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 0x55d15c0a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb277fe640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c004540 tx=0x7fdb1c009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56b6427c54ee761 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c01f860 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdb1c01fa00 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2009ce50 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdb1c01fc00 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 10 client.21303 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d15bcd2170 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb2009ce50 con 0x55d15c09aca0 2024-03-19T02:47:55.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdb1c028410 con 0x55d15c09aca0 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 10 client.21303 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 10 client.21303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 20 client.21303 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c0b3b30 0x55d15c0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 4 client.21303 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d15c00b6a0 con 0x55d15c0b3b30 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.839+0000 7fdb27fff640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c0b3b30 0x55d15c0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.840+0000 7fdb27fff640 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c0b3b30 0x55d15c0b5f90 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857399143ac1d036 server_cookie=c5e13dac75ae74d4 in_seq=0 out_seq=0 2024-03-19T02:47:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.840+0000 7fdb077fe640 10 client.21303 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:55.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.841+0000 7fdb077fe640 1 -- 172.21.15.116:0/812525989 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d15c00b6a0 con 0x55d15c0b3b30 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.841+0000 7fdb077fe640 10 client.21303 handle_command_reply: tid=0 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.841+0000 7fdb311c8740 1 client.21303 shutdown 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 msgr2=0x55d15c0a0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c004540 tx=0x7fdb1c009ba0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 shutdown_connections 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c0b3b30 0x55d15c0b5f90 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c0a3880 0x55d15c0a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb077fe640 0 client.21303 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c09aca0 0x55d15c0a0ce0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 --2- 172.21.15.116:0/812525989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 >> 172.21.15.116:0/812525989 conn(0x55d15c094ab0 msgr2=0x55d15c096800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 shutdown_connections 2024-03-19T02:47:55.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 1 -- 172.21.15.116:0/812525989 wait complete. 2024-03-19T02:47:55.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.842+0000 7fdb311c8740 20 client.21303 trim_cache size 0 max 16384 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.867+0000 7fdb311c8740 1 Processor -- start 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 -- start start 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 0x55d15c0af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281aec70 con 0x55d15c0a3880 2024-03-19T02:47:55.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281b07b0 con 0x55d15c09a380 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb311c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281af340 con 0x55d15c09aca0 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 0x55d15c0af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47670/0 (socket says 172.21.15.116:47670) 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/4149274307 learned_addr learned my addr 172.21.15.116:0/4149274307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb27fff640 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/4149274307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 msgr2=0x55d15c0af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 0x55d15c0af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/4149274307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 msgr2=0x55d15c0a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb2ca9c640 1 -- 172.21.15.116:0/4149274307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15c0948e0 con 0x55d15c0a3880 2024-03-19T02:47:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.868+0000 7fdb277fe640 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 0x55d15c0af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.869+0000 7fdb2ca9c640 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fdb200cccc0 tx=0x7fdb200a42c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299da16c312b9b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.869+0000 7fdb0dffb640 1 -- 172.21.15.116:0/4149274307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2009c220 con 0x55d15c0a3880 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.869+0000 7fdb0dffb640 1 -- 172.21.15.116:0/4149274307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb2009ce50 con 0x55d15c0a3880 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.869+0000 7fdb0dffb640 1 -- 172.21.15.116:0/4149274307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2009b480 con 0x55d15c0a3880 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- 172.21.15.116:0/4149274307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 msgr2=0x55d15c0939b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fdb200cccc0 tx=0x7fdb200a42c0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- 172.21.15.116:0/4149274307 shutdown_connections 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c0939b0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09aca0 0x55d15c0af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 --2- 172.21.15.116:0/4149274307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09a380 0x55d15c0a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- 172.21.15.116:0/4149274307 >> 172.21.15.116:0/4149274307 conn(0x55d15c094ab0 msgr2=0x55d15c0932a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- 172.21.15.116:0/4149274307 shutdown_connections 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- 172.21.15.116:0/4149274307 wait complete. 2024-03-19T02:47:55.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 -- start start 2024-03-19T02:47:55.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.870+0000 7fdb311c8740 1 Processor -- start 2024-03-19T02:47:55.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c09db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c09e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281b00e0 con 0x55d15c0a3880 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281afa10 con 0x55d15c09aca0 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281acc70 con 0x55d15c09a380 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb311c8740 10 client.0 _ll_register_callbacks cb 0x7fdb1c004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb2ca9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56574/0 (socket says 172.21.15.116:56574) 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 -- 172.21.15.116:0/2849863191 learned_addr learned my addr 172.21.15.116:0/2849863191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c09db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 -- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 msgr2=0x55d15c09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c09db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 -- 172.21.15.116:0/2849863191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 msgr2=0x55d15c09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c09e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 -- 172.21.15.116:0/2849863191 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15c1190e0 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb27fff640 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdb1802c210 tx=0x7fdb18004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa3ba97ddcfe4b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb077fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.871+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1802cc40 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7fdb1802cde0 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2009ce50 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fdb18059050 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 10 client.31012 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d15c11a420 con 0x55d15c09a380 2024-03-19T02:47:55.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb2009ce50 con 0x55d15c09a380 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7fdb18004c90 con 0x55d15c09a380 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 10 client.31012 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 10 client.31012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 20 client.31012 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c11aba0 0x55d15c11cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 4 client.31012 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d15c0059f0 con 0x55d15c11aba0 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.872+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c11aba0 0x55d15c11cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.873+0000 7fdb077fe640 10 client.31012 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.873+0000 7fdb277fe640 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c11aba0 0x55d15c11cf90 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c17cf02add27977 server_cookie=50c7eef5a9140cec in_seq=0 out_seq=0 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb077fe640 1 -- 172.21.15.116:0/2849863191 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d15c0059f0 con 0x55d15c11aba0 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb077fe640 10 client.31012 handle_command_reply: tid=0 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 client.31012 shutdown 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 msgr2=0x55d15c09d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdb1802c210 tx=0x7fdb18004610 comp rx=0 tx=0).stop 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 shutdown_connections 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55d15c11aba0 0x55d15c11cf90 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d15c0a3880 0x55d15c09e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d15c09aca0 0x55d15c09db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 --2- 172.21.15.116:0/2849863191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c09a380 0x55d15c09d610 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 >> 172.21.15.116:0/2849863191 conn(0x55d15c094ab0 msgr2=0x55d15c093b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.877+0000 7fdb077fe640 0 client.31012 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 shutdown_connections 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb311c8740 1 -- 172.21.15.116:0/2849863191 wait complete. 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb311c8740 20 client.31012 trim_cache size 0 max 16384 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:47:55.882 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "00e66c1f-0582-4412-96e0-7dcc3c847c62", 2024-03-19T02:47:55.883 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:47:55.883 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 msgr2=0x7fdb100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 0x7fdb100a4e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c011de0 tx=0x7fdb1c009c70 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 msgr2=0x7fdb2819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.878+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000eb10 tx=0x7fdb2000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 shutdown_connections 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb28107810 0x7fdb2819c810 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb2806a5a0 0x7fdb281984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 --2- 172.21.15.116:0/2795825121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb28069c00 0x7fdb28197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 >> 172.21.15.116:0/2795825121 conn(0x7fdb280777b0 msgr2=0x7fdb280ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 shutdown_connections 2024-03-19T02:47:55.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:55.879+0000 7fdb2e4ff640 1 -- 172.21.15.116:0/2795825121 wait complete. 2024-03-19T02:47:55.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:47:55.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:47:56.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 -- 172.21.15.116:0/3335431285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101050 msgr2=0x7f113010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 --2- 172.21.15.116:0/3335431285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101050 0x7f113010d560 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f1128009cd0 tx=0x7f112802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 -- 172.21.15.116:0/3335431285 shutdown_connections 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 --2- 172.21.15.116:0/3335431285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101050 0x7f113010d560 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 --2- 172.21.15.116:0/3335431285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1130100690 0x7f1130100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 --2- 172.21.15.116:0/3335431285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11301066a0 0x7f1130106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 -- 172.21.15.116:0/3335431285 >> 172.21.15.116:0/3335431285 conn(0x7f1130077820 msgr2=0x7f1130077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 -- 172.21.15.116:0/3335431285 shutdown_connections 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.010+0000 7f1136ec6640 1 -- 172.21.15.116:0/3335431285 wait complete. 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 Processor -- start 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 -- start start 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 0x7f1130199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11301066a0 0x7f11301a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1130110960 con 0x7f1130100690 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11301107e0 con 0x7f1130101050 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1136ec6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1130110ae0 con 0x7f11301066a0 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47690/0 (socket says 172.21.15.116:47690) 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 -- 172.21.15.116:0/1539694496 learned_addr learned my addr 172.21.15.116:0/1539694496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1127fff640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 0x7f1130199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1135463640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11301066a0 0x7f11301a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 -- 172.21.15.116:0/1539694496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11301066a0 msgr2=0x7f11301a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11301066a0 0x7f11301a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 -- 172.21.15.116:0/1539694496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 msgr2=0x7f1130199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 0x7f1130199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 -- 172.21.15.116:0/1539694496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11301a43e0 con 0x7f1130100690 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1127fff640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 0x7f1130199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.011+0000 7f1134c62640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f112c00e9f0 tx=0x7f112c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.012+0000 7f1125ffb640 1 -- 172.21.15.116:0/1539694496 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c00cd80 con 0x7f1130100690 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.012+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11301a45b0 con 0x7f1130100690 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.012+0000 7f1125ffb640 1 -- 172.21.15.116:0/1539694496 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f112c0108e0 con 0x7f1130100690 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.012+0000 7f1125ffb640 1 -- 172.21.15.116:0/1539694496 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c018920 con 0x7f1130100690 2024-03-19T02:47:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.012+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11301a4b20 con 0x7f1130100690 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.014+0000 7f1125ffb640 1 -- 172.21.15.116:0/1539694496 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f112c010a90 con 0x7f1130100690 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.014+0000 7f1139b94740 1 Processor -- start 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 -- start start 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1125ffb640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 0x7f11180a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf2ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf38320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1135463640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 0x7f11180a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f8be90500 con 0x559f8bf35f30 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559f8be38be0 con 0x559f8bf2ca30 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1139b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559f8be8c5a0 con 0x559f8bf2d350 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1125ffb640 1 -- 172.21.15.116:0/1539694496 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f112c0cf450 con 0x7f1130100690 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56600/0 (socket says 172.21.15.116:56600) 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 -- 172.21.15.116:0/2733311887 learned_addr learned my addr 172.21.15.116:0/2733311887 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1134c62640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf2ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1135463640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf38320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 -- 172.21.15.116:0/2733311887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 msgr2=0x559f8bf2ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf2ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 -- 172.21.15.116:0/2733311887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 msgr2=0x559f8bf38320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf38320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1127fff640 1 -- 172.21.15.116:0/2733311887 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8be5a000 con 0x559f8bf2d350 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1134c62640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf2ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.015+0000 7f1135463640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf38320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1135463640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 0x7f11180a4eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1128005cc0 tx=0x7f1128009900 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1127fff640 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f11200114f0 tx=0x7f11200119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=601f045ca8344314 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1116ffd640 1 -- 172.21.15.116:0/2733311887 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120004020 con 0x559f8bf2d350 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1116ffd640 1 -- 172.21.15.116:0/2733311887 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11200041c0 con 0x559f8bf2d350 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1116ffd640 1 -- 172.21.15.116:0/2733311887 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112000f860 con 0x559f8bf2d350 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1139b94740 1 -- 172.21.15.116:0/2733311887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 msgr2=0x559f8bf359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.016+0000 7f1139b94740 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f11200114f0 tx=0x7f11200119b0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- 172.21.15.116:0/2733311887 shutdown_connections 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf38320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf359f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- 172.21.15.116:0/2733311887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf2ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- 172.21.15.116:0/2733311887 >> 172.21.15.116:0/2733311887 conn(0x559f8bf27160 msgr2=0x559f8bf27540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- 172.21.15.116:0/2733311887 shutdown_connections 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- 172.21.15.116:0/2733311887 wait complete. 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- start start 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 Processor -- start 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf333a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf338e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f8bf3d130 con 0x559f8bf2ca30 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559f8be4c160 con 0x559f8bf35f30 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.017+0000 7f1139b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559f8bf3d2b0 con 0x559f8bf2d350 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1139b94740 10 client.0 _ll_register_callbacks cb 0x559f8be88a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47706/0 (socket says 172.21.15.116:47706) 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 -- 172.21.15.116:0/3130332680 learned_addr learned my addr 172.21.15.116:0/3130332680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1127fff640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1135463640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 -- 172.21.15.116:0/3130332680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 msgr2=0x559f8bf333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 -- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 msgr2=0x559f8bf338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf338e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 -- 172.21.15.116:0/3130332680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8bc6e7d0 con 0x559f8bf2ca30 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1134c62640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f112c098150 tx=0x7f112c010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4a7e7ed64763a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f110ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c097df0 con 0x559f8bf2ca30 2024-03-19T02:47:56.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f112c0cf0c0 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11200041c0 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f112c097f90 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1139b94740 10 client.21318 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.018+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559f8bba43f0 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11200041c0 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f112c09ac60 con 0x559f8bf2ca30 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 10 client.21318 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 10 client.21318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 20 client.21318 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bf4c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 4 client.21318 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f8be9da40 con 0x559f8bf4a480 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.019+0000 7f1127fff640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bf4c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.020+0000 7f1127fff640 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bf4c890 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=153a3eeb71090a00 server_cookie=adfe37ed621d72d1 in_seq=0 out_seq=0 2024-03-19T02:47:56.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.020+0000 7f110ffff640 10 client.21318 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.023+0000 7f110ffff640 1 -- 172.21.15.116:0/3130332680 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f8be9da40 con 0x559f8bf4a480 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.023+0000 7f110ffff640 10 client.21318 handle_command_reply: tid=0 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.023+0000 7f1139b94740 1 client.21318 shutdown 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 msgr2=0x559f8bf32e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f112c098150 tx=0x7f112c010f20 comp rx=0 tx=0).stop 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 shutdown_connections 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bf4c890 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f110ffff640 0 client.21318 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 --2- 172.21.15.116:0/3130332680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf32e60 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 >> 172.21.15.116:0/3130332680 conn(0x559f8bf27160 msgr2=0x559f8bf28eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 shutdown_connections 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 1 -- 172.21.15.116:0/3130332680 wait complete. 2024-03-19T02:47:56.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.024+0000 7f1139b94740 20 client.21318 trim_cache size 0 max 16384 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 Processor -- start 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 -- start start 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf45ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf52130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11301acad0 con 0x559f8bf2ca30 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11301ae610 con 0x559f8bf35f30 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1139b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11301ad1a0 con 0x559f8bf2d350 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47718/0 (socket says 172.21.15.116:47718) 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 -- 172.21.15.116:0/1488181748 learned_addr learned my addr 172.21.15.116:0/1488181748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1127fff640 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf45ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1135463640 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf52130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 -- 172.21.15.116:0/1488181748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 msgr2=0x559f8bf45ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf45ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 -- 172.21.15.116:0/1488181748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 msgr2=0x559f8bf52130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf52130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 -- 172.21.15.116:0/1488181748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8bb640f0 con 0x559f8bf2ca30 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.047+0000 7f1134c62640 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f112c0a3940 tx=0x7f112c09b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a6c85e28c16baf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.048+0000 7f11167fc640 1 -- 172.21.15.116:0/1488181748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c097df0 con 0x559f8bf2ca30 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.048+0000 7f11167fc640 1 -- 172.21.15.116:0/1488181748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f112c097820 con 0x559f8bf2ca30 2024-03-19T02:47:56.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.048+0000 7f11167fc640 1 -- 172.21.15.116:0/1488181748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c097a20 con 0x559f8bf2ca30 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- 172.21.15.116:0/1488181748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 msgr2=0x559f8bf35590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f112c0a3940 tx=0x7f112c09b900 comp rx=0 tx=0).stop 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- 172.21.15.116:0/1488181748 shutdown_connections 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf35f30 0x559f8bf52130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf45ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- 172.21.15.116:0/1488181748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf2ca30 0x559f8bf35590 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- 172.21.15.116:0/1488181748 >> 172.21.15.116:0/1488181748 conn(0x559f8bf27160 msgr2=0x559f8bf32e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- 172.21.15.116:0/1488181748 shutdown_connections 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- 172.21.15.116:0/1488181748 wait complete. 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- start start 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 Processor -- start 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf3e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf3e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11301adf40 con 0x559f8bf35f30 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11301ad870 con 0x559f8bf2ca30 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.049+0000 7f1139b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11301aaad0 con 0x559f8bf2d350 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1139b94740 10 client.0 _ll_register_callbacks cb 0x7f112c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1134c62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf3e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1134c62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:51576/0 (socket says 172.21.15.116:51576) 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1134c62640 1 -- 172.21.15.116:0/771888433 learned_addr learned my addr 172.21.15.116:0/771888433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1127fff640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf3e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 -- 172.21.15.116:0/771888433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 msgr2=0x559f8bf3e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf3e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 -- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 msgr2=0x559f8bf3df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 -- 172.21.15.116:0/771888433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8bfaf8c0 con 0x559f8bf35f30 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1134c62640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1127fff640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf3e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f1135463640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf3e9c0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f112802c630 tx=0x7f112802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7e50f73df37698 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f110ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:47:56.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.050+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128052020 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9510+0+0 (secure 0 0 0) 0x7f1128003ab0 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f112c097820 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f112805d050 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 10 client.21324 fetch_fsmap learned FSMap version 38 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559f8bfb0bb0 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f112c097820 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8362+0+0 (secure 0 0 0) 0x7f112803f040 con 0x559f8bf35f30 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 10 client.21324 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 10 client.21324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 20 client.21324 populate_metadata read hostname 'smithi116' 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bfb3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 4 client.21324 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f8be97d90 con 0x559f8bf4a480 2024-03-19T02:47:56.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.051+0000 7f1134c62640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bfb3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:56.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.052+0000 7f110ffff640 10 client.21324 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:56.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.052+0000 7f1134c62640 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bfb3390 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf2839fb775ae11 server_cookie=5b2e8411290885b0 in_seq=0 out_seq=0 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f110ffff640 1 -- 172.21.15.116:0/771888433 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x559f8be97d90 con 0x559f8bf4a480 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f110ffff640 10 client.21324 handle_command_reply: tid=0 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 client.21324 shutdown 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 msgr2=0x559f8bf3e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf3e9c0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f112802c630 tx=0x7f112802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 shutdown_connections 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x559f8bf4a480 0x559f8bfb3390 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f8bf35f30 0x559f8bf3e9c0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559f8bf2d350 0x559f8bf3e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 --2- 172.21.15.116:0/771888433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f8bf2ca30 0x559f8bf3df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f110ffff640 0 client.21324 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 >> 172.21.15.116:0/771888433 conn(0x559f8bf27160 msgr2=0x559f8bf336b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 shutdown_connections 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 1 -- 172.21.15.116:0/771888433 wait complete. 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.054+0000 7f1139b94740 20 client.21324 trim_cache size 0 max 16384 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (553 inodes in the stack)", 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:47:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "00e66c1f-0582-4412-96e0-7dcc3c847c62": { 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "00e66c1f-0582-4412-96e0-7dcc3c847c62", 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:47:56.061 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 msgr2=0x7f11180a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 0x7f11180a4eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1128005cc0 tx=0x7f1128009900 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 msgr2=0x7f1130198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f112c00e9f0 tx=0x7f112c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 shutdown_connections 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f11180a29f0 0x7f11180a4eb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11301066a0 0x7f11301a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1130101050 0x7f1130199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 --2- 172.21.15.116:0/1539694496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130100690 0x7f1130198e20 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 >> 172.21.15.116:0/1539694496 conn(0x7f1130077820 msgr2=0x7f1130102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 shutdown_connections 2024-03-19T02:47:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:47:56.055+0000 7f1136ec6640 1 -- 172.21.15.116:0/1539694496 wait complete. 2024-03-19T02:47:56.069 INFO:tasks.cephfs.filesystem:scrub status for tag:00e66c1f-0582-4412-96e0-7dcc3c847c62 - {'path': '/', 'tag': '00e66c1f-0582-4412-96e0-7dcc3c847c62', 'options': 'recursive,force'} 2024-03-19T02:47:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:55 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:56.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:55 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:55 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:56.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:55 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:56.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49277 6.01 MB/sec execute 318 sec latency 231.353 ms 2024-03-19T02:47:56.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:55 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:55 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: pgmap v634: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 180 op/s 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:56 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: pgmap v634: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 180 op/s 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:56 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.332 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49369 6.01 MB/sec execute 319 sec latency 194.278 ms 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: pgmap v634: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 180 op/s 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:47:57.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:57.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:56 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:47:58.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49445 5.99 MB/sec execute 320 sec latency 324.958 ms 2024-03-19T02:47:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:47:58 smithi154 ceph-mon[35693]: pgmap v635: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:47:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:47:58 smithi116 ceph-mon[32263]: pgmap v635: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:47:59.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49553 5.97 MB/sec execute 321 sec latency 194.904 ms 2024-03-19T02:47:59.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:47:58 smithi162 ceph-mon[35761]: pgmap v635: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:48:00.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49639 5.96 MB/sec execute 322 sec latency 330.570 ms 2024-03-19T02:48:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:00 smithi154 ceph-mon[35693]: pgmap v636: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:48:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:00 smithi116 ceph-mon[32263]: pgmap v636: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:48:01.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49738 5.95 MB/sec execute 323 sec latency 176.425 ms 2024-03-19T02:48:01.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:00 smithi162 ceph-mon[35761]: pgmap v636: 129 pgs: 129 active+clean; 1.0 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 119 op/s 2024-03-19T02:48:02.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49886 5.96 MB/sec execute 324 sec latency 168.796 ms 2024-03-19T02:48:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:02 smithi154 ceph-mon[35693]: pgmap v637: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 298 op/s 2024-03-19T02:48:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:02 smithi116 ceph-mon[32263]: pgmap v637: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 298 op/s 2024-03-19T02:48:03.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 49978 5.96 MB/sec execute 325 sec latency 157.658 ms 2024-03-19T02:48:03.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:02 smithi162 ceph-mon[35761]: pgmap v637: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.1 MiB/s wr, 298 op/s 2024-03-19T02:48:04.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50095 5.96 MB/sec execute 326 sec latency 156.600 ms 2024-03-19T02:48:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:04 smithi154 ceph-mon[35693]: pgmap v638: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-03-19T02:48:05.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:04 smithi116 ceph-mon[32263]: pgmap v638: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-03-19T02:48:05.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50180 5.97 MB/sec execute 327 sec latency 238.776 ms 2024-03-19T02:48:05.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:04 smithi162 ceph-mon[35761]: pgmap v638: 129 pgs: 129 active+clean; 1.0 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-03-19T02:48:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:06 smithi154 ceph-mon[35693]: osdmap e124: 12 total, 12 up, 12 in 2024-03-19T02:48:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:06 smithi154 ceph-mon[35693]: pgmap v640: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:06 smithi116 ceph-mon[32263]: osdmap e124: 12 total, 12 up, 12 in 2024-03-19T02:48:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:06 smithi116 ceph-mon[32263]: pgmap v640: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:06.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50278 5.96 MB/sec execute 328 sec latency 143.283 ms 2024-03-19T02:48:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:06 smithi162 ceph-mon[35761]: osdmap e124: 12 total, 12 up, 12 in 2024-03-19T02:48:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:06 smithi162 ceph-mon[35761]: pgmap v640: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:07.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50399 5.96 MB/sec execute 329 sec latency 158.021 ms 2024-03-19T02:48:08.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50513 5.97 MB/sec execute 330 sec latency 190.054 ms 2024-03-19T02:48:08.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:08 smithi162 ceph-mon[35761]: pgmap v641: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:08 smithi154 ceph-mon[35693]: pgmap v641: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:08 smithi116 ceph-mon[32263]: pgmap v641: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:09.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50624 5.98 MB/sec execute 331 sec latency 167.121 ms 2024-03-19T02:48:10.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50728 5.97 MB/sec execute 332 sec latency 242.066 ms 2024-03-19T02:48:10.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:10 smithi162 ceph-mon[35761]: pgmap v642: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:10.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:10.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:10 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:10 smithi154 ceph-mon[35693]: pgmap v642: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:10 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:10 smithi116 ceph-mon[32263]: pgmap v642: 129 pgs: 129 active+clean; 972 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.9 MiB/s wr, 393 op/s 2024-03-19T02:48:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:10 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:11.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50839 5.96 MB/sec execute 333 sec latency 237.239 ms 2024-03-19T02:48:11.409 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:11.409 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:11.410 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:11 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:11 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:11 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:12.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 50964 5.95 MB/sec execute 334 sec latency 138.238 ms 2024-03-19T02:48:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:12 smithi162 ceph-mon[35761]: pgmap v643: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-03-19T02:48:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:12 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:12 smithi162 ceph-mon[35761]: osdmap e125: 12 total, 12 up, 12 in 2024-03-19T02:48:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:12 smithi154 ceph-mon[35693]: pgmap v643: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-03-19T02:48:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:12 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:12 smithi154 ceph-mon[35693]: osdmap e125: 12 total, 12 up, 12 in 2024-03-19T02:48:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:12 smithi116 ceph-mon[32263]: pgmap v643: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-03-19T02:48:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:12 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:12.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:12 smithi116 ceph-mon[32263]: osdmap e125: 12 total, 12 up, 12 in 2024-03-19T02:48:13.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51113 5.95 MB/sec execute 335 sec latency 134.372 ms 2024-03-19T02:48:14.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51247 5.95 MB/sec execute 336 sec latency 137.163 ms 2024-03-19T02:48:14.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:14 smithi162 ceph-mon[35761]: pgmap v645: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-19T02:48:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:14 smithi154 ceph-mon[35693]: pgmap v645: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-19T02:48:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:14 smithi116 ceph-mon[32263]: pgmap v645: 129 pgs: 129 active+clean; 971 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-19T02:48:15.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51341 5.94 MB/sec execute 337 sec latency 185.527 ms 2024-03-19T02:48:16.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51434 5.93 MB/sec execute 338 sec latency 244.781 ms 2024-03-19T02:48:16.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:16 smithi162 ceph-mon[35761]: pgmap v646: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:16.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:16 smithi154 ceph-mon[35693]: pgmap v646: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:16.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:16 smithi116 ceph-mon[32263]: pgmap v646: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:17.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51509 5.92 MB/sec execute 339 sec latency 1037.106 ms 2024-03-19T02:48:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:17 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:17.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:17 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:17.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:17 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:48:18.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51594 5.92 MB/sec execute 340 sec latency 2037.132 ms 2024-03-19T02:48:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:18 smithi162 ceph-mon[35761]: pgmap v647: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:18.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:18 smithi154 ceph-mon[35693]: pgmap v647: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:18.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:18 smithi116 ceph-mon[32263]: pgmap v647: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:19.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51678 5.92 MB/sec execute 341 sec latency 3037.158 ms 2024-03-19T02:48:20.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51787 5.93 MB/sec execute 342 sec latency 4037.186 ms 2024-03-19T02:48:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:20 smithi162 ceph-mon[35761]: pgmap v648: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:20 smithi154 ceph-mon[35693]: pgmap v648: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:20 smithi116 ceph-mon[32263]: pgmap v648: 129 pgs: 129 active+clean; 981 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 179 op/s 2024-03-19T02:48:21.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 51904 5.95 MB/sec execute 343 sec latency 4558.100 ms 2024-03-19T02:48:22.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52048 5.96 MB/sec execute 344 sec latency 151.816 ms 2024-03-19T02:48:22.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:22 smithi162 ceph-mon[35761]: pgmap v649: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-03-19T02:48:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:22 smithi154 ceph-mon[35693]: pgmap v649: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-03-19T02:48:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:22 smithi116 ceph-mon[32263]: pgmap v649: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 162 op/s 2024-03-19T02:48:23.333 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52166 5.96 MB/sec execute 345 sec latency 159.591 ms 2024-03-19T02:48:24.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52332 5.98 MB/sec execute 346 sec latency 147.707 ms 2024-03-19T02:48:24.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:24 smithi162 ceph-mon[35761]: pgmap v650: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-19T02:48:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:24 smithi154 ceph-mon[35693]: pgmap v650: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-19T02:48:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:24 smithi116 ceph-mon[32263]: pgmap v650: 129 pgs: 129 active+clean; 991 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-19T02:48:25.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52455 5.99 MB/sec execute 347 sec latency 171.156 ms 2024-03-19T02:48:26.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:48:26.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 -- 172.21.15.116:0/1405795194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 msgr2=0x7f3068100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 --2- 172.21.15.116:0/1405795194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068100b40 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f305c009900 tx=0x7f305c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 -- 172.21.15.116:0/1405795194 shutdown_connections 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 --2- 172.21.15.116:0/1405795194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f306810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 --2- 172.21.15.116:0/1405795194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068100b40 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 --2- 172.21.15.116:0/1405795194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 0x7f30681059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 -- 172.21.15.116:0/1405795194 >> 172.21.15.116:0/1405795194 conn(0x7f30680777b0 msgr2=0x7f3068077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.208+0000 7f306e9e1640 1 -- 172.21.15.116:0/1405795194 shutdown_connections 2024-03-19T02:48:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.209+0000 7f306e9e1640 1 -- 172.21.15.116:0/1405795194 wait complete. 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.209+0000 7f306e9e1640 1 Processor -- start 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 -- start start 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f3068197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 0x7f3068197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f3067fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f30677fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f3068197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f3067fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33188/0 (socket says 172.21.15.116:33188) 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f3067fff640 1 -- 172.21.15.116:0/2741011139 learned_addr learned my addr 172.21.15.116:0/2741011139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3068110960 con 0x7f30681006e0 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 0x7f3068197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30681107e0 con 0x7f30681055c0 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.210+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3068110ae0 con 0x7f3068101080 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 -- 172.21.15.116:0/2741011139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 msgr2=0x7f3068197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 0x7f3068197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 -- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 msgr2=0x7f3068196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30681a6480 con 0x7f3068101080 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30677fe640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f3068197110 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c210 tx=0x7f305c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.211+0000 7f30657fa640 1 -- 172.21.15.116:0/2741011139 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c042020 con 0x7f3068101080 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.212+0000 7f30657fa640 1 -- 172.21.15.116:0/2741011139 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f305c004e00 con 0x7f3068101080 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.212+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30681a6710 con 0x7f3068101080 2024-03-19T02:48:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.212+0000 7f30657fa640 1 -- 172.21.15.116:0/2741011139 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c003970 con 0x7f3068101080 2024-03-19T02:48:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.212+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30681a6cb0 con 0x7f3068101080 2024-03-19T02:48:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30657fa640 1 -- 172.21.15.116:0/2741011139 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f305c045020 con 0x7f3068101080 2024-03-19T02:48:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 Processor -- start 2024-03-19T02:48:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 -- start start 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c07e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c08350 0x563ad1c109f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ad1b6b500 con 0x563ad1c08350 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ad1b13be0 con 0x563ad1c07a30 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30715c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ad1b675a0 con 0x563ad1c10f30 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30677fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30657fa640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 0x7f30500a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30677fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45018/0 (socket says 172.21.15.116:45018) 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f30677fe640 1 -- 172.21.15.116:0/2941452419 learned_addr learned my addr 172.21.15.116:0/2941452419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.215+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c07e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f3067fff640 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c08350 0x563ad1c109f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30657fa640 1 -- 172.21.15.116:0/2741011139 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f305c0f0c30 con 0x7f3068101080 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 0x7f30500a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 -- 172.21.15.116:0/2941452419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 msgr2=0x563ad1c07e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c07e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 -- 172.21.15.116:0/2941452419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c08350 msgr2=0x563ad1c109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c08350 0x563ad1c109f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 -- 172.21.15.116:0/2941452419 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ad1b35000 con 0x563ad1c10f30 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 0x7f30500a4f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3060005fd0 tx=0x7f3060009bb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.216+0000 7f30677fe640 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f305c005820 tx=0x7f305c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20cff392e5c104cd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f304e7fc640 1 -- 172.21.15.116:0/2941452419 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c0d10d0 con 0x563ad1c10f30 2024-03-19T02:48:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f304e7fc640 1 -- 172.21.15.116:0/2941452419 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f305c0bd660 con 0x563ad1c10f30 2024-03-19T02:48:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f304e7fc640 1 -- 172.21.15.116:0/2941452419 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c003b10 con 0x563ad1c10f30 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 -- 172.21.15.116:0/2941452419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 msgr2=0x563ad1c13320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f305c005820 tx=0x7f305c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 -- 172.21.15.116:0/2941452419 shutdown_connections 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c13320 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c08350 0x563ad1c109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 --2- 172.21.15.116:0/2941452419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c07e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 -- 172.21.15.116:0/2941452419 >> 172.21.15.116:0/2941452419 conn(0x563ad1c02160 msgr2=0x563ad1c02540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.217+0000 7f30715c5740 1 -- 172.21.15.116:0/2941452419 shutdown_connections 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.218+0000 7f30715c5740 1 -- 172.21.15.116:0/2941452419 wait complete. 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.218+0000 7f30715c5740 1 -- start start 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.218+0000 7f30715c5740 1 Processor -- start 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.218+0000 7f30715c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c0de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c08350 0x563ad1c0e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ad1c13ae0 con 0x563ad1c10f30 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ad1b27160 con 0x563ad1c07a30 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ad1c13c60 con 0x563ad1c08350 2024-03-19T02:48:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30715c5740 10 client.0 _ll_register_callbacks cb 0x563ad1b63a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f3067fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c0de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33214/0 (socket says 172.21.15.116:33214) 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 -- 172.21.15.116:0/2724392747 learned_addr learned my addr 172.21.15.116:0/2724392747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 -- 172.21.15.116:0/2724392747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c08350 msgr2=0x563ad1c0e390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c08350 0x563ad1c0e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 -- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 msgr2=0x563ad1c0de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c0de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 -- 172.21.15.116:0/2724392747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ad19497d0 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3060013450 tx=0x7f306001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f9e001f9de7af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.219+0000 7f30477fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060010dc0 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f3060004b20 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f305c0bd660 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3060032050 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30715c5740 10 client.21336 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563ad187f3f0 con 0x563ad1c10f30 2024-03-19T02:48:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3060004d20 con 0x563ad1c10f30 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f306001f420 con 0x563ad1c10f30 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30715c5740 10 client.21336 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.220+0000 7f30715c5740 10 client.21336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f30715c5740 20 client.21336 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c21300 0x563ad1c23780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f30715c5740 4 client.21336 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f305c0bd660 con 0x563ad1c21300 2024-03-19T02:48:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f3067fff640 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c21300 0x563ad1c23780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f30477fe640 10 client.21336 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.221+0000 7f3067fff640 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c21300 0x563ad1c23780 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92b52d111e711bf1 server_cookie=4a0149360cd71e1e in_seq=0 out_seq=0 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.223+0000 7f30477fe640 1 -- 172.21.15.116:0/2724392747 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f305c0bd660 con 0x563ad1c21300 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.223+0000 7f30477fe640 10 client.21336 handle_command_reply: tid=0 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 client.21336 shutdown 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 msgr2=0x563ad1c0e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3060013450 tx=0x7f306001e000 comp rx=0 tx=0).stop 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 shutdown_connections 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c21300 0x563ad1c23780 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c0e8d0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c08350 0x563ad1c0e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 --2- 172.21.15.116:0/2724392747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c07a30 0x563ad1c0de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 >> 172.21.15.116:0/2724392747 conn(0x563ad1c02160 msgr2=0x563ad1c03eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30477fe640 0 client.21336 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 shutdown_connections 2024-03-19T02:48:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 1 -- 172.21.15.116:0/2724392747 wait complete. 2024-03-19T02:48:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.224+0000 7f30715c5740 20 client.21336 trim_cache size 0 max 16384 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 Processor -- start 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 -- start start 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c07a30 0x563ad1c10580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c29090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30681b0ed0 con 0x563ad1c10f30 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30681b2a70 con 0x563ad1c08350 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30715c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30681b15a0 con 0x563ad1c07a30 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30677fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f306cf7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c29090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30677fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:39990/0 (socket says 172.21.15.116:39990) 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f3067fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c07a30 0x563ad1c10580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.247+0000 7f30677fe640 1 -- 172.21.15.116:0/2251972142 learned_addr learned my addr 172.21.15.116:0/2251972142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 -- 172.21.15.116:0/2251972142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c07a30 msgr2=0x563ad1c10580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c07a30 0x563ad1c10580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 -- 172.21.15.116:0/2251972142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 msgr2=0x563ad1c29090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c29090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 -- 172.21.15.116:0/2251972142 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ad183f0f0 con 0x563ad1c08350 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f306cf7e640 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c29090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f30677fe640 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c580 tx=0x7f305c009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d686255227e36de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f304dffb640 1 -- 172.21.15.116:0/2251972142 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c0bd660 con 0x563ad1c08350 2024-03-19T02:48:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f304dffb640 1 -- 172.21.15.116:0/2251972142 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f305c0c10b0 con 0x563ad1c08350 2024-03-19T02:48:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.248+0000 7f304dffb640 1 -- 172.21.15.116:0/2251972142 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c003c90 con 0x563ad1c08350 2024-03-19T02:48:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 -- 172.21.15.116:0/2251972142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 msgr2=0x563ad1c03ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c580 tx=0x7f305c009290 comp rx=0 tx=0).stop 2024-03-19T02:48:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 -- 172.21.15.116:0/2251972142 shutdown_connections 2024-03-19T02:48:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c10f30 0x563ad1c29090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c03ca0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 --2- 172.21.15.116:0/2251972142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c07a30 0x563ad1c10580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 -- 172.21.15.116:0/2251972142 >> 172.21.15.116:0/2251972142 conn(0x563ad1c02160 msgr2=0x563ad1c03590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 -- 172.21.15.116:0/2251972142 shutdown_connections 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.249+0000 7f30715c5740 1 -- 172.21.15.116:0/2251972142 wait complete. 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 -- start start 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 Processor -- start 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c0ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c0b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30681b23a0 con 0x563ad1c07a30 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30681b1c70 con 0x563ad1c08350 2024-03-19T02:48:26.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30681aeed0 con 0x563ad1c10f30 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30715c5740 10 client.0 _ll_register_callbacks cb 0x7f3060013cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f3067fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f30677fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c0ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f3067fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33240/0 (socket says 172.21.15.116:33240) 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.250+0000 7f3067fff640 1 -- 172.21.15.116:0/1328198396 learned_addr learned my addr 172.21.15.116:0/1328198396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f306cf7e640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c0b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 -- 172.21.15.116:0/1328198396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 msgr2=0x563ad1c0b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c0b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 -- 172.21.15.116:0/1328198396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 msgr2=0x563ad1c0a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 -- 172.21.15.116:0/1328198396 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ad1c866c0 con 0x563ad1c08350 2024-03-19T02:48:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f3067fff640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30677fe640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c0ae40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x563ad1c07730 tx=0x7f305c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59f8475a0e8388c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30477fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c0bcf60 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f305c005320 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.251+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ad1b78a40 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f305c0c5700 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 10 client.31070 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563ad1c87930 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563ad1b78a40 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f305c0bc540 con 0x563ad1c08350 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 10 client.31070 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 10 client.31070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 20 client.31070 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c87cd0 0x563ad1c8a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 4 client.31070 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ad1b72d90 con 0x563ad1c87cd0 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.252+0000 7f3067fff640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c87cd0 0x563ad1c8a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.253+0000 7f30477fe640 10 client.31070 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.253+0000 7f3067fff640 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c87cd0 0x563ad1c8a0c0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2dd316fe44065a1 server_cookie=b23a3ec752fb4f1 in_seq=0 out_seq=0 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30477fe640 1 -- 172.21.15.116:0/1328198396 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ad1b72d90 con 0x563ad1c87cd0 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30477fe640 10 client.31070 handle_command_reply: tid=0 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 client.31070 shutdown 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 msgr2=0x563ad1c0ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c0ae40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x563ad1c07730 tx=0x7f305c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 shutdown_connections 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563ad1c87cd0 0x563ad1c8a0c0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ad1c10f30 0x563ad1c0b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ad1c08350 0x563ad1c0ae40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 --2- 172.21.15.116:0/1328198396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ad1c07a30 0x563ad1c0a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30477fe640 0 client.31070 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 >> 172.21.15.116:0/1328198396 conn(0x563ad1c02160 msgr2=0x563ad1c03e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.254+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 shutdown_connections 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f30715c5740 1 -- 172.21.15.116:0/1328198396 wait complete. 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f30715c5740 20 client.31070 trim_cache size 0 max 16384 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:48:26.261 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:48:26.262 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:48:26.262 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:48:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 msgr2=0x7f30500a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 0x7f30500a4f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3060005fd0 tx=0x7f3060009bb0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 msgr2=0x7f3068197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.255+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f3068197110 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c210 tx=0x7f305c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 shutdown_connections 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f30500a2a70 0x7f30500a4f30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30681055c0 0x7f3068197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3068101080 0x7f3068197110 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 --2- 172.21.15.116:0/2741011139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30681006e0 0x7f3068196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 >> 172.21.15.116:0/2741011139 conn(0x7f30680777b0 msgr2=0x7f30680fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 shutdown_connections 2024-03-19T02:48:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.256+0000 7f306e9e1640 1 -- 172.21.15.116:0/2741011139 wait complete. 2024-03-19T02:48:26.272 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:48:26.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:48:26.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52558 5.99 MB/sec execute 348 sec latency 160.120 ms 2024-03-19T02:48:26.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 -- 172.21.15.116:0/4000423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 msgr2=0x7f62d810e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/4000423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d810e300 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009be0 tx=0x7f62d002bef0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 -- 172.21.15.116:0/4000423500 shutdown_connections 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/4000423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d810e300 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/4000423500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d81033b0 0x7f62d810b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/4000423500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d8102a90 0x7f62d8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 -- 172.21.15.116:0/4000423500 >> 172.21.15.116:0/4000423500 conn(0x7f62d8077820 msgr2=0x7f62d8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 -- 172.21.15.116:0/4000423500 shutdown_connections 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.401+0000 7f62dcfb9640 1 -- 172.21.15.116:0/4000423500 wait complete. 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 Processor -- start 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 -- start start 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 0x7f62d8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8111b10 con 0x7f62d8102a90 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8111990 con 0x7f62d810be60 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62dcfb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62d8111c90 con 0x7f62d81033b0 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 0x7f62d8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45054/0 (socket says 172.21.15.116:45054) 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62cffff640 1 -- 172.21.15.116:0/740549549 learned_addr learned my addr 172.21.15.116:0/740549549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d6d9d640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 -- 172.21.15.116:0/740549549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 msgr2=0x7f62d81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 -- 172.21.15.116:0/740549549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 msgr2=0x7f62d819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62d659c640 1 -- 172.21.15.116:0/740549549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d819cfa0 con 0x7f62d8102a90 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.402+0000 7f62cffff640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62d659c640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 0x7f62d8197fb0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005fd0 tx=0x7f62c800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c800be60 con 0x7f62d8102a90 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62c80035f0 con 0x7f62d8102a90 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d819d290 con 0x7f62d8102a90 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8019670 con 0x7f62d8102a90 2024-03-19T02:48:26.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.403+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d8100610 con 0x7f62d8102a90 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.404+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d8068610 con 0x7f62d8102a90 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.405+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f62c8004000 con 0x7f62d8102a90 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.406+0000 7f62cf7fe640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 0x7f62c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.406+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f62c80d2a90 con 0x7f62d8102a90 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.406+0000 7f62cffff640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 0x7f62c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.406+0000 7f62cffff640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 0x7f62c40a4e30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f62c00095f0 tx=0x7f62c0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.411+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62c809edc0 con 0x7f62d8102a90 2024-03-19T02:48:26.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:26 smithi162 ceph-mon[35761]: pgmap v651: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 196 op/s 2024-03-19T02:48:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:26 smithi154 ceph-mon[35693]: pgmap v651: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 196 op/s 2024-03-19T02:48:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:26 smithi116 ceph-mon[32263]: pgmap v651: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 196 op/s 2024-03-19T02:48:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.754+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d8106f50 con 0x7f62d8102a90 2024-03-19T02:48:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.755+0000 7f62cf7fe640 1 -- 172.21.15.116:0/740549549 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11953 (secure 0 0 0) 0x7f62c80a2e60 con 0x7f62d8102a90 2024-03-19T02:48:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:48:26.756 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:48:15.836587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:48:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2024-03-19T02:48:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 msgr2=0x7f62c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 0x7f62c40a4e30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f62c00095f0 tx=0x7f62c0005f90 comp rx=0 tx=0).stop 2024-03-19T02:48:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 msgr2=0x7f62d8197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 0x7f62d8197fb0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005fd0 tx=0x7f62c800b690 comp rx=0 tx=0).stop 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 shutdown_connections 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f62c40a2970 0x7f62c40a4e30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62d810be60 0x7f62d819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62d81033b0 0x7f62d81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 --2- 172.21.15.116:0/740549549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62d8102a90 0x7f62d8197fb0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.758+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 >> 172.21.15.116:0/740549549 conn(0x7f62d8077820 msgr2=0x7f62d80fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.759+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 shutdown_connections 2024-03-19T02:48:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.759+0000 7f62dcfb9640 1 -- 172.21.15.116:0/740549549 wait complete. 2024-03-19T02:48:26.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- 172.21.15.116:0/984512602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 msgr2=0x7f050410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 --2- 172.21.15.116:0/984512602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f050410cd20 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009ca0 tx=0x7f04fc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- 172.21.15.116:0/984512602 shutdown_connections 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 --2- 172.21.15.116:0/984512602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f050410cd20 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 --2- 172.21.15.116:0/984512602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0504100a30 0x7f0504104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 --2- 172.21.15.116:0/984512602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 0x7f0504100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- 172.21.15.116:0/984512602 >> 172.21.15.116:0/984512602 conn(0x7f0504076c50 msgr2=0x7f05040fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- 172.21.15.116:0/984512602 shutdown_connections 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- 172.21.15.116:0/984512602 wait complete. 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 Processor -- start 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.910+0000 7f050b019640 1 -- start start 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 0x7f0504197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f050419c850 0x7f050419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f830 con 0x7f0504105150 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f6b0 con 0x7f050419c850 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f050b019640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f050410f9b0 con 0x7f0504100080 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33266/0 (socket says 172.21.15.116:33266) 2024-03-19T02:48:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 -- 172.21.15.116:0/509270236 learned_addr learned my addr 172.21.15.116:0/509270236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0508db5640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 0x7f0504197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f05095b6640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f050419c850 0x7f050419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 -- 172.21.15.116:0/509270236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 msgr2=0x7f0504197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 0x7f0504197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 -- 172.21.15.116:0/509270236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f050419c850 msgr2=0x7f050419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f050419c850 0x7f050419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 -- 172.21.15.116:0/509270236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050419d340 con 0x7f0504105150 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0503fff640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f04f000e9f0 tx=0x7f04f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.911+0000 7f0501ffb640 1 -- 172.21.15.116:0/509270236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f000cce0 con 0x7f0504105150 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.912+0000 7f0501ffb640 1 -- 172.21.15.116:0/509270236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04f00054d0 con 0x7f0504105150 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.912+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050419d630 con 0x7f0504105150 2024-03-19T02:48:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.912+0000 7f0501ffb640 1 -- 172.21.15.116:0/509270236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0016410 con 0x7f0504105150 2024-03-19T02:48:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.912+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05041a4e20 con 0x7f0504105150 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.914+0000 7f0501ffb640 1 -- 172.21.15.116:0/509270236 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f04f00046c0 con 0x7f0504105150 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.914+0000 7f050dc06740 1 Processor -- start 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.914+0000 7f050dc06740 1 -- start start 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0501ffb640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 0x7f04f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae53290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae55bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa7adad4b0 con 0x55aa7ae537d0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aa7ad55d70 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f050dc06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aa7ada9550 con 0x55aa7ae4abf0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f05095b6640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 0x7f04f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0501ffb640 1 -- 172.21.15.116:0/509270236 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f04f00cef40 con 0x7f0504105150 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40036/0 (socket says 172.21.15.116:40036) 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 -- 172.21.15.116:0/2769055220 learned_addr learned my addr 172.21.15.116:0/2769055220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f05095b6640 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae55bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0503fff640 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae53290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 -- 172.21.15.116:0/2769055220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4abf0 msgr2=0x55aa7ae53290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae53290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 -- 172.21.15.116:0/2769055220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 msgr2=0x55aa7ae55bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae55bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 -- 172.21.15.116:0/2769055220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7ad78210 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f05095b6640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 0x7f04f40a4e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f04fc004af0 tx=0x7f04fc032e20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.915+0000 7f0508db5640 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009cf0 tx=0x7f04f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0645cfa189a3483 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f04eeffd640 1 -- 172.21.15.116:0/2769055220 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8004230 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f04eeffd640 1 -- 172.21.15.116:0/2769055220 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04f80043d0 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f04eeffd640 1 -- 172.21.15.116:0/2769055220 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f80197c0 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 -- 172.21.15.116:0/2769055220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 msgr2=0x55aa7ae4a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009cf0 tx=0x7f04f8011810 comp rx=0 tx=0).stop 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 -- 172.21.15.116:0/2769055220 shutdown_connections 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae55bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae53290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 --2- 172.21.15.116:0/2769055220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae4a6b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.916+0000 7f050dc06740 1 -- 172.21.15.116:0/2769055220 >> 172.21.15.116:0/2769055220 conn(0x55aa7ae44a00 msgr2=0x55aa7ae44de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.917+0000 7f050dc06740 1 -- 172.21.15.116:0/2769055220 shutdown_connections 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.917+0000 7f050dc06740 1 -- 172.21.15.116:0/2769055220 wait complete. 2024-03-19T02:48:26.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.917+0000 7f050dc06740 1 -- start start 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.917+0000 7f050dc06740 1 Processor -- start 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae50c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa7ae565c0 con 0x55aa7ae537d0 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aa7ad69310 con 0x55aa7ae4abf0 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aa7ae56740 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f050dc06740 10 client.0 _ll_register_callbacks cb 0x55aa7ad5d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33298/0 (socket says 172.21.15.116:33298) 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 -- 172.21.15.116:0/3043247718 learned_addr learned my addr 172.21.15.116:0/3043247718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f0508db5640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f0503fff640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae50c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 -- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 msgr2=0x55aa7ae50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae50730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 -- 172.21.15.116:0/3043247718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4abf0 msgr2=0x55aa7ae50c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae50c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 -- 172.21.15.116:0/3043247718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7aac13f0 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f0508db5640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f05095b6640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0328e0 tx=0x7f04fc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e086bc69d966d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f04e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.918+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc052060 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f04fc004250 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f80043d0 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f04fc059050 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 10 client.21360 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55aa7ab8b7d0 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04f80043d0 con 0x55aa7ae537d0 2024-03-19T02:48:26.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f04fc03f070 con 0x55aa7ae537d0 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 10 client.21360 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 10 client.21360 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 20 client.21360 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7ae63b50 0x55aa7ae65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 4 client.21360 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.919+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa7ae0b3e0 con 0x55aa7ae63b50 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.920+0000 7f0508db5640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7ae63b50 0x55aa7ae65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.920+0000 7f04e3fff640 10 client.21360 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:26.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.920+0000 7f0508db5640 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7ae63b50 0x55aa7ae65fb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=946c514654ec098 server_cookie=d0f105e1bbb17e7e in_seq=0 out_seq=0 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f04e3fff640 1 -- 172.21.15.116:0/3043247718 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa7ae0b3e0 con 0x55aa7ae63b50 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f04e3fff640 10 client.21360 handle_command_reply: tid=0 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 client.21360 shutdown 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 msgr2=0x55aa7ae511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0328e0 tx=0x7f04fc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 shutdown_connections 2024-03-19T02:48:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7ae63b50 0x55aa7ae65fb0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae537d0 0x55aa7ae511b0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae50c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f04e3fff640 0 client.21360 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 --2- 172.21.15.116:0/3043247718 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.921+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 >> 172.21.15.116:0/3043247718 conn(0x55aa7ae44a00 msgr2=0x55aa7ae46750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.922+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 shutdown_connections 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.922+0000 7f050dc06740 1 -- 172.21.15.116:0/3043247718 wait complete. 2024-03-19T02:48:26.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.922+0000 7f050dc06740 20 client.21360 trim_cache size 0 max 16384 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.938+0000 7f050dc06740 1 Processor -- start 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.938+0000 7f050dc06740 1 -- start start 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae52e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae537d0 0x55aa7ae6b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05041aeb60 con 0x55aa7ae4abf0 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05041b0700 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f050dc06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05041af230 con 0x55aa7ae537d0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33314/0 (socket says 172.21.15.116:33314) 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 -- 172.21.15.116:0/2602478867 learned_addr learned my addr 172.21.15.116:0/2602478867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0508db5640 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae52e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f05095b6640 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae537d0 0x55aa7ae6b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 -- 172.21.15.116:0/2602478867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae537d0 msgr2=0x55aa7ae6b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae537d0 0x55aa7ae6b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 -- 172.21.15.116:0/2602478867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 msgr2=0x55aa7ae52e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae52e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 -- 172.21.15.116:0/2602478867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7ae44830 con 0x55aa7ae4abf0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.939+0000 7f0503fff640 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f04f00976d0 tx=0x7f04f0097640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c8295707063aac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.940+0000 7f04ee7fc640 1 -- 172.21.15.116:0/2602478867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f00672a0 con 0x55aa7ae4abf0 2024-03-19T02:48:26.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.940+0000 7f04ee7fc640 1 -- 172.21.15.116:0/2602478867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04f009b270 con 0x55aa7ae4abf0 2024-03-19T02:48:26.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.940+0000 7f04ee7fc640 1 -- 172.21.15.116:0/2602478867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f00978e0 con 0x55aa7ae4abf0 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- 172.21.15.116:0/2602478867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 msgr2=0x55aa7ae5f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f04f00976d0 tx=0x7f04f0097640 comp rx=0 tx=0).stop 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- 172.21.15.116:0/2602478867 shutdown_connections 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae537d0 0x55aa7ae6b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae5f390 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 --2- 172.21.15.116:0/2602478867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae52e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- 172.21.15.116:0/2602478867 >> 172.21.15.116:0/2602478867 conn(0x55aa7ae44a00 msgr2=0x55aa7ae43ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- 172.21.15.116:0/2602478867 shutdown_connections 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- 172.21.15.116:0/2602478867 wait complete. 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 -- start start 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 Processor -- start 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.941+0000 7f050dc06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae57ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae537d0 0x55aa7aec8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05041b0030 con 0x55aa7ae4abf0 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05041af900 con 0x55aa7ae537d0 2024-03-19T02:48:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f05041acb60 con 0x55aa7ae4a2d0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 10 client.0 _ll_register_callbacks cb 0x7f04f0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33320/0 (socket says 172.21.15.116:33320) 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 -- 172.21.15.116:0/3862585304 learned_addr learned my addr 172.21.15.116:0/3862585304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f05095b6640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae537d0 0x55aa7aec8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0508db5640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae57ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 -- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 msgr2=0x55aa7ae57ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae57ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 -- 172.21.15.116:0/3862585304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae537d0 msgr2=0x55aa7aec8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae537d0 0x55aa7aec8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 -- 172.21.15.116:0/3862585304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7aec9420 con 0x55aa7ae4abf0 2024-03-19T02:48:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f0503fff640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f04f0097c40 tx=0x7f04f00cc280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626f0dac481aa4ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f04e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0066470 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f04f009b4d0 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.942+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f009b270 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f04f0082050 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 10 client.21366 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55aa7aeca720 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04f009b270 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f04f000b040 con 0x55aa7ae4abf0 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 10 client.21366 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 10 client.21366 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:48:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 20 client.21366 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7aecae30 0x55aa7aecd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 4 client.21366 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa7adba9f0 con 0x55aa7aecae30 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.943+0000 7f0508db5640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7aecae30 0x55aa7aecd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f04e3fff640 10 client.21366 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f0508db5640 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7aecae30 0x55aa7aecd250 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=52f09bad99382453 server_cookie=11eaf58c69923425 in_seq=0 out_seq=0 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f04e3fff640 1 -- 172.21.15.116:0/3862585304 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa7adba9f0 con 0x55aa7aecae30 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f04e3fff640 10 client.21366 handle_command_reply: tid=0 2024-03-19T02:48:26.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f050dc06740 1 client.21366 shutdown 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 msgr2=0x55aa7ae58000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f04f0097c40 tx=0x7f04f00cc280 comp rx=0 tx=0).stop 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 shutdown_connections 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.944+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55aa7aecae30 0x55aa7aecd250 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa7ae537d0 0x55aa7aec8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa7ae4abf0 0x55aa7ae58000 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 --2- 172.21.15.116:0/3862585304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aa7ae4a2d0 0x55aa7ae57ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f04e3fff640 0 client.21366 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 >> 172.21.15.116:0/3862585304 conn(0x55aa7ae44a00 msgr2=0x55aa7ae506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 shutdown_connections 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 1 -- 172.21.15.116:0/3862585304 wait complete. 2024-03-19T02:48:26.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050dc06740 20 client.21366 trim_cache size 0 max 16384 2024-03-19T02:48:26.953 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.945+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 msgr2=0x7f04f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 0x7f04f40a4e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f04fc004af0 tx=0x7f04fc032e20 comp rx=0 tx=0).stop 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 msgr2=0x7f05041984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f04f000e9f0 tx=0x7f04f000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 shutdown_connections 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f04f40a2970 0x7f04f40a4e30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f050419c850 0x7f050419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0504105150 0x7f05041984d0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 --2- 172.21.15.116:0/509270236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0504100080 0x7f0504197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 >> 172.21.15.116:0/509270236 conn(0x7f0504076c50 msgr2=0x7f0504103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 shutdown_connections 2024-03-19T02:48:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:26.946+0000 7f050b019640 1 -- 172.21.15.116:0/509270236 wait complete. 2024-03-19T02:48:26.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:48:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/667480012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c0073c50 msgr2=0x7fc4c0074030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/667480012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c0073c50 0x7fc4c0074030 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009900 tx=0x7fc4bc02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/667480012 shutdown_connections 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/667480012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c010dfd0 0x7fc4c0110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/667480012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0074570 0x7fc4c010da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/667480012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c0073c50 0x7fc4c0074030 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/667480012 >> 172.21.15.116:0/667480012 conn(0x7fc4c00fd2c0 msgr2=0x7fc4c00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/667480012 shutdown_connections 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.089+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/667480012 wait complete. 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 Processor -- start 2024-03-19T02:48:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 -- start start 2024-03-19T02:48:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 0x7fc4c019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 0x7fc4c019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c0113c70 con 0x7fc4c0074570 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c0113af0 con 0x7fc4c0073c50 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c6c8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c0113df0 con 0x7fc4c010dfd0 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 0x7fc4c019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4c5227640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 0x7fc4c019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33338/0 (socket says 172.21.15.116:33338) 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.090+0000 7fc4b7fff640 1 -- 172.21.15.116:0/2034805482 learned_addr learned my addr 172.21.15.116:0/2034805482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2034805482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 msgr2=0x7fc4c019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 0x7fc4c019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 msgr2=0x7fc4c019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2034805482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c019f200 con 0x7fc4c0073c50 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c5227640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 0x7fc4c019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 0x7fc4c019a210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc02c210 tx=0x7fc4bc02c700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4b5ffb640 1 -- 172.21.15.116:0/2034805482 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc043020 con 0x7fc4c0073c50 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c019f490 con 0x7fc4c0073c50 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c01a6d80 con 0x7fc4c0073c50 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4b5ffb640 1 -- 172.21.15.116:0/2034805482 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4bc02dba0 con 0x7fc4c0073c50 2024-03-19T02:48:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.091+0000 7fc4b5ffb640 1 -- 172.21.15.116:0/2034805482 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc003850 con 0x7fc4c0073c50 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.094+0000 7fc4b5ffb640 1 -- 172.21.15.116:0/2034805482 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc4bc046020 con 0x7fc4c0073c50 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 Processor -- start 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4b5ffb640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 0x7fc4a80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 -- start start 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4b5ffb640 1 -- 172.21.15.116:0/2034805482 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc4bc0f0f50 con 0x7fc4c0073c50 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b749200290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c5227640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 0x7fc4a80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749202bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b74915a4b0 con 0x55b7491f72d0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b749102d70 con 0x55b7491f7bf0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c9871740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b749156550 con 0x55b7492007d0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33348/0 (socket says 172.21.15.116:33348) 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c4a26640 1 -- 172.21.15.116:0/637350142 learned_addr learned my addr 172.21.15.116:0/637350142 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.095+0000 7fc4c5227640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749202bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b749200290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 -- 172.21.15.116:0/637350142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 msgr2=0x55b749202bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749202bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 -- 172.21.15.116:0/637350142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 msgr2=0x55b749200290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b749200290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 -- 172.21.15.116:0/637350142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b749125210 con 0x55b7491f72d0 2024-03-19T02:48:27.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b749200290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:48:27.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c5227640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 0x7fc4a80a4eb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8005fd0 tx=0x7fc4b8011f40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.096+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0bd390 tx=0x7fc4bc0bcf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcc4a60872044fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.099+0000 7fc4a6ffd640 1 -- 172.21.15.116:0/637350142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0bd400 con 0x55b7491f72d0 2024-03-19T02:48:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.099+0000 7fc4a6ffd640 1 -- 172.21.15.116:0/637350142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4bc0c1550 con 0x55b7491f72d0 2024-03-19T02:48:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.099+0000 7fc4a6ffd640 1 -- 172.21.15.116:0/637350142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0c1120 con 0x55b7491f72d0 2024-03-19T02:48:27.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- 172.21.15.116:0/637350142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 msgr2=0x55b7491f76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0bd390 tx=0x7fc4bc0bcf40 comp rx=0 tx=0).stop 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- 172.21.15.116:0/637350142 shutdown_connections 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749202bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b749200290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 --2- 172.21.15.116:0/637350142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491f76b0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- 172.21.15.116:0/637350142 >> 172.21.15.116:0/637350142 conn(0x55b7491f1a00 msgr2=0x55b7491f1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- 172.21.15.116:0/637350142 shutdown_connections 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- 172.21.15.116:0/637350142 wait complete. 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.100+0000 7fc4c9871740 1 -- start start 2024-03-19T02:48:27.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 Processor -- start 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 0x55b7491fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7492007d0 0x55b7491fe280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b749203380 con 0x55b7491f7bf0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b749116310 con 0x55b7492007d0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b749203500 con 0x55b7491f72d0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c9871740 10 client.0 _ll_register_callbacks cb 0x55b74910a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33352/0 (socket says 172.21.15.116:33352) 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 0x55b7491fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4b7fff640 1 -- 172.21.15.116:0/1805833950 learned_addr learned my addr 172.21.15.116:0/1805833950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.101+0000 7fc4c5227640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7492007d0 0x55b7491fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 -- 172.21.15.116:0/1805833950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7492007d0 msgr2=0x55b7491fe280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7492007d0 0x55b7491fe280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 -- 172.21.15.116:0/1805833950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 msgr2=0x55b7491fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 -- 172.21.15.116:0/1805833950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b748e6e3f0 con 0x55b7491f72d0 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 0x55b7491fd800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0c5d60 tx=0x7fc4bc0bce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c46543e117051bfb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.102+0000 7fc49ffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:48:27.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.104+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0b9ae0 con 0x55b7491f72d0 2024-03-19T02:48:27.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.104+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc4bc0c1120 con 0x55b7491f72d0 2024-03-19T02:48:27.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.104+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4bc0c1550 con 0x55b7491f72d0 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.109+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc4bc0c1320 con 0x55b7491f72d0 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.109+0000 7fc4c9871740 10 client.31081 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.109+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b748f387d0 con 0x55b7491f72d0 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4bc0c1550 con 0x55b7491f72d0 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc4bc0bc2c0 con 0x55b7491f72d0 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 10 client.31081 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 10 client.31081 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:48:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 20 client.31081 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749210760 0x55b749212b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 4 client.31081 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7491b83e0 con 0x55b749210760 2024-03-19T02:48:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.110+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749210760 0x55b749212b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc49ffff640 10 client.31081 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749210760 0x55b749212b50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b8e2d14002a70756 server_cookie=6a9680cfd36d200e in_seq=0 out_seq=0 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc49ffff640 1 -- 172.21.15.116:0/1805833950 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b7491b83e0 con 0x55b749210760 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc49ffff640 10 client.31081 handle_command_reply: tid=0 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 client.31081 shutdown 2024-03-19T02:48:27.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 msgr2=0x55b7491fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 0x55b7491fd800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0c5d60 tx=0x7fc4bc0bce60 comp rx=0 tx=0).stop 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 shutdown_connections 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7492007d0 0x55b7491fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749210760 0x55b749212b50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 --2- 172.21.15.116:0/1805833950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7491f72d0 0x55b7491fd800 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc49ffff640 0 client.31081 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 >> 172.21.15.116:0/1805833950 conn(0x55b7491f1a00 msgr2=0x55b7491f3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.111+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 shutdown_connections 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.112+0000 7fc4c9871740 1 -- 172.21.15.116:0/1805833950 wait complete. 2024-03-19T02:48:27.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.112+0000 7fc4c9871740 20 client.31081 trim_cache size 0 max 16384 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 Processor -- start 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 -- start start 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b7491fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749218420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac00ff60 con 0x55b7491f72d0 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac011b00 con 0x55b7491f7bf0 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c9871740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac010630 con 0x55b7492007d0 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33364/0 (socket says 172.21.15.116:33364) 2024-03-19T02:48:27.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2810671656 learned_addr learned my addr 172.21.15.116:0/2810671656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c5227640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749218420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b7491fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2810671656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 msgr2=0x55b749218420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749218420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2810671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 msgr2=0x55b7491fe0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b7491fe0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c4a26640 1 -- 172.21.15.116:0/2810671656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b749202bc0 con 0x55b7491f72d0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.126+0000 7fc4c5227640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749218420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0bd100 tx=0x7fc4bc02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955392f46e5dfce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4a67fc640 1 -- 172.21.15.116:0/2810671656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0c5ba0 con 0x55b7491f72d0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4a67fc640 1 -- 172.21.15.116:0/2810671656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4bc0c1550 con 0x55b7491f72d0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4a67fc640 1 -- 172.21.15.116:0/2810671656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0b9b00 con 0x55b7491f72d0 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 -- 172.21.15.116:0/2810671656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 msgr2=0x55b7491fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0bd100 tx=0x7fc4bc02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 -- 172.21.15.116:0/2810671656 shutdown_connections 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b749218420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f7bf0 0x55b7491fe0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 --2- 172.21.15.116:0/2810671656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f72d0 0x55b7491fff30 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.127+0000 7fc4c9871740 1 -- 172.21.15.116:0/2810671656 >> 172.21.15.116:0/2810671656 conn(0x55b7491f1a00 msgr2=0x55b7491f2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- 172.21.15.116:0/2810671656 shutdown_connections 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- 172.21.15.116:0/2810671656 wait complete. 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- start start 2024-03-19T02:48:27.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 Processor -- start 2024-03-19T02:48:27.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 0x55b7491fa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b7491faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac011430 con 0x55b7491f7bf0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac010d00 con 0x55b7491f72d0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac00df60 con 0x55b7492007d0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c9871740 10 client.0 _ll_register_callbacks cb 0x7fc4bc0b9770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33374/0 (socket says 172.21.15.116:33374) 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4b7fff640 1 -- 172.21.15.116:0/4138394999 learned_addr learned my addr 172.21.15.116:0/4138394999 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 0x55b7491fa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.128+0000 7fc4c5227640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b7491faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 -- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 msgr2=0x55b7491faf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b7491faf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 -- 172.21.15.116:0/4138394999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 msgr2=0x55b7491fa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 0x55b7491fa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 -- 172.21.15.116:0/4138394999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b749275bb0 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 0x55b7491fa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4b7fff640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005d00 tx=0x7fc4b001f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56177e10e84e82d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc49ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0005220 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc4b00053c0 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4bc0c1550 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc4b0032050 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4c9871740 10 client.21384 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b749276ec0 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.129+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4bc0c1550 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc4b0022410 con 0x55b7491f7bf0 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 10 client.21384 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 10 client.21384 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:48:27.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 20 client.21384 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749277670 0x55b749279a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 4 client.21384 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b7491679f0 con 0x55b749277670 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749277670 0x55b749279a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c4a26640 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749277670 0x55b749279a90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86f4490883fecbee server_cookie=5ec81db84d0f68a7 in_seq=0 out_seq=0 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc49ffff640 10 client.21384 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc49ffff640 1 -- 172.21.15.116:0/4138394999 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b7491679f0 con 0x55b749277670 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc49ffff640 10 client.21384 handle_command_reply: tid=0 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 1 client.21384 shutdown 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 msgr2=0x55b7491faa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.130+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005d00 tx=0x7fc4b001f880 comp rx=0 tx=0).stop 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 shutdown_connections 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b7492007d0 0x55b7491faf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55b749277670 0x55b749279a90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7491f7bf0 0x55b7491faa10 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 --2- 172.21.15.116:0/4138394999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b7491f72d0 0x55b7491fa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc49ffff640 0 client.21384 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 >> 172.21.15.116:0/4138394999 conn(0x55b7491f1a00 msgr2=0x55b7491f3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 shutdown_connections 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 1 -- 172.21.15.116:0/4138394999 wait complete. 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c9871740 20 client.21384 trim_cache size 0 max 16384 2024-03-19T02:48:27.139 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 msgr2=0x7fc4a80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 0x7fc4a80a4eb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8005fd0 tx=0x7fc4b8011f40 comp rx=0 tx=0).stop 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 msgr2=0x7fc4c019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 0x7fc4c019a210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc02c210 tx=0x7fc4bc02c700 comp rx=0 tx=0).stop 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 shutdown_connections 2024-03-19T02:48:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4a80a29f0 0x7fc4a80a4eb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4c010dfd0 0x7fc4c019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4c0074570 0x7fc4c019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 --2- 172.21.15.116:0/2034805482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4c0073c50 0x7fc4c019a210 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 >> 172.21.15.116:0/2034805482 conn(0x7fc4c00fd2c0 msgr2=0x7fc4c00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 shutdown_connections 2024-03-19T02:48:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.131+0000 7fc4c6c8a640 1 -- 172.21.15.116:0/2034805482 wait complete. 2024-03-19T02:48:27.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:48:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 -- 172.21.15.116:0/67255625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001066b0 msgr2=0x7f2100106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 --2- 172.21.15.116:0/67255625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001066b0 0x7f2100106a90 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009930 tx=0x7f20fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 -- 172.21.15.116:0/67255625 shutdown_connections 2024-03-19T02:48:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 --2- 172.21.15.116:0/67255625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2100101060 0x7f210010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 --2- 172.21.15.116:0/67255625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21001006a0 0x7f2100100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 --2- 172.21.15.116:0/67255625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001066b0 0x7f2100106a90 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 -- 172.21.15.116:0/67255625 >> 172.21.15.116:0/67255625 conn(0x7f21000778d0 msgr2=0x7f2100077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 -- 172.21.15.116:0/67255625 shutdown_connections 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.268+0000 7f2107c12640 1 -- 172.21.15.116:0/67255625 wait complete. 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 Processor -- start 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 -- start start 2024-03-19T02:48:27.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 0x7f21001984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21001066b0 0x7f210019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2100110970 con 0x7f21001006a0 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21001107f0 con 0x7f2100101060 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f2107c12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2100110af0 con 0x7f21001066b0 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f21059ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f21059ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33384/0 (socket says 172.21.15.116:33384) 2024-03-19T02:48:27.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f21059ae640 1 -- 172.21.15.116:0/3476945297 learned_addr learned my addr 172.21.15.116:0/3476945297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.269+0000 7f21051ad640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 0x7f21001984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21061af640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21001066b0 0x7f210019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 -- 172.21.15.116:0/3476945297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21001066b0 msgr2=0x7f210019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21001066b0 0x7f210019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 -- 172.21.15.116:0/3476945297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 msgr2=0x7f21001984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 0x7f21001984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 -- 172.21.15.116:0/3476945297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210019cfa0 con 0x7f21001006a0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21051ad640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 0x7f21001984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f21059ae640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009230 tx=0x7f20fc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f20f6ffd640 1 -- 172.21.15.116:0/3476945297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc043020 con 0x7f21001006a0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210019d230 con 0x7f21001006a0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f20f6ffd640 1 -- 172.21.15.116:0/3476945297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20fc02ca70 con 0x7f21001006a0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f20f6ffd640 1 -- 172.21.15.116:0/3476945297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc003a50 con 0x7f21001006a0 2024-03-19T02:48:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.270+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21001a4b20 con 0x7f21001006a0 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.272+0000 7f20f6ffd640 1 -- 172.21.15.116:0/3476945297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f20fc046020 con 0x7f21001006a0 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 Processor -- start 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 -- start start 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f20f6ffd640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 0x7f20ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622bd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c6622174b0 con 0x55c6622bd7d0 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c6621bfd70 con 0x55c6622b4bf0 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f210a802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c662213550 con 0x55c6622b42d0 2024-03-19T02:48:27.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f21061af640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 0x7f20ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f20f6ffd640 1 -- 172.21.15.116:0/3476945297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f20fc0f1110 con 0x7f21001006a0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.273+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21051ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622bd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45196/0 (socket says 172.21.15.116:45196) 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 -- 172.21.15.116:0/1307036430 learned_addr learned my addr 172.21.15.116:0/1307036430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21061af640 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 -- 172.21.15.116:0/1307036430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 msgr2=0x55c6622bd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622bd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 -- 172.21.15.116:0/1307036430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 msgr2=0x55c6622bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 -- 172.21.15.116:0/1307036430 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6621e2210 con 0x55c6622b42d0 2024-03-19T02:48:27.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21061af640 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21059ae640 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f20fc005ec0 tx=0x7f20fc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f1733607c00f597 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f21061af640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 0x7f20ec0a4e30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55c6622afb90 tx=0x7f20f8009aa0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f20ebfff640 1 -- 172.21.15.116:0/1307036430 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc030c60 con 0x55c6622b42d0 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f20ebfff640 1 -- 172.21.15.116:0/1307036430 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20fc0bdba0 con 0x55c6622b42d0 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.274+0000 7f20ebfff640 1 -- 172.21.15.116:0/1307036430 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0c5c40 con 0x55c6622b42d0 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 -- 172.21.15.116:0/1307036430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 msgr2=0x55c6622b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f20fc005ec0 tx=0x7f20fc009260 comp rx=0 tx=0).stop 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 -- 172.21.15.116:0/1307036430 shutdown_connections 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622bd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 --2- 172.21.15.116:0/1307036430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b46b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 -- 172.21.15.116:0/1307036430 >> 172.21.15.116:0/1307036430 conn(0x55c6622aea00 msgr2=0x55c6622aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 -- 172.21.15.116:0/1307036430 shutdown_connections 2024-03-19T02:48:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.275+0000 7f210a802740 1 -- 172.21.15.116:0/1307036430 wait complete. 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 -- start start 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 Processor -- start 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 0x55c6622bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622bd7d0 0x55c6622bb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c6622c0380 con 0x55c6622b4bf0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c6621d3310 con 0x55c6622bd7d0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c6622c0500 con 0x55c6622b42d0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f210a802740 10 client.0 _ll_register_callbacks cb 0x55c6621c72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21051ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 0x55c6622bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45204/0 (socket says 172.21.15.116:45204) 2024-03-19T02:48:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 -- 172.21.15.116:0/3171223813 learned_addr learned my addr 172.21.15.116:0/3171223813 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21061af640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622bd7d0 0x55c6622bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 -- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622bd7d0 msgr2=0x55c6622bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622bd7d0 0x55c6622bb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 -- 172.21.15.116:0/3171223813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 msgr2=0x55c6622bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 0x55c6622bacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21059ae640 1 -- 172.21.15.116:0/3171223813 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c661f2b3f0 con 0x55c6622b42d0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.276+0000 7f21051ad640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 0x55c6622bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f21059ae640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f20fc0c17b0 tx=0x7f20fc007ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac877988325c7779 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0c5c40 con 0x55c6622b42d0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f20fc0053d0 con 0x55c6622b42d0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20fc0bdba0 con 0x55c6622b42d0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f20fc007bc0 con 0x55c6622b42d0 2024-03-19T02:48:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f210a802740 10 client.31099 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c661ff57d0 con 0x55c6622b42d0 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20fc007d60 con 0x55c6622b42d0 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.277+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f20fc003bf0 con 0x55c6622b42d0 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 10 client.31099 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 10 client.31099 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 20 client.31099 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c6622cd620 0x55c6622cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 4 client.31099 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f20fc0bdba0 con 0x55c6622cd620 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f21051ad640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c6622cd620 0x55c6622cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f20e97fa640 10 client.31099 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.278+0000 7f21051ad640 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c6622cd620 0x55c6622cfa10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65493a954026de35 server_cookie=d15707d98c2ff897 in_seq=0 out_seq=0 2024-03-19T02:48:27.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f20e97fa640 1 -- 172.21.15.116:0/3171223813 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f20fc0bdba0 con 0x55c6622cd620 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f20e97fa640 10 client.31099 handle_command_reply: tid=0 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 client.31099 shutdown 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 msgr2=0x55c6622ba790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f20fc0c17b0 tx=0x7f20fc007ae0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 shutdown_connections 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c6622cd620 0x55c6622cfa10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622bd7d0 0x55c6622bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622b4bf0 0x55c6622bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 --2- 172.21.15.116:0/3171223813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622ba790 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 >> 172.21.15.116:0/3171223813 conn(0x55c6622aea00 msgr2=0x55c6622b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f20e97fa640 0 client.31099 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 shutdown_connections 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 1 -- 172.21.15.116:0/3171223813 wait complete. 2024-03-19T02:48:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.280+0000 7f210a802740 20 client.31099 trim_cache size 0 max 16384 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.303+0000 7f210a802740 1 Processor -- start 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.303+0000 7f210a802740 1 -- start start 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b42d0 0x55c6622c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b4bf0 0x55c6622bb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21001aec00 con 0x55c6622bd7d0 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21001b07a0 con 0x55c6622b42d0 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f210a802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21001af2d0 con 0x55c6622b4bf0 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21051ad640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b4bf0 0x55c6622bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33408/0 (socket says 172.21.15.116:33408) 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 -- 172.21.15.116:0/1996266923 learned_addr learned my addr 172.21.15.116:0/1996266923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21059ae640 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b42d0 0x55c6622c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 -- 172.21.15.116:0/1996266923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b4bf0 msgr2=0x55c6622bb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b4bf0 0x55c6622bb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 -- 172.21.15.116:0/1996266923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b42d0 msgr2=0x55c6622c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b42d0 0x55c6622c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.304+0000 7f21061af640 1 -- 172.21.15.116:0/1996266923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6622ae830 con 0x55c6622bd7d0 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.305+0000 7f21061af640 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f20f80039c0 tx=0x7f20f8015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa18615da4e6154 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.305+0000 7f20eb7fe640 1 -- 172.21.15.116:0/1996266923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80047d0 con 0x55c6622bd7d0 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.305+0000 7f20eb7fe640 1 -- 172.21.15.116:0/1996266923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20f80104f0 con 0x55c6622bd7d0 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.305+0000 7f20eb7fe640 1 -- 172.21.15.116:0/1996266923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8028610 con 0x55c6622bd7d0 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- 172.21.15.116:0/1996266923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 msgr2=0x55c6622d5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f20f80039c0 tx=0x7f20f8015040 comp rx=0 tx=0).stop 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- 172.21.15.116:0/1996266923 shutdown_connections 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622d5200 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b4bf0 0x55c6622bb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 --2- 172.21.15.116:0/1996266923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b42d0 0x55c6622c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- 172.21.15.116:0/1996266923 >> 172.21.15.116:0/1996266923 conn(0x55c6622aea00 msgr2=0x55c6622adba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- 172.21.15.116:0/1996266923 shutdown_connections 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- 172.21.15.116:0/1996266923 wait complete. 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 -- start start 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.306+0000 7f210a802740 1 Processor -- start 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21001b00d0 con 0x55c6622bd7d0 2024-03-19T02:48:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21001af9a0 con 0x55c6622b4bf0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f21001acc00 con 0x55c6622b42d0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f210a802740 10 client.0 _ll_register_callbacks cb 0x7f20f0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21051ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21059ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45212/0 (socket says 172.21.15.116:45212) 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33418/0 (socket says 172.21.15.116:33418) 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21059ae640 1 -- 172.21.15.116:0/2162667761 learned_addr learned my addr 172.21.15.116:0/2162667761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 -- 172.21.15.116:0/2162667761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 msgr2=0x55c6622b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 -- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 msgr2=0x55c6622b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.307+0000 7f21061af640 1 -- 172.21.15.116:0/2162667761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6622c0250 con 0x55c6622bd7d0 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f21061af640 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f20f8004420 tx=0x7f20f8004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabbacf883c6fca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f20e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80108b0 con 0x55c6622bd7d0 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f20f8010a50 con 0x55c6622bd7d0 2024-03-19T02:48:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20f80104f0 con 0x55c6622bd7d0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f20f8032440 con 0x55c6622bd7d0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f210a802740 10 client.21399 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.308+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c6622bfbc0 con 0x55c6622bd7d0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20f80104f0 con 0x55c6622bd7d0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f20f8030020 con 0x55c6622bd7d0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 10 client.21399 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 10 client.21399 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 20 client.21399 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c662334030 0x55c662336420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 4 client.21399 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6622753e0 con 0x55c662334030 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f21059ae640 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c662334030 0x55c662336420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f20e97fa640 10 client.21399 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.309+0000 7f21059ae640 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c662334030 0x55c662336420 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3dc522c9bc0423a server_cookie=9a4a549aa2ca2f9d in_seq=0 out_seq=0 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.310+0000 7f20e97fa640 1 -- 172.21.15.116:0/2162667761 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6622753e0 con 0x55c662334030 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.310+0000 7f20e97fa640 10 client.21399 handle_command_reply: tid=0 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.310+0000 7f210a802740 1 client.21399 shutdown 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.310+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 msgr2=0x55c6622b7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.310+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f20f8004420 tx=0x7f20f8004c70 comp rx=0 tx=0).stop 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 shutdown_connections 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55c662334030 0x55c662336420 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6622bd7d0 0x55c6622b7ff0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6622b4bf0 0x55c6622b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 --2- 172.21.15.116:0/2162667761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6622b42d0 0x55c6622b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 >> 172.21.15.116:0/2162667761 conn(0x55c6622aea00 msgr2=0x55c6622afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f20e97fa640 0 client.21399 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:48:27.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 shutdown_connections 2024-03-19T02:48:27.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 1 -- 172.21.15.116:0/2162667761 wait complete. 2024-03-19T02:48:27.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.311+0000 7f210a802740 20 client.21399 trim_cache size 0 max 16384 2024-03-19T02:48:27.321 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 msgr2=0x7f20ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 0x7f20ec0a4e30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55c6622afb90 tx=0x7f20f8009aa0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 msgr2=0x7f2100197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009230 tx=0x7f20fc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 shutdown_connections 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f20ec0a2970 0x7f20ec0a4e30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21001066b0 0x7f210019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2100101060 0x7f21001984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 --2- 172.21.15.116:0/3476945297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21001006a0 0x7f2100197fb0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 >> 172.21.15.116:0/3476945297 conn(0x7f21000778d0 msgr2=0x7f2100068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 shutdown_connections 2024-03-19T02:48:27.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.312+0000 7f2107c12640 1 -- 172.21.15.116:0/3476945297 wait complete. 2024-03-19T02:48:27.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:48:27.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52611 5.97 MB/sec execute 349 sec latency 158.150 ms 2024-03-19T02:48:27.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:27 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:27 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/740549549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:27 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:27 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/740549549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.454+0000 7fe6ad377640 1 -- 172.21.15.116:0/95177189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a8074750 msgr2=0x7fe6a810da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.454+0000 7fe6ad377640 1 --2- 172.21.15.116:0/95177189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a8074750 0x7fe6a810da90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009900 tx=0x7fe69c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 -- 172.21.15.116:0/95177189 shutdown_connections 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 --2- 172.21.15.116:0/95177189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a810dfd0 0x7fe6a8110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 --2- 172.21.15.116:0/95177189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a8074750 0x7fe6a810da90 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 --2- 172.21.15.116:0/95177189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8073e30 0x7fe6a8074210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 -- 172.21.15.116:0/95177189 >> 172.21.15.116:0/95177189 conn(0x7fe6a80fd2c0 msgr2=0x7fe6a80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 -- 172.21.15.116:0/95177189 shutdown_connections 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 -- 172.21.15.116:0/95177189 wait complete. 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 Processor -- start 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.455+0000 7fe6ad377640 1 -- start start 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a8073e30 0x7fe6a819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8074750 0x7fe6a819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113c80 con 0x7fe6a810dfd0 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113b00 con 0x7fe6a8073e30 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a8113e00 con 0x7fe6a8074750 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33432/0 (socket says 172.21.15.116:33432) 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 -- 172.21.15.116:0/1540387027 learned_addr learned my addr 172.21.15.116:0/1540387027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8074750 0x7fe6a819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a8073e30 0x7fe6a819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 -- 172.21.15.116:0/1540387027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8074750 msgr2=0x7fe6a819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8074750 0x7fe6a819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 -- 172.21.15.116:0/1540387027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a8073e30 msgr2=0x7fe6a819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a8073e30 0x7fe6a819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 -- 172.21.15.116:0/1540387027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a819f210 con 0x7fe6a810dfd0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6a77fe640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000eb10 tx=0x7fe6a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe68ffff640 1 -- 172.21.15.116:0/1540387027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0004b20 con 0x7fe6a810dfd0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.456+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a819f500 con 0x7fe6a810dfd0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.457+0000 7fe68ffff640 1 -- 172.21.15.116:0/1540387027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6a0004cc0 con 0x7fe6a810dfd0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.457+0000 7fe68ffff640 1 -- 172.21.15.116:0/1540387027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0010690 con 0x7fe6a810dfd0 2024-03-19T02:48:27.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.457+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a81a6d90 con 0x7fe6a810dfd0 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe68ffff640 1 -- 172.21.15.116:0/1540387027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe6a0005020 con 0x7fe6a810dfd0 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 Processor -- start 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 -- start start 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e23bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1d7b4b0 con 0x55a2b1e182d0 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1d23d70 con 0x55a2b1e18bf0 2024-03-19T02:48:27.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1d77550 con 0x55a2b1e217d0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.459+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33434/0 (socket says 172.21.15.116:33434) 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40200/0 (socket says 172.21.15.116:40200) 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2082552620 learned_addr learned my addr 172.21.15.116:0/2082552620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e23bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe68ffff640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 0x7fe6900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a77fe640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 0x7fe6900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2082552620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 msgr2=0x55a2b1e23bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e23bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2082552620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 msgr2=0x55a2b1e21290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2082552620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2b1d46210 con 0x55a2b1e182d0 2024-03-19T02:48:27.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe68ffff640 1 -- 172.21.15.116:0/1540387027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fe6a00cf790 con 0x7fe6a810dfd0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe69c02c780 tx=0x7fe69c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df879fa503e94c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe68d7fa640 1 -- 172.21.15.116:0/2082552620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c04a070 con 0x55a2b1e182d0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe6a77fe640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 0x7fe6900a4e30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6a009b710 tx=0x7fe6a0086540 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe68d7fa640 1 -- 172.21.15.116:0/2082552620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe69c02cc90 con 0x55a2b1e182d0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.460+0000 7fe68d7fa640 1 -- 172.21.15.116:0/2082552620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c034440 con 0x55a2b1e182d0 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2082552620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 msgr2=0x55a2b1e186b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe69c02c780 tx=0x7fe69c005b50 comp rx=0 tx=0).stop 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2082552620 shutdown_connections 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e23bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e21290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2082552620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e182d0 0x55a2b1e186b0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.461+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2082552620 >> 172.21.15.116:0/2082552620 conn(0x55a2b1e12a00 msgr2=0x55a2b1e12de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2082552620 shutdown_connections 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2082552620 wait complete. 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- start start 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 Processor -- start 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1e24560 con 0x55a2b1e18bf0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1d37310 con 0x55a2b1e182d0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2b1e246e0 con 0x55a2b1e217d0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6afbfd740 10 client.0 _ll_register_callbacks cb 0x55a2b1d2b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33450/0 (socket says 172.21.15.116:33450) 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.462+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/4105988637 learned_addr learned my addr 172.21.15.116:0/4105988637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 msgr2=0x55a2b1e1f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/4105988637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 msgr2=0x55a2b1e1e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/4105988637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2b1a8f3f0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fe698012770 tx=0x7fe698012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1a2a1e971f7116 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe68aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698004050 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fe6980041f0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.463+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c02cc90 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe6980043f0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 10 client.21411 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a2b1b597d0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe69c02cc90 con 0x55a2b1e18bf0 2024-03-19T02:48:27.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe698003ba0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 10 client.21411 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 10 client.21411 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 20 client.21411 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1e3a390 0x55a2b1e3c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 4 client.21411 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2b1dd93e0 con 0x55a2b1e3a390 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.464+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1e3a390 0x55a2b1e3c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.465+0000 7fe68aff5640 10 client.21411 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:48:27.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.465+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1e3a390 0x55a2b1e3c810 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b14a76565a1b145 server_cookie=144c213bd303a5d1 in_seq=0 out_seq=0 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe68aff5640 1 -- 172.21.15.116:0/4105988637 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2b1dd93e0 con 0x55a2b1e3a390 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe68aff5640 10 client.21411 handle_command_reply: tid=0 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 client.21411 shutdown 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 msgr2=0x55a2b1e1ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fe698012770 tx=0x7fe698012c30 comp rx=0 tx=0).stop 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 shutdown_connections 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1e3a390 0x55a2b1e3c810 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ed50 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/4105988637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 >> 172.21.15.116:0/4105988637 conn(0x55a2b1e12a00 msgr2=0x55a2b1e14750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe68aff5640 0 client.21411 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 shutdown_connections 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 1 -- 172.21.15.116:0/4105988637 wait complete. 2024-03-19T02:48:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.466+0000 7fe6afbfd740 20 client.21411 trim_cache size 0 max 16384 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 Processor -- start 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 -- start start 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 0x55a2b1e20f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e217d0 0x55a2b1e42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81aca20 con 0x55a2b1e18bf0 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81ae5c0 con 0x55a2b1e217d0 2024-03-19T02:48:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81ad0f0 con 0x55a2b1e182d0 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33458/0 (socket says 172.21.15.116:33458) 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6a67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 0x55a2b1e20f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/1552017284 learned_addr learned my addr 172.21.15.116:0/1552017284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.481+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e217d0 0x55a2b1e42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 -- 172.21.15.116:0/1552017284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e217d0 msgr2=0x55a2b1e42120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e217d0 0x55a2b1e42120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 -- 172.21.15.116:0/1552017284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 msgr2=0x55a2b1e35bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 -- 172.21.15.116:0/1552017284 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2b1e18a20 con 0x55a2b1e182d0 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 0x55a2b1e20f40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe69c036040 tx=0x7fe69c02c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=259f83558051611b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe68cff9640 1 -- 172.21.15.116:0/1552017284 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c04a040 con 0x55a2b1e182d0 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe68cff9640 1 -- 172.21.15.116:0/1552017284 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe69c04b040 con 0x55a2b1e182d0 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.482+0000 7fe68cff9640 1 -- 172.21.15.116:0/1552017284 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c058320 con 0x55a2b1e182d0 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- 172.21.15.116:0/1552017284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 msgr2=0x55a2b1e20f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 0x55a2b1e20f40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe69c036040 tx=0x7fe69c02c630 comp rx=0 tx=0).stop 2024-03-19T02:48:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- 172.21.15.116:0/1552017284 shutdown_connections 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e217d0 0x55a2b1e42120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e35bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/1552017284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e182d0 0x55a2b1e20f40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- 172.21.15.116:0/1552017284 >> 172.21.15.116:0/1552017284 conn(0x55a2b1e12a00 msgr2=0x55a2b1e11ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- 172.21.15.116:0/1552017284 shutdown_connections 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- 172.21.15.116:0/1552017284 wait complete. 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- start start 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 Processor -- start 2024-03-19T02:48:27.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81adef0 con 0x55a2b1e18bf0 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.483+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81ad7c0 con 0x55a2b1e182d0 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6afbfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a81aaa20 con 0x55a2b1e217d0 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6afbfd740 10 client.0 _ll_register_callbacks cb 0x7fe698009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33468/0 (socket says 172.21.15.116:33468) 2024-03-19T02:48:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a6ffd640 1 -- 172.21.15.116:0/2948553388 learned_addr learned my addr 172.21.15.116:0/2948553388 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2948553388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 msgr2=0x55a2b1e1b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2948553388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 msgr2=0x55a2b1e1ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 -- 172.21.15.116:0/2948553388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2b1e9f9a0 con 0x55a2b1e217d0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6a67fc640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1bfb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009850 tx=0x7fe69c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75866815c01f1928 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe68aff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c04a040 con 0x55a2b1e217d0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fe69c0035f0 con 0x55a2b1e217d0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.484+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c04b040 con 0x55a2b1e217d0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe69c06c050 con 0x55a2b1e217d0 2024-03-19T02:48:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 10 client.31123 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a2b1ea0c40 con 0x55a2b1e217d0 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe69c04b040 con 0x55a2b1e217d0 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe69c0584d0 con 0x55a2b1e217d0 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 10 client.31123 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 10 client.31123 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 20 client.31123 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1ea0f20 0x55a2b1ea3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 4 client.31123 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2b1d889f0 con 0x55a2b1ea0f20 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.485+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1ea0f20 0x55a2b1ea3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6a6ffd640 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1ea0f20 0x55a2b1ea3340 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41d4725a3dee6623 server_cookie=dc728b6f5679cf58 in_seq=0 out_seq=0 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe68aff5640 10 client.31123 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe68aff5640 1 -- 172.21.15.116:0/2948553388 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2b1d889f0 con 0x55a2b1ea0f20 2024-03-19T02:48:27.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe68aff5640 10 client.31123 handle_command_reply: tid=0 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 client.31123 shutdown 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 msgr2=0x55a2b1e1bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1bfb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009850 tx=0x7fe69c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 shutdown_connections 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55a2b1ea0f20 0x55a2b1ea3340 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2b1e217d0 0x55a2b1e1bfb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe68aff5640 0 client.31123 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2b1e18bf0 0x55a2b1e1ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 --2- 172.21.15.116:0/2948553388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2b1e182d0 0x55a2b1e1b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 >> 172.21.15.116:0/2948553388 conn(0x55a2b1e12a00 msgr2=0x55a2b1e11ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 shutdown_connections 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 1 -- 172.21.15.116:0/2948553388 wait complete. 2024-03-19T02:48:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.486+0000 7fe6afbfd740 20 client.31123 trim_cache size 0 max 16384 2024-03-19T02:48:27.497 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 msgr2=0x7fe6900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 0x7fe6900a4e30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6a009b710 tx=0x7fe6a0086540 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 msgr2=0x7fe6a819eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000eb10 tx=0x7fe6a000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 shutdown_connections 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6900a2970 0x7fe6900a4e30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6a810dfd0 0x7fe6a819eae0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6a8074750 0x7fe6a819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 --2- 172.21.15.116:0/1540387027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a8073e30 0x7fe6a819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 >> 172.21.15.116:0/1540387027 conn(0x7fe6a80fd2c0 msgr2=0x7fe6a80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 shutdown_connections 2024-03-19T02:48:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.487+0000 7fe6ad377640 1 -- 172.21.15.116:0/1540387027 wait complete. 2024-03-19T02:48:27.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:48:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:27 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:27 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/740549549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:48:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 -- 172.21.15.116:0/3673698251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 msgr2=0x7fad34100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 --2- 172.21.15.116:0/3673698251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34100b20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fad24009930 tx=0x7fad2402bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 -- 172.21.15.116:0/3673698251 shutdown_connections 2024-03-19T02:48:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 --2- 172.21.15.116:0/3673698251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad3410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 --2- 172.21.15.116:0/3673698251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34100b20 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 --2- 172.21.15.116:0/3673698251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad341066b0 0x7fad34106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 -- 172.21.15.116:0/3673698251 >> 172.21.15.116:0/3673698251 conn(0x7fad340778d0 msgr2=0x7fad34077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 -- 172.21.15.116:0/3673698251 shutdown_connections 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.679+0000 7fad3b685640 1 -- 172.21.15.116:0/3673698251 wait complete. 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 Processor -- start 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 -- start start 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad34196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad341066b0 0x7fad3419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410f860 con 0x7fad341006a0 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410f6e0 con 0x7fad34101060 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.680+0000 7fad3b685640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410f9e0 con 0x7fad341066b0 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56336/0 (socket says 172.21.15.116:56336) 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 -- 172.21.15.116:0/703019761 learned_addr learned my addr 172.21.15.116:0/703019761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad38c20640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad34196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 -- 172.21.15.116:0/703019761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad341066b0 msgr2=0x7fad3419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad341066b0 0x7fad3419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 -- 172.21.15.116:0/703019761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 msgr2=0x7fad34196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad34196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 -- 172.21.15.116:0/703019761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3419ae00 con 0x7fad341006a0 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad38c20640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad34196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:48:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.681+0000 7fad39421640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fad30010fc0 tx=0x7fad3000e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.682+0000 7fad2a7fc640 1 -- 172.21.15.116:0/703019761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3001a020 con 0x7fad341006a0 2024-03-19T02:48:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.682+0000 7fad2a7fc640 1 -- 172.21.15.116:0/703019761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad30016930 con 0x7fad341006a0 2024-03-19T02:48:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.682+0000 7fad2a7fc640 1 -- 172.21.15.116:0/703019761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad300159a0 con 0x7fad341006a0 2024-03-19T02:48:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.683+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3419b0f0 con 0x7fad341006a0 2024-03-19T02:48:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.683+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a2980 con 0x7fad341006a0 2024-03-19T02:48:27.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 Processor -- start 2024-03-19T02:48:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 -- start start 2024-03-19T02:48:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x5653047536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530475c7d0 0x56530475ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5653046b64b0 con 0x5653047532d0 2024-03-19T02:48:27.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56530465ed70 con 0x565304753bf0 2024-03-19T02:48:27.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.687+0000 7fad3e279740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5653046b2550 con 0x56530475c7d0 2024-03-19T02:48:27.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.688+0000 7fad2a7fc640 1 -- 172.21.15.116:0/703019761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fad3001d020 con 0x7fad341006a0 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad2a7fc640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 0x7fad200a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad2a7fc640 1 -- 172.21.15.116:0/703019761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fad300d1ab0 con 0x7fad341006a0 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad39421640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad39421640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57276/0 (socket says 172.21.15.116:57276) 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad39421640 1 -- 172.21.15.116:0/3484295947 learned_addr learned my addr 172.21.15.116:0/3484295947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.689+0000 7fad39c22640 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x5653047536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39c22640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 0x7fad200a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39421640 1 -- 172.21.15.116:0/3484295947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530475c7d0 msgr2=0x56530475ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39421640 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530475c7d0 0x56530475ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39421640 1 -- 172.21.15.116:0/3484295947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 msgr2=0x5653047536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39421640 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x5653047536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.690+0000 7fad39421640 1 -- 172.21.15.116:0/3484295947 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565304681210 con 0x565304753bf0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.691+0000 7fad39c22640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 0x7fad200a4f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009fd0 tx=0x7fad2c0097c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.691+0000 7fad39421640 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad300a5f40 tx=0x7fad3009d940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d12d432e20795c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.691+0000 7fad1f7fe640 1 -- 172.21.15.116:0/3484295947 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3009dcd0 con 0x565304753bf0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.691+0000 7fad1f7fe640 1 -- 172.21.15.116:0/3484295947 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad300a1e20 con 0x565304753bf0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.691+0000 7fad1f7fe640 1 -- 172.21.15.116:0/3484295947 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad300cf040 con 0x565304753bf0 2024-03-19T02:48:27.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 -- 172.21.15.116:0/3484295947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 msgr2=0x56530475c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad300a5f40 tx=0x7fad3009d940 comp rx=0 tx=0).stop 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 -- 172.21.15.116:0/3484295947 shutdown_connections 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530475c7d0 0x56530475ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565304753bf0 0x56530475c290 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 --2- 172.21.15.116:0/3484295947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x5653047536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 -- 172.21.15.116:0/3484295947 >> 172.21.15.116:0/3484295947 conn(0x56530474da00 msgr2=0x56530474dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 -- 172.21.15.116:0/3484295947 shutdown_connections 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.692+0000 7fad3e279740 1 -- 172.21.15.116:0/3484295947 wait complete. 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 -- start start 2024-03-19T02:48:27.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 Processor -- start 2024-03-19T02:48:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304759d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565304770510 con 0x5653047532d0 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565304672310 con 0x56530475c7d0 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565304770690 con 0x565304753bf0 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad3e279740 10 client.0 _ll_register_callbacks cb 0x5653046662d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56350/0 (socket says 172.21.15.116:56350) 2024-03-19T02:48:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.693+0000 7fad39421640 1 -- 172.21.15.116:0/787151902 learned_addr learned my addr 172.21.15.116:0/787151902 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad38c20640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304759d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39c22640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 -- 172.21.15.116:0/787151902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 msgr2=0x565304759d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304759d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 -- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 msgr2=0x56530475a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 -- 172.21.15.116:0/787151902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653043ca3f0 con 0x5653047532d0 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad39421640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fad3000bcf0 tx=0x7fad3000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb95082c1c464208 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.694+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3009dcd0 con 0x5653047532d0 2024-03-19T02:48:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fad300bc2c0 con 0x5653047532d0 2024-03-19T02:48:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad300a1e20 con 0x5653047532d0 2024-03-19T02:48:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fad300a6470 con 0x5653047532d0 2024-03-19T02:48:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 10 client.21429 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5653044947d0 con 0x5653047532d0 2024-03-19T02:48:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad300a1e20 con 0x5653047532d0 2024-03-19T02:48:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fad30084030 con 0x5653047532d0 2024-03-19T02:48:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 10 client.21429 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 10 client.21429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 20 client.21429 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56530477db30 0x56530477ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 4 client.21429 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.695+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653047143e0 con 0x56530477db30 2024-03-19T02:48:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.696+0000 7fad38c20640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56530477db30 0x56530477ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.696+0000 7fad38c20640 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56530477db30 0x56530477ff20 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a93bbd307d173c4d server_cookie=fa2ee6ce51ae35f4 in_seq=0 out_seq=0 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.696+0000 7fad1cff9640 10 client.21429 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad1cff9640 1 -- 172.21.15.116:0/787151902 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5653047143e0 con 0x56530477db30 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad1cff9640 10 client.21429 handle_command_reply: tid=0 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 client.21429 shutdown 2024-03-19T02:48:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 msgr2=0x565304759800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fad3000bcf0 tx=0x7fad3000b410 comp rx=0 tx=0).stop 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 shutdown_connections 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56530477db30 0x56530477ff20 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304759d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 --2- 172.21.15.116:0/787151902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5653047532d0 0x565304759800 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 >> 172.21.15.116:0/787151902 conn(0x56530474da00 msgr2=0x56530474f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad1cff9640 0 client.21429 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.698+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 shutdown_connections 2024-03-19T02:48:27.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.699+0000 7fad3e279740 1 -- 172.21.15.116:0/787151902 wait complete. 2024-03-19T02:48:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.699+0000 7fad3e279740 20 client.21429 trim_cache size 0 max 16384 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.719+0000 7fad3e279740 1 Processor -- start 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.719+0000 7fad3e279740 1 -- start start 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530474fa10 0x5653047857b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a65f0 con 0x565304753bf0 2024-03-19T02:48:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a4380 con 0x56530475c7d0 2024-03-19T02:48:27.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad3e279740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a5f20 con 0x56530474fa10 2024-03-19T02:48:27.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39c22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530474fa10 0x5653047857b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56362/0 (socket says 172.21.15.116:56362) 2024-03-19T02:48:27.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 -- 172.21.15.116:0/4184734194 learned_addr learned my addr 172.21.15.116:0/4184734194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 -- 172.21.15.116:0/4184734194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530474fa10 msgr2=0x5653047857b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530474fa10 0x5653047857b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 -- 172.21.15.116:0/4184734194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 msgr2=0x56530475a0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.720+0000 7fad39421640 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.721+0000 7fad39421640 1 -- 172.21.15.116:0/4184734194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565304723880 con 0x565304753bf0 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.721+0000 7fad39421640 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fad300d1de0 tx=0x7fad30087120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef21a0ffd51f516d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.721+0000 7fad1effd640 1 -- 172.21.15.116:0/4184734194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3009dcd0 con 0x565304753bf0 2024-03-19T02:48:27.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.721+0000 7fad1effd640 1 -- 172.21.15.116:0/4184734194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad300a1e20 con 0x565304753bf0 2024-03-19T02:48:27.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.721+0000 7fad1effd640 1 -- 172.21.15.116:0/4184734194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3009d120 con 0x565304753bf0 2024-03-19T02:48:27.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.722+0000 7fad3e279740 1 -- 172.21.15.116:0/4184734194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 msgr2=0x5653047792b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.722+0000 7fad3e279740 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fad300d1de0 tx=0x7fad30087120 comp rx=0 tx=0).stop 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 -- 172.21.15.116:0/4184734194 shutdown_connections 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56530474fa10 0x5653047857b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x56530475a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 --2- 172.21.15.116:0/4184734194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565304753bf0 0x5653047792b0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 -- 172.21.15.116:0/4184734194 >> 172.21.15.116:0/4184734194 conn(0x56530474da00 msgr2=0x56530474cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 -- 172.21.15.116:0/4184734194 shutdown_connections 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.723+0000 7fad3e279740 1 -- 172.21.15.116:0/4184734194 wait complete. 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 -- start start 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 Processor -- start 2024-03-19T02:48:27.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304770e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x5653047713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a5120 con 0x56530474fa10 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a4a50 con 0x56530475c7d0 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad341a5850 con 0x565304753bf0 2024-03-19T02:48:27.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad3e279740 10 client.0 _ll_register_callbacks cb 0x7fad3009e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad39421640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56364/0 (socket says 172.21.15.116:56364) 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad39421640 1 -- 172.21.15.116:0/3391913294 learned_addr learned my addr 172.21.15.116:0/3391913294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.724+0000 7fad38c20640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304770e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39c22640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x5653047713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 -- 172.21.15.116:0/3391913294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 msgr2=0x565304770e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304770e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 -- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 msgr2=0x5653047713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x5653047713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 -- 172.21.15.116:0/3391913294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56530475ebc0 con 0x56530474fa10 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad38c20640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304770e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad39421640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fad300d1040 tx=0x7fad30000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1a4d3d2922b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:27.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30099a20 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fad30016690 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.725+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad300a1e20 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fad3009dcd0 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 10 client.21435 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56530475ebc0 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad300a1e20 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fad30084070 con 0x56530474fa10 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 10 client.21435 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 10 client.21435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 20 client.21435 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5653047e3d20 0x5653047e6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 4 client.21435 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:27.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653046c39f0 con 0x5653047e3d20 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.726+0000 7fad38c20640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5653047e3d20 0x5653047e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.727+0000 7fad1cff9640 10 client.21435 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.727+0000 7fad38c20640 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5653047e3d20 0x5653047e6110 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e813a76fa46034eb server_cookie=d6b0dae48f25a042 in_seq=0 out_seq=0 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad1cff9640 1 -- 172.21.15.116:0/3391913294 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653046c39f0 con 0x5653047e3d20 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad1cff9640 10 client.21435 handle_command_reply: tid=0 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 client.21435 shutdown 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 msgr2=0x5653047759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fad300d1040 tx=0x7fad30000f40 comp rx=0 tx=0).stop 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 shutdown_connections 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5653047e3d20 0x5653047e6110 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56530475c7d0 0x5653047713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565304753bf0 0x565304770e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.728+0000 7fad1cff9640 0 client.21435 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:27.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3e279740 1 --2- 172.21.15.116:0/3391913294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56530474fa10 0x5653047759e0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 >> 172.21.15.116:0/3391913294 conn(0x56530474da00 msgr2=0x56530474ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 shutdown_connections 2024-03-19T02:48:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3e279740 1 -- 172.21.15.116:0/3391913294 wait complete. 2024-03-19T02:48:27.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3e279740 20 client.21435 trim_cache size 0 max 16384 2024-03-19T02:48:27.734 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 msgr2=0x7fad200a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 0x7fad200a4f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009fd0 tx=0x7fad2c0097c0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 msgr2=0x7fad34195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fad30010fc0 tx=0x7fad3000e490 comp rx=0 tx=0).stop 2024-03-19T02:48:27.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 shutdown_connections 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fad200a2a70 0x7fad200a4f30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad341066b0 0x7fad3419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad34101060 0x7fad34196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 --2- 172.21.15.116:0/703019761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad341006a0 0x7fad34195e10 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 >> 172.21.15.116:0/703019761 conn(0x7fad340778d0 msgr2=0x7fad34103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 shutdown_connections 2024-03-19T02:48:27.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:27.729+0000 7fad3b685640 1 -- 172.21.15.116:0/703019761 wait complete. 2024-03-19T02:48:28.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:28 smithi116 ceph-mon[32263]: pgmap v652: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:28.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:28 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:28 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52684 5.97 MB/sec execute 350 sec latency 153.578 ms 2024-03-19T02:48:28.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:28 smithi162 ceph-mon[35761]: pgmap v652: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:28.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:28 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:28 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:28 smithi154 ceph-mon[35693]: pgmap v652: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:28 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:28 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:28.742 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:48:28.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:48:28.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/2158156794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e4101080 msgr2=0x7fc3e410d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/2158156794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e4101080 0x7fc3e410d540 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009ca0 tx=0x7fc3dc02c180 comp rx=0 tx=0).stop 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/2158156794 shutdown_connections 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/2158156794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e4101080 0x7fc3e410d540 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/2158156794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 0x7fc3e4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/2158156794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e41055c0 0x7fc3e41059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.884+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/2158156794 >> 172.21.15.116:0/2158156794 conn(0x7fc3e40777b0 msgr2=0x7fc3e4077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.885+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/2158156794 shutdown_connections 2024-03-19T02:48:28.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.885+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/2158156794 wait complete. 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.885+0000 7fc3ea5a9640 1 Processor -- start 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.885+0000 7fc3ea5a9640 1 -- start start 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 0x7fc3e4198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e4101080 0x7fc3e4199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4110960 con 0x7fc3e41055c0 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41107e0 con 0x7fc3e4101080 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3ea5a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4110ae0 con 0x7fc3e41006e0 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56372/0 (socket says 172.21.15.116:56372) 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 -- 172.21.15.116:0/4243748766 learned_addr learned my addr 172.21.15.116:0/4243748766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e4101080 0x7fc3e4199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 0x7fc3e4198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 -- 172.21.15.116:0/4243748766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 msgr2=0x7fc3e4198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 0x7fc3e4198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 -- 172.21.15.116:0/4243748766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e4101080 msgr2=0x7fc3e4199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e4101080 0x7fc3e4199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 -- 172.21.15.116:0/4243748766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e41a4380 con 0x7fc3e41055c0 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.886+0000 7fc3e8b46640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc02c790 tx=0x7fc3dc035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.887+0000 7fc3e17fa640 1 -- 172.21.15.116:0/4243748766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc039040 con 0x7fc3e41055c0 2024-03-19T02:48:28.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.887+0000 7fc3e17fa640 1 -- 172.21.15.116:0/4243748766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3dc044070 con 0x7fc3e41055c0 2024-03-19T02:48:28.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.887+0000 7fc3e17fa640 1 -- 172.21.15.116:0/4243748766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc032850 con 0x7fc3e41055c0 2024-03-19T02:48:28.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.887+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e41a4550 con 0x7fc3e41055c0 2024-03-19T02:48:28.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.887+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e41a49b0 con 0x7fc3e41055c0 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.889+0000 7fc3e17fa640 1 -- 172.21.15.116:0/4243748766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc3dc0329f0 con 0x7fc3e41055c0 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 Processor -- start 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 -- start start 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc7833d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78346340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78348c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cc782a1160 con 0x55cc78346880 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cc78249e10 con 0x55cc7833dca0 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3ed194740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc7829d200 con 0x55cc7833d380 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc7833d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc7833d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51686/0 (socket says 172.21.15.116:51686) 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78346340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e37fe640 1 -- 172.21.15.116:0/1743989258 learned_addr learned my addr 172.21.15.116:0/1743989258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78348c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e17fa640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 0x7fc3cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e8b46640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 0x7fc3cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 -- 172.21.15.116:0/1743989258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 msgr2=0x55cc7833d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc7833d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 -- 172.21.15.116:0/1743989258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 msgr2=0x55cc78348c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78348c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.890+0000 7fc3e3fff640 1 -- 172.21.15.116:0/1743989258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc7807d7e0 con 0x55cc7833dca0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78348c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3e17fa640 1 -- 172.21.15.116:0/4243748766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc3dc0f0e70 con 0x7fc3e41055c0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78346340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009c20 tx=0x7fc3d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4766d0fc40d7ced server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3ca7fc640 1 -- 172.21.15.116:0/1743989258 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4004030 con 0x55cc7833dca0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3e8b46640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 0x7fc3cc0a4e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ee7e0 tx=0x7fc3dc0af040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3ca7fc640 1 -- 172.21.15.116:0/1743989258 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3d40035f0 con 0x55cc7833dca0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.891+0000 7fc3ca7fc640 1 -- 172.21.15.116:0/1743989258 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4019620 con 0x55cc7833dca0 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- 172.21.15.116:0/1743989258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 msgr2=0x55cc78346340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78346340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009c20 tx=0x7fc3d4011810 comp rx=0 tx=0).stop 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- 172.21.15.116:0/1743989258 shutdown_connections 2024-03-19T02:48:28.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78348c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78346340 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 --2- 172.21.15.116:0/1743989258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc7833d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- 172.21.15.116:0/1743989258 >> 172.21.15.116:0/1743989258 conn(0x55cc78337ab0 msgr2=0x55cc78337e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- 172.21.15.116:0/1743989258 shutdown_connections 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- 172.21.15.116:0/1743989258 wait complete. 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.892+0000 7fc3ed194740 1 -- start start 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 Processor -- start 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56396/0 (socket says 172.21.15.116:56396) 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3e37fe640 1 -- 172.21.15.116:0/3521258383 learned_addr learned my addr 172.21.15.116:0/3521258383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc78343d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc78346880 0x55cc78344280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cc7834da80 con 0x55cc7833d380 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cc7825d1a0 con 0x55cc78346880 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.893+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc7834dc00 con 0x55cc7833dca0 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc78343d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 -- 172.21.15.116:0/3521258383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 msgr2=0x55cc78343d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc78343d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 -- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc78346880 msgr2=0x55cc78344280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc78346880 0x55cc78344280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc780ddf70 con 0x55cc7833d380 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.894+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8003870 tx=0x7fc3d8003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6c6c0b70a37c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3c37fe640 1 -- 172.21.15.116:0/3521258383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8023060 con 0x55cc7833d380 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 10 client.0 _ll_register_callbacks cb 0x55cc77fb4c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3c37fe640 1 -- 172.21.15.116:0/3521258383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc3d8011c30 con 0x55cc7833d380 2024-03-19T02:48:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d40035f0 con 0x55cc7833d380 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3c37fe640 1 -- 172.21.15.116:0/3521258383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc3d802a050 con 0x55cc7833d380 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 10 client.21447 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=40}) -- 0x55cc77f75170 con 0x55cc7833d380 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3c37fe640 1 -- 172.21.15.116:0/3521258383 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc3d801d030 con 0x55cc7833d380 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 10 client.21447 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 10 client.21447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 20 client.21447 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc7835b2a0 0x55cc7835d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 4 client.21447 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.895+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc782ae6a0 con 0x55cc7835b2a0 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.896+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc7835b2a0 0x55cc7835d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.896+0000 7fc3c37fe640 10 client.21447 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.896+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc7835b2a0 0x55cc7835d760 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d15b2250270da5e server_cookie=be21a8daaf25b086 in_seq=0 out_seq=0 2024-03-19T02:48:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3c37fe640 1 -- 172.21.15.116:0/3521258383 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc782ae6a0 con 0x55cc7835b2a0 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3c37fe640 10 client.21447 handle_command_reply: tid=0 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 client.21447 shutdown 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 msgr2=0x55cc78343800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8003870 tx=0x7fc3d8003d30 comp rx=0 tx=0).stop 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 shutdown_connections 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc7835b2a0 0x55cc7835d760 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc78346880 0x55cc78344280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc78343d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3521258383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc7833d380 0x55cc78343800 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3c37fe640 0 client.21447 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 >> 172.21.15.116:0/3521258383 conn(0x55cc78337ab0 msgr2=0x55cc78339800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 shutdown_connections 2024-03-19T02:48:28.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 1 -- 172.21.15.116:0/3521258383 wait complete. 2024-03-19T02:48:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.898+0000 7fc3ed194740 20 client.21447 trim_cache size 0 max 16384 2024-03-19T02:48:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 Processor -- start 2024-03-19T02:48:28.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 -- start start 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833d380 0x55cc78356820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78363070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ac920 con 0x55cc78346880 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ae4c0 con 0x55cc7833d380 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3ed194740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41acff0 con 0x55cc7833dca0 2024-03-19T02:48:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51710/0 (socket says 172.21.15.116:51710) 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 -- 172.21.15.116:0/3665793616 learned_addr learned my addr 172.21.15.116:0/3665793616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78363070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833d380 0x55cc78356820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 -- 172.21.15.116:0/3665793616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833d380 msgr2=0x55cc78356820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833d380 0x55cc78356820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 -- 172.21.15.116:0/3665793616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 msgr2=0x55cc78363070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.921+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78363070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3e3fff640 1 -- 172.21.15.116:0/3665793616 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc78338840 con 0x55cc7833dca0 2024-03-19T02:48:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78363070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009b40 tx=0x7fc3d4005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a85976ea71c421 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3c9ffb640 1 -- 172.21.15.116:0/3665793616 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d401f040 con 0x55cc7833dca0 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3c9ffb640 1 -- 172.21.15.116:0/3665793616 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3d4003b90 con 0x55cc7833dca0 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.922+0000 7fc3c9ffb640 1 -- 172.21.15.116:0/3665793616 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4025450 con 0x55cc7833dca0 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- 172.21.15.116:0/3665793616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 msgr2=0x55cc783440a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009b40 tx=0x7fc3d4005d30 comp rx=0 tx=0).stop 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- 172.21.15.116:0/3665793616 shutdown_connections 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc78363070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833dca0 0x55cc783440a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 --2- 172.21.15.116:0/3665793616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833d380 0x55cc78356820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- 172.21.15.116:0/3665793616 >> 172.21.15.116:0/3665793616 conn(0x55cc78337ab0 msgr2=0x55cc78336b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- 172.21.15.116:0/3665793616 shutdown_connections 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- 172.21.15.116:0/3665793616 wait complete. 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 -- start start 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.923+0000 7fc3ed194740 1 Processor -- start 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc78353040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78340090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41addf0 con 0x55cc78346880 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ad6c0 con 0x55cc7833dca0 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41aa920 con 0x55cc7833d380 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3ed194740 10 client.0 _ll_register_callbacks cb 0x7fc3d8009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56410/0 (socket says 172.21.15.116:56410) 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78340090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 -- 172.21.15.116:0/4260813885 learned_addr learned my addr 172.21.15.116:0/4260813885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc78353040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 -- 172.21.15.116:0/4260813885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 msgr2=0x55cc78353040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc78353040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 -- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 msgr2=0x55cc78340090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78340090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 -- 172.21.15.116:0/4260813885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc78340c40 con 0x55cc78346880 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78340090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3e37fe640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x55cc78341190 tx=0x7fc3d8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa740951c66425f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.924+0000 7fc3c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d802f860 con 0x55cc78346880 2024-03-19T02:48:28.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d4003b90 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc3d802fa00 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc3d8023030 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 10 client.21453 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cc783c1590 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3d4003b90 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc3d803b030 con 0x55cc78346880 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 10 client.21453 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 10 client.21453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 20 client.21453 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc783c1c50 0x55cc783c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 4 client.21453 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.925+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cc782a89f0 con 0x55cc783c1c50 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.926+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc783c1c50 0x55cc783c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.926+0000 7fc3e3fff640 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc783c1c50 0x55cc783c4040 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd1432f29dec3472 server_cookie=6f110c8ad45a7b5a in_seq=0 out_seq=0 2024-03-19T02:48:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.926+0000 7fc3c37fe640 10 client.21453 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3c37fe640 1 -- 172.21.15.116:0/4260813885 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cc782a89f0 con 0x55cc783c1c50 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3c37fe640 10 client.21453 handle_command_reply: tid=0 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 client.21453 shutdown 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 msgr2=0x55cc783405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x55cc78341190 tx=0x7fc3d8013000 comp rx=0 tx=0).stop 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 shutdown_connections 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55cc783c1c50 0x55cc783c4040 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cc78346880 0x55cc783405d0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3c37fe640 0 client.21453 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc7833dca0 0x55cc78340090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 --2- 172.21.15.116:0/4260813885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc7833d380 0x55cc78353040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 >> 172.21.15.116:0/4260813885 conn(0x55cc78337ab0 msgr2=0x55cc78338ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.929+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 shutdown_connections 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ed194740 1 -- 172.21.15.116:0/4260813885 wait complete. 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ed194740 20 client.21453 trim_cache size 0 max 16384 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:48:28.938 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "f94a8215-1c5b-4d5c-8c3b-767f5cadeb82", 2024-03-19T02:48:28.939 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:48:28.939 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:48:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 msgr2=0x7fc3cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 0x7fc3cc0a4e30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ee7e0 tx=0x7fc3dc0af040 comp rx=0 tx=0).stop 2024-03-19T02:48:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 msgr2=0x7fc3e41a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc02c790 tx=0x7fc3dc035db0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 shutdown_connections 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3cc0a2970 0x7fc3cc0a4e30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3e41055c0 0x7fc3e41a3e40 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e4101080 0x7fc3e4199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 --2- 172.21.15.116:0/4243748766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3e41006e0 0x7fc3e4198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 >> 172.21.15.116:0/4243748766 conn(0x7fc3e40777b0 msgr2=0x7fc3e40fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 shutdown_connections 2024-03-19T02:48:28.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:28.930+0000 7fc3ea5a9640 1 -- 172.21.15.116:0/4243748766 wait complete. 2024-03-19T02:48:28.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:48:28.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:48:29.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 -- 172.21.15.116:0/23315539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25701066a0 msgr2=0x7f2570106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 --2- 172.21.15.116:0/23315539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25701066a0 0x7f2570106a80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f256c009930 tx=0x7f256c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 -- 172.21.15.116:0/23315539 shutdown_connections 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 --2- 172.21.15.116:0/23315539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 0x7f257010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 --2- 172.21.15.116:0/23315539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2570100690 0x7f2570100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 --2- 172.21.15.116:0/23315539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25701066a0 0x7f2570106a80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 -- 172.21.15.116:0/23315539 >> 172.21.15.116:0/23315539 conn(0x7f2570077820 msgr2=0x7f2570077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 -- 172.21.15.116:0/23315539 shutdown_connections 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 -- 172.21.15.116:0/23315539 wait complete. 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.066+0000 7f257826e640 1 Processor -- start 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 -- start start 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 0x7f2570197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 0x7f257019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2570110920 con 0x7f2570100690 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25701107a0 con 0x7f2570101050 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257826e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2570110aa0 con 0x7f25701066a0 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257600a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257600a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56426/0 (socket says 172.21.15.116:56426) 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257600a640 1 -- 172.21.15.116:0/3441644361 learned_addr learned my addr 172.21.15.116:0/3441644361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 0x7f257019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f2575809640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 0x7f2570197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 -- 172.21.15.116:0/3441644361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 msgr2=0x7f2570197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 0x7f2570197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 -- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 msgr2=0x7f2570196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257680b640 1 -- 172.21.15.116:0/3441644361 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019beb0 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.067+0000 7f257600a640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 0x7f257019b780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2568009fd0 tx=0x7f256800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f25677fe640 1 -- 172.21.15.116:0/3441644361 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800ce20 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019c1a0 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f25677fe640 1 -- 172.21.15.116:0/3441644361 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2568005020 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f25677fe640 1 -- 172.21.15.116:0/3441644361 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568010690 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.068+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25701a3a30 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 Processor -- start 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f25677fe640 1 -- 172.21.15.116:0/3441644361 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f25680051d0 con 0x7f25701066a0 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 -- start start 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 0x55ad836009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad83600f30 0x55ad83603320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ad8355b500 con 0x55ad835f7a30 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ad83503be0 con 0x55ad835f8350 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257ae56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ad835575a0 con 0x55ad83600f30 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.071+0000 7f257600a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 0x55ad836009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56436/0 (socket says 172.21.15.116:56436) 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f2575809640 1 -- 172.21.15.116:0/1114658058 learned_addr learned my addr 172.21.15.116:0/1114658058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f2575809640 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad83600f30 0x55ad83603320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f25677fe640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 0x7f255c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 -- 172.21.15.116:0/1114658058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad83600f30 msgr2=0x55ad83603320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad83600f30 0x55ad83603320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 -- 172.21.15.116:0/1114658058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 msgr2=0x55ad835f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 -- 172.21.15.116:0/1114658058 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad83525000 con 0x55ad835f8350 2024-03-19T02:48:29.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 0x7f255c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f2575809640 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f25677fe640 1 -- 172.21.15.116:0/3441644361 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f25680cf880 con 0x7f25701066a0 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257600a640 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 0x55ad836009f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f256c009900 tx=0x7f256c009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ddcf0a0310116c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f257680b640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 0x7f255c0a4e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f256809ba90 tx=0x7f2568071040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.072+0000 7f2564bfe640 1 -- 172.21.15.116:0/1114658058 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c045070 con 0x55ad835f8350 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f2564bfe640 1 -- 172.21.15.116:0/1114658058 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256c02c7e0 con 0x55ad835f8350 2024-03-19T02:48:29.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f2564bfe640 1 -- 172.21.15.116:0/1114658058 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c003a20 con 0x55ad835f8350 2024-03-19T02:48:29.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 -- 172.21.15.116:0/1114658058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 msgr2=0x55ad836009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 0x55ad836009f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f256c009900 tx=0x7f256c009670 comp rx=0 tx=0).stop 2024-03-19T02:48:29.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 -- 172.21.15.116:0/1114658058 shutdown_connections 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad83600f30 0x55ad83603320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f8350 0x55ad836009f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 --2- 172.21.15.116:0/1114658058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.073+0000 7f257ae56740 1 -- 172.21.15.116:0/1114658058 >> 172.21.15.116:0/1114658058 conn(0x55ad835f2160 msgr2=0x55ad835f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- 172.21.15.116:0/1114658058 shutdown_connections 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- 172.21.15.116:0/1114658058 wait complete. 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- start start 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 Processor -- start 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad835fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ad83603cc0 con 0x55ad835f7a30 2024-03-19T02:48:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ad83517160 con 0x55ad83600f30 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ad83603e40 con 0x55ad835f8350 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257ae56740 10 client.0 _ll_register_callbacks cb 0x55ad83553a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56448/0 (socket says 172.21.15.116:56448) 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f2575809640 1 -- 172.21.15.116:0/1686985991 learned_addr learned my addr 172.21.15.116:0/1686985991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.074+0000 7f257600a640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad835fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 -- 172.21.15.116:0/1686985991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 msgr2=0x55ad835fe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 -- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 msgr2=0x55ad835fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad835fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 -- 172.21.15.116:0/1686985991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad833397d0 con 0x55ad835f7a30 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f257600a640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:29.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2575809640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f25600135c0 tx=0x7f2560005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e5288d3ffe7d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560011da0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f25600047a0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256c02c7e0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f25600049a0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f257ae56740 10 client.21465 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.075+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ad8326f3f0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256c02c7e0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f25600214f0 con 0x55ad835f7a30 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 10 client.21465 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 10 client.21465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 20 client.21465 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83611590 0x55ad83613a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 4 client.21465 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad83568a40 con 0x55ad83611590 2024-03-19T02:48:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.076+0000 7f257600a640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83611590 0x55ad83613a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.077+0000 7f2559ffb640 10 client.21465 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.077+0000 7f257600a640 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83611590 0x55ad83613a10 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e2bb602d6d4805 server_cookie=76f55fb3f2e8884c in_seq=0 out_seq=0 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f2559ffb640 1 -- 172.21.15.116:0/1686985991 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ad83568a40 con 0x55ad83611590 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f2559ffb640 10 client.21465 handle_command_reply: tid=0 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 client.21465 shutdown 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 msgr2=0x55ad835fde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f25600135c0 tx=0x7f2560005f90 comp rx=0 tx=0).stop 2024-03-19T02:48:29.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 shutdown_connections 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83611590 0x55ad83613a10 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad835fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 --2- 172.21.15.116:0/1686985991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f7a30 0x55ad835fde30 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f2559ffb640 0 client.21465 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 >> 172.21.15.116:0/1686985991 conn(0x55ad835f2160 msgr2=0x55ad835f3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 shutdown_connections 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 1 -- 172.21.15.116:0/1686985991 wait complete. 2024-03-19T02:48:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.082+0000 7f257ae56740 20 client.21465 trim_cache size 0 max 16384 2024-03-19T02:48:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 Processor -- start 2024-03-19T02:48:29.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 -- start start 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f7a30 0x55ad83600560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ac9c0 con 0x55ad83600f30 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ae560 con 0x55ad835f7a30 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257ae56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ad090 con 0x55ad835f8350 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f2575809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56456/0 (socket says 172.21.15.116:56456) 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f2575809640 1 -- 172.21.15.116:0/4107204416 learned_addr learned my addr 172.21.15.116:0/4107204416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f2575809640 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f7a30 0x55ad83600560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.119+0000 7f257600a640 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 -- 172.21.15.116:0/4107204416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 msgr2=0x55ad835fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 -- 172.21.15.116:0/4107204416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f7a30 msgr2=0x55ad83600560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f7a30 0x55ad83600560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 -- 172.21.15.116:0/4107204416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8322f0f0 con 0x55ad83600f30 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f2575809640 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f25600139f0 tx=0x7f25600097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a346c378e87ddf00 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f255bfff640 1 -- 172.21.15.116:0/4107204416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560011da0 con 0x55ad83600f30 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f255bfff640 1 -- 172.21.15.116:0/4107204416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25600046c0 con 0x55ad83600f30 2024-03-19T02:48:29.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.120+0000 7f255bfff640 1 -- 172.21.15.116:0/4107204416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560004420 con 0x55ad83600f30 2024-03-19T02:48:29.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- 172.21.15.116:0/4107204416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 msgr2=0x55ad8360ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f25600139f0 tx=0x7f25600097a0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- 172.21.15.116:0/4107204416 shutdown_connections 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad83600f30 0x55ad8360ca60 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f8350 0x55ad835fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 --2- 172.21.15.116:0/4107204416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad835f7a30 0x55ad83600560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- 172.21.15.116:0/4107204416 >> 172.21.15.116:0/4107204416 conn(0x55ad835f2160 msgr2=0x55ad835f3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- 172.21.15.116:0/4107204416 shutdown_connections 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- 172.21.15.116:0/4107204416 wait complete. 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.121+0000 7f257ae56740 1 -- start start 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 Processor -- start 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 0x55ad83604020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad83604aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ade90 con 0x55ad835f8350 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ad760 con 0x55ad83600f30 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25701aa9c0 con 0x55ad835f7a30 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257ae56740 10 client.0 _ll_register_callbacks cb 0x7f2560009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257600a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 0x55ad83604020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257600a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56458/0 (socket says 172.21.15.116:56458) 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257600a640 1 -- 172.21.15.116:0/3829620538 learned_addr learned my addr 172.21.15.116:0/3829620538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad83604aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 -- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 msgr2=0x55ad83604aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad83604aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 -- 172.21.15.116:0/3829620538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 msgr2=0x55ad83604560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f2575809640 1 -- 172.21.15.116:0/3829620538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad83675aa0 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.122+0000 7f257600a640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2575809640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 0x55ad83604020 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2560021930 tx=0x7f2560013a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a80450a2c6a6c80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560004030 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25600046c0 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f25600041d0 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f25600043d0 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 10 client.31171 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ad83676d60 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25600046c0 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f256003f040 con 0x55ad835f7a30 2024-03-19T02:48:29.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 10 client.31171 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 10 client.31171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 20 client.31171 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83677460 0x55ad83679880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 4 client.31171 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.123+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ad83562d90 con 0x55ad83677460 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.124+0000 7f257600a640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83677460 0x55ad83679880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.124+0000 7f2559ffb640 10 client.31171 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:29.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.124+0000 7f257600a640 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83677460 0x55ad83679880 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81951fa5444ee528 server_cookie=67a4faeae2187b7d in_seq=0 out_seq=0 2024-03-19T02:48:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f2559ffb640 1 -- 172.21.15.116:0/3829620538 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55ad83562d90 con 0x55ad83677460 2024-03-19T02:48:29.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f2559ffb640 10 client.31171 handle_command_reply: tid=0 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 client.31171 shutdown 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 msgr2=0x55ad83604020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 0x55ad83604020 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2560021930 tx=0x7f2560013a20 comp rx=0 tx=0).stop 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 shutdown_connections 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ad83677460 0x55ad83679880 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad83600f30 0x55ad83604aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ad835f8350 0x55ad83604560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f2559ffb640 0 client.31171 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 --2- 172.21.15.116:0/3829620538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ad835f7a30 0x55ad83604020 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 >> 172.21.15.116:0/3829620538 conn(0x55ad835f2160 msgr2=0x55ad835f3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.126+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 shutdown_connections 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257ae56740 1 -- 172.21.15.116:0/3829620538 wait complete. 2024-03-19T02:48:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257ae56740 20 client.31171 trim_cache size 0 max 16384 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (539 inodes in the stack)", 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "f94a8215-1c5b-4d5c-8c3b-767f5cadeb82": { 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "f94a8215-1c5b-4d5c-8c3b-767f5cadeb82", 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:48:29.133 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:48:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 msgr2=0x7f255c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 0x7f255c0a4e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f256809ba90 tx=0x7f2568071040 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 msgr2=0x7f257019b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 0x7f257019b780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2568009fd0 tx=0x7f256800ef10 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 shutdown_connections 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f255c0a2970 0x7f255c0a4e30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25701066a0 0x7f257019b780 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2570101050 0x7f2570197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 --2- 172.21.15.116:0/3441644361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2570100690 0x7f2570196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 >> 172.21.15.116:0/3441644361 conn(0x7f2570077820 msgr2=0x7f2570102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 shutdown_connections 2024-03-19T02:48:29.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:29.127+0000 7f257826e640 1 -- 172.21.15.116:0/3441644361 wait complete. 2024-03-19T02:48:29.142 INFO:tasks.cephfs.filesystem:scrub status for tag:f94a8215-1c5b-4d5c-8c3b-767f5cadeb82 - {'path': '/', 'tag': 'f94a8215-1c5b-4d5c-8c3b-767f5cadeb82', 'options': 'recursive,force'} 2024-03-19T02:48:29.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52771 5.97 MB/sec execute 351 sec latency 230.316 ms 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:29.859 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:48:30.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:30.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.170 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:29 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:48:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:48:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:29 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi162 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:29 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:30.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52849 5.97 MB/sec execute 352 sec latency 161.465 ms 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: pgmap v653: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:30 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:48:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: pgmap v653: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:30 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: pgmap v653: 129 pgs: 129 active+clean; 1014 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 121 op/s 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:30 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:31.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 52940 5.98 MB/sec execute 353 sec latency 197.183 ms 2024-03-19T02:48:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:48:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:48:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:32.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:32.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:48:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:32.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:48:32.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53037 5.98 MB/sec execute 354 sec latency 398.776 ms 2024-03-19T02:48:33.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:32 smithi162 ceph-mon[35761]: pgmap v654: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 221 op/s 2024-03-19T02:48:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:32 smithi162 ceph-mon[35761]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:48:33.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:32 smithi162 ceph-mon[35761]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:48:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:32 smithi154 ceph-mon[35693]: pgmap v654: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 221 op/s 2024-03-19T02:48:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:32 smithi154 ceph-mon[35693]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:48:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:32 smithi154 ceph-mon[35693]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:48:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:32 smithi116 ceph-mon[32263]: pgmap v654: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.3 MiB/s wr, 221 op/s 2024-03-19T02:48:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:32 smithi116 ceph-mon[32263]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-03-19T02:48:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:32 smithi116 ceph-mon[32263]: Unable to set osd_memory_target on smithi154 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-03-19T02:48:33.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53139 5.99 MB/sec execute 355 sec latency 1398.802 ms 2024-03-19T02:48:34.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53223 5.99 MB/sec execute 356 sec latency 2398.832 ms 2024-03-19T02:48:35.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:34 smithi162 ceph-mon[35761]: pgmap v655: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-03-19T02:48:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:34 smithi154 ceph-mon[35693]: pgmap v655: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-03-19T02:48:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:34 smithi116 ceph-mon[32263]: pgmap v655: 129 pgs: 129 active+clean; 1022 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 160 op/s 2024-03-19T02:48:35.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53290 5.98 MB/sec execute 357 sec latency 3398.858 ms 2024-03-19T02:48:36.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53357 5.96 MB/sec execute 358 sec latency 3931.325 ms 2024-03-19T02:48:37.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:36 smithi162 ceph-mon[35761]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 216 op/s 2024-03-19T02:48:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:36 smithi154 ceph-mon[35693]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 216 op/s 2024-03-19T02:48:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:36 smithi116 ceph-mon[32263]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 216 op/s 2024-03-19T02:48:37.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53411 5.95 MB/sec execute 359 sec latency 1014.063 ms 2024-03-19T02:48:38.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53491 5.94 MB/sec execute 360 sec latency 2014.093 ms 2024-03-19T02:48:39.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:38 smithi162 ceph-mon[35761]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:38 smithi154 ceph-mon[35693]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:38 smithi116 ceph-mon[32263]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:39.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53559 5.93 MB/sec execute 361 sec latency 3014.124 ms 2024-03-19T02:48:40.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:39 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:39 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:39 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:40.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53638 5.91 MB/sec execute 362 sec latency 4014.154 ms 2024-03-19T02:48:41.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:40 smithi116 ceph-mon[32263]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:41.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:41.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:41.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:41.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:40 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:48:41.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:40 smithi162 ceph-mon[35761]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:41.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:41.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:41.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:41.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:40 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:48:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:40 smithi154 ceph-mon[35693]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:41.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:40 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:48:41.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53730 5.91 MB/sec execute 363 sec latency 4537.092 ms 2024-03-19T02:48:42.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53832 5.90 MB/sec execute 364 sec latency 182.873 ms 2024-03-19T02:48:43.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:42 smithi162 ceph-mon[35761]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 211 op/s 2024-03-19T02:48:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:42 smithi154 ceph-mon[35693]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 211 op/s 2024-03-19T02:48:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:42 smithi116 ceph-mon[32263]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 211 op/s 2024-03-19T02:48:43.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 53939 5.89 MB/sec execute 365 sec latency 199.783 ms 2024-03-19T02:48:44.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54058 5.90 MB/sec execute 366 sec latency 140.574 ms 2024-03-19T02:48:45.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:44 smithi162 ceph-mon[35761]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 111 op/s 2024-03-19T02:48:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:44 smithi154 ceph-mon[35693]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 111 op/s 2024-03-19T02:48:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:44 smithi116 ceph-mon[32263]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 111 op/s 2024-03-19T02:48:45.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54172 5.90 MB/sec execute 367 sec latency 143.841 ms 2024-03-19T02:48:46.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54280 5.89 MB/sec execute 368 sec latency 144.445 ms 2024-03-19T02:48:47.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:46 smithi162 ceph-mon[35761]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 210 op/s 2024-03-19T02:48:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:46 smithi154 ceph-mon[35693]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 210 op/s 2024-03-19T02:48:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:46 smithi116 ceph-mon[32263]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 210 op/s 2024-03-19T02:48:47.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54410 5.90 MB/sec execute 369 sec latency 176.584 ms 2024-03-19T02:48:48.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54525 5.90 MB/sec execute 370 sec latency 151.871 ms 2024-03-19T02:48:49.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:48 smithi162 ceph-mon[35761]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:48 smithi154 ceph-mon[35693]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:48 smithi116 ceph-mon[32263]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:49.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54652 5.92 MB/sec execute 371 sec latency 140.079 ms 2024-03-19T02:48:50.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54772 5.91 MB/sec execute 372 sec latency 137.736 ms 2024-03-19T02:48:51.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:50 smithi162 ceph-mon[35761]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:50 smithi154 ceph-mon[35693]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:50 smithi116 ceph-mon[32263]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 155 op/s 2024-03-19T02:48:51.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54867 5.90 MB/sec execute 373 sec latency 138.449 ms 2024-03-19T02:48:52.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 54973 5.89 MB/sec execute 374 sec latency 852.426 ms 2024-03-19T02:48:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:52 smithi162 ceph-mon[35761]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-03-19T02:48:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:52 smithi154 ceph-mon[35693]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-03-19T02:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:52 smithi116 ceph-mon[32263]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 228 op/s 2024-03-19T02:48:53.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55091 5.89 MB/sec execute 375 sec latency 1852.453 ms 2024-03-19T02:48:54.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55201 5.88 MB/sec execute 376 sec latency 2852.478 ms 2024-03-19T02:48:55.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:54 smithi162 ceph-mon[35761]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 171 op/s 2024-03-19T02:48:55.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:54 smithi154 ceph-mon[35693]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 171 op/s 2024-03-19T02:48:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:54 smithi116 ceph-mon[32263]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 171 op/s 2024-03-19T02:48:55.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55305 5.87 MB/sec execute 377 sec latency 3852.511 ms 2024-03-19T02:48:56.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55412 5.87 MB/sec execute 378 sec latency 4364.053 ms 2024-03-19T02:48:57.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:56 smithi162 ceph-mon[35761]: pgmap v666: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-03-19T02:48:57.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:56 smithi154 ceph-mon[35693]: pgmap v666: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-03-19T02:48:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:56 smithi116 ceph-mon[32263]: pgmap v666: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-03-19T02:48:57.334 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55496 5.87 MB/sec execute 379 sec latency 1235.772 ms 2024-03-19T02:48:58.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55561 5.87 MB/sec execute 380 sec latency 2235.804 ms 2024-03-19T02:48:59.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:48:59.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:48:59.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:58 smithi162 ceph-mon[35761]: pgmap v667: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:48:59.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:58 smithi116 ceph-mon[32263]: pgmap v667: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:48:59.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:58 smithi154 ceph-mon[35693]: pgmap v667: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 -- 172.21.15.116:0/1716974352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c1066a0 msgr2=0x7f3e4c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 --2- 172.21.15.116:0/1716974352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c106a80 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009930 tx=0x7f3e3c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 -- 172.21.15.116:0/1716974352 shutdown_connections 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 --2- 172.21.15.116:0/1716974352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c101050 0x7f3e4c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 --2- 172.21.15.116:0/1716974352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c100690 0x7f3e4c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 --2- 172.21.15.116:0/1716974352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c106a80 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 -- 172.21.15.116:0/1716974352 >> 172.21.15.116:0/1716974352 conn(0x7f3e4c077820 msgr2=0x7f3e4c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 -- 172.21.15.116:0/1716974352 shutdown_connections 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.273+0000 7f3e5241f640 1 -- 172.21.15.116:0/1716974352 wait complete. 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 Processor -- start 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 -- start start 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c100690 0x7f3e4c198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c1a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c110960 con 0x7f3e4c101050 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1107e0 con 0x7f3e4c100690 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e5241f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c110ae0 con 0x7f3e4c1066a0 2024-03-19T02:48:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c100690 0x7f3e4c198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51606/0 (socket says 172.21.15.116:51606) 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e509bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c1a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/2382608542 learned_addr learned my addr 172.21.15.116:0/2382608542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/2382608542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 msgr2=0x7f3e4c1a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c1a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/2382608542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c100690 msgr2=0x7f3e4c198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.274+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c100690 0x7f3e4c198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/2382608542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c1a4440 con 0x7f3e4c101050 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e509bc640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9f0 tx=0x7f3e4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e497fa640 1 -- 172.21.15.116:0/2382608542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4000cd80 con 0x7f3e4c101050 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c1a4610 con 0x7f3e4c101050 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e497fa640 1 -- 172.21.15.116:0/2382608542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e400054d0 con 0x7f3e4c101050 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e497fa640 1 -- 172.21.15.116:0/2382608542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40010490 con 0x7f3e4c101050 2024-03-19T02:48:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.275+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c1a4b30 con 0x7f3e4c101050 2024-03-19T02:48:59.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.277+0000 7f3e497fa640 1 -- 172.21.15.116:0/2382608542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3e40005020 con 0x7f3e4c101050 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.277+0000 7f3e55005740 1 Processor -- start 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 -- start start 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e497fa640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 0x7f3e340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 0x55ebdda80e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda89f30 0x55ebdda8c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd9e4500 con 0x55ebdda89f30 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd98cbe0 con 0x55ebdda81350 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e55005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd9e05a0 con 0x55ebdda80a30 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e497fa640 1 -- 172.21.15.116:0/2382608542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f3e400cf5a0 con 0x7f3e4c101050 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e509bc640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 0x7f3e340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 0x55ebdda80e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45076/0 (socket says 172.21.15.116:45076) 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/338843356 learned_addr learned my addr 172.21.15.116:0/338843356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e509bc640 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda89f30 0x55ebdda8c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/338843356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 msgr2=0x55ebdda80e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 0x55ebdda80e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/338843356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda89f30 msgr2=0x55ebdda8c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda89f30 0x55ebdda8c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/338843356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdd9ae000 con 0x55ebdda81350 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 0x55ebdda80e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e509bc640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 0x7f3e340a4e30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009fd0 tx=0x7f3e440116b0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.278+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3e400a3a90 tx=0x7f3e4009b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd8b92533668337 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e327fc640 1 -- 172.21.15.116:0/338843356 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4009bf20 con 0x55ebdda81350 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e327fc640 1 -- 172.21.15.116:0/338843356 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e4009b820 con 0x55ebdda81350 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e327fc640 1 -- 172.21.15.116:0/338843356 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400ccb90 con 0x55ebdda81350 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 -- 172.21.15.116:0/338843356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 msgr2=0x55ebdda899f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3e400a3a90 tx=0x7f3e4009b420 comp rx=0 tx=0).stop 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 -- 172.21.15.116:0/338843356 shutdown_connections 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda89f30 0x55ebdda8c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda899f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 --2- 172.21.15.116:0/338843356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda80a30 0x55ebdda80e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 -- 172.21.15.116:0/338843356 >> 172.21.15.116:0/338843356 conn(0x55ebdda7b160 msgr2=0x55ebdda7b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 -- 172.21.15.116:0/338843356 shutdown_connections 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.279+0000 7f3e55005740 1 -- 172.21.15.116:0/338843356 wait complete. 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 -- start start 2024-03-19T02:48:59.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 Processor -- start 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda80a30 0x55ebdda86fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebdda87a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebdda8cd20 con 0x55ebdda81350 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd9a0160 con 0x55ebdda80a30 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ebdda8cea0 con 0x55ebdda89f30 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e55005740 10 client.0 _ll_register_callbacks cb 0x55ebdd9dca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e509bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebdda87a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51632/0 (socket says 172.21.15.116:51632) 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e4b7fe640 1 -- 172.21.15.116:0/3067510802 learned_addr learned my addr 172.21.15.116:0/3067510802 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.280+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda80a30 0x55ebdda86fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 -- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda80a30 msgr2=0x55ebdda86fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda80a30 0x55ebdda86fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 -- 172.21.15.116:0/3067510802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 msgr2=0x55ebdda87500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 -- 172.21.15.116:0/3067510802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdd7c27d0 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e509bc640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebdda87a40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55ebdda7c2f0 tx=0x7f3e4401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c07cd5db859ec95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44024070 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e4009b820 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f3e44020ad0 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3e4401ee40 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e55005740 10 client.31183 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.281+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ebdd6f83f0 con 0x55ebdda89f30 2024-03-19T02:48:59.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e4009b820 con 0x55ebdda89f30 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f3e4402ab50 con 0x55ebdda89f30 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 10 client.31183 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 10 client.31183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 20 client.31183 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebdda9a300 0x55ebdda9c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 4 client.31183 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebdd9f1a40 con 0x55ebdda9a300 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebdda9a300 0x55ebdda9c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e2b7fe640 10 client.31183 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.282+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebdda9a300 0x55ebdda9c6f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95da2315f104cb9d server_cookie=b1873b5304756248 in_seq=0 out_seq=0 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/3067510802 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ebdd9f1a40 con 0x55ebdda9a300 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e2b7fe640 10 client.31183 handle_command_reply: tid=0 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 client.31183 shutdown 2024-03-19T02:48:59.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 msgr2=0x55ebdda87a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebdda87a40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55ebdda7c2f0 tx=0x7f3e4401d000 comp rx=0 tx=0).stop 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 shutdown_connections 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebdda9a300 0x55ebdda9c6f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebdda87a40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e2b7fe640 0 client.31183 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda81350 0x55ebdda87500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 --2- 172.21.15.116:0/3067510802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda80a30 0x55ebdda86fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 >> 172.21.15.116:0/3067510802 conn(0x55ebdda7b160 msgr2=0x55ebdda7ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 shutdown_connections 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 1 -- 172.21.15.116:0/3067510802 wait complete. 2024-03-19T02:48:59.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.284+0000 7f3e55005740 20 client.31183 trim_cache size 0 max 16384 2024-03-19T02:48:59.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.304+0000 7f3e55005740 1 Processor -- start 2024-03-19T02:48:59.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.304+0000 7f3e55005740 1 -- start start 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda95ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebddaa1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1aec80 con 0x55ebdda80a30 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1b0820 con 0x55ebdda81350 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e55005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1af350 con 0x55ebdda89f30 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51634/0 (socket says 172.21.15.116:51634) 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 -- 172.21.15.116:0/651718914 learned_addr learned my addr 172.21.15.116:0/651718914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e509bc640 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebddaa1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda95ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 -- 172.21.15.116:0/651718914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 msgr2=0x55ebddaa1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebddaa1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 -- 172.21.15.116:0/651718914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 msgr2=0x55ebdda95ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda95ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 -- 172.21.15.116:0/651718914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdda81180 con 0x55ebdda80a30 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009930 tx=0x7f3e3c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b7e0f2b7447804 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e31ffb640 1 -- 172.21.15.116:0/651718914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c004280 con 0x55ebdda80a30 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e31ffb640 1 -- 172.21.15.116:0/651718914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e3c02cc90 con 0x55ebdda80a30 2024-03-19T02:48:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.305+0000 7f3e31ffb640 1 -- 172.21.15.116:0/651718914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0046c0 con 0x55ebdda80a30 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- 172.21.15.116:0/651718914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 msgr2=0x55ebdda7a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009930 tx=0x7f3e3c005d50 comp rx=0 tx=0).stop 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- 172.21.15.116:0/651718914 shutdown_connections 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda89f30 0x55ebddaa1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda81350 0x55ebdda95ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 --2- 172.21.15.116:0/651718914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda7a1e0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- 172.21.15.116:0/651718914 >> 172.21.15.116:0/651718914 conn(0x55ebdda7b160 msgr2=0x55ebdda86f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- 172.21.15.116:0/651718914 shutdown_connections 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- 172.21.15.116:0/651718914 wait complete. 2024-03-19T02:48:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.306+0000 7f3e55005740 1 -- start start 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 Processor -- start 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 0x55ebdda848f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda89f30 0x55ebddafd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1b0150 con 0x55ebdda80a30 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1afa20 con 0x55ebdda89f30 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c1acc80 con 0x55ebdda81350 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e55005740 10 client.0 _ll_register_callbacks cb 0x7f3e44011860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51646/0 (socket says 172.21.15.116:51646) 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 -- 172.21.15.116:0/414720439 learned_addr learned my addr 172.21.15.116:0/414720439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 0x55ebdda848f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e509bc640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda89f30 0x55ebddafd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 -- 172.21.15.116:0/414720439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 msgr2=0x55ebdda848f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 0x55ebdda848f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 -- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda89f30 msgr2=0x55ebddafd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda89f30 0x55ebddafd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4bfff640 1 -- 172.21.15.116:0/414720439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdd6b80f0 con 0x55ebdda80a30 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.307+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 0x55ebdda848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e4bfff640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c02c170 tx=0x7f3e3c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608a0f920de1e179 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c056020 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f3e3c005440 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e3c02cc90 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f3e3c061050 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 10 client.21489 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ebddb05150 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e3c02cc90 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f3e3c004c20 con 0x55ebdda80a30 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 10 client.21489 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 10 client.21489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:48:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 20 client.21489 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebddb05870 0x55ebddb07c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 4 client.21489 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.308+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebdd9ebd90 con 0x55ebddb05870 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.309+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebddb05870 0x55ebddb07c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.309+0000 7f3e4b7fe640 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebddb05870 0x55ebddb07c90 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0746b9d14e607e server_cookie=a8d4ec4dc27ab145 in_seq=0 out_seq=0 2024-03-19T02:48:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.309+0000 7f3e2b7fe640 10 client.21489 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e2b7fe640 1 -- 172.21.15.116:0/414720439 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ebdd9ebd90 con 0x55ebddb05870 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e2b7fe640 10 client.21489 handle_command_reply: tid=0 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 client.21489 shutdown 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 msgr2=0x55ebdda843b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c02c170 tx=0x7f3e3c005040 comp rx=0 tx=0).stop 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 shutdown_connections 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55ebddb05870 0x55ebddb07c90 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebdda89f30 0x55ebddafd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e2b7fe640 0 client.21489 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ebdda81350 0x55ebdda848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 --2- 172.21.15.116:0/414720439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebdda80a30 0x55ebdda843b0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 >> 172.21.15.116:0/414720439 conn(0x55ebdda7b160 msgr2=0x55ebdda87810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 shutdown_connections 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 1 -- 172.21.15.116:0/414720439 wait complete. 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.310+0000 7f3e55005740 20 client.21489 trim_cache size 0 max 16384 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:48:59.320 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:48:59.321 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:48:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 msgr2=0x7f3e340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 0x7f3e340a4e30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009fd0 tx=0x7f3e440116b0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 msgr2=0x7f3e4c1993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9f0 tx=0x7f3e4000edf0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 shutdown_connections 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3e340a2970 0x7f3e340a4e30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e4c1066a0 0x7f3e4c1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e4c101050 0x7f3e4c1993c0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 --2- 172.21.15.116:0/2382608542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e4c100690 0x7f3e4c198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 >> 172.21.15.116:0/2382608542 conn(0x7f3e4c077820 msgr2=0x7f3e4c103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 shutdown_connections 2024-03-19T02:48:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.311+0000 7f3e5241f640 1 -- 172.21.15.116:0/2382608542 wait complete. 2024-03-19T02:48:59.323 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:48:59.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:48:59.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55660 5.87 MB/sec execute 381 sec latency 3235.831 ms 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- 172.21.15.116:0/3258680866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 msgr2=0x7f7ed8106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/3258680866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed8106b10 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009930 tx=0x7f7ec802bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- 172.21.15.116:0/3258680866 shutdown_connections 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/3258680866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 0x7f7ed810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/3258680866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 0x7f7ed8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/3258680866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed8106b10 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- 172.21.15.116:0/3258680866 >> 172.21.15.116:0/3258680866 conn(0x7f7ed8077820 msgr2=0x7f7ed8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- 172.21.15.116:0/3258680866 shutdown_connections 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- 172.21.15.116:0/3258680866 wait complete. 2024-03-19T02:48:59.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 Processor -- start 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.446+0000 7f7edcbe8640 1 -- start start 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 0x7f7ed8198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 0x7f7ed81993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed81109b0 con 0x7f7ed8106730 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed8110830 con 0x7f7ed8100720 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7edcbe8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed8110b30 con 0x7f7ed81010e0 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51666/0 (socket says 172.21.15.116:51666) 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 -- 172.21.15.116:0/801143170 learned_addr learned my addr 172.21.15.116:0/801143170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ecdd9b640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 0x7f7ed81993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed659c640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 0x7f7ed8198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 -- 172.21.15.116:0/801143170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 msgr2=0x7f7ed81993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 0x7f7ed81993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 -- 172.21.15.116:0/801143170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 msgr2=0x7f7ed8198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 0x7f7ed8198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 -- 172.21.15.116:0/801143170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed81a4470 con 0x7f7ed8106730 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.447+0000 7f7ed6d9d640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009fd0 tx=0x7f7ed000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.448+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0017070 con 0x7f7ed8106730 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.448+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed81a4640 con 0x7f7ed8106730 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.448+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ed0003e20 con 0x7f7ed8106730 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.448+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed81a4b90 con 0x7f7ed8106730 2024-03-19T02:48:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.448+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0010460 con 0x7f7ed8106730 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.449+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed8068610 con 0x7f7ed8106730 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.450+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7ed0003970 con 0x7f7ed8106730 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.450+0000 7f7ecffff640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 0x7f7ec40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.451+0000 7f7ed659c640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 0x7f7ec40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.451+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7ed00cef90 con 0x7f7ed8106730 2024-03-19T02:48:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.451+0000 7f7ed659c640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 0x7f7ec40a4e30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005ab0 tx=0x7f7ec8005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.459+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ed009b2c0 con 0x7f7ed8106730 2024-03-19T02:48:59.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.718+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ed8108c90 con 0x7f7ed8106730 2024-03-19T02:48:59.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.718+0000 7f7ecffff640 1 -- 172.21.15.116:0/801143170 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11953 (secure 0 0 0) 0x7f7ed009f360 con 0x7f7ed8106730 2024-03-19T02:48:59.720 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:48:59.720 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:48:15.836587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:48:59.720 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 msgr2=0x7f7ec40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 0x7f7ec40a4e30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005ab0 tx=0x7f7ec8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 msgr2=0x7f7ed81a3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009fd0 tx=0x7f7ed000b260 comp rx=0 tx=0).stop 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 shutdown_connections 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ec40a2970 0x7f7ec40a4e30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ed8106730 0x7f7ed81a3f30 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ed81010e0 0x7f7ed81993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 --2- 172.21.15.116:0/801143170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed8100720 0x7f7ed8198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 >> 172.21.15.116:0/801143170 conn(0x7f7ed8077820 msgr2=0x7f7ed8103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 shutdown_connections 2024-03-19T02:48:59.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.721+0000 7f7edcbe8640 1 -- 172.21.15.116:0/801143170 wait complete. 2024-03-19T02:48:59.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:48:59.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 -- 172.21.15.116:0/263072287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 msgr2=0x7f1da4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/263072287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da4106a90 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009930 tx=0x7f1da002bc30 comp rx=0 tx=0).stop 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 -- 172.21.15.116:0/263072287 shutdown_connections 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/263072287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da4101060 0x7f1da410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/263072287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da41006a0 0x7f1da4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/263072287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da4106a90 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 -- 172.21.15.116:0/263072287 >> 172.21.15.116:0/263072287 conn(0x7f1da40778d0 msgr2=0x7f1da4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 -- 172.21.15.116:0/263072287 shutdown_connections 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.865+0000 7f1dab5c4640 1 -- 172.21.15.116:0/263072287 wait complete. 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 Processor -- start 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 -- start start 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da41006a0 0x7f1da4198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da4101060 0x7f1da4199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1da4110970 con 0x7f1da41066b0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41107f0 con 0x7f1da41006a0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1dab5c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1da4110af0 con 0x7f1da4101060 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da4101060 0x7f1da4199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51688/0 (socket says 172.21.15.116:51688) 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 -- 172.21.15.116:0/1338238859 learned_addr learned my addr 172.21.15.116:0/1338238859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9360640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da41006a0 0x7f1da4198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 -- 172.21.15.116:0/1338238859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da4101060 msgr2=0x7f1da4199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da4101060 0x7f1da4199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 -- 172.21.15.116:0/1338238859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da41006a0 msgr2=0x7f1da4198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da41006a0 0x7f1da4198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.866+0000 7f1da9b61640 1 -- 172.21.15.116:0/1338238859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da41a4410 con 0x7f1da41066b0 2024-03-19T02:48:59.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1da9b61640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00eb10 tx=0x7f1d9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1d9a7fc640 1 -- 172.21.15.116:0/1338238859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c004b20 con 0x7f1da41066b0 2024-03-19T02:48:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1d9a7fc640 1 -- 172.21.15.116:0/1338238859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d9c004cc0 con 0x7f1da41066b0 2024-03-19T02:48:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da41a45e0 con 0x7f1da41066b0 2024-03-19T02:48:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1d9a7fc640 1 -- 172.21.15.116:0/1338238859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c010690 con 0x7f1da41066b0 2024-03-19T02:48:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.867+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41a4aa0 con 0x7f1da41066b0 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.869+0000 7f1d9a7fc640 1 -- 172.21.15.116:0/1338238859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1d9c005020 con 0x7f1da41066b0 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 Processor -- start 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 -- start start 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986f0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986f07d0 0x55ea986f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea9864a4b0 con 0x55ea986e72d0 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ea985f2d70 con 0x55ea986f07d0 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea98646550 con 0x55ea986e7bf0 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1d9a7fc640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 0x7f1d900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51700/0 (socket says 172.21.15.116:51700) 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2480681175 learned_addr learned my addr 172.21.15.116:0/2480681175 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9b61640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 0x7f1d900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986f07d0 0x55ea986f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1d9a7fc640 1 -- 172.21.15.116:0/1338238859 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f1d9c0cf730 con 0x7f1da41066b0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 -- 172.21.15.116:0/2480681175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986f07d0 msgr2=0x55ea986f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986f07d0 0x55ea986f2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 -- 172.21.15.116:0/2480681175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 msgr2=0x55ea986e76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.870+0000 7f1da9360640 1 -- 172.21.15.116:0/2480681175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea98615210 con 0x55ea986e7bf0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1da9b61640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 0x7f1d900a4eb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c09b6b0 tx=0x7f1d9c004e70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1da9360640 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986f0290 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009230 tx=0x7f1da0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467b8cff18de8d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1d8f7fe640 1 -- 172.21.15.116:0/2480681175 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0045020 con 0x55ea986e7bf0 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1d8f7fe640 1 -- 172.21.15.116:0/2480681175 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1da002c7e0 con 0x55ea986e7bf0 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.871+0000 7f1d8f7fe640 1 -- 172.21.15.116:0/2480681175 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005330 con 0x55ea986e7bf0 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2480681175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 msgr2=0x55ea986f0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986f0290 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009230 tx=0x7f1da0009560 comp rx=0 tx=0).stop 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2480681175 shutdown_connections 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986f07d0 0x55ea986f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986f0290 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2480681175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986e72d0 0x55ea986e76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2480681175 >> 172.21.15.116:0/2480681175 conn(0x55ea986e1a00 msgr2=0x55ea986e1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2480681175 shutdown_connections 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2480681175 wait complete. 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 -- start start 2024-03-19T02:48:59.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 Processor -- start 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.872+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986edd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea986f3560 con 0x55ea986f07d0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ea98606310 con 0x55ea986e7bf0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ea986f36e0 con 0x55ea986e72d0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 10 client.0 _ll_register_callbacks cb 0x55ea985fa2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51712/0 (socket says 172.21.15.116:51712) 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2073173573 learned_addr learned my addr 172.21.15.116:0/2073173573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da9360640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986edd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 msgr2=0x55ea986ed830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2073173573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 msgr2=0x55ea986edd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986edd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2073173573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea9835e3f0 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f1d94013ea0 tx=0x7f1d94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dae1360a8505d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400f690 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f1d9400f830 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1da002c7e0 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1d9402d050 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 10 client.21507 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.873+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ea984287d0 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1da002c7e0 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f1d940258e0 con 0x55ea986f07d0 2024-03-19T02:48:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 10 client.21507 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 10 client.21507 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 20 client.21507 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea98700ac0 0x55ea98702f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 4 client.21507 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea986a83e0 con 0x55ea98700ac0 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1da9360640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea98700ac0 0x55ea98702f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1d8cff9640 10 client.21507 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.874+0000 7f1da9360640 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea98700ac0 0x55ea98702f00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dcfe3155f8253 server_cookie=c279cec58c082d90 in_seq=0 out_seq=0 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2073173573 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea986a83e0 con 0x55ea98700ac0 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1d8cff9640 10 client.21507 handle_command_reply: tid=0 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 client.21507 shutdown 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 msgr2=0x55ea986ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f1d94013ea0 tx=0x7f1d94005f90 comp rx=0 tx=0).stop 2024-03-19T02:48:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 shutdown_connections 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea98700ac0 0x55ea98702f00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea986ee2b0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1d8cff9640 0 client.21507 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986edd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2073173573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 >> 172.21.15.116:0/2073173573 conn(0x55ea986e1a00 msgr2=0x55ea986e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 shutdown_connections 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2073173573 wait complete. 2024-03-19T02:48:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.875+0000 7f1dae1ad740 20 client.21507 trim_cache size 0 max 16384 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.890+0000 7f1dae1ad740 1 Processor -- start 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.890+0000 7f1dae1ad740 1 -- start start 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986fc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41aebc0 con 0x55ea986f07d0 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41b0760 con 0x55ea986e7bf0 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41af290 con 0x55ea986e72d0 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51718/0 (socket says 172.21.15.116:51718) 2024-03-19T02:48:59.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 -- 172.21.15.116:0/652096268 learned_addr learned my addr 172.21.15.116:0/652096268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da9360640 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 -- 172.21.15.116:0/652096268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 msgr2=0x55ea986ed110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 -- 172.21.15.116:0/652096268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 msgr2=0x55ea986fc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986fc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 -- 172.21.15.116:0/652096268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea986dd2c0 con 0x55ea986f07d0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f1d94001eb0 tx=0x7f1d94015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30866664cd77616 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1d8effd640 1 -- 172.21.15.116:0/652096268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94004220 con 0x55ea986f07d0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.891+0000 7f1d8effd640 1 -- 172.21.15.116:0/652096268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d940054e0 con 0x55ea986f07d0 2024-03-19T02:48:59.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1d8effd640 1 -- 172.21.15.116:0/652096268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d940257a0 con 0x55ea986f07d0 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 -- 172.21.15.116:0/652096268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 msgr2=0x55ea987087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f1d94001eb0 tx=0x7f1d94015040 comp rx=0 tx=0).stop 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 -- 172.21.15.116:0/652096268 shutdown_connections 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea986f07d0 0x55ea987087d0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e7bf0 0x55ea986fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/652096268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e72d0 0x55ea986ed110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 -- 172.21.15.116:0/652096268 >> 172.21.15.116:0/652096268 conn(0x55ea986e1a00 msgr2=0x55ea986ef7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 -- 172.21.15.116:0/652096268 shutdown_connections 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.892+0000 7f1dae1ad740 1 -- 172.21.15.116:0/652096268 wait complete. 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 -- start start 2024-03-19T02:48:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 Processor -- start 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e72d0 0x55ea986f8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986e9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41b0090 con 0x55ea98700ac0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41af960 con 0x55ea986e72d0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41acbc0 con 0x55ea986e7bf0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1dae1ad740 10 client.0 _ll_register_callbacks cb 0x7f1d94013d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1da9360640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986e9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1da8b5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51732/0 (socket says 172.21.15.116:51732) 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2422931743 learned_addr learned my addr 172.21.15.116:0/2422931743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.893+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e72d0 0x55ea986f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 msgr2=0x55ea986e9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986e9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2422931743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e72d0 msgr2=0x55ea986f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e72d0 0x55ea986f8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 -- 172.21.15.116:0/2422931743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea986ea930 con 0x55ea98700ac0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da9360640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986e9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1da8b5f640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1d94025d80 tx=0x7f1d94036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27357949b2865743 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.894+0000 7f1d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400e3d0 con 0x55ea98700ac0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f1d94004460 con 0x55ea98700ac0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d940054e0 con 0x55ea98700ac0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1d94025520 con 0x55ea98700ac0 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1dae1ad740 10 client.21513 fetch_fsmap learned FSMap version 40 2024-03-19T02:48:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ea98766b50 con 0x55ea98700ac0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.895+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d940054e0 con 0x55ea98700ac0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f1d9400fb20 con 0x55ea98700ac0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 10 client.21513 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 10 client.21513 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 20 client.21513 populate_metadata read hostname 'smithi116' 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea987671c0 0x55ea987695e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 4 client.21513 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea986579f0 con 0x55ea987671c0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1da9360640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea987671c0 0x55ea987695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1d8cff9640 10 client.21513 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1da9360640 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea987671c0 0x55ea987695e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cec40eee9853c105 server_cookie=5e4438202875146 in_seq=0 out_seq=0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1d8cff9640 1 -- 172.21.15.116:0/2422931743 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea986579f0 con 0x55ea987671c0 2024-03-19T02:48:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1d8cff9640 10 client.21513 handle_command_reply: tid=0 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.896+0000 7f1dae1ad740 1 client.21513 shutdown 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 msgr2=0x55ea986ea3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1d94025d80 tx=0x7f1d94036040 comp rx=0 tx=0).stop 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 shutdown_connections 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55ea987671c0 0x55ea987695e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea98700ac0 0x55ea986ea3f0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea986e7bf0 0x55ea986e9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1d8cff9640 0 client.21513 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 --2- 172.21.15.116:0/2422931743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ea986e72d0 0x55ea986f8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 >> 172.21.15.116:0/2422931743 conn(0x55ea986e1a00 msgr2=0x55ea986ed7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 shutdown_connections 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 1 -- 172.21.15.116:0/2422931743 wait complete. 2024-03-19T02:48:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dae1ad740 20 client.21513 trim_cache size 0 max 16384 2024-03-19T02:48:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:48:59.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 msgr2=0x7f1d900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 0x7f1d900a4eb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c09b6b0 tx=0x7f1d9c004e70 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 msgr2=0x7f1da41a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.897+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00eb10 tx=0x7f1d9c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 shutdown_connections 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1d900a29f0 0x7f1d900a4eb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1da41066b0 0x7f1da41a3ed0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da4101060 0x7f1da4199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 --2- 172.21.15.116:0/1338238859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da41006a0 0x7f1da4198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 >> 172.21.15.116:0/1338238859 conn(0x7f1da40778d0 msgr2=0x7f1da4102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 shutdown_connections 2024-03-19T02:48:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:48:59.898+0000 7f1dab5c4640 1 -- 172.21.15.116:0/1338238859 wait complete. 2024-03-19T02:48:59.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:49:00.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:48:59 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/801143170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 -- 172.21.15.116:0/3647973464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece410be10 msgr2=0x7fece410e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 --2- 172.21.15.116:0/3647973464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece410be10 0x7fece410e2b0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009ca0 tx=0x7fecdc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 -- 172.21.15.116:0/3647973464 shutdown_connections 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 --2- 172.21.15.116:0/3647973464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece410be10 0x7fece410e2b0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 --2- 172.21.15.116:0/3647973464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4103360 0x7fece410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 --2- 172.21.15.116:0/3647973464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece4102a40 0x7fece4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 -- 172.21.15.116:0/3647973464 >> 172.21.15.116:0/3647973464 conn(0x7fece40778b0 msgr2=0x7fece4077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 -- 172.21.15.116:0/3647973464 shutdown_connections 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.027+0000 7fece9681640 1 -- 172.21.15.116:0/3647973464 wait complete. 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 Processor -- start 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 -- start start 2024-03-19T02:49:00.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece410be10 0x7fece419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fece4111ac0 con 0x7fece4103360 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fece4111940 con 0x7fece4102a40 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece9681640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fece4111c40 con 0x7fece410be10 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51746/0 (socket says 172.21.15.116:51746) 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45226/0 (socket says 172.21.15.116:45226) 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece27fc640 1 -- 172.21.15.116:0/99164214 learned_addr learned my addr 172.21.15.116:0/99164214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.028+0000 7fece37fe640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece410be10 0x7fece419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 -- 172.21.15.116:0/99164214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece410be10 msgr2=0x7fece419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece410be10 0x7fece419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 -- 172.21.15.116:0/99164214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 msgr2=0x7fece4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 -- 172.21.15.116:0/99164214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece419cf10 con 0x7fece4103360 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece2ffd640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece27fc640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fecd800e9f0 tx=0x7fecd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7feccbfff640 1 -- 172.21.15.116:0/99164214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd800cd80 con 0x7fece4103360 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece419d200 con 0x7fece4103360 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7feccbfff640 1 -- 172.21.15.116:0/99164214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecd80108e0 con 0x7fece4103360 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece41a4a90 con 0x7fece4103360 2024-03-19T02:49:00.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.029+0000 7feccbfff640 1 -- 172.21.15.116:0/99164214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd8026060 con 0x7fece4103360 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.031+0000 7feccbfff640 1 -- 172.21.15.116:0/99164214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fecd8010a90 con 0x7fece4103360 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7feccbfff640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 0x7feccc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 Processor -- start 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece2ffd640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 0x7feccc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 -- start start 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7feccbfff640 1 -- 172.21.15.116:0/99164214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fecd80cf5a0 con 0x7fece4103360 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 0x55704a786ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a78fbe0 0x55704a791fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55704a6e54b0 con 0x55704a787000 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55704a68dd70 con 0x55704a7866e0 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fecec26b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55704a6e1550 con 0x55704a78fbe0 2024-03-19T02:49:00.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 0x55704a786ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51754/0 (socket says 172.21.15.116:51754) 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece27fc640 1 -- 172.21.15.116:0/3728323713 learned_addr learned my addr 172.21.15.116:0/3728323713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.032+0000 7fece37fe640 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a78fbe0 0x55704a791fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 0x7feccc0a4e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fecd4011490 tx=0x7fecd4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 -- 172.21.15.116:0/3728323713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a78fbe0 msgr2=0x55704a791fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a78fbe0 0x55704a791fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 -- 172.21.15.116:0/3728323713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 msgr2=0x55704a78f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 -- 172.21.15.116:0/3728323713 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704a6b0210 con 0x55704a7866e0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece27fc640 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fece2ffd640 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 0x55704a786ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fecd4016000 tx=0x7fecd4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a78ac4d270e362e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fecc97fa640 1 -- 172.21.15.116:0/3728323713 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd400f4e0 con 0x55704a7866e0 2024-03-19T02:49:00.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fecc97fa640 1 -- 172.21.15.116:0/3728323713 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecd400fb30 con 0x55704a7866e0 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.033+0000 7fecc97fa640 1 -- 172.21.15.116:0/3728323713 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd4026a10 con 0x55704a7866e0 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- 172.21.15.116:0/3728323713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 msgr2=0x55704a786ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 0x55704a786ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fecd4016000 tx=0x7fecd4019000 comp rx=0 tx=0).stop 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- 172.21.15.116:0/3728323713 shutdown_connections 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a78fbe0 0x55704a791fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a787000 0x55704a78f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 --2- 172.21.15.116:0/3728323713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a7866e0 0x55704a786ac0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- 172.21.15.116:0/3728323713 >> 172.21.15.116:0/3728323713 conn(0x55704a780e10 msgr2=0x55704a7811f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- 172.21.15.116:0/3728323713 shutdown_connections 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- 172.21.15.116:0/3728323713 wait complete. 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 -- start start 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.034+0000 7fecec26b740 1 Processor -- start 2024-03-19T02:49:00.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a78d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a78d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55704a7929d0 con 0x55704a78fbe0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55704a6a1310 con 0x55704a787000 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55704a792b50 con 0x55704a7866e0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecec26b740 10 client.0 _ll_register_callbacks cb 0x55704a6952d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a78d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece27fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a78d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a78d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51758/0 (socket says 172.21.15.116:51758) 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece37fe640 1 -- 172.21.15.116:0/2452279502 learned_addr learned my addr 172.21.15.116:0/2452279502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 -- 172.21.15.116:0/2452279502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 msgr2=0x55704a78d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a78d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 -- 172.21.15.116:0/2452279502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 msgr2=0x55704a78d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a78d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 -- 172.21.15.116:0/2452279502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704a3f93f0 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fece2ffd640 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78cc10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009e80 tx=0x7fecd400f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda5fd9782c34823 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.035+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd400fce0 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecd400fb30 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fecd4005040 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fecd4038050 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 10 client.31225 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55704a4c37d0 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecd400fb30 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fecd4024020 con 0x55704a7866e0 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 10 client.31225 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 10 client.31225 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 20 client.31225 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a79fdf0 0x55704a7a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 4 client.31225 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55704a7433e0 con 0x55704a79fdf0 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.036+0000 7fece27fc640 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a79fdf0 0x55704a7a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.037+0000 7fecc6ff5640 10 client.31225 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.037+0000 7fece27fc640 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a79fdf0 0x55704a7a21e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=44ec93ca7c04a555 server_cookie=838a1976b0f55239 in_seq=0 out_seq=0 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.037+0000 7fecc6ff5640 1 -- 172.21.15.116:0/2452279502 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55704a7433e0 con 0x55704a79fdf0 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.037+0000 7fecc6ff5640 10 client.31225 handle_command_reply: tid=0 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 client.31225 shutdown 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 msgr2=0x55704a78cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78cc10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009e80 tx=0x7fecd400f990 comp rx=0 tx=0).stop 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 shutdown_connections 2024-03-19T02:49:00.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a79fdf0 0x55704a7a21e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a78d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a78d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecc6ff5640 0 client.31225 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 --2- 172.21.15.116:0/2452279502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78cc10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 >> 172.21.15.116:0/2452279502 conn(0x55704a780e10 msgr2=0x55704a77bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 shutdown_connections 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 1 -- 172.21.15.116:0/2452279502 wait complete. 2024-03-19T02:49:00.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.038+0000 7fecec26b740 20 client.31225 trim_cache size 0 max 16384 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 Processor -- start 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 -- start start 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a79b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fece41aca60 con 0x55704a78fbe0 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fece41ae600 con 0x55704a787000 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fecec26b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fece41ad130 con 0x55704a7866e0 2024-03-19T02:49:00.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51774/0 (socket says 172.21.15.116:51774) 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 -- 172.21.15.116:0/1952765059 learned_addr learned my addr 172.21.15.116:0/1952765059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece2ffd640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece27fc640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a79b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 -- 172.21.15.116:0/1952765059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 msgr2=0x55704a78f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 -- 172.21.15.116:0/1952765059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 msgr2=0x55704a79b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a79b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece37fe640 1 -- 172.21.15.116:0/1952765059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704a786e30 con 0x55704a78fbe0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.055+0000 7fece2ffd640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fece37fe640 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fecdc02c700 tx=0x7fecdc005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea71fa2a1815847 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fecc8ff9640 1 -- 172.21.15.116:0/1952765059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc039040 con 0x55704a78fbe0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fecc8ff9640 1 -- 172.21.15.116:0/1952765059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecdc045070 con 0x55704a78fbe0 2024-03-19T02:49:00.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fecc8ff9640 1 -- 172.21.15.116:0/1952765059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc004040 con 0x55704a78fbe0 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fecec26b740 1 -- 172.21.15.116:0/1952765059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 msgr2=0x55704a77ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.056+0000 7fecec26b740 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fecdc02c700 tx=0x7fecdc005c50 comp rx=0 tx=0).stop 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- 172.21.15.116:0/1952765059 shutdown_connections 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a77ba20 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a79b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- 172.21.15.116:0/1952765059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a78f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- 172.21.15.116:0/1952765059 >> 172.21.15.116:0/1952765059 conn(0x55704a780e10 msgr2=0x55704a77b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- 172.21.15.116:0/1952765059 shutdown_connections 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- 172.21.15.116:0/1952765059 wait complete. 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- start start 2024-03-19T02:49:00.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 Processor -- start 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a798170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a793280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fece41adf30 con 0x55704a78fbe0 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fece41ad800 con 0x55704a787000 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fece41aaa60 con 0x55704a7866e0 2024-03-19T02:49:00.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.057+0000 7fecec26b740 10 client.0 _ll_register_callbacks cb 0x7fecd8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43562/0 (socket says 172.21.15.116:43562) 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a793280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece2ffd640 1 -- 172.21.15.116:0/4291581920 learned_addr learned my addr 172.21.15.116:0/4291581920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece27fc640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a798170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 -- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 msgr2=0x55704a797c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 -- 172.21.15.116:0/4291581920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 msgr2=0x55704a798170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a798170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 -- 172.21.15.116:0/4291581920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704a793ab0 con 0x55704a78fbe0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece2ffd640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fece37fe640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a793280 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009ca0 tx=0x7fecdc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39db65dcfe8c4787 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc039040 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fecdc0324f0 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecdc045070 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fecdc032730 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecec26b740 10 client.21531 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.058+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55704a805bc0 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecdc045070 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fecdc053500 con 0x55704a78fbe0 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 10 client.21531 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 10 client.21531 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 20 client.21531 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a8062b0 0x55704a8086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 4 client.21531 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55704a6f29f0 con 0x55704a8062b0 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fece2ffd640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a8062b0 0x55704a8086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fecc6ff5640 10 client.21531 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.059+0000 7fece2ffd640 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a8062b0 0x55704a8086d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=211f3cd10f272cb0 server_cookie=a2c7e49257c17c5d in_seq=0 out_seq=0 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecc6ff5640 1 -- 172.21.15.116:0/4291581920 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55704a6f29f0 con 0x55704a8062b0 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecc6ff5640 10 client.21531 handle_command_reply: tid=0 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 client.21531 shutdown 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 msgr2=0x55704a793280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a793280 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009ca0 tx=0x7fecdc003970 comp rx=0 tx=0).stop 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 shutdown_connections 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55704a8062b0 0x55704a8086d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55704a78fbe0 0x55704a793280 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55704a787000 0x55704a798170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 --2- 172.21.15.116:0/4291581920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55704a7866e0 0x55704a797c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 >> 172.21.15.116:0/4291581920 conn(0x55704a780e10 msgr2=0x55704a77bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecc6ff5640 0 client.21531 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 shutdown_connections 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 1 -- 172.21.15.116:0/4291581920 wait complete. 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.060+0000 7fecec26b740 20 client.21531 trim_cache size 0 max 16384 2024-03-19T02:49:00.068 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 msgr2=0x7feccc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 0x7feccc0a4e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fecd4011490 tx=0x7fecd4005f90 comp rx=0 tx=0).stop 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 msgr2=0x7fece4198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fecd800e9f0 tx=0x7fecd800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 shutdown_connections 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7feccc0a2970 0x7feccc0a4e30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fece410be10 0x7fece419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece4103360 0x7fece4198460 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 --2- 172.21.15.116:0/99164214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fece4102a40 0x7fece4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 >> 172.21.15.116:0/99164214 conn(0x7fece40778b0 msgr2=0x7fece40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 shutdown_connections 2024-03-19T02:49:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.061+0000 7fece9681640 1 -- 172.21.15.116:0/99164214 wait complete. 2024-03-19T02:49:00.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:49:00.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:48:59 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/801143170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 -- 172.21.15.116:0/1541707086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241066b0 msgr2=0x7f1724106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 --2- 172.21.15.116:0/1541707086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241066b0 0x7f1724106a90 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f1720009930 tx=0x7f172002bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 -- 172.21.15.116:0/1541707086 shutdown_connections 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 --2- 172.21.15.116:0/1541707086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1724101060 0x7f172410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 --2- 172.21.15.116:0/1541707086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241006a0 0x7f1724100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 --2- 172.21.15.116:0/1541707086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241066b0 0x7f1724106a90 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 -- 172.21.15.116:0/1541707086 >> 172.21.15.116:0/1541707086 conn(0x7f17240778d0 msgr2=0x7f1724077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.194+0000 7f172bd44640 1 -- 172.21.15.116:0/1541707086 shutdown_connections 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 -- 172.21.15.116:0/1541707086 wait complete. 2024-03-19T02:49:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 Processor -- start 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 -- start start 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 0x7f1724197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241066b0 0x7f172419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1724110930 con 0x7f1724101060 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17241107b0 con 0x7f17241006a0 2024-03-19T02:49:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f172bd44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1724110ab0 con 0x7f17241066b0 2024-03-19T02:49:00.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.195+0000 7f17292df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51798/0 (socket says 172.21.15.116:51798) 2024-03-19T02:49:00.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 0x7f1724197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 -- 172.21.15.116:0/4109776625 learned_addr learned my addr 172.21.15.116:0/4109776625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f172a2e1640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241066b0 0x7f172419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 -- 172.21.15.116:0/4109776625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241066b0 msgr2=0x7f172419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241066b0 0x7f172419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 -- 172.21.15.116:0/4109776625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 msgr2=0x7f1724197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 0x7f1724197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 -- 172.21.15.116:0/4109776625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172419cf70 con 0x7f1724101060 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f1729ae0640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 0x7f1724197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f17292df640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f171400e9f0 tx=0x7f171400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f171affd640 1 -- 172.21.15.116:0/4109776625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171400cd80 con 0x7f1724101060 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172419d260 con 0x7f1724101060 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f171affd640 1 -- 172.21.15.116:0/4109776625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17140108e0 con 0x7f1724101060 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17241a4af0 con 0x7f1724101060 2024-03-19T02:49:00.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.196+0000 7f171affd640 1 -- 172.21.15.116:0/4109776625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714018920 con 0x7f1724101060 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.198+0000 7f171affd640 1 -- 172.21.15.116:0/4109776625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1714010a90 con 0x7f1724101060 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 Processor -- start 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 -- start start 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 0x5605055436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x56050554c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x56050554ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605054a64b0 con 0x5605055432d0 2024-03-19T02:49:00.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56050544ed70 con 0x56050554c7d0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172ea05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5605054a2550 con 0x560505543bf0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 0x5605055436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f17292df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x56050554ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x56050554c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 0x5605055436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51812/0 (socket says 172.21.15.116:51812) 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f17292df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x56050554ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45290/0 (socket says 172.21.15.116:45290) 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172a2e1640 1 -- 172.21.15.116:0/40883091 learned_addr learned my addr 172.21.15.116:0/40883091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f171affd640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 0x7f17100a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f172a2e1640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 0x7f17100a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 -- 172.21.15.116:0/40883091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 msgr2=0x56050554ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x56050554ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 -- 172.21.15.116:0/40883091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 msgr2=0x5605055436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 0x5605055436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f1729ae0640 1 -- 172.21.15.116:0/40883091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560505471210 con 0x560505543bf0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.199+0000 7f171affd640 1 -- 172.21.15.116:0/4109776625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f17140cf4c0 con 0x7f1724101060 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.200+0000 7f1729ae0640 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x56050554c290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1720009230 tx=0x7f17200095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e38742c3ffbad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.200+0000 7f172a2e1640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 0x7f17100a4eb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f171c0049d0 tx=0x7f171c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.200+0000 7f170ffff640 1 -- 172.21.15.116:0/40883091 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720004040 con 0x560505543bf0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.200+0000 7f170ffff640 1 -- 172.21.15.116:0/40883091 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172002cc90 con 0x560505543bf0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.200+0000 7f170ffff640 1 -- 172.21.15.116:0/40883091 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172003e9f0 con 0x560505543bf0 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- 172.21.15.116:0/40883091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 msgr2=0x56050554c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x56050554c290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1720009230 tx=0x7f17200095f0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- 172.21.15.116:0/40883091 shutdown_connections 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x56050554ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x56050554c290 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- 172.21.15.116:0/40883091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605055432d0 0x5605055436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- 172.21.15.116:0/40883091 >> 172.21.15.116:0/40883091 conn(0x56050553da00 msgr2=0x56050553dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- 172.21.15.116:0/40883091 shutdown_connections 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- 172.21.15.116:0/40883091 wait complete. 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- start start 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 Processor -- start 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x5605055498d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505549e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56050554f560 con 0x56050554c7d0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560505462310 con 0x5605055432d0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56050554f6e0 con 0x560505543bf0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172ea05740 10 client.0 _ll_register_callbacks cb 0x5605054562d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x5605055498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x5605055498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45302/0 (socket says 172.21.15.116:45302) 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51820/0 (socket says 172.21.15.116:51820) 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f1729ae0640 1 -- 172.21.15.116:0/2892522970 learned_addr learned my addr 172.21.15.116:0/2892522970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.201+0000 7f17292df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505549e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 -- 172.21.15.116:0/2892522970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 msgr2=0x560505549e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505549e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 -- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 msgr2=0x5605055498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x5605055498d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 -- 172.21.15.116:0/2892522970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605051ba3f0 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172a2e1640 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f171c005c40 tx=0x7f171c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb2500bc081a588 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f170d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c00fe60 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f171c004d20 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172002cc90 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f171c032050 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172ea05740 10 client.21543 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5605052847d0 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.202+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172002cc90 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f171c01f040 con 0x56050554c7d0 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 10 client.21543 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 10 client.21543 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 20 client.21543 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56050555ccd0 0x56050555f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 4 client.21543 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605055043e0 con 0x56050555ccd0 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f1729ae0640 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56050555ccd0 0x56050555f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f1729ae0640 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56050555ccd0 0x56050555f120 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3395135831fe994b server_cookie=5bbede6db0fffa81 in_seq=0 out_seq=0 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.203+0000 7f170d7fa640 10 client.21543 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f170d7fa640 1 -- 172.21.15.116:0/2892522970 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605055043e0 con 0x56050555ccd0 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f170d7fa640 10 client.21543 handle_command_reply: tid=0 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 client.21543 shutdown 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 msgr2=0x56050554a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f171c005c40 tx=0x7f171c004a40 comp rx=0 tx=0).stop 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 shutdown_connections 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x56050555ccd0 0x56050555f120 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x56050554a350 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505549e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 --2- 172.21.15.116:0/2892522970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x5605055498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 >> 172.21.15.116:0/2892522970 conn(0x56050553da00 msgr2=0x56050553f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.204+0000 7f170d7fa640 0 client.21543 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.205+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 shutdown_connections 2024-03-19T02:49:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.205+0000 7f172ea05740 1 -- 172.21.15.116:0/2892522970 wait complete. 2024-03-19T02:49:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.205+0000 7f172ea05740 20 client.21543 trim_cache size 0 max 16384 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.219+0000 7f172ea05740 1 Processor -- start 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.219+0000 7f172ea05740 1 -- start start 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.219+0000 7f172ea05740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x56050554c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.219+0000 7f172ea05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505558300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172ea05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172ea05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17241aca40 con 0x56050554c7d0 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172ea05740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17241ae5e0 con 0x5605055432d0 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172ea05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17241ad110 con 0x560505543bf0 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x56050554c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51822/0 (socket says 172.21.15.116:51822) 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 -- 172.21.15.116:0/793075593 learned_addr learned my addr 172.21.15.116:0/793075593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f17292df640 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505558300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 -- 172.21.15.116:0/793075593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 msgr2=0x560505558300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505558300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 -- 172.21.15.116:0/793075593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 msgr2=0x56050554c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x56050554c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 -- 172.21.15.116:0/793075593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050554ef30 con 0x56050554c7d0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f1729ae0640 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x56050554c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f172a2e1640 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f171c005b40 tx=0x7f171c013480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b7b9d779376f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f170f7fe640 1 -- 172.21.15.116:0/793075593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c03c020 con 0x56050554c7d0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f170f7fe640 1 -- 172.21.15.116:0/793075593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f171c02f070 con 0x56050554c7d0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.220+0000 7f170f7fe640 1 -- 172.21.15.116:0/793075593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c0386a0 con 0x56050554c7d0 2024-03-19T02:49:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- 172.21.15.116:0/793075593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 msgr2=0x5605055649f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f171c005b40 tx=0x7f171c013480 comp rx=0 tx=0).stop 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- 172.21.15.116:0/793075593 shutdown_connections 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56050554c7d0 0x5605055649f0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560505543bf0 0x560505558300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 --2- 172.21.15.116:0/793075593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605055432d0 0x56050554c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- 172.21.15.116:0/793075593 >> 172.21.15.116:0/793075593 conn(0x56050553da00 msgr2=0x56050553c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- 172.21.15.116:0/793075593 shutdown_connections 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- 172.21.15.116:0/793075593 wait complete. 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 -- start start 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.221+0000 7f172ea05740 1 Processor -- start 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 0x560505546450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x560505546ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17241adf10 con 0x560505543bf0 2024-03-19T02:49:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17241ad7e0 con 0x56050554c7d0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17241aaa40 con 0x5605055432d0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172ea05740 10 client.0 _ll_register_callbacks cb 0x7f171c0135a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f1729ae0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 0x560505546450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51824/0 (socket says 172.21.15.116:51824) 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 -- 172.21.15.116:0/4039821072 learned_addr learned my addr 172.21.15.116:0/4039821072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172a2e1640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x560505546ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 -- 172.21.15.116:0/4039821072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 msgr2=0x560505546450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 0x560505546450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 -- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 msgr2=0x560505546ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x560505546ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 -- 172.21.15.116:0/4039821072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560505542a70 con 0x560505543bf0 2024-03-19T02:49:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f172a2e1640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x560505546ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f1729ae0640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 0x560505546450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f17292df640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f17140cc890 tx=0x7f171409b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841664d9a416c667 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f170d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140a3ee0 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.222+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f17140cf160 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f171c02f070 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f171409b7f0 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 10 client.21549 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56050555b820 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f171c02f070 con 0x560505543bf0 2024-03-19T02:49:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f1714080070 con 0x560505543bf0 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 10 client.21549 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 10 client.21549 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 20 client.21549 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5605055c3670 0x5605055c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 4 client.21549 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56050555f4b0 con 0x5605055c3670 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.223+0000 7f1729ae0640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5605055c3670 0x5605055c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f170d7fa640 10 client.21549 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f1729ae0640 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5605055c3670 0x5605055c5a90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f09027f3ce21316d server_cookie=89a09a85f7819573 in_seq=0 out_seq=0 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f170d7fa640 1 -- 172.21.15.116:0/4039821072 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56050555f4b0 con 0x5605055c3670 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f170d7fa640 10 client.21549 handle_command_reply: tid=0 2024-03-19T02:49:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 client.21549 shutdown 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 msgr2=0x560505546990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f17140cc890 tx=0x7f171409b6f0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 shutdown_connections 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5605055c3670 0x5605055c5a90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56050554c7d0 0x560505546ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560505543bf0 0x560505546990 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 --2- 172.21.15.116:0/4039821072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605055432d0 0x560505546450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f170d7fa640 0 client.21549 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 >> 172.21.15.116:0/4039821072 conn(0x56050553da00 msgr2=0x56050553f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 shutdown_connections 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 1 -- 172.21.15.116:0/4039821072 wait complete. 2024-03-19T02:49:00.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.224+0000 7f172ea05740 20 client.21549 trim_cache size 0 max 16384 2024-03-19T02:49:00.233 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 msgr2=0x7f17100a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 0x7f17100a4eb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f171c0049d0 tx=0x7f171c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 msgr2=0x7f17241984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f171400e9f0 tx=0x7f171400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 shutdown_connections 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f17100a29f0 0x7f17100a4eb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17241066b0 0x7f172419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1724101060 0x7f17241984c0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 --2- 172.21.15.116:0/4109776625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17241006a0 0x7f1724197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 >> 172.21.15.116:0/4109776625 conn(0x7f17240778d0 msgr2=0x7f1724068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 shutdown_connections 2024-03-19T02:49:00.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.225+0000 7f172bd44640 1 -- 172.21.15.116:0/4109776625 wait complete. 2024-03-19T02:49:00.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:49:00.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:48:59 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/801143170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:00.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55766 5.88 MB/sec execute 382 sec latency 4235.857 ms 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 -- 172.21.15.116:0/1824319218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc1066b0 msgr2=0x7f7ccc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 --2- 172.21.15.116:0/1824319218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc106a90 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009cd0 tx=0x7f7cbc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 -- 172.21.15.116:0/1824319218 shutdown_connections 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 --2- 172.21.15.116:0/1824319218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc101060 0x7f7ccc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 --2- 172.21.15.116:0/1824319218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 --2- 172.21.15.116:0/1824319218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc106a90 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 -- 172.21.15.116:0/1824319218 >> 172.21.15.116:0/1824319218 conn(0x7f7ccc0778d0 msgr2=0x7f7ccc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 -- 172.21.15.116:0/1824319218 shutdown_connections 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.364+0000 7f7cd1412640 1 -- 172.21.15.116:0/1824319218 wait complete. 2024-03-19T02:49:00.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 Processor -- start 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 -- start start 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc110970 con 0x7f7ccc101060 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1107f0 con 0x7f7ccc1006a0 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.365+0000 7f7cd1412640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc110af0 con 0x7f7ccc1066b0 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51832/0 (socket says 172.21.15.116:51832) 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 -- 172.21.15.116:0/4230218799 learned_addr learned my addr 172.21.15.116:0/4230218799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 -- 172.21.15.116:0/4230218799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc1066b0 msgr2=0x7f7ccc19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 -- 172.21.15.116:0/4230218799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 msgr2=0x7f7ccc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 -- 172.21.15.116:0/4230218799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19cf70 con 0x7f7ccc101060 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9f0 tx=0x7f7cc000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cb3fff640 1 -- 172.21.15.116:0/4230218799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000cd80 con 0x7f7ccc101060 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cb3fff640 1 -- 172.21.15.116:0/4230218799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cc00108e0 con 0x7f7ccc101060 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cb3fff640 1 -- 172.21.15.116:0/4230218799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0018920 con 0x7f7ccc101060 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19d260 con 0x7f7ccc101060 2024-03-19T02:49:00.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.366+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc1a4af0 con 0x7f7ccc101060 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.369+0000 7f7cb3fff640 1 -- 172.21.15.116:0/4230218799 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7cc0010a90 con 0x7f7ccc101060 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.369+0000 7f7cd3fff740 1 Processor -- start 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.369+0000 7f7cd3fff740 1 -- start start 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b21800290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b21802bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b2175a4b0 con 0x561b217f7bf0 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b21702d70 con 0x561b218007d0 2024-03-19T02:49:00.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b21756550 con 0x561b217f72d0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b21800290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43620/0 (socket says 172.21.15.116:43620) 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cca7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b21802bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 -- 172.21.15.116:0/1009309323 learned_addr learned my addr 172.21.15.116:0/1009309323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cb3fff640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 0x7f7cb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 -- 172.21.15.116:0/1009309323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 msgr2=0x561b21802bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b21802bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 -- 172.21.15.116:0/1009309323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 msgr2=0x561b21800290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b21800290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 -- 172.21.15.116:0/1009309323 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b21725210 con 0x561b217f72d0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7cb3fff640 1 -- 172.21.15.116:0/4230218799 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7cc00cf4a0 con 0x7f7ccc101060 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.370+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 0x7f7cb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.371+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009c20 tx=0x7f7cc4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a747f2fc2fe2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.371+0000 7f7cb17fa640 1 -- 172.21.15.116:0/1009309323 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc401b850 con 0x561b217f72d0 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.371+0000 7f7cb17fa640 1 -- 172.21.15.116:0/1009309323 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cc4019d80 con 0x561b217f72d0 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.371+0000 7f7cb17fa640 1 -- 172.21.15.116:0/1009309323 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400ed30 con 0x561b217f72d0 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1009309323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 msgr2=0x561b217f76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 0x7f7cb40a4e30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009fd0 tx=0x7f7cc4021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009c20 tx=0x7f7cc4011810 comp rx=0 tx=0).stop 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1009309323 shutdown_connections 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b21802bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b21800290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1009309323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217f76b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1009309323 >> 172.21.15.116:0/1009309323 conn(0x561b217f1a00 msgr2=0x561b217f1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1009309323 shutdown_connections 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1009309323 wait complete. 2024-03-19T02:49:00.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 -- start start 2024-03-19T02:49:00.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.372+0000 7f7cd3fff740 1 Processor -- start 2024-03-19T02:49:00.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b217fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b217fe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b218035c0 con 0x561b217f7bf0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b21716310 con 0x561b218007d0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b21803740 con 0x561b217f72d0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 10 client.0 _ll_register_callbacks cb 0x561b2170a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43636/0 (socket says 172.21.15.116:43636) 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 -- 172.21.15.116:0/1430726466 learned_addr learned my addr 172.21.15.116:0/1430726466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b217fe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b217fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 -- 172.21.15.116:0/1430726466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 msgr2=0x561b217fe250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b217fe250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 -- 172.21.15.116:0/1430726466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 msgr2=0x561b217fdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b217fdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 -- 172.21.15.116:0/1430726466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2146e3f0 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000f80 tx=0x7f7cbc0329a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0ef30932b0299e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7caeff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc04d060 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7cbc003d80 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.373+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cc4019d80 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7cbc054050 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 10 client.31261 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561b215387d0 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cc4019d80 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7cbc03d850 con 0x561b217f72d0 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 10 client.31261 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 10 client.31261 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:49:00.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 20 client.31261 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21810b10 0x561b21812fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 4 client.31261 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b217b83e0 con 0x561b21810b10 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.374+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21810b10 0x561b21812fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.375+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21810b10 0x561b21812fd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4079b931ecf5258a server_cookie=ec9a129dbc13bedb in_seq=0 out_seq=0 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.375+0000 7f7caeff5640 10 client.31261 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7caeff5640 1 -- 172.21.15.116:0/1430726466 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b217b83e0 con 0x561b21810b10 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7caeff5640 10 client.31261 handle_command_reply: tid=0 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 client.31261 shutdown 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 msgr2=0x561b217fd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000f80 tx=0x7f7cbc0329a0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 shutdown_connections 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21810b10 0x561b21812fd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b217fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f7bf0 0x561b217fdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/1430726466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f72d0 0x561b217fd7d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7caeff5640 0 client.31261 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 >> 172.21.15.116:0/1430726466 conn(0x561b217f1a00 msgr2=0x561b217f3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 shutdown_connections 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 1 -- 172.21.15.116:0/1430726466 wait complete. 2024-03-19T02:49:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.376+0000 7f7cd3fff740 20 client.31261 trim_cache size 0 max 16384 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 Processor -- start 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 -- start start 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b217fe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218188a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1acae0 con 0x561b217f72d0 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1ae680 con 0x561b218007d0 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1ad1b0 con 0x561b217f7bf0 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51850/0 (socket says 172.21.15.116:51850) 2024-03-19T02:49:00.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3873374620 learned_addr learned my addr 172.21.15.116:0/3873374620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b217fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218188a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3873374620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 msgr2=0x561b217fe070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b217fe070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3873374620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 msgr2=0x561b218188a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218188a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3873374620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b217ebfb0 con 0x561b217f72d0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.391+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc033000 tx=0x7f7cbc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180e0e5ba56994ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cb0ff9640 1 -- 172.21.15.116:0/3873374620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc04d070 con 0x561b217f72d0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cb0ff9640 1 -- 172.21.15.116:0/3873374620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cbc034440 con 0x561b217f72d0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cb0ff9640 1 -- 172.21.15.116:0/3873374620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc039040 con 0x561b217f72d0 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3873374620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 msgr2=0x561b2180c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc033000 tx=0x7f7cbc009b60 comp rx=0 tx=0).stop 2024-03-19T02:49:00.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3873374620 shutdown_connections 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b217fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3873374620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b2180c360 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.392+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3873374620 >> 172.21.15.116:0/3873374620 conn(0x561b217f1a00 msgr2=0x561b217f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3873374620 shutdown_connections 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3873374620 wait complete. 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- start start 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 Processor -- start 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b218040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1adfb0 con 0x561b217f72d0 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1ad880 con 0x561b218007d0 2024-03-19T02:49:00.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1aaae0 con 0x561b217f7bf0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cd3fff740 10 client.0 _ll_register_callbacks cb 0x7f7cc4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7cca7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b218040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.393+0000 7f7ccaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51864/0 (socket says 172.21.15.116:51864) 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3039775667 learned_addr learned my addr 172.21.15.116:0/3039775667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccb7fe640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 msgr2=0x561b218040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b218040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3039775667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 msgr2=0x561b218046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218046e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 -- 172.21.15.116:0/3039775667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b21804f10 con 0x561b217f72d0 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b218040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7ccaffd640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005cc0 tx=0x7f7cbc03d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca55a28a4f1d98c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7caeff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc05d030 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7cbc0348d0 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cbc034440 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7cbc067050 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7cd3fff740 10 client.21567 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.394+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561b21877000 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cbc034440 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7cbc059980 con 0x561b217f72d0 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 10 client.21567 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 10 client.21567 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 20 client.21567 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21877640 0x561b21879a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 4 client.21567 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b217679f0 con 0x561b21877640 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21877640 0x561b21879a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7caeff5640 10 client.21567 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.395+0000 7f7cca7fc640 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21877640 0x561b21879a60 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd8ab447c45bc46d server_cookie=386b61ebc226cf64 in_seq=0 out_seq=0 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7caeff5640 1 -- 172.21.15.116:0/3039775667 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b217679f0 con 0x561b21877640 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7caeff5640 10 client.21567 handle_command_reply: tid=0 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 client.21567 shutdown 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 msgr2=0x561b21808d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005cc0 tx=0x7f7cbc03d8a0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 shutdown_connections 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x561b21877640 0x561b21879a60 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b218007d0 0x561b218046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b217f7bf0 0x561b218040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7caeff5640 0 client.21567 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 --2- 172.21.15.116:0/3039775667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b217f72d0 0x561b21808d60 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 >> 172.21.15.116:0/3039775667 conn(0x561b217f1a00 msgr2=0x561b217f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 shutdown_connections 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 1 -- 172.21.15.116:0/3039775667 wait complete. 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd3fff740 20 client.21567 trim_cache size 0 max 16384 2024-03-19T02:49:00.404 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 msgr2=0x7f7cb40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 0x7f7cb40a4e30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009fd0 tx=0x7f7cc4021000 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 msgr2=0x7f7ccc198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9f0 tx=0x7f7cc000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 shutdown_connections 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7cb40a2970 0x7f7cb40a4e30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ccc1066b0 0x7f7ccc19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.396+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ccc101060 0x7f7ccc198510 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.397+0000 7f7cd1412640 1 --2- 172.21.15.116:0/4230218799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc1006a0 0x7f7ccc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.397+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 >> 172.21.15.116:0/4230218799 conn(0x7f7ccc0778d0 msgr2=0x7f7ccc0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.397+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 shutdown_connections 2024-03-19T02:49:00.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.397+0000 7f7cd1412640 1 -- 172.21.15.116:0/4230218799 wait complete. 2024-03-19T02:49:00.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:49:00.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 -- 172.21.15.116:0/1538303962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d810bd80 msgr2=0x7fc2d810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 --2- 172.21.15.116:0/1538303962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d810bd80 0x7fc2d810e220 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009ca0 tx=0x7fc2d002c180 comp rx=0 tx=0).stop 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 -- 172.21.15.116:0/1538303962 shutdown_connections 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 --2- 172.21.15.116:0/1538303962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d810bd80 0x7fc2d810e220 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 --2- 172.21.15.116:0/1538303962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d8103360 0x7fc2d810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.514+0000 7fc2de812640 1 --2- 172.21.15.116:0/1538303962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 0x7fc2d8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- 172.21.15.116:0/1538303962 >> 172.21.15.116:0/1538303962 conn(0x7fc2d80778d0 msgr2=0x7fc2d8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- 172.21.15.116:0/1538303962 shutdown_connections 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- 172.21.15.116:0/1538303962 wait complete. 2024-03-19T02:49:00.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 Processor -- start 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- start start 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 0x7fc2d8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d810bd80 0x7fc2d819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d8111a30 con 0x7fc2d8103360 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81118b0 con 0x7fc2d810bd80 2024-03-19T02:49:00.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.515+0000 7fc2de812640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d8111bb0 con 0x7fc2d8102a40 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51874/0 (socket says 172.21.15.116:51874) 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 -- 172.21.15.116:0/135032783 learned_addr learned my addr 172.21.15.116:0/135032783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 0x7fc2d8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d810bd80 0x7fc2d819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 -- 172.21.15.116:0/135032783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 msgr2=0x7fc2d8197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 0x7fc2d8197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 -- 172.21.15.116:0/135032783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d810bd80 msgr2=0x7fc2d819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d810bd80 0x7fc2d819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 -- 172.21.15.116:0/135032783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d819cf80 con 0x7fc2d8103360 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e9f0 tx=0x7fc2cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d57fa640 1 -- 172.21.15.116:0/135032783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc00cd80 con 0x7fc2d8103360 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d819d270 con 0x7fc2d8103360 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d57fa640 1 -- 172.21.15.116:0/135032783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2cc0108e0 con 0x7fc2d8103360 2024-03-19T02:49:00.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2d57fa640 1 -- 172.21.15.116:0/135032783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc018920 con 0x7fc2d8103360 2024-03-19T02:49:00.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.516+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d81a4b00 con 0x7fc2d8103360 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.518+0000 7fc2d57fa640 1 -- 172.21.15.116:0/135032783 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc2cc010a90 con 0x7fc2d8103360 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 Processor -- start 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 -- start start 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d57fa640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 0x7fc2c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 0x55e1509d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1509354b0 con 0x55e1509d28a0 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e1508ddd70 con 0x55e1509d1f80 2024-03-19T02:49:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2e1403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e150931550 con 0x55e1509dfbe0 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 0x7fc2c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d57fa640 1 -- 172.21.15.116:0/135032783 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc2cc0cf4f0 con 0x7fc2d8103360 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 0x55e1509d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51886/0 (socket says 172.21.15.116:51886) 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 -- 172.21.15.116:0/2312769409 learned_addr learned my addr 172.21.15.116:0/2312769409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 -- 172.21.15.116:0/2312769409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 msgr2=0x55e1509e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509e1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 -- 172.21.15.116:0/2312769409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 msgr2=0x55e1509d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 0x55e1509d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 -- 172.21.15.116:0/2312769409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e150900210 con 0x55e1509d28a0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 0x55e1509d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 0x7fc2c00a4e30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0004af0 tx=0x7fc2d0005fa0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.519+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0981f0 tx=0x7fc2cc010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709fa5aabfee2033 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2be7fc640 1 -- 172.21.15.116:0/2312769409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0a3f10 con 0x55e1509d28a0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2be7fc640 1 -- 172.21.15.116:0/2312769409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2cc099b20 con 0x55e1509d28a0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2be7fc640 1 -- 172.21.15.116:0/2312769409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc09ac70 con 0x55e1509d28a0 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 -- 172.21.15.116:0/2312769409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 msgr2=0x55e1509df6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0981f0 tx=0x7fc2cc010de0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 -- 172.21.15.116:0/2312769409 shutdown_connections 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509e1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509df6a0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 --2- 172.21.15.116:0/2312769409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d1f80 0x55e1509d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 -- 172.21.15.116:0/2312769409 >> 172.21.15.116:0/2312769409 conn(0x55e1509cca00 msgr2=0x55e1509ccde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 -- 172.21.15.116:0/2312769409 shutdown_connections 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.520+0000 7fc2e1403740 1 -- 172.21.15.116:0/2312769409 wait complete. 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 -- start start 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 Processor -- start 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509dcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509dd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1509e2960 con 0x55e1509dfbe0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e1508f1310 con 0x55e1509d28a0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1509e2ae0 con 0x55e1509d1f80 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2e1403740 10 client.0 _ll_register_callbacks cb 0x55e1508e52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51900/0 (socket says 172.21.15.116:51900) 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 -- 172.21.15.116:0/297647753 learned_addr learned my addr 172.21.15.116:0/297647753 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509dcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 -- 172.21.15.116:0/297647753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 msgr2=0x55e1509dcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509dcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 -- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 msgr2=0x55e1509dd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509dd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2dcdaf640 1 -- 172.21.15.116:0/297647753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1506493f0 con 0x55e1509dfbe0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.521+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009ca0 tx=0x7fc2d002c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeeddbb0c830db62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0053060 con 0x55e1509dfbe0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc2d0034850 con 0x55e1509dfbe0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2cc099b20 con 0x55e1509dfbe0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc2d005a050 con 0x55e1509dfbe0 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 10 client.21579 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e1507137d0 con 0x55e1509dfbe0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2cc099b20 con 0x55e1509dfbe0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc2d004ec00 con 0x55e1509dfbe0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 10 client.21579 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 10 client.21579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 20 client.21579 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1509efed0 0x55e1509f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 4 client.21579 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1509933e0 con 0x55e1509efed0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.522+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1509efed0 0x55e1509f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.523+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1509efed0 0x55e1509f22c0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8feaeca3510270ee server_cookie=e91788b26bc49c54 in_seq=0 out_seq=0 2024-03-19T02:49:00.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.523+0000 7fc2b77fe640 10 client.21579 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.523+0000 7fc2b77fe640 1 -- 172.21.15.116:0/297647753 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e1509933e0 con 0x55e1509efed0 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.523+0000 7fc2b77fe640 10 client.21579 handle_command_reply: tid=0 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.523+0000 7fc2e1403740 1 client.21579 shutdown 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 msgr2=0x55e1509dd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009ca0 tx=0x7fc2d002c180 comp rx=0 tx=0).stop 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 shutdown_connections 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e1509efed0 0x55e1509f22c0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509dfbe0 0x55e1509dd5b0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2b77fe640 0 client.21579 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 --2- 172.21.15.116:0/297647753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 >> 172.21.15.116:0/297647753 conn(0x55e1509cca00 msgr2=0x55e1509ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 shutdown_connections 2024-03-19T02:49:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 1 -- 172.21.15.116:0/297647753 wait complete. 2024-03-19T02:49:00.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.524+0000 7fc2e1403740 20 client.21579 trim_cache size 0 max 16384 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 Processor -- start 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 -- start start 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509eb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81aca10 con 0x55e1509d1f80 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81ae5b0 con 0x55e1509d28a0 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2e1403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81ad0e0 con 0x55e1509dfbe0 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51908/0 (socket says 172.21.15.116:51908) 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 -- 172.21.15.116:0/3329860978 learned_addr learned my addr 172.21.15.116:0/3329860978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 -- 172.21.15.116:0/3329860978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 msgr2=0x55e1509f7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509f7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.537+0000 7fc2d7fff640 1 -- 172.21.15.116:0/3329860978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 msgr2=0x55e1509eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2d7fff640 1 -- 172.21.15.116:0/3329860978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1509cd5d0 con 0x55e1509d1f80 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x55e1509f2b00 tx=0x7fc2c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7f0936ea7194cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2bdffb640 1 -- 172.21.15.116:0/3329860978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8003e20 con 0x55e1509d1f80 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2bdffb640 1 -- 172.21.15.116:0/3329860978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c80205a0 con 0x55e1509d1f80 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2bdffb640 1 -- 172.21.15.116:0/3329860978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8010cf0 con 0x55e1509d1f80 2024-03-19T02:49:00.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 -- 172.21.15.116:0/3329860978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 msgr2=0x55e1509df260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x55e1509f2b00 tx=0x7fc2c8005f90 comp rx=0 tx=0).stop 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 -- 172.21.15.116:0/3329860978 shutdown_connections 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509dfbe0 0x55e1509f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509d28a0 0x55e1509eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3329860978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d1f80 0x55e1509df260 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 -- 172.21.15.116:0/3329860978 >> 172.21.15.116:0/3329860978 conn(0x55e1509cca00 msgr2=0x55e1509cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 -- 172.21.15.116:0/3329860978 shutdown_connections 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.538+0000 7fc2e1403740 1 -- 172.21.15.116:0/3329860978 wait complete. 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 -- start start 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 Processor -- start 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509e7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509dfbe0 0x55e1509e2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81adee0 con 0x55e1509d28a0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81ad7b0 con 0x55e1509dfbe0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81aaa10 con 0x55e1509d1f80 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2e1403740 10 client.0 _ll_register_callbacks cb 0x7fc2d0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51912/0 (socket says 172.21.15.116:51912) 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 -- 172.21.15.116:0/3415154472 learned_addr learned my addr 172.21.15.116:0/3415154472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2dcdaf640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509dfbe0 0x55e1509e2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 -- 172.21.15.116:0/3415154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 msgr2=0x55e1509e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 -- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509dfbe0 msgr2=0x55e1509e2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:00.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.539+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509dfbe0 0x55e1509e2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2d77fe640 1 -- 172.21.15.116:0/3415154472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1509e3600 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2d77fe640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0cce60 tx=0x7fc2cc097490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5604dc755a397b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0afb50 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7fc2cc010430 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c80205a0 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc2cc010630 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 10 client.21585 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e150a55c60 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c80205a0 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fc2cc018ac0 con 0x55e1509d28a0 2024-03-19T02:49:00.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 10 client.21585 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 10 client.21585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 20 client.21585 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e150a56060 0x55e150a58450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 4 client.21585 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1509429f0 con 0x55e150a56060 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.540+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e150a56060 0x55e150a58450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2d7fff640 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e150a56060 0x55e150a58450 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5f63791779e3bfc server_cookie=d076a4e4f8b03f84 in_seq=0 out_seq=0 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2b77fe640 10 client.21585 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2b77fe640 1 -- 172.21.15.116:0/3415154472 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1509429f0 con 0x55e150a56060 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2b77fe640 10 client.21585 handle_command_reply: tid=0 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 client.21585 shutdown 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 msgr2=0x55e1509e7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc0cce60 tx=0x7fc2cc097490 comp rx=0 tx=0).stop 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 shutdown_connections 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e150a56060 0x55e150a58450 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e1509dfbe0 0x55e1509e2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1509d28a0 0x55e1509e7e80 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2b77fe640 0 client.21585 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 --2- 172.21.15.116:0/3415154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1509d1f80 0x55e1509e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 >> 172.21.15.116:0/3415154472 conn(0x55e1509cca00 msgr2=0x55e1509dcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 shutdown_connections 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 1 -- 172.21.15.116:0/3415154472 wait complete. 2024-03-19T02:49:00.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.541+0000 7fc2e1403740 20 client.21585 trim_cache size 0 max 16384 2024-03-19T02:49:00.551 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 msgr2=0x7fc2c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 0x7fc2c00a4e30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0004af0 tx=0x7fc2d0005fa0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 msgr2=0x7fc2d81984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00e9f0 tx=0x7fc2cc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 shutdown_connections 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc2c00a2970 0x7fc2c00a4e30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2d810bd80 0x7fc2d819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d8103360 0x7fc2d81984d0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 --2- 172.21.15.116:0/135032783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc2d8102a40 0x7fc2d8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 >> 172.21.15.116:0/135032783 conn(0x7fc2d80778d0 msgr2=0x7fc2d80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:00.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 shutdown_connections 2024-03-19T02:49:00.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:00.542+0000 7fc2de812640 1 -- 172.21.15.116:0/135032783 wait complete. 2024-03-19T02:49:01.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:00 smithi162 ceph-mon[35761]: pgmap v668: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:49:01.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:00 smithi154 ceph-mon[35693]: pgmap v668: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:49:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:00 smithi116 ceph-mon[32263]: pgmap v668: 129 pgs: 129 active+clean; 994 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 141 op/s 2024-03-19T02:49:01.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55822 5.87 MB/sec execute 383 sec latency 4759.561 ms 2024-03-19T02:49:01.554 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:49:01.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:49:01.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:49:01.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 -- 172.21.15.116:0/78760895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 msgr2=0x7f6ffc106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 --2- 172.21.15.116:0/78760895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc106a80 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009930 tx=0x7f6ff802bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:01.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 -- 172.21.15.116:0/78760895 shutdown_connections 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 --2- 172.21.15.116:0/78760895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 --2- 172.21.15.116:0/78760895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 0x7f6ffc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 --2- 172.21.15.116:0/78760895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc106a80 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 -- 172.21.15.116:0/78760895 >> 172.21.15.116:0/78760895 conn(0x7f6ffc077820 msgr2=0x7f6ffc077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 -- 172.21.15.116:0/78760895 shutdown_connections 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.680+0000 7f7003abf640 1 -- 172.21.15.116:0/78760895 wait complete. 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 Processor -- start 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 -- start start 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 0x7f6ffc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc110960 con 0x7f6ffc1066a0 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1107e0 con 0x7f6ffc100690 2024-03-19T02:49:01.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f7003abf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc110ae0 con 0x7f6ffc101050 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f700105a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f700185b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 0x7f6ffc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f700205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f700105a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43714/0 (socket says 172.21.15.116:43714) 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.681+0000 7f700105a640 1 -- 172.21.15.116:0/1331386174 learned_addr learned my addr 172.21.15.116:0/1331386174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 -- 172.21.15.116:0/1331386174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 msgr2=0x7f6ffc198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 -- 172.21.15.116:0/1331386174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 msgr2=0x7f6ffc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 0x7f6ffc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 -- 172.21.15.116:0/1331386174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc19cfc0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700105a640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f700205c640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc19c890 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009fd0 tx=0x7f6ff400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f6ff2ffd640 1 -- 172.21.15.116:0/1331386174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4004b20 con 0x7f6ffc1066a0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ffc19d2b0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f6ff2ffd640 1 -- 172.21.15.116:0/1331386174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ff4004cc0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f6ff2ffd640 1 -- 172.21.15.116:0/1331386174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40053c0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.682+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ffc0fd5f0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.685+0000 7f6ff2ffd640 1 -- 172.21.15.116:0/1331386174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6ff40057f0 con 0x7f6ffc1066a0 2024-03-19T02:49:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 Processor -- start 2024-03-19T02:49:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 -- start start 2024-03-19T02:49:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f6ff2ffd640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 0x7f6fe80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f681ca0 0x561d4f68a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f6ff2ffd640 1 -- 172.21.15.116:0/1331386174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f6ff40d3280 con 0x7f6ffc1066a0 2024-03-19T02:49:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f68a880 0x561d4f68cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f700105a640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 0x7f6fe80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561d4f5e5160 con 0x561d4f681ca0 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d4f58de10 con 0x561d4f68a880 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.686+0000 7f7006789740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d4f5e1200 con 0x561d4f681380 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43718/0 (socket says 172.21.15.116:43718) 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 -- 172.21.15.116:0/1999279066 learned_addr learned my addr 172.21.15.116:0/1999279066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700205c640 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f68a880 0x561d4f68cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 -- 172.21.15.116:0/1999279066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f68a880 msgr2=0x561d4f68cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f68a880 0x561d4f68cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 -- 172.21.15.116:0/1999279066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f681ca0 msgr2=0x561d4f68a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f681ca0 0x561d4f68a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 -- 172.21.15.116:0/1999279066 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d4f3c17e0 con 0x561d4f681380 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700105a640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 0x7f6fe80a4e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009620 tx=0x7f6fec009530 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f700185b640 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8036040 tx=0x7f6ff8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=377a3aed157d0bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f6fe7fff640 1 -- 172.21.15.116:0/1999279066 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8045020 con 0x561d4f681380 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f6fe7fff640 1 -- 172.21.15.116:0/1999279066 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ff802c7e0 con 0x561d4f681380 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.687+0000 7f6fe7fff640 1 -- 172.21.15.116:0/1999279066 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8005450 con 0x561d4f681380 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 -- 172.21.15.116:0/1999279066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 msgr2=0x561d4f681760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8036040 tx=0x7f6ff8009560 comp rx=0 tx=0).stop 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 -- 172.21.15.116:0/1999279066 shutdown_connections 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f68a880 0x561d4f68cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f681ca0 0x561d4f68a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 --2- 172.21.15.116:0/1999279066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f681760 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 -- 172.21.15.116:0/1999279066 >> 172.21.15.116:0/1999279066 conn(0x561d4f67bab0 msgr2=0x561d4f67be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 -- 172.21.15.116:0/1999279066 shutdown_connections 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.688+0000 7f7006789740 1 -- 172.21.15.116:0/1999279066 wait complete. 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 -- start start 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 Processor -- start 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 0x561d4f687dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f688310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561d4f68d610 con 0x561d4f68a880 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d4f5a11a0 con 0x561d4f681ca0 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561d4f68d790 con 0x561d4f681380 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f7006789740 10 client.0 _ll_register_callbacks cb 0x561d4f2f8c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f700205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f688310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f700185b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f700105a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 0x561d4f687dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f700185b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43734/0 (socket says 172.21.15.116:43734) 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.689+0000 7f700185b640 1 -- 172.21.15.116:0/3462169954 learned_addr learned my addr 172.21.15.116:0/3462169954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 -- 172.21.15.116:0/3462169954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 msgr2=0x561d4f687890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 -- 172.21.15.116:0/3462169954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 msgr2=0x561d4f688310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f688310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 -- 172.21.15.116:0/3462169954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d4f421f70 con 0x561d4f681ca0 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700185b640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700205c640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f688310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f700105a640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 0x561d4f687dd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0097f0 tx=0x7f6fec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d634ec85007cc71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f6fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec00f5c0 con 0x561d4f681ca0 2024-03-19T02:49:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff802c7e0 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f6fec00f760 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6fec00f960 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.690+0000 7f7006789740 10 client.31322 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561d4f2b9170 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ff802c7e0 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f6fec010bf0 con 0x561d4f681ca0 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 10 client.31322 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 10 client.31322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 20 client.31322 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f69ac30 0x561d4f69d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 4 client.31322 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d4f5f26a0 con 0x561d4f69ac30 2024-03-19T02:49:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.691+0000 7f700185b640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f69ac30 0x561d4f69d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.692+0000 7f700185b640 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f69ac30 0x561d4f69d050 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3666b5aa48e130d server_cookie=13fd04b8c08e5ea2 in_seq=0 out_seq=0 2024-03-19T02:49:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.692+0000 7f6fe57fa640 10 client.31322 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f6fe57fa640 1 -- 172.21.15.116:0/3462169954 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d4f5f26a0 con 0x561d4f69ac30 2024-03-19T02:49:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f6fe57fa640 10 client.31322 handle_command_reply: tid=0 2024-03-19T02:49:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 client.31322 shutdown 2024-03-19T02:49:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 msgr2=0x561d4f687dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 0x561d4f687dd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0097f0 tx=0x7f6fec0046c0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 shutdown_connections 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f69ac30 0x561d4f69d050 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f688310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681ca0 0x561d4f687dd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f6fe57fa640 0 client.31322 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 --2- 172.21.15.116:0/3462169954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681380 0x561d4f687890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 >> 172.21.15.116:0/3462169954 conn(0x561d4f67bab0 msgr2=0x561d4f67d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 shutdown_connections 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 1 -- 172.21.15.116:0/3462169954 wait complete. 2024-03-19T02:49:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.695+0000 7f7006789740 20 client.31322 trim_cache size 0 max 16384 2024-03-19T02:49:01.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 Processor -- start 2024-03-19T02:49:01.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 -- start start 2024-03-19T02:49:01.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f6963b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f6a2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1ac9f0 con 0x561d4f68a880 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1ae590 con 0x561d4f681380 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f7006789740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1ad0c0 con 0x561d4f681ca0 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f6a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700105a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f6963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700185b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700185b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45470/0 (socket says 172.21.15.116:45470) 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700105a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f6963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43746/0 (socket says 172.21.15.116:43746) 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700185b640 1 -- 172.21.15.116:0/2974813901 learned_addr learned my addr 172.21.15.116:0/2974813901 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 -- 172.21.15.116:0/2974813901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 msgr2=0x561d4f6963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f6963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 -- 172.21.15.116:0/2974813901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 msgr2=0x561d4f689fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700205c640 1 -- 172.21.15.116:0/2974813901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d4f681ad0 con 0x561d4f68a880 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.716+0000 7f700185b640 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f700205c640 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f6a2920 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40d05f0 tx=0x7f6ff401b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5381c9ab0db828af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f6fe77fe640 1 -- 172.21.15.116:0/2974813901 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40d30c0 con 0x561d4f68a880 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f6fe77fe640 1 -- 172.21.15.116:0/2974813901 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ff409f5b0 con 0x561d4f68a880 2024-03-19T02:49:01.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f6fe77fe640 1 -- 172.21.15.116:0/2974813901 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40a33c0 con 0x561d4f68a880 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f7006789740 1 -- 172.21.15.116:0/2974813901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 msgr2=0x561d4f6a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.717+0000 7f7006789740 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f6a2920 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40d05f0 tx=0x7f6ff401b040 comp rx=0 tx=0).stop 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- 172.21.15.116:0/2974813901 shutdown_connections 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f6a2920 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f6963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- 172.21.15.116:0/2974813901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f689fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- 172.21.15.116:0/2974813901 >> 172.21.15.116:0/2974813901 conn(0x561d4f67bab0 msgr2=0x561d4f6880e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- 172.21.15.116:0/2974813901 shutdown_connections 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- 172.21.15.116:0/2974813901 wait complete. 2024-03-19T02:49:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- start start 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 Processor -- start 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f684600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f684b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1adec0 con 0x561d4f68a880 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1ad790 con 0x561d4f681380 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc1aa9f0 con 0x561d4f681ca0 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.718+0000 7f7006789740 10 client.0 _ll_register_callbacks cb 0x7f6fec0051a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700105a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f684b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700185b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f684600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51960/0 (socket says 172.21.15.116:51960) 2024-03-19T02:49:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 -- 172.21.15.116:0/146253974 learned_addr learned my addr 172.21.15.116:0/146253974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 -- 172.21.15.116:0/146253974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 msgr2=0x561d4f684b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f684b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 -- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 msgr2=0x561d4f684600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f684600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 -- 172.21.15.116:0/146253974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d4f7001a0 con 0x561d4f68a880 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700105a640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f684b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f700205c640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f6ff409bf80 tx=0x7f6ff409be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e17eb753068fa0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f6fe57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40a3360 con 0x561d4f68a880 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f6ff4019450 con 0x561d4f68a880 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff409f5b0 con 0x561d4f68a880 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6ff4086050 con 0x561d4f68a880 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f7006789740 10 client.21600 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x561d4f701480 con 0x561d4f68a880 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.719+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ff409f5b0 con 0x561d4f68a880 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f6ff4088af0 con 0x561d4f68a880 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 10 client.21600 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 10 client.21600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 20 client.21600 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f701ba0 0x561d4f703fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 4 client.21600 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561d4f5ec9f0 con 0x561d4f701ba0 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f700185b640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f701ba0 0x561d4f703fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f700185b640 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f701ba0 0x561d4f703fc0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb50d10a2057ebee server_cookie=f05a957bd4b17286 in_seq=0 out_seq=0 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.720+0000 7f6fe57fa640 10 client.21600 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f6fe57fa640 1 -- 172.21.15.116:0/146253974 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561d4f5ec9f0 con 0x561d4f701ba0 2024-03-19T02:49:01.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f6fe57fa640 10 client.21600 handle_command_reply: tid=0 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 client.21600 shutdown 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 msgr2=0x561d4f685080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f6ff409bf80 tx=0x7f6ff409be00 comp rx=0 tx=0).stop 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 shutdown_connections 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561d4f701ba0 0x561d4f703fc0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561d4f68a880 0x561d4f685080 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561d4f681ca0 0x561d4f684b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 --2- 172.21.15.116:0/146253974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d4f681380 0x561d4f684600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f6fe57fa640 0 client.21600 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 >> 172.21.15.116:0/146253974 conn(0x561d4f67bab0 msgr2=0x561d4f67a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 shutdown_connections 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 1 -- 172.21.15.116:0/146253974 wait complete. 2024-03-19T02:49:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.723+0000 7f7006789740 20 client.21600 trim_cache size 0 max 16384 2024-03-19T02:49:01.730 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:49:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:49:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "728423ef-bc05-42b2-9322-7eb66834d23c", 2024-03-19T02:49:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:49:01.730 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 msgr2=0x7f6fe80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 0x7f6fe80a4e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009620 tx=0x7f6fec009530 comp rx=0 tx=0).stop 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 msgr2=0x7f6ffc19c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc19c890 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009fd0 tx=0x7f6ff400b690 comp rx=0 tx=0).stop 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 shutdown_connections 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6fe80a2970 0x7f6fe80a4e30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ffc1066a0 0x7f6ffc19c890 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ffc101050 0x7f6ffc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 --2- 172.21.15.116:0/1331386174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ffc100690 0x7f6ffc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 >> 172.21.15.116:0/1331386174 conn(0x7f6ffc077820 msgr2=0x7f6ffc0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 shutdown_connections 2024-03-19T02:49:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.724+0000 7f7003abf640 1 -- 172.21.15.116:0/1331386174 wait complete. 2024-03-19T02:49:01.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:49:01.737 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:49:01.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 -- 172.21.15.116:0/582215905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f141099d0 msgr2=0x7f7f14109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 --2- 172.21.15.116:0/582215905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f141099d0 0x7f7f14109db0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009930 tx=0x7f7f0402bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:01.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 -- 172.21.15.116:0/582215905 shutdown_connections 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 --2- 172.21.15.116:0/582215905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f14104380 0x7f7f14110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 --2- 172.21.15.116:0/582215905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141039c0 0x7f7f14103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 --2- 172.21.15.116:0/582215905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f141099d0 0x7f7f14109db0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 -- 172.21.15.116:0/582215905 >> 172.21.15.116:0/582215905 conn(0x7f7f140ff490 msgr2=0x7f7f141018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 -- 172.21.15.116:0/582215905 shutdown_connections 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.857+0000 7f7f18f80640 1 -- 172.21.15.116:0/582215905 wait complete. 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 Processor -- start 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 -- start start 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141099d0 0x7f7f1419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14113ce0 con 0x7f7f14104380 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14113b60 con 0x7f7f141099d0 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f18f80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14113e60 con 0x7f7f141039c0 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43776/0 (socket says 172.21.15.116:43776) 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f1259c640 1 -- 172.21.15.116:0/4093368731 learned_addr learned my addr 172.21.15.116:0/4093368731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51966/0 (socket says 172.21.15.116:51966) 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141099d0 0x7f7f1419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 -- 172.21.15.116:0/4093368731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 msgr2=0x7f7f1419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 -- 172.21.15.116:0/4093368731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141099d0 msgr2=0x7f7f1419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141099d0 0x7f7f1419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f0bfff640 1 -- 172.21.15.116:0/4093368731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1419f210 con 0x7f7f14104380 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.858+0000 7f7f1259c640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e9f0 tx=0x7f7efc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/4093368731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00ccc0 con 0x7f7f14104380 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1419f500 con 0x7f7f14104380 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/4093368731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7efc0108e0 con 0x7f7f14104380 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/4093368731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc018880 con 0x7f7f14104380 2024-03-19T02:49:01.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.859+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f141a6d90 con 0x7f7f14104380 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.861+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/4093368731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7efc010a90 con 0x7f7f14104380 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.861+0000 7f7f1bb68740 1 Processor -- start 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 -- start start 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f73d204500 con 0x55f73d2a1350 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f73d1acbe0 con 0x55f73d2a0a30 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f73d2005a0 con 0x55f73d2a9f30 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f0b7fe640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 0x7f7f000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f0bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f0bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43784/0 (socket says 172.21.15.116:43784) 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1854007609 learned_addr learned my addr 172.21.15.116:0/1854007609 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 0x7f7f000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.862+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 -- 172.21.15.116:0/1854007609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 msgr2=0x55f73d2ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 -- 172.21.15.116:0/1854007609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 msgr2=0x55f73d2a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 -- 172.21.15.116:0/1854007609 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f73d1ce000 con 0x55f73d2a1350 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/4093368731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7efc0cf5b0 con 0x7f7f14104380 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f1259c640 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a99f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f7f04000c00 tx=0x7f7f04009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7911a1e984f50e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 0x7f7f000a4e30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c009d70 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f08ff9640 1 -- 172.21.15.116:0/1854007609 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04045070 con 0x55f73d2a1350 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f08ff9640 1 -- 172.21.15.116:0/1854007609 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f0402c7e0 con 0x55f73d2a1350 2024-03-19T02:49:01.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.863+0000 7f7f08ff9640 1 -- 172.21.15.116:0/1854007609 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04003a30 con 0x55f73d2a1350 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.864+0000 7f7f1bb68740 1 -- 172.21.15.116:0/1854007609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 msgr2=0x55f73d2a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.864+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a99f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f7f04000c00 tx=0x7f7f04009670 comp rx=0 tx=0).stop 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 -- 172.21.15.116:0/1854007609 shutdown_connections 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a99f0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/1854007609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 -- 172.21.15.116:0/1854007609 >> 172.21.15.116:0/1854007609 conn(0x55f73d29b160 msgr2=0x55f73d29b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 -- 172.21.15.116:0/1854007609 shutdown_connections 2024-03-19T02:49:01.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.866+0000 7f7f1bb68740 1 -- 172.21.15.116:0/1854007609 wait complete. 2024-03-19T02:49:01.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.867+0000 7f7f1bb68740 1 -- start start 2024-03-19T02:49:01.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.867+0000 7f7f1bb68740 1 Processor -- start 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2a7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f73d2accc0 con 0x55f73d2a1350 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f73d1c0160 con 0x55f73d2a0a30 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f73d2ace40 con 0x55f73d2a9f30 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1bb68740 10 client.0 _ll_register_callbacks cb 0x55f73d1fca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:01.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:45518/0 (socket says 172.21.15.116:45518) 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.868+0000 7f7f1259c640 1 -- 172.21.15.116:0/2421570408 learned_addr learned my addr 172.21.15.116:0/2421570408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 -- 172.21.15.116:0/2421570408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 msgr2=0x55f73d2a7930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2a7930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 -- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 msgr2=0x55f73d2a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 -- 172.21.15.116:0/2421570408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f73cfe27d0 con 0x55f73d2a1350 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a73f0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x55f73d29c2f0 tx=0x7f7efc09b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dab23f50711d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.869+0000 7f7ef9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7f1259c640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0862b0 con 0x55f73d2a1350 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7efc0a3b40 con 0x55f73d2a1350 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0402c7e0 con 0x55f73d2a1350 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7efc010e00 con 0x55f73d2a1350 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7f1bb68740 10 client.21612 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:01.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f73cf183f0 con 0x55f73d2a1350 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f0402c7e0 con 0x55f73d2a1350 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.870+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7efc0cf1f0 con 0x55f73d2a1350 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 10 client.21612 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 10 client.21612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 20 client.21612 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d2c2940 0x55f73d2c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 4 client.21612 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:01.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.871+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f73d211a40 con 0x55f73d2c2940 2024-03-19T02:49:01.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.872+0000 7f7f1259c640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d2c2940 0x55f73d2c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.874+0000 7f7f1259c640 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d2c2940 0x55f73d2c4d50 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93ebc52b680ea459 server_cookie=d2b678ecc8a3a423 in_seq=0 out_seq=0 2024-03-19T02:49:01.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.874+0000 7f7ef9ffb640 10 client.21612 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.882+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/2421570408 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f73d211a40 con 0x55f73d2c2940 2024-03-19T02:49:01.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.882+0000 7f7ef9ffb640 10 client.21612 handle_command_reply: tid=0 2024-03-19T02:49:01.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.882+0000 7f7f1bb68740 1 client.21612 shutdown 2024-03-19T02:49:01.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.883+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 msgr2=0x55f73d2a73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.883+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a73f0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x55f73d29c2f0 tx=0x7f7efc09b770 comp rx=0 tx=0).stop 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 shutdown_connections 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d2c2940 0x55f73d2c4d50 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2a7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7ef9ffb640 0 client.21612 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a1350 0x55f73d2a73f0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/2421570408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a0a30 0x55f73d2a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 >> 172.21.15.116:0/2421570408 conn(0x55f73d29b160 msgr2=0x55f73d29ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.884+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 shutdown_connections 2024-03-19T02:49:01.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.885+0000 7f7f1bb68740 1 -- 172.21.15.116:0/2421570408 wait complete. 2024-03-19T02:49:01.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.886+0000 7f7f1bb68740 20 client.21612 trim_cache size 0 max 16384 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 Processor -- start 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 -- start start 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a0a30 0x55f73d2a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2be2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141acad0 con 0x55f73d2a9f30 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ae670 con 0x55f73d2a1350 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ad1a0 con 0x55f73d2a0a30 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51990/0 (socket says 172.21.15.116:51990) 2024-03-19T02:49:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a0a30 0x55f73d2a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 -- 172.21.15.116:0/188940133 learned_addr learned my addr 172.21.15.116:0/188940133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2be2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 -- 172.21.15.116:0/188940133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a0a30 msgr2=0x55f73d2a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a0a30 0x55f73d2a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 -- 172.21.15.116:0/188940133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 msgr2=0x55f73d2be2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2be2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.910+0000 7f7f12d9d640 1 -- 172.21.15.116:0/188940133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f73ced80f0 con 0x55f73d2a9f30 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0132a0 tx=0x7f7f0c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecba0a7e21294761 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7efbfff640 1 -- 172.21.15.116:0/188940133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c024070 con 0x55f73d2a9f30 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7efbfff640 1 -- 172.21.15.116:0/188940133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f0c0132d0 con 0x55f73d2a9f30 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7efbfff640 1 -- 172.21.15.116:0/188940133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c010690 con 0x55f73d2a9f30 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7f1bb68740 1 -- 172.21.15.116:0/188940133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 msgr2=0x55f73d2ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.911+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0132a0 tx=0x7f7f0c01e000 comp rx=0 tx=0).stop 2024-03-19T02:49:01.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- 172.21.15.116:0/188940133 shutdown_connections 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a9f30 0x55f73d2ca5f0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2be2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/188940133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a0a30 0x55f73d2a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- 172.21.15.116:0/188940133 >> 172.21.15.116:0/188940133 conn(0x55f73d29b160 msgr2=0x55f73d29a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- 172.21.15.116:0/188940133 shutdown_connections 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- 172.21.15.116:0/188940133 wait complete. 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- start start 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 Processor -- start 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141adfa0 con 0x55f73d2a0a30 2024-03-19T02:49:01.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ad870 con 0x55f73d2a1350 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141aaad0 con 0x55f73d2a9f30 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1bb68740 10 client.0 _ll_register_callbacks cb 0x7f7efc00ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52000/0 (socket says 172.21.15.116:52000) 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f1259c640 1 -- 172.21.15.116:0/3253403826 learned_addr learned my addr 172.21.15.116:0/3253403826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f12d9d640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.912+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 -- 172.21.15.116:0/3253403826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 msgr2=0x55f73d2b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 -- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 msgr2=0x55f73d2b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 -- 172.21.15.116:0/3253403826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f73d2b83a0 con 0x55f73d2a0a30 2024-03-19T02:49:01.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1259c640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f7f04036040 tx=0x7f7f0402c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a00acfadcb8db11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04059070 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 9510+0+0 (secure 0 0 0) 0x7f7f0404a860 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0c0132d0 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7f04062050 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 10 client.21618 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f73d3299e0 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f0c0132d0 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7f0405c020 con 0x55f73d2a0a30 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 10 client.21618 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 10 client.21618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 20 client.21618 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d329d40 0x55f73d32c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 4 client.21618 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:49:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.913+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f73d20bd90 con 0x55f73d329d40 2024-03-19T02:49:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.914+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d329d40 0x55f73d32c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.914+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d329d40 0x55f73d32c130 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd08b0dbd70726cf server_cookie=a24554386227c11 in_seq=0 out_seq=0 2024-03-19T02:49:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.914+0000 7f7ef9ffb640 10 client.21618 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7ef9ffb640 1 -- 172.21.15.116:0/3253403826 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f73d20bd90 con 0x55f73d329d40 2024-03-19T02:49:01.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7ef9ffb640 10 client.21618 handle_command_reply: tid=0 2024-03-19T02:49:01.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 client.21618 shutdown 2024-03-19T02:49:01.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 msgr2=0x55f73d2ba920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f7f04036040 tx=0x7f7f0402c2b0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 shutdown_connections 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f73d329d40 0x55f73d32c130 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f73d2a9f30 0x55f73d2b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f73d2a1350 0x55f73d2b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 --2- 172.21.15.116:0/3253403826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f73d2a0a30 0x55f73d2ba920 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7ef9ffb640 0 client.21618 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 >> 172.21.15.116:0/3253403826 conn(0x55f73d29b160 msgr2=0x55f73d2a6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 shutdown_connections 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 1 -- 172.21.15.116:0/3253403826 wait complete. 2024-03-19T02:49:01.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.918+0000 7f7f1bb68740 20 client.21618 trim_cache size 0 max 16384 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (456 inodes in the stack)", 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "728423ef-bc05-42b2-9322-7eb66834d23c": { 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "728423ef-bc05-42b2-9322-7eb66834d23c", 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:49:01.924 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:49:01.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 msgr2=0x7f7f000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 0x7f7f000a4e30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c009d70 comp rx=0 tx=0).stop 2024-03-19T02:49:01.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 msgr2=0x7f7f1419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:01.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e9f0 tx=0x7f7efc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 shutdown_connections 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f000a2970 0x7f7f000a4e30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f141099d0 0x7f7f1419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f14104380 0x7f7f1419a760 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 --2- 172.21.15.116:0/4093368731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f141039c0 0x7f7f1419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 >> 172.21.15.116:0/4093368731 conn(0x7f7f140ff490 msgr2=0x7f7f140ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 shutdown_connections 2024-03-19T02:49:01.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:01.919+0000 7f7f18f80640 1 -- 172.21.15.116:0/4093368731 wait complete. 2024-03-19T02:49:01.931 INFO:tasks.cephfs.filesystem:scrub status for tag:728423ef-bc05-42b2-9322-7eb66834d23c - {'path': '/', 'tag': '728423ef-bc05-42b2-9322-7eb66834d23c', 'options': 'recursive,force'} 2024-03-19T02:49:02.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 55948 5.89 MB/sec execute 384 sec latency 368.076 ms 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: pgmap v669: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.169 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:02 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: pgmap v669: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:03.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:02 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: pgmap v669: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:03.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:02 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:03.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56091 5.90 MB/sec execute 385 sec latency 134.837 ms 2024-03-19T02:49:04.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56243 5.91 MB/sec execute 386 sec latency 160.919 ms 2024-03-19T02:49:05.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:04 smithi162 ceph-mon[35761]: pgmap v670: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T02:49:05.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:04 smithi154 ceph-mon[35693]: pgmap v670: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T02:49:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:04 smithi116 ceph-mon[32263]: pgmap v670: 129 pgs: 129 active+clean; 1008 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T02:49:05.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56336 5.91 MB/sec execute 387 sec latency 195.281 ms 2024-03-19T02:49:06.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:05 smithi162 ceph-mon[35761]: osdmap e126: 12 total, 12 up, 12 in 2024-03-19T02:49:06.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:05 smithi154 ceph-mon[35693]: osdmap e126: 12 total, 12 up, 12 in 2024-03-19T02:49:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:05 smithi116 ceph-mon[32263]: osdmap e126: 12 total, 12 up, 12 in 2024-03-19T02:49:06.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56413 5.91 MB/sec execute 388 sec latency 143.067 ms 2024-03-19T02:49:07.167 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:06 smithi162 ceph-mon[35761]: pgmap v672: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-19T02:49:07.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:06 smithi154 ceph-mon[35693]: pgmap v672: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-19T02:49:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:06 smithi116 ceph-mon[32263]: pgmap v672: 129 pgs: 2 active+clean+snaptrim, 121 active+clean, 6 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 204 op/s 2024-03-19T02:49:07.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56471 5.90 MB/sec execute 389 sec latency 152.109 ms 2024-03-19T02:49:08.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56564 5.90 MB/sec execute 390 sec latency 129.461 ms 2024-03-19T02:49:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:08 smithi154 ceph-mon[35693]: pgmap v673: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:08 smithi116 ceph-mon[32263]: pgmap v673: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:09.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56628 5.89 MB/sec execute 391 sec latency 139.003 ms 2024-03-19T02:49:09.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:08 smithi162 ceph-mon[35761]: pgmap v673: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:10.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56759 5.91 MB/sec execute 392 sec latency 153.968 ms 2024-03-19T02:49:11.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:10 smithi154 ceph-mon[35693]: pgmap v674: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:11.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:10 smithi116 ceph-mon[32263]: pgmap v674: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:11.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56842 5.91 MB/sec execute 393 sec latency 169.882 ms 2024-03-19T02:49:11.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:10 smithi162 ceph-mon[35761]: pgmap v674: 129 pgs: 3 active+clean+snaptrim, 116 active+clean, 10 active+clean+snaptrim_wait; 1.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 208 op/s 2024-03-19T02:49:12.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 56949 5.92 MB/sec execute 394 sec latency 146.039 ms 2024-03-19T02:49:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:12 smithi154 ceph-mon[35693]: pgmap v675: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-03-19T02:49:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:12 smithi154 ceph-mon[35693]: osdmap e127: 12 total, 12 up, 12 in 2024-03-19T02:49:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:12 smithi116 ceph-mon[32263]: pgmap v675: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-03-19T02:49:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:12 smithi116 ceph-mon[32263]: osdmap e127: 12 total, 12 up, 12 in 2024-03-19T02:49:13.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57028 5.92 MB/sec execute 395 sec latency 1012.435 ms 2024-03-19T02:49:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:12 smithi162 ceph-mon[35761]: pgmap v675: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-03-19T02:49:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:12 smithi162 ceph-mon[35761]: osdmap e127: 12 total, 12 up, 12 in 2024-03-19T02:49:14.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57094 5.91 MB/sec execute 396 sec latency 2012.461 ms 2024-03-19T02:49:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:14 smithi154 ceph-mon[35693]: pgmap v677: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.9 MiB/s wr, 118 op/s 2024-03-19T02:49:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:14 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:14 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:15.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:14 smithi116 ceph-mon[32263]: pgmap v677: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.9 MiB/s wr, 118 op/s 2024-03-19T02:49:15.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:14 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:15.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:14 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:15.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57154 5.90 MB/sec execute 397 sec latency 3012.486 ms 2024-03-19T02:49:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:14 smithi162 ceph-mon[35761]: pgmap v677: 129 pgs: 129 active+clean; 1017 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.9 MiB/s wr, 118 op/s 2024-03-19T02:49:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:14 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:14 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:16.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:15 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:16.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:16.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:16.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:15 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:16.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57214 5.89 MB/sec execute 398 sec latency 3538.438 ms 2024-03-19T02:49:16.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:16.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:16.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:15 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:16 smithi154 ceph-mon[35693]: pgmap v678: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 132 op/s 2024-03-19T02:49:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:16 smithi116 ceph-mon[32263]: pgmap v678: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 132 op/s 2024-03-19T02:49:17.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57280 5.88 MB/sec execute 399 sec latency 947.488 ms 2024-03-19T02:49:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:16 smithi162 ceph-mon[35761]: pgmap v678: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 132 op/s 2024-03-19T02:49:18.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57353 5.87 MB/sec execute 400 sec latency 1947.514 ms 2024-03-19T02:49:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:18 smithi154 ceph-mon[35693]: pgmap v679: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:18 smithi116 ceph-mon[32263]: pgmap v679: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:19.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57420 5.85 MB/sec execute 401 sec latency 2947.541 ms 2024-03-19T02:49:19.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:18 smithi162 ceph-mon[35761]: pgmap v679: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:20.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57497 5.85 MB/sec execute 402 sec latency 3947.567 ms 2024-03-19T02:49:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:20 smithi154 ceph-mon[35693]: pgmap v680: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:20 smithi116 ceph-mon[32263]: pgmap v680: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:21.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57582 5.84 MB/sec execute 403 sec latency 4463.366 ms 2024-03-19T02:49:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:20 smithi162 ceph-mon[35761]: pgmap v680: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 129 op/s 2024-03-19T02:49:22.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57684 5.83 MB/sec execute 404 sec latency 148.112 ms 2024-03-19T02:49:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:22 smithi154 ceph-mon[35693]: pgmap v681: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 122 op/s 2024-03-19T02:49:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:22 smithi116 ceph-mon[32263]: pgmap v681: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 122 op/s 2024-03-19T02:49:23.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57785 5.82 MB/sec execute 405 sec latency 162.353 ms 2024-03-19T02:49:23.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:22 smithi162 ceph-mon[35761]: pgmap v681: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 122 op/s 2024-03-19T02:49:24.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 57906 5.83 MB/sec execute 406 sec latency 142.936 ms 2024-03-19T02:49:25.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:24 smithi162 ceph-mon[35761]: pgmap v682: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-03-19T02:49:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:24 smithi154 ceph-mon[35693]: pgmap v682: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-03-19T02:49:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:24 smithi116 ceph-mon[32263]: pgmap v682: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-03-19T02:49:25.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58021 5.84 MB/sec execute 407 sec latency 182.798 ms 2024-03-19T02:49:26.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58124 5.83 MB/sec execute 408 sec latency 160.691 ms 2024-03-19T02:49:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:26 smithi154 ceph-mon[35693]: pgmap v683: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 199 op/s 2024-03-19T02:49:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:26 smithi116 ceph-mon[32263]: pgmap v683: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 199 op/s 2024-03-19T02:49:27.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58257 5.83 MB/sec execute 409 sec latency 143.241 ms 2024-03-19T02:49:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:26 smithi162 ceph-mon[35761]: pgmap v683: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.1 MiB/s wr, 199 op/s 2024-03-19T02:49:28.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58375 5.84 MB/sec execute 410 sec latency 138.819 ms 2024-03-19T02:49:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:28 smithi154 ceph-mon[35693]: pgmap v684: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:28 smithi116 ceph-mon[32263]: pgmap v684: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:29.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58505 5.85 MB/sec execute 411 sec latency 144.483 ms 2024-03-19T02:49:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:28 smithi162 ceph-mon[35761]: pgmap v684: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:30.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58623 5.84 MB/sec execute 412 sec latency 151.682 ms 2024-03-19T02:49:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:30 smithi154 ceph-mon[35693]: pgmap v685: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:30 smithi116 ceph-mon[32263]: pgmap v685: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:31.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58714 5.84 MB/sec execute 413 sec latency 128.946 ms 2024-03-19T02:49:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:30 smithi162 ceph-mon[35761]: pgmap v685: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 146 op/s 2024-03-19T02:49:31.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:49:31.932 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:49:32.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 -- 172.21.15.116:0/3403723671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad8100f60 msgr2=0x7f6ad810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 --2- 172.21.15.116:0/3403723671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad8100f60 0x7f6ad810d420 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009ca0 tx=0x7f6ad002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 -- 172.21.15.116:0/3403723671 shutdown_connections 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 --2- 172.21.15.116:0/3403723671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad8100f60 0x7f6ad810d420 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 --2- 172.21.15.116:0/3403723671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad81005a0 0x7f6ad8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 --2- 172.21.15.116:0/3403723671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 0x7f6ad8106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 -- 172.21.15.116:0/3403723671 >> 172.21.15.116:0/3403723671 conn(0x7f6ad80fc070 msgr2=0x7f6ad80fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 -- 172.21.15.116:0/3403723671 shutdown_connections 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.070+0000 7f6add19f640 1 -- 172.21.15.116:0/3403723671 wait complete. 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 Processor -- start 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 -- start start 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 0x7f6ad810c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 0x7f6ad810cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 0x7f6ad819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad806a410 con 0x7f6ad81005a0 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad806a290 con 0x7f6ad8106660 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6add19f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad806a590 con 0x7f6ad8100f60 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 0x7f6ad810c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.071+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 0x7f6ad810cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 0x7f6ad810c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49002/0 (socket says 172.21.15.116:49002) 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad7fff640 1 -- 172.21.15.116:0/1765232229 learned_addr learned my addr 172.21.15.116:0/1765232229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6adc99e640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 0x7f6ad819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 -- 172.21.15.116:0/1765232229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 msgr2=0x7f6ad819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 0x7f6ad819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 -- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 msgr2=0x7f6ad810c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 0x7f6ad810c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 -- 172.21.15.116:0/1765232229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad819efc0 con 0x7f6ad8100f60 2024-03-19T02:49:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 0x7f6ad810cdf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00e9f0 tx=0x7f6acc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad57fa640 1 -- 172.21.15.116:0/1765232229 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00ccc0 con 0x7f6ad8100f60 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad819f2b0 con 0x7f6ad8100f60 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad57fa640 1 -- 172.21.15.116:0/1765232229 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6acc005020 con 0x7f6ad8100f60 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6ad57fa640 1 -- 172.21.15.116:0/1765232229 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc004a00 con 0x7f6ad8100f60 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.072+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad81a6b40 con 0x7f6ad8100f60 2024-03-19T02:49:32.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.075+0000 7f6ad57fa640 1 -- 172.21.15.116:0/1765232229 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6acc004ba0 con 0x7f6ad8100f60 2024-03-19T02:49:32.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 Processor -- start 2024-03-19T02:49:32.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 -- start start 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc5c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 0x56190cc653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 0x56190cc67cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56190cbb7500 con 0x56190cc65900 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56190cb5fbe0 con 0x56190cc5cd20 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.076+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56190cbb35a0 con 0x56190cc5c400 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6adc99e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc5c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 0x56190cc67cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6adc99e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc5c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37460/0 (socket says 172.21.15.116:37460) 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6adc99e640 1 -- 172.21.15.116:0/2309056207 learned_addr learned my addr 172.21.15.116:0/2309056207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 0x56190cc653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad77fe640 1 -- 172.21.15.116:0/2309056207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 msgr2=0x56190cc5c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc5c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad77fe640 1 -- 172.21.15.116:0/2309056207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 msgr2=0x56190cc653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.077+0000 7f6ad57fa640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 0x7f6ac00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.078+0000 7f6ad57fa640 1 -- 172.21.15.116:0/1765232229 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f6acc0cf230 con 0x7f6ad8100f60 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.078+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 0x56190cc653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.078+0000 7f6ad77fe640 1 -- 172.21.15.116:0/2309056207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190cb81000 con 0x56190cc65900 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.078+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 0x56190cc653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6adc99e640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 0x7f6ac00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 0x56190cc67cf0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0cc540 tx=0x7f6acc0054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61e963bcbbae958 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6abe7fc640 1 -- 172.21.15.116:0/2309056207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc01c030 con 0x56190cc65900 2024-03-19T02:49:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6adc99e640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 0x7f6ac00a4e30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0037040 tx=0x7f6ad002e040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6abe7fc640 1 -- 172.21.15.116:0/2309056207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6acc09bb50 con 0x56190cc65900 2024-03-19T02:49:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.079+0000 7f6abe7fc640 1 -- 172.21.15.116:0/2309056207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc0a3bf0 con 0x56190cc65900 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/2309056207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 msgr2=0x56190cc67cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 0x56190cc67cf0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0cc540 tx=0x7f6acc0054d0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/2309056207 shutdown_connections 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc65900 0x56190cc67cf0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc5cd20 0x56190cc653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/2309056207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc5c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/2309056207 >> 172.21.15.116:0/2309056207 conn(0x56190cc56b30 msgr2=0x56190cc56f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.080+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/2309056207 shutdown_connections 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.081+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/2309056207 wait complete. 2024-03-19T02:49:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.081+0000 7f6ae0fe0740 1 -- start start 2024-03-19T02:49:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.081+0000 7f6ae0fe0740 1 Processor -- start 2024-03-19T02:49:32.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.081+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5cd20 0x56190cc62e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc63340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5cd20 0x56190cc62e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49008/0 (socket says 172.21.15.116:49008) 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 -- 172.21.15.116:0/4273192615 learned_addr learned my addr 172.21.15.116:0/4273192615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56190cc68690 con 0x56190cc5c400 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56190cb73160 con 0x56190cc65900 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56190cc68810 con 0x56190cc5cd20 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6adc99e640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc63340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ae0fe0740 10 client.0 _ll_register_callbacks cb 0x56190cbafa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 -- 172.21.15.116:0/4273192615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5cd20 msgr2=0x56190cc62e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5cd20 0x56190cc62e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 -- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 msgr2=0x56190cc63340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc63340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.082+0000 7f6ad7fff640 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190c9957d0 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80127a0 tx=0x7f6ac8012c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a83b7a27115479 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac800f940 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f6ac800fae0 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6acc09bb50 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6ac800fce0 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ae0fe0740 10 client.21630 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56190c8cb3f0 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6acc09bb50 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f6ac801c420 con 0x56190cc5c400 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.083+0000 7f6ae0fe0740 10 client.21630 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ae0fe0740 10 client.21630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ae0fe0740 20 client.21630 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190cc78140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ae0fe0740 4 client.21630 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56190cbc4a40 con 0x56190cc75c80 2024-03-19T02:49:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190cc78140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190cc78140 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6883c161220c3c7d server_cookie=7d5bc43731f0e760 in_seq=0 out_seq=0 2024-03-19T02:49:32.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.084+0000 7f6ab77fe640 10 client.21630 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ab77fe640 1 -- 172.21.15.116:0/4273192615 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56190cbc4a40 con 0x56190cc75c80 2024-03-19T02:49:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ab77fe640 10 client.21630 handle_command_reply: tid=0 2024-03-19T02:49:32.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 client.21630 shutdown 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 msgr2=0x56190cc628c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80127a0 tx=0x7f6ac8012c60 comp rx=0 tx=0).stop 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 shutdown_connections 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190cc78140 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc63340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5cd20 0x56190cc62e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ab77fe640 0 client.21630 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/4273192615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5c400 0x56190cc628c0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 >> 172.21.15.116:0/4273192615 conn(0x56190cc56b30 msgr2=0x56190cc4d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 shutdown_connections 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/4273192615 wait complete. 2024-03-19T02:49:32.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.086+0000 7f6ae0fe0740 20 client.21630 trim_cache size 0 max 16384 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 Processor -- start 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 -- start start 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc64ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc71430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81ac840 con 0x56190cc5cd20 2024-03-19T02:49:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81ae3e0 con 0x56190cc65900 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81acf10 con 0x56190cc5c400 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49024/0 (socket says 172.21.15.116:49024) 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ad77fe640 1 -- 172.21.15.116:0/3465327796 learned_addr learned my addr 172.21.15.116:0/3465327796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc64ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.119+0000 7f6adc99e640 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc71430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 -- 172.21.15.116:0/3465327796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 msgr2=0x56190cc64ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc64ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 -- 172.21.15.116:0/3465327796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 msgr2=0x56190cc71430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc71430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 -- 172.21.15.116:0/3465327796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190cc61870 con 0x56190cc5cd20 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0a36c0 tx=0x7f6acc09f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad53bd3df44383d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6abdffb640 1 -- 172.21.15.116:0/3465327796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc01c040 con 0x56190cc5cd20 2024-03-19T02:49:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6abdffb640 1 -- 172.21.15.116:0/3465327796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6acc09bb50 con 0x56190cc5cd20 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.120+0000 7f6abdffb640 1 -- 172.21.15.116:0/3465327796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc0051d0 con 0x56190cc5cd20 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/3465327796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 msgr2=0x56190cc49330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0a36c0 tx=0x7f6acc09f5a0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/3465327796 shutdown_connections 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc71430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc49330 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/3465327796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc64ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/3465327796 >> 172.21.15.116:0/3465327796 conn(0x56190cc56b30 msgr2=0x56190cc4d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/3465327796 shutdown_connections 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/3465327796 wait complete. 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- start start 2024-03-19T02:49:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 Processor -- start 2024-03-19T02:49:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc6da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc6af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81add10 con 0x56190cc5cd20 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.121+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81ad5e0 con 0x56190cc65900 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ae0fe0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81aa860 con 0x56190cc5c400 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ae0fe0740 10 client.0 _ll_register_callbacks cb 0x7f6ac8009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc6da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6adc99e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc6af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49034/0 (socket says 172.21.15.116:49034) 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 -- 172.21.15.116:0/541018038 learned_addr learned my addr 172.21.15.116:0/541018038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 -- 172.21.15.116:0/541018038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 msgr2=0x56190cc6da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc6da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 -- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 msgr2=0x56190cc6af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc6af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 -- 172.21.15.116:0/541018038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190c88b0f0 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6adc99e640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc6af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ad77fe640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6acc097e20 tx=0x7f6acc0977f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8de8a2f313164c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ab77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc01c070 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f6acc0051d0 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.122+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6acc09bb50 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6acc0d2dd0 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 10 client.21636 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56190ccdcb40 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6acc09bb50 con 0x56190cc5cd20 2024-03-19T02:49:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f6acc09f170 con 0x56190cc5cd20 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 10 client.21636 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 10 client.21636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 20 client.21636 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190ccded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 4 client.21636 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56190cbbed90 con 0x56190cc75c80 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.123+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190ccded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.124+0000 7f6ab77fe640 10 client.21636 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.124+0000 7f6ad7fff640 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190ccded80 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c19f7dd296a2fa server_cookie=ecf02d0364790f7 in_seq=0 out_seq=0 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.125+0000 7f6ab77fe640 1 -- 172.21.15.116:0/541018038 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56190cbbed90 con 0x56190cc75c80 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.125+0000 7f6ab77fe640 10 client.21636 handle_command_reply: tid=0 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.125+0000 7f6ae0fe0740 1 client.21636 shutdown 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 msgr2=0x56190cc6aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6acc097e20 tx=0x7f6acc0977f0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 shutdown_connections 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56190cc75c80 0x56190ccded80 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56190cc65900 0x56190cc6af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56190cc5cd20 0x56190cc6aa30 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 --2- 172.21.15.116:0/541018038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56190cc5c400 0x56190cc6da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 >> 172.21.15.116:0/541018038 conn(0x56190cc56b30 msgr2=0x56190cc62880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ab77fe640 0 client.21636 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 shutdown_connections 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 1 -- 172.21.15.116:0/541018038 wait complete. 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6ae0fe0740 20 client.21636 trim_cache size 0 max 16384 2024-03-19T02:49:32.131 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:49:32.132 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:49:32.132 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:49:32.132 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:49:32.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 msgr2=0x7f6ac00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 0x7f6ac00a4e30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0037040 tx=0x7f6ad002e040 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 msgr2=0x7f6ad810cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 0x7f6ad810cdf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00e9f0 tx=0x7f6acc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 shutdown_connections 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6ac00a2970 0x7f6ac00a4e30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ad8106660 0x7f6ad819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad8100f60 0x7f6ad810cdf0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 --2- 172.21.15.116:0/1765232229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ad81005a0 0x7f6ad810c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 >> 172.21.15.116:0/1765232229 conn(0x7f6ad80fc070 msgr2=0x7f6ad80fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 shutdown_connections 2024-03-19T02:49:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.126+0000 7f6add19f640 1 -- 172.21.15.116:0/1765232229 wait complete. 2024-03-19T02:49:32.139 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:49:32.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:49:32.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:31 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:31 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 -- 172.21.15.116:0/124539779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4100720 msgr2=0x7f68d4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 --2- 172.21.15.116:0/124539779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4100720 0x7f68d4100ba0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009890 tx=0x7f68bc02bb70 comp rx=0 tx=0).stop 2024-03-19T02:49:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 -- 172.21.15.116:0/124539779 shutdown_connections 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 --2- 172.21.15.116:0/124539779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 --2- 172.21.15.116:0/124539779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4100720 0x7f68d4100ba0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 --2- 172.21.15.116:0/124539779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4106730 0x7f68d4106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 -- 172.21.15.116:0/124539779 >> 172.21.15.116:0/124539779 conn(0x7f68d4077820 msgr2=0x7f68d4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 -- 172.21.15.116:0/124539779 shutdown_connections 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.267+0000 7f68d9390640 1 -- 172.21.15.116:0/124539779 wait complete. 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 Processor -- start 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 -- start start 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4100720 0x7f68d4198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4106730 0x7f68d41a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68d41109f0 con 0x7f68d41010e0 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4110870 con 0x7f68d4100720 2024-03-19T02:49:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d9390640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68d4110b70 con 0x7f68d4106730 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49040/0 (socket says 172.21.15.116:49040) 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 -- 172.21.15.116:0/2178673801 learned_addr learned my addr 172.21.15.116:0/2178673801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d37fe640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4106730 0x7f68d41a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68d2ffd640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4100720 0x7f68d4198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 -- 172.21.15.116:0/2178673801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4106730 msgr2=0x7f68d41a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4106730 0x7f68d41a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 -- 172.21.15.116:0/2178673801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4100720 msgr2=0x7f68d4198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.268+0000 7f68ca7fc640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4100720 0x7f68d4198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68ca7fc640 1 -- 172.21.15.116:0/2178673801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d41a4430 con 0x7f68d41010e0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68ca7fc640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc041020 con 0x7f68d41010e0 2024-03-19T02:49:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d41a4600 con 0x7f68d41010e0 2024-03-19T02:49:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68bc02c9c0 con 0x7f68d41010e0 2024-03-19T02:49:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68bc039040 con 0x7f68d41010e0 2024-03-19T02:49:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.269+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d41a4a70 con 0x7f68d41010e0 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.270+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d4068610 con 0x7f68d41010e0 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.271+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f68bc044020 con 0x7f68d41010e0 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.271+0000 7f68d0ff9640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 0x7f68c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.272+0000 7f68d2ffd640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 0x7f68c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.272+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f68bc0f08c0 con 0x7f68d41010e0 2024-03-19T02:49:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.272+0000 7f68d2ffd640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 0x7f68c00a4e30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009b50 tx=0x7f68c4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.278+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f68bc04b020 con 0x7f68d41010e0 2024-03-19T02:49:32.335 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58810 5.83 MB/sec execute 414 sec latency 638.671 ms 2024-03-19T02:49:32.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:31 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:32.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.593+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68d4108c90 con 0x7f68d41010e0 2024-03-19T02:49:32.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.595+0000 7f68d0ff9640 1 -- 172.21.15.116:0/2178673801 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11953 (secure 0 0 0) 0x7f68bc0bd350 con 0x7f68d41010e0 2024-03-19T02:49:32.596 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:49:32.596 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:48:15.836587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24451,24494,24484,24463,24470]},"id":1}]} 2024-03-19T02:49:32.596 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2024-03-19T02:49:32.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 msgr2=0x7f68c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 0x7f68c00a4e30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009b50 tx=0x7f68c4005f90 comp rx=0 tx=0).stop 2024-03-19T02:49:32.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 msgr2=0x7f68d41993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc035990 comp rx=0 tx=0).stop 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 shutdown_connections 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68c00a2970 0x7f68c00a4e30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68d4106730 0x7f68d41a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68d41010e0 0x7f68d41993b0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 --2- 172.21.15.116:0/2178673801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68d4100720 0x7f68d4198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 >> 172.21.15.116:0/2178673801 conn(0x7f68d4077820 msgr2=0x7f68d4103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 shutdown_connections 2024-03-19T02:49:32.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.597+0000 7f68d9390640 1 -- 172.21.15.116:0/2178673801 wait complete. 2024-03-19T02:49:32.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:49:32.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/615895043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01006a0 msgr2=0x7fe9e0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/615895043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01006a0 0x7fe9e0100b20 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009930 tx=0x7fe9d002bc20 comp rx=0 tx=0).stop 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/615895043 shutdown_connections 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/615895043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/615895043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01006a0 0x7fe9e0100b20 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/615895043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01066b0 0x7fe9e0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/615895043 >> 172.21.15.116:0/615895043 conn(0x7fe9e00778d0 msgr2=0x7fe9e0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/615895043 shutdown_connections 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.742+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/615895043 wait complete. 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 Processor -- start 2024-03-19T02:49:32.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 -- start start 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 0x7fe9e0196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e01974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 0x7fe9e019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0110970 con 0x7fe9e01066b0 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01107f0 con 0x7fe9e0101060 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e7bcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0110af0 con 0x7fe9e01006a0 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e01974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 0x7fe9e0196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 0x7fe9e019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 0x7fe9e0196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37538/0 (socket says 172.21.15.116:37538) 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e5969640 1 -- 172.21.15.116:0/1869828855 learned_addr learned my addr 172.21.15.116:0/1869828855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 -- 172.21.15.116:0/1869828855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 msgr2=0x7fe9e0196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 0x7fe9e0196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 -- 172.21.15.116:0/1869828855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 msgr2=0x7fe9e01974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e01974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.743+0000 7fe9e616a640 1 -- 172.21.15.116:0/1869828855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e019bf60 con 0x7fe9e01066b0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9e5168640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e01974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9e616a640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 0x7fe9e019b830 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800ec50 tx=0x7fe9d800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9d6ffd640 1 -- 172.21.15.116:0/1869828855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8003b70 con 0x7fe9e01066b0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9d6ffd640 1 -- 172.21.15.116:0/1869828855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9d8003d10 con 0x7fe9e01066b0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9d6ffd640 1 -- 172.21.15.116:0/1869828855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8010760 con 0x7fe9e01066b0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e019c250 con 0x7fe9e01066b0 2024-03-19T02:49:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.744+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e01a3ae0 con 0x7fe9e01066b0 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 Processor -- start 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9d6ffd640 1 -- 172.21.15.116:0/1869828855 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe9d8004020 con 0x7fe9e01066b0 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 -- start start 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608ea1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eaedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d608e044b0 con 0x55d608ea0f90 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d608dacd70 con 0x55d608ea18b0 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.747+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d608e00550 con 0x55d608eac9e0 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608ea1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608ea1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49070/0 (socket says 172.21.15.116:49070) 2024-03-19T02:49:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5168640 1 -- 172.21.15.116:0/3186549185 learned_addr learned my addr 172.21.15.116:0/3186549185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5168640 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eaedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9d6ffd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 0x7fe9cc0a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 -- 172.21.15.116:0/3186549185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 msgr2=0x55d608eaedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eaedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 -- 172.21.15.116:0/3186549185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 msgr2=0x55d608ea1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608ea1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 -- 172.21.15.116:0/3186549185 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d608dcf210 con 0x55d608ea18b0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9d6ffd640 1 -- 172.21.15.116:0/1869828855 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fe9d80cf700 con 0x7fe9e01066b0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e616a640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 0x7fe9cc0a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eac4a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009c70 tx=0x7fe9dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5df089733e0ca0a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9cbfff640 1 -- 172.21.15.116:0/3186549185 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc009e20 con 0x55d608ea18b0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9cbfff640 1 -- 172.21.15.116:0/3186549185 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9dc0054d0 con 0x55d608ea18b0 2024-03-19T02:49:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.748+0000 7fe9cbfff640 1 -- 172.21.15.116:0/3186549185 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc019670 con 0x55d608ea18b0 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3186549185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 msgr2=0x55d608eac4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9e616a640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 0x7fe9cc0a4f80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80a3bf0 tx=0x7fe9d80045e0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eac4a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009c70 tx=0x7fe9dc011810 comp rx=0 tx=0).stop 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3186549185 shutdown_connections 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eaedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eac4a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3186549185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608ea1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3186549185 >> 172.21.15.116:0/3186549185 conn(0x55d608e9ba00 msgr2=0x55d608e9bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3186549185 shutdown_connections 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.749+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3186549185 wait complete. 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9ea7b8740 1 -- start start 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9ea7b8740 1 Processor -- start 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.750+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49084/0 (socket says 172.21.15.116:49084) 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59724/0 (socket says 172.21.15.116:59724) 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 -- 172.21.15.116:0/715129853 learned_addr learned my addr 172.21.15.116:0/715129853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d608ea2530 con 0x55d608ea18b0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d608dc0310 con 0x55d608ea0f90 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d608ea26b0 con 0x55d608eac9e0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9ea7b8740 10 client.0 _ll_register_callbacks cb 0x55d608db42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5168640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 -- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 msgr2=0x55d608eba920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eba920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 -- 172.21.15.116:0/715129853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 msgr2=0x55d608ea80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d608b183f0 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5168640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9e5969640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009c40 tx=0x7fe9dc005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10aaba982f00de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc028070 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fe9dc024440 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.751+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9dc0054d0 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe9dc034050 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 10 client.21654 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d608be27d0 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9dc009880 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe9dc022410 con 0x55d608ea18b0 2024-03-19T02:49:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 10 client.21654 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 10 client.21654 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 20 client.21654 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608ebf2a0 0x55d608ec1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 4 client.21654 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe9dc0054d0 con 0x55d608ebf2a0 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.752+0000 7fe9e5168640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608ebf2a0 0x55d608ec1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.753+0000 7fe9c97fa640 10 client.21654 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.753+0000 7fe9e5168640 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608ebf2a0 0x55d608ec1740 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f88aed806fb539b9 server_cookie=92960e8928d2b081 in_seq=0 out_seq=0 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9c97fa640 1 -- 172.21.15.116:0/715129853 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe9dc0054d0 con 0x55d608ebf2a0 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9c97fa640 10 client.21654 handle_command_reply: tid=0 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 client.21654 shutdown 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 msgr2=0x55d608ea85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009c40 tx=0x7fe9dc005060 comp rx=0 tx=0).stop 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 shutdown_connections 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608ebf2a0 0x55d608ec1740 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eba920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ea85e0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9c97fa640 0 client.21654 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/715129853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 >> 172.21.15.116:0/715129853 conn(0x55d608e9ba00 msgr2=0x55d608e9d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 shutdown_connections 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/715129853 wait complete. 2024-03-19T02:49:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.754+0000 7fe9ea7b8740 20 client.21654 trim_cache size 0 max 16384 2024-03-19T02:49:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.770+0000 7fe9ea7b8740 1 Processor -- start 2024-03-19T02:49:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.770+0000 7fe9ea7b8740 1 -- start start 2024-03-19T02:49:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ec7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608ea7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ac9a0 con 0x55d608ea18b0 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ae540 con 0x55d608ea0f90 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ad070 con 0x55d608eac9e0 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608ea7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ec7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5168640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608ea7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37568/0 (socket says 172.21.15.116:37568) 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5168640 1 -- 172.21.15.116:0/449522316 learned_addr learned my addr 172.21.15.116:0/449522316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5168640 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 -- 172.21.15.116:0/449522316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 msgr2=0x55d608ea7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608ea7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 -- 172.21.15.116:0/449522316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 msgr2=0x55d608ea1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.771+0000 7fe9e5969640 1 -- 172.21.15.116:0/449522316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d608ea8dc0 con 0x55d608ea18b0 2024-03-19T02:49:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.772+0000 7fe9e5969640 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ec7050 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0227e0 tx=0x7fe9dc011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13109fe0849e6d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.772+0000 7fe9cb7fe640 1 -- 172.21.15.116:0/449522316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc031030 con 0x55d608ea18b0 2024-03-19T02:49:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.772+0000 7fe9cb7fe640 1 -- 172.21.15.116:0/449522316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9dc004af0 con 0x55d608ea18b0 2024-03-19T02:49:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.772+0000 7fe9cb7fe640 1 -- 172.21.15.116:0/449522316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc028040 con 0x55d608ea18b0 2024-03-19T02:49:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/449522316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 msgr2=0x55d608ec7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ec7050 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0227e0 tx=0x7fe9dc011dd0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/449522316 shutdown_connections 2024-03-19T02:49:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608ea7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea18b0 0x55d608ec7050 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/449522316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea0f90 0x55d608ea1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/449522316 >> 172.21.15.116:0/449522316 conn(0x55d608e9ba00 msgr2=0x55d608e9bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/449522316 shutdown_connections 2024-03-19T02:49:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/449522316 wait complete. 2024-03-19T02:49:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.773+0000 7fe9ea7b8740 1 -- start start 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 Processor -- start 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608eb5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eb1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ade70 con 0x55d608ea0f90 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ad740 con 0x55d608ea18b0 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01aa9a0 con 0x55d608eac9e0 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9ea7b8740 10 client.0 _ll_register_callbacks cb 0x7fe9dc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59738/0 (socket says 172.21.15.116:59738) 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 -- 172.21.15.116:0/3515454203 learned_addr learned my addr 172.21.15.116:0/3515454203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5168640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608eb5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 -- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 msgr2=0x55d608eb1130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eb1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 -- 172.21.15.116:0/3515454203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 msgr2=0x55d608eb5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608eb5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5969640 1 -- 172.21.15.116:0/3515454203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d608eb1960 con 0x55d608ea18b0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.774+0000 7fe9e5168640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608eb5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9e5969640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc012ff0 tx=0x7fe9dc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3757d3a3f0e8bce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc03c070 con 0x55d608ea18b0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fe9dc0229d0 con 0x55d608ea18b0 2024-03-19T02:49:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9dc004af0 con 0x55d608ea18b0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe9dc031070 con 0x55d608ea18b0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9ea7b8740 10 client.31388 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.775+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d608eba4e0 con 0x55d608ea18b0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9dc004af0 con 0x55d608ea18b0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe9dc024c70 con 0x55d608ea18b0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 10 client.31388 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 10 client.31388 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 20 client.31388 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608f29e70 0x55d608f2c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 4 client.31388 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d608e623e0 con 0x55d608f29e70 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.776+0000 7fe9e5168640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608f29e70 0x55d608f2c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9c97fa640 10 client.31388 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9e5168640 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608f29e70 0x55d608f2c260 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18f709e4217924d0 server_cookie=182b1e7d0d112193 in_seq=0 out_seq=0 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9c97fa640 1 -- 172.21.15.116:0/3515454203 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d608e623e0 con 0x55d608f29e70 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9c97fa640 10 client.31388 handle_command_reply: tid=0 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 client.31388 shutdown 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 msgr2=0x55d608eb6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc012ff0 tx=0x7fe9dc009f90 comp rx=0 tx=0).stop 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 shutdown_connections 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55d608f29e70 0x55d608f2c260 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d608eac9e0 0x55d608eb1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d608ea18b0 0x55d608eb6270 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 --2- 172.21.15.116:0/3515454203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d608ea0f90 0x55d608eb5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 >> 172.21.15.116:0/3515454203 conn(0x55d608e9ba00 msgr2=0x55d608ea7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9c97fa640 0 client.31388 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 shutdown_connections 2024-03-19T02:49:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 1 -- 172.21.15.116:0/3515454203 wait complete. 2024-03-19T02:49:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.777+0000 7fe9ea7b8740 20 client.31388 trim_cache size 0 max 16384 2024-03-19T02:49:32.785 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 msgr2=0x7fe9cc0a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 0x7fe9cc0a4f80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80a3bf0 tx=0x7fe9d80045e0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 msgr2=0x7fe9e019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 0x7fe9e019b830 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800ec50 tx=0x7fe9d800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 shutdown_connections 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9cc0a2ac0 0x7fe9cc0a4f80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9e01066b0 0x7fe9e019b830 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9e0101060 0x7fe9e01974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 --2- 172.21.15.116:0/1869828855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9e01006a0 0x7fe9e0196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 >> 172.21.15.116:0/1869828855 conn(0x7fe9e00778d0 msgr2=0x7fe9e0103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 shutdown_connections 2024-03-19T02:49:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.778+0000 7fe9e7bcd640 1 -- 172.21.15.116:0/1869828855 wait complete. 2024-03-19T02:49:32.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.932+0000 7fdd1741f640 1 -- 172.21.15.116:0/424483276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 msgr2=0x7fdd080a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.932+0000 7fdd1741f640 1 --2- 172.21.15.116:0/424483276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd080a1ab0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdd10067630 tx=0x7fdd10097830 comp rx=0 tx=0).stop 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.933+0000 7fdd1741f640 1 -- 172.21.15.116:0/424483276 shutdown_connections 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.933+0000 7fdd1741f640 1 --2- 172.21.15.116:0/424483276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd080a1ab0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.933+0000 7fdd1741f640 1 --2- 172.21.15.116:0/424483276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd080952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.933+0000 7fdd1741f640 1 --2- 172.21.15.116:0/424483276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.933+0000 7fdd1741f640 1 -- 172.21.15.116:0/424483276 >> 172.21.15.116:0/424483276 conn(0x7fdd0800a730 msgr2=0x7fdd0800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.935+0000 7fdd1741f640 1 -- 172.21.15.116:0/424483276 shutdown_connections 2024-03-19T02:49:32.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.935+0000 7fdd1741f640 1 -- 172.21.15.116:0/424483276 wait complete. 2024-03-19T02:49:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.936+0000 7fdd1741f640 1 Processor -- start 2024-03-19T02:49:32.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.936+0000 7fdd1741f640 1 -- start start 2024-03-19T02:49:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd0812ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080a80c0 con 0x7fdd08099bb0 2024-03-19T02:49:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080a7f40 con 0x7fdd08094e40 2024-03-19T02:49:32.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.937+0000 7fdd1741f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080a8240 con 0x7fdd080957e0 2024-03-19T02:49:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59744/0 (socket says 172.21.15.116:59744) 2024-03-19T02:49:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 -- 172.21.15.116:0/2331273736 learned_addr learned my addr 172.21.15.116:0/2331273736 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd0812ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 -- 172.21.15.116:0/2331273736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 msgr2=0x7fdd0812ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd0812ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 -- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 msgr2=0x7fdd08132f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08132f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd1641d640 1 -- 172.21.15.116:0/2331273736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd08133680 con 0x7fdd08094e40 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:49:32.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.939+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd0812ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.940+0000 7fdd1641d640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0097d0 tx=0x7fdd0c009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.940+0000 7fdd077fe640 1 -- 172.21.15.116:0/2331273736 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00eab0 con 0x7fdd08094e40 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.940+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd08133970 con 0x7fdd08094e40 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.940+0000 7fdd077fe640 1 -- 172.21.15.116:0/2331273736 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd0c019070 con 0x7fdd08094e40 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.940+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0813b200 con 0x7fdd08094e40 2024-03-19T02:49:32.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.941+0000 7fdd077fe640 1 -- 172.21.15.116:0/2331273736 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c013380 con 0x7fdd08094e40 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 Processor -- start 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 -- start start 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c30c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647a6b824b0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5647a6b2ad70 con 0x5647a6c1fc70 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.943+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647a6b7e550 con 0x5647a6c311c0 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd077fe640 1 -- 172.21.15.116:0/2331273736 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdd0c014420 con 0x7fdd08094e40 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49120/0 (socket says 172.21.15.116:49120) 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2617061948 learned_addr learned my addr 172.21.15.116:0/2617061948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2617061948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 msgr2=0x5647a6c335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c335b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2617061948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 msgr2=0x5647a6c30c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c30c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2617061948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a6b4d210 con 0x5647a6c1f2c0 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fdd10097d70 tx=0x7fdd100676c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29a7cc10e8186c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd077fe640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 0x7fdcfc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.944+0000 7fdd04ff9640 1 -- 172.21.15.116:0/2617061948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd100752d0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd077fe640 1 -- 172.21.15.116:0/2331273736 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fdd0c0cf110 con 0x7fdd08094e40 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd04ff9640 1 -- 172.21.15.116:0/2617061948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd10075710 con 0x5647a6c1f2c0 2024-03-19T02:49:32.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd04ff9640 1 -- 172.21.15.116:0/2617061948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd100693f0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd1641d640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 0x7fdcfc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2617061948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 msgr2=0x5647a6c1f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd1641d640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 0x7fdcfc0a4eb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x5647a6c1ae70 tx=0x7fdd0c009e80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.945+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fdd10097d70 tx=0x7fdd100676c0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2617061948 shutdown_connections 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c30c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2617061948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c1f6a0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2617061948 >> 172.21.15.116:0/2617061948 conn(0x5647a6c19be0 msgr2=0x5647a6c19fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2617061948 shutdown_connections 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2617061948 wait complete. 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 -- start start 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.946+0000 7fdd1b26f740 1 Processor -- start 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 0x5647a6c2e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c2e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 0x5647a6c2ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647a6c33f20 con 0x5647a6c311c0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5647a6b3e310 con 0x5647a6c1f2c0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647a6c340a0 con 0x5647a6c1fc70 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd1b26f740 10 client.0 _ll_register_callbacks cb 0x5647a6b322d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 0x5647a6c2ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 0x5647a6c2ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49128/0 (socket says 172.21.15.116:49128) 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2482674348 learned_addr learned my addr 172.21.15.116:0/2482674348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c2e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 0x5647a6c2e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 -- 172.21.15.116:0/2482674348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 msgr2=0x5647a6c2e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 0x5647a6c2e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 -- 172.21.15.116:0/2482674348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 msgr2=0x5647a6c2ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 0x5647a6c2ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 -- 172.21.15.116:0/2482674348 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a68963f0 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 0x5647a6c2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.947+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c2e770 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdd08092590 tx=0x7fdd00012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a637d9dca0ae8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00011bb0 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd10075710 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fdd00011d50 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd00025050 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 10 client.31369 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5647a69607d0 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd10075710 con 0x5647a6c1fc70 2024-03-19T02:49:32.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fdd00003ca0 con 0x5647a6c1fc70 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 10 client.31369 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 10 client.31369 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.948+0000 7fdd1b26f740 20 client.31369 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6c43b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdd1b26f740 4 client.31369 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647a6be03e0 con 0x5647a6c41750 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6c43b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdcf9ffb640 10 client.31369 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.949+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6c43b70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7329bc5b2574c1ad server_cookie=6b3f0c2dc37683e7 in_seq=0 out_seq=0 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/2482674348 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5647a6be03e0 con 0x5647a6c41750 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdcf9ffb640 10 client.31369 handle_command_reply: tid=0 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 client.31369 shutdown 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 msgr2=0x5647a6c2e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c2e770 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdd08092590 tx=0x7fdd00012b60 comp rx=0 tx=0).stop 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 shutdown_connections 2024-03-19T02:49:32.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6c43b70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c311c0 0x5647a6c2ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c2e770 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdcf9ffb640 0 client.31369 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2482674348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1f2c0 0x5647a6c2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 >> 172.21.15.116:0/2482674348 conn(0x5647a6c19be0 msgr2=0x5647a6c1b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 shutdown_connections 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2482674348 wait complete. 2024-03-19T02:49:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.950+0000 7fdd1b26f740 20 client.31369 trim_cache size 0 max 16384 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 Processor -- start 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 -- start start 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c18d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c311c0 0x5647a6c2ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08140ef0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08142a90 con 0x5647a6c311c0 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081415c0 con 0x5647a6c1fc70 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49136/0 (socket says 172.21.15.116:49136) 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.968+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2249294857 learned_addr learned my addr 172.21.15.116:0/2249294857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c18d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2249294857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 msgr2=0x5647a6c18d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c18d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2249294857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c311c0 msgr2=0x5647a6c2ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c311c0 0x5647a6c2ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 -- 172.21.15.116:0/2249294857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a6c2d1e0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c18d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:49:32.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.969+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000b60 tx=0x7fdd10097fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58897c94727f61d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.973+0000 7fdcfbfff640 1 -- 172.21.15.116:0/2249294857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd100752d0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.974+0000 7fdcfbfff640 1 -- 172.21.15.116:0/2249294857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd10075590 con 0x5647a6c1f2c0 2024-03-19T02:49:32.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.974+0000 7fdcfbfff640 1 -- 172.21.15.116:0/2249294857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10069100 con 0x5647a6c1f2c0 2024-03-19T02:49:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2249294857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 msgr2=0x5647a6c3cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000b60 tx=0x7fdd10097fb0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2249294857 shutdown_connections 2024-03-19T02:49:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c311c0 0x5647a6c2ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c1fc70 0x5647a6c18d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/2249294857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c3cd60 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2249294857 >> 172.21.15.116:0/2249294857 conn(0x5647a6c19be0 msgr2=0x5647a6c19fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2249294857 shutdown_connections 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- 172.21.15.116:0/2249294857 wait complete. 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 -- start start 2024-03-19T02:49:32.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.975+0000 7fdd1b26f740 1 Processor -- start 2024-03-19T02:49:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c39610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c34f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081423c0 con 0x5647a6c1f2c0 2024-03-19T02:49:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08141c90 con 0x5647a6c1fc70 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0813ef70 con 0x5647a6c311c0 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd1b26f740 10 client.0 _ll_register_callbacks cb 0x7fdd00009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd16c1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c39610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59776/0 (socket says 172.21.15.116:59776) 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 -- 172.21.15.116:0/1776296083 learned_addr learned my addr 172.21.15.116:0/1776296083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c34f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 -- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 msgr2=0x5647a6c34f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c34f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 -- 172.21.15.116:0/1776296083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 msgr2=0x5647a6c39610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c39610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd15c1c640 1 -- 172.21.15.116:0/1776296083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a6c35740 con 0x5647a6c1fc70 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.976+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c34f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdd15c1c640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdd00000c00 tx=0x7fdd0000f980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eff8db1bb637a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdcf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0002d020 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd10075590 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7fdd00018040 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdd00038050 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdd1b26f740 10 client.31400 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5647a6ca7a10 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.977+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd10075590 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fdd00004660 con 0x5647a6c1fc70 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 10 client.31400 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 10 client.31400 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 20 client.31400 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6caa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:32.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 4 client.31400 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647a6b8f9f0 con 0x5647a6c41750 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6caa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdcf9ffb640 10 client.31400 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.978+0000 7fdd16c1e640 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6caa160 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb863463c19dd2fc server_cookie=97f08532ffb20123 in_seq=0 out_seq=0 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdcf9ffb640 1 -- 172.21.15.116:0/1776296083 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647a6b8f9f0 con 0x5647a6c41750 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdcf9ffb640 10 client.31400 handle_command_reply: tid=0 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 client.31400 shutdown 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 msgr2=0x5647a6c349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdd00000c00 tx=0x7fdd0000f980 comp rx=0 tx=0).stop 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 shutdown_connections 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647a6c311c0 0x5647a6c34f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647a6c1fc70 0x5647a6c349d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5647a6c41750 0x5647a6caa160 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 --2- 172.21.15.116:0/1776296083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647a6c1f2c0 0x5647a6c39610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 >> 172.21.15.116:0/1776296083 conn(0x5647a6c19be0 msgr2=0x5647a6c18d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdcf9ffb640 0 client.31400 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 shutdown_connections 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 1 -- 172.21.15.116:0/1776296083 wait complete. 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.979+0000 7fdd1b26f740 20 client.31400 trim_cache size 0 max 16384 2024-03-19T02:49:32.985 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 msgr2=0x7fdcfc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 0x7fdcfc0a4eb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x5647a6c1ae70 tx=0x7fdd0c009e80 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 msgr2=0x7fdd0812e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0097d0 tx=0x7fdd0c009c90 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 shutdown_connections 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdcfc0a29f0 0x7fdcfc0a4eb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd08099bb0 0x7fdd08132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdd080957e0 0x7fdd0812ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 --2- 172.21.15.116:0/2331273736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd08094e40 0x7fdd0812e690 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0097d0 tx=0x7fdd0c009c90 comp rx=0 tx=0).stop 2024-03-19T02:49:32.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 >> 172.21.15.116:0/2331273736 conn(0x7fdd0800a730 msgr2=0x7fdd08092e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 shutdown_connections 2024-03-19T02:49:32.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:32.980+0000 7fdd1741f640 1 -- 172.21.15.116:0/2331273736 wait complete. 2024-03-19T02:49:32.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 -- 172.21.15.116:0/4117634276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60105690 msgr2=0x7f1e60105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/4117634276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60105690 0x7f1e60105a70 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009900 tx=0x7f1e5002bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 -- 172.21.15.116:0/4117634276 shutdown_connections 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/4117634276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60101330 0x7f1e6010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/4117634276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60100970 0x7f1e60100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/4117634276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60105690 0x7f1e60105a70 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 -- 172.21.15.116:0/4117634276 >> 172.21.15.116:0/4117634276 conn(0x7f1e600774b0 msgr2=0x7f1e600778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 -- 172.21.15.116:0/4117634276 shutdown_connections 2024-03-19T02:49:33.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.130+0000 7f1e64eb2640 1 -- 172.21.15.116:0/4117634276 wait complete. 2024-03-19T02:49:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.131+0000 7f1e64eb2640 1 Processor -- start 2024-03-19T02:49:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.131+0000 7f1e64eb2640 1 -- start start 2024-03-19T02:49:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.131+0000 7f1e64eb2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 0x7f1e60198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.131+0000 7f1e64eb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.131+0000 7f1e64eb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60105690 0x7f1e6019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e64eb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60111b60 con 0x7f1e60101330 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e64eb2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601119e0 con 0x7f1e60100970 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e64eb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60111ce0 con 0x7f1e60105690 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 0x7f1e60198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49166/0 (socket says 172.21.15.116:49166) 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 -- 172.21.15.116:0/3813654120 learned_addr learned my addr 172.21.15.116:0/3813654120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60105690 0x7f1e6019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 -- 172.21.15.116:0/3813654120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60105690 msgr2=0x7f1e6019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60105690 0x7f1e6019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 -- 172.21.15.116:0/3813654120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 msgr2=0x7f1e60198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 0x7f1e60198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 -- 172.21.15.116:0/3813654120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6019d010 con 0x7f1e60101330 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 0x7f1e60198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e57fff640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800e9f0 tx=0x7f1e4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.132+0000 7f1e577fe640 1 -- 172.21.15.116:0/3813654120 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800cd80 con 0x7f1e60101330 2024-03-19T02:49:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.133+0000 7f1e577fe640 1 -- 172.21.15.116:0/3813654120 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e480054d0 con 0x7f1e60101330 2024-03-19T02:49:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.133+0000 7f1e577fe640 1 -- 172.21.15.116:0/3813654120 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48010490 con 0x7f1e60101330 2024-03-19T02:49:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.133+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6019d300 con 0x7f1e60101330 2024-03-19T02:49:33.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.133+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e6010e220 con 0x7f1e60101330 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.135+0000 7f1e577fe640 1 -- 172.21.15.116:0/3813654120 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f1e48005020 con 0x7f1e60101330 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.135+0000 7f1e67b7c740 1 Processor -- start 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.135+0000 7f1e67b7c740 1 -- start start 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee22d0 0x562268ee26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e577fe640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 0x7f1e4c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268eedbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562268e454b0 con 0x562268ee22d0 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562268dedd70 con 0x562268eeb7d0 2024-03-19T02:49:33.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562268e41550 con 0x562268ee2bf0 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee22d0 0x562268ee26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37660/0 (socket says 172.21.15.116:37660) 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 -- 172.21.15.116:0/533452191 learned_addr learned my addr 172.21.15.116:0/533452191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e57fff640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 0x7f1e4c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e577fe640 1 -- 172.21.15.116:0/3813654120 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f1e480cf5f0 con 0x7f1e60101330 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 -- 172.21.15.116:0/533452191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 msgr2=0x562268eedbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268eedbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268eedbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 -- 172.21.15.116:0/533452191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee22d0 msgr2=0x562268ee26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee22d0 0x562268ee26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.136+0000 7f1e5e59c640 1 -- 172.21.15.116:0/533452191 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562268e10210 con 0x562268ee2bf0 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.137+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1e50000c00 tx=0x7f1e50005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1c40a76140b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.137+0000 7f1e57fff640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 0x7f1e4c0a4e30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1e480cca10 tx=0x7f1e48086040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.137+0000 7f1e54ff9640 1 -- 172.21.15.116:0/533452191 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e500042c0 con 0x562268ee2bf0 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.137+0000 7f1e54ff9640 1 -- 172.21.15.116:0/533452191 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e5002cc30 con 0x562268ee2bf0 2024-03-19T02:49:33.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.137+0000 7f1e54ff9640 1 -- 172.21.15.116:0/533452191 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5003c5d0 con 0x562268ee2bf0 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- 172.21.15.116:0/533452191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 msgr2=0x562268eeb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1e50000c00 tx=0x7f1e50005a20 comp rx=0 tx=0).stop 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- 172.21.15.116:0/533452191 shutdown_connections 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268eedbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee2bf0 0x562268eeb290 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/533452191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee22d0 0x562268ee26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- 172.21.15.116:0/533452191 >> 172.21.15.116:0/533452191 conn(0x562268edca00 msgr2=0x562268edcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- 172.21.15.116:0/533452191 shutdown_connections 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- 172.21.15.116:0/533452191 wait complete. 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- start start 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 Processor -- start 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ee8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268ee92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562268eee560 con 0x562268ee2bf0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562268e01310 con 0x562268ee22d0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562268eee6e0 con 0x562268eeb7d0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.138+0000 7f1e67b7c740 10 client.0 _ll_register_callbacks cb 0x562268df52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ee8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49196/0 (socket says 172.21.15.116:49196) 2024-03-19T02:49:33.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 -- 172.21.15.116:0/166917235 learned_addr learned my addr 172.21.15.116:0/166917235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268ee92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 -- 172.21.15.116:0/166917235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 msgr2=0x562268ee92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268ee92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 -- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 msgr2=0x562268ee8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ee8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 -- 172.21.15.116:0/166917235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562268b593f0 con 0x562268ee2bf0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268ee92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002c720 tx=0x7f1e5002ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db484be361c9d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e500042c0 con 0x562268ee2bf0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e5002cc30 con 0x562268ee2bf0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f1e50004460 con 0x562268ee2bf0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1e50061050 con 0x562268ee2bf0 2024-03-19T02:49:33.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e67b7c740 10 client.21687 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.139+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562268c237d0 con 0x562268ee2bf0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e5002cc30 con 0x562268ee2bf0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f1e50047070 con 0x562268ee2bf0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 10 client.21687 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 10 client.21687 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 20 client.21687 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268efb9d0 0x562268efddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 4 client.21687 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562268ea33e0 con 0x562268efb9d0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268efb9d0 0x562268efddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268efb9d0 0x562268efddc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc0681ac6e40338b server_cookie=13fd1eb04de31dd in_seq=0 out_seq=0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.140+0000 7f1e45ffb640 10 client.21687 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e45ffb640 1 -- 172.21.15.116:0/166917235 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562268ea33e0 con 0x562268efb9d0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e45ffb640 10 client.21687 handle_command_reply: tid=0 2024-03-19T02:49:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e67b7c740 1 client.21687 shutdown 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 msgr2=0x562268ee8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002c720 tx=0x7f1e5002ca30 comp rx=0 tx=0).stop 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.141+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 shutdown_connections 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268efb9d0 0x562268efddc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268ee92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e45ffb640 0 client.21687 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee8d90 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/166917235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ee8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 >> 172.21.15.116:0/166917235 conn(0x562268edca00 msgr2=0x562268ede750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 shutdown_connections 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 1 -- 172.21.15.116:0/166917235 wait complete. 2024-03-19T02:49:33.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.142+0000 7f1e67b7c740 20 client.21687 trim_cache size 0 max 16384 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 Processor -- start 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 -- start start 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ef7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268edb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601ac930 con 0x562268ee2bf0 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601ae4d0 con 0x562268ee22d0 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601ad000 con 0x562268eeb7d0 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49200/0 (socket says 172.21.15.116:49200) 2024-03-19T02:49:33.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 -- 172.21.15.116:0/3480788464 learned_addr learned my addr 172.21.15.116:0/3480788464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ef7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268edb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 -- 172.21.15.116:0/3480788464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 msgr2=0x562268edb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268edb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 -- 172.21.15.116:0/3480788464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 msgr2=0x562268ef7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ef7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 -- 172.21.15.116:0/3480788464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562268efe130 con 0x562268ee2bf0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.158+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f1e50053540 tx=0x7f1e50005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7a477d6bf44b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e47fff640 1 -- 172.21.15.116:0/3480788464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50005850 con 0x562268ee2bf0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e47fff640 1 -- 172.21.15.116:0/3480788464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e500044a0 con 0x562268ee2bf0 2024-03-19T02:49:33.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e47fff640 1 -- 172.21.15.116:0/3480788464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50053ca0 con 0x562268ee2bf0 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 -- 172.21.15.116:0/3480788464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 msgr2=0x562268f03360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f1e50053540 tx=0x7f1e50005a20 comp rx=0 tx=0).stop 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 -- 172.21.15.116:0/3480788464 shutdown_connections 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268eeb7d0 0x562268edb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268f03360 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/3480788464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268ee22d0 0x562268ef7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.159+0000 7f1e67b7c740 1 -- 172.21.15.116:0/3480788464 >> 172.21.15.116:0/3480788464 conn(0x562268edca00 msgr2=0x562268ee3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- 172.21.15.116:0/3480788464 shutdown_connections 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- 172.21.15.116:0/3480788464 wait complete. 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- start start 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 Processor -- start 2024-03-19T02:49:33.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 0x562268ef3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268ee56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601ade00 con 0x562268ee2bf0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601ad6d0 con 0x562268eeb7d0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601aa930 con 0x562268ee22d0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e67b7c740 10 client.0 _ll_register_callbacks cb 0x7f1e58009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e5ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 0x562268ef3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e5ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 0x562268ef3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37696/0 (socket says 172.21.15.116:37696) 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e5ed9d640 1 -- 172.21.15.116:0/1120591923 learned_addr learned my addr 172.21.15.116:0/1120591923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.160+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268ee56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 -- 172.21.15.116:0/1120591923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 msgr2=0x562268ef3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 0x562268ef3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 -- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 msgr2=0x562268ee56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268ee56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 -- 172.21.15.116:0/1120591923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562268ee5f10 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e5e59c640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee5180 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x562268eee290 tx=0x7f1e50009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d921fda12c4bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5002cc10 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f1e50003ff0 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e500044a0 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1e500041f0 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e67b7c740 10 client.21693 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562268f61a20 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.161+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e500044a0 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f1e50047040 con 0x562268ee2bf0 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 10 client.21693 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 10 client.21693 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 20 client.21693 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268f61e50 0x562268f64240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 4 client.21693 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562268e529f0 con 0x562268f61e50 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268f61e50 0x562268f64240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e45ffb640 10 client.21693 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e5ed9d640 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268f61e50 0x562268f64240 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6b02cb295cf9f64 server_cookie=506d52fbb112084b in_seq=0 out_seq=0 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e45ffb640 1 -- 172.21.15.116:0/1120591923 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562268e529f0 con 0x562268f61e50 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e45ffb640 10 client.21693 handle_command_reply: tid=0 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.162+0000 7f1e67b7c740 1 client.21693 shutdown 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 msgr2=0x562268ee5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee5180 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x562268eee290 tx=0x7f1e50009780 comp rx=0 tx=0).stop 2024-03-19T02:49:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 shutdown_connections 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x562268f61e50 0x562268f64240 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562268eeb7d0 0x562268ee56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562268ee2bf0 0x562268ee5180 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e45ffb640 0 client.21693 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 --2- 172.21.15.116:0/1120591923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562268ee22d0 0x562268ef3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 >> 172.21.15.116:0/1120591923 conn(0x562268edca00 msgr2=0x562268edb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 shutdown_connections 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 1 -- 172.21.15.116:0/1120591923 wait complete. 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e67b7c740 20 client.21693 trim_cache size 0 max 16384 2024-03-19T02:49:33.170 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 msgr2=0x7f1e4c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 0x7f1e4c0a4e30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1e480cca10 tx=0x7f1e48086040 comp rx=0 tx=0).stop 2024-03-19T02:49:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 msgr2=0x7f1e60198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800e9f0 tx=0x7f1e4800edf0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 shutdown_connections 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f1e4c0a2970 0x7f1e4c0a4e30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e60105690 0x7f1e6019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e60101330 0x7f1e60198560 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 --2- 172.21.15.116:0/3813654120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e60100970 0x7f1e60198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.163+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 >> 172.21.15.116:0/3813654120 conn(0x7f1e600774b0 msgr2=0x7f1e600fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.164+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 shutdown_connections 2024-03-19T02:49:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.164+0000 7f1e64eb2640 1 -- 172.21.15.116:0/3813654120 wait complete. 2024-03-19T02:49:33.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:49:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: pgmap v686: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-03-19T02:49:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:32 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2178673801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: pgmap v686: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:32 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2178673801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 -- 172.21.15.116:0/1035450349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541066b0 msgr2=0x7f5654106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 --2- 172.21.15.116:0/1035450349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541066b0 0x7f5654106a90 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5650009930 tx=0x7f565002bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 -- 172.21.15.116:0/1035450349 shutdown_connections 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 --2- 172.21.15.116:0/1035450349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f565410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 --2- 172.21.15.116:0/1035450349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541006a0 0x7f5654100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 --2- 172.21.15.116:0/1035450349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541066b0 0x7f5654106a90 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 -- 172.21.15.116:0/1035450349 >> 172.21.15.116:0/1035450349 conn(0x7f56540778d0 msgr2=0x7f5654077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 -- 172.21.15.116:0/1035450349 shutdown_connections 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 -- 172.21.15.116:0/1035450349 wait complete. 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.318+0000 7f565bb10640 1 Processor -- start 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 -- start start 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f5654103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541066b0 0x7f5654103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56541109c0 con 0x7f56541006a0 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5654110840 con 0x7f5654101060 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565bb10640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5654110b40 con 0x7f56541066b0 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49222/0 (socket says 172.21.15.116:49222) 2024-03-19T02:49:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 -- 172.21.15.116:0/2184101220 learned_addr learned my addr 172.21.15.116:0/2184101220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56590ab640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f5654103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f565a0ad640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541066b0 0x7f5654103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 -- 172.21.15.116:0/2184101220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541066b0 msgr2=0x7f5654103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541066b0 0x7f5654103740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 -- 172.21.15.116:0/2184101220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 msgr2=0x7f5654103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f5654103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56598ac640 1 -- 172.21.15.116:0/2184101220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56541a64d0 con 0x7f56541006a0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.319+0000 7f56590ab640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f5654103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f56598ac640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5650000c00 tx=0x7f565002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f564affd640 1 -- 172.21.15.116:0/2184101220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650043020 con 0x7f56541006a0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f564affd640 1 -- 172.21.15.116:0/2184101220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f565002caf0 con 0x7f56541006a0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56541a6760 con 0x7f56541006a0 2024-03-19T02:49:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f564affd640 1 -- 172.21.15.116:0/2184101220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650003a80 con 0x7f56541006a0 2024-03-19T02:49:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.320+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56541a6cb0 con 0x7f56541006a0 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.322+0000 7f564affd640 1 -- 172.21.15.116:0/2184101220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5650046020 con 0x7f56541006a0 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 Processor -- start 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 -- start start 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f564affd640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 0x7f56400a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abed290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abed7d0 0x56421abefbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56421ab474b0 con 0x56421abe4bf0 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56421aaefd70 con 0x56421abe42d0 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565e7dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56421ab43550 con 0x56421abed7d0 2024-03-19T02:49:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.323+0000 7f565a0ad640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 0x7f56400a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59846/0 (socket says 172.21.15.116:59846) 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 -- 172.21.15.116:0/3375450373 learned_addr learned my addr 172.21.15.116:0/3375450373 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56590ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f565a0ad640 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abed7d0 0x56421abefbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f564affd640 1 -- 172.21.15.116:0/2184101220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f56500f1030 con 0x7f56541006a0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 -- 172.21.15.116:0/3375450373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abed7d0 msgr2=0x56421abefbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abed7d0 0x56421abefbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 -- 172.21.15.116:0/3375450373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 msgr2=0x56421abed290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abed290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 -- 172.21.15.116:0/3375450373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421ab12210 con 0x56421abe42d0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f565a0ad640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 0x7f56400a4f80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f564c009fd0 tx=0x7f564c009de0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56590ab640 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abed290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f56598ac640 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5650041000 tx=0x7f5650009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a092cd3f0979431c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f563ffff640 1 -- 172.21.15.116:0/3375450373 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500bcd20 con 0x56421abe42d0 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f563ffff640 1 -- 172.21.15.116:0/3375450373 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56500bdac0 con 0x56421abe42d0 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.324+0000 7f563ffff640 1 -- 172.21.15.116:0/3375450373 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500cfdc0 con 0x56421abe42d0 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 -- 172.21.15.116:0/3375450373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 msgr2=0x56421abe46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5650041000 tx=0x7f5650009260 comp rx=0 tx=0).stop 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 -- 172.21.15.116:0/3375450373 shutdown_connections 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abed7d0 0x56421abefbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abed290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3375450373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abe46b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 -- 172.21.15.116:0/3375450373 >> 172.21.15.116:0/3375450373 conn(0x56421abdea00 msgr2=0x56421abdede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 -- 172.21.15.116:0/3375450373 shutdown_connections 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.325+0000 7f565e7dc740 1 -- 172.21.15.116:0/3375450373 wait complete. 2024-03-19T02:49:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 -- start start 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 Processor -- start 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe4bf0 0x56421abead00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421abeb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56421abf0380 con 0x56421abed7d0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56421ab03310 con 0x56421abe4bf0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56421abf0500 con 0x56421abe42d0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565e7dc740 10 client.0 _ll_register_callbacks cb 0x56421aaf72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37732/0 (socket says 172.21.15.116:37732) 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 -- 172.21.15.116:0/3482335437 learned_addr learned my addr 172.21.15.116:0/3482335437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565a0ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421abeb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56590ab640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe4bf0 0x56421abead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 -- 172.21.15.116:0/3482335437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe4bf0 msgr2=0x56421abead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe4bf0 0x56421abead00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 -- 172.21.15.116:0/3482335437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 msgr2=0x56421abeb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421abeb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f56598ac640 1 -- 172.21.15.116:0/3482335437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421a85b3f0 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.326+0000 7f565a0ad640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421abeb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f56598ac640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f56500bd350 tx=0x7f56500bb8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc36208d9a51e7e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f563d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500cffa0 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f56500cfb50 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56500bdac0 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f56500d0180 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f565e7dc740 10 client.31402 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.327+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56421a9257d0 con 0x56421abe42d0 2024-03-19T02:49:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56500bdac0 con 0x56421abe42d0 2024-03-19T02:49:33.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f56500d60d0 con 0x56421abe42d0 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 10 client.31402 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 10 client.31402 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 20 client.31402 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421abfd680 0x56421abffa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 4 client.31402 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.328+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56421aba53e0 con 0x56421abfd680 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.329+0000 7f56590ab640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421abfd680 0x56421abffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.329+0000 7f56590ab640 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421abfd680 0x56421abffa70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa2826415b32662b server_cookie=83ddbd0895d650c3 in_seq=0 out_seq=0 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.329+0000 7f563d7fa640 10 client.31402 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f563d7fa640 1 -- 172.21.15.116:0/3482335437 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56421aba53e0 con 0x56421abfd680 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f563d7fa640 10 client.31402 handle_command_reply: tid=0 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 client.31402 shutdown 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 msgr2=0x56421abea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f56500bd350 tx=0x7f56500bb8d0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 shutdown_connections 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421abfd680 0x56421abffa70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421abeb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe4bf0 0x56421abead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f563d7fa640 0 client.31402 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3482335437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abea7c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.330+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 >> 172.21.15.116:0/3482335437 conn(0x56421abdea00 msgr2=0x56421abe0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.331+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 shutdown_connections 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.331+0000 7f565e7dc740 1 -- 172.21.15.116:0/3482335437 wait complete. 2024-03-19T02:49:33.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.331+0000 7f565e7dc740 20 client.31402 trim_cache size 0 max 16384 2024-03-19T02:49:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 58904 5.82 MB/sec execute 415 sec latency 1638.703 ms 2024-03-19T02:49:33.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.347+0000 7f565e7dc740 1 Processor -- start 2024-03-19T02:49:33.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.347+0000 7f565e7dc740 1 -- start start 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abf9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421ac05380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b0e80 con 0x56421abed7d0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b2a20 con 0x56421abe42d0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565e7dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b1550 con 0x56421abe4bf0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565a0ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421ac05380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37738/0 (socket says 172.21.15.116:37738) 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 -- 172.21.15.116:0/616526079 learned_addr learned my addr 172.21.15.116:0/616526079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56598ac640 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abf9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 -- 172.21.15.116:0/616526079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 msgr2=0x56421abf9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abf9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 -- 172.21.15.116:0/616526079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 msgr2=0x56421ac05380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421ac05380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 -- 172.21.15.116:0/616526079 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421abe4a20 con 0x56421abe4bf0 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f565a0ad640 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421ac05380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.348+0000 7f56590ab640 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5644005d00 tx=0x7f5644005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0668b4696ecabb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.349+0000 7f563f7fe640 1 -- 172.21.15.116:0/616526079 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644023030 con 0x56421abe4bf0 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.349+0000 7f563f7fe640 1 -- 172.21.15.116:0/616526079 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56440054d0 con 0x56421abe4bf0 2024-03-19T02:49:33.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.349+0000 7f563f7fe640 1 -- 172.21.15.116:0/616526079 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564401f110 con 0x56421abe4bf0 2024-03-19T02:49:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.349+0000 7f565e7dc740 1 -- 172.21.15.116:0/616526079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 msgr2=0x56421abeb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.349+0000 7f565e7dc740 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5644005d00 tx=0x7f5644005f90 comp rx=0 tx=0).stop 2024-03-19T02:49:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- 172.21.15.116:0/616526079 shutdown_connections 2024-03-19T02:49:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abed7d0 0x56421ac05380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe4bf0 0x56421abeb060 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- 172.21.15.116:0/616526079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abe42d0 0x56421abf9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- 172.21.15.116:0/616526079 >> 172.21.15.116:0/616526079 conn(0x56421abdea00 msgr2=0x56421abdd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- 172.21.15.116:0/616526079 shutdown_connections 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- 172.21.15.116:0/616526079 wait complete. 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- start start 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 Processor -- start 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abf5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abed7d0 0x56421abf1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b2350 con 0x56421abe4bf0 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b1c20 con 0x56421abed7d0 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56541aee80 con 0x56421abe42d0 2024-03-19T02:49:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f565e7dc740 10 client.0 _ll_register_callbacks cb 0x7f5644009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f56590ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.350+0000 7f56590ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49268/0 (socket says 172.21.15.116:49268) 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56598ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abf5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 -- 172.21.15.116:0/3739985196 learned_addr learned my addr 172.21.15.116:0/3739985196 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f565a0ad640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abed7d0 0x56421abf1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 -- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 msgr2=0x56421abf5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abf5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 -- 172.21.15.116:0/3739985196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abed7d0 msgr2=0x56421abf1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abed7d0 0x56421abf1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 -- 172.21.15.116:0/3739985196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421abf1a00 con 0x56421abe4bf0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f56590ab640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f5644014010 tx=0x7f56440056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a0da1a321cbb76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f563d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644023040 con 0x56421abe4bf0 2024-03-19T02:49:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f5644005220 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.351+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56440054d0 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5644039050 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 10 client.21711 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56421ac639c0 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56440054d0 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f5644028440 con 0x56421abe4bf0 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 10 client.21711 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 10 client.21711 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 20 client.21711 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421ac64010 0x56421ac66430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 4 client.21711 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.352+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56421ab549f0 con 0x56421ac64010 2024-03-19T02:49:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f56598ac640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421ac64010 0x56421ac66430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f56598ac640 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421ac64010 0x56421ac66430 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2f3cade7b4e16c server_cookie=9e4a98716d949338 in_seq=0 out_seq=0 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f563d7fa640 10 client.21711 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f563d7fa640 1 -- 172.21.15.116:0/3739985196 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56421ab549f0 con 0x56421ac64010 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f563d7fa640 10 client.21711 handle_command_reply: tid=0 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 client.21711 shutdown 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 msgr2=0x56421abf0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f5644014010 tx=0x7f56440056f0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 shutdown_connections 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x56421ac64010 0x56421ac66430 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56421abed7d0 0x56421abf1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f563d7fa640 0 client.21711 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56421abe4bf0 0x56421abf0ce0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 --2- 172.21.15.116:0/3739985196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56421abe42d0 0x56421abf5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 >> 172.21.15.116:0/3739985196 conn(0x56421abdea00 msgr2=0x56421abddba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.353+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 shutdown_connections 2024-03-19T02:49:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565e7dc740 1 -- 172.21.15.116:0/3739985196 wait complete. 2024-03-19T02:49:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565e7dc740 20 client.21711 trim_cache size 0 max 16384 2024-03-19T02:49:33.360 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 msgr2=0x7f56400a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 0x7f56400a4f80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f564c009fd0 tx=0x7f564c009de0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 msgr2=0x7f5654102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f5650000c00 tx=0x7f565002c660 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 shutdown_connections 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f56400a2ac0 0x7f56400a4f80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56541066b0 0x7f5654103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5654101060 0x7f5654103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 --2- 172.21.15.116:0/2184101220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56541006a0 0x7f5654102cc0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 >> 172.21.15.116:0/2184101220 conn(0x7f56540778d0 msgr2=0x7f5654068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 shutdown_connections 2024-03-19T02:49:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.354+0000 7f565bb10640 1 -- 172.21.15.116:0/2184101220 wait complete. 2024-03-19T02:49:33.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: pgmap v686: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 214 op/s 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:33.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:32 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2178673801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:49:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 -- 172.21.15.116:0/3622350949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 msgr2=0x7f5714100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 --2- 172.21.15.116:0/3622350949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714100b20 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f5704009930 tx=0x7f570402bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 -- 172.21.15.116:0/3622350949 shutdown_connections 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 --2- 172.21.15.116:0/3622350949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5714101060 0x7f571410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 --2- 172.21.15.116:0/3622350949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714100b20 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 --2- 172.21.15.116:0/3622350949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57141066b0 0x7f5714106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 -- 172.21.15.116:0/3622350949 >> 172.21.15.116:0/3622350949 conn(0x7f57140778d0 msgr2=0x7f5714077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 -- 172.21.15.116:0/3622350949 shutdown_connections 2024-03-19T02:49:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.500+0000 7f571ba01640 1 -- 172.21.15.116:0/3622350949 wait complete. 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 Processor -- start 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 -- start start 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5714101060 0x7f5714197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57141066b0 0x7f571419b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5714110970 con 0x7f57141006a0 2024-03-19T02:49:33.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57141107f0 con 0x7f5714101060 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571ba01640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5714110af0 con 0x7f57141066b0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49272/0 (socket says 172.21.15.116:49272) 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 -- 172.21.15.116:0/3984367089 learned_addr learned my addr 172.21.15.116:0/3984367089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f5718f9c640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5714101060 0x7f5714197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f5719f9e640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57141066b0 0x7f571419b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 -- 172.21.15.116:0/3984367089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57141066b0 msgr2=0x7f571419b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57141066b0 0x7f571419b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 -- 172.21.15.116:0/3984367089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5714101060 msgr2=0x7f5714197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5714101060 0x7f5714197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.501+0000 7f571979d640 1 -- 172.21.15.116:0/3984367089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571419bec0 con 0x7f57141006a0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f571979d640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f571000ecd0 tx=0x7f571000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f570a7fc640 1 -- 172.21.15.116:0/3984367089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571001c070 con 0x7f57141006a0 2024-03-19T02:49:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f570a7fc640 1 -- 172.21.15.116:0/3984367089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57100178c0 con 0x7f57141006a0 2024-03-19T02:49:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571419c1b0 con 0x7f57141006a0 2024-03-19T02:49:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f570a7fc640 1 -- 172.21.15.116:0/3984367089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710016880 con 0x7f57141006a0 2024-03-19T02:49:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.502+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5714068980 con 0x7f57141006a0 2024-03-19T02:49:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.504+0000 7f570a7fc640 1 -- 172.21.15.116:0/3984367089 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f571001f070 con 0x7f57141006a0 2024-03-19T02:49:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 Processor -- start 2024-03-19T02:49:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 -- start start 2024-03-19T02:49:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe6bf0 0x5593e6fef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593e6f494b0 con 0x5593e6fe62d0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593e6ef1d70 con 0x5593e6fef7d0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571e5ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593e6f45550 con 0x5593e6fe6bf0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f570a7fc640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 0x7f57000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49276/0 (socket says 172.21.15.116:49276) 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 -- 172.21.15.116:0/557850590 learned_addr learned my addr 172.21.15.116:0/557850590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f571979d640 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe6bf0 0x5593e6fef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5718f9c640 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f570a7fc640 1 -- 172.21.15.116:0/3984367089 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f57100d0b00 con 0x7f57141006a0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 0x7f57000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 -- 172.21.15.116:0/557850590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe6bf0 msgr2=0x5593e6fef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe6bf0 0x5593e6fef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 -- 172.21.15.116:0/557850590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 msgr2=0x5593e6ff1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.505+0000 7f5719f9e640 1 -- 172.21.15.116:0/557850590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593e6f14210 con 0x5593e6fe62d0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.506+0000 7f5719f9e640 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f570c0138b0 tx=0x7f570c013d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eac34a97cba569c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.506+0000 7f5719f9e640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 0x7f57000a4e30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f570c00f790 tx=0x7f570c020000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.506+0000 7f56ff7fe640 1 -- 172.21.15.116:0/557850590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c0120d0 con 0x5593e6fe62d0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.506+0000 7f56ff7fe640 1 -- 172.21.15.116:0/557850590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c023020 con 0x5593e6fe62d0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.506+0000 7f56ff7fe640 1 -- 172.21.15.116:0/557850590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c01fdb0 con 0x5593e6fe62d0 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- 172.21.15.116:0/557850590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 msgr2=0x5593e6fe66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f570c0138b0 tx=0x7f570c013d70 comp rx=0 tx=0).stop 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- 172.21.15.116:0/557850590 shutdown_connections 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe6bf0 0x5593e6fef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 --2- 172.21.15.116:0/557850590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe62d0 0x5593e6fe66b0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- 172.21.15.116:0/557850590 >> 172.21.15.116:0/557850590 conn(0x5593e6fe0a00 msgr2=0x5593e6fe0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- 172.21.15.116:0/557850590 shutdown_connections 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- 172.21.15.116:0/557850590 wait complete. 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.507+0000 7f571e5ea740 1 -- start start 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 Processor -- start 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 0x5593e6fec840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fef7d0 0x5593e6fed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593e6ff2500 con 0x5593e6fe6bf0 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593e6f05310 con 0x5593e6fe62d0 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593e6ff2680 con 0x5593e6fef7d0 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f5718f9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f5718f9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49278/0 (socket says 172.21.15.116:49278) 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f5718f9c640 1 -- 172.21.15.116:0/2229161169 learned_addr learned my addr 172.21.15.116:0/2229161169 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 0x5593e6fec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f5719f9e640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fef7d0 0x5593e6fed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571e5ea740 10 client.0 _ll_register_callbacks cb 0x5593e6ef92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 -- 172.21.15.116:0/2229161169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fef7d0 msgr2=0x5593e6fed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fef7d0 0x5593e6fed2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 -- 172.21.15.116:0/2229161169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 msgr2=0x5593e6fecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f571979d640 1 -- 172.21.15.116:0/2229161169 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593e6c5d3f0 con 0x5593e6fe62d0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.508+0000 7f5718f9c640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f571979d640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 0x5593e6fec840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f57100cde70 tx=0x7f571009c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b93ce620b56033bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f56fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57100a5520 con 0x5593e6fe62d0 2024-03-19T02:49:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570c023020 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f5710017410 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5710086120 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f571e5ea740 10 client.31445 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.509+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5593e6d277d0 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c023020 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f5710098c10 con 0x5593e6fe62d0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 10 client.31445 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 10 client.31445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 20 client.31445 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e6fffaf0 0x5593e7001f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 4 client.31445 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593e6fa73e0 con 0x5593e6fffaf0 2024-03-19T02:49:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f5718f9c640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e6fffaf0 0x5593e7001f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f56fcff9640 10 client.31445 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.510+0000 7f5718f9c640 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e6fffaf0 0x5593e7001f60 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51d62d52c11bed89 server_cookie=b3eeb354a8c6d9ae in_seq=0 out_seq=0 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f56fcff9640 1 -- 172.21.15.116:0/2229161169 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593e6fa73e0 con 0x5593e6fffaf0 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f56fcff9640 10 client.31445 handle_command_reply: tid=0 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 client.31445 shutdown 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 msgr2=0x5593e6fec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 0x5593e6fec840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f57100cde70 tx=0x7f571009c850 comp rx=0 tx=0).stop 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 shutdown_connections 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e6fffaf0 0x5593e7001f60 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fef7d0 0x5593e6fed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6fecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2229161169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe62d0 0x5593e6fec840 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f56fcff9640 0 client.31445 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 >> 172.21.15.116:0/2229161169 conn(0x5593e6fe0a00 msgr2=0x5593e6fe2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 shutdown_connections 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 1 -- 172.21.15.116:0/2229161169 wait complete. 2024-03-19T02:49:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.512+0000 7f571e5ea740 20 client.31445 trim_cache size 0 max 16384 2024-03-19T02:49:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 Processor -- start 2024-03-19T02:49:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 -- start start 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe6bf0 0x5593e6ffb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57141acad0 con 0x5593e6fef7d0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57141ae670 con 0x5593e6fe6bf0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571e5ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57141ad1a0 con 0x5593e6fe62d0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f5719f9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571979d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f5719f9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49294/0 (socket says 172.21.15.116:49294) 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f571979d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37784/0 (socket says 172.21.15.116:37784) 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f5719f9e640 1 -- 172.21.15.116:0/2483156746 learned_addr learned my addr 172.21.15.116:0/2483156746 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.531+0000 7f5718f9c640 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe6bf0 0x5593e6ffb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 -- 172.21.15.116:0/2483156746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe6bf0 msgr2=0x5593e6ffb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe6bf0 0x5593e6ffb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 -- 172.21.15.116:0/2483156746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 msgr2=0x5593e6febff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 -- 172.21.15.116:0/2483156746 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593e6fe15d0 con 0x5593e6fe62d0 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f5719f9e640 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f571979d640 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f571009b0e0 tx=0x7f571009c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ac1c7572506255d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f56feffd640 1 -- 172.21.15.116:0/2483156746 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571009c970 con 0x5593e6fe62d0 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f56feffd640 1 -- 172.21.15.116:0/2483156746 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57100a5520 con 0x5593e6fe62d0 2024-03-19T02:49:33.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.532+0000 7f56feffd640 1 -- 172.21.15.116:0/2483156746 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571009c280 con 0x5593e6fe62d0 2024-03-19T02:49:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 -- 172.21.15.116:0/2483156746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 msgr2=0x5593e6fdf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f571009b0e0 tx=0x7f571009c8c0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 -- 172.21.15.116:0/2483156746 shutdown_connections 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fef7d0 0x5593e6febff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fe6bf0 0x5593e6ffb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 --2- 172.21.15.116:0/2483156746 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6fdf310 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 -- 172.21.15.116:0/2483156746 >> 172.21.15.116:0/2483156746 conn(0x5593e6fe0a00 msgr2=0x5593e6fed090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 -- 172.21.15.116:0/2483156746 shutdown_connections 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.533+0000 7f571e5ea740 1 -- 172.21.15.116:0/2483156746 wait complete. 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 -- start start 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 Processor -- start 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6ff7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57141adfa0 con 0x5593e6fe6bf0 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57141ad870 con 0x5593e6fef7d0 2024-03-19T02:49:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57141aaad0 con 0x5593e6fe62d0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571e5ea740 10 client.0 _ll_register_callbacks cb 0x7f57100099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49306/0 (socket says 172.21.15.116:49306) 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 -- 172.21.15.116:0/1740410393 learned_addr learned my addr 172.21.15.116:0/1740410393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f571979d640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6ff7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5719f9e640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 -- 172.21.15.116:0/1740410393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 msgr2=0x5593e6ff7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6ff7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 -- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 msgr2=0x5593e6ff3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.534+0000 7f5718f9c640 1 -- 172.21.15.116:0/1740410393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593e7064e60 con 0x5593e6fe6bf0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f5718f9c640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f5704036040 tx=0x7f570402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6d985758a7dab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570402cc20 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f570404bab0 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57100a5520 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5704059050 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 10 client.21729 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5593e70660b0 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57100a5520 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f57040053d0 con 0x5593e6fe6bf0 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 10 client.21729 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 10 client.21729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 20 client.21729 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e7066860 0x5593e7068c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 4 client.21729 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.535+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593e6f569f0 con 0x5593e7066860 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.536+0000 7f571979d640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e7066860 0x5593e7068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.536+0000 7f571979d640 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e7066860 0x5593e7068c80 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9215a33dcccc4b50 server_cookie=120e34c57d1ca5c3 in_seq=0 out_seq=0 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.536+0000 7f56fcff9640 10 client.21729 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f56fcff9640 1 -- 172.21.15.116:0/1740410393 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593e6f569f0 con 0x5593e7066860 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f56fcff9640 10 client.21729 handle_command_reply: tid=0 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 client.21729 shutdown 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 msgr2=0x5593e6ff3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f5704036040 tx=0x7f570402c7e0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 shutdown_connections 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5593e7066860 0x5593e7068c80 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e6fef7d0 0x5593e6ff3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593e6fe6bf0 0x5593e6ff3150 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f56fcff9640 0 client.21729 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 --2- 172.21.15.116:0/1740410393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593e6fe62d0 0x5593e6ff7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 >> 172.21.15.116:0/1740410393 conn(0x5593e6fe0a00 msgr2=0x5593e6fedef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 shutdown_connections 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 1 -- 172.21.15.116:0/1740410393 wait complete. 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571e5ea740 20 client.21729 trim_cache size 0 max 16384 2024-03-19T02:49:33.544 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 msgr2=0x7f57000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 0x7f57000a4e30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f570c00f790 tx=0x7f570c020000 comp rx=0 tx=0).stop 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 msgr2=0x7f5714196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f571000ecd0 tx=0x7f571000b690 comp rx=0 tx=0).stop 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 shutdown_connections 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f57000a2970 0x7f57000a4e30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57141066b0 0x7f571419b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5714101060 0x7f5714197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.538+0000 7f571ba01640 1 --2- 172.21.15.116:0/3984367089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57141006a0 0x7f5714196ed0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.539+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 >> 172.21.15.116:0/3984367089 conn(0x7f57140778d0 msgr2=0x7f5714102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.539+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 shutdown_connections 2024-03-19T02:49:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:33.539+0000 7f571ba01640 1 -- 172.21.15.116:0/3984367089 wait complete. 2024-03-19T02:49:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:33 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:33 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:33 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:33 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:33 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:33 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:34.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59017 5.81 MB/sec execute 416 sec latency 2638.733 ms 2024-03-19T02:49:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:33 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:33 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:33 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:49:34.553 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:49:34.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:49:34.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/2596403849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 msgr2=0x7f5e18106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/2596403849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e18106a80 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009930 tx=0x7f5e1402bc30 comp rx=0 tx=0).stop 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/2596403849 shutdown_connections 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/2596403849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e1810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/2596403849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/2596403849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e18106a80 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.669+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/2596403849 >> 172.21.15.116:0/2596403849 conn(0x7f5e18077820 msgr2=0x7f5e18077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/2596403849 shutdown_connections 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/2596403849 wait complete. 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 Processor -- start 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- start start 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e181984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18110920 con 0x7f5e181066a0 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181107a0 con 0x7f5e18101050 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1fe0c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18110aa0 con 0x7f5e18100690 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1e3a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1dba8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1e3a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49324/0 (socket says 172.21.15.116:49324) 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1e3a9640 1 -- 172.21.15.116:0/3350956810 learned_addr learned my addr 172.21.15.116:0/3350956810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.670+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e181984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 -- 172.21.15.116:0/3350956810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 msgr2=0x7f5e18197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 -- 172.21.15.116:0/3350956810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 msgr2=0x7f5e181984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e181984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 -- 172.21.15.116:0/3350956810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1819cf50 con 0x7f5e181066a0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e181984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1e3a9640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000eb10 tx=0x7f5e1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e0effd640 1 -- 172.21.15.116:0/3350956810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10004b20 con 0x7f5e181066a0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e0effd640 1 -- 172.21.15.116:0/3350956810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e10004cc0 con 0x7f5e181066a0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1819d240 con 0x7f5e181066a0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e0effd640 1 -- 172.21.15.116:0/3350956810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10010690 con 0x7f5e181066a0 2024-03-19T02:49:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.671+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e181a4ad0 con 0x7f5e181066a0 2024-03-19T02:49:34.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e0effd640 1 -- 172.21.15.116:0/3350956810 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5e10005020 con 0x7f5e181066a0 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 Processor -- start 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 -- start start 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cca5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cca7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56397cc00160 con 0x56397cc9c380 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56397cba8e10 con 0x56397cca5880 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e229f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56397cbfc200 con 0x56397cc9cca0 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e0effd640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 0x7f5e040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49328/0 (socket says 172.21.15.116:49328) 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/974569708 learned_addr learned my addr 172.21.15.116:0/974569708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cca5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cca7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.674+0000 7f5e1e3a9640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 0x7f5e040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e0effd640 1 -- 172.21.15.116:0/3350956810 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f5e100cf790 con 0x7f5e181066a0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/974569708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 msgr2=0x56397cca5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cca5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/974569708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 msgr2=0x56397cca7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cca7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/974569708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56397c9dc7e0 con 0x56397cc9c380 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f5e08013a20 tx=0x7f5e08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37bd3f51f455f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e1e3a9640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 0x7f5e040a4e30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5e1009b710 tx=0x7f5e10086040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e03fff640 1 -- 172.21.15.116:0/974569708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800f980 con 0x56397cc9c380 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e03fff640 1 -- 172.21.15.116:0/974569708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e08005020 con 0x56397cc9c380 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.675+0000 7f5e03fff640 1 -- 172.21.15.116:0/974569708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08004b20 con 0x56397cc9c380 2024-03-19T02:49:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- 172.21.15.116:0/974569708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 msgr2=0x56397cc9c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f5e08013a20 tx=0x7f5e08009b50 comp rx=0 tx=0).stop 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- 172.21.15.116:0/974569708 shutdown_connections 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cca7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cca5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- 172.21.15.116:0/974569708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9c760 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- 172.21.15.116:0/974569708 >> 172.21.15.116:0/974569708 conn(0x56397cc96ab0 msgr2=0x56397cc96e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- 172.21.15.116:0/974569708 shutdown_connections 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- 172.21.15.116:0/974569708 wait complete. 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 -- start start 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 Processor -- start 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9c380 0x56397cca27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.676+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397cca3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56397cca8670 con 0x56397cc9cca0 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56397cbbc1a0 con 0x56397cc9c380 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56397cca87f0 con 0x56397cca5880 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 10 client.0 _ll_register_callbacks cb 0x56397c913c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49344/0 (socket says 172.21.15.116:49344) 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 -- 172.21.15.116:0/1155352880 learned_addr learned my addr 172.21.15.116:0/1155352880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9c380 0x56397cca27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397cca3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 -- 172.21.15.116:0/1155352880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 msgr2=0x56397cca3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397cca3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 -- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9c380 msgr2=0x56397cca27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9c380 0x56397cca27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 -- 172.21.15.116:0/1155352880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56397ca3cf70 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397cca3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5e14005e00 tx=0x7f5e1402c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41d8993cdf3573 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e017fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1402c5d0 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f5e140035f0 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e08005020 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5e14054050 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.677+0000 7f5e229f1740 10 client.21741 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56397c8d4170 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e08005020 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f5e1404b940 con 0x56397cc9cca0 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 10 client.21741 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:34.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 10 client.21741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 20 client.21741 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397ccb5b00 0x56397ccb7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 4 client.21741 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56397cc0d6a0 con 0x56397ccb5b00 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.678+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397ccb5b00 0x56397ccb7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.679+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397ccb5b00 0x56397ccb7f10 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baef31adef93333c server_cookie=4e74e5e57f703bea in_seq=0 out_seq=0 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.679+0000 7f5e017fa640 10 client.21741 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e017fa640 1 -- 172.21.15.116:0/1155352880 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56397cc0d6a0 con 0x56397ccb5b00 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e017fa640 10 client.21741 handle_command_reply: tid=0 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 client.21741 shutdown 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 msgr2=0x56397cca2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5e14005e00 tx=0x7f5e1402c9f0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 shutdown_connections 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397ccb5b00 0x56397ccb7f10 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397cca3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e017fa640 0 client.21741 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9cca0 0x56397cca2d10 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 --2- 172.21.15.116:0/1155352880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9c380 0x56397cca27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 >> 172.21.15.116:0/1155352880 conn(0x56397cc96ab0 msgr2=0x56397cc98800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 shutdown_connections 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 1 -- 172.21.15.116:0/1155352880 wait complete. 2024-03-19T02:49:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.680+0000 7f5e229f1740 20 client.21741 trim_cache size 0 max 16384 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.701+0000 7f5e229f1740 1 Processor -- start 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 -- start start 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9cca0 0x56397ccb1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397ccbd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181aecc0 con 0x56397cc9c380 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b0860 con 0x56397cc9cca0 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e229f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181af390 con 0x56397cca5880 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49352/0 (socket says 172.21.15.116:49352) 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2975884576 learned_addr learned my addr 172.21.15.116:0/2975884576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9cca0 0x56397ccb1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397ccbd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2975884576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 msgr2=0x56397ccbd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397ccbd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2975884576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9cca0 msgr2=0x56397ccb1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9cca0 0x56397ccb1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2975884576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56397cc9cad0 con 0x56397cc9c380 2024-03-19T02:49:34.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.702+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009ce0 tx=0x7f5e08009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464abca690250a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e037fe640 1 -- 172.21.15.116:0/2975884576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08028040 con 0x56397cc9c380 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e037fe640 1 -- 172.21.15.116:0/2975884576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080054b0 con 0x56397cc9c380 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e037fe640 1 -- 172.21.15.116:0/2975884576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800fe10 con 0x56397cc9c380 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 -- 172.21.15.116:0/2975884576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 msgr2=0x56397cca4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009ce0 tx=0x7f5e08009d10 comp rx=0 tx=0).stop 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 -- 172.21.15.116:0/2975884576 shutdown_connections 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cca5880 0x56397ccbd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cc9cca0 0x56397ccb1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2975884576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cca4f00 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.703+0000 7f5e229f1740 1 -- 172.21.15.116:0/2975884576 >> 172.21.15.116:0/2975884576 conn(0x56397cc96ab0 msgr2=0x56397cc952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- 172.21.15.116:0/2975884576 shutdown_connections 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- 172.21.15.116:0/2975884576 wait complete. 2024-03-19T02:49:34.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- start start 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 Processor -- start 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cc9fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cd1ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181b0190 con 0x56397cc9c380 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181afa60 con 0x56397cca5880 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e181accc0 con 0x56397cc9cca0 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e229f1740 10 client.0 _ll_register_callbacks cb 0x7f5e14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e1d3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49364/0 (socket says 172.21.15.116:49364) 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2808171229 learned_addr learned my addr 172.21.15.116:0/2808171229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.704+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cc9fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2808171229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 msgr2=0x56397cc9fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cc9fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 msgr2=0x56397cd1ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cd1ae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 -- 172.21.15.116:0/2808171229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56397cd1b350 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cc9fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e1d3a7640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f5e080056d0 tx=0x7f5e0800f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b861fd8971137091 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e017fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800fb90 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f5e0800fd30 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e080054b0 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5e0803f050 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e229f1740 10 client.21747 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.705+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56397cd1c530 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080054b0 con 0x56397cc9c380 2024-03-19T02:49:34.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f5e0801f9b0 con 0x56397cc9c380 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 10 client.21747 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 10 client.21747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 20 client.21747 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397cd1ca00 0x56397cd1edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 4 client.21747 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56397cc079f0 con 0x56397cd1ca00 2024-03-19T02:49:34.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.706+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397cd1ca00 0x56397cd1edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.707+0000 7f5e1dba8640 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397cd1ca00 0x56397cd1edf0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f07e5b7c2d759bc server_cookie=2f532cbb3def52e5 in_seq=0 out_seq=0 2024-03-19T02:49:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.707+0000 7f5e017fa640 10 client.21747 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e017fa640 1 -- 172.21.15.116:0/2808171229 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56397cc079f0 con 0x56397cd1ca00 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e017fa640 10 client.21747 handle_command_reply: tid=0 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 client.21747 shutdown 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 msgr2=0x56397cc9f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f5e080056d0 tx=0x7f5e0800f980 comp rx=0 tx=0).stop 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 shutdown_connections 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56397cd1ca00 0x56397cd1edf0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.709+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56397cca5880 0x56397cd1ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56397cc9cca0 0x56397cc9fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 1 --2- 172.21.15.116:0/2808171229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56397cc9c380 0x56397cc9f930 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 >> 172.21.15.116:0/2808171229 conn(0x56397cc96ab0 msgr2=0x56397cca3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e017fa640 0 client.21747 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 shutdown_connections 2024-03-19T02:49:34.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 1 -- 172.21.15.116:0/2808171229 wait complete. 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e229f1740 20 client.21747 trim_cache size 0 max 16384 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "6495bcf5-547b-4db9-87a5-99817958a422", 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:49:34.715 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 msgr2=0x7f5e040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 0x7f5e040a4e30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5e1009b710 tx=0x7f5e10086040 comp rx=0 tx=0).stop 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 msgr2=0x7f5e1819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000eb10 tx=0x7f5e1000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 shutdown_connections 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5e040a2970 0x7f5e040a4e30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e181066a0 0x7f5e1819c820 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e18101050 0x7f5e181984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 --2- 172.21.15.116:0/3350956810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e18100690 0x7f5e18197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 >> 172.21.15.116:0/3350956810 conn(0x7f5e18077820 msgr2=0x7f5e18068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 shutdown_connections 2024-03-19T02:49:34.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.710+0000 7f5e1fe0c640 1 -- 172.21.15.116:0/3350956810 wait complete. 2024-03-19T02:49:34.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:49:34.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:49:34.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 -- 172.21.15.116:0/1186452281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee0100680 msgr2=0x7f7ee010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 --2- 172.21.15.116:0/1186452281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee0100680 0x7f7ee010e640 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009ca0 tx=0x7f7ed802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 -- 172.21.15.116:0/1186452281 shutdown_connections 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 --2- 172.21.15.116:0/1186452281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee0100680 0x7f7ee010e640 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 --2- 172.21.15.116:0/1186452281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 0x7f7ee0100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 --2- 172.21.15.116:0/1186452281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee01077c0 0x7f7ee0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 -- 172.21.15.116:0/1186452281 >> 172.21.15.116:0/1186452281 conn(0x7f7ee0076c40 msgr2=0x7f7ee00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 -- 172.21.15.116:0/1186452281 shutdown_connections 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.845+0000 7f7ee4819640 1 -- 172.21.15.116:0/1186452281 wait complete. 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 Processor -- start 2024-03-19T02:49:34.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 -- start start 2024-03-19T02:49:34.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 0x7f7ee019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee0100680 0x7f7ee019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee0111a60 con 0x7f7ee01077c0 2024-03-19T02:49:34.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01118e0 con 0x7f7ee00ffcc0 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.846+0000 7f7ee4819640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee0111be0 con 0x7f7ee0100680 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49372/0 (socket says 172.21.15.116:49372) 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 -- 172.21.15.116:0/592728016 learned_addr learned my addr 172.21.15.116:0/592728016 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee0100680 0x7f7ee019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ede59c640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 0x7f7ee019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 -- 172.21.15.116:0/592728016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee0100680 msgr2=0x7f7ee019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee0100680 0x7f7ee019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 -- 172.21.15.116:0/592728016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 msgr2=0x7f7ee019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 0x7f7ee019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 -- 172.21.15.116:0/592728016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee019f150 con 0x7f7ee01077c0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7eded9d640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802c700 tx=0x7f7ed8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ed77fe640 1 -- 172.21.15.116:0/592728016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8039040 con 0x7f7ee01077c0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee019f3e0 con 0x7f7ee01077c0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ed77fe640 1 -- 172.21.15.116:0/592728016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ed8044070 con 0x7f7ee01077c0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ed77fe640 1 -- 172.21.15.116:0/592728016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80042a0 con 0x7f7ee01077c0 2024-03-19T02:49:34.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.847+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee01a6cd0 con 0x7f7ee01077c0 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ed77fe640 1 -- 172.21.15.116:0/592728016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7ed8003970 con 0x7f7ee01077c0 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 Processor -- start 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 -- start start 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ed77fe640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 0x7f7ecc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 0x7f7ecc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a02be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ed77fe640 1 -- 172.21.15.116:0/592728016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f7ed80f14a0 con 0x7f7ee01077c0 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a0349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a037320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564969f8f500 con 0x56496a034f30 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564969f37be0 con 0x56496a02ba30 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ee7408740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564969f8b5a0 con 0x56496a02c350 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.850+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a02be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a0349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a037320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a0349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37882/0 (socket says 172.21.15.116:37882) 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed7fff640 1 -- 172.21.15.116:0/3452639834 learned_addr learned my addr 172.21.15.116:0/3452639834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 -- 172.21.15.116:0/3452639834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 msgr2=0x56496a0349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a0349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 -- 172.21.15.116:0/3452639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 msgr2=0x56496a02be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 0x7f7ecc0a4e30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8011550 tx=0x7f7ec8011950 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a02be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 -- 172.21.15.116:0/3452639834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564969f59000 con 0x56496a034f30 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7eded9d640 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a037320 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80ee9c0 tx=0x7f7ed8035f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7483b52193c43ff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed4ff9640 1 -- 172.21.15.116:0/3452639834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80326f0 con 0x56496a034f30 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed4ff9640 1 -- 172.21.15.116:0/3452639834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ed8034700 con 0x56496a034f30 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.851+0000 7f7ed4ff9640 1 -- 172.21.15.116:0/3452639834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80f11b0 con 0x56496a034f30 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- 172.21.15.116:0/3452639834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 msgr2=0x56496a037320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a037320 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80ee9c0 tx=0x7f7ed8035f50 comp rx=0 tx=0).stop 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- 172.21.15.116:0/3452639834 shutdown_connections 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a037320 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a0349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 --2- 172.21.15.116:0/3452639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a02be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- 172.21.15.116:0/3452639834 >> 172.21.15.116:0/3452639834 conn(0x56496a026160 msgr2=0x56496a026540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- 172.21.15.116:0/3452639834 shutdown_connections 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- 172.21.15.116:0/3452639834 wait complete. 2024-03-19T02:49:34.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 -- start start 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.852+0000 7f7ee7408740 1 Processor -- start 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a031f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a032460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56496a037ae0 con 0x56496a034f30 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564969f4b160 con 0x56496a02c350 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56496a037c60 con 0x56496a02ba30 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 10 client.0 _ll_register_callbacks cb 0x564969f87a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a031f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49396/0 (socket says 172.21.15.116:49396) 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ed7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a032460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a031f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37890/0 (socket says 172.21.15.116:37890) 2024-03-19T02:49:34.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 -- 172.21.15.116:0/1367781336 learned_addr learned my addr 172.21.15.116:0/1367781336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 -- 172.21.15.116:0/1367781336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 msgr2=0x56496a031f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a031f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 -- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 msgr2=0x56496a032460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a032460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 -- 172.21.15.116:0/1367781336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564969d6d7d0 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80eea70 tx=0x7f7ed8003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378139112b8f2fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80bdfd0 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f7ed80d66a0 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.853+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564969f9ca40 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7ed80bd8d0 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 10 client.21759 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x564969ca33f0 con 0x56496a034f30 2024-03-19T02:49:34.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564969f9ca40 con 0x56496a034f30 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7ed803c030 con 0x56496a034f30 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 10 client.21759 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 10 client.21759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 20 client.21759 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a044e20 0x56496a047210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 4 client.21759 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564969f96d90 con 0x56496a044e20 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ede59c640 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a044e20 0x56496a047210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ede59c640 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a044e20 0x56496a047210 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78cafa2cc8b31c28 server_cookie=5db585b879e7778d in_seq=0 out_seq=0 2024-03-19T02:49:34.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.854+0000 7f7ec5ffb640 10 client.21759 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/1367781336 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564969f96d90 con 0x56496a044e20 2024-03-19T02:49:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ec5ffb640 10 client.21759 handle_command_reply: tid=0 2024-03-19T02:49:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 client.21759 shutdown 2024-03-19T02:49:34.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 msgr2=0x56496a0329a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80eea70 tx=0x7f7ed8003e90 comp rx=0 tx=0).stop 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 shutdown_connections 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a044e20 0x56496a047210 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0329a0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a032460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ec5ffb640 0 client.21759 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1367781336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a031f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 >> 172.21.15.116:0/1367781336 conn(0x56496a026160 msgr2=0x56496a027eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 shutdown_connections 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 1 -- 172.21.15.116:0/1367781336 wait complete. 2024-03-19T02:49:34.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.859+0000 7f7ee7408740 20 client.21759 trim_cache size 0 max 16384 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 Processor -- start 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 -- start start 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a030e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a02d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01b0d20 con 0x56496a034f30 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01b28c0 con 0x56496a02c350 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ee7408740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01b13f0 con 0x56496a02ba30 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a030e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49404/0 (socket says 172.21.15.116:49404) 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7eded9d640 1 -- 172.21.15.116:0/1174675857 learned_addr learned my addr 172.21.15.116:0/1174675857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.877+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 -- 172.21.15.116:0/1174675857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 msgr2=0x56496a030e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a030e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 -- 172.21.15.116:0/1174675857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 msgr2=0x56496a02d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a02d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 -- 172.21.15.116:0/1174675857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56496a02c180 con 0x56496a034f30 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7ede59c640 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a030e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7eded9d640 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80bd5d0 tx=0x7f7ed80327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102494dce2c07118 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7ec7fff640 1 -- 172.21.15.116:0/1174675857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80bdfd0 con 0x56496a034f30 2024-03-19T02:49:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7ec7fff640 1 -- 172.21.15.116:0/1174675857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ed80c1a20 con 0x56496a034f30 2024-03-19T02:49:34.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.878+0000 7f7ec7fff640 1 -- 172.21.15.116:0/1174675857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80b9f40 con 0x56496a034f30 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- 172.21.15.116:0/1174675857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 msgr2=0x56496a0407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80bd5d0 tx=0x7f7ed80327b0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- 172.21.15.116:0/1174675857 shutdown_connections 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0407d0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02c350 0x56496a02d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 --2- 172.21.15.116:0/1174675857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02ba30 0x56496a030e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- 172.21.15.116:0/1174675857 >> 172.21.15.116:0/1174675857 conn(0x56496a026160 msgr2=0x56496a0251e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- 172.21.15.116:0/1174675857 shutdown_connections 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- 172.21.15.116:0/1174675857 wait complete. 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 -- start start 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.879+0000 7f7ee7408740 1 Processor -- start 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01b21f0 con 0x56496a034f30 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01b1ac0 con 0x56496a02ba30 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee01aed20 con 0x56496a02c350 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ee7408740 10 client.0 _ll_register_callbacks cb 0x7f7ed00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7eded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49408/0 (socket says 172.21.15.116:49408) 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ede59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a037ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:60058/0 (socket says 172.21.15.116:60058) 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7eded9d640 1 -- 172.21.15.116:0/196609073 learned_addr learned my addr 172.21.15.116:0/196609073 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37922/0 (socket says 172.21.15.116:37922) 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 -- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 msgr2=0x56496a037ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a037ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 -- 172.21.15.116:0/196609073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 msgr2=0x56496a0a3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 -- 172.21.15.116:0/196609073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564969c630f0 con 0x56496a02c350 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7eded9d640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ed7fff640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x56496a031780 tx=0x7f7ec8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1322a5bbc2d371a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.880+0000 7f7ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec801a040 con 0x56496a02c350 2024-03-19T02:49:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ed80c1a20 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f7ec8005020 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7ec8032050 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 10 client.31462 fetch_fsmap learned FSMap version 40 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56496a0aa8a0 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ed80c1a20 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f7ec80035f0 con 0x56496a02c350 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 10 client.31462 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 10 client.31462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 20 client.31462 populate_metadata read hostname 'smithi116' 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a0ab000 0x56496a0ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 4 client.31462 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.881+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564969f14540 con 0x56496a0ab000 2024-03-19T02:49:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.882+0000 7f7ede59c640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a0ab000 0x56496a0ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.882+0000 7f7ec5ffb640 10 client.31462 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.882+0000 7f7ede59c640 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a0ab000 0x56496a0ad3f0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633290f48cccb22c server_cookie=73d7c56f6668dfcd in_seq=0 out_seq=0 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ec5ffb640 1 -- 172.21.15.116:0/196609073 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x564969f14540 con 0x56496a0ab000 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ec5ffb640 10 client.31462 handle_command_reply: tid=0 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 client.31462 shutdown 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 msgr2=0x56496a038530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x56496a031780 tx=0x7f7ec8004730 comp rx=0 tx=0).stop 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 shutdown_connections 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x56496a0ab000 0x56496a0ad3f0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56496a034f30 0x56496a0a3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56496a02c350 0x56496a038530 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 --2- 172.21.15.116:0/196609073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56496a02ba30 0x56496a037ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ec5ffb640 0 client.31462 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.884+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 >> 172.21.15.116:0/196609073 conn(0x56496a026160 msgr2=0x56496a024950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.885+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 shutdown_connections 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.885+0000 7f7ee7408740 1 -- 172.21.15.116:0/196609073 wait complete. 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.885+0000 7f7ee7408740 20 client.31462 trim_cache size 0 max 16384 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (480 inodes in the stack)", 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "6495bcf5-547b-4db9-87a5-99817958a422": { 2024-03-19T02:49:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "6495bcf5-547b-4db9-87a5-99817958a422", 2024-03-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:49:34.891 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 msgr2=0x7f7ecc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 0x7f7ecc0a4e30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8011550 tx=0x7f7ec8011950 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 msgr2=0x7f7ee019ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802c700 tx=0x7f7ed8035d60 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 shutdown_connections 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ecc0a2970 0x7f7ecc0a4e30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ee01077c0 0x7f7ee019ea20 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ee0100680 0x7f7ee019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 --2- 172.21.15.116:0/592728016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00ffcc0 0x7f7ee019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 >> 172.21.15.116:0/592728016 conn(0x7f7ee0076c40 msgr2=0x7f7ee010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 shutdown_connections 2024-03-19T02:49:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:49:34.886+0000 7f7ee4819640 1 -- 172.21.15.116:0/592728016 wait complete. 2024-03-19T02:49:34.901 INFO:tasks.cephfs.filesystem:scrub status for tag:6495bcf5-547b-4db9-87a5-99817958a422 - {'path': '/', 'tag': '6495bcf5-547b-4db9-87a5-99817958a422', 'options': 'recursive,force'} 2024-03-19T02:49:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:34 smithi154 ceph-mon[35693]: pgmap v687: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 167 op/s 2024-03-19T02:49:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:34 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:34 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:34 smithi116 ceph-mon[32263]: pgmap v687: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 167 op/s 2024-03-19T02:49:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:34 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:34 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:35.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59102 5.81 MB/sec execute 417 sec latency 3047.635 ms 2024-03-19T02:49:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:34 smithi162 ceph-mon[35761]: pgmap v687: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 167 op/s 2024-03-19T02:49:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:34 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:34 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:49:36.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:35 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:49:36.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:35 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59214 5.80 MB/sec execute 418 sec latency 543.240 ms 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:36.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:35 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:49:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:36 smithi154 ceph-mon[35693]: pgmap v688: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-03-19T02:49:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:36 smithi116 ceph-mon[32263]: pgmap v688: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-03-19T02:49:37.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59324 5.80 MB/sec execute 419 sec latency 129.679 ms 2024-03-19T02:49:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:36 smithi162 ceph-mon[35761]: pgmap v688: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 244 op/s 2024-03-19T02:49:38.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59453 5.80 MB/sec execute 420 sec latency 154.900 ms 2024-03-19T02:49:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:38 smithi154 ceph-mon[35693]: pgmap v689: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-03-19T02:49:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:38 smithi116 ceph-mon[32263]: pgmap v689: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-03-19T02:49:39.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59556 5.80 MB/sec execute 421 sec latency 155.197 ms 2024-03-19T02:49:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:38 smithi162 ceph-mon[35761]: pgmap v689: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-03-19T02:49:40.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59673 5.81 MB/sec execute 422 sec latency 164.198 ms 2024-03-19T02:49:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:40 smithi154 ceph-mon[35693]: pgmap v690: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-03-19T02:49:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:40 smithi116 ceph-mon[32263]: pgmap v690: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-03-19T02:49:41.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59798 5.81 MB/sec execute 423 sec latency 162.392 ms 2024-03-19T02:49:41.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:40 smithi162 ceph-mon[35761]: pgmap v690: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-03-19T02:49:42.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 59932 5.82 MB/sec execute 424 sec latency 171.983 ms 2024-03-19T02:49:43.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:42 smithi154 ceph-mon[35693]: pgmap v691: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 261 op/s 2024-03-19T02:49:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:42 smithi116 ceph-mon[32263]: pgmap v691: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 261 op/s 2024-03-19T02:49:43.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60073 5.83 MB/sec execute 425 sec latency 141.277 ms 2024-03-19T02:49:43.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:42 smithi162 ceph-mon[35761]: pgmap v691: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 261 op/s 2024-03-19T02:49:44.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60285 5.86 MB/sec execute 426 sec latency 128.365 ms 2024-03-19T02:49:45.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:44 smithi154 ceph-mon[35693]: pgmap v692: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 194 op/s 2024-03-19T02:49:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:44 smithi116 ceph-mon[32263]: pgmap v692: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 194 op/s 2024-03-19T02:49:45.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60393 5.87 MB/sec execute 427 sec latency 147.623 ms 2024-03-19T02:49:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:44 smithi162 ceph-mon[35761]: pgmap v692: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 194 op/s 2024-03-19T02:49:46.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60498 5.87 MB/sec execute 428 sec latency 155.967 ms 2024-03-19T02:49:47.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:46 smithi154 ceph-mon[35693]: pgmap v693: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 257 op/s 2024-03-19T02:49:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:46 smithi116 ceph-mon[32263]: pgmap v693: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 257 op/s 2024-03-19T02:49:47.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60530 5.86 MB/sec execute 429 sec latency 1155.993 ms 2024-03-19T02:49:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:46 smithi162 ceph-mon[35761]: pgmap v693: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 257 op/s 2024-03-19T02:49:48.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60581 5.86 MB/sec execute 430 sec latency 2156.018 ms 2024-03-19T02:49:49.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:48 smithi154 ceph-mon[35693]: pgmap v694: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-03-19T02:49:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:48 smithi116 ceph-mon[32263]: pgmap v694: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-03-19T02:49:49.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60619 5.85 MB/sec execute 431 sec latency 3156.044 ms 2024-03-19T02:49:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:48 smithi162 ceph-mon[35761]: pgmap v694: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.5 MiB/s wr, 179 op/s 2024-03-19T02:49:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:50.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:50 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:50 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:50.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60687 5.86 MB/sec execute 432 sec latency 4156.070 ms 2024-03-19T02:49:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:50.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:50 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:51 smithi154 ceph-mon[35693]: pgmap v695: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 173 op/s 2024-03-19T02:49:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:51.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:51 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:49:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:51 smithi116 ceph-mon[32263]: pgmap v695: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 173 op/s 2024-03-19T02:49:51.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:51.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:51.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:51 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:49:51.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60776 5.86 MB/sec execute 433 sec latency 4669.370 ms 2024-03-19T02:49:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:51 smithi162 ceph-mon[35761]: pgmap v695: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 173 op/s 2024-03-19T02:49:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:51 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:49:52.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60857 5.86 MB/sec execute 434 sec latency 997.181 ms 2024-03-19T02:49:53.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:53 smithi154 ceph-mon[35693]: pgmap v696: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 211 op/s 2024-03-19T02:49:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:53 smithi116 ceph-mon[32263]: pgmap v696: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 211 op/s 2024-03-19T02:49:53.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60926 5.86 MB/sec execute 435 sec latency 1997.209 ms 2024-03-19T02:49:53.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:53 smithi162 ceph-mon[35761]: pgmap v696: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 211 op/s 2024-03-19T02:49:54.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 60989 5.85 MB/sec execute 436 sec latency 2997.234 ms 2024-03-19T02:49:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:55 smithi116 ceph-mon[32263]: pgmap v697: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-03-19T02:49:55.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61061 5.84 MB/sec execute 437 sec latency 3997.262 ms 2024-03-19T02:49:55.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:55 smithi162 ceph-mon[35761]: pgmap v697: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-03-19T02:49:55.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:55 smithi154 ceph-mon[35693]: pgmap v697: 129 pgs: 129 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 101 op/s 2024-03-19T02:49:56.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61131 5.83 MB/sec execute 438 sec latency 4510.997 ms 2024-03-19T02:49:57.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61207 5.82 MB/sec execute 439 sec latency 189.864 ms 2024-03-19T02:49:57.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:57 smithi162 ceph-mon[35761]: pgmap v698: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 149 op/s 2024-03-19T02:49:57.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:57 smithi154 ceph-mon[35693]: pgmap v698: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 149 op/s 2024-03-19T02:49:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:57 smithi116 ceph-mon[32263]: pgmap v698: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 149 op/s 2024-03-19T02:49:58.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61300 5.81 MB/sec execute 440 sec latency 197.655 ms 2024-03-19T02:49:58.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:49:58 smithi162 ceph-mon[35761]: pgmap v699: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:49:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:49:58 smithi154 ceph-mon[35693]: pgmap v699: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:49:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:49:58 smithi116 ceph-mon[32263]: pgmap v699: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:49:59.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61388 5.81 MB/sec execute 441 sec latency 213.441 ms 2024-03-19T02:50:00.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61488 5.80 MB/sec execute 442 sec latency 187.520 ms 2024-03-19T02:50:00.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:00 smithi162 ceph-mon[35761]: pgmap v700: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:50:00.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:00 smithi162 ceph-mon[35761]: overall HEALTH_OK 2024-03-19T02:50:00.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:00 smithi154 ceph-mon[35693]: pgmap v700: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:50:00.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:00 smithi154 ceph-mon[35693]: overall HEALTH_OK 2024-03-19T02:50:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:00 smithi116 ceph-mon[32263]: pgmap v700: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 86 op/s 2024-03-19T02:50:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:00 smithi116 ceph-mon[32263]: overall HEALTH_OK 2024-03-19T02:50:01.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61593 5.79 MB/sec execute 443 sec latency 201.692 ms 2024-03-19T02:50:02.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61705 5.79 MB/sec execute 444 sec latency 187.584 ms 2024-03-19T02:50:02.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:02 smithi162 ceph-mon[35761]: pgmap v701: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.7 MiB/s wr, 149 op/s 2024-03-19T02:50:02.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:02 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:02 smithi154 ceph-mon[35693]: pgmap v701: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.7 MiB/s wr, 149 op/s 2024-03-19T02:50:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:02 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:02 smithi116 ceph-mon[32263]: pgmap v701: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.7 MiB/s wr, 149 op/s 2024-03-19T02:50:02.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:02 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:03.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61807 5.79 MB/sec execute 445 sec latency 205.400 ms 2024-03-19T02:50:04.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 61926 5.79 MB/sec execute 446 sec latency 141.794 ms 2024-03-19T02:50:04.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:04 smithi162 ceph-mon[35761]: pgmap v702: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 111 op/s 2024-03-19T02:50:04.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:04 smithi154 ceph-mon[35693]: pgmap v702: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 111 op/s 2024-03-19T02:50:04.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:04 smithi116 ceph-mon[32263]: pgmap v702: 129 pgs: 129 active+clean; 1.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 111 op/s 2024-03-19T02:50:04.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:50:04.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:50:05.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 -- 172.21.15.116:0/3485472151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 msgr2=0x7f8060107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 --2- 172.21.15.116:0/3485472151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f8060107c50 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f8050009930 tx=0x7f805002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 -- 172.21.15.116:0/3485472151 shutdown_connections 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 --2- 172.21.15.116:0/3485472151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 0x7f806010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 --2- 172.21.15.116:0/3485472151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 --2- 172.21.15.116:0/3485472151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f8060107c50 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 -- 172.21.15.116:0/3485472151 >> 172.21.15.116:0/3485472151 conn(0x7f8060077820 msgr2=0x7f8060077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 -- 172.21.15.116:0/3485472151 shutdown_connections 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.037+0000 7f806627e640 1 -- 172.21.15.116:0/3485472151 wait complete. 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 Processor -- start 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 -- start start 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 0x7f8060198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f806019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8060111a30 con 0x7f8060107870 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80601118b0 con 0x7f80600693e0 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806627e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8060111bb0 con 0x7f8060068a20 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58938/0 (socket says 172.21.15.116:58938) 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 -- 172.21.15.116:0/1268962342 learned_addr learned my addr 172.21.15.116:0/1268962342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805f7fe640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 0x7f8060198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f806481b640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f806019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 -- 172.21.15.116:0/1268962342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 msgr2=0x7f8060198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 0x7f8060198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 -- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 msgr2=0x7f806019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f806019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.038+0000 7f805ffff640 1 -- 172.21.15.116:0/1268962342 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806019cf90 con 0x7f8060068a20 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f805ffff640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8050005940 tx=0x7f8050005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f805d7fa640 1 -- 172.21.15.116:0/1268962342 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050043020 con 0x7f8060068a20 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806019d220 con 0x7f8060068a20 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f805d7fa640 1 -- 172.21.15.116:0/1268962342 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8050004350 con 0x7f8060068a20 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80601a4b10 con 0x7f8060068a20 2024-03-19T02:50:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.039+0000 7f805d7fa640 1 -- 172.21.15.116:0/1268962342 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050003a80 con 0x7f8060068a20 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 Processor -- start 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f805d7fa640 1 -- 172.21.15.116:0/1268962342 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8050046020 con 0x7f8060068a20 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 -- start start 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e312350 0x55c60e31a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e31d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c60e275500 con 0x55c60e311a30 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c60e21dbe0 con 0x55c60e31af30 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f8068e68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c60e2715a0 con 0x55c60e312350 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f806481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f806481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60702/0 (socket says 172.21.15.116:60702) 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f806481b640 1 -- 172.21.15.116:0/3488076870 learned_addr learned my addr 172.21.15.116:0/3488076870 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.042+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e31d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805d7fa640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 0x7f80480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 -- 172.21.15.116:0/3488076870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e312350 msgr2=0x55c60e31a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e312350 0x55c60e31a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 -- 172.21.15.116:0/3488076870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 msgr2=0x55c60e311e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805d7fa640 1 -- 172.21.15.116:0/1268962342 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f80500f08c0 con 0x7f8060068a20 2024-03-19T02:50:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 -- 172.21.15.116:0/3488076870 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60e23f000 con 0x55c60e31af30 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f806481b640 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f806481b640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 0x7f80480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e31d320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8054009bf0 tx=0x7f8054011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=950a7c0af283e09a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.043+0000 7f80467fc640 1 -- 172.21.15.116:0/3488076870 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054004050 con 0x55c60e31af30 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.044+0000 7f80467fc640 1 -- 172.21.15.116:0/3488076870 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80540054d0 con 0x55c60e31af30 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.044+0000 7f80467fc640 1 -- 172.21.15.116:0/3488076870 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054004d30 con 0x55c60e31af30 2024-03-19T02:50:05.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.044+0000 7f806481b640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 0x7f80480a4e30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8058003df0 tx=0x7f8058004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- 172.21.15.116:0/3488076870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 msgr2=0x55c60e31d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e31d320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8054009bf0 tx=0x7f8054011810 comp rx=0 tx=0).stop 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- 172.21.15.116:0/3488076870 shutdown_connections 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e31d320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e312350 0x55c60e31a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 --2- 172.21.15.116:0/3488076870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e311e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- 172.21.15.116:0/3488076870 >> 172.21.15.116:0/3488076870 conn(0x55c60e30c160 msgr2=0x55c60e30c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- 172.21.15.116:0/3488076870 shutdown_connections 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- 172.21.15.116:0/3488076870 wait complete. 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.045+0000 7f8068e68740 1 -- start start 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 Processor -- start 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e317ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e318960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c60e31dd20 con 0x55c60e312350 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c60e231160 con 0x55c60e31af30 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c60e31dea0 con 0x55c60e311a30 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60718/0 (socket says 172.21.15.116:60718) 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 -- 172.21.15.116:0/3621049104 learned_addr learned my addr 172.21.15.116:0/3621049104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805ffff640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e317ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f8068e68740 10 client.0 _ll_register_callbacks cb 0x55c60e26da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f806481b640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e318960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 -- 172.21.15.116:0/3621049104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 msgr2=0x55c60e317ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e317ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 -- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 msgr2=0x55c60e318960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e318960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.046+0000 7f805f7fe640 1 -- 172.21.15.116:0/3621049104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60e0537d0 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f805f7fe640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8054009fd0 tx=0x7f8054004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce8ab044ae30c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054029070 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80540054d0 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f8054003a00 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8054035050 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 10 client.21777 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c60df893f0 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80540054d0 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f8054005060 con 0x55c60e312350 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 10 client.21777 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 10 client.21777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.047+0000 7f8068e68740 20 client.21777 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e32b5f0 0x55c60e32dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f8068e68740 4 client.21777 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c60e282a40 con 0x55c60e32b5f0 2024-03-19T02:50:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f805ffff640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e32b5f0 0x55c60e32dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f805ffff640 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e32b5f0 0x55c60e32dab0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9020f600876bbd07 server_cookie=e695049b6872a61b in_seq=0 out_seq=0 2024-03-19T02:50:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.048+0000 7f803f7fe640 10 client.21777 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f803f7fe640 1 -- 172.21.15.116:0/3621049104 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c60e282a40 con 0x55c60e32b5f0 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f803f7fe640 10 client.21777 handle_command_reply: tid=0 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 client.21777 shutdown 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 msgr2=0x55c60e318420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8054009fd0 tx=0x7f8054004940 comp rx=0 tx=0).stop 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 shutdown_connections 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e32b5f0 0x55c60e32dab0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e31af30 0x55c60e318960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e312350 0x55c60e318420 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 --2- 172.21.15.116:0/3621049104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e317ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 >> 172.21.15.116:0/3621049104 conn(0x55c60e30c160 msgr2=0x55c60e30deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f803f7fe640 0 client.21777 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 shutdown_connections 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 1 -- 172.21.15.116:0/3621049104 wait complete. 2024-03-19T02:50:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.051+0000 7f8068e68740 20 client.21777 trim_cache size 0 max 16384 2024-03-19T02:50:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 Processor -- start 2024-03-19T02:50:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 -- start start 2024-03-19T02:50:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e316e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e31af30 0x55c60e313210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f804c00ff60 con 0x55c60e311a30 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f804c011ad0 con 0x55c60e312350 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.074+0000 7f8068e68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f804c010630 con 0x55c60e31af30 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60720/0 (socket says 172.21.15.116:60720) 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 -- 172.21.15.116:0/4009596362 learned_addr learned my addr 172.21.15.116:0/4009596362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805f7fe640 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e316e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f806481b640 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e31af30 0x55c60e313210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 -- 172.21.15.116:0/4009596362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e31af30 msgr2=0x55c60e313210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e31af30 0x55c60e313210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 -- 172.21.15.116:0/4009596362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 msgr2=0x55c60e316e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e316e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 -- 172.21.15.116:0/4009596362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60df490f0 con 0x55c60e311a30 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f805ffff640 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f80500bcb90 tx=0x7f8050000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c6477837e5125c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f8045ffb640 1 -- 172.21.15.116:0/4009596362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500d4030 con 0x55c60e311a30 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f8045ffb640 1 -- 172.21.15.116:0/4009596362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80500c0da0 con 0x55c60e311a30 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.075+0000 7f8045ffb640 1 -- 172.21.15.116:0/4009596362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500b92c0 con 0x55c60e311a30 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- 172.21.15.116:0/4009596362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 msgr2=0x55c60e31a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f80500bcb90 tx=0x7f8050000bc0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- 172.21.15.116:0/4009596362 shutdown_connections 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e31af30 0x55c60e313210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e316e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 --2- 172.21.15.116:0/4009596362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e311a30 0x55c60e31a610 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- 172.21.15.116:0/4009596362 >> 172.21.15.116:0/4009596362 conn(0x55c60e30c160 msgr2=0x55c60e30b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- 172.21.15.116:0/4009596362 shutdown_connections 2024-03-19T02:50:05.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- 172.21.15.116:0/4009596362 wait complete. 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 -- start start 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.076+0000 7f8068e68740 1 Processor -- start 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e3335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e330990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f804c011400 con 0x55c60e31af30 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f804c010d00 con 0x55c60e312350 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f804c00df60 con 0x55c60e311a30 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f8068e68740 10 client.0 _ll_register_callbacks cb 0x7f8058009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f806481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e3335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f806481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60722/0 (socket says 172.21.15.116:60722) 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f806481b640 1 -- 172.21.15.116:0/2839665499 learned_addr learned my addr 172.21.15.116:0/2839665499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805f7fe640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e330990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 -- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 msgr2=0x55c60e330990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e330990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 -- 172.21.15.116:0/2839665499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 msgr2=0x55c60e330fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 -- 172.21.15.116:0/2839665499 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60e3317a0 con 0x55c60e311a30 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f806481b640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f805ffff640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e3335b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8050005c40 tx=0x7f80500edd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3a3699e6c6f3f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f803f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.077+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500d4070 con 0x55c60e311a30 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80500c0da0 con 0x55c60e311a30 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9510+0+0 (secure 0 0 0) 0x7f80500c4f60 con 0x55c60e311a30 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f80500c53f0 con 0x55c60e311a30 2024-03-19T02:50:05.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 10 client.31477 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c60e391980 con 0x55c60e311a30 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80500c0da0 con 0x55c60e311a30 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f80500b8ac0 con 0x55c60e311a30 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 10 client.31477 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 10 client.31477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 20 client.31477 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e391fe0 0x55c60e3943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 4 client.31477 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c60e27cd90 con 0x55c60e391fe0 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.078+0000 7f805f7fe640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e391fe0 0x55c60e3943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.079+0000 7f803f7fe640 10 client.31477 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:05.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.079+0000 7f805f7fe640 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e391fe0 0x55c60e3943d0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b3bbba9704c5915 server_cookie=b0bebc59b18700db in_seq=0 out_seq=0 2024-03-19T02:50:05.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f803f7fe640 1 -- 172.21.15.116:0/2839665499 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c60e27cd90 con 0x55c60e391fe0 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f803f7fe640 10 client.31477 handle_command_reply: tid=0 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 client.31477 shutdown 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 msgr2=0x55c60e3335b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e3335b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8050005c40 tx=0x7f80500edd10 comp rx=0 tx=0).stop 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 shutdown_connections 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c60e391fe0 0x55c60e3943d0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c60e31af30 0x55c60e330fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c60e312350 0x55c60e330990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 --2- 172.21.15.116:0/2839665499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c60e311a30 0x55c60e3335b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 >> 172.21.15.116:0/2839665499 conn(0x55c60e30c160 msgr2=0x55c60e30a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f803f7fe640 0 client.31477 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 shutdown_connections 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 1 -- 172.21.15.116:0/2839665499 wait complete. 2024-03-19T02:50:05.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.080+0000 7f8068e68740 20 client.31477 trim_cache size 0 max 16384 2024-03-19T02:50:05.088 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:50:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:50:05.088 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 msgr2=0x7f80480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 0x7f80480a4e30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8058003df0 tx=0x7f8058004a40 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 msgr2=0x7f8060197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8050005940 tx=0x7f8050005dc0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 shutdown_connections 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f80480a2970 0x7f80480a4e30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8060107870 0x7f806019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600693e0 0x7f8060198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 --2- 172.21.15.116:0/1268962342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8060068a20 0x7f8060197ff0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 >> 172.21.15.116:0/1268962342 conn(0x7f8060077820 msgr2=0x7f80600fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 shutdown_connections 2024-03-19T02:50:05.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.081+0000 7f806627e640 1 -- 172.21.15.116:0/1268962342 wait complete. 2024-03-19T02:50:05.097 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:50:05.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 -- 172.21.15.116:0/3363930063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c10e080 msgr2=0x7fe78c10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 --2- 172.21.15.116:0/3363930063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c10e080 0x7fe78c10e500 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009ca0 tx=0x7fe77c02c100 comp rx=0 tx=0).stop 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 -- 172.21.15.116:0/3363930063 shutdown_connections 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 --2- 172.21.15.116:0/3363930063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c11b000 0x7fe78c11d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 --2- 172.21.15.116:0/3363930063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c10e080 0x7fe78c10e500 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 --2- 172.21.15.116:0/3363930063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 0x7fe78c10db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.284+0000 7fe793897640 1 -- 172.21.15.116:0/3363930063 >> 172.21.15.116:0/3363930063 conn(0x7fe78c06c910 msgr2=0x7fe78c06cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.287+0000 7fe793897640 1 -- 172.21.15.116:0/3363930063 shutdown_connections 2024-03-19T02:50:05.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.287+0000 7fe793897640 1 -- 172.21.15.116:0/3363930063 wait complete. 2024-03-19T02:50:05.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.287+0000 7fe793897640 1 Processor -- start 2024-03-19T02:50:05.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 -- start start 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c10e080 0x7fe78c1a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 0x7fe78c1a7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c120880 con 0x7fe78c110050 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c120700 con 0x7fe78c10e080 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe793897640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c120a00 con 0x7fe78c11b000 2024-03-19T02:50:05.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe791e34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe791e34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58996/0 (socket says 172.21.15.116:58996) 2024-03-19T02:50:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe791e34640 1 -- 172.21.15.116:0/3828995301 learned_addr learned my addr 172.21.15.116:0/3828995301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe790e32640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 0x7fe78c1a7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.288+0000 7fe791633640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c10e080 0x7fe78c1a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.290+0000 7fe791e34640 1 -- 172.21.15.116:0/3828995301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c10e080 msgr2=0x7fe78c1a6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.290+0000 7fe791e34640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c10e080 0x7fe78c1a6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.290+0000 7fe791e34640 1 -- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 msgr2=0x7fe78c1a7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.290+0000 7fe791e34640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 0x7fe78c1a7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.290+0000 7fe791e34640 1 -- 172.21.15.116:0/3828995301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c1abdf0 con 0x7fe78c11b000 2024-03-19T02:50:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.291+0000 7fe791e34640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.293+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784007bb0 con 0x7fe78c11b000 2024-03-19T02:50:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.294+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78c1ac0e0 con 0x7fe78c11b000 2024-03-19T02:50:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.294+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe78c1b3970 con 0x7fe78c11b000 2024-03-19T02:50:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.294+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe784007d50 con 0x7fe78c11b000 2024-03-19T02:50:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.295+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784018730 con 0x7fe78c11b000 2024-03-19T02:50:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.297+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe78400f450 con 0x7fe78c11b000 2024-03-19T02:50:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.297+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe778005290 con 0x7fe78c11b000 2024-03-19T02:50:05.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.298+0000 7fe78a7fc640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 0x7fe7740a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.298+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7fe7840cf250 con 0x7fe78c11b000 2024-03-19T02:50:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.299+0000 7fe791633640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 0x7fe7740a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.301+0000 7fe791633640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 0x7fe7740a4f10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe7800058c0 tx=0x7fe78000a4c0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.313+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe784014030 con 0x7fe78c11b000 2024-03-19T02:50:05.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62062 5.80 MB/sec execute 447 sec latency 224.342 ms 2024-03-19T02:50:05.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.601+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe778005850 con 0x7fe78c11b000 2024-03-19T02:50:05.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.602+0000 7fe78a7fc640 1 -- 172.21.15.116:0/3828995301 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11951 (secure 0 0 0) 0x7fe78409bb20 con 0x7fe78c11b000 2024-03-19T02:50:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:50:05.603 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:50:00.839067+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:50:05.604 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 msgr2=0x7fe7740a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 0x7fe7740a4f10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe7800058c0 tx=0x7fe78000a4c0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 msgr2=0x7fe78c1ab6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).stop 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 shutdown_connections 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe7740a2a50 0x7fe7740a4f10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe78c11b000 0x7fe78c1ab6c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe78c110050 0x7fe78c1a7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 --2- 172.21.15.116:0/3828995301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe78c10e080 0x7fe78c1a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 >> 172.21.15.116:0/3828995301 conn(0x7fe78c06c910 msgr2=0x7fe78c11bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 shutdown_connections 2024-03-19T02:50:05.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.605+0000 7fe793897640 1 -- 172.21.15.116:0/3828995301 wait complete. 2024-03-19T02:50:05.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:50:05.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 -- 172.21.15.116:0/2593464926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 msgr2=0x7f452410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 --2- 172.21.15.116:0/2593464926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f452410e220 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f451c009ca0 tx=0x7f451c02c180 comp rx=0 tx=0).stop 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 -- 172.21.15.116:0/2593464926 shutdown_connections 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 --2- 172.21.15.116:0/2593464926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f452410e220 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 --2- 172.21.15.116:0/2593464926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524103360 0x7f452410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 --2- 172.21.15.116:0/2593464926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524102a40 0x7f4524102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 -- 172.21.15.116:0/2593464926 >> 172.21.15.116:0/2593464926 conn(0x7f45240778d0 msgr2=0x7f4524077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 -- 172.21.15.116:0/2593464926 shutdown_connections 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.748+0000 7f452c0f1640 1 -- 172.21.15.116:0/2593464926 wait complete. 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 Processor -- start 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 -- start start 2024-03-19T02:50:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524102a40 0x7f4524100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 0x7f4524100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111a30 con 0x7f452410bd80 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45241118b0 con 0x7f4524103360 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452c0f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111bb0 con 0x7f4524102a40 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f4529e8d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524102a40 0x7f4524100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60760/0 (socket says 172.21.15.116:60760) 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452968c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 0x7f4524100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 -- 172.21.15.116:0/978597099 learned_addr learned my addr 172.21.15.116:0/978597099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 -- 172.21.15.116:0/978597099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524102a40 msgr2=0x7f4524100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524102a40 0x7f4524100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 -- 172.21.15.116:0/978597099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 msgr2=0x7f4524100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 0x7f4524100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 -- 172.21.15.116:0/978597099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45241a63f0 con 0x7f452410bd80 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452968c640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 0x7f4524100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.749+0000 7f452a68e640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f451c02c790 tx=0x7f451c035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.750+0000 7f451affd640 1 -- 172.21.15.116:0/978597099 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c039040 con 0x7f452410bd80 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.750+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241a6680 con 0x7f452410bd80 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.750+0000 7f451affd640 1 -- 172.21.15.116:0/978597099 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451c044070 con 0x7f452410bd80 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.750+0000 7f451affd640 1 -- 172.21.15.116:0/978597099 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c032850 con 0x7f452410bd80 2024-03-19T02:50:05.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.750+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a6b70 con 0x7f452410bd80 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.752+0000 7f452edbb740 1 Processor -- start 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.752+0000 7f451affd640 1 -- 172.21.15.116:0/978597099 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f451c0329f0 con 0x7f452410bd80 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 -- start start 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caead3bf0 0x558caeadc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caeadc7d0 0x558caeadebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558caea364b0 con 0x558caead32d0 2024-03-19T02:50:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558cae9ded70 con 0x558caeadc7d0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452edbb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558caea32550 con 0x558caead3bf0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60766/0 (socket says 172.21.15.116:60766) 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 -- 172.21.15.116:0/613699747 learned_addr learned my addr 172.21.15.116:0/613699747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f4529e8d640 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caead3bf0 0x558caeadc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caeadc7d0 0x558caeadebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 -- 172.21.15.116:0/613699747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caead3bf0 msgr2=0x558caeadc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caead3bf0 0x558caeadc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 -- 172.21.15.116:0/613699747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caeadc7d0 msgr2=0x558caeadebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caeadc7d0 0x558caeadebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.753+0000 7f452968c640 1 -- 172.21.15.116:0/613699747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558caea01210 con 0x558caead32d0 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f451affd640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 0x7f45100a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f451affd640 1 -- 172.21.15.116:0/978597099 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f451c0f15a0 con 0x7f452410bd80 2024-03-19T02:50:05.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f452968c640 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f45140139a0 tx=0x7f4514009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d31e245b735fc4e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f4529e8d640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 0x7f45100a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f450ffff640 1 -- 172.21.15.116:0/613699747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514004030 con 0x558caead32d0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f450ffff640 1 -- 172.21.15.116:0/613699747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451400f8c0 con 0x558caead32d0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.754+0000 7f450ffff640 1 -- 172.21.15.116:0/613699747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45140038e0 con 0x558caead32d0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 -- 172.21.15.116:0/613699747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 msgr2=0x558caead36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f45140139a0 tx=0x7f4514009d80 comp rx=0 tx=0).stop 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f4529e8d640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 0x7f45100a4eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x558caead2da0 tx=0x7f4520005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 -- 172.21.15.116:0/613699747 shutdown_connections 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caeadc7d0 0x558caeadebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caead3bf0 0x558caeadc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 --2- 172.21.15.116:0/613699747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead36b0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 -- 172.21.15.116:0/613699747 >> 172.21.15.116:0/613699747 conn(0x558caeacda00 msgr2=0x558caeacdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.755+0000 7f452edbb740 1 -- 172.21.15.116:0/613699747 shutdown_connections 2024-03-19T02:50:05.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.756+0000 7f452edbb740 1 -- 172.21.15.116:0/613699747 wait complete. 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.756+0000 7f452edbb740 1 -- start start 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.756+0000 7f452edbb740 1 Processor -- start 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.756+0000 7f452edbb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caead9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeada230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558caeadf5c0 con 0x558caead32d0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558cae9f2310 con 0x558caead3bf0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558caeadf740 con 0x558caeadc7d0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452edbb740 10 client.0 _ll_register_callbacks cb 0x558cae9e62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f4529e8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeada230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f4529e8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60778/0 (socket says 172.21.15.116:60778) 2024-03-19T02:50:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f4529e8d640 1 -- 172.21.15.116:0/3284175626 learned_addr learned my addr 172.21.15.116:0/3284175626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 -- 172.21.15.116:0/3284175626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 msgr2=0x558caead9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caead9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 -- 172.21.15.116:0/3284175626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 msgr2=0x558caead97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.757+0000 7f452a68e640 1 -- 172.21.15.116:0/3284175626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cae74a3f0 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f452a68e640 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeada230 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f451c0ee970 tx=0x7f451c0bd3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1206ed4c416f11c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f450d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c0bdf80 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f451c0f11b0 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f451400f8c0 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f451c0bd880 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f452edbb740 10 client.31495 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.758+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558cae8147d0 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451400f8c0 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f451c03c030 con 0x558caeadc7d0 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 10 client.31495 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 10 client.31495 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 20 client.31495 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeaecad0 0x558caeaeef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 4 client.31495 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558caea943e0 con 0x558caeaecad0 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.759+0000 7f4529e8d640 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeaecad0 0x558caeaeef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.760+0000 7f4529e8d640 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeaecad0 0x558caeaeef70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9c2bfd252743c914 server_cookie=a302af82fe997944 in_seq=0 out_seq=0 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.760+0000 7f450d7fa640 10 client.31495 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.760+0000 7f450d7fa640 1 -- 172.21.15.116:0/3284175626 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558caea943e0 con 0x558caeaecad0 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f450d7fa640 10 client.31495 handle_command_reply: tid=0 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 client.31495 shutdown 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 msgr2=0x558caeada230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeada230 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f451c0ee970 tx=0x7f451c0bd3c0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 shutdown_connections 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeaecad0 0x558caeaeef70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeada230 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f450d7fa640 0 client.31495 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caead9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 --2- 172.21.15.116:0/3284175626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caead97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 >> 172.21.15.116:0/3284175626 conn(0x558caeacda00 msgr2=0x558caeacf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 shutdown_connections 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 1 -- 172.21.15.116:0/3284175626 wait complete. 2024-03-19T02:50:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.761+0000 7f452edbb740 20 client.31495 trim_cache size 0 max 16384 2024-03-19T02:50:05.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 Processor -- start 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 -- start start 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead32d0 0x558caeadbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b0f80 con 0x558caead3bf0 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b2af0 con 0x558caead32d0 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452edbb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b1650 con 0x558caeadc7d0 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452968c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452a68e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452968c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60794/0 (socket says 172.21.15.116:60794) 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452a68e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59056/0 (socket says 172.21.15.116:59056) 2024-03-19T02:50:05.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f452968c640 1 -- 172.21.15.116:0/2016744390 learned_addr learned my addr 172.21.15.116:0/2016744390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.781+0000 7f4529e8d640 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead32d0 0x558caeadbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 -- 172.21.15.116:0/2016744390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead32d0 msgr2=0x558caeadbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead32d0 0x558caeadbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 -- 172.21.15.116:0/2016744390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 msgr2=0x558caeac8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 -- 172.21.15.116:0/2016744390 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558caead8760 con 0x558caeadc7d0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452968c640 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f452a68e640 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f451c0b9ce0 tx=0x7f451c0bd4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f81e3acf346d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f450f7fe640 1 -- 172.21.15.116:0/2016744390 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c0bdf80 con 0x558caeadc7d0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f450f7fe640 1 -- 172.21.15.116:0/2016744390 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451c0eeae0 con 0x558caeadc7d0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.782+0000 7f450f7fe640 1 -- 172.21.15.116:0/2016744390 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c0f11b0 con 0x558caeadc7d0 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- 172.21.15.116:0/2016744390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 msgr2=0x558caeae8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f451c0b9ce0 tx=0x7f451c0bd4f0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- 172.21.15.116:0/2016744390 shutdown_connections 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae8390 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead3bf0 0x558caeac8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- 172.21.15.116:0/2016744390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead32d0 0x558caeadbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- 172.21.15.116:0/2016744390 >> 172.21.15.116:0/2016744390 conn(0x558caeacda00 msgr2=0x558caeaccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- 172.21.15.116:0/2016744390 shutdown_connections 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- 172.21.15.116:0/2016744390 wait complete. 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- start start 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 Processor -- start 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caeae1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b2420 con 0x558caead32d0 2024-03-19T02:50:05.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45241b1d20 con 0x558caead3bf0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.783+0000 7f452edbb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f45241aef80 con 0x558caeadc7d0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452edbb740 10 client.0 _ll_register_callbacks cb 0x7f451c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f4529e8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452968c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caeae1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f4529e8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60802/0 (socket says 172.21.15.116:60802) 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f4529e8d640 1 -- 172.21.15.116:0/2843611481 learned_addr learned my addr 172.21.15.116:0/2843611481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 -- 172.21.15.116:0/2843611481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 msgr2=0x558caeae1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caeae1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 -- 172.21.15.116:0/2843611481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 msgr2=0x558caeae49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 -- 172.21.15.116:0/2843611481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558caeae2410 con 0x558caeadc7d0 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f4529e8d640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452a68e640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae1ed0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f451c0bd580 tx=0x7f451c0f1460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f41ce26ad83a0077 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f450d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c0bd1e0 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f451c0eeae0 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.784+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f451c0c5b90 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f451c0bdf80 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 10 client.31501 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x558caeae2410 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451c0eeae0 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f451c0b9040 con 0x558caeadc7d0 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 10 client.31501 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 10 client.31501 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 20 client.31501 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeb53db0 0x558caeb561a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 4 client.31501 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558caea439f0 con 0x558caeb53db0 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.785+0000 7f4529e8d640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeb53db0 0x558caeb561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f450d7fa640 10 client.31501 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f4529e8d640 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeb53db0 0x558caeb561a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0ff0bfaacdd68a7 server_cookie=4fe26b09743a25c9 in_seq=0 out_seq=0 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f450d7fa640 1 -- 172.21.15.116:0/2843611481 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558caea439f0 con 0x558caeb53db0 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f450d7fa640 10 client.31501 handle_command_reply: tid=0 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 client.31501 shutdown 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 msgr2=0x558caeae1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae1ed0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f451c0bd580 tx=0x7f451c0f1460 comp rx=0 tx=0).stop 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 shutdown_connections 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558caeb53db0 0x558caeb561a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558caeadc7d0 0x558caeae1ed0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558caead3bf0 0x558caeae1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f450d7fa640 0 client.31501 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 --2- 172.21.15.116:0/2843611481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558caead32d0 0x558caeae49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 >> 172.21.15.116:0/2843611481 conn(0x558caeacda00 msgr2=0x558caeaced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 shutdown_connections 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 1 -- 172.21.15.116:0/2843611481 wait complete. 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.786+0000 7f452edbb740 20 client.31501 trim_cache size 0 max 16384 2024-03-19T02:50:05.794 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:05.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 msgr2=0x7f45100a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 0x7f45100a4eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x558caead2da0 tx=0x7f4520005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:05.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 msgr2=0x7f4524101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f451c02c790 tx=0x7f451c035db0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 shutdown_connections 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f45100a29f0 0x7f45100a4eb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f452410bd80 0x7f4524101110 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4524103360 0x7f4524100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 --2- 172.21.15.116:0/978597099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524102a40 0x7f4524100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 >> 172.21.15.116:0/978597099 conn(0x7f45240778d0 msgr2=0x7f45240fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 shutdown_connections 2024-03-19T02:50:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.787+0000 7f452c0f1640 1 -- 172.21.15.116:0/978597099 wait complete. 2024-03-19T02:50:05.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- 172.21.15.116:0/3231147449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241066b0 msgr2=0x7f9724106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3231147449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241066b0 0x7f9724106a90 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9714009930 tx=0x7f971402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- 172.21.15.116:0/3231147449 shutdown_connections 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3231147449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9724101060 0x7f972410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3231147449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97241006a0 0x7f9724100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3231147449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241066b0 0x7f9724106a90 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- 172.21.15.116:0/3231147449 >> 172.21.15.116:0/3231147449 conn(0x7f97240778d0 msgr2=0x7f9724077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- 172.21.15.116:0/3231147449 shutdown_connections 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- 172.21.15.116:0/3231147449 wait complete. 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 Processor -- start 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.931+0000 7f9729cc0640 1 -- start start 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9724101060 0x7f9724199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97241066b0 0x7f97241a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9724110930 con 0x7f97241006a0 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97241107b0 con 0x7f97241066b0 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9729cc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9724110ab0 con 0x7f9724101060 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60830/0 (socket says 172.21.15.116:60830) 2024-03-19T02:50:05.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 -- 172.21.15.116:0/3661999662 learned_addr learned my addr 172.21.15.116:0/3661999662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9722ffd640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9724101060 0x7f9724199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f9723fff640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97241066b0 0x7f97241a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 -- 172.21.15.116:0/3661999662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9724101060 msgr2=0x7f9724199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9724101060 0x7f9724199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 -- 172.21.15.116:0/3661999662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97241066b0 msgr2=0x7f97241a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97241066b0 0x7f97241a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 -- 172.21.15.116:0/3661999662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97241a4390 con 0x7f97241006a0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.932+0000 7f97237fe640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9714009230 tx=0x7f97140045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.933+0000 7f9720ff9640 1 -- 172.21.15.116:0/3661999662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714043020 con 0x7f97241006a0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.933+0000 7f9720ff9640 1 -- 172.21.15.116:0/3661999662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971402cb70 con 0x7f97241006a0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.933+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97241a4560 con 0x7f97241006a0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.933+0000 7f9720ff9640 1 -- 172.21.15.116:0/3661999662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714003b50 con 0x7f97241006a0 2024-03-19T02:50:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.933+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97241a4a50 con 0x7f97241006a0 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f9720ff9640 1 -- 172.21.15.116:0/3661999662 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9714046020 con 0x7f97241006a0 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 Processor -- start 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 -- start start 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 0x5555908a4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5555908074b0 con 0x5555908a46f0 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555907afd70 con 0x5555908a3d40 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f972c987740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555590803550 con 0x5555908b5f00 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60838/0 (socket says 172.21.15.116:60838) 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f97237fe640 1 -- 172.21.15.116:0/1464040370 learned_addr learned my addr 172.21.15.116:0/1464040370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f9720ff9640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 0x7f970c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f9723fff640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 0x5555908a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.936+0000 7f9722ffd640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9723fff640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 0x7f970c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 -- 172.21.15.116:0/1464040370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 msgr2=0x5555908a4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 0x5555908a4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 -- 172.21.15.116:0/1464040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 msgr2=0x5555908a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 -- 172.21.15.116:0/1464040370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555907d2210 con 0x5555908b5f00 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9720ff9640 1 -- 172.21.15.116:0/3661999662 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f97140f1310 con 0x7f97241006a0 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9723fff640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 0x5555908a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f97237fe640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9722ffd640 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b82f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9718005d00 tx=0x7f97180107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=627d7b2c2e4ecbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9709ffb640 1 -- 172.21.15.116:0/1464040370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971801c480 con 0x5555908b5f00 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9709ffb640 1 -- 172.21.15.116:0/1464040370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971801cad0 con 0x5555908b5f00 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.937+0000 7f9709ffb640 1 -- 172.21.15.116:0/1464040370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718011ab0 con 0x5555908b5f00 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 -- 172.21.15.116:0/1464040370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 msgr2=0x5555908b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b82f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9718005d00 tx=0x7f97180107a0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f9723fff640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 0x7f970c0a4e30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f971c005fd0 tx=0x7f971c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 -- 172.21.15.116:0/1464040370 shutdown_connections 2024-03-19T02:50:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b82f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a46f0 0x5555908a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 --2- 172.21.15.116:0/1464040370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a3d40 0x5555908a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 -- 172.21.15.116:0/1464040370 >> 172.21.15.116:0/1464040370 conn(0x55559089ea00 msgr2=0x55559089ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 -- 172.21.15.116:0/1464040370 shutdown_connections 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.938+0000 7f972c987740 1 -- 172.21.15.116:0/1464040370 wait complete. 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 -- start start 2024-03-19T02:50:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 Processor -- start 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908aff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60850/0 (socket says 172.21.15.116:60850) 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f9723fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f9722ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f9723fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59080/0 (socket says 172.21.15.116:59080) 2024-03-19T02:50:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f97237fe640 1 -- 172.21.15.116:0/2461673021 learned_addr learned my addr 172.21.15.116:0/2461673021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5555908a9a30 con 0x5555908a3d40 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555907c3310 con 0x5555908a46f0 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.939+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5555908a9bb0 con 0x5555908b5f00 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 -- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 msgr2=0x5555908b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f972c987740 10 client.0 _ll_register_callbacks cb 0x5555907b72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 -- 172.21.15.116:0/2461673021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 msgr2=0x5555908aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559051b3f0 con 0x5555908a3d40 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9723fff640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971801cad0 con 0x5555908a3d40 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f97237fe640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f97140bd5e0 tx=0x7f97140b9240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec8a06350151ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9702ffd640 10 client.21816 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9702ffd640 1 -- 172.21.15.116:0/2461673021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140bd420 con 0x5555908a3d40 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9702ffd640 1 -- 172.21.15.116:0/2461673021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f97140b9cc0 con 0x5555908a3d40 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.940+0000 7f9702ffd640 1 -- 172.21.15.116:0/2461673021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f97140f1170 con 0x5555908a3d40 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 10 client.21816 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=41}) -- 0x5555905e57d0 con 0x5555908a3d40 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f9702ffd640 1 -- 172.21.15.116:0/2461673021 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f9714005500 con 0x5555908a3d40 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 10 client.21816 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 10 client.21816 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 20 client.21816 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555908d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 4 client.21816 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555908653e0 con 0x5555908cea30 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.941+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555908d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.942+0000 7f9702ffd640 10 client.21816 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.942+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555908d0e40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=53c99930e1abb7b5 server_cookie=81257422ff43cb22 in_seq=0 out_seq=0 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f9702ffd640 1 -- 172.21.15.116:0/2461673021 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555908653e0 con 0x5555908cea30 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f9702ffd640 10 client.21816 handle_command_reply: tid=0 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 client.21816 shutdown 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 msgr2=0x5555908afa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f97140bd5e0 tx=0x7f97140b9240 comp rx=0 tx=0).stop 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 shutdown_connections 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555908d0e40 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 --2- 172.21.15.116:0/2461673021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908afa00 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f9702ffd640 0 client.21816 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 >> 172.21.15.116:0/2461673021 conn(0x55559089ea00 msgr2=0x5555908a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 shutdown_connections 2024-03-19T02:50:05.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 1 -- 172.21.15.116:0/2461673021 wait complete. 2024-03-19T02:50:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.943+0000 7f972c987740 20 client.21816 trim_cache size 0 max 16384 2024-03-19T02:50:05.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 Processor -- start 2024-03-19T02:50:05.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 -- start start 2024-03-19T02:50:05.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908a46f0 0x5555908a0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908b5f00 0x5555908d6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97241acb20 con 0x5555908a3d40 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ae690 con 0x5555908b5f00 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f972c987740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ad1f0 con 0x5555908a46f0 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60856/0 (socket says 172.21.15.116:60856) 2024-03-19T02:50:05.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f9722ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908a46f0 0x5555908a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f97237fe640 1 -- 172.21.15.116:0/4042183862 learned_addr learned my addr 172.21.15.116:0/4042183862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.958+0000 7f97237fe640 1 -- 172.21.15.116:0/4042183862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908a46f0 msgr2=0x5555908a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97237fe640 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908a46f0 0x5555908a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97237fe640 1 -- 172.21.15.116:0/4042183862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908b5f00 msgr2=0x5555908d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97237fe640 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908b5f00 0x5555908d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97237fe640 1 -- 172.21.15.116:0/4042183862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55559089fc30 con 0x5555908a3d40 2024-03-19T02:50:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97237fe640 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f97140b9330 tx=0x7f9714040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9643fbde3d0c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97097fa640 1 -- 172.21.15.116:0/4042183862 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140bcfb0 con 0x5555908a3d40 2024-03-19T02:50:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97097fa640 1 -- 172.21.15.116:0/4042183862 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971801cad0 con 0x5555908a3d40 2024-03-19T02:50:05.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.959+0000 7f97097fa640 1 -- 172.21.15.116:0/4042183862 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140bd190 con 0x5555908a3d40 2024-03-19T02:50:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- 172.21.15.116:0/4042183862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 msgr2=0x555590899a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f97140b9330 tx=0x7f9714040000 comp rx=0 tx=0).stop 2024-03-19T02:50:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- 172.21.15.116:0/4042183862 shutdown_connections 2024-03-19T02:50:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908b5f00 0x5555908d6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908a46f0 0x5555908a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 --2- 172.21.15.116:0/4042183862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x555590899a30 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- 172.21.15.116:0/4042183862 >> 172.21.15.116:0/4042183862 conn(0x55559089ea00 msgr2=0x55559089fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- 172.21.15.116:0/4042183862 shutdown_connections 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- 172.21.15.116:0/4042183862 wait complete. 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.960+0000 7f972c987740 1 -- start start 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 Processor -- start 2024-03-19T02:50:05.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908e1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97241adfc0 con 0x5555908a3d40 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97241ad8c0 con 0x5555908a46f0 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97241aab20 con 0x5555908b5f00 2024-03-19T02:50:05.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f972c987740 10 client.0 _ll_register_callbacks cb 0x7f97140035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f97237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60862/0 (socket says 172.21.15.116:60862) 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f97237fe640 1 -- 172.21.15.116:0/2472307222 learned_addr learned my addr 172.21.15.116:0/2472307222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.961+0000 7f9723fff640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908e1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 -- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 msgr2=0x5555908e1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908e1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 -- 172.21.15.116:0/2472307222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 msgr2=0x5555908e04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908e04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 -- 172.21.15.116:0/2472307222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555908aa130 con 0x5555908a3d40 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908e04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f97237fe640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f971402cdd0 tx=0x7f9714005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7689bff6c1495dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:05.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140bd420 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f97140d6490 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971801cad0 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f97140c17a0 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f972c987740 10 client.21822 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5555908aa130 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.962+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971801cad0 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f97140a6b90 con 0x5555908a3d40 2024-03-19T02:50:05.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 10 client.21822 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 10 client.21822 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 20 client.21822 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555909373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 4 client.21822 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555908149f0 con 0x5555908cea30 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555909373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:05.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f9702ffd640 10 client.21822 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.963+0000 7f9722ffd640 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555909373c0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a92415ac5b88440d server_cookie=bbe43bab9a8511a4 in_seq=0 out_seq=0 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f9702ffd640 1 -- 172.21.15.116:0/2472307222 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555908149f0 con 0x5555908cea30 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f9702ffd640 10 client.21822 handle_command_reply: tid=0 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 client.21822 shutdown 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 msgr2=0x5555908dffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f971402cdd0 tx=0x7f9714005730 comp rx=0 tx=0).stop 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 shutdown_connections 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5555908b5f00 0x5555908e1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555908a46f0 0x5555908e04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x5555908cea30 0x5555909373c0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 --2- 172.21.15.116:0/2472307222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555908a3d40 0x5555908dffb0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 >> 172.21.15.116:0/2472307222 conn(0x55559089ea00 msgr2=0x5555908a06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f9702ffd640 0 client.21822 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 shutdown_connections 2024-03-19T02:50:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 1 -- 172.21.15.116:0/2472307222 wait complete. 2024-03-19T02:50:05.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.964+0000 7f972c987740 20 client.21822 trim_cache size 0 max 16384 2024-03-19T02:50:05.980 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 msgr2=0x7f970c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 0x7f970c0a4e30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f971c005fd0 tx=0x7f971c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 msgr2=0x7f9724198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9714009230 tx=0x7f97140045e0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 shutdown_connections 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f970c0a2970 0x7f970c0a4e30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97241066b0 0x7f97241a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9724101060 0x7f9724199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 --2- 172.21.15.116:0/3661999662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97241006a0 0x7f9724198dd0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 >> 172.21.15.116:0/3661999662 conn(0x7f97240778d0 msgr2=0x7f9724103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 shutdown_connections 2024-03-19T02:50:05.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:05.965+0000 7f9729cc0640 1 -- 172.21.15.116:0/3661999662 wait complete. 2024-03-19T02:50:05.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:50:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.106+0000 7f4b3bea1640 1 -- 172.21.15.116:0/453561662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b341066b0 msgr2=0x7f4b34106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.106+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/453561662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b341066b0 0x7f4b34106a90 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009930 tx=0x7f4b3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.106+0000 7f4b3bea1640 1 -- 172.21.15.116:0/453561662 shutdown_connections 2024-03-19T02:50:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.106+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/453561662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34101060 0x7f4b3410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/453561662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b34100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/453561662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b341066b0 0x7f4b34106a90 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- 172.21.15.116:0/453561662 >> 172.21.15.116:0/453561662 conn(0x7f4b340778d0 msgr2=0x7f4b34077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- 172.21.15.116:0/453561662 shutdown_connections 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- 172.21.15.116:0/453561662 wait complete. 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 Processor -- start 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- start start 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b340fd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b341066b0 0x7f4b340fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34110980 con 0x7f4b34101060 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34110800 con 0x7f4b341066b0 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.107+0000 7f4b3bea1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34110b00 con 0x7f4b341006a0 2024-03-19T02:50:06.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60886/0 (socket says 172.21.15.116:60886) 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b340fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3943c640 1 -- 172.21.15.116:0/2662051042 learned_addr learned my addr 172.21.15.116:0/2662051042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b341066b0 0x7f4b340fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2662051042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b341066b0 msgr2=0x7f4b340fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b341066b0 0x7f4b340fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 msgr2=0x7f4b340fdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2662051042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b341a6490 con 0x7f4b341006a0 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3943c640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b340fd670 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4b30036040 tx=0x7f4b3002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b2affd640 1 -- 172.21.15.116:0/2662051042 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b30043020 con 0x7f4b341006a0 2024-03-19T02:50:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b341a6720 con 0x7f4b341006a0 2024-03-19T02:50:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b2affd640 1 -- 172.21.15.116:0/2662051042 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b3002ca70 con 0x7f4b341006a0 2024-03-19T02:50:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b2affd640 1 -- 172.21.15.116:0/2662051042 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300039f0 con 0x7f4b341006a0 2024-03-19T02:50:06.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.108+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b341a6c90 con 0x7f4b341006a0 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.111+0000 7f4b3eb6a740 1 Processor -- start 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.111+0000 7f4b2affd640 1 -- 172.21.15.116:0/2662051042 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f4b30046020 con 0x7f4b341006a0 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.111+0000 7f4b3eb6a740 1 -- start start 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a94290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3a96bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5619e39ee4b0 con 0x5619e3a8bbf0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619e3996d70 con 0x5619e3a947d0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5619e39ea550 con 0x5619e3a8b2d0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3a96bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a94290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59126/0 (socket says 172.21.15.116:59126) 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 -- 172.21.15.116:0/3335768825 learned_addr learned my addr 172.21.15.116:0/3335768825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 -- 172.21.15.116:0/3335768825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 msgr2=0x5619e3a96bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3a96bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b2affd640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 0x7f4b200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 -- 172.21.15.116:0/3335768825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 msgr2=0x5619e3a94290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a94290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 -- 172.21.15.116:0/3335768825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619e39b9210 con 0x5619e3a8b2d0 2024-03-19T02:50:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.112+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 0x7f4b200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.113+0000 7f4b2affd640 1 -- 172.21.15.116:0/2662051042 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f4b300f1110 con 0x7f4b341006a0 2024-03-19T02:50:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.113+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009f50 tx=0x7f4b2c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7445c4c1cfdbe597 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.113+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 0x7f4b200a4e60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c012fd0 tx=0x7f4b2c021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.114+0000 7f4b1ffff640 1 -- 172.21.15.116:0/3335768825 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c00f680 con 0x5619e3a8b2d0 2024-03-19T02:50:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.114+0000 7f4b1ffff640 1 -- 172.21.15.116:0/3335768825 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2c003cb0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.114+0000 7f4b1ffff640 1 -- 172.21.15.116:0/3335768825 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c01ba90 con 0x5619e3a8b2d0 2024-03-19T02:50:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3335768825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 msgr2=0x5619e3a8b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009f50 tx=0x7f4b2c0118d0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3335768825 shutdown_connections 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3a96bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a94290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3335768825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8b6b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3335768825 >> 172.21.15.116:0/3335768825 conn(0x5619e3a85a00 msgr2=0x5619e3a85de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3335768825 shutdown_connections 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3335768825 wait complete. 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.115+0000 7f4b3eb6a740 1 -- start start 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 Processor -- start 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8bbf0 0x5619e3a91da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a947d0 0x5619e3a922e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5619e3a975c0 con 0x5619e3a947d0 2024-03-19T02:50:06.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619e39aa310 con 0x5619e3a8bbf0 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5619e3a97740 con 0x5619e3a8b2d0 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59130/0 (socket says 172.21.15.116:59130) 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3206268579 learned_addr learned my addr 172.21.15.116:0/3206268579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8bbf0 0x5619e3a91da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3eb6a740 10 client.0 _ll_register_callbacks cb 0x5619e399e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a947d0 0x5619e3a922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8bbf0 msgr2=0x5619e3a91da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8bbf0 0x5619e3a91da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3206268579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a947d0 msgr2=0x5619e3a922e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a947d0 0x5619e3a922e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.116+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3206268579 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619e37023f0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4b30006010 tx=0x7f4b300f4850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd2d1a6d0392ddde server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300bcf90 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f4b300b9ac0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5619e3a4c3e0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f4b300bcdb0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b3eb6a740 10 client.31531 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5619e37cc7d0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5619e3a4c3e0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.117+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f4b300c1150 con 0x5619e3a8b2d0 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 10 client.31531 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 10 client.31531 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 20 client.31531 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3aa4b80 0x5619e3aa7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 4 client.31531 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:06.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619e39fb9f0 con 0x5619e3aa4b80 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3aa4b80 0x5619e3aa7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3aa4b80 0x5619e3aa7000 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=258f7db0c8cc5058 server_cookie=61362c76d26b29b6 in_seq=0 out_seq=0 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.118+0000 7f4b1d7fa640 10 client.31531 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3206268579 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5619e39fb9f0 con 0x5619e3aa4b80 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b1d7fa640 10 client.31531 handle_command_reply: tid=0 2024-03-19T02:50:06.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 client.31531 shutdown 2024-03-19T02:50:06.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 msgr2=0x5619e3a91860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4b30006010 tx=0x7f4b300f4850 comp rx=0 tx=0).stop 2024-03-19T02:50:06.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 shutdown_connections 2024-03-19T02:50:06.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3aa4b80 0x5619e3aa7000 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a947d0 0x5619e3a922e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b1d7fa640 0 client.31531 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8bbf0 0x5619e3a91da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3206268579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3a91860 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 >> 172.21.15.116:0/3206268579 conn(0x5619e3a85a00 msgr2=0x5619e3a87750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 shutdown_connections 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3206268579 wait complete. 2024-03-19T02:50:06.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.119+0000 7f4b3eb6a740 20 client.31531 trim_cache size 0 max 16384 2024-03-19T02:50:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.133+0000 7f4b3eb6a740 1 Processor -- start 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.133+0000 7f4b3eb6a740 1 -- start start 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3aac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341acb80 con 0x5619e3a8bbf0 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ae6f0 con 0x5619e3a947d0 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ad250 con 0x5619e3a8b2d0 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59140/0 (socket says 172.21.15.116:59140) 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2508036927 learned_addr learned my addr 172.21.15.116:0/2508036927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60914/0 (socket says 172.21.15.116:60914) 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3a43e640 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3aac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2508036927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 msgr2=0x5619e3aac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3aac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2508036927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 msgr2=0x5619e3a92120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 -- 172.21.15.116:0/2508036927 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619e3a812c0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b3943c640 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.134+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4b300bd3e0 tx=0x7f4b3002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=badb72fb4f48bc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b1f7fe640 1 -- 172.21.15.116:0/2508036927 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300bcf90 con 0x5619e3a8b2d0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b1f7fe640 1 -- 172.21.15.116:0/2508036927 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b300bdb50 con 0x5619e3a8b2d0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b1f7fe640 1 -- 172.21.15.116:0/2508036927 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300b9c60 con 0x5619e3a8b2d0 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/2508036927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 msgr2=0x5619e3aa03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4b300bd3e0 tx=0x7f4b3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/2508036927 shutdown_connections 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a947d0 0x5619e3aac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a92120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/2508036927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a8b2d0 0x5619e3aa03b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.135+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/2508036927 >> 172.21.15.116:0/2508036927 conn(0x5619e3a85a00 msgr2=0x5619e3a84ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/2508036927 shutdown_connections 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/2508036927 wait complete. 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- start start 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 Processor -- start 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a8e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a947d0 0x5619e3a8ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ae020 con 0x5619e3a8bbf0 2024-03-19T02:50:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341ad920 con 0x5619e3a8b2d0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341aabe0 con 0x5619e3a947d0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3eb6a740 10 client.0 _ll_register_callbacks cb 0x7f4b300f4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3943c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a8e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b3a43e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a947d0 0x5619e3a8ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b39c3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48164/0 (socket says 172.21.15.116:48164) 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.136+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3799422780 learned_addr learned my addr 172.21.15.116:0/3799422780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3799422780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a947d0 msgr2=0x5619e3a8ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a947d0 0x5619e3a8ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3799422780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 msgr2=0x5619e3a8e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a8e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 -- 172.21.15.116:0/3799422780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619e3b09f00 con 0x5619e3a8b2d0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a8e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b39c3d640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4b3002c170 tx=0x7f4b300f48e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28466d6e2e5b9868 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b300bcf90 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b300bdb50 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.137+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f4b300c5760 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f4b300bd450 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 10 client.31553 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5619e3b09f00 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b300bdb50 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f4b300dd0b0 con 0x5619e3a8b2d0 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 10 client.31553 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 10 client.31553 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 20 client.31553 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3b0b440 0x5619e3b0d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 4 client.31553 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5619e39f5d40 con 0x5619e3b0b440 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.138+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3b0b440 0x5619e3b0d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b1d7fa640 10 client.31553 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3943c640 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3b0b440 0x5619e3b0d830 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6900dbd8cac10a08 server_cookie=40c01ea307378d22 in_seq=0 out_seq=0 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b1d7fa640 1 -- 172.21.15.116:0/3799422780 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5619e39f5d40 con 0x5619e3b0b440 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b1d7fa640 10 client.31553 handle_command_reply: tid=0 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 client.31553 shutdown 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 msgr2=0x5619e3a8e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4b3002c170 tx=0x7f4b300f48e0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 shutdown_connections 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5619e3b0b440 0x5619e3b0d830 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619e3a947d0 0x5619e3a8ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5619e3a8bbf0 0x5619e3a8e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 --2- 172.21.15.116:0/3799422780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e3a8b2d0 0x5619e3a8e3b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 >> 172.21.15.116:0/3799422780 conn(0x5619e3a85a00 msgr2=0x5619e3a86bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b1d7fa640 0 client.31553 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 shutdown_connections 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.139+0000 7f4b3eb6a740 1 -- 172.21.15.116:0/3799422780 wait complete. 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3eb6a740 20 client.31553 trim_cache size 0 max 16384 2024-03-19T02:50:06.151 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 msgr2=0x7f4b200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 0x7f4b200a4e60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c012fd0 tx=0x7f4b2c021000 comp rx=0 tx=0).stop 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 msgr2=0x7f4b340fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b340fd670 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4b30036040 tx=0x7f4b3002c640 comp rx=0 tx=0).stop 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 shutdown_connections 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f4b200a29a0 0x7f4b200a4e60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b341066b0 0x7f4b340fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b34101060 0x7f4b340fdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 --2- 172.21.15.116:0/2662051042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b341006a0 0x7f4b340fd670 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 >> 172.21.15.116:0/2662051042 conn(0x7f4b340778d0 msgr2=0x7f4b340ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 shutdown_connections 2024-03-19T02:50:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.140+0000 7f4b3bea1640 1 -- 172.21.15.116:0/2662051042 wait complete. 2024-03-19T02:50:06.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:50:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:06 smithi116 ceph-mon[32263]: pgmap v703: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 222 op/s 2024-03-19T02:50:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:06 smithi116 ceph-mon[32263]: osdmap e128: 12 total, 12 up, 12 in 2024-03-19T02:50:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:06 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3828995301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.280+0000 7f61f82dc640 1 -- 172.21.15.116:0/4284385751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 msgr2=0x7f61f0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.280+0000 7f61f82dc640 1 --2- 172.21.15.116:0/4284385751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0100b20 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009930 tx=0x7f61e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- 172.21.15.116:0/4284385751 shutdown_connections 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- 172.21.15.116:0/4284385751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- 172.21.15.116:0/4284385751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0100b20 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- 172.21.15.116:0/4284385751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 0x7f61f0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- 172.21.15.116:0/4284385751 >> 172.21.15.116:0/4284385751 conn(0x7f61f00778d0 msgr2=0x7f61f0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- 172.21.15.116:0/4284385751 shutdown_connections 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- 172.21.15.116:0/4284385751 wait complete. 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 Processor -- start 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- start start 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 0x7f61f019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0110970 con 0x7f61f01006a0 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01107f0 con 0x7f61f01066b0 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.281+0000 7f61f82dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0110af0 con 0x7f61f0101060 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f5877640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60928/0 (socket says 172.21.15.116:60928) 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6879640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 0x7f61f019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 -- 172.21.15.116:0/1556712056 learned_addr learned my addr 172.21.15.116:0/1556712056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 -- 172.21.15.116:0/1556712056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 msgr2=0x7f61f0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 -- 172.21.15.116:0/1556712056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 msgr2=0x7f61f019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 0x7f61f019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 -- 172.21.15.116:0/1556712056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f019cfb0 con 0x7f61f01006a0 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f5877640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f6078640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9f0 tx=0x7f61ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61e77fe640 1 -- 172.21.15.116:0/1556712056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec00cd80 con 0x7f61f01006a0 2024-03-19T02:50:06.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f019d2a0 con 0x7f61f01006a0 2024-03-19T02:50:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61e77fe640 1 -- 172.21.15.116:0/1556712056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec0108e0 con 0x7f61f01006a0 2024-03-19T02:50:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61e77fe640 1 -- 172.21.15.116:0/1556712056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec018920 con 0x7f61f01006a0 2024-03-19T02:50:06.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.282+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f01a4b30 con 0x7f61f01006a0 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.285+0000 7f61e77fe640 1 -- 172.21.15.116:0/1556712056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f61ec010a90 con 0x7f61f01006a0 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.285+0000 7f61fafa1740 1 Processor -- start 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.285+0000 7f61fafa1740 1 -- start start 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb84416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb844cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559cb83a44b0 con 0x559cb84412d0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559cb834cd70 con 0x559cb844a7d0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61fafa1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559cb83a0550 con 0x559cb8441bf0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61e77fe640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 0x7f61dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6879640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb84416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f5877640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb844cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59160/0 (socket says 172.21.15.116:59160) 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f5877640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb844cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48200/0 (socket says 172.21.15.116:48200) 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 -- 172.21.15.116:0/241252038 learned_addr learned my addr 172.21.15.116:0/241252038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6879640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 0x7f61dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61e77fe640 1 -- 172.21.15.116:0/1556712056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f61ec0cf540 con 0x7f61f01006a0 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 -- 172.21.15.116:0/241252038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 msgr2=0x559cb844cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb844cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 -- 172.21.15.116:0/241252038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 msgr2=0x559cb84416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb84416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 -- 172.21.15.116:0/241252038 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb836f210 con 0x559cb8441bf0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6879640 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb84416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6078640 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9c0 tx=0x7f61ec09b180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca114d194388e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.286+0000 7f61f6879640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 0x7f61dc0a4e30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f61e8005c40 tx=0x7f61e8004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61e4bfe640 1 -- 172.21.15.116:0/241252038 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec09f8c0 con 0x559cb8441bf0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61e4bfe640 1 -- 172.21.15.116:0/241252038 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec0a3f10 con 0x559cb8441bf0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61e4bfe640 1 -- 172.21.15.116:0/241252038 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0cf170 con 0x559cb8441bf0 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 -- 172.21.15.116:0/241252038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 msgr2=0x559cb844a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9c0 tx=0x7f61ec09b180 comp rx=0 tx=0).stop 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 -- 172.21.15.116:0/241252038 shutdown_connections 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb844cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb8441bf0 0x559cb844a290 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 --2- 172.21.15.116:0/241252038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb84416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 -- 172.21.15.116:0/241252038 >> 172.21.15.116:0/241252038 conn(0x559cb843ba00 msgr2=0x559cb843bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 -- 172.21.15.116:0/241252038 shutdown_connections 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.287+0000 7f61fafa1740 1 -- 172.21.15.116:0/241252038 wait complete. 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 -- start start 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 Processor -- start 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8447dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559cb844d560 con 0x559cb844a7d0 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559cb8360310 con 0x559cb8441bf0 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559cb844d6e0 con 0x559cb84412d0 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61fafa1740 10 client.0 _ll_register_callbacks cb 0x559cb83542d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f6879640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f6078640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f5877640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8447dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f6879640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60960/0 (socket says 172.21.15.116:60960) 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f6879640 1 -- 172.21.15.116:0/2732776961 learned_addr learned my addr 172.21.15.116:0/2732776961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.288+0000 7f61f6078640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59170/0 (socket says 172.21.15.116:59170) 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 -- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 msgr2=0x559cb8447880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 -- 172.21.15.116:0/2732776961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 msgr2=0x559cb8447dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8447dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 -- 172.21.15.116:0/2732776961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb80b83f0 con 0x559cb844a7d0 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6078640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f5877640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8447dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61f6879640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f61e8003c20 tx=0x7f61e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c188ef15058fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8026030 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f61e801c070 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ec0a3f10 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f61e8032050 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61fafa1740 10 client.21852 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559cb81827d0 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec0a3f10 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f61e801f050 con 0x559cb844a7d0 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61fafa1740 10 client.21852 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.289+0000 7f61fafa1740 10 client.21852 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61fafa1740 20 client.21852 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb845ab20 0x559cb845cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61fafa1740 4 client.21852 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cb84023e0 con 0x559cb845ab20 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61f6078640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb845ab20 0x559cb845cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61d9ffb640 10 client.21852 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.290+0000 7f61f6078640 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb845ab20 0x559cb845cf30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b950b74bdf23ab3f server_cookie=1ee7edeb1dcaa9ee in_seq=0 out_seq=0 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2732776961 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559cb84023e0 con 0x559cb845ab20 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61d9ffb640 10 client.21852 handle_command_reply: tid=0 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 client.21852 shutdown 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 msgr2=0x559cb8448300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f61e8003c20 tx=0x7f61e8004a40 comp rx=0 tx=0).stop 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 shutdown_connections 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb845ab20 0x559cb845cf30 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb844a7d0 0x559cb8448300 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8447dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2732776961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8447880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61d9ffb640 0 client.21852 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 >> 172.21.15.116:0/2732776961 conn(0x559cb843ba00 msgr2=0x559cb843d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 shutdown_connections 2024-03-19T02:50:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 1 -- 172.21.15.116:0/2732776961 wait complete. 2024-03-19T02:50:06.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.291+0000 7f61fafa1740 20 client.21852 trim_cache size 0 max 16384 2024-03-19T02:50:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 Processor -- start 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 -- start start 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8456300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb844a7d0 0x559cb8462800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01aee30 con 0x559cb84412d0 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01b09a0 con 0x559cb8441bf0 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61fafa1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01af500 con 0x559cb844a7d0 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61f6078640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61f6078640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60976/0 (socket says 172.21.15.116:60976) 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61f6078640 1 -- 172.21.15.116:0/1700614224 learned_addr learned my addr 172.21.15.116:0/1700614224 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.311+0000 7f61f6879640 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb844a7d0 0x559cb8462800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f5877640 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8456300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 -- 172.21.15.116:0/1700614224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb844a7d0 msgr2=0x559cb8462800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb844a7d0 0x559cb8462800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 -- 172.21.15.116:0/1700614224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 msgr2=0x559cb8456300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8456300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 -- 172.21.15.116:0/1700614224 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb8411880 con 0x559cb84412d0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61f6078640 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0cc910 tx=0x7f61ec0cf840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379ceb1fdfae1398 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61dbfff640 1 -- 172.21.15.116:0/1700614224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0a3f10 con 0x559cb84412d0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61dbfff640 1 -- 172.21.15.116:0/1700614224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec097f00 con 0x559cb84412d0 2024-03-19T02:50:06.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.312+0000 7f61dbfff640 1 -- 172.21.15.116:0/1700614224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec09ac70 con 0x559cb84412d0 2024-03-19T02:50:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 -- 172.21.15.116:0/1700614224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 msgr2=0x559cb8449fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0cc910 tx=0x7f61ec0cf840 comp rx=0 tx=0).stop 2024-03-19T02:50:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 -- 172.21.15.116:0/1700614224 shutdown_connections 2024-03-19T02:50:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb844a7d0 0x559cb8462800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb8441bf0 0x559cb8456300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 --2- 172.21.15.116:0/1700614224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb84412d0 0x559cb8449fb0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 -- 172.21.15.116:0/1700614224 >> 172.21.15.116:0/1700614224 conn(0x559cb843ba00 msgr2=0x559cb843a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 -- 172.21.15.116:0/1700614224 shutdown_connections 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.313+0000 7f61fafa1740 1 -- 172.21.15.116:0/1700614224 wait complete. 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 -- start start 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 Processor -- start 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8452b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb8444630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01b02d0 con 0x559cb8441bf0 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01afbd0 con 0x559cb844a7d0 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61f01ace30 con 0x559cb84412d0 2024-03-19T02:50:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61fafa1740 10 client.0 _ll_register_callbacks cb 0x7f61e8013970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60988/0 (socket says 172.21.15.116:60988) 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 -- 172.21.15.116:0/2981051859 learned_addr learned my addr 172.21.15.116:0/2981051859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f6879640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb8444630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f6078640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8452b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 -- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 msgr2=0x559cb8452b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8452b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 -- 172.21.15.116:0/2981051859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 msgr2=0x559cb8444630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb8444630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.314+0000 7f61f5877640 1 -- 172.21.15.116:0/2981051859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb8444e30 con 0x559cb8441bf0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61f5877640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f61e0005e00 tx=0x7f61e002cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e2d3e8d004f9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0003800 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f61e00039a0 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ec097f00 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f61e0003ba0 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 10 client.21858 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559cb84c0d50 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ec097f00 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f61e0040420 con 0x559cb8441bf0 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 10 client.21858 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 10 client.21858 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 20 client.21858 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.315+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb84c1580 0x559cb84c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 4 client.21858 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559cb83b19f0 con 0x559cb84c1580 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61f6078640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb84c1580 0x559cb84c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61d9ffb640 10 client.21858 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61f6078640 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb84c1580 0x559cb84c39a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3dc5199eb716879f server_cookie=318188ce1873aba0 in_seq=0 out_seq=0 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61d9ffb640 1 -- 172.21.15.116:0/2981051859 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559cb83b19f0 con 0x559cb84c1580 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61d9ffb640 10 client.21858 handle_command_reply: tid=0 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 client.21858 shutdown 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 msgr2=0x559cb84440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f61e0005e00 tx=0x7f61e002cb90 comp rx=0 tx=0).stop 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 shutdown_connections 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x559cb84c1580 0x559cb84c39a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559cb844a7d0 0x559cb8444630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cb8441bf0 0x559cb84440f0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 --2- 172.21.15.116:0/2981051859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559cb84412d0 0x559cb8452b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 >> 172.21.15.116:0/2981051859 conn(0x559cb843ba00 msgr2=0x559cb84480b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.316+0000 7f61d9ffb640 0 client.21858 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 shutdown_connections 2024-03-19T02:50:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61fafa1740 1 -- 172.21.15.116:0/2981051859 wait complete. 2024-03-19T02:50:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61fafa1740 20 client.21858 trim_cache size 0 max 16384 2024-03-19T02:50:06.325 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 msgr2=0x7f61dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 0x7f61dc0a4e30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f61e8005c40 tx=0x7f61e8004ab0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 msgr2=0x7f61f0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9f0 tx=0x7f61ec00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 shutdown_connections 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61dc0a2970 0x7f61dc0a4e30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61f01066b0 0x7f61f019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61f0101060 0x7f61f0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 --2- 172.21.15.116:0/1556712056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61f01006a0 0x7f61f0197fc0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 >> 172.21.15.116:0/1556712056 conn(0x7f61f00778d0 msgr2=0x7f61f00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 shutdown_connections 2024-03-19T02:50:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.317+0000 7f61f82dc640 1 -- 172.21.15.116:0/1556712056 wait complete. 2024-03-19T02:50:06.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:50:06.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62141 5.80 MB/sec execute 448 sec latency 258.685 ms 2024-03-19T02:50:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:06 smithi162 ceph-mon[35761]: pgmap v703: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 222 op/s 2024-03-19T02:50:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:06 smithi162 ceph-mon[35761]: osdmap e128: 12 total, 12 up, 12 in 2024-03-19T02:50:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:06 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3828995301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 -- 172.21.15.116:0/983783763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c1039b0 msgr2=0x7f926c103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 --2- 172.21.15.116:0/983783763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c1039b0 0x7f926c103e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9260009890 tx=0x7f926002bb70 comp rx=0 tx=0).stop 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 -- 172.21.15.116:0/983783763 shutdown_connections 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 --2- 172.21.15.116:0/983783763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c104370 0x7f926c110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 --2- 172.21.15.116:0/983783763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c1039b0 0x7f926c103e30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 --2- 172.21.15.116:0/983783763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 -- 172.21.15.116:0/983783763 >> 172.21.15.116:0/983783763 conn(0x7f926c0ff4a0 msgr2=0x7f926c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 -- 172.21.15.116:0/983783763 shutdown_connections 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.464+0000 7f9270f0a640 1 -- 172.21.15.116:0/983783763 wait complete. 2024-03-19T02:50:06.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 Processor -- start 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 -- start start 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 0x7f926c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f926c113c80 con 0x7f926c1039b0 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f926c113b00 con 0x7f926c1099c0 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9270f0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f926c113e00 con 0x7f926c104370 2024-03-19T02:50:06.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f926a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 0x7f926c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:48256/0 (socket says 172.21.15.116:48256) 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59208/0 (socket says 172.21.15.116:59208) 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.465+0000 7f926ad9d640 1 -- 172.21.15.116:0/1504627049 learned_addr learned my addr 172.21.15.116:0/1504627049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 -- 172.21.15.116:0/1504627049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 msgr2=0x7f926c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 -- 172.21.15.116:0/1504627049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 msgr2=0x7f926c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 -- 172.21.15.116:0/1504627049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c19f210 con 0x7f926c1039b0 2024-03-19T02:50:06.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f926a59c640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 0x7f926c19a220 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e9f0 tx=0x7f925c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f925b7fe640 1 -- 172.21.15.116:0/1504627049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c00ccc0 con 0x7f926c1039b0 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c19f500 con 0x7f926c1039b0 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f925b7fe640 1 -- 172.21.15.116:0/1504627049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f925c005020 con 0x7f926c1039b0 2024-03-19T02:50:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f925b7fe640 1 -- 172.21.15.116:0/1504627049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c004a00 con 0x7f926c1039b0 2024-03-19T02:50:06.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.466+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c1a6d90 con 0x7f926c1039b0 2024-03-19T02:50:06.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.468+0000 7f925b7fe640 1 -- 172.21.15.116:0/1504627049 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f925c004ba0 con 0x7f926c1039b0 2024-03-19T02:50:06.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 Processor -- start 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 -- start start 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60704290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60706bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f925b7fe640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 0x7f92540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd6065e4b0 con 0x55fd606fb2d0 2024-03-19T02:50:06.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd60606d70 con 0x55fd607047d0 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9273bcd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd6065a550 con 0x55fd606fbbf0 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 0x7f92540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f926ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32786/0 (socket says 172.21.15.116:32786) 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f926a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60704290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f926ad9d640 1 -- 172.21.15.116:0/1560402538 learned_addr learned my addr 172.21.15.116:0/1560402538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.469+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60706bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f925b7fe640 1 -- 172.21.15.116:0/1504627049 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f925c0cf1b0 con 0x7f926c1039b0 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 -- 172.21.15.116:0/1560402538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 msgr2=0x55fd60704290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60704290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 -- 172.21.15.116:0/1560402538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 msgr2=0x55fd60706bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60706bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 -- 172.21.15.116:0/1560402538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd60629210 con 0x55fd606fb2d0 2024-03-19T02:50:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926a59c640 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60704290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 0x7f92540a4e30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f92600097b0 tx=0x7f92600058d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f926400c530 tx=0x7f9264009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6890ccb252d003a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f9258ff9640 1 -- 172.21.15.116:0/1560402538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926400b770 con 0x55fd606fb2d0 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f9258ff9640 1 -- 172.21.15.116:0/1560402538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9264018040 con 0x55fd606fb2d0 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.470+0000 7f9258ff9640 1 -- 172.21.15.116:0/1560402538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926401ea10 con 0x55fd606fb2d0 2024-03-19T02:50:06.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 -- 172.21.15.116:0/1560402538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 msgr2=0x55fd606fb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f926400c530 tx=0x7f9264009d10 comp rx=0 tx=0).stop 2024-03-19T02:50:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 -- 172.21.15.116:0/1560402538 shutdown_connections 2024-03-19T02:50:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60706bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60704290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1560402538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd606fb6b0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 -- 172.21.15.116:0/1560402538 >> 172.21.15.116:0/1560402538 conn(0x55fd606f5a00 msgr2=0x55fd606f5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 -- 172.21.15.116:0/1560402538 shutdown_connections 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.471+0000 7f9273bcd740 1 -- 172.21.15.116:0/1560402538 wait complete. 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 -- start start 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 Processor -- start 2024-03-19T02:50:06.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60701cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60702230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd607075c0 con 0x55fd606fb2d0 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd6061a310 con 0x55fd607047d0 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fd60707740 con 0x55fd606fbbf0 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9273bcd740 10 client.0 _ll_register_callbacks cb 0x55fd6060e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f926a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60701cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f926a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32796/0 (socket says 172.21.15.116:32796) 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f926a59c640 1 -- 172.21.15.116:0/2079251510 learned_addr learned my addr 172.21.15.116:0/2079251510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f926ad9d640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60702230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9269d9b640 1 -- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 msgr2=0x55fd60702230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9269d9b640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60702230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9269d9b640 1 -- 172.21.15.116:0/2079251510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 msgr2=0x55fd607017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.472+0000 7f9269d9b640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9269d9b640 1 -- 172.21.15.116:0/2079251510 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd603723f0 con 0x55fd606fbbf0 2024-03-19T02:50:06.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f926a59c640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9269d9b640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60701cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f926002c0b0 tx=0x7f926002bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=239df78c2156da4d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260044070 con 0x55fd606fbbf0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f926002c920 con 0x55fd606fbbf0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9264018040 con 0x55fd606fbbf0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9260059050 con 0x55fd606fbbf0 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9273bcd740 10 client.31567 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fd6043c7d0 con 0x55fd606fbbf0 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9264018040 con 0x55fd606fbbf0 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.473+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f926004b490 con 0x55fd606fbbf0 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 10 client.31567 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 10 client.31567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 20 client.31567 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd60714b00 0x55fd60716f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 4 client.31567 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd606bc3e0 con 0x55fd60714b00 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f926a59c640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd60714b00 0x55fd60716f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f926a59c640 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd60714b00 0x55fd60716f80 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=849af950a2629655 server_cookie=9b91f870aa560ec9 in_seq=0 out_seq=0 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.474+0000 7f9251ffb640 10 client.31567 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.476+0000 7f9251ffb640 1 -- 172.21.15.116:0/2079251510 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd606bc3e0 con 0x55fd60714b00 2024-03-19T02:50:06.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.476+0000 7f9251ffb640 10 client.31567 handle_command_reply: tid=0 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 client.31567 shutdown 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 msgr2=0x55fd60701cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60701cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f926002c0b0 tx=0x7f926002bb70 comp rx=0 tx=0).stop 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 shutdown_connections 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd60714b00 0x55fd60716f80 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60702230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fbbf0 0x55fd60701cf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9251ffb640 0 client.31567 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2079251510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fb2d0 0x55fd607017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 >> 172.21.15.116:0/2079251510 conn(0x55fd606f5a00 msgr2=0x55fd606f7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 shutdown_connections 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 1 -- 172.21.15.116:0/2079251510 wait complete. 2024-03-19T02:50:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.477+0000 7f9273bcd740 20 client.31567 trim_cache size 0 max 16384 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 Processor -- start 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 -- start start 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd60703ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60710360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b0fa0 con 0x55fd606fbbf0 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b2b10 con 0x55fd607047d0 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.494+0000 7f9273bcd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b1670 con 0x55fd606fb2d0 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f926a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd60703ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32808/0 (socket says 172.21.15.116:32808) 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 -- 172.21.15.116:0/1346147364 learned_addr learned my addr 172.21.15.116:0/1346147364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f926ad9d640 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60710360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 -- 172.21.15.116:0/1346147364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 msgr2=0x55fd60703ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd60703ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 -- 172.21.15.116:0/1346147364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 msgr2=0x55fd60710360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60710360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 -- 172.21.15.116:0/1346147364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd60700760 con 0x55fd606fbbf0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9269d9b640 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f926004b900 tx=0x7f9260005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d951ee903ec656ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9253fff640 1 -- 172.21.15.116:0/1346147364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260044070 con 0x55fd606fbbf0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9253fff640 1 -- 172.21.15.116:0/1346147364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9260050070 con 0x55fd606fbbf0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.495+0000 7f9253fff640 1 -- 172.21.15.116:0/1346147364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260039040 con 0x55fd606fbbf0 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 -- 172.21.15.116:0/1346147364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 msgr2=0x55fd606f0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f926004b900 tx=0x7f9260005c90 comp rx=0 tx=0).stop 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 -- 172.21.15.116:0/1346147364 shutdown_connections 2024-03-19T02:50:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd60710360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606f0a30 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 --2- 172.21.15.116:0/1346147364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd60703ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.496+0000 7f9273bcd740 1 -- 172.21.15.116:0/1346147364 >> 172.21.15.116:0/1346147364 conn(0x55fd606f5a00 msgr2=0x55fd606f4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- 172.21.15.116:0/1346147364 shutdown_connections 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- 172.21.15.116:0/1346147364 wait complete. 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- start start 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 Processor -- start 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd606fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b2440 con 0x55fd606fbbf0 2024-03-19T02:50:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1b1d40 con 0x55fd607047d0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1aefa0 con 0x55fd606fb2d0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9273bcd740 10 client.0 _ll_register_callbacks cb 0x7f9264009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f926a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f926a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59256/0 (socket says 172.21.15.116:59256) 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f926a59c640 1 -- 172.21.15.116:0/2927309783 learned_addr learned my addr 172.21.15.116:0/2927309783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f9269d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32816/0 (socket says 172.21.15.116:32816) 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.497+0000 7f926ad9d640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd606fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 -- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 msgr2=0x55fd606fefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd606fefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 -- 172.21.15.116:0/2927309783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 msgr2=0x55fd606fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606fea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 -- 172.21.15.116:0/2927309783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd6077a080 con 0x55fd606fb2d0 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f926a59c640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f925c09b0e0 tx=0x7f925c0096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a525bba78bf1ab82 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9251ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c01c040 con 0x55fd606fb2d0 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f925c0051d0 con 0x55fd606fb2d0 2024-03-19T02:50:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9260050070 con 0x55fd606fb2d0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f925c09b3e0 con 0x55fd606fb2d0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9273bcd740 10 client.31573 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fd6077b380 con 0x55fd606fb2d0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.498+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9260050070 con 0x55fd606fb2d0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f925c09f1b0 con 0x55fd606fb2d0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 10 client.31573 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 10 client.31573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 20 client.31573 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd6077bb00 0x55fd6077df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 4 client.31573 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd6066b9f0 con 0x55fd6077bb00 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9269d9b640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd6077bb00 0x55fd6077df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9251ffb640 10 client.31573 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.499+0000 7f9269d9b640 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd6077bb00 0x55fd6077df20 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e444345ce28f6fe server_cookie=727e17da9ea97258 in_seq=0 out_seq=0 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.500+0000 7f9251ffb640 1 -- 172.21.15.116:0/2927309783 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd6066b9f0 con 0x55fd6077bb00 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.500+0000 7f9251ffb640 10 client.31573 handle_command_reply: tid=0 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.500+0000 7f9273bcd740 1 client.31573 shutdown 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 msgr2=0x55fd606fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f925c09b0e0 tx=0x7f925c0096e0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 shutdown_connections 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55fd6077bb00 0x55fd6077df20 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd607047d0 0x55fd606fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9251ffb640 0 client.31573 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd606fbbf0 0x55fd606fea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 --2- 172.21.15.116:0/2927309783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fd606fb2d0 0x55fd606fe550 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 >> 172.21.15.116:0/2927309783 conn(0x55fd606f5a00 msgr2=0x55fd60701770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 shutdown_connections 2024-03-19T02:50:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 1 -- 172.21.15.116:0/2927309783 wait complete. 2024-03-19T02:50:06.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9273bcd740 20 client.31573 trim_cache size 0 max 16384 2024-03-19T02:50:06.511 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 msgr2=0x7f92540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 0x7f92540a4e30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f92600097b0 tx=0x7f92600058d0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 msgr2=0x7f926c19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 0x7f926c19a220 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e9f0 tx=0x7f925c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 shutdown_connections 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f92540a2970 0x7f92540a4e30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c1099c0 0x7f926c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f926c104370 0x7f926c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 --2- 172.21.15.116:0/1504627049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f926c1039b0 0x7f926c19a220 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 >> 172.21.15.116:0/1504627049 conn(0x7f926c0ff4a0 msgr2=0x7f926c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 shutdown_connections 2024-03-19T02:50:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:06.501+0000 7f9270f0a640 1 -- 172.21.15.116:0/1504627049 wait complete. 2024-03-19T02:50:06.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:06 smithi154 ceph-mon[35693]: pgmap v703: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 222 op/s 2024-03-19T02:50:06.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:06 smithi154 ceph-mon[35693]: osdmap e128: 12 total, 12 up, 12 in 2024-03-19T02:50:06.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:06 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3828995301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:07.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62252 5.80 MB/sec execute 449 sec latency 148.433 ms 2024-03-19T02:50:07.517 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:50:07.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:50:07.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:50:07.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 -- 172.21.15.116:0/81667194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 msgr2=0x7f7c00100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 --2- 172.21.15.116:0/81667194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00100b40 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009900 tx=0x7f7be802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 -- 172.21.15.116:0/81667194 shutdown_connections 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 --2- 172.21.15.116:0/81667194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 0x7f7c0010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 --2- 172.21.15.116:0/81667194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00100b40 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 --2- 172.21.15.116:0/81667194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 0x7f7c001059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 -- 172.21.15.116:0/81667194 >> 172.21.15.116:0/81667194 conn(0x7f7c000777b0 msgr2=0x7f7c00077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 -- 172.21.15.116:0/81667194 shutdown_connections 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.635+0000 7f7c056c6640 1 -- 172.21.15.116:0/81667194 wait complete. 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 Processor -- start 2024-03-19T02:50:07.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 -- start start 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 0x7f7c00199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 0x7f7c001a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00110960 con 0x7f7c001006e0 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001107e0 con 0x7f7c001055c0 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7c056c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00110ae0 con 0x7f7c00101080 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53918/0 (socket says 172.21.15.116:53918) 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 -- 172.21.15.116:0/1984110921 learned_addr learned my addr 172.21.15.116:0/1984110921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 0x7f7c001a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 0x7f7c00199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 -- 172.21.15.116:0/1984110921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 msgr2=0x7f7c00199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 0x7f7c00199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 -- 172.21.15.116:0/1984110921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 msgr2=0x7f7c001a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 0x7f7c001a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.636+0000 7f7bfeffd640 1 -- 172.21.15.116:0/1984110921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c001a43e0 con 0x7f7c001006e0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000e9f0 tx=0x7f7bf000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7bfcff9640 1 -- 172.21.15.116:0/1984110921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf000cd80 con 0x7f7c001006e0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c001a45b0 con 0x7f7c001006e0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c001a4b20 con 0x7f7c001006e0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7bfcff9640 1 -- 172.21.15.116:0/1984110921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf00108e0 con 0x7f7c001006e0 2024-03-19T02:50:07.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.637+0000 7f7bfcff9640 1 -- 172.21.15.116:0/1984110921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0018920 con 0x7f7c001006e0 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.639+0000 7f7bfcff9640 1 -- 172.21.15.116:0/1984110921 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7bf0010a90 con 0x7f7c001006e0 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 Processor -- start 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 -- start start 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a17340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a19c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7c5972160 con 0x55c7c5a0eca0 2024-03-19T02:50:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c7c591ae10 con 0x55c7c5a17880 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7c0838e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7c596e200 con 0x55c7c5a0e380 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bfcff9640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 0x7f7bec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a17340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bf7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a19c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34852/0 (socket says 172.21.15.116:34852) 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 -- 172.21.15.116:0/4002182838 learned_addr learned my addr 172.21.15.116:0/4002182838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 0x7f7bec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 -- 172.21.15.116:0/4002182838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 msgr2=0x55c7c5a19c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a19c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 -- 172.21.15.116:0/4002182838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 msgr2=0x55c7c5a17340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bfcff9640 1 -- 172.21.15.116:0/1984110921 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f7bf00cf4a0 con 0x7f7c001006e0 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a17340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bff7fe640 1 -- 172.21.15.116:0/4002182838 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7c574e7e0 con 0x55c7c5a0e380 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.640+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a17340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8011e20 tx=0x7f7bf800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=354302372f81c47a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 0x7f7bec0a4eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800fc90 tx=0x7f7bf801e000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7bf57fa640 1 -- 172.21.15.116:0/4002182838 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf801dcb0 con 0x55c7c5a0e380 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7bf57fa640 1 -- 172.21.15.116:0/4002182838 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf801de50 con 0x55c7c5a0e380 2024-03-19T02:50:07.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7bf57fa640 1 -- 172.21.15.116:0/4002182838 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8004290 con 0x55c7c5a0e380 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7c0838e740 1 -- 172.21.15.116:0/4002182838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 msgr2=0x55c7c5a0e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7c0838e740 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8011e20 tx=0x7f7bf800f790 comp rx=0 tx=0).stop 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7c0838e740 1 -- 172.21.15.116:0/4002182838 shutdown_connections 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7c0838e740 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a19c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.641+0000 7f7c0838e740 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a17340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 --2- 172.21.15.116:0/4002182838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a0e760 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- 172.21.15.116:0/4002182838 >> 172.21.15.116:0/4002182838 conn(0x55c7c5a08ab0 msgr2=0x55c7c5a08e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- 172.21.15.116:0/4002182838 shutdown_connections 2024-03-19T02:50:07.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- 172.21.15.116:0/4002182838 wait complete. 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- start start 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 Processor -- start 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7c5a1a5b0 con 0x55c7c5a17880 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c7c592e1a0 con 0x55c7c5a0eca0 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7c5a1a730 con 0x55c7c5a0e380 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7c0838e740 10 client.0 _ll_register_callbacks cb 0x55c7c5685c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34862/0 (socket says 172.21.15.116:34862) 2024-03-19T02:50:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53924/0 (socket says 172.21.15.116:53924) 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bfeffd640 1 -- 172.21.15.116:0/3271920066 learned_addr learned my addr 172.21.15.116:0/3271920066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.642+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 -- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 msgr2=0x55c7c5a14df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a14df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 -- 172.21.15.116:0/3271920066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 msgr2=0x55c7c5a15330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 -- 172.21.15.116:0/3271920066 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7c57aef70 con 0x55c7c5a0e380 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000c6a0 tx=0x7f7bf0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35886523452f20f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7be67fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf009f820 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f7bf0097df0 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf801de50 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7bf00a3e70 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7c0838e740 10 client.31585 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.643+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c7c5646170 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf801de50 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f7bf00cc3a0 con 0x55c7c5a0e380 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 10 client.31585 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 10 client.31585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 20 client.31585 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a27bf0 0x55c7c5a2a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 4 client.31585 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7c597f6a0 con 0x55c7c5a27bf0 2024-03-19T02:50:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a27bf0 0x55c7c5a2a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7be67fc640 10 client.31585 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.644+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a27bf0 0x55c7c5a2a020 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a8e8e79a0200759 server_cookie=ca6893168582681d in_seq=0 out_seq=0 2024-03-19T02:50:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7be67fc640 1 -- 172.21.15.116:0/3271920066 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7c597f6a0 con 0x55c7c5a27bf0 2024-03-19T02:50:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7be67fc640 10 client.31585 handle_command_reply: tid=0 2024-03-19T02:50:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 client.31585 shutdown 2024-03-19T02:50:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 msgr2=0x55c7c5a148b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000c6a0 tx=0x7f7bf0009310 comp rx=0 tx=0).stop 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 shutdown_connections 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a27bf0 0x55c7c5a2a020 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a15330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7be67fc640 0 client.31585 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 --2- 172.21.15.116:0/3271920066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a148b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 >> 172.21.15.116:0/3271920066 conn(0x55c7c5a08ab0 msgr2=0x55c7c5a0a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 shutdown_connections 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 1 -- 172.21.15.116:0/3271920066 wait complete. 2024-03-19T02:50:07.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.647+0000 7f7c0838e740 20 client.31585 trim_cache size 0 max 16384 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.667+0000 7f7c0838e740 1 Processor -- start 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.667+0000 7f7c0838e740 1 -- start start 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a23350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a2f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001aed50 con 0x55c7c5a0e380 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001b08c0 con 0x55c7c5a17880 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7c0838e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001af420 con 0x55c7c5a0eca0 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53932/0 (socket says 172.21.15.116:53932) 2024-03-19T02:50:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 -- 172.21.15.116:0/2442807881 learned_addr learned my addr 172.21.15.116:0/2442807881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a23350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a2f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 -- 172.21.15.116:0/2442807881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0eca0 msgr2=0x55c7c5a23350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a23350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 -- 172.21.15.116:0/2442807881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 msgr2=0x55c7c5a2f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a2f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 -- 172.21.15.116:0/2442807881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7c5a19c70 con 0x55c7c5a0e380 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a2f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7bf009b430 tx=0x7f7bf009b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a143cd83cfc6651e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.668+0000 7f7bf4ff9640 1 -- 172.21.15.116:0/2442807881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0065070 con 0x55c7c5a0e380 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7bf4ff9640 1 -- 172.21.15.116:0/2442807881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf009aef0 con 0x55c7c5a0e380 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7bf4ff9640 1 -- 172.21.15.116:0/2442807881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0010430 con 0x55c7c5a0e380 2024-03-19T02:50:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 -- 172.21.15.116:0/2442807881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 msgr2=0x55c7c5a14190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7bf009b430 tx=0x7f7bf009b900 comp rx=0 tx=0).stop 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 -- 172.21.15.116:0/2442807881 shutdown_connections 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a17880 0x55c7c5a2f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a23350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 --2- 172.21.15.116:0/2442807881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a0e380 0x55c7c5a14190 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 -- 172.21.15.116:0/2442807881 >> 172.21.15.116:0/2442807881 conn(0x55c7c5a08ab0 msgr2=0x55c7c5a15100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 -- 172.21.15.116:0/2442807881 shutdown_connections 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.669+0000 7f7c0838e740 1 -- 172.21.15.116:0/2442807881 wait complete. 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 -- start start 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 Processor -- start 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a1b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a1b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001b01f0 con 0x55c7c5a17880 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001afaf0 con 0x55c7c5a0eca0 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c001acd50 con 0x55c7c5a0e380 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7c0838e740 10 client.0 _ll_register_callbacks cb 0x7f7bf000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7bff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53942/0 (socket says 172.21.15.116:53942) 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7bff7fe640 1 -- 172.21.15.116:0/1483359962 learned_addr learned my addr 172.21.15.116:0/1483359962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7bf7fff640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a1b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.670+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a1b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 -- 172.21.15.116:0/1483359962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 msgr2=0x55c7c5a1b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a1b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 -- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 msgr2=0x55c7c5a1b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a1b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 -- 172.21.15.116:0/1483359962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7c5a8d0f0 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7bff7fe640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8013010 tx=0x7f7bf8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5327fc21ff95a3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7be67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8011230 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f7bf80050b0 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf009aef0 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7bf803f050 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7c0838e740 10 client.21894 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.671+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c7c5a8e3f0 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf009aef0 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f7bf803a670 con 0x55c7c5a17880 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 10 client.21894 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:07.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 10 client.21894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 20 client.21894 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a8ea50 0x55c7c5a90e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 4 client.21894 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c7c5a104b0 con 0x55c7c5a8ea50 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a8ea50 0x55c7c5a90e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7be67fc640 10 client.21894 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.672+0000 7f7bfeffd640 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a8ea50 0x55c7c5a90e70 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a6ca2c8fad09d server_cookie=27148d7a5056fea6 in_seq=0 out_seq=0 2024-03-19T02:50:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7be67fc640 1 -- 172.21.15.116:0/1483359962 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c7c5a104b0 con 0x55c7c5a8ea50 2024-03-19T02:50:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7be67fc640 10 client.21894 handle_command_reply: tid=0 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 client.21894 shutdown 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 msgr2=0x55c7c5a1bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8013010 tx=0x7f7bf8009500 comp rx=0 tx=0).stop 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 shutdown_connections 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55c7c5a8ea50 0x55c7c5a90e70 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7c5a17880 0x55c7c5a1bf20 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c7c5a0eca0 0x55c7c5a1b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 --2- 172.21.15.116:0/1483359962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7c5a0e380 0x55c7c5a1b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 >> 172.21.15.116:0/1483359962 conn(0x55c7c5a08ab0 msgr2=0x55c7c5a072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7be67fc640 0 client.21894 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 shutdown_connections 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 1 -- 172.21.15.116:0/1483359962 wait complete. 2024-03-19T02:50:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.676+0000 7f7c0838e740 20 client.21894 trim_cache size 0 max 16384 2024-03-19T02:50:07.682 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:07.682 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:50:07.682 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "b34ecd1f-2e79-4d15-83d5-38fb33dde85e", 2024-03-19T02:50:07.682 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:50:07.682 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:07.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 msgr2=0x7f7bec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 0x7f7bec0a4eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800fc90 tx=0x7f7bf801e000 comp rx=0 tx=0).stop 2024-03-19T02:50:07.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 msgr2=0x7f7c00198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000e9f0 tx=0x7f7bf000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 shutdown_connections 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7bec0a29f0 0x7f7bec0a4eb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c001055c0 0x7f7c001a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c00101080 0x7f7c00199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 --2- 172.21.15.116:0/1984110921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c001006e0 0x7f7c00198e20 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 >> 172.21.15.116:0/1984110921 conn(0x7f7c000777b0 msgr2=0x7f7c000fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 shutdown_connections 2024-03-19T02:50:07.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.677+0000 7f7c056c6640 1 -- 172.21.15.116:0/1984110921 wait complete. 2024-03-19T02:50:07.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:50:07.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:50:07.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- 172.21.15.116:0/3998230934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9424101790 msgr2=0x7f9424101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3998230934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9424101790 0x7f9424101c10 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f9414009930 tx=0x7f941402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- 172.21.15.116:0/3998230934 shutdown_connections 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3998230934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f942410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3998230934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9424101790 0x7f9424101c10 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3998230934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94241077a0 0x7f9424107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- 172.21.15.116:0/3998230934 >> 172.21.15.116:0/3998230934 conn(0x7f94240fd280 msgr2=0x7f94240ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- 172.21.15.116:0/3998230934 shutdown_connections 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- 172.21.15.116:0/3998230934 wait complete. 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 Processor -- start 2024-03-19T02:50:07.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.810+0000 7f942c3cd640 1 -- start start 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f94241985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9424111a60 con 0x7f94241077a0 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94241118e0 con 0x7f9424101790 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942c3cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9424111be0 con 0x7f9424102150 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:40566/0 (socket says 172.21.15.116:40566) 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f9429968640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f94241985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 -- 172.21.15.116:0/3926464828 learned_addr learned my addr 172.21.15.116:0/3926464828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53964/0 (socket says 172.21.15.116:53964) 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 -- 172.21.15.116:0/3926464828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 msgr2=0x7f94241985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f94241985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 -- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 msgr2=0x7f942419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a169640 1 -- 172.21.15.116:0/3926464828 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942419d060 con 0x7f9424101790 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f9429968640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f94241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.811+0000 7f942a96a640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f942a169640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9f0 tx=0x7f942000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f941b7fe640 1 -- 172.21.15.116:0/3926464828 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942000cd80 con 0x7f9424101790 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942419d350 con 0x7f9424101790 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f941b7fe640 1 -- 172.21.15.116:0/3926464828 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94200108e0 con 0x7f9424101790 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f941b7fe640 1 -- 172.21.15.116:0/3926464828 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420018920 con 0x7f9424101790 2024-03-19T02:50:07.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.812+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94241a4be0 con 0x7f9424101790 2024-03-19T02:50:07.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.815+0000 7f942efbc740 1 Processor -- start 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.815+0000 7f942efbc740 1 -- start start 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.815+0000 7f941b7fe640 1 -- 172.21.15.116:0/3926464828 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9420010a90 con 0x7f9424101790 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d54e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d5d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9d60320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5615a9cb8500 con 0x5615a9d5df30 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615a9c60be0 con 0x5615a9d55350 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942efbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615a9cb45a0 con 0x5615a9d54a30 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942a96a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d54e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9d60320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942a96a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d54e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34888/0 (socket says 172.21.15.116:34888) 2024-03-19T02:50:07.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942a169640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d5d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942a96a640 1 -- 172.21.15.116:0/2429719184 learned_addr learned my addr 172.21.15.116:0/2429719184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f941b7fe640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 0x7f94100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 -- 172.21.15.116:0/2429719184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 msgr2=0x5615a9d54e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d54e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 -- 172.21.15.116:0/2429719184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 msgr2=0x5615a9d5d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d5d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 -- 172.21.15.116:0/2429719184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a9c82000 con 0x5615a9d5df30 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f942a96a640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 0x7f94100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f941b7fe640 1 -- 172.21.15.116:0/3926464828 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f94200cf540 con 0x7f9424101790 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.816+0000 7f9429968640 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9d60320 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f9414009230 tx=0x7f94140095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2c6c7156583a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f9418bfe640 1 -- 172.21.15.116:0/2429719184 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414045070 con 0x5615a9d5df30 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942a96a640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 0x7f94100a4e30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f941c009fd0 tx=0x7f941c004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f9418bfe640 1 -- 172.21.15.116:0/2429719184 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f941402c7e0 con 0x5615a9d5df30 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f9418bfe640 1 -- 172.21.15.116:0/2429719184 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414003a00 con 0x5615a9d5df30 2024-03-19T02:50:07.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 -- 172.21.15.116:0/2429719184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 msgr2=0x5615a9d60320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9d60320 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f9414009230 tx=0x7f94140095f0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 -- 172.21.15.116:0/2429719184 shutdown_connections 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9d60320 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d5d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 --2- 172.21.15.116:0/2429719184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d54e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 -- 172.21.15.116:0/2429719184 >> 172.21.15.116:0/2429719184 conn(0x5615a9d4f160 msgr2=0x5615a9d4f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 -- 172.21.15.116:0/2429719184 shutdown_connections 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.817+0000 7f942efbc740 1 -- 172.21.15.116:0/2429719184 wait complete. 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 -- start start 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 Processor -- start 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d5ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d5b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5615a9d60cc0 con 0x5615a9d55350 2024-03-19T02:50:07.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615a9c74160 con 0x5615a9d5df30 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5615a9d60e40 con 0x5615a9d54a30 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f9429968640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942a169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d5ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f9429968640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53994/0 (socket says 172.21.15.116:53994) 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942a169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d5ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34894/0 (socket says 172.21.15.116:34894) 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942a96a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d5b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f9429968640 1 -- 172.21.15.116:0/770007282 learned_addr learned my addr 172.21.15.116:0/770007282 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.818+0000 7f942efbc740 10 client.0 _ll_register_callbacks cb 0x5615a9cb0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 -- 172.21.15.116:0/770007282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 msgr2=0x5615a9d5ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d5ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 -- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 msgr2=0x5615a9d5b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d5b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 -- 172.21.15.116:0/770007282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a9a967d0 con 0x5615a9d55350 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f9429968640 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f9414006010 tx=0x7f9414005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dccd88808046a99f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f940dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414051070 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f941403fc80 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f941402c7e0 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f941405d050 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f942efbc740 10 client.21906 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5615a99cc3f0 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.819+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f941402c7e0 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f9414034440 con 0x5615a9d55350 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 10 client.21906 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 10 client.21906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 20 client.21906 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9d70380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 4 client.21906 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615a9cc5a40 con 0x5615a9d6df90 2024-03-19T02:50:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942a169640 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9d70380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f942a169640 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9d70380 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b98bfa8c43b5039 server_cookie=cfef0f80f00ce411 in_seq=0 out_seq=0 2024-03-19T02:50:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.820+0000 7f940dffb640 10 client.21906 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f940dffb640 1 -- 172.21.15.116:0/770007282 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615a9cc5a40 con 0x5615a9d6df90 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f940dffb640 10 client.21906 handle_command_reply: tid=0 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 client.21906 shutdown 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 msgr2=0x5615a9d5b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f9414006010 tx=0x7f9414005e50 comp rx=0 tx=0).stop 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 shutdown_connections 2024-03-19T02:50:07.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9d70380 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f940dffb640 0 client.21906 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d5b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d55350 0x5615a9d5b350 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 --2- 172.21.15.116:0/770007282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d5ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 >> 172.21.15.116:0/770007282 conn(0x5615a9d4f160 msgr2=0x5615a9d50eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 shutdown_connections 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 1 -- 172.21.15.116:0/770007282 wait complete. 2024-03-19T02:50:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.825+0000 7f942efbc740 20 client.21906 trim_cache size 0 max 16384 2024-03-19T02:50:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.847+0000 7f942efbc740 1 Processor -- start 2024-03-19T02:50:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.847+0000 7f942efbc740 1 -- start start 2024-03-19T02:50:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d55350 0x5615a9d69a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d50e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94241aedd0 con 0x5615a9d54a30 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b0940 con 0x5615a9d5df30 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942efbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94241af4a0 con 0x5615a9d55350 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54006/0 (socket says 172.21.15.116:54006) 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 -- 172.21.15.116:0/207174768 learned_addr learned my addr 172.21.15.116:0/207174768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f9429968640 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d55350 0x5615a9d69a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a96a640 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d50e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 -- 172.21.15.116:0/207174768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d55350 msgr2=0x5615a9d69a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d55350 0x5615a9d69a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 -- 172.21.15.116:0/207174768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 msgr2=0x5615a9d50e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d50e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 -- 172.21.15.116:0/207174768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a9d706f0 con 0x5615a9d54a30 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f942a169640 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f942000c6a0 tx=0x7f94200d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc62b502f768b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f940ffff640 1 -- 172.21.15.116:0/207174768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942009b5f0 con 0x5615a9d54a30 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f940ffff640 1 -- 172.21.15.116:0/207174768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9420010de0 con 0x5615a9d54a30 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.848+0000 7f940ffff640 1 -- 172.21.15.116:0/207174768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942009b180 con 0x5615a9d54a30 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 -- 172.21.15.116:0/207174768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 msgr2=0x5615a9d5d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f942000c6a0 tx=0x7f94200d1040 comp rx=0 tx=0).stop 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 -- 172.21.15.116:0/207174768 shutdown_connections 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d5df30 0x5615a9d50e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d55350 0x5615a9d69a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 --2- 172.21.15.116:0/207174768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d54a30 0x5615a9d5d540 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 -- 172.21.15.116:0/207174768 >> 172.21.15.116:0/207174768 conn(0x5615a9d4f160 msgr2=0x5615a9d59d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 -- 172.21.15.116:0/207174768 shutdown_connections 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.849+0000 7f942efbc740 1 -- 172.21.15.116:0/207174768 wait complete. 2024-03-19T02:50:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 -- start start 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 Processor -- start 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d58750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94241b0270 con 0x5615a9d5df30 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94241afb70 con 0x5615a9d55350 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94241acdd0 con 0x5615a9d54a30 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942efbc740 10 client.0 _ll_register_callbacks cb 0x7f94200d5060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:34924/0 (socket says 172.21.15.116:34924) 2024-03-19T02:50:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a96a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54010/0 (socket says 172.21.15.116:54010) 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 -- 172.21.15.116:0/1127156972 learned_addr learned my addr 172.21.15.116:0/1127156972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f9429968640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d58750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 -- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 msgr2=0x5615a9d58750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d58750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 -- 172.21.15.116:0/1127156972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 msgr2=0x5615a9dccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a169640 1 -- 172.21.15.116:0/1127156972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a998c0f0 con 0x5615a9d54a30 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.850+0000 7f942a96a640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942a169640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f94200cce60 tx=0x7f942009b290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ddec452c11497b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942009b360 con 0x5615a9d54a30 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5615a9cbfd90 con 0x5615a9d54a30 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9598+0+0 (secure 0 0 0) 0x7f9420097e90 con 0x5615a9d54a30 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f942009f8c0 con 0x5615a9d54a30 2024-03-19T02:50:07.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 10 client.31609 fetch_fsmap learned FSMap version 41 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5615a9dd47e0 con 0x5615a9d54a30 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5615a9cbfd90 con 0x5615a9d54a30 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f9420010430 con 0x5615a9d54a30 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 10 client.31609 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 10 client.31609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 20 client.31609 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9dd6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 4 client.31609 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.851+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5615a9c3d540 con 0x5615a9d6df90 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.852+0000 7f9429968640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9dd6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.852+0000 7f940dffb640 10 client.31609 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.852+0000 7f9429968640 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9dd6f20 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b9154df3673002a server_cookie=95f9dcc8b637127c in_seq=0 out_seq=0 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f940dffb640 1 -- 172.21.15.116:0/1127156972 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5615a9c3d540 con 0x5615a9d6df90 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f940dffb640 10 client.31609 handle_command_reply: tid=0 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 client.31609 shutdown 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 msgr2=0x5615a9d58210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f94200cce60 tx=0x7f942009b290 comp rx=0 tx=0).stop 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 shutdown_connections 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5615a9d6df90 0x5615a9dd6f20 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615a9d5df30 0x5615a9dccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615a9d55350 0x5615a9d58750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f940dffb640 0 client.31609 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 --2- 172.21.15.116:0/1127156972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615a9d54a30 0x5615a9d58210 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 >> 172.21.15.116:0/1127156972 conn(0x5615a9d4f160 msgr2=0x5615a9d5b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 shutdown_connections 2024-03-19T02:50:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 1 -- 172.21.15.116:0/1127156972 wait complete. 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942efbc740 20 client.31609 trim_cache size 0 max 16384 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (449 inodes in the stack)", 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "b34ecd1f-2e79-4d15-83d5-38fb33dde85e": { 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "b34ecd1f-2e79-4d15-83d5-38fb33dde85e", 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:50:07.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:50:07.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:50:07.862 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 msgr2=0x7f94100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 0x7f94100a4e30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f941c009fd0 tx=0x7f941c004a40 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 msgr2=0x7f9424198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9f0 tx=0x7f942000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 shutdown_connections 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f94100a2970 0x7f94100a4e30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94241077a0 0x7f942419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9424102150 0x7f94241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.856+0000 7f942c3cd640 1 --2- 172.21.15.116:0/3926464828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9424101790 0x7f9424198070 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.857+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 >> 172.21.15.116:0/3926464828 conn(0x7f94240fd280 msgr2=0x7f94240fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.857+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 shutdown_connections 2024-03-19T02:50:07.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:07.857+0000 7f942c3cd640 1 -- 172.21.15.116:0/3926464828 wait complete. 2024-03-19T02:50:07.871 INFO:tasks.cephfs.filesystem:scrub status for tag:b34ecd1f-2e79-4d15-83d5-38fb33dde85e - {'path': '/', 'tag': 'b34ecd1f-2e79-4d15-83d5-38fb33dde85e', 'options': 'recursive,force'} 2024-03-19T02:50:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:07 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:07 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:07 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:07 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:08.336 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62330 5.80 MB/sec execute 450 sec latency 317.869 ms 2024-03-19T02:50:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:07 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:07 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: pgmap v705: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:09.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:08 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: pgmap v705: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:08 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62421 5.80 MB/sec execute 451 sec latency 1085.078 ms 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: pgmap v705: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:09.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:09.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:08 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:10.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62504 5.80 MB/sec execute 452 sec latency 2085.107 ms 2024-03-19T02:50:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:10 smithi154 ceph-mon[35693]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:10 smithi116 ceph-mon[32263]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:11.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62593 5.79 MB/sec execute 453 sec latency 2606.784 ms 2024-03-19T02:50:11.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:10 smithi162 ceph-mon[35761]: pgmap v706: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 209 op/s 2024-03-19T02:50:12.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62704 5.78 MB/sec execute 454 sec latency 145.035 ms 2024-03-19T02:50:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:12 smithi154 ceph-mon[35693]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:12 smithi116 ceph-mon[32263]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:13.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62818 5.77 MB/sec execute 455 sec latency 148.144 ms 2024-03-19T02:50:13.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:12 smithi162 ceph-mon[35761]: pgmap v707: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:14.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 62936 5.76 MB/sec execute 456 sec latency 129.439 ms 2024-03-19T02:50:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:14 smithi154 ceph-mon[35693]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:14 smithi116 ceph-mon[32263]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:15.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63057 5.76 MB/sec execute 457 sec latency 141.691 ms 2024-03-19T02:50:15.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:14 smithi162 ceph-mon[35761]: pgmap v708: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-19T02:50:16.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63171 5.75 MB/sec execute 458 sec latency 147.522 ms 2024-03-19T02:50:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:16 smithi154 ceph-mon[35693]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:16 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:16 smithi154 ceph-mon[35693]: osdmap e129: 12 total, 12 up, 12 in 2024-03-19T02:50:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:16 smithi116 ceph-mon[32263]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:16 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:16 smithi116 ceph-mon[32263]: osdmap e129: 12 total, 12 up, 12 in 2024-03-19T02:50:17.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63275 5.74 MB/sec execute 459 sec latency 126.365 ms 2024-03-19T02:50:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:16 smithi162 ceph-mon[35761]: pgmap v709: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:16 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:50:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:16 smithi162 ceph-mon[35761]: osdmap e129: 12 total, 12 up, 12 in 2024-03-19T02:50:18.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63403 5.75 MB/sec execute 460 sec latency 137.640 ms 2024-03-19T02:50:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:18 smithi154 ceph-mon[35693]: pgmap v711: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:18 smithi116 ceph-mon[32263]: pgmap v711: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:19.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63534 5.76 MB/sec execute 461 sec latency 133.001 ms 2024-03-19T02:50:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:18 smithi162 ceph-mon[35761]: pgmap v711: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:19 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:19 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:19 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:19 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:20.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63668 5.77 MB/sec execute 462 sec latency 151.364 ms 2024-03-19T02:50:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:19 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:19 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:20 smithi154 ceph-mon[35693]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:20 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:20 smithi116 ceph-mon[32263]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:20 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:21.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63770 5.77 MB/sec execute 463 sec latency 149.134 ms 2024-03-19T02:50:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:20 smithi162 ceph-mon[35761]: pgmap v712: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-19T02:50:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:20 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:22.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 63885 5.78 MB/sec execute 464 sec latency 653.862 ms 2024-03-19T02:50:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:22 smithi154 ceph-mon[35693]: pgmap v713: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:22 smithi116 ceph-mon[32263]: pgmap v713: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:23.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64038 5.80 MB/sec execute 465 sec latency 1653.893 ms 2024-03-19T02:50:23.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:22 smithi162 ceph-mon[35761]: pgmap v713: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:24.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64162 5.81 MB/sec execute 466 sec latency 2653.923 ms 2024-03-19T02:50:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:24 smithi154 ceph-mon[35693]: pgmap v714: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:24 smithi116 ceph-mon[32263]: pgmap v714: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:25.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64288 5.82 MB/sec execute 467 sec latency 3653.952 ms 2024-03-19T02:50:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:24 smithi162 ceph-mon[35761]: pgmap v714: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 183 op/s 2024-03-19T02:50:26.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64372 5.82 MB/sec execute 468 sec latency 4166.826 ms 2024-03-19T02:50:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:26 smithi154 ceph-mon[35693]: pgmap v715: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 152 op/s 2024-03-19T02:50:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:26 smithi116 ceph-mon[32263]: pgmap v715: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 152 op/s 2024-03-19T02:50:27.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64392 5.80 MB/sec execute 469 sec latency 1061.936 ms 2024-03-19T02:50:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:26 smithi162 ceph-mon[35761]: pgmap v715: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 152 op/s 2024-03-19T02:50:28.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64437 5.80 MB/sec execute 470 sec latency 2061.961 ms 2024-03-19T02:50:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:28 smithi154 ceph-mon[35693]: pgmap v716: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 151 op/s 2024-03-19T02:50:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:28 smithi116 ceph-mon[32263]: pgmap v716: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 151 op/s 2024-03-19T02:50:29.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64485 5.80 MB/sec execute 471 sec latency 3061.986 ms 2024-03-19T02:50:29.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:28 smithi162 ceph-mon[35761]: pgmap v716: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 151 op/s 2024-03-19T02:50:30.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64562 5.80 MB/sec execute 472 sec latency 4062.023 ms 2024-03-19T02:50:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:30 smithi154 ceph-mon[35693]: pgmap v717: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-03-19T02:50:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:30 smithi116 ceph-mon[32263]: pgmap v717: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-03-19T02:50:31.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64654 5.81 MB/sec execute 473 sec latency 4578.392 ms 2024-03-19T02:50:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:30 smithi162 ceph-mon[35761]: pgmap v717: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 127 op/s 2024-03-19T02:50:32.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64754 5.81 MB/sec execute 474 sec latency 176.199 ms 2024-03-19T02:50:33.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:32 smithi154 ceph-mon[35693]: pgmap v718: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 164 op/s 2024-03-19T02:50:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:32 smithi116 ceph-mon[32263]: pgmap v718: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 164 op/s 2024-03-19T02:50:33.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64852 5.81 MB/sec execute 475 sec latency 137.892 ms 2024-03-19T02:50:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:32 smithi162 ceph-mon[35761]: pgmap v718: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.2 MiB/s wr, 164 op/s 2024-03-19T02:50:34.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:33 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:33 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:34.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 64933 5.80 MB/sec execute 476 sec latency 195.250 ms 2024-03-19T02:50:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:33 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:34 smithi154 ceph-mon[35693]: pgmap v719: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 90 op/s 2024-03-19T02:50:35.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:34 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:34 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:35.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:34 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:50:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:34 smithi116 ceph-mon[32263]: pgmap v719: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 90 op/s 2024-03-19T02:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:34 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:34 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:34 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:50:35.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65028 5.80 MB/sec execute 477 sec latency 203.297 ms 2024-03-19T02:50:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:34 smithi162 ceph-mon[35761]: pgmap v719: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 90 op/s 2024-03-19T02:50:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:34 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:34 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:34 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:50:36.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65108 5.79 MB/sec execute 478 sec latency 240.329 ms 2024-03-19T02:50:37.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:36 smithi154 ceph-mon[35693]: pgmap v720: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 149 op/s 2024-03-19T02:50:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:36 smithi116 ceph-mon[32263]: pgmap v720: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 149 op/s 2024-03-19T02:50:37.337 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65197 5.78 MB/sec execute 479 sec latency 196.712 ms 2024-03-19T02:50:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:36 smithi162 ceph-mon[35761]: pgmap v720: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.5 MiB/s wr, 149 op/s 2024-03-19T02:50:37.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:50:37.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:50:37.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 -- 172.21.15.116:0/520088363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 msgr2=0x7fc020102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:37.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 --2- 172.21.15.116:0/520088363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020102e00 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009900 tx=0x7fc01c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 -- 172.21.15.116:0/520088363 shutdown_connections 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 --2- 172.21.15.116:0/520088363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 --2- 172.21.15.116:0/520088363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc02010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 --2- 172.21.15.116:0/520088363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020102e00 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 -- 172.21.15.116:0/520088363 >> 172.21.15.116:0/520088363 conn(0x7fc0200774d0 msgr2=0x7fc0200feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 -- 172.21.15.116:0/520088363 shutdown_connections 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.994+0000 7fc02806c640 1 -- 172.21.15.116:0/520088363 wait complete. 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 Processor -- start 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 -- start start 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc0201984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:37.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc020111a90 con 0x7fc020102a20 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc020111910 con 0x7fc02010bdf0 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc02806c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc020111c10 con 0x7fc020103340 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc025e08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc025e08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49426/0 (socket says 172.21.15.116:49426) 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc025e08640 1 -- 172.21.15.116:0/727775930 learned_addr learned my addr 172.21.15.116:0/727775930 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.995+0000 7fc026609640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025607640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc0201984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 -- 172.21.15.116:0/727775930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 msgr2=0x7fc0201984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc0201984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 -- 172.21.15.116:0/727775930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 msgr2=0x7fc02019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 -- 172.21.15.116:0/727775930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02019cfa0 con 0x7fc020102a20 2024-03-19T02:50:37.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025607640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc0201984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc026609640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.996+0000 7fc025e08640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000c00 tx=0x7fc01c02c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.999+0000 7fc016ffd640 1 -- 172.21.15.116:0/727775930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c043020 con 0x7fc020102a20 2024-03-19T02:50:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.999+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02019d230 con 0x7fc020102a20 2024-03-19T02:50:38.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.999+0000 7fc016ffd640 1 -- 172.21.15.116:0/727775930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc01c02dbe0 con 0x7fc020102a20 2024-03-19T02:50:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.999+0000 7fc016ffd640 1 -- 172.21.15.116:0/727775930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c003970 con 0x7fc020102a20 2024-03-19T02:50:38.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:37.999+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0201005a0 con 0x7fc020102a20 2024-03-19T02:50:38.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.002+0000 7fc016ffd640 1 -- 172.21.15.116:0/727775930 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc01c046020 con 0x7fc020102a20 2024-03-19T02:50:38.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.002+0000 7fc02ac4b740 1 Processor -- start 2024-03-19T02:50:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 -- start start 2024-03-19T02:50:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b350a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3515b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f8b3465500 con 0x55f8b3513740 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f8b340dbe0 con 0x55f8b350a240 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f8b34615a0 con 0x55f8b350ab60 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc016ffd640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 0x7fc00c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc025e08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc026609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b350a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc025e08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50774/0 (socket says 172.21.15.116:50774) 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc026609640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b350a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33700/0 (socket says 172.21.15.116:33700) 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc025e08640 1 -- 172.21.15.116:0/3441616136 learned_addr learned my addr 172.21.15.116:0/3441616136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc016ffd640 1 -- 172.21.15.116:0/727775930 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fc01c0f0fe0 con 0x7fc020102a20 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.003+0000 7fc026609640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 0x7fc00c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 -- 172.21.15.116:0/3441616136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 msgr2=0x55f8b350a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b350a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 -- 172.21.15.116:0/3441616136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 msgr2=0x55f8b3515b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3515b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 -- 172.21.15.116:0/3441616136 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b342f000 con 0x55f8b350ab60 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc025e08640 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc01c005ec0 tx=0x7fc01c0b9760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b79144b2e6283e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc00bfff640 1 -- 172.21.15.116:0/3441616136 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0bdb00 con 0x55f8b350ab60 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc00bfff640 1 -- 172.21.15.116:0/3441616136 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc01c0bd400 con 0x55f8b350ab60 2024-03-19T02:50:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.004+0000 7fc00bfff640 1 -- 172.21.15.116:0/3441616136 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0ee720 con 0x55f8b350ab60 2024-03-19T02:50:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3441616136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 msgr2=0x55f8b3513200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc026609640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 0x7fc00c0a4e30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc0180099e0 tx=0x7fc018009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc01c005ec0 tx=0x7fc01c0b9760 comp rx=0 tx=0).stop 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3441616136 shutdown_connections 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3515b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3513200 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3441616136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b350a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.005+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3441616136 >> 172.21.15.116:0/3441616136 conn(0x55f8b3504970 msgr2=0x55f8b3504d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.006+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3441616136 shutdown_connections 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.006+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3441616136 wait complete. 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.006+0000 7fc02ac4b740 1 -- start start 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.006+0000 7fc02ac4b740 1 Processor -- start 2024-03-19T02:50:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3511210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025e08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025e08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33712/0 (socket says 172.21.15.116:33712) 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50782/0 (socket says 172.21.15.116:50782) 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3511210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3511210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49440/0 (socket says 172.21.15.116:49440) 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc026609640 1 -- 172.21.15.116:0/3726809235 learned_addr learned my addr 172.21.15.116:0/3726809235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f8b35162f0 con 0x55f8b3513740 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f8b3421160 con 0x55f8b350a240 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f8b3516470 con 0x55f8b350ab60 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc02ac4b740 10 client.0 _ll_register_callbacks cb 0x55f8b345da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 -- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 msgr2=0x55f8b3510790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 -- 172.21.15.116:0/3726809235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 msgr2=0x55f8b3511210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3511210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025607640 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8b32437d0 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.007+0000 7fc025e08640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc025607640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc02010e220 tx=0x7fc010011e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67fe954955fd3504 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc0097fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01000fbd0 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fc01000fd70 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc01c0bd400 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc010024050 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc02ac4b740 10 client.31621 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f8b31793f0 con 0x55f8b350ab60 2024-03-19T02:50:38.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.008+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc01c0bd400 con 0x55f8b350ab60 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc0100184d0 con 0x55f8b350ab60 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 10 client.31621 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 10 client.31621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 20 client.31621 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b3523cf0 0x55f8b3526150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 4 client.31621 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8b3472a40 con 0x55f8b3523cf0 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.009+0000 7fc025e08640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b3523cf0 0x55f8b3526150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.010+0000 7fc0097fa640 10 client.31621 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:38.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.010+0000 7fc025e08640 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b3523cf0 0x55f8b3526150 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abddaef21e9d256f server_cookie=2ac3cd4f0ea97500 in_seq=0 out_seq=0 2024-03-19T02:50:38.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.011+0000 7fc0097fa640 1 -- 172.21.15.116:0/3726809235 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8b3472a40 con 0x55f8b3523cf0 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.011+0000 7fc0097fa640 10 client.31621 handle_command_reply: tid=0 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 client.31621 shutdown 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 msgr2=0x55f8b3510cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc02010e220 tx=0x7fc010011e40 comp rx=0 tx=0).stop 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 shutdown_connections 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b3523cf0 0x55f8b3526150 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3511210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3510cd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3726809235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3510790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 >> 172.21.15.116:0/3726809235 conn(0x55f8b3504970 msgr2=0x55f8b35066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc0097fa640 0 client.31621 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 shutdown_connections 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3726809235 wait complete. 2024-03-19T02:50:38.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.012+0000 7fc02ac4b740 20 client.31621 trim_cache size 0 max 16384 2024-03-19T02:50:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 Processor -- start 2024-03-19T02:50:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 -- start start 2024-03-19T02:50:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 0x55f8b3512ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350ab60 0x55f8b34fb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201aca20 con 0x55f8b3513740 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201ae560 con 0x55f8b350ab60 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201ad0f0 con 0x55f8b350a240 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc025e08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 0x55f8b3512ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49446/0 (socket says 172.21.15.116:49446) 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc026609640 1 -- 172.21.15.116:0/496008291 learned_addr learned my addr 172.21.15.116:0/496008291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.030+0000 7fc025607640 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350ab60 0x55f8b34fb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 -- 172.21.15.116:0/496008291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 msgr2=0x55f8b3512ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 0x55f8b3512ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 -- 172.21.15.116:0/496008291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350ab60 msgr2=0x55f8b34fb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350ab60 0x55f8b34fb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 -- 172.21.15.116:0/496008291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b31390f0 con 0x55f8b3513740 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc025e08640 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 0x55f8b3512ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc026609640 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fc01800fe30 tx=0x7fc018005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be14aadb056a4478 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc00b7fe640 1 -- 172.21.15.116:0/496008291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01801ed90 con 0x55f8b3513740 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc00b7fe640 1 -- 172.21.15.116:0/496008291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc018003e20 con 0x55f8b3513740 2024-03-19T02:50:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.031+0000 7fc00b7fe640 1 -- 172.21.15.116:0/496008291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01802a6c0 con 0x55f8b3513740 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- 172.21.15.116:0/496008291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 msgr2=0x55f8b352ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fc01800fe30 tx=0x7fc018005bd0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- 172.21.15.116:0/496008291 shutdown_connections 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b352ba60 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350ab60 0x55f8b34fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/496008291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350a240 0x55f8b3512ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- 172.21.15.116:0/496008291 >> 172.21.15.116:0/496008291 conn(0x55f8b3504970 msgr2=0x55f8b34fa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- 172.21.15.116:0/496008291 shutdown_connections 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- 172.21.15.116:0/496008291 wait complete. 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 -- start start 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.032+0000 7fc02ac4b740 1 Processor -- start 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3582940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3516410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201ade90 con 0x55f8b3513740 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201ad7c0 con 0x55f8b350a240 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc0201aaa20 con 0x55f8b350ab60 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3516410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025e08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3582940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc02ac4b740 10 client.0 _ll_register_callbacks cb 0x7fc010009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025e08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33728/0 (socket says 172.21.15.116:33728) 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025e08640 1 -- 172.21.15.116:0/3671324338 learned_addr learned my addr 172.21.15.116:0/3671324338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc026609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3516410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49462/0 (socket says 172.21.15.116:49462) 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 -- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 msgr2=0x55f8b3582400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 -- 172.21.15.116:0/3671324338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 msgr2=0x55f8b3516410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3516410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025607640 1 -- 172.21.15.116:0/3671324338 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8b3516b00 con 0x55f8b350ab60 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.033+0000 7fc025e08640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc025607640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3582940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc0100188f0 tx=0x7fc01000fe00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f590f44828fcf4a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01002c020 con 0x55f8b350ab60 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fc0100183b0 con 0x55f8b350ab60 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc018003e20 con 0x55f8b350ab60 2024-03-19T02:50:38.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc010037050 con 0x55f8b350ab60 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 10 client.31627 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f8b3589b60 con 0x55f8b350ab60 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc018003e20 con 0x55f8b350ab60 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc0100035f0 con 0x55f8b350ab60 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 10 client.31627 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 10 client.31627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.034+0000 7fc02ac4b740 20 client.31627 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b358a360 0x55f8b358c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc02ac4b740 4 client.31627 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8b346cd90 con 0x55f8b358a360 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc025e08640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b358a360 0x55f8b358c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc0097fa640 10 client.31627 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.035+0000 7fc025e08640 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b358a360 0x55f8b358c780 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db787646155e046 server_cookie=326b0079d42d4848 in_seq=0 out_seq=0 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.036+0000 7fc0097fa640 1 -- 172.21.15.116:0/3671324338 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f8b346cd90 con 0x55f8b358a360 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.036+0000 7fc0097fa640 10 client.31627 handle_command_reply: tid=0 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.036+0000 7fc02ac4b740 1 client.31627 shutdown 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 msgr2=0x55f8b3582940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3582940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc0100188f0 tx=0x7fc01000fe00 comp rx=0 tx=0).stop 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 shutdown_connections 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f8b358a360 0x55f8b358c780 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8b3513740 0x55f8b3516410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f8b350ab60 0x55f8b3582940 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 --2- 172.21.15.116:0/3671324338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f8b350a240 0x55f8b3582400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 >> 172.21.15.116:0/3671324338 conn(0x55f8b3504970 msgr2=0x55f8b3510fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc0097fa640 0 client.31627 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:38.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 shutdown_connections 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 1 -- 172.21.15.116:0/3671324338 wait complete. 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02ac4b740 20 client.31627 trim_cache size 0 max 16384 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:50:38.044 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 msgr2=0x7fc00c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 0x7fc00c0a4e30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc0180099e0 tx=0x7fc018009500 comp rx=0 tx=0).stop 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 msgr2=0x7fc020197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000c00 tx=0x7fc01c02c6d0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 shutdown_connections 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc00c0a2970 0x7fc00c0a4e30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc02010bdf0 0x7fc02019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc020103340 0x7fc0201984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 --2- 172.21.15.116:0/727775930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc020102a20 0x7fc020197fb0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 >> 172.21.15.116:0/727775930 conn(0x7fc0200774d0 msgr2=0x7fc0200fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.037+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 shutdown_connections 2024-03-19T02:50:38.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.038+0000 7fc02806c640 1 -- 172.21.15.116:0/727775930 wait complete. 2024-03-19T02:50:38.050 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:50:38.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 -- 172.21.15.116:0/757876157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 msgr2=0x7fa5b010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/757876157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b010e680 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009cd0 tx=0x7fa5a802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 -- 172.21.15.116:0/757876157 shutdown_connections 2024-03-19T02:50:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/757876157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b010e680 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/757876157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 0x7fa5b0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/757876157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 0x7fa5b0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 -- 172.21.15.116:0/757876157 >> 172.21.15.116:0/757876157 conn(0x7fa5b00fd2c0 msgr2=0x7fa5b00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 -- 172.21.15.116:0/757876157 shutdown_connections 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 -- 172.21.15.116:0/757876157 wait complete. 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.174+0000 7fa5b58c6640 1 Processor -- start 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 -- start start 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 0x7fa5b0073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 0x7fa5b00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111a80 con 0x7fa5b0102170 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111900 con 0x7fa5b01017b0 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b58c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111c00 con 0x7fa5b01077c0 2024-03-19T02:50:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49484/0 (socket says 172.21.15.116:49484) 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 -- 172.21.15.116:0/2229760766 learned_addr learned my addr 172.21.15.116:0/2229760766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5aeffd640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 0x7fa5b0073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5af7fe640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 0x7fa5b00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 -- 172.21.15.116:0/2229760766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 msgr2=0x7fa5b00745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 0x7fa5b00745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 -- 172.21.15.116:0/2229760766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 msgr2=0x7fa5b0073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 0x7fa5b0073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 -- 172.21.15.116:0/2229760766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b01a64e0 con 0x7fa5b0102170 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5ae7fc640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400e9f0 tx=0x7fa5a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.175+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a400cd80 con 0x7fa5b0102170 2024-03-19T02:50:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.176+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b01a67d0 con 0x7fa5b0102170 2024-03-19T02:50:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.176+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5a40108e0 con 0x7fa5b0102170 2024-03-19T02:50:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.176+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4018920 con 0x7fa5b0102170 2024-03-19T02:50:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.176+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a6d70 con 0x7fa5b0102170 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.177+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b0068610 con 0x7fa5b0102170 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.178+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa5a4010a90 con 0x7fa5b0102170 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.178+0000 7fa5b48c4640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 0x7fa5980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.178+0000 7fa5aeffd640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 0x7fa5980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.178+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fa5a40cf4c0 con 0x7fa5b0102170 2024-03-19T02:50:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.179+0000 7fa5aeffd640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 0x7fa5980a4e30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009b50 tx=0x7fa5a0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.183+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5a409b890 con 0x7fa5b0102170 2024-03-19T02:50:38.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65281 5.78 MB/sec execute 480 sec latency 237.694 ms 2024-03-19T02:50:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.479+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5b0109d20 con 0x7fa5b0102170 2024-03-19T02:50:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.480+0000 7fa5b48c4640 1 -- 172.21.15.116:0/2229760766 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11950 (secure 0 0 0) 0x7fa5a409f930 con 0x7fa5b0102170 2024-03-19T02:50:38.481 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:50:38.481 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:50:15.839334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:50:38.482 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 msgr2=0x7fa5980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 0x7fa5980a4e30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009b50 tx=0x7fa5a0005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 msgr2=0x7fa5b0074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400e9f0 tx=0x7fa5a400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 shutdown_connections 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa5980a2970 0x7fa5980a4e30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5b01077c0 0x7fa5b00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b0102170 0x7fa5b0074060 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 --2- 172.21.15.116:0/2229760766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5b01017b0 0x7fa5b0073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 >> 172.21.15.116:0/2229760766 conn(0x7fa5b00fd2c0 msgr2=0x7fa5b00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 shutdown_connections 2024-03-19T02:50:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.483+0000 7fa5b58c6640 1 -- 172.21.15.116:0/2229760766 wait complete. 2024-03-19T02:50:38.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:50:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 -- 172.21.15.116:0/2899460277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44100970 msgr2=0x7fac44100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 --2- 172.21.15.116:0/2899460277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44100970 0x7fac44100df0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009900 tx=0x7fac2c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 -- 172.21.15.116:0/2899460277 shutdown_connections 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 --2- 172.21.15.116:0/2899460277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44101330 0x7fac4410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 --2- 172.21.15.116:0/2899460277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44100970 0x7fac44100df0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 --2- 172.21.15.116:0/2899460277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44105660 0x7fac44105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 -- 172.21.15.116:0/2899460277 >> 172.21.15.116:0/2899460277 conn(0x7fac440774c0 msgr2=0x7fac440778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 -- 172.21.15.116:0/2899460277 shutdown_connections 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 -- 172.21.15.116:0/2899460277 wait complete. 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.627+0000 7fac4926e640 1 Processor -- start 2024-03-19T02:50:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 -- start start 2024-03-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 0x7fac4419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac44110930 con 0x7fac44101330 2024-03-19T02:50:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac441107b0 con 0x7fac44105660 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac4926e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac44110ab0 con 0x7fac44100970 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49496/0 (socket says 172.21.15.116:49496) 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 0x7fac4419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1132731758 learned_addr learned my addr 172.21.15.116:0/1132731758 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50842/0 (socket says 172.21.15.116:50842) 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1132731758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 msgr2=0x7fac44197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1132731758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 msgr2=0x7fac4419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 0x7fac4419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1132731758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4419cfc0 con 0x7fac44101330 2024-03-19T02:50:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac42ffd640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.628+0000 7fac437fe640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 0x7fac4419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fac2c005940 tx=0x7fac2c02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac40ff9640 1 -- 172.21.15.116:0/1132731758 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c043020 con 0x7fac44101330 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4419d250 con 0x7fac44101330 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac40ff9640 1 -- 172.21.15.116:0/1132731758 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c02ca70 con 0x7fac44101330 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac40ff9640 1 -- 172.21.15.116:0/1132731758 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c003a80 con 0x7fac44101330 2024-03-19T02:50:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.629+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac441a4b40 con 0x7fac44101330 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 Processor -- start 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac40ff9640 1 -- 172.21.15.116:0/1132731758 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fac2c046020 con 0x7fac44101330 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 -- start start 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdb77d0 0x56239cdb9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56239cd114b0 con 0x56239cdaebf0 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56239ccb9d70 con 0x56239cdae2d0 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac4be56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56239cd0d550 con 0x56239cdb77d0 2024-03-19T02:50:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49498/0 (socket says 172.21.15.116:49498) 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33782/0 (socket says 172.21.15.116:33782) 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdb77d0 0x56239cdb9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.632+0000 7fac437fe640 1 -- 172.21.15.116:0/2323581083 learned_addr learned my addr 172.21.15.116:0/2323581083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 -- 172.21.15.116:0/2323581083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdb77d0 msgr2=0x56239cdb9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdb77d0 0x56239cdb9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 -- 172.21.15.116:0/2323581083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 msgr2=0x56239cdb7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 -- 172.21.15.116:0/2323581083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56239ccdc210 con 0x56239cdae2d0 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac40ff9640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 0x7fac300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac437fe640 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fac3c011540 tx=0x7fac3c011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6070d9179fecc96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac40ff9640 1 -- 172.21.15.116:0/1132731758 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fac2c0f1160 con 0x7fac44101330 2024-03-19T02:50:38.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac42ffd640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 0x7fac300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac397fa640 1 -- 172.21.15.116:0/2323581083 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00f5c0 con 0x56239cdae2d0 2024-03-19T02:50:38.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac397fa640 1 -- 172.21.15.116:0/2323581083 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac3c00fc10 con 0x56239cdae2d0 2024-03-19T02:50:38.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.633+0000 7fac397fa640 1 -- 172.21.15.116:0/2323581083 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c019c30 con 0x56239cdae2d0 2024-03-19T02:50:38.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac42ffd640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 0x7fac300a4e30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56239cdadda0 tx=0x7fac34009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 -- 172.21.15.116:0/2323581083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 msgr2=0x56239cdae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fac3c011540 tx=0x7fac3c011a00 comp rx=0 tx=0).stop 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 -- 172.21.15.116:0/2323581083 shutdown_connections 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdb77d0 0x56239cdb9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 --2- 172.21.15.116:0/2323581083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdae6b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 -- 172.21.15.116:0/2323581083 >> 172.21.15.116:0/2323581083 conn(0x56239cda8a00 msgr2=0x56239cda8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 -- 172.21.15.116:0/2323581083 shutdown_connections 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.634+0000 7fac4be56740 1 -- 172.21.15.116:0/2323581083 wait complete. 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 -- start start 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 Processor -- start 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdb4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 0x56239cdb52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56239cdba5c0 con 0x56239cdb77d0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56239cccd310 con 0x56239cdae2d0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56239cdba740 con 0x56239cdaebf0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac4be56740 10 client.0 _ll_register_callbacks cb 0x56239ccc12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 0x56239cdb52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdb4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50866/0 (socket says 172.21.15.116:50866) 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdb4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33792/0 (socket says 172.21.15.116:33792) 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.635+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1191511329 learned_addr learned my addr 172.21.15.116:0/1191511329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1191511329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 msgr2=0x56239cdb4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdb4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1191511329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 msgr2=0x56239cdb52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 0x56239cdb52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 -- 172.21.15.116:0/1191511329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56239ca253f0 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac437fe640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 0x56239cdb52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0bd1f0 tx=0x7fac2c0f1460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31955271be846c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0bd2c0 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fac2c0bd750 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac3c00fc10 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fac2c0f4e60 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac4be56740 10 client.31645 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56239caef7d0 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.636+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac3c00fc10 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fac2c0c5800 con 0x56239cdaebf0 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 10 client.31645 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 10 client.31645 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 20 client.31645 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239cdca000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 4 client.31645 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56239cd6f3e0 con 0x56239cdc7c10 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac42ffd640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239cdca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac2a7fc640 10 client.31645 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.637+0000 7fac42ffd640 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239cdca000 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6e3a71ba4518c713 server_cookie=e14264621692e386 in_seq=0 out_seq=0 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.638+0000 7fac2a7fc640 1 -- 172.21.15.116:0/1191511329 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56239cd6f3e0 con 0x56239cdc7c10 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.638+0000 7fac2a7fc640 10 client.31645 handle_command_reply: tid=0 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.638+0000 7fac4be56740 1 client.31645 shutdown 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 msgr2=0x56239cdb4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0bd1f0 tx=0x7fac2c0f1460 comp rx=0 tx=0).stop 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 shutdown_connections 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239cdca000 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdb77d0 0x56239cdb52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac2a7fc640 0 client.31645 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdaebf0 0x56239cdb4d70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 --2- 172.21.15.116:0/1191511329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdae2d0 0x56239cdb4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 >> 172.21.15.116:0/1191511329 conn(0x56239cda8a00 msgr2=0x56239cdaa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 shutdown_connections 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 1 -- 172.21.15.116:0/1191511329 wait complete. 2024-03-19T02:50:38.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.639+0000 7fac4be56740 20 client.31645 trim_cache size 0 max 16384 2024-03-19T02:50:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.656+0000 7fac4be56740 1 Processor -- start 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.656+0000 7fac4be56740 1 -- start start 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdcf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac441aeba0 con 0x56239cdaebf0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac441b06e0 con 0x56239cdb77d0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac4be56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac441af270 con 0x56239cdae2d0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49512/0 (socket says 172.21.15.116:49512) 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4064052495 learned_addr learned my addr 172.21.15.116:0/4064052495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac42ffd640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac437fe640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdcf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4064052495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 msgr2=0x56239cdb6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4064052495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 msgr2=0x56239cdcf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdcf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4064052495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56239cd7e880 con 0x56239cdaebf0 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac42ffd640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8750e0ed007ca1da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.657+0000 7fac38ff9640 1 -- 172.21.15.116:0/4064052495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0bce50 con 0x56239cdaebf0 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac38ff9640 1 -- 172.21.15.116:0/4064052495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c02cc20 con 0x56239cdaebf0 2024-03-19T02:50:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac38ff9640 1 -- 172.21.15.116:0/4064052495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0bd030 con 0x56239cdaebf0 2024-03-19T02:50:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 -- 172.21.15.116:0/4064052495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 msgr2=0x56239cdc3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000c00 tx=0x7fac2c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 -- 172.21.15.116:0/4064052495 shutdown_connections 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdcf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdc3360 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 --2- 172.21.15.116:0/4064052495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.658+0000 7fac4be56740 1 -- 172.21.15.116:0/4064052495 >> 172.21.15.116:0/4064052495 conn(0x56239cda8a00 msgr2=0x56239cdaa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- 172.21.15.116:0/4064052495 shutdown_connections 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- 172.21.15.116:0/4064052495 wait complete. 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- start start 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 Processor -- start 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdb1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac441b0010 con 0x56239cdaebf0 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac441af940 con 0x56239cdb77d0 2024-03-19T02:50:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac441acba0 con 0x56239cdae2d0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac4be56740 10 client.0 _ll_register_callbacks cb 0x7fac2c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac3a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49516/0 (socket says 172.21.15.116:49516) 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4036372424 learned_addr learned my addr 172.21.15.116:0/4036372424 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac42ffd640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.659+0000 7fac437fe640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdb1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 msgr2=0x56239cdb14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4036372424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 msgr2=0x56239cdb1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdb1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 -- 172.21.15.116:0/4036372424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56239ce2cf50 con 0x56239cdaebf0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac3a7fc640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b9ec0 tx=0x7fac2c0bda00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78269b807cbd7822 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0bd030 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fac2c0c13c0 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac2c02cc20 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fac2c0bd2c0 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac4be56740 10 client.21951 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56239ce2cf50 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.660+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c02cc20 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fac2c0d60d0 con 0x56239cdaebf0 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 10 client.21951 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 10 client.21951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 20 client.21951 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239ce30530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 4 client.21951 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:38.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56239cd1e9f0 con 0x56239cdc7c10 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac42ffd640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239ce30530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac42ffd640 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239ce30530 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27fb8fe6c95b3870 server_cookie=8cee2e209e7652ff in_seq=0 out_seq=0 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac2a7fc640 10 client.21951 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac2a7fc640 1 -- 172.21.15.116:0/4036372424 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56239cd1e9f0 con 0x56239cdc7c10 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac2a7fc640 10 client.21951 handle_command_reply: tid=0 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.661+0000 7fac4be56740 1 client.21951 shutdown 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 msgr2=0x56239cdb1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0b9ec0 tx=0x7fac2c0bda00 comp rx=0 tx=0).stop 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 shutdown_connections 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x56239cdc7c10 0x56239ce30530 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56239cdb77d0 0x56239cdb1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56239cdaebf0 0x56239cdb1a00 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac2a7fc640 0 client.21951 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 --2- 172.21.15.116:0/4036372424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56239cdae2d0 0x56239cdb14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 >> 172.21.15.116:0/4036372424 conn(0x56239cda8a00 msgr2=0x56239cda7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 shutdown_connections 2024-03-19T02:50:38.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 1 -- 172.21.15.116:0/4036372424 wait complete. 2024-03-19T02:50:38.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4be56740 20 client.21951 trim_cache size 0 max 16384 2024-03-19T02:50:38.669 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 msgr2=0x7fac300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 0x7fac300a4e30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56239cdadda0 tx=0x7fac34009500 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 msgr2=0x7fac44198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fac2c005940 tx=0x7fac2c02c5f0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 shutdown_connections 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fac300a2970 0x7fac300a4e30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac44105660 0x7fac4419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac44101330 0x7fac44198510 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 --2- 172.21.15.116:0/1132731758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac44100970 0x7fac44197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.662+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 >> 172.21.15.116:0/1132731758 conn(0x7fac440774c0 msgr2=0x7fac4406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.663+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 shutdown_connections 2024-03-19T02:50:38.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.663+0000 7fac4926e640 1 -- 172.21.15.116:0/1132731758 wait complete. 2024-03-19T02:50:38.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:50:38.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 -- 172.21.15.116:0/3915496174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904100a00 msgr2=0x7f5904100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 --2- 172.21.15.116:0/3915496174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904100a00 0x7f5904100e80 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009900 tx=0x7f58f402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:38.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 -- 172.21.15.116:0/3915496174 shutdown_connections 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 --2- 172.21.15.116:0/3915496174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59041013c0 0x7f590410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 --2- 172.21.15.116:0/3915496174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904100a00 0x7f5904100e80 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 --2- 172.21.15.116:0/3915496174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5904105680 0x7f5904105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 -- 172.21.15.116:0/3915496174 >> 172.21.15.116:0/3915496174 conn(0x7f5904077700 msgr2=0x7f5904077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 -- 172.21.15.116:0/3915496174 shutdown_connections 2024-03-19T02:50:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.809+0000 7f590b77e640 1 -- 172.21.15.116:0/3915496174 wait complete. 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 Processor -- start 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 -- start start 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59041013c0 0x7f59041992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904105680 0x7f59041a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5904110970 con 0x7f5904105680 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59041107f0 con 0x7f59041013c0 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590b77e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5904110af0 con 0x7f5904100a00 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590951a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f5909d1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904105680 0x7f59041a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590951a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50900/0 (socket says 172.21.15.116:50900) 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f590951a640 1 -- 172.21.15.116:0/3907244775 learned_addr learned my addr 172.21.15.116:0/3907244775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.810+0000 7f5908d19640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59041013c0 0x7f59041992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 -- 172.21.15.116:0/3907244775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59041013c0 msgr2=0x7f59041992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59041013c0 0x7f59041992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 -- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904105680 msgr2=0x7f59041a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904105680 0x7f59041a3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 -- 172.21.15.116:0/3907244775 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59041a4330 con 0x7f5904100a00 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590951a640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f590000e9c0 tx=0x7f590000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f58fa7fc640 1 -- 172.21.15.116:0/3907244775 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590000cd80 con 0x7f5904100a00 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5904196b60 con 0x7f5904100a00 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f58fa7fc640 1 -- 172.21.15.116:0/3907244775 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59000035f0 con 0x7f5904100a00 2024-03-19T02:50:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f58fa7fc640 1 -- 172.21.15.116:0/3907244775 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900010690 con 0x7f5904100a00 2024-03-19T02:50:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.811+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5904197030 con 0x7f5904100a00 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.814+0000 7f58fa7fc640 1 -- 172.21.15.116:0/3907244775 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f59000037a0 con 0x7f5904100a00 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.814+0000 7f590e36c740 1 Processor -- start 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.814+0000 7f590e36c740 1 -- start start 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4054f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 0x55bb40558290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb4055abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb404b24b0 con 0x55bb4054fbf0 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb4045ad70 con 0x55bb405587d0 2024-03-19T02:50:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590e36c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb404ae550 con 0x55bb4054f2d0 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f58fa7fc640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 0x7f58f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f5908d19640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb4055abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f5909d1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4054f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f5909d1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4054f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50916/0 (socket says 172.21.15.116:50916) 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f5909d1b640 1 -- 172.21.15.116:0/2663550863 learned_addr learned my addr 172.21.15.116:0/2663550863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 0x55bb40558290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f5909d1b640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 0x7f58f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f58fa7fc640 1 -- 172.21.15.116:0/3907244775 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f59000cf720 con 0x7f5904100a00 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 -- 172.21.15.116:0/2663550863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 msgr2=0x55bb4054f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4054f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 -- 172.21.15.116:0/2663550863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 msgr2=0x55bb4055abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb4055abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.815+0000 7f590951a640 1 -- 172.21.15.116:0/2663550863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb4047d210 con 0x55bb4054fbf0 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.816+0000 7f590951a640 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 0x55bb40558290 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f59000cca90 tx=0x7f590009b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05416eea00e0236 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.816+0000 7f5909d1b640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 0x7f58f00a4e30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0099e0 tx=0x7f58fc009c80 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.816+0000 7f58ef7fe640 1 -- 172.21.15.116:0/2663550863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900003b10 con 0x55bb4054fbf0 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.816+0000 7f58ef7fe640 1 -- 172.21.15.116:0/2663550863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f590009c0a0 con 0x55bb4054fbf0 2024-03-19T02:50:38.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.816+0000 7f58ef7fe640 1 -- 172.21.15.116:0/2663550863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000ccd10 con 0x55bb4054fbf0 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- 172.21.15.116:0/2663550863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 msgr2=0x55bb40558290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 0x55bb40558290 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f59000cca90 tx=0x7f590009b4e0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- 172.21.15.116:0/2663550863 shutdown_connections 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb4055abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054fbf0 0x55bb40558290 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 --2- 172.21.15.116:0/2663550863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4054f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- 172.21.15.116:0/2663550863 >> 172.21.15.116:0/2663550863 conn(0x55bb40549a00 msgr2=0x55bb40549de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- 172.21.15.116:0/2663550863 shutdown_connections 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- 172.21.15.116:0/2663550863 wait complete. 2024-03-19T02:50:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 -- start start 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.817+0000 7f590e36c740 1 Processor -- start 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054fbf0 0x55bb40555d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb405562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bb4055b5c0 con 0x55bb4054f2d0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bb4046e310 con 0x55bb405587d0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bb4055b740 con 0x55bb4054fbf0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590e36c740 10 client.0 _ll_register_callbacks cb 0x55bb404622d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49544/0 (socket says 172.21.15.116:49544) 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 -- 172.21.15.116:0/3240044230 learned_addr learned my addr 172.21.15.116:0/3240044230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f5909d1b640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb405562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f5908d19640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054fbf0 0x55bb40555d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 -- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054fbf0 msgr2=0x55bb40555d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054fbf0 0x55bb40555d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 -- 172.21.15.116:0/3240044230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 msgr2=0x55bb405562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb405562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.818+0000 7f590951a640 1 -- 172.21.15.116:0/3240044230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb401c63f0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f590951a640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f590000cfd0 tx=0x7f590000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330efaececbc872e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000b3020 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f590009bc00 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590009c0a0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f590009b9a0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f590e36c740 10 client.21963 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bb402907d0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f590009c0a0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.819+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f59000a3cb0 con 0x55bb4054f2d0 2024-03-19T02:50:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 10 client.21963 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 10 client.21963 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 20 client.21963 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb4056adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 4 client.21963 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb405103e0 con 0x55bb40568a00 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f5908d19640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb4056adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f5908d19640 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb4056adf0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1fa33b912cab0eaa server_cookie=5559a2f90a4fe51f in_seq=0 out_seq=0 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.820+0000 7f58ecff9640 10 client.21963 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f58ecff9640 1 -- 172.21.15.116:0/3240044230 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb405103e0 con 0x55bb40568a00 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f58ecff9640 10 client.21963 handle_command_reply: tid=0 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 client.21963 shutdown 2024-03-19T02:50:38.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 msgr2=0x55bb40555820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f590000cfd0 tx=0x7f590000c840 comp rx=0 tx=0).stop 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 shutdown_connections 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb405587d0 0x55bb405562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054fbf0 0x55bb40555d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb4056adf0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f58ecff9640 0 client.21963 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 --2- 172.21.15.116:0/3240044230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb4054f2d0 0x55bb40555820 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.821+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 >> 172.21.15.116:0/3240044230 conn(0x55bb40549a00 msgr2=0x55bb4054b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.822+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 shutdown_connections 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.822+0000 7f590e36c740 1 -- 172.21.15.116:0/3240044230 wait complete. 2024-03-19T02:50:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.822+0000 7f590e36c740 20 client.21963 trim_cache size 0 max 16384 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 Processor -- start 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 -- start start 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb40557f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb405560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59041ac920 con 0x55bb405587d0 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59041ae460 con 0x55bb4054fbf0 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.846+0000 7f590e36c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59041acff0 con 0x55bb4054f2d0 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f590951a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb40557f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49550/0 (socket says 172.21.15.116:49550) 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 -- 172.21.15.116:0/2723104782 learned_addr learned my addr 172.21.15.116:0/2723104782 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5908d19640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb405560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 -- 172.21.15.116:0/2723104782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 msgr2=0x55bb40557f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb40557f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 -- 172.21.15.116:0/2723104782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 msgr2=0x55bb405560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb405560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 -- 172.21.15.116:0/2723104782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb40550030 con 0x55bb405587d0 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f590951a640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb40557f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5908d19640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb405560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f5909d1b640 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0206a0 tx=0x7f58fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a0c97a1d94f842 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f58eeffd640 1 -- 172.21.15.116:0/2723104782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc026040 con 0x55bb405587d0 2024-03-19T02:50:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f58eeffd640 1 -- 172.21.15.116:0/2723104782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58fc003970 con 0x55bb405587d0 2024-03-19T02:50:38.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.847+0000 7f58eeffd640 1 -- 172.21.15.116:0/2723104782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0043b0 con 0x55bb405587d0 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.848+0000 7f590e36c740 1 -- 172.21.15.116:0/2723104782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 msgr2=0x55bb4056de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.848+0000 7f590e36c740 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0206a0 tx=0x7f58fc017000 comp rx=0 tx=0).stop 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- 172.21.15.116:0/2723104782 shutdown_connections 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056de10 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb405560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- 172.21.15.116:0/2723104782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb40557f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- 172.21.15.116:0/2723104782 >> 172.21.15.116:0/2723104782 conn(0x55bb40549a00 msgr2=0x55bb40548310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- 172.21.15.116:0/2723104782 shutdown_connections 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- 172.21.15.116:0/2723104782 wait complete. 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- start start 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 Processor -- start 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb4056b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59041add90 con 0x55bb405587d0 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59041ad6c0 con 0x55bb4054fbf0 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.849+0000 7f590e36c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59041aa920 con 0x55bb4054f2d0 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590e36c740 10 client.0 _ll_register_callbacks cb 0x7f58fc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50932/0 (socket says 172.21.15.116:50932) 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 -- 172.21.15.116:0/3636557158 learned_addr learned my addr 172.21.15.116:0/3636557158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:38.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f5908d19640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb4056b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f5909d1b640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 -- 172.21.15.116:0/3636557158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 msgr2=0x55bb4056b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb4056b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 -- 172.21.15.116:0/3636557158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 msgr2=0x55bb4056b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 -- 172.21.15.116:0/3636557158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb4056bd00 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590951a640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f590009b930 tx=0x7f590000c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7eb0cc9d1808ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f58ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59000b3020 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58fc003970 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f59000cf470 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.850+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f590009b300 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 10 client.31669 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bb405ce750 con 0x55bb4054f2d0 2024-03-19T02:50:38.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58fc003970 con 0x55bb4054f2d0 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f590009bc00 con 0x55bb4054f2d0 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 10 client.31669 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 10 client.31669 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 20 client.31669 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb405d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 4 client.31669 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb404bf9f0 con 0x55bb40568a00 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.851+0000 7f5908d19640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb405d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f58ecff9640 10 client.31669 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f5908d19640 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb405d0d30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=79c90658925ce43f server_cookie=b4667e62984fb635 in_seq=0 out_seq=0 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f58ecff9640 1 -- 172.21.15.116:0/3636557158 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb404bf9f0 con 0x55bb40568a00 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f58ecff9640 10 client.31669 handle_command_reply: tid=0 2024-03-19T02:50:38.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 client.31669 shutdown 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 msgr2=0x55bb4056de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f590009b930 tx=0x7f590000c7d0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 shutdown_connections 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55bb40568a00 0x55bb405d0d30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bb405587d0 0x55bb4056b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bb4054fbf0 0x55bb4056b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f58ecff9640 0 client.31669 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 --2- 172.21.15.116:0/3636557158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bb4054f2d0 0x55bb4056de70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 >> 172.21.15.116:0/3636557158 conn(0x55bb40549a00 msgr2=0x55bb4054adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 shutdown_connections 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 1 -- 172.21.15.116:0/3636557158 wait complete. 2024-03-19T02:50:38.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.852+0000 7f590e36c740 20 client.31669 trim_cache size 0 max 16384 2024-03-19T02:50:38.860 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:38.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 msgr2=0x7f58f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 0x7f58f00a4e30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0099e0 tx=0x7f58fc009c80 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 msgr2=0x7f5904198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f590000e9c0 tx=0x7f590000ee80 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 shutdown_connections 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f58f00a2970 0x7f58f00a4e30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5904105680 0x7f59041a3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59041013c0 0x7f59041992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 --2- 172.21.15.116:0/3907244775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5904100a00 0x7f5904198d70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 >> 172.21.15.116:0/3907244775 conn(0x7f5904077700 msgr2=0x7f59040fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 shutdown_connections 2024-03-19T02:50:38.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.853+0000 7f590b77e640 1 -- 172.21.15.116:0/3907244775 wait complete. 2024-03-19T02:50:38.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:50:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 -- 172.21.15.116:0/314502924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 msgr2=0x7fce0c100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 --2- 172.21.15.116:0/314502924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c100df0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009900 tx=0x7fcdf402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 -- 172.21.15.116:0/314502924 shutdown_connections 2024-03-19T02:50:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 --2- 172.21.15.116:0/314502924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c101330 0x7fce0c069ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 --2- 172.21.15.116:0/314502924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c100df0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 --2- 172.21.15.116:0/314502924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c1068b0 0x7fce0c106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 -- 172.21.15.116:0/314502924 >> 172.21.15.116:0/314502924 conn(0x7fce0c0774e0 msgr2=0x7fce0c0778f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 -- 172.21.15.116:0/314502924 shutdown_connections 2024-03-19T02:50:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.995+0000 7fce111a9640 1 -- 172.21.15.116:0/314502924 wait complete. 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 Processor -- start 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 -- start start 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 0x7fce0c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 0x7fce0c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c111b20 con 0x7fce0c100970 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1119a0 con 0x7fce0c101330 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce111a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c111ca0 con 0x7fce0c1068b0 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce0b59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 0x7fce0c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49578/0 (socket says 172.21.15.116:49578) 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce0259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 0x7fce0c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.996+0000 7fce0ad9d640 1 -- 172.21.15.116:0/3097258624 learned_addr learned my addr 172.21.15.116:0/3097258624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 -- 172.21.15.116:0/3097258624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 msgr2=0x7fce0c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 0x7fce0c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 -- 172.21.15.116:0/3097258624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 msgr2=0x7fce0c1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 0x7fce0c1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 -- 172.21.15.116:0/3097258624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0c19cf90 con 0x7fce0c100970 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 0x7fce0c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0259c640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 0x7fce0c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00ea60 tx=0x7fcdfc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce08d99640 1 -- 172.21.15.116:0/3097258624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00cde0 con 0x7fce0c100970 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce0c19d280 con 0x7fce0c100970 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce08d99640 1 -- 172.21.15.116:0/3097258624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdfc0035f0 con 0x7fce0c100970 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce08d99640 1 -- 172.21.15.116:0/3097258624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc010690 con 0x7fce0c100970 2024-03-19T02:50:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:38.997+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce0c1a4b10 con 0x7fce0c100970 2024-03-19T02:50:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 Processor -- start 2024-03-19T02:50:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 -- start start 2024-03-19T02:50:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x5580384a4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5580383fe4b0 con 0x5580384a47d0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5580383a6d70 con 0x55803849bbf0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce13d8f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5580383fa550 con 0x55803849b2d0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce0259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce0b59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce0259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49590/0 (socket says 172.21.15.116:49590) 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.000+0000 7fce0259c640 1 -- 172.21.15.116:0/377993470 learned_addr learned my addr 172.21.15.116:0/377993470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x5580384a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 -- 172.21.15.116:0/377993470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 msgr2=0x55803849b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 -- 172.21.15.116:0/377993470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 msgr2=0x5580384a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x5580384a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 -- 172.21.15.116:0/377993470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580383c9210 con 0x5580384a47d0 2024-03-19T02:50:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.001+0000 7fce0259c640 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009900 tx=0x7fcdf40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ebd1da048f88da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.003+0000 7fce0159a640 1 -- 172.21.15.116:0/377993470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4045070 con 0x5580384a47d0 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.003+0000 7fce0159a640 1 -- 172.21.15.116:0/377993470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdf402c7e0 con 0x5580384a47d0 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.003+0000 7fce0159a640 1 -- 172.21.15.116:0/377993470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4003ab0 con 0x5580384a47d0 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 -- 172.21.15.116:0/377993470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 msgr2=0x5580384a6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009900 tx=0x7fcdf40095f0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 -- 172.21.15.116:0/377993470 shutdown_connections 2024-03-19T02:50:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384a6bc0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x5580384a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 --2- 172.21.15.116:0/377993470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.004+0000 7fce13d8f740 1 -- 172.21.15.116:0/377993470 >> 172.21.15.116:0/377993470 conn(0x558038495a00 msgr2=0x558038495de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.005+0000 7fce08d99640 1 -- 172.21.15.116:0/3097258624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcdfc0037a0 con 0x7fce0c100970 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.005+0000 7fce13d8f740 1 -- 172.21.15.116:0/377993470 shutdown_connections 2024-03-19T02:50:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.005+0000 7fce13d8f740 1 -- 172.21.15.116:0/377993470 wait complete. 2024-03-19T02:50:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.005+0000 7fce13d8f740 1 -- start start 2024-03-19T02:50:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce08d99640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 0x7fcdf80a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 Processor -- start 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce08d99640 1 -- 172.21.15.116:0/3097258624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fcdfc0cf880 con 0x7fce0c100970 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 0x7fcdf80a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 0x5580384a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580384a47d0 0x5580384a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5580384a7560 con 0x55803849bbf0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5580383ba310 con 0x5580384a47d0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5580384a76e0 con 0x55803849b2d0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce13d8f740 10 client.0 _ll_register_callbacks cb 0x5580383ae2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50966/0 (socket says 172.21.15.116:50966) 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce0ad9d640 1 -- 172.21.15.116:0/232989340 learned_addr learned my addr 172.21.15.116:0/232989340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.006+0000 7fce0259c640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 0x5580384a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 0x7fcdf80a4dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fce04009fd0 tx=0x7fce04009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 -- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580384a47d0 msgr2=0x5580384a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580384a47d0 0x5580384a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 -- 172.21.15.116:0/232989340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 msgr2=0x5580384a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 0x5580384a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 -- 172.21.15.116:0/232989340 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580381123f0 con 0x55803849b2d0 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0259c640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 0x5580384a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc09ba90 tx=0x7fcdfc09b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3960c3df1a6ad72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fcdf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc026030 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdf402c7e0 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.007+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fcdfc09af50 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fcdfc09c200 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 10 client.31681 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5580381dc7d0 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdfc09bb00 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fcdfc0b4bc0 con 0x55803849b2d0 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 10 client.31681 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 10 client.31681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 20 client.31681 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5580384b4990 0x5580384b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 4 client.31681 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcdf402c7e0 con 0x5580384b4990 2024-03-19T02:50:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.008+0000 7fce0259c640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5580384b4990 0x5580384b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.009+0000 7fcdf27fc640 10 client.31681 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.009+0000 7fce0259c640 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5580384b4990 0x5580384b6d80 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ace2cc5c083a6fa server_cookie=f050d6ed555d71cb in_seq=0 out_seq=0 2024-03-19T02:50:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.010+0000 7fcdf27fc640 1 -- 172.21.15.116:0/232989340 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fcdf402c7e0 con 0x5580384b4990 2024-03-19T02:50:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.010+0000 7fcdf27fc640 10 client.31681 handle_command_reply: tid=0 2024-03-19T02:50:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.010+0000 7fce13d8f740 1 client.31681 shutdown 2024-03-19T02:50:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 msgr2=0x5580384a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc09ba90 tx=0x7fcdfc09b770 comp rx=0 tx=0).stop 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 shutdown_connections 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5580384b4990 0x5580384b6d80 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580384a47d0 0x5580384a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fcdf27fc640 0 client.31681 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55803849bbf0 0x5580384a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 --2- 172.21.15.116:0/232989340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x5580384a1820 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 >> 172.21.15.116:0/232989340 conn(0x558038495a00 msgr2=0x558038497750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 shutdown_connections 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 1 -- 172.21.15.116:0/232989340 wait complete. 2024-03-19T02:50:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.011+0000 7fce13d8f740 20 client.31681 trim_cache size 0 max 16384 2024-03-19T02:50:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 Processor -- start 2024-03-19T02:50:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 -- start start 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 0x5580384b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 0x5580384a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1aeb70 con 0x5580384a47d0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1b06b0 con 0x55803849b2d0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce13d8f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1af240 con 0x55803849bbf0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce0b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce0259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 0x5580384a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce0b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49618/0 (socket says 172.21.15.116:49618) 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 0x5580384b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.030+0000 7fce0b59e640 1 -- 172.21.15.116:0/3190124285 learned_addr learned my addr 172.21.15.116:0/3190124285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 -- 172.21.15.116:0/3190124285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 msgr2=0x5580384a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 0x5580384a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 -- 172.21.15.116:0/3190124285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 msgr2=0x5580384b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 0x5580384b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 -- 172.21.15.116:0/3190124285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580384965d0 con 0x5580384a47d0 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0259c640 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 0x5580384a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 0x5580384b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce0b59e640 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fce04003c40 tx=0x7fce04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee126862bcc05a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce00d99640 1 -- 172.21.15.116:0/3190124285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0400fe60 con 0x5580384a47d0 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce00d99640 1 -- 172.21.15.116:0/3190124285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce04012570 con 0x5580384a47d0 2024-03-19T02:50:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.031+0000 7fce00d99640 1 -- 172.21.15.116:0/3190124285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04029c70 con 0x5580384a47d0 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 -- 172.21.15.116:0/3190124285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 msgr2=0x5580384bc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fce04003c40 tx=0x7fce04004a40 comp rx=0 tx=0).stop 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 -- 172.21.15.116:0/3190124285 shutdown_connections 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x5580384bc410 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849bbf0 0x5580384a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 --2- 172.21.15.116:0/3190124285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849b2d0 0x5580384b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.032+0000 7fce13d8f740 1 -- 172.21.15.116:0/3190124285 >> 172.21.15.116:0/3190124285 conn(0x558038495a00 msgr2=0x558038494310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- 172.21.15.116:0/3190124285 shutdown_connections 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- 172.21.15.116:0/3190124285 wait complete. 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- start start 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 Processor -- start 2024-03-19T02:50:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x55803849e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x55803849edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1affe0 con 0x5580384a47d0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1af910 con 0x55803849bbf0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c1acb70 con 0x55803849b2d0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.033+0000 7fce13d8f740 10 client.0 _ll_register_callbacks cb 0x7fcdfc009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x55803849e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50978/0 (socket says 172.21.15.116:50978) 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 -- 172.21.15.116:0/1403534717 learned_addr learned my addr 172.21.15.116:0/1403534717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0b59e640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x55803849edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 -- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 msgr2=0x55803849e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x55803849e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 -- 172.21.15.116:0/1403534717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 msgr2=0x55803849edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x55803849edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 -- 172.21.15.116:0/1403534717 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558038515a20 con 0x55803849b2d0 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0259c640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x55803849e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0b59e640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x55803849edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce0ad9d640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc09b800 tx=0x7fcdfc004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6514867738bf7353 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fcdf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc026070 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fcdfc09af50 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.034+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55803845c3e0 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fcdfc09b8d0 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 10 client.31687 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558038515a20 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55803845c3e0 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fcdfc0a0b70 con 0x55803849b2d0 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 10 client.31687 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 10 client.31687 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 20 client.31687 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558038523930 0x558038525d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 4 client.31687 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55803840b9f0 con 0x558038523930 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.035+0000 7fce0259c640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558038523930 0x558038525d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fce0259c640 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558038523930 0x558038525d20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4db57b5c5497548c server_cookie=23bf149d1cc9fb0 in_seq=0 out_seq=0 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fcdf27fc640 10 client.31687 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fcdf27fc640 1 -- 172.21.15.116:0/1403534717 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55803840b9f0 con 0x558038523930 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fcdf27fc640 10 client.31687 handle_command_reply: tid=0 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fce13d8f740 1 client.31687 shutdown 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 msgr2=0x55803849e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.036+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc09b800 tx=0x7fcdfc004560 comp rx=0 tx=0).stop 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 shutdown_connections 2024-03-19T02:50:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x558038523930 0x558038525d20 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5580384a47d0 0x55803849edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55803849bbf0 0x55803849e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 --2- 172.21.15.116:0/1403534717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55803849b2d0 0x55803849e330 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 >> 172.21.15.116:0/1403534717 conn(0x558038495a00 msgr2=0x558038494ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fcdf27fc640 0 client.31687 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 shutdown_connections 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 1 -- 172.21.15.116:0/1403534717 wait complete. 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce13d8f740 20 client.31687 trim_cache size 0 max 16384 2024-03-19T02:50:39.045 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 msgr2=0x7fcdf80a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 0x7fcdf80a4dc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fce04009fd0 tx=0x7fce04009500 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.037+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 msgr2=0x7fce0c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00ea60 tx=0x7fcdfc00ef20 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 shutdown_connections 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdf80a2900 0x7fcdf80a4dc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce0c1068b0 0x7fce0c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce0c101330 0x7fce0c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 --2- 172.21.15.116:0/3097258624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce0c100970 0x7fce0c197fa0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 >> 172.21.15.116:0/3097258624 conn(0x7fce0c0774e0 msgr2=0x7fce0c10b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 shutdown_connections 2024-03-19T02:50:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.038+0000 7fce111a9640 1 -- 172.21.15.116:0/3097258624 wait complete. 2024-03-19T02:50:39.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:50:39.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 -- 172.21.15.116:0/2173406073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974101330 msgr2=0x7f9974069b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 --2- 172.21.15.116:0/2173406073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974101330 0x7f9974069b80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9970009900 tx=0x7f997002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:39.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 -- 172.21.15.116:0/2173406073 shutdown_connections 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 --2- 172.21.15.116:0/2173406073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974101330 0x7f9974069b80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 --2- 172.21.15.116:0/2173406073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974100970 0x7f9974100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 --2- 172.21.15.116:0/2173406073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974105690 0x7f9974105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 -- 172.21.15.116:0/2173406073 >> 172.21.15.116:0/2173406073 conn(0x7f9974077580 msgr2=0x7f9974077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 -- 172.21.15.116:0/2173406073 shutdown_connections 2024-03-19T02:50:39.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.186+0000 7f997b5f3640 1 -- 172.21.15.116:0/2173406073 wait complete. 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 Processor -- start 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 -- start start 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974100970 0x7f997410cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974101330 0x7f997410d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f997410f8b0 con 0x7f9974101330 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f997410f730 con 0x7f9974105690 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.187+0000 7f997b5f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f997410fa30 con 0x7f9974100970 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f997938f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974100970 0x7f997410cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33932/0 (socket says 172.21.15.116:33932) 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 -- 172.21.15.116:0/1585161543 learned_addr learned my addr 172.21.15.116:0/1585161543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9978b8e640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974101330 0x7f997410d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 -- 172.21.15.116:0/1585161543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974100970 msgr2=0x7f997410cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974100970 0x7f997410cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 -- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974101330 msgr2=0x7f997410d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974101330 0x7f997410d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.188+0000 7f9979b90640 1 -- 172.21.15.116:0/1585161543 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99741a4710 con 0x7f9974105690 2024-03-19T02:50:39.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f9979b90640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f997002c1c0 tx=0x7f99700045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f99667fc640 1 -- 172.21.15.116:0/1585161543 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970043020 con 0x7f9974105690 2024-03-19T02:50:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f99667fc640 1 -- 172.21.15.116:0/1585161543 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f997002ca70 con 0x7f9974105690 2024-03-19T02:50:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f99667fc640 1 -- 172.21.15.116:0/1585161543 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970003a10 con 0x7f9974105690 2024-03-19T02:50:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99741a49a0 con 0x7f9974105690 2024-03-19T02:50:39.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.189+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99741a4ea0 con 0x7f9974105690 2024-03-19T02:50:39.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.193+0000 7f99667fc640 1 -- 172.21.15.116:0/1585161543 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f9970046020 con 0x7f9974105690 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.193+0000 7f997e1d6740 1 Processor -- start 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.193+0000 7f997e1d6740 1 -- start start 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e3b2d0 0x5613b5e3b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e46bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5613b5d9e4b0 con 0x5613b5e3bbf0 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613b5d46d70 con 0x5613b5e3b2d0 2024-03-19T02:50:39.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997e1d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613b5d9a550 con 0x5613b5e447d0 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f99667fc640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 0x7f99600a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997938f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997938f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49652/0 (socket says 172.21.15.116:49652) 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f997938f640 1 -- 172.21.15.116:0/2821486186 learned_addr learned my addr 172.21.15.116:0/2821486186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f9979b90640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 0x7f99600a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.194+0000 7f9978b8e640 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e46bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f9978b8e640 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e3b2d0 0x5613b5e3b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 -- 172.21.15.116:0/2821486186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 msgr2=0x5613b5e46bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e46bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 -- 172.21.15.116:0/2821486186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e3b2d0 msgr2=0x5613b5e3b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e3b2d0 0x5613b5e3b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 -- 172.21.15.116:0/2821486186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b5d69210 con 0x5613b5e3bbf0 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f997938f640 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f996c0114f0 tx=0x7f996c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d438a43c56d39 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f99667fc640 1 -- 172.21.15.116:0/1585161543 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f99700f11b0 con 0x7f9974105690 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.195+0000 7f9979b90640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 0x7f99600a4f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f99700c5750 tx=0x7f99700b9870 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.196+0000 7f995f7fe640 1 -- 172.21.15.116:0/2821486186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c01e070 con 0x5613b5e3bbf0 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.196+0000 7f995f7fe640 1 -- 172.21.15.116:0/2821486186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f996c00fad0 con 0x5613b5e3bbf0 2024-03-19T02:50:39.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.196+0000 7f995f7fe640 1 -- 172.21.15.116:0/2821486186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c019b40 con 0x5613b5e3bbf0 2024-03-19T02:50:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 -- 172.21.15.116:0/2821486186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 msgr2=0x5613b5e44290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f996c0114f0 tx=0x7f996c0119b0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 -- 172.21.15.116:0/2821486186 shutdown_connections 2024-03-19T02:50:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e46bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3bbf0 0x5613b5e44290 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 --2- 172.21.15.116:0/2821486186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e3b2d0 0x5613b5e3b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 -- 172.21.15.116:0/2821486186 >> 172.21.15.116:0/2821486186 conn(0x5613b5e35a00 msgr2=0x5613b5e35de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 -- 172.21.15.116:0/2821486186 shutdown_connections 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.197+0000 7f997e1d6740 1 -- 172.21.15.116:0/2821486186 wait complete. 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.198+0000 7f997e1d6740 1 -- start start 2024-03-19T02:50:39.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.198+0000 7f997e1d6740 1 Processor -- start 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e41850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f9978b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e41850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f9978b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e41850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49662/0 (socket says 172.21.15.116:49662) 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 0x5613b5e422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5613b5e475c0 con 0x5613b5e3b2d0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613b5d5a310 con 0x5613b5e447d0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5613b5e47740 con 0x5613b5e3bbf0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997e1d6740 10 client.0 _ll_register_callbacks cb 0x5613b5d4e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997938f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997938f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51020/0 (socket says 172.21.15.116:51020) 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.199+0000 7f997938f640 1 -- 172.21.15.116:0/1281023124 learned_addr learned my addr 172.21.15.116:0/1281023124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f9978b8e640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 0x5613b5e422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 -- 172.21.15.116:0/1281023124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 msgr2=0x5613b5e422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 0x5613b5e422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 -- 172.21.15.116:0/1281023124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 msgr2=0x5613b5e41850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e41850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 -- 172.21.15.116:0/1281023124 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b5ab23f0 con 0x5613b5e3bbf0 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f9978b8e640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 0x5613b5e422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f997938f640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f996c00f850 tx=0x7f996c00f880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=424382a4335ef9f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.200+0000 7f995cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:39.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.202+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c026020 con 0x5613b5e3bbf0 2024-03-19T02:50:39.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.202+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f996c004a40 con 0x5613b5e3bbf0 2024-03-19T02:50:39.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.202+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996c00fad0 con 0x5613b5e3bbf0 2024-03-19T02:50:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.203+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f996c01e040 con 0x5613b5e3bbf0 2024-03-19T02:50:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.203+0000 7f997e1d6740 10 client.31699 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.203+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5613b5b7c7d0 con 0x5613b5e3bbf0 2024-03-19T02:50:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.203+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f996c003e20 con 0x5613b5e3bbf0 2024-03-19T02:50:39.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f996c021070 con 0x5613b5e3bbf0 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 10 client.31699 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 10 client.31699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 20 client.31699 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5e54b50 0x5613b5e56f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 4 client.31699 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.204+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f996c00fad0 con 0x5613b5e54b50 2024-03-19T02:50:39.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.205+0000 7f9978b8e640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5e54b50 0x5613b5e56f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.206+0000 7f9978b8e640 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5e54b50 0x5613b5e56f40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab12184fb5ae7225 server_cookie=10193c1d910ecb31 in_seq=0 out_seq=0 2024-03-19T02:50:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.206+0000 7f995cff9640 10 client.31699 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.208+0000 7f995cff9640 1 -- 172.21.15.116:0/1281023124 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f996c00fad0 con 0x5613b5e54b50 2024-03-19T02:50:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.208+0000 7f995cff9640 10 client.31699 handle_command_reply: tid=0 2024-03-19T02:50:39.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.208+0000 7f997e1d6740 1 client.31699 shutdown 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 msgr2=0x5613b5e41d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f996c00f850 tx=0x7f996c00f880 comp rx=0 tx=0).stop 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 shutdown_connections 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5e54b50 0x5613b5e56f40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f995cff9640 0 client.31699 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e447d0 0x5613b5e422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3bbf0 0x5613b5e41d90 secure :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f996c00f850 tx=0x7f996c00f880 comp rx=0 tx=0).stop 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 --2- 172.21.15.116:0/1281023124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e41850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.211+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 >> 172.21.15.116:0/1281023124 conn(0x5613b5e35a00 msgr2=0x5613b5e37750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.213+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 shutdown_connections 2024-03-19T02:50:39.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.213+0000 7f997e1d6740 1 -- 172.21.15.116:0/1281023124 wait complete. 2024-03-19T02:50:39.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.213+0000 7f997e1d6740 20 client.31699 trim_cache size 0 max 16384 2024-03-19T02:50:39.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.242+0000 7f997e1d6740 1 Processor -- start 2024-03-19T02:50:39.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f997e1d6740 1 -- start start 2024-03-19T02:50:39.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e50390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e5c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f997e1d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99741bdb00 con 0x5613b5e3b2d0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f9978b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.243+0000 7f9978b8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49672/0 (socket says 172.21.15.116:49672) 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f9978b8e640 1 -- 172.21.15.116:0/3802835971 learned_addr learned my addr 172.21.15.116:0/3802835971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e50390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99741bf640 con 0x5613b5e376c0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f99741be1d0 con 0x5613b5e447d0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f9978b8e640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e5c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 -- 172.21.15.116:0/3802835971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 msgr2=0x5613b5e5c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e5c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 -- 172.21.15.116:0/3802835971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 msgr2=0x5613b5e43f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f997938f640 1 -- 172.21.15.116:0/3802835971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b5e40800 con 0x5613b5e447d0 2024-03-19T02:50:39.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.244+0000 7f9978b8e640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.245+0000 7f997938f640 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e50390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x5613b5e428e0 tx=0x7f996c0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8865beac7dfdbad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.245+0000 7f995effd640 1 -- 172.21.15.116:0/3802835971 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c026070 con 0x5613b5e447d0 2024-03-19T02:50:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.245+0000 7f995effd640 1 -- 172.21.15.116:0/3802835971 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f996c019410 con 0x5613b5e447d0 2024-03-19T02:50:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.245+0000 7f995effd640 1 -- 172.21.15.116:0/3802835971 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c003e10 con 0x5613b5e447d0 2024-03-19T02:50:39.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.246+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 msgr2=0x5613b5e50390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.246+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e50390 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x5613b5e428e0 tx=0x7f996c0119b0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.246+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 shutdown_connections 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.246+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e5c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.247+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e447d0 0x5613b5e50390 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.247+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3802835971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e3b2d0 0x5613b5e43f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.247+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 >> 172.21.15.116:0/3802835971 conn(0x5613b5e35a00 msgr2=0x5613b5e35de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.248+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 shutdown_connections 2024-03-19T02:50:39.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.248+0000 7f997e1d6740 1 -- 172.21.15.116:0/3802835971 wait complete. 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.248+0000 7f997e1d6740 1 -- start start 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.248+0000 7f997e1d6740 1 Processor -- start 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e4cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e447d0 0x5613b5e49ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99741bef70 con 0x5613b5e447d0 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99741be8a0 con 0x5613b5e376c0 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f997410df20 con 0x5613b5e3b2d0 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997e1d6740 10 client.0 _ll_register_callbacks cb 0x7f996c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f9978b8e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e4cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51030/0 (socket says 172.21.15.116:51030) 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 -- 172.21.15.116:0/3936508864 learned_addr learned my addr 172.21.15.116:0/3936508864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f9978b8e640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e447d0 0x5613b5e49ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 -- 172.21.15.116:0/3936508864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 msgr2=0x5613b5e4cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e4cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 -- 172.21.15.116:0/3936508864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e447d0 msgr2=0x5613b5e49ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e447d0 0x5613b5e49ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.249+0000 7f997938f640 1 -- 172.21.15.116:0/3936508864 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b5e4a5c0 con 0x5613b5e3b2d0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f997938f640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f996c006010 tx=0x7f996c003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dc691da380c6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f995cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c026040 con 0x5613b5e3b2d0 2024-03-19T02:50:39.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f996c003af0 con 0x5613b5e3b2d0 2024-03-19T02:50:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996c019410 con 0x5613b5e3b2d0 2024-03-19T02:50:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f996c019670 con 0x5613b5e3b2d0 2024-03-19T02:50:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f997e1d6740 10 client.31705 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.250+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5613b5ebbb80 con 0x5613b5e3b2d0 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f996c019410 con 0x5613b5e3b2d0 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f996c021070 con 0x5613b5e3b2d0 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 10 client.31705 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 10 client.31705 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 20 client.31705 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5ebbe70 0x5613b5ebe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 4 client.31705 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.252+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613b5dfc3e0 con 0x5613b5ebbe70 2024-03-19T02:50:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.254+0000 7f9978b8e640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5ebbe70 0x5613b5ebe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:38 smithi154 ceph-mon[35693]: pgmap v721: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:39.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:38 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2229760766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.256+0000 7f9978b8e640 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5ebbe70 0x5613b5ebe310 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=17ecf61c2c7013ef server_cookie=1f642454d18a616c in_seq=0 out_seq=0 2024-03-19T02:50:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.256+0000 7f995cff9640 10 client.31705 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.256+0000 7f995cff9640 1 -- 172.21.15.116:0/3936508864 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613b5dfc3e0 con 0x5613b5ebbe70 2024-03-19T02:50:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.256+0000 7f995cff9640 10 client.31705 handle_command_reply: tid=0 2024-03-19T02:50:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.256+0000 7f997e1d6740 1 client.31705 shutdown 2024-03-19T02:50:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.257+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 msgr2=0x5613b5e49990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.257+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f996c006010 tx=0x7f996c003e40 comp rx=0 tx=0).stop 2024-03-19T02:50:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:38 smithi116 ceph-mon[32263]: pgmap v721: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:38 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2229760766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 shutdown_connections 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5613b5ebbe70 0x5613b5ebe310 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5613b5e447d0 0x5613b5e49ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5613b5e3b2d0 0x5613b5e49990 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f996c006010 tx=0x7f996c003e40 comp rx=0 tx=0).stop 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 --2- 172.21.15.116:0/3936508864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613b5e376c0 0x5613b5e4cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 >> 172.21.15.116:0/3936508864 conn(0x5613b5e35a00 msgr2=0x5613b5e41810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.258+0000 7f995cff9640 0 client.31705 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:50:39.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.259+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 shutdown_connections 2024-03-19T02:50:39.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.259+0000 7f997e1d6740 1 -- 172.21.15.116:0/3936508864 wait complete. 2024-03-19T02:50:39.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.260+0000 7f997e1d6740 20 client.31705 trim_cache size 0 max 16384 2024-03-19T02:50:39.264 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:39.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 msgr2=0x7f99600a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 0x7f99600a4f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f99700c5750 tx=0x7f99700b9870 comp rx=0 tx=0).stop 2024-03-19T02:50:39.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 msgr2=0x7f99741a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f997002c1c0 tx=0x7f99700045e0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 shutdown_connections 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f99600a2ab0 0x7f99600a4f70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9974105690 0x7f99741a4090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9974101330 0x7f997410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 --2- 172.21.15.116:0/1585161543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9974100970 0x7f997410cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.264+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 >> 172.21.15.116:0/1585161543 conn(0x7f9974077580 msgr2=0x7f99740ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.265+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 shutdown_connections 2024-03-19T02:50:39.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.265+0000 7f997b5f3640 1 -- 172.21.15.116:0/1585161543 wait complete. 2024-03-19T02:50:39.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:50:39.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65368 5.77 MB/sec execute 481 sec latency 181.889 ms 2024-03-19T02:50:39.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:38 smithi162 ceph-mon[35761]: pgmap v721: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:38 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2229760766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 -- 172.21.15.116:0/3157315245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0788101060 msgr2=0x7f078810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 --2- 172.21.15.116:0/3157315245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0788101060 0x7f078810d570 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f0780009cd0 tx=0x7f078002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 -- 172.21.15.116:0/3157315245 shutdown_connections 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 --2- 172.21.15.116:0/3157315245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0788101060 0x7f078810d570 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 --2- 172.21.15.116:0/3157315245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07881006a0 0x7f0788100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 --2- 172.21.15.116:0/3157315245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881066b0 0x7f0788106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 -- 172.21.15.116:0/3157315245 >> 172.21.15.116:0/3157315245 conn(0x7f07880778d0 msgr2=0x7f0788077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 -- 172.21.15.116:0/3157315245 shutdown_connections 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.425+0000 7f078eddf640 1 -- 172.21.15.116:0/3157315245 wait complete. 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 Processor -- start 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- start start 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881006a0 0x7f0788197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101060 0x7f0788198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0788110970 con 0x7f07881066b0 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07881107f0 con 0x7f07881006a0 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0788110af0 con 0x7f0788101060 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49696/0 (socket says 172.21.15.116:49696) 2024-03-19T02:50:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 -- 172.21.15.116:0/2404482716 learned_addr learned my addr 172.21.15.116:0/2404482716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881006a0 0x7f0788197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f0787fff640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101060 0x7f0788198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 -- 172.21.15.116:0/2404482716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101060 msgr2=0x7f0788198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101060 0x7f0788198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 -- 172.21.15.116:0/2404482716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881006a0 msgr2=0x7f0788197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881006a0 0x7f0788197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 -- 172.21.15.116:0/2404482716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078819cf00 con 0x7f07881066b0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078d37c640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f0780004af0 tx=0x7f0780035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f0785ffb640 1 -- 172.21.15.116:0/2404482716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780039040 con 0x7f07881066b0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f0785ffb640 1 -- 172.21.15.116:0/2404482716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0780044070 con 0x7f07881066b0 2024-03-19T02:50:39.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078819d190 con 0x7f07881066b0 2024-03-19T02:50:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f0785ffb640 1 -- 172.21.15.116:0/2404482716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07800042a0 con 0x7f07881066b0 2024-03-19T02:50:39.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.426+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a4a80 con 0x7f07881066b0 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0785ffb640 1 -- 172.21.15.116:0/2404482716 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0780003970 con 0x7f07881066b0 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 Processor -- start 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 -- start start 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060ccfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562060c274b0 con 0x562060cc4bf0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562060bcfd70 con 0x562060cc42d0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f07919c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562060c23550 con 0x562060ccd7d0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0785ffb640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 0x7f07780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33998/0 (socket says 172.21.15.116:33998) 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 -- 172.21.15.116:0/2767491466 learned_addr learned my addr 172.21.15.116:0/2767491466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060ccfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f078d37c640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 0x7f07780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 -- 172.21.15.116:0/2767491466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 msgr2=0x562060ccfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060ccfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 -- 172.21.15.116:0/2767491466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 msgr2=0x562060ccd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 -- 172.21.15.116:0/2767491466 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562060bf2210 con 0x562060cc42d0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0785ffb640 1 -- 172.21.15.116:0/2404482716 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f07800f14a0 con 0x7f07881066b0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f0787fff640 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f07740058c0 tx=0x7f0774005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77be1608623a7b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.430+0000 7f078d37c640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 0x7f07780a4e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f07800eea70 tx=0x7f07800c1040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f0772ffd640 1 -- 172.21.15.116:0/2767491466 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077401a310 con 0x562060cc42d0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f0772ffd640 1 -- 172.21.15.116:0/2767491466 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0774011840 con 0x562060cc42d0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f0772ffd640 1 -- 172.21.15.116:0/2767491466 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774003850 con 0x562060cc42d0 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- 172.21.15.116:0/2767491466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 msgr2=0x562060cc46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f07740058c0 tx=0x7f0774005850 comp rx=0 tx=0).stop 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- 172.21.15.116:0/2767491466 shutdown_connections 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060ccfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- 172.21.15.116:0/2767491466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc42d0 0x562060cc46b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- 172.21.15.116:0/2767491466 >> 172.21.15.116:0/2767491466 conn(0x562060cbea00 msgr2=0x562060cbede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- 172.21.15.116:0/2767491466 shutdown_connections 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- 172.21.15.116:0/2767491466 wait complete. 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- start start 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 Processor -- start 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060ccb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562060cd0560 con 0x562060cc4bf0 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562060be3310 con 0x562060ccd7d0 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562060cd06e0 con 0x562060cc42d0 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f07919c9740 10 client.0 _ll_register_callbacks cb 0x562060bd72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f078cb7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f078cb7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49714/0 (socket says 172.21.15.116:49714) 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f078cb7b640 1 -- 172.21.15.116:0/1180466584 learned_addr learned my addr 172.21.15.116:0/1180466584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f0787fff640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060ccb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.432+0000 7f0787fff640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 -- 172.21.15.116:0/1180466584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 msgr2=0x562060cca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 -- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 msgr2=0x562060ccb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060ccb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 -- 172.21.15.116:0/1180466584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56206093b3f0 con 0x562060cc4bf0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f078cb7b640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f077c012710 tx=0x7f077c012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9befd906d333f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c004020 con 0x562060cc4bf0 2024-03-19T02:50:39.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f077c0041c0 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0774011840 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f077c025050 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f07919c9740 10 client.22017 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562060a057d0 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0774011840 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.434+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f077c003960 con 0x562060cc4bf0 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 10 client.22017 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 10 client.22017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 20 client.22017 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060cddc20 0x562060ce0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 4 client.22017 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562060c853e0 con 0x562060cddc20 2024-03-19T02:50:39.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f0787fff640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060cddc20 0x562060ce0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f0787fff640 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060cddc20 0x562060ce0010 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3d1865d125629b4 server_cookie=7f0ba5abaf26b248 in_seq=0 out_seq=0 2024-03-19T02:50:39.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.435+0000 7f0767fff640 10 client.22017 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f0767fff640 1 -- 172.21.15.116:0/1180466584 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562060c853e0 con 0x562060cddc20 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f0767fff640 10 client.22017 handle_command_reply: tid=0 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 client.22017 shutdown 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 msgr2=0x562060ccad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f077c012710 tx=0x7f077c012bd0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 shutdown_connections 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060cddc20 0x562060ce0010 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f0767fff640 0 client.22017 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060ccb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060ccad40 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 --2- 172.21.15.116:0/1180466584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 >> 172.21.15.116:0/1180466584 conn(0x562060cbea00 msgr2=0x562060cc0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 shutdown_connections 2024-03-19T02:50:39.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.439+0000 7f07919c9740 1 -- 172.21.15.116:0/1180466584 wait complete. 2024-03-19T02:50:39.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.440+0000 7f07919c9740 20 client.22017 trim_cache size 0 max 16384 2024-03-19T02:50:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 Processor -- start 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 -- start start 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc4bf0 0x562060cd9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060cbda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07881aca20 con 0x562060cc42d0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.456+0000 7f07919c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ae560 con 0x562060cc4bf0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f07919c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ad0f0 con 0x562060ccd7d0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49724/0 (socket says 172.21.15.116:49724) 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 -- 172.21.15.116:0/69964183 learned_addr learned my addr 172.21.15.116:0/69964183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f078cb7b640 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc4bf0 0x562060cd9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060cbda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 -- 172.21.15.116:0/69964183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 msgr2=0x562060cbda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060cbda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 -- 172.21.15.116:0/69964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc4bf0 msgr2=0x562060cd9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc4bf0 0x562060cd9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 -- 172.21.15.116:0/69964183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562060cc4a20 con 0x562060cc42d0 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f0787fff640 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f0774009fd0 tx=0x7f077401c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4651e68ebca8df6b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f07727fc640 1 -- 172.21.15.116:0/69964183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07740050d0 con 0x562060cc42d0 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f07727fc640 1 -- 172.21.15.116:0/69964183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0774033320 con 0x562060cc42d0 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.457+0000 7f07727fc640 1 -- 172.21.15.116:0/69964183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774011c00 con 0x562060cc42d0 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- 172.21.15.116:0/69964183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 msgr2=0x562060cccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f0774009fd0 tx=0x7f077401c040 comp rx=0 tx=0).stop 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- 172.21.15.116:0/69964183 shutdown_connections 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060ccd7d0 0x562060cbda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060cc4bf0 0x562060cd9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 --2- 172.21.15.116:0/69964183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc42d0 0x562060cccf30 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- 172.21.15.116:0/69964183 >> 172.21.15.116:0/69964183 conn(0x562060cbea00 msgr2=0x562060cbd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- 172.21.15.116:0/69964183 shutdown_connections 2024-03-19T02:50:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- 172.21.15.116:0/69964183 wait complete. 2024-03-19T02:50:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.458+0000 7f07919c9740 1 -- start start 2024-03-19T02:50:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 Processor -- start 2024-03-19T02:50:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cd5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060cd14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ade90 con 0x562060cc4bf0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ad7c0 con 0x562060ccd7d0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07881aaa20 con 0x562060cc42d0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f07919c9740 10 client.0 _ll_register_callbacks cb 0x7f077c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f0787fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cd5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49734/0 (socket says 172.21.15.116:49734) 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 -- 172.21.15.116:0/2065755186 learned_addr learned my addr 172.21.15.116:0/2065755186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f0787fff640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060cd14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 -- 172.21.15.116:0/2065755186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 msgr2=0x562060cd5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cd5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 -- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 msgr2=0x562060cd14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060cd14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.459+0000 7f078cb7b640 1 -- 172.21.15.116:0/2065755186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562060cd1ca0 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f078cb7b640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f077c0059f0 tx=0x7f077c00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60d690e5ca64628 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c02e060 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7f077c00fd70 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0774033320 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f077c035050 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 10 client.22023 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562060d43ef0 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0774033320 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f077c005460 con 0x562060cc4bf0 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 10 client.22023 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:39.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 10 client.22023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 20 client.22023 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060d446a0 0x562060d46ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 4 client.22023 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.460+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562060c349f0 con 0x562060d446a0 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.461+0000 7f0787fff640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060d446a0 0x562060d46ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.461+0000 7f0767fff640 10 client.22023 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.461+0000 7f0787fff640 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060d446a0 0x562060d46ac0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86b1ffa90549b78f server_cookie=2b6fe8b9d271239f in_seq=0 out_seq=0 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.462+0000 7f0767fff640 1 -- 172.21.15.116:0/2065755186 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562060c349f0 con 0x562060d446a0 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.462+0000 7f0767fff640 10 client.22023 handle_command_reply: tid=0 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.462+0000 7f07919c9740 1 client.22023 shutdown 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 msgr2=0x562060cd0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f077c0059f0 tx=0x7f077c00f810 comp rx=0 tx=0).stop 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 shutdown_connections 2024-03-19T02:50:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x562060d446a0 0x562060d46ac0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562060ccd7d0 0x562060cd14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562060cc4bf0 0x562060cd0f80 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 --2- 172.21.15.116:0/2065755186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562060cc42d0 0x562060cd5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f0767fff640 0 client.22023 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 >> 172.21.15.116:0/2065755186 conn(0x562060cbea00 msgr2=0x562060cbdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 shutdown_connections 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 1 -- 172.21.15.116:0/2065755186 wait complete. 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f07919c9740 20 client.22023 trim_cache size 0 max 16384 2024-03-19T02:50:39.468 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 msgr2=0x7f07780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 0x7f07780a4e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f07800eea70 tx=0x7f07800c1040 comp rx=0 tx=0).stop 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 msgr2=0x7f078819c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f0780004af0 tx=0x7f0780035ce0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 shutdown_connections 2024-03-19T02:50:39.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f07780a2970 0x7f07780a4e30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07881066b0 0x7f078819c7d0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0788101060 0x7f0788198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 --2- 172.21.15.116:0/2404482716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07881006a0 0x7f0788197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.463+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 >> 172.21.15.116:0/2404482716 conn(0x7f07880778d0 msgr2=0x7f07881035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.464+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 shutdown_connections 2024-03-19T02:50:39.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:39.464+0000 7f078eddf640 1 -- 172.21.15.116:0/2404482716 wait complete. 2024-03-19T02:50:40.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65498 5.77 MB/sec execute 482 sec latency 182.436 ms 2024-03-19T02:50:40.476 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:50:40.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:50:40.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 -- 172.21.15.116:0/2112181139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 msgr2=0x7fe644068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/2112181139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe644068980 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009be0 tx=0x7fe63c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 -- 172.21.15.116:0/2112181139 shutdown_connections 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/2112181139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe644068980 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/2112181139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe644100690 0x7fe644100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/2112181139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6441066a0 0x7fe644106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.599+0000 7fe64c3f3640 1 -- 172.21.15.116:0/2112181139 >> 172.21.15.116:0/2112181139 conn(0x7fe644077820 msgr2=0x7fe644077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- 172.21.15.116:0/2112181139 shutdown_connections 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- 172.21.15.116:0/2112181139 wait complete. 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 Processor -- start 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- start start 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe644100690 0x7fe64419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6441066a0 0x7fe64419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe644113d00 con 0x7fe644101050 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe644113b80 con 0x7fe6441066a0 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.600+0000 7fe64c3f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe644113e80 con 0x7fe644100690 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49740/0 (socket says 172.21.15.116:49740) 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64a990640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6441066a0 0x7fe64419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe644100690 0x7fe64419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 -- 172.21.15.116:0/3247669674 learned_addr learned my addr 172.21.15.116:0/3247669674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 -- 172.21.15.116:0/3247669674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe644100690 msgr2=0x7fe64419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe644100690 0x7fe64419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 -- 172.21.15.116:0/3247669674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6441066a0 msgr2=0x7fe64419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6441066a0 0x7fe64419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 -- 172.21.15.116:0/3247669674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64419f140 con 0x7fe644101050 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64998e640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9c0 tx=0x7fe63400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe63b7fe640 1 -- 172.21.15.116:0/3247669674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63400cde0 con 0x7fe644101050 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64419f430 con 0x7fe644101050 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe63b7fe640 1 -- 172.21.15.116:0/3247669674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6340035f0 con 0x7fe644101050 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6441a6cc0 con 0x7fe644101050 2024-03-19T02:50:40.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.601+0000 7fe63b7fe640 1 -- 172.21.15.116:0/3247669674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634010690 con 0x7fe644101050 2024-03-19T02:50:40.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.604+0000 7fe63b7fe640 1 -- 172.21.15.116:0/3247669674 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe6340037a0 con 0x7fe644101050 2024-03-19T02:50:40.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.604+0000 7fe64efd7740 1 Processor -- start 2024-03-19T02:50:40.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.604+0000 7fe64efd7740 1 -- start start 2024-03-19T02:50:40.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e0983380 0x5625e0983760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e098c880 0x5625e098ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5625e08e7160 con 0x5625e098c880 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5625e088fe10 con 0x5625e0983ca0 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64efd7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5625e08e3200 con 0x5625e0983380 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe63b7fe640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 0x7fe6300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64998e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e098c880 0x5625e098ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34032/0 (socket says 172.21.15.116:34032) 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 -- 172.21.15.116:0/113155543 learned_addr learned my addr 172.21.15.116:0/113155543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a990640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 0x7fe6300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a990640 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e0983380 0x5625e0983760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe63b7fe640 1 -- 172.21.15.116:0/3247669674 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fe6340cf530 con 0x7fe644101050 2024-03-19T02:50:40.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 -- 172.21.15.116:0/113155543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e0983380 msgr2=0x5625e0983760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e0983380 0x5625e0983760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 -- 172.21.15.116:0/113155543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e098c880 msgr2=0x5625e098ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e098c880 0x5625e098ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 -- 172.21.15.116:0/113155543 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e06c37e0 con 0x5625e0983ca0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.605+0000 7fe64a18f640 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe640009c70 tx=0x7fe640011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5b49d73f2cbadb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64a990640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 0x7fe6300a4e30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe63c006010 tx=0x7fe63c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe638bfe640 1 -- 172.21.15.116:0/113155543 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640004230 con 0x5625e0983ca0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe638bfe640 1 -- 172.21.15.116:0/113155543 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6400043d0 con 0x5625e0983ca0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe638bfe640 1 -- 172.21.15.116:0/113155543 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640019740 con 0x5625e0983ca0 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 -- 172.21.15.116:0/113155543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 msgr2=0x5625e098c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe640009c70 tx=0x7fe640011810 comp rx=0 tx=0).stop 2024-03-19T02:50:40.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 -- 172.21.15.116:0/113155543 shutdown_connections 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e098c880 0x5625e098ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098c340 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 --2- 172.21.15.116:0/113155543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e0983380 0x5625e0983760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.606+0000 7fe64efd7740 1 -- 172.21.15.116:0/113155543 >> 172.21.15.116:0/113155543 conn(0x5625e097dab0 msgr2=0x5625e097de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- 172.21.15.116:0/113155543 shutdown_connections 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- 172.21.15.116:0/113155543 wait complete. 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- start start 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 Processor -- start 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0989cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e098a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5625e098f670 con 0x5625e0983380 2024-03-19T02:50:40.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5625e08a31a0 con 0x5625e0983ca0 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5625e098f7f0 con 0x5625e098c880 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49752/0 (socket says 172.21.15.116:49752) 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64a18f640 1 -- 172.21.15.116:0/1655846506 learned_addr learned my addr 172.21.15.116:0/1655846506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.607+0000 7fe64efd7740 10 client.0 _ll_register_callbacks cb 0x5625e05fac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64998e640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0989cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a990640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e098a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 -- 172.21.15.116:0/1655846506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 msgr2=0x5625e098a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e098a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 -- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 msgr2=0x5625e0989cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0989cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 -- 172.21.15.116:0/1655846506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e0723f70 con 0x5625e0983380 2024-03-19T02:50:40.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64a18f640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe640005db0 tx=0x7fe6400056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73da28ca9ff1291d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe62dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640027070 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fe640003800 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6400043d0 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe640033050 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64efd7740 10 client.22035 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.608+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5625e05bb170 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6400043d0 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fe640018070 con 0x5625e0983380 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 10 client.22035 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 10 client.22035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 20 client.22035 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e099c9f0 0x5625e099ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 4 client.22035 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:40.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625e08f46a0 con 0x5625e099c9f0 2024-03-19T02:50:40.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64998e640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e099c9f0 0x5625e099ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe62dffb640 10 client.22035 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.609+0000 7fe64998e640 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e099c9f0 0x5625e099ede0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bc4c1def616eea9 server_cookie=20b92db537eb7a2f in_seq=0 out_seq=0 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe62dffb640 1 -- 172.21.15.116:0/1655846506 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625e08f46a0 con 0x5625e099c9f0 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe62dffb640 10 client.22035 handle_command_reply: tid=0 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 client.22035 shutdown 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 msgr2=0x5625e09897b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe640005db0 tx=0x7fe6400056d0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 shutdown_connections 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e099c9f0 0x5625e099ede0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e098a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0989cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 --2- 172.21.15.116:0/1655846506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e09897b0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 >> 172.21.15.116:0/1655846506 conn(0x5625e097dab0 msgr2=0x5625e097f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe62dffb640 0 client.22035 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 shutdown_connections 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.611+0000 7fe64efd7740 1 -- 172.21.15.116:0/1655846506 wait complete. 2024-03-19T02:50:40.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.612+0000 7fe64efd7740 20 client.22035 trim_cache size 0 max 16384 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 Processor -- start 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 -- start start 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0998410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e09a4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b0e70 con 0x5625e0983380 2024-03-19T02:50:40.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b29b0 con 0x5625e0983ca0 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64efd7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b1540 con 0x5625e098c880 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49760/0 (socket says 172.21.15.116:49760) 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 -- 172.21.15.116:0/869374522 learned_addr learned my addr 172.21.15.116:0/869374522 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64998e640 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0998410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a990640 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e09a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 -- 172.21.15.116:0/869374522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 msgr2=0x5625e09a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e09a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 -- 172.21.15.116:0/869374522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 msgr2=0x5625e0998410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0998410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.635+0000 7fe64a18f640 1 -- 172.21.15.116:0/869374522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e0979160 con 0x5625e0983380 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe64a18f640 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe640005bc0 tx=0x7fe640001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdebf008a20d5e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe62ffff640 1 -- 172.21.15.116:0/869374522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640027030 con 0x5625e0983380 2024-03-19T02:50:40.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe62ffff640 1 -- 172.21.15.116:0/869374522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe640015070 con 0x5625e0983380 2024-03-19T02:50:40.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe62ffff640 1 -- 172.21.15.116:0/869374522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6400053e0 con 0x5625e0983380 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe64efd7740 1 -- 172.21.15.116:0/869374522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 msgr2=0x5625e098bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.636+0000 7fe64efd7740 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe640005bc0 tx=0x7fe640001e50 comp rx=0 tx=0).stop 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- 172.21.15.116:0/869374522 shutdown_connections 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e09a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e0998410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- 172.21.15.116:0/869374522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e098bee0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- 172.21.15.116:0/869374522 >> 172.21.15.116:0/869374522 conn(0x5625e097dab0 msgr2=0x5625e09887e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- 172.21.15.116:0/869374522 shutdown_connections 2024-03-19T02:50:40.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- 172.21.15.116:0/869374522 wait complete. 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- start start 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 Processor -- start 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e0990500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b22e0 con 0x5625e0983380 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441b1c10 con 0x5625e0983ca0 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441aee70 con 0x5625e098c880 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64efd7740 10 client.0 _ll_register_callbacks cb 0x7fe634003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64a18f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49770/0 (socket says 172.21.15.116:49770) 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64a18f640 1 -- 172.21.15.116:0/263200577 learned_addr learned my addr 172.21.15.116:0/263200577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64998e640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.637+0000 7fe64a990640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e0990500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 -- 172.21.15.116:0/263200577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 msgr2=0x5625e0990500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e0990500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 -- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 msgr2=0x5625e098ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 -- 172.21.15.116:0/263200577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e0984650 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64a18f640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe640014040 tx=0x7fe6400092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cc4aa3653f90a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe62dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640027040 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fe640003800 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe640015070 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe640003a00 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64efd7740 10 client.22041 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5625e0a02890 con 0x5625e0983380 2024-03-19T02:50:40.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.638+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe640015070 con 0x5625e0983380 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fe64003e070 con 0x5625e0983380 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 10 client.22041 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 10 client.22041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 20 client.22041 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e0a02e40 0x5625e0a05260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 4 client.22041 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5625e08ee9f0 con 0x5625e0a02e40 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64998e640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e0a02e40 0x5625e0a05260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe62dffb640 10 client.22041 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.639+0000 7fe64998e640 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e0a02e40 0x5625e0a05260 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12492d79bb6ce7ad server_cookie=ad81de791c64f924 in_seq=0 out_seq=0 2024-03-19T02:50:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe62dffb640 1 -- 172.21.15.116:0/263200577 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5625e08ee9f0 con 0x5625e0a02e40 2024-03-19T02:50:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe62dffb640 10 client.22041 handle_command_reply: tid=0 2024-03-19T02:50:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 client.22041 shutdown 2024-03-19T02:50:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 msgr2=0x5625e0994a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe640014040 tx=0x7fe6400092d0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 shutdown_connections 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5625e0a02e40 0x5625e0a05260 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe62dffb640 0 client.22041 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5625e098c880 0x5625e0990500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625e0983ca0 0x5625e098ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 --2- 172.21.15.116:0/263200577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625e0983380 0x5625e0994a10 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 >> 172.21.15.116:0/263200577 conn(0x5625e097dab0 msgr2=0x5625e0987f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 shutdown_connections 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 1 -- 172.21.15.116:0/263200577 wait complete. 2024-03-19T02:50:40.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.643+0000 7fe64efd7740 20 client.22041 trim_cache size 0 max 16384 2024-03-19T02:50:40.647 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:40.647 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:50:40.647 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "19898826-94c9-4ca6-bcb7-c15e0dc4843b", 2024-03-19T02:50:40.647 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:50:40.647 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 msgr2=0x7fe6300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 0x7fe6300a4e30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe63c006010 tx=0x7fe63c009500 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 msgr2=0x7fe64419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9c0 tx=0x7fe63400ee80 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 shutdown_connections 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe6300a2970 0x7fe6300a4e30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6441066a0 0x7fe64419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe644101050 0x7fe64419a690 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 --2- 172.21.15.116:0/3247669674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe644100690 0x7fe64419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 >> 172.21.15.116:0/3247669674 conn(0x7fe644077820 msgr2=0x7fe6441041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 shutdown_connections 2024-03-19T02:50:40.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.644+0000 7fe64c3f3640 1 -- 172.21.15.116:0/3247669674 wait complete. 2024-03-19T02:50:40.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:50:40.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 -- 172.21.15.116:0/525520178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 msgr2=0x7fef1c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 --2- 172.21.15.116:0/525520178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c106a80 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009930 tx=0x7fef0c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 -- 172.21.15.116:0/525520178 shutdown_connections 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 --2- 172.21.15.116:0/525520178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c101050 0x7fef1c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 --2- 172.21.15.116:0/525520178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c100690 0x7fef1c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 --2- 172.21.15.116:0/525520178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c106a80 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 -- 172.21.15.116:0/525520178 >> 172.21.15.116:0/525520178 conn(0x7fef1c077820 msgr2=0x7fef1c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 -- 172.21.15.116:0/525520178 shutdown_connections 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.772+0000 7fef21d09640 1 -- 172.21.15.116:0/525520178 wait complete. 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 Processor -- start 2024-03-19T02:50:40.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 -- start start 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c100690 0x7fef1c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c110960 con 0x7fef1c1066a0 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1107e0 con 0x7fef1c100690 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.773+0000 7fef21d09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c110ae0 con 0x7fef1c101050 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51158/0 (socket says 172.21.15.116:51158) 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 -- 172.21.15.116:0/3424832748 learned_addr learned my addr 172.21.15.116:0/3424832748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c100690 0x7fef1c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 -- 172.21.15.116:0/3424832748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c100690 msgr2=0x7fef1c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c100690 0x7fef1c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 -- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 msgr2=0x7fef1c19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 -- 172.21.15.116:0/3424832748 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1c19cf90 con 0x7fef1c101050 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef1affd640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e9f0 tx=0x7fef1000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef18ff9640 1 -- 172.21.15.116:0/3424832748 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1000cd80 con 0x7fef1c101050 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1c19d280 con 0x7fef1c101050 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef18ff9640 1 -- 172.21.15.116:0/3424832748 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef100054d0 con 0x7fef1c101050 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef18ff9640 1 -- 172.21.15.116:0/3424832748 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10010490 con 0x7fef1c101050 2024-03-19T02:50:40.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.774+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1c1a4b10 con 0x7fef1c101050 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.777+0000 7fef18ff9640 1 -- 172.21.15.116:0/3424832748 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fef10005020 con 0x7fef1c101050 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.777+0000 7fef249d3740 1 Processor -- start 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.777+0000 7fef249d3740 1 -- start start 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d14320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef18ff9640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0c6c500 con 0x55f1d0d08a30 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0c14be0 con 0x55f1d0d09350 2024-03-19T02:50:40.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef249d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0c685a0 con 0x55f1d0d11f30 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49790/0 (socket says 172.21.15.116:49790) 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 -- 172.21.15.116:0/3139922653 learned_addr learned my addr 172.21.15.116:0/3139922653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34062/0 (socket says 172.21.15.116:34062) 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1affd640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d14320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef18ff9640 1 -- 172.21.15.116:0/3424832748 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fef100cf5b0 con 0x7fef1c101050 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 -- 172.21.15.116:0/3139922653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 msgr2=0x55f1d0d14320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d14320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 -- 172.21.15.116:0/3139922653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 msgr2=0x55f1d0d08e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 -- 172.21.15.116:0/3139922653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d0c36000 con 0x55f1d0d09350 2024-03-19T02:50:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1affd640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d14320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1bfff640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fef140098b0 tx=0x7fef14009600 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009230 tx=0x7fef0c009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c690609fc730e8c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.778+0000 7fef01ffb640 1 -- 172.21.15.116:0/3139922653 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c045020 con 0x55f1d0d09350 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef01ffb640 1 -- 172.21.15.116:0/3139922653 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0c02c7e0 con 0x55f1d0d09350 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef01ffb640 1 -- 172.21.15.116:0/3139922653 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c005540 con 0x55f1d0d09350 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 -- 172.21.15.116:0/3139922653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 msgr2=0x55f1d0d119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009230 tx=0x7fef0c009610 comp rx=0 tx=0).stop 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 -- 172.21.15.116:0/3139922653 shutdown_connections 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d14320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d09350 0x55f1d0d119f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 --2- 172.21.15.116:0/3139922653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d08a30 0x55f1d0d08e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 -- 172.21.15.116:0/3139922653 >> 172.21.15.116:0/3139922653 conn(0x55f1d0d03160 msgr2=0x55f1d0d03540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 -- 172.21.15.116:0/3139922653 shutdown_connections 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.779+0000 7fef249d3740 1 -- 172.21.15.116:0/3139922653 wait complete. 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 -- start start 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 Processor -- start 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d0eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d0f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0d14cc0 con 0x55f1d0d09350 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0c28160 con 0x55f1d0d11f30 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1d0d14e40 con 0x55f1d0d08a30 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef249d3740 10 client.0 _ll_register_callbacks cb 0x55f1d0c64a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d0eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49806/0 (socket says 172.21.15.116:49806) 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d0f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.780+0000 7fef1affd640 1 -- 172.21.15.116:0/760030071 learned_addr learned my addr 172.21.15.116:0/760030071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 -- 172.21.15.116:0/760030071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 msgr2=0x55f1d0d0eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d0eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 -- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 msgr2=0x55f1d0d0f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d0f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 -- 172.21.15.116:0/760030071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d0a4a7d0 con 0x55f1d0d09350 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef1affd640 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fef100cc980 tx=0x7fef1009b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3a19feae1c7420 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10099df0 con 0x55f1d0d09350 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fef100860c0 con 0x55f1d0d09350 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef0c02c7e0 con 0x55f1d0d09350 2024-03-19T02:50:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fef100a3740 con 0x55f1d0d09350 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef249d3740 10 client.22053 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f1d09803f0 con 0x55f1d0d09350 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0c02c7e0 con 0x55f1d0d09350 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fef10097ea0 con 0x55f1d0d09350 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.781+0000 7fef249d3740 10 client.22053 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef249d3740 10 client.22053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef249d3740 20 client.22053 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d22170 0x55f1d0d245a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef249d3740 4 client.22053 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d0c79a40 con 0x55f1d0d22170 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d22170 0x55f1d0d245a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d22170 0x55f1d0d245a0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806ccccccc4227e3 server_cookie=c306f4c66f476bc5 in_seq=0 out_seq=0 2024-03-19T02:50:40.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.782+0000 7feefaffd640 10 client.22053 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7feefaffd640 1 -- 172.21.15.116:0/760030071 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1d0c79a40 con 0x55f1d0d22170 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7feefaffd640 10 client.22053 handle_command_reply: tid=0 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 client.22053 shutdown 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 msgr2=0x55f1d0d0f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fef100cc980 tx=0x7fef1009b370 comp rx=0 tx=0).stop 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 shutdown_connections 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d22170 0x55f1d0d245a0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d0f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7feefaffd640 0 client.22053 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0f410 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 --2- 172.21.15.116:0/760030071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d0eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 >> 172.21.15.116:0/760030071 conn(0x55f1d0d03160 msgr2=0x55f1d0d04eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 shutdown_connections 2024-03-19T02:50:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 1 -- 172.21.15.116:0/760030071 wait complete. 2024-03-19T02:50:40.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.790+0000 7fef249d3740 20 client.22053 trim_cache size 0 max 16384 2024-03-19T02:50:40.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 Processor -- start 2024-03-19T02:50:40.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 -- start start 2024-03-19T02:50:40.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d08a30 0x55f1d0d11600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d1da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1aeaf0 con 0x55f1d0d09350 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b0630 con 0x55f1d0d08a30 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef249d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1af1c0 con 0x55f1d0d11f30 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.809+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49810/0 (socket says 172.21.15.116:49810) 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 -- 172.21.15.116:0/4123734806 learned_addr learned my addr 172.21.15.116:0/4123734806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d08a30 0x55f1d0d11600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1bfff640 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d1da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 -- 172.21.15.116:0/4123734806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 msgr2=0x55f1d0d1da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d1da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 -- 172.21.15.116:0/4123734806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d08a30 msgr2=0x55f1d0d11600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d08a30 0x55f1d0d11600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 -- 172.21.15.116:0/4123734806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d09400f0 con 0x55f1d0d09350 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef1affd640 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fef100ccf20 tx=0x7fef10004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1449a65f8fd8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef017fa640 1 -- 172.21.15.116:0/4123734806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef100b9070 con 0x55f1d0d09350 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef017fa640 1 -- 172.21.15.116:0/4123734806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef10005310 con 0x55f1d0d09350 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.810+0000 7fef017fa640 1 -- 172.21.15.116:0/4123734806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef100a3930 con 0x55f1d0d09350 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- 172.21.15.116:0/4123734806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 msgr2=0x55f1d0d0de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fef100ccf20 tx=0x7fef10004b10 comp rx=0 tx=0).stop 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- 172.21.15.116:0/4123734806 shutdown_connections 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d11f30 0x55f1d0d1da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d0de10 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 --2- 172.21.15.116:0/4123734806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d08a30 0x55f1d0d11600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- 172.21.15.116:0/4123734806 >> 172.21.15.116:0/4123734806 conn(0x55f1d0d03160 msgr2=0x55f1d0d0f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- 172.21.15.116:0/4123734806 shutdown_connections 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- 172.21.15.116:0/4123734806 wait complete. 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 -- start start 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.811+0000 7fef249d3740 1 Processor -- start 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d16170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d874a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1aff60 con 0x55f1d0d09350 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1af890 con 0x55f1d0d11f30 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1acaf0 con 0x55f1d0d08a30 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 10 client.0 _ll_register_callbacks cb 0x7fef1000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d16170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49826/0 (socket says 172.21.15.116:49826) 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1affd640 1 -- 172.21.15.116:0/605501555 learned_addr learned my addr 172.21.15.116:0/605501555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1bfff640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d874a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 -- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 msgr2=0x55f1d0d874a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d874a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 -- 172.21.15.116:0/605501555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 msgr2=0x55f1d0d166b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 -- 172.21.15.116:0/605501555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d0d87a90 con 0x55f1d0d08a30 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1affd640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef1b7fe640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d16170 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fef0c02c170 tx=0x7fef0c0050e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5556dada1d54a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7feefaffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c045040 con 0x55f1d0d08a30 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1d0c73d90 con 0x55f1d0d08a30 2024-03-19T02:50:40.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.812+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9510+0+0 (secure 0 0 0) 0x7fef0c005150 con 0x55f1d0d08a30 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fef0c05f050 con 0x55f1d0d08a30 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 10 client.31759 fetch_fsmap learned FSMap version 42 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f1d0d88de0 con 0x55f1d0d08a30 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f1d0c73d90 con 0x55f1d0d08a30 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fef0c004c60 con 0x55f1d0d08a30 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 10 client.31759 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 10 client.31759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 20 client.31759 populate_metadata read hostname 'smithi116' 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d89120 0x55f1d0d8b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 4 client.31759 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f1d0bf1540 con 0x55f1d0d89120 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.813+0000 7fef1affd640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d89120 0x55f1d0d8b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:40.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.814+0000 7fef1affd640 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d89120 0x55f1d0d8b510 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e25961492d1ce5 server_cookie=6dd7f6f743cf8150 in_seq=0 out_seq=0 2024-03-19T02:50:40.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.814+0000 7feefaffd640 10 client.31759 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7feefaffd640 1 -- 172.21.15.116:0/605501555 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f1d0bf1540 con 0x55f1d0d89120 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7feefaffd640 10 client.31759 handle_command_reply: tid=0 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 client.31759 shutdown 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 msgr2=0x55f1d0d16170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d16170 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fef0c02c170 tx=0x7fef0c0050e0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 shutdown_connections 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55f1d0d89120 0x55f1d0d8b510 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1d0d11f30 0x55f1d0d874a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7feefaffd640 0 client.31759 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1d0d09350 0x55f1d0d166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 --2- 172.21.15.116:0/605501555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d0d08a30 0x55f1d0d16170 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 >> 172.21.15.116:0/605501555 conn(0x55f1d0d03160 msgr2=0x55f1d0d01950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 shutdown_connections 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 1 -- 172.21.15.116:0/605501555 wait complete. 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.818+0000 7fef249d3740 20 client.31759 trim_cache size 0 max 16384 2024-03-19T02:50:40.822 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (454 inodes in the stack)", 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "19898826-94c9-4ca6-bcb7-c15e0dc4843b": { 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "19898826-94c9-4ca6-bcb7-c15e0dc4843b", 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:50:40.823 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 msgr2=0x7fef040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fef140098b0 tx=0x7fef14009600 comp rx=0 tx=0).stop 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 msgr2=0x7fef1c198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e9f0 tx=0x7fef1000edf0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 shutdown_connections 2024-03-19T02:50:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fef040a2970 0x7fef040a4e30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef1c1066a0 0x7fef1c19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef1c101050 0x7fef1c198530 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 --2- 172.21.15.116:0/3424832748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef1c100690 0x7fef1c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 >> 172.21.15.116:0/3424832748 conn(0x7fef1c077820 msgr2=0x7fef1c068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 shutdown_connections 2024-03-19T02:50:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:50:40.819+0000 7fef21d09640 1 -- 172.21.15.116:0/3424832748 wait complete. 2024-03-19T02:50:40.831 INFO:tasks.cephfs.filesystem:scrub status for tag:19898826-94c9-4ca6-bcb7-c15e0dc4843b - {'path': '/', 'tag': '19898826-94c9-4ca6-bcb7-c15e0dc4843b', 'options': 'recursive,force'} 2024-03-19T02:50:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:41 smithi154 ceph-mon[35693]: pgmap v722: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:41 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:41.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:41 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:41 smithi116 ceph-mon[32263]: pgmap v722: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:41 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:41 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:41.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65565 5.76 MB/sec execute 483 sec latency 249.170 ms 2024-03-19T02:50:41.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:41 smithi162 ceph-mon[35761]: pgmap v722: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 96 op/s 2024-03-19T02:50:41.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:41 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:41.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:41 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:42 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65669 5.76 MB/sec execute 484 sec latency 181.685 ms 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:50:42.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:42 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:42.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:42 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:50:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:43 smithi116 ceph-mon[32263]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 157 op/s 2024-03-19T02:50:43.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65759 5.75 MB/sec execute 485 sec latency 979.412 ms 2024-03-19T02:50:43.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:43 smithi162 ceph-mon[35761]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 157 op/s 2024-03-19T02:50:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:43 smithi154 ceph-mon[35693]: pgmap v723: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.1 MiB/s wr, 157 op/s 2024-03-19T02:50:44.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65844 5.75 MB/sec execute 486 sec latency 1979.439 ms 2024-03-19T02:50:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:45 smithi116 ceph-mon[32263]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 119 op/s 2024-03-19T02:50:45.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 65927 5.75 MB/sec execute 487 sec latency 2979.465 ms 2024-03-19T02:50:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:45 smithi162 ceph-mon[35761]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 119 op/s 2024-03-19T02:50:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:45 smithi154 ceph-mon[35693]: pgmap v724: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 119 op/s 2024-03-19T02:50:46.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66024 5.75 MB/sec execute 488 sec latency 3505.222 ms 2024-03-19T02:50:47.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66101 5.74 MB/sec execute 489 sec latency 982.008 ms 2024-03-19T02:50:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:47 smithi162 ceph-mon[35761]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 256 op/s 2024-03-19T02:50:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:47 smithi154 ceph-mon[35693]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 256 op/s 2024-03-19T02:50:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:47 smithi116 ceph-mon[32263]: pgmap v725: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 256 op/s 2024-03-19T02:50:48.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66188 5.76 MB/sec execute 490 sec latency 1982.035 ms 2024-03-19T02:50:49.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66259 5.75 MB/sec execute 491 sec latency 2982.066 ms 2024-03-19T02:50:49.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:49 smithi162 ceph-mon[35761]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:49 smithi154 ceph-mon[35693]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:49.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:49 smithi116 ceph-mon[32263]: pgmap v726: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:50.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66344 5.75 MB/sec execute 492 sec latency 3982.095 ms 2024-03-19T02:50:51.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66435 5.74 MB/sec execute 493 sec latency 4500.226 ms 2024-03-19T02:50:51.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:51 smithi162 ceph-mon[35761]: pgmap v727: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:51 smithi154 ceph-mon[35693]: pgmap v727: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:51.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:51 smithi116 ceph-mon[32263]: pgmap v727: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-19T02:50:52.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66536 5.73 MB/sec execute 494 sec latency 141.338 ms 2024-03-19T02:50:53.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66651 5.72 MB/sec execute 495 sec latency 143.966 ms 2024-03-19T02:50:53.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:53 smithi162 ceph-mon[35761]: pgmap v728: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 249 op/s 2024-03-19T02:50:53.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:53 smithi154 ceph-mon[35693]: pgmap v728: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 249 op/s 2024-03-19T02:50:53.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:53 smithi116 ceph-mon[32263]: pgmap v728: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 249 op/s 2024-03-19T02:50:54.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66773 5.72 MB/sec execute 496 sec latency 147.922 ms 2024-03-19T02:50:54.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:54 smithi162 ceph-mon[35761]: pgmap v729: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-19T02:50:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:54 smithi154 ceph-mon[35693]: pgmap v729: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-19T02:50:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:54 smithi116 ceph-mon[32263]: pgmap v729: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 188 op/s 2024-03-19T02:50:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:55.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:55 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:55.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:55.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:55 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:55.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 66889 5.72 MB/sec execute 497 sec latency 141.461 ms 2024-03-19T02:50:55.407 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:55.408 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:55 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:56.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67011 5.71 MB/sec execute 498 sec latency 141.673 ms 2024-03-19T02:50:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:56 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:56 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:56 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:50:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:56 smithi162 ceph-mon[35761]: pgmap v730: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-03-19T02:50:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:56 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:56 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:56 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:50:56.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:56 smithi154 ceph-mon[35693]: pgmap v730: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-03-19T02:50:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:56 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:56 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:56 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:50:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:56 smithi116 ceph-mon[32263]: pgmap v730: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 267 op/s 2024-03-19T02:50:57.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67121 5.70 MB/sec execute 499 sec latency 139.797 ms 2024-03-19T02:50:58.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67247 5.70 MB/sec execute 500 sec latency 141.671 ms 2024-03-19T02:50:58.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:50:58 smithi162 ceph-mon[35761]: pgmap v731: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:50:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:50:58 smithi154 ceph-mon[35693]: pgmap v731: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:50:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:50:58 smithi116 ceph-mon[32263]: pgmap v731: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:50:59.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67384 5.71 MB/sec execute 501 sec latency 135.446 ms 2024-03-19T02:51:00.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67471 5.71 MB/sec execute 502 sec latency 244.548 ms 2024-03-19T02:51:00.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:00 smithi162 ceph-mon[35761]: pgmap v732: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:51:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:00 smithi154 ceph-mon[35693]: pgmap v732: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:51:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:00 smithi116 ceph-mon[32263]: pgmap v732: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 131 op/s 2024-03-19T02:51:01.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67572 5.72 MB/sec execute 503 sec latency 244.664 ms 2024-03-19T02:51:02.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67678 5.72 MB/sec execute 504 sec latency 550.356 ms 2024-03-19T02:51:02.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:02 smithi162 ceph-mon[35761]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-19T02:51:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:02 smithi154 ceph-mon[35693]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-19T02:51:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:02 smithi116 ceph-mon[32263]: pgmap v733: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-19T02:51:03.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67807 5.73 MB/sec execute 505 sec latency 1550.381 ms 2024-03-19T02:51:04.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 67929 5.75 MB/sec execute 506 sec latency 2550.406 ms 2024-03-19T02:51:04.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:04 smithi162 ceph-mon[35761]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:51:04.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:04 smithi154 ceph-mon[35693]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:51:04.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:04 smithi116 ceph-mon[32263]: pgmap v734: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 154 op/s 2024-03-19T02:51:05.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68054 5.76 MB/sec execute 507 sec latency 3550.441 ms 2024-03-19T02:51:06.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68182 5.76 MB/sec execute 508 sec latency 4074.315 ms 2024-03-19T02:51:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:06 smithi162 ceph-mon[35761]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-03-19T02:51:06.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:06 smithi162 ceph-mon[35761]: osdmap e130: 12 total, 12 up, 12 in 2024-03-19T02:51:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:06 smithi154 ceph-mon[35693]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-03-19T02:51:06.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:06 smithi154 ceph-mon[35693]: osdmap e130: 12 total, 12 up, 12 in 2024-03-19T02:51:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:06 smithi116 ceph-mon[32263]: pgmap v735: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 222 op/s 2024-03-19T02:51:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:06 smithi116 ceph-mon[32263]: osdmap e130: 12 total, 12 up, 12 in 2024-03-19T02:51:07.338 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68246 5.76 MB/sec execute 509 sec latency 1048.508 ms 2024-03-19T02:51:08.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68294 5.75 MB/sec execute 510 sec latency 2048.537 ms 2024-03-19T02:51:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:08 smithi162 ceph-mon[35761]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:08 smithi154 ceph-mon[35693]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:08 smithi116 ceph-mon[32263]: pgmap v737: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:09.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68337 5.75 MB/sec execute 511 sec latency 3048.564 ms 2024-03-19T02:51:10.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68370 5.74 MB/sec execute 512 sec latency 4048.601 ms 2024-03-19T02:51:10.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:10 smithi162 ceph-mon[35761]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:10 smithi154 ceph-mon[35693]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:10 smithi116 ceph-mon[32263]: pgmap v738: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 172 op/s 2024-03-19T02:51:10.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:51:10.832 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:51:10.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 -- 172.21.15.116:0/3448875734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 msgr2=0x7f6868107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 --2- 172.21.15.116:0/3448875734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f6868107bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6864009860 tx=0x7f686402bb70 comp rx=0 tx=0).stop 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 -- 172.21.15.116:0/3448875734 shutdown_connections 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 --2- 172.21.15.116:0/3448875734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6868102180 0x7f686810e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 --2- 172.21.15.116:0/3448875734 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f68681017c0 0x7f6868101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 --2- 172.21.15.116:0/3448875734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f6868107bb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 -- 172.21.15.116:0/3448875734 >> 172.21.15.116:0/3448875734 conn(0x7f68680fd250 msgr2=0x7f68680ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 -- 172.21.15.116:0/3448875734 shutdown_connections 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.960+0000 7f686f353640 1 -- 172.21.15.116:0/3448875734 wait complete. 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 Processor -- start 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 -- start start 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6868102180 0x7f6868198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f686819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6868111a50 con 0x7f68681017c0 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68681118d0 con 0x7f68681077d0 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686f353640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6868111bd0 con 0x7f6868102180 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45360/0 (socket says 172.21.15.116:45360) 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 -- 172.21.15.116:0/3972798436 learned_addr learned my addr 172.21.15.116:0/3972798436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686c8ee640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6868102180 0x7f6868198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d8f0640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f686819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 -- 172.21.15.116:0/3972798436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6868102180 msgr2=0x7f6868198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6868102180 0x7f6868198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 -- 172.21.15.116:0/3972798436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 msgr2=0x7f686819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f686819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 -- 172.21.15.116:0/3972798436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686819d000 con 0x7f68681017c0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.961+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6864037040 tx=0x7f6864004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.962+0000 7f685e7fc640 1 -- 172.21.15.116:0/3972798436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864041020 con 0x7f68681017c0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.962+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686819d290 con 0x7f68681017c0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.962+0000 7f685e7fc640 1 -- 172.21.15.116:0/3972798436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686402c9c0 con 0x7f68681017c0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.962+0000 7f685e7fc640 1 -- 172.21.15.116:0/3972798436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864039040 con 0x7f68681017c0 2024-03-19T02:51:10.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.962+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68681a4b80 con 0x7f68681017c0 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.964+0000 7f685e7fc640 1 -- 172.21.15.116:0/3972798436 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6864044020 con 0x7f68681017c0 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.964+0000 7f687201e740 1 Processor -- start 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.964+0000 7f687201e740 1 -- start start 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.964+0000 7f685e7fc640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 0x7f68540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686d8f0640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 0x7f68540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7ca30 0x561c28a7ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f685e7fc640 1 -- 172.21.15.116:0/3972798436 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f68640f0890 con 0x7f68681017c0 2024-03-19T02:51:10.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a88320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561c289e0500 con 0x561c28a7d350 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561c28988be0 con 0x561c28a7ca30 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f687201e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c289dc5a0 con 0x561c28a85f30 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45364/0 (socket says 172.21.15.116:45364) 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 -- 172.21.15.116:0/2154524893 learned_addr learned my addr 172.21.15.116:0/2154524893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7ca30 0x561c28a7ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686d8f0640 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a88320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 -- 172.21.15.116:0/2154524893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 msgr2=0x561c28a88320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a88320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 -- 172.21.15.116:0/2154524893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7ca30 msgr2=0x561c28a7ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7ca30 0x561c28a7ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 -- 172.21.15.116:0/2154524893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c289aa000 con 0x561c28a7d350 2024-03-19T02:51:10.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686d8f0640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 0x7f68540a4e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6860005fd0 tx=0x7f6860011810 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6858009cd0 tx=0x7f68580117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0986b1c2ae1a023 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.965+0000 7f68537fe640 1 -- 172.21.15.116:0/2154524893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858004250 con 0x561c28a7d350 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f68537fe640 1 -- 172.21.15.116:0/2154524893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6858016070 con 0x561c28a7d350 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f68537fe640 1 -- 172.21.15.116:0/2154524893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580126b0 con 0x561c28a7d350 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 -- 172.21.15.116:0/2154524893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 msgr2=0x561c28a859f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f6858009cd0 tx=0x7f68580117b0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 -- 172.21.15.116:0/2154524893 shutdown_connections 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a88320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a859f0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 --2- 172.21.15.116:0/2154524893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7ca30 0x561c28a7ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 -- 172.21.15.116:0/2154524893 >> 172.21.15.116:0/2154524893 conn(0x561c28a77160 msgr2=0x561c28a77540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:10.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.966+0000 7f687201e740 1 -- 172.21.15.116:0/2154524893 shutdown_connections 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 -- 172.21.15.116:0/2154524893 wait complete. 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 -- start start 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 Processor -- start 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7d350 0x561c28a833d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a83910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.967+0000 7f687201e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561c28a88d20 con 0x561c28a7ca30 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f687201e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561c2899c160 con 0x561c28a7d350 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f687201e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561c28a88ea0 con 0x561c28a85f30 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f687201e740 10 client.0 _ll_register_callbacks cb 0x561c289d8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d8f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a83910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45376/0 (socket says 172.21.15.116:45376) 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 -- 172.21.15.116:0/3571796247 learned_addr learned my addr 172.21.15.116:0/3571796247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:10.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686c8ee640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7d350 0x561c28a833d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 -- 172.21.15.116:0/3571796247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 msgr2=0x561c28a83910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a83910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 -- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7d350 msgr2=0x561c28a833d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7d350 0x561c28a833d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 -- 172.21.15.116:0/3571796247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c287be7d0 con 0x561c28a7ca30 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f686d0ef640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f6864009830 tx=0x7f68640bd390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c4c517f9bdf4a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f6850ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68640c5370 con 0x561c28a7ca30 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6858016070 con 0x561c28a7ca30 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f68640035f0 con 0x561c28a7ca30 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.968+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f68640b9240 con 0x561c28a7ca30 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 10 client.22071 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:10.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561c286f43f0 con 0x561c28a7ca30 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6858016070 con 0x561c28a7ca30 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f686403c030 con 0x561c28a7ca30 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 10 client.22071 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 10 client.22071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 20 client.22071 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28a96200 0x561c28a98620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 4 client.22071 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c289eda40 con 0x561c28a96200 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.969+0000 7f686c8ee640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28a96200 0x561c28a98620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.970+0000 7f6850ff9640 10 client.22071 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:10.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.970+0000 7f686c8ee640 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28a96200 0x561c28a98620 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb3f7386b8789959 server_cookie=e55a13c9e42d9223 in_seq=0 out_seq=0 2024-03-19T02:51:10.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.971+0000 7f6850ff9640 1 -- 172.21.15.116:0/3571796247 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c289eda40 con 0x561c28a96200 2024-03-19T02:51:10.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.971+0000 7f6850ff9640 10 client.22071 handle_command_reply: tid=0 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.971+0000 7f687201e740 1 client.22071 shutdown 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 msgr2=0x561c28a82e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f6864009830 tx=0x7f68640bd390 comp rx=0 tx=0).stop 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 shutdown_connections 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28a96200 0x561c28a98620 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a85f30 0x561c28a83910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f6850ff9640 0 client.22071 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a7d350 0x561c28a833d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 --2- 172.21.15.116:0/3571796247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7ca30 0x561c28a82e90 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 >> 172.21.15.116:0/3571796247 conn(0x561c28a77160 msgr2=0x561c28a78eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 shutdown_connections 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 1 -- 172.21.15.116:0/3571796247 wait complete. 2024-03-19T02:51:10.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.972+0000 7f687201e740 20 client.22071 trim_cache size 0 max 16384 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 Processor -- start 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 -- start start 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28a81dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a91b50 0x561c28a9df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f684c010080 con 0x561c28a7d350 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f684c011bc0 con 0x561c28a85f30 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f687201e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f684c010750 con 0x561c28a91b50 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45378/0 (socket says 172.21.15.116:45378) 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d0ef640 1 -- 172.21.15.116:0/2066269294 learned_addr learned my addr 172.21.15.116:0/2066269294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d8f0640 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a91b50 0x561c28a9df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28a81dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d0ef640 1 -- 172.21.15.116:0/2066269294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a91b50 msgr2=0x561c28a9df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.995+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a91b50 0x561c28a9df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f686d0ef640 1 -- 172.21.15.116:0/2066269294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 msgr2=0x561c28a81dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28a81dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f686d0ef640 1 -- 172.21.15.116:0/2066269294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c28a88690 con 0x561c28a7d350 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f6864009980 tx=0x7f68640c0ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4522351b7d11b9f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f6852ffd640 1 -- 172.21.15.116:0/2066269294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68640c5370 con 0x561c28a7d350 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f6852ffd640 1 -- 172.21.15.116:0/2066269294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686402c7e0 con 0x561c28a7d350 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.996+0000 7f6852ffd640 1 -- 172.21.15.116:0/2066269294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68640b93e0 con 0x561c28a7d350 2024-03-19T02:51:11.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- 172.21.15.116:0/2066269294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 msgr2=0x561c28a855c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f6864009980 tx=0x7f68640c0ea0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- 172.21.15.116:0/2066269294 shutdown_connections 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28a91b50 0x561c28a9df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28a81dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- 172.21.15.116:0/2066269294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a7d350 0x561c28a855c0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- 172.21.15.116:0/2066269294 >> 172.21.15.116:0/2066269294 conn(0x561c28a77160 msgr2=0x561c28a78dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- 172.21.15.116:0/2066269294 shutdown_connections 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- 172.21.15.116:0/2066269294 wait complete. 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- start start 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 Processor -- start 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28afa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 0x561c28afaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28af4830 0x561c28af4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f684c0114f0 con 0x561c28a91b50 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f684c010e20 con 0x561c28a85f30 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.997+0000 7f687201e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f684c00e080 con 0x561c28af4830 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f687201e740 10 client.0 _ll_register_callbacks cb 0x7f68640b8ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28afa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 0x561c28afaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686d8f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28af4830 0x561c28af4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686d0ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28afa4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:33918/0 (socket says 172.21.15.116:33918) 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686d0ef640 1 -- 172.21.15.116:0/2889088322 learned_addr learned my addr 172.21.15.116:0/2889088322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 -- 172.21.15.116:0/2889088322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28af4830 msgr2=0x561c28af4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28af4830 0x561c28af4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 -- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 msgr2=0x561c28afa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28afa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 -- 172.21.15.116:0/2889088322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c286b40f0 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f686c8ee640 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 0x561c28afaa00 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f68580096c0 tx=0x7f6858009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4d211f4642347b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f6850ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858030060 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f686402c7e0 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f6858005250 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6858037050 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f687201e740 10 client.22077 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.998+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561c28b01ed0 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686402c7e0 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f6858023040 con 0x561c28a91b50 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 10 client.22077 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 10 client.22077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 20 client.22077 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28b02370 0x561c28b04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 4 client.22077 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:51:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561c289e7d90 con 0x561c28b02370 2024-03-19T02:51:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:10.999+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28b02370 0x561c28b04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.000+0000 7f686d0ef640 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28b02370 0x561c28b04760 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2720ed0ca8d2968c server_cookie=e4ef89cac0c43100 in_seq=0 out_seq=0 2024-03-19T02:51:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.000+0000 7f6850ff9640 10 client.22077 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f6850ff9640 1 -- 172.21.15.116:0/2889088322 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561c289e7d90 con 0x561c28b02370 2024-03-19T02:51:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f6850ff9640 10 client.22077 handle_command_reply: tid=0 2024-03-19T02:51:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 client.22077 shutdown 2024-03-19T02:51:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 msgr2=0x561c28afaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 0x561c28afaa00 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f68580096c0 tx=0x7f6858009fc0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 shutdown_connections 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561c28b02370 0x561c28b04760 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f6850ff9640 0 client.22077 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c28af4830 0x561c28af4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c28a91b50 0x561c28afaa00 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 --2- 172.21.15.116:0/2889088322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c28a85f30 0x561c28afa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 >> 172.21.15.116:0/2889088322 conn(0x561c28a77160 msgr2=0x561c28a90b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 shutdown_connections 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 1 -- 172.21.15.116:0/2889088322 wait complete. 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.001+0000 7f687201e740 20 client.22077 trim_cache size 0 max 16384 2024-03-19T02:51:11.008 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 msgr2=0x7f68540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 0x7f68540a4e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6860005fd0 tx=0x7f6860011810 comp rx=0 tx=0).stop 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 msgr2=0x7f6868198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f6864037040 tx=0x7f6864004040 comp rx=0 tx=0).stop 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 shutdown_connections 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68540a2970 0x7f68540a4e30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68681077d0 0x7f686819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6868102180 0x7f6868198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 --2- 172.21.15.116:0/3972798436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68681017c0 0x7f6868198010 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 >> 172.21.15.116:0/3972798436 conn(0x7f68680fd250 msgr2=0x7f68680fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 shutdown_connections 2024-03-19T02:51:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.002+0000 7f686f353640 1 -- 172.21.15.116:0/3972798436 wait complete. 2024-03-19T02:51:11.015 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:51:11.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:51:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 -- 172.21.15.116:0/2808430493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 msgr2=0x7f32e41069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/2808430493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e41069e0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009900 tx=0x7f32d802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 -- 172.21.15.116:0/2808430493 shutdown_connections 2024-03-19T02:51:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/2808430493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 0x7f32e410d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/2808430493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 0x7f32e4100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/2808430493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e41069e0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 -- 172.21.15.116:0/2808430493 >> 172.21.15.116:0/2808430493 conn(0x7f32e4076130 msgr2=0x7f32e4076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 -- 172.21.15.116:0/2808430493 shutdown_connections 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.147+0000 7f32e9c8f640 1 -- 172.21.15.116:0/2808430493 wait complete. 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 Processor -- start 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 -- start start 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 0x7f32e4197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 0x7f32e41983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32e4110830 con 0x7f32e4106600 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32e41106b0 con 0x7f32e41007d0 2024-03-19T02:51:11.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e9c8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32e41109b0 con 0x7f32e4101190 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e948e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e948e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45398/0 (socket says 172.21.15.116:45398) 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e948e640 1 -- 172.21.15.116:0/1915628597 learned_addr learned my addr 172.21.15.116:0/1915628597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e8c8d640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 0x7f32e4197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.148+0000 7f32e3fff640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 0x7f32e41983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 -- 172.21.15.116:0/1915628597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 msgr2=0x7f32e41983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 0x7f32e41983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 -- 172.21.15.116:0/1915628597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 msgr2=0x7f32e4197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 0x7f32e4197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 -- 172.21.15.116:0/1915628597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e419ce60 con 0x7f32e4106600 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e948e640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00eb10 tx=0x7f32dc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00ce20 con 0x7f32e4106600 2024-03-19T02:51:11.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32dc005020 con 0x7f32e4106600 2024-03-19T02:51:11.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc010690 con 0x7f32e4106600 2024-03-19T02:51:11.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e419d150 con 0x7f32e4106600 2024-03-19T02:51:11.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.149+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e41a49e0 con 0x7f32e4106600 2024-03-19T02:51:11.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.150+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e4068530 con 0x7f32e4106600 2024-03-19T02:51:11.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.152+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f32dc0051d0 con 0x7f32e4106600 2024-03-19T02:51:11.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.153+0000 7f32e1ffb640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 0x7f32d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.153+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f32dc0cf950 con 0x7f32e4106600 2024-03-19T02:51:11.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.153+0000 7f32e8c8d640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 0x7f32d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.154+0000 7f32e8c8d640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 0x7f32d40a4e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005ab0 tx=0x7f32d8005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.158+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32dc09c280 con 0x7f32e4106600 2024-03-19T02:51:11.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68456 5.75 MB/sec execute 513 sec latency 4562.533 ms 2024-03-19T02:51:11.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.452+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32e41a4d50 con 0x7f32e4106600 2024-03-19T02:51:11.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.453+0000 7f32e1ffb640 1 -- 172.21.15.116:0/1915628597 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11950 (secure 0 0 0) 0x7f32dc09bb80 con 0x7f32e4106600 2024-03-19T02:51:11.454 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:51:11.455 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:50:15.839334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:51:11.455 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-03-19T02:51:11.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 msgr2=0x7f32d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 0x7f32d40a4e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005ab0 tx=0x7f32d8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 msgr2=0x7f32e419c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00eb10 tx=0x7f32dc00ef10 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 shutdown_connections 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f32d40a2970 0x7f32d40a4e30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4106600 0x7f32e419c730 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e4101190 0x7f32e41983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 --2- 172.21.15.116:0/1915628597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32e41007d0 0x7f32e4197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 >> 172.21.15.116:0/1915628597 conn(0x7f32e4076130 msgr2=0x7f32e40ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 shutdown_connections 2024-03-19T02:51:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.456+0000 7f32e9c8f640 1 -- 172.21.15.116:0/1915628597 wait complete. 2024-03-19T02:51:11.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:51:11.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 -- 172.21.15.116:0/395455117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81006a0 msgr2=0x7f90b8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/395455117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81006a0 0x7f90b8100b20 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009930 tx=0x7f90a802bc50 comp rx=0 tx=0).stop 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 -- 172.21.15.116:0/395455117 shutdown_connections 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/395455117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90b8101060 0x7f90b810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/395455117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81006a0 0x7f90b8100b20 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/395455117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b81066b0 0x7f90b8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 -- 172.21.15.116:0/395455117 >> 172.21.15.116:0/395455117 conn(0x7f90b80778d0 msgr2=0x7f90b8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 -- 172.21.15.116:0/395455117 shutdown_connections 2024-03-19T02:51:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.610+0000 7f90bfa7e640 1 -- 172.21.15.116:0/395455117 wait complete. 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 Processor -- start 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 -- start start 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90b81006a0 0x7f90b8115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b8101060 0x7f90b81162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90b810f870 con 0x7f90b81066b0 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90b810f6f0 con 0x7f90b8101060 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.611+0000 7f90bfa7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f90b810f9f0 con 0x7f90b81006a0 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45412/0 (socket says 172.21.15.116:45412) 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 -- 172.21.15.116:0/2875167592 learned_addr learned my addr 172.21.15.116:0/2875167592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90bd019640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b8101060 0x7f90b81162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 -- 172.21.15.116:0/2875167592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90b81006a0 msgr2=0x7f90b8115d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90b81006a0 0x7f90b8115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 -- 172.21.15.116:0/2875167592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b8101060 msgr2=0x7f90b81162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b8101060 0x7f90b81162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 -- 172.21.15.116:0/2875167592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b81117a0 con 0x7f90b81066b0 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90be01b640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90b000eb10 tx=0x7f90b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.612+0000 7f90aeffd640 1 -- 172.21.15.116:0/2875167592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0004b20 con 0x7f90b81066b0 2024-03-19T02:51:11.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.613+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b8111a90 con 0x7f90b81066b0 2024-03-19T02:51:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.613+0000 7f90aeffd640 1 -- 172.21.15.116:0/2875167592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b0004cc0 con 0x7f90b81066b0 2024-03-19T02:51:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.613+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b811bc70 con 0x7f90b81066b0 2024-03-19T02:51:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.614+0000 7f90aeffd640 1 -- 172.21.15.116:0/2875167592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0010690 con 0x7f90b81066b0 2024-03-19T02:51:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.616+0000 7f90aeffd640 1 -- 172.21.15.116:0/2875167592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f90b00108f0 con 0x7f90b81066b0 2024-03-19T02:51:11.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.616+0000 7f90aeffd640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 0x7f90a40a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90bd81a640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 0x7f90a40a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90aeffd640 1 -- 172.21.15.116:0/2875167592 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f90b00cf910 con 0x7f90b81066b0 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90c2662740 1 Processor -- start 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90bd81a640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 0x7f90a40a4e70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009ad0 tx=0x7f90b4009590 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90c2662740 1 -- start start 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.617+0000 7f90c2662740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90c2662740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90c2662740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90c2662740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f2d563f4b0 con 0x55f2d56dc2d0 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90c2662740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f2d55e7d70 con 0x55f2d56e57d0 2024-03-19T02:51:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90c2662740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d563b550 con 0x55f2d56dcbf0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46204/0 (socket says 172.21.15.116:46204) 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90be01b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd019640 1 -- 172.21.15.116:0/412861036 learned_addr learned my addr 172.21.15.116:0/412861036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 -- 172.21.15.116:0/412861036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 msgr2=0x55f2d56e5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 -- 172.21.15.116:0/412861036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 msgr2=0x55f2d56e7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 -- 172.21.15.116:0/412861036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d560a210 con 0x55f2d56dc2d0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd019640 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.618+0000 7f90bd81a640 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56dc6b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f90b400ed00 tx=0x7f90b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789cc28a94dedf7f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.619+0000 7f90a3fff640 1 -- 172.21.15.116:0/412861036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4020070 con 0x55f2d56dc2d0 2024-03-19T02:51:11.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.619+0000 7f90a3fff640 1 -- 172.21.15.116:0/412861036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b4025910 con 0x55f2d56dc2d0 2024-03-19T02:51:11.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.619+0000 7f90a3fff640 1 -- 172.21.15.116:0/412861036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b401c7c0 con 0x55f2d56dc2d0 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.619+0000 7f90c2662740 1 -- 172.21.15.116:0/412861036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 msgr2=0x55f2d56dc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.619+0000 7f90c2662740 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56dc6b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f90b400ed00 tx=0x7f90b4009f90 comp rx=0 tx=0).stop 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 -- 172.21.15.116:0/412861036 shutdown_connections 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 --2- 172.21.15.116:0/412861036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56dc6b0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 -- 172.21.15.116:0/412861036 >> 172.21.15.116:0/412861036 conn(0x55f2d56d6a00 msgr2=0x55f2d56d6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 -- 172.21.15.116:0/412861036 shutdown_connections 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 -- 172.21.15.116:0/412861036 wait complete. 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 -- start start 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.620+0000 7f90c2662740 1 Processor -- start 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 0x55f2d56e3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90bd019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f2d56e85c0 con 0x55f2d56e57d0 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f2d55fb310 con 0x55f2d56dc2d0 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90bd019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46220/0 (socket says 172.21.15.116:46220) 2024-03-19T02:51:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2d56e8740 con 0x55f2d56dcbf0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90bd019640 1 -- 172.21.15.116:0/800903792 learned_addr learned my addr 172.21.15.116:0/800903792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90be01b640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 0x55f2d56e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90c2662740 10 client.0 _ll_register_callbacks cb 0x55f2d55ef2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.621+0000 7f90bd81a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 -- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 msgr2=0x55f2d56e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 -- 172.21.15.116:0/800903792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 msgr2=0x55f2d56e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 -- 172.21.15.116:0/800903792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d53533f0 con 0x55f2d56e57d0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90bd81a640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90be01b640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 0x55f2d56e3380 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f90b00ccc30 tx=0x7f90b009c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9669e2d9a9f1542 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b00b9060 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90b4025910 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.622+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f90b009bda0 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f90b00b3030 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 10 client.22095 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f2d541d7d0 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b0062020 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f90b00a3e50 con 0x55f2d56e57d0 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 10 client.22095 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 10 client.22095 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 20 client.22095 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d56f5c30 0x55f2d56f8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 4 client.22095 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2d569d3e0 con 0x55f2d56f5c30 2024-03-19T02:51:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.623+0000 7f90bd81a640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d56f5c30 0x55f2d56f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.624+0000 7f90bd81a640 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d56f5c30 0x55f2d56f8020 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66622178cf800798 server_cookie=231e29f396306235 in_seq=0 out_seq=0 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.624+0000 7f90a17fa640 10 client.22095 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90a17fa640 1 -- 172.21.15.116:0/800903792 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2d569d3e0 con 0x55f2d56f5c30 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90a17fa640 10 client.22095 handle_command_reply: tid=0 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 client.22095 shutdown 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 msgr2=0x55f2d56e3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 0x55f2d56e3380 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f90b00ccc30 tx=0x7f90b009c300 comp rx=0 tx=0).stop 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 shutdown_connections 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d56f5c30 0x55f2d56f8020 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56e57d0 0x55f2d56e3380 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 --2- 172.21.15.116:0/800903792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90a17fa640 0 client.22095 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 >> 172.21.15.116:0/800903792 conn(0x55f2d56d6a00 msgr2=0x55f2d56d8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 shutdown_connections 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 1 -- 172.21.15.116:0/800903792 wait complete. 2024-03-19T02:51:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.625+0000 7f90c2662740 20 client.22095 trim_cache size 0 max 16384 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 Processor -- start 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 -- start start 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dcbf0 0x55f2d56f1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 0x55f2d56fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81bdb40 con 0x55f2d56dc2d0 2024-03-19T02:51:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81bf680 con 0x55f2d56dcbf0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90c2662740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81be210 con 0x55f2d56e57d0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45446/0 (socket says 172.21.15.116:45446) 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 -- 172.21.15.116:0/623142282 learned_addr learned my addr 172.21.15.116:0/623142282 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90be01b640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 0x55f2d56fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd019640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dcbf0 0x55f2d56f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 -- 172.21.15.116:0/623142282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 msgr2=0x55f2d56fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 0x55f2d56fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 -- 172.21.15.116:0/623142282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dcbf0 msgr2=0x55f2d56f1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dcbf0 0x55f2d56f1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90bd81a640 1 -- 172.21.15.116:0/623142282 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2d56e18b0 con 0x55f2d56dc2d0 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.644+0000 7f90be01b640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 0x55f2d56fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.645+0000 7f90bd81a640 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f90b400e720 tx=0x7f90b40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e079d8fcbd45a6e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.645+0000 7f90a37fe640 1 -- 172.21.15.116:0/623142282 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4014070 con 0x55f2d56dc2d0 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.645+0000 7f90a37fe640 1 -- 172.21.15.116:0/623142282 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b402b040 con 0x55f2d56dc2d0 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.645+0000 7f90a37fe640 1 -- 172.21.15.116:0/623142282 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b4020040 con 0x55f2d56dc2d0 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- 172.21.15.116:0/623142282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 msgr2=0x55f2d56e5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f90b400e720 tx=0x7f90b40058d0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- 172.21.15.116:0/623142282 shutdown_connections 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56e57d0 0x55f2d56fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56dcbf0 0x55f2d56f1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 --2- 172.21.15.116:0/623142282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dc2d0 0x55f2d56e5030 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- 172.21.15.116:0/623142282 >> 172.21.15.116:0/623142282 conn(0x55f2d56d6a00 msgr2=0x55f2d56d5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- 172.21.15.116:0/623142282 shutdown_connections 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- 172.21.15.116:0/623142282 wait complete. 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 -- start start 2024-03-19T02:51:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.646+0000 7f90c2662740 1 Processor -- start 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dc2d0 0x55f2d56edd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81befb0 con 0x55f2d56dcbf0 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81be8e0 con 0x55f2d56e57d0 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81bbb40 con 0x55f2d56dc2d0 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 10 client.0 _ll_register_callbacks cb 0x7f90b009b910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45448/0 (socket says 172.21.15.116:45448) 2024-03-19T02:51:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 -- 172.21.15.116:0/398323316 learned_addr learned my addr 172.21.15.116:0/398323316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd81a640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dc2d0 0x55f2d56edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90be01b640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 -- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dc2d0 msgr2=0x55f2d56edd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dc2d0 0x55f2d56edd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 -- 172.21.15.116:0/398323316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 msgr2=0x55f2d56e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 -- 172.21.15.116:0/398323316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2d56e9ee0 con 0x55f2d56dcbf0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90be01b640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90bd019640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005ab0 tx=0x7f90a802ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5bb6ef8d22e033 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8005030 con 0x55f2d56dcbf0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f90a803a040 con 0x55f2d56dcbf0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.647+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90b402b040 con 0x55f2d56dcbf0 2024-03-19T02:51:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f90a80051d0 con 0x55f2d56dcbf0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 10 client.22101 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f2d575c050 con 0x55f2d56dcbf0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90b402b040 con 0x55f2d56dcbf0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f90a80035f0 con 0x55f2d56dcbf0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 10 client.22101 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 10 client.22101 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 20 client.22101 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d575c800 0x55f2d575ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 4 client.22101 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2d564c9f0 con 0x55f2d575c800 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.648+0000 7f90bd81a640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d575c800 0x55f2d575ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90a17fa640 10 client.22101 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90bd81a640 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d575c800 0x55f2d575ec20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a12fea681ccfac97 server_cookie=11125a43537ef52d in_seq=0 out_seq=0 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90a17fa640 1 -- 172.21.15.116:0/398323316 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2d564c9f0 con 0x55f2d575c800 2024-03-19T02:51:11.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90a17fa640 10 client.22101 handle_command_reply: tid=0 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 client.22101 shutdown 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 msgr2=0x55f2d56e90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005ab0 tx=0x7f90a802ca60 comp rx=0 tx=0).stop 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 shutdown_connections 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55f2d575c800 0x55f2d575ec20 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2d56e57d0 0x55f2d56e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90a17fa640 0 client.22101 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2d56dcbf0 0x55f2d56e90d0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 --2- 172.21.15.116:0/398323316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2d56dc2d0 0x55f2d56edd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 >> 172.21.15.116:0/398323316 conn(0x55f2d56d6a00 msgr2=0x55f2d56d8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 shutdown_connections 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 1 -- 172.21.15.116:0/398323316 wait complete. 2024-03-19T02:51:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.649+0000 7f90c2662740 20 client.22101 trim_cache size 0 max 16384 2024-03-19T02:51:11.658 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 msgr2=0x7f90a40a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 0x7f90a40a4e70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009ad0 tx=0x7f90b4009590 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 msgr2=0x7f90b8110ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90b000eb10 tx=0x7f90b000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 shutdown_connections 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f90a40a29b0 0x7f90a40a4e70 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90b81066b0 0x7f90b8110ee0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b8101060 0x7f90b81162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 --2- 172.21.15.116:0/2875167592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90b81006a0 0x7f90b8115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 >> 172.21.15.116:0/2875167592 conn(0x7f90b80778d0 msgr2=0x7f90b8102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 shutdown_connections 2024-03-19T02:51:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.650+0000 7f90bfa7e640 1 -- 172.21.15.116:0/2875167592 wait complete. 2024-03-19T02:51:11.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 -- 172.21.15.116:0/3153241040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da01006a0 msgr2=0x7f3da0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/3153241040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da01006a0 0x7f3da0100b20 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009930 tx=0x7f3d9402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 -- 172.21.15.116:0/3153241040 shutdown_connections 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/3153241040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da0101060 0x7f3da010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/3153241040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da01006a0 0x7f3da0100b20 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/3153241040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 -- 172.21.15.116:0/3153241040 >> 172.21.15.116:0/3153241040 conn(0x7f3da00778d0 msgr2=0x7f3da0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 -- 172.21.15.116:0/3153241040 shutdown_connections 2024-03-19T02:51:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.790+0000 7f3da5a9a640 1 -- 172.21.15.116:0/3153241040 wait complete. 2024-03-19T02:51:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 Processor -- start 2024-03-19T02:51:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 -- start start 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da01006a0 0x7f3da0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0110970 con 0x7f3da0101060 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01107f0 con 0x7f3da01006a0 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3da5a9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0110af0 con 0x7f3da01066b0 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.791+0000 7f3d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45474/0 (socket says 172.21.15.116:45474) 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da01006a0 0x7f3da0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 -- 172.21.15.116:0/1548304993 learned_addr learned my addr 172.21.15.116:0/1548304993 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 -- 172.21.15.116:0/1548304993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 msgr2=0x7f3da019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 -- 172.21.15.116:0/1548304993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da01006a0 msgr2=0x7f3da0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da01006a0 0x7f3da0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 -- 172.21.15.116:0/1548304993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da019cfb0 con 0x7f3da0101060 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009230 tx=0x7f3d9402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9cff9640 1 -- 172.21.15.116:0/1548304993 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94043020 con 0x7f3da0101060 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da019d240 con 0x7f3da0101060 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9cff9640 1 -- 172.21.15.116:0/1548304993 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d9402ca70 con 0x7f3da0101060 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3d9cff9640 1 -- 172.21.15.116:0/1548304993 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003a50 con 0x7f3da0101060 2024-03-19T02:51:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.792+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da0102c90 con 0x7f3da0101060 2024-03-19T02:51:11.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 Processor -- start 2024-03-19T02:51:11.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 -- start start 2024-03-19T02:51:11.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3d9cff9640 1 -- 172.21.15.116:0/1548304993 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3d94046020 con 0x7f3da0101060 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 0x56338db6d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6dbf0 0x56338db76290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56338dad04b0 con 0x56338db767d0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56338da78d70 con 0x56338db6dbf0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3da8763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56338dacc550 con 0x56338db6d2d0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.795+0000 7f3d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45484/0 (socket says 172.21.15.116:45484) 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 -- 172.21.15.116:0/3606386014 learned_addr learned my addr 172.21.15.116:0/3606386014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6dbf0 0x56338db76290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 0x56338db6d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 -- 172.21.15.116:0/3606386014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 msgr2=0x56338db6d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 0x56338db6d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 -- 172.21.15.116:0/3606386014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6dbf0 msgr2=0x56338db76290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6dbf0 0x56338db76290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 -- 172.21.15.116:0/3606386014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338da9b210 con 0x56338db767d0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 0x56338db6d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9cff9640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 0x7f3d880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f3d940ee460 tx=0x7f3d940f9270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cc04bdd1685d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 0x7f3d880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.796+0000 7f3d9cff9640 1 -- 172.21.15.116:0/1548304993 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f3d940f10c0 con 0x7f3da0101060 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3d85ffb640 1 -- 172.21.15.116:0/3606386014 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940b96a0 con 0x56338db767d0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3d85ffb640 1 -- 172.21.15.116:0/3606386014 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d940bdb00 con 0x56338db767d0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3d85ffb640 1 -- 172.21.15.116:0/3606386014 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940bcd60 con 0x56338db767d0 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 -- 172.21.15.116:0/3606386014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 msgr2=0x56338db78bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 0x7f3d880a4eb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009fd0 tx=0x7f3d98009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f3d940ee460 tx=0x7f3d940f9270 comp rx=0 tx=0).stop 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 -- 172.21.15.116:0/3606386014 shutdown_connections 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db78bc0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6dbf0 0x56338db76290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 --2- 172.21.15.116:0/3606386014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6d2d0 0x56338db6d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 -- 172.21.15.116:0/3606386014 >> 172.21.15.116:0/3606386014 conn(0x56338db67a00 msgr2=0x56338db67de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 -- 172.21.15.116:0/3606386014 shutdown_connections 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.797+0000 7f3da8763740 1 -- 172.21.15.116:0/3606386014 wait complete. 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 -- start start 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 Processor -- start 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db73d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56338dad04b0 con 0x56338db767d0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56338da8c310 con 0x56338db6d2d0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56338db79380 con 0x56338db6dbf0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3da8763740 10 client.0 _ll_register_callbacks cb 0x56338da802d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db73d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db742b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45488/0 (socket says 172.21.15.116:45488) 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9ffff640 1 -- 172.21.15.116:0/1099679975 learned_addr learned my addr 172.21.15.116:0/1099679975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.798+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34008/0 (socket says 172.21.15.116:34008) 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 -- 172.21.15.116:0/1099679975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 msgr2=0x56338db73830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 -- 172.21.15.116:0/1099679975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 msgr2=0x56338db742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db742b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 -- 172.21.15.116:0/1099679975 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338d7e43f0 con 0x56338db6dbf0 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d9effd640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db73d70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009900 tx=0x7f3d940f13c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d2ae5c3a1fb7f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d7effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940bcf40 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d940bdb00 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f3d940bb700 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3d940bd1d0 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3da8763740 10 client.31810 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.799+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56338d8ae7d0 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d940bdb00 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f3d940b9a70 con 0x56338db6dbf0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 10 client.31810 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 10 client.31810 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 20 client.31810 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338db88a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 4 client.31810 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56338db2e3e0 con 0x56338db86680 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338db88a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3d7effd640 10 client.31810 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.800+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338db88a70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9888dbda2012c4ab server_cookie=6c227a8ee6b05180 in_seq=0 out_seq=0 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3d7effd640 1 -- 172.21.15.116:0/1099679975 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56338db2e3e0 con 0x56338db86680 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3d7effd640 10 client.31810 handle_command_reply: tid=0 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 client.31810 shutdown 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 msgr2=0x56338db73d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db73d70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009900 tx=0x7f3d940f13c0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 shutdown_connections 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338db88a70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db767d0 0x56338db742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.801+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db73d70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3d7effd640 0 client.31810 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3da8763740 1 --2- 172.21.15.116:0/1099679975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db73830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 >> 172.21.15.116:0/1099679975 conn(0x56338db67a00 msgr2=0x56338db69750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 shutdown_connections 2024-03-19T02:51:11.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3da8763740 1 -- 172.21.15.116:0/1099679975 wait complete. 2024-03-19T02:51:11.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.802+0000 7f3da8763740 20 client.31810 trim_cache size 0 max 16384 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 Processor -- start 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 -- start start 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db75f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6dbf0 0x56338db81fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01acb80 con 0x56338db6dbf0 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01ae6c0 con 0x56338db6d2d0 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3da8763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01ad250 con 0x56338db767d0 2024-03-19T02:51:11.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db75f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db75f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34012/0 (socket says 172.21.15.116:34012) 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46288/0 (socket says 172.21.15.116:46288) 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9f7fe640 1 -- 172.21.15.116:0/44781875 learned_addr learned my addr 172.21.15.116:0/44781875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9effd640 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6dbf0 0x56338db81fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 -- 172.21.15.116:0/44781875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 msgr2=0x56338db75f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db75f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 -- 172.21.15.116:0/44781875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6dbf0 msgr2=0x56338db81fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6dbf0 0x56338db81fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.821+0000 7f3d9ffff640 1 -- 172.21.15.116:0/44781875 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56338db6f400 con 0x56338db767d0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.822+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005fd0 tx=0x7f3d9801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c52fa4ba612a345 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.822+0000 7f3d857fa640 1 -- 172.21.15.116:0/44781875 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98023bb0 con 0x56338db767d0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.822+0000 7f3d857fa640 1 -- 172.21.15.116:0/44781875 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d98034070 con 0x56338db767d0 2024-03-19T02:51:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.822+0000 7f3d857fa640 1 -- 172.21.15.116:0/44781875 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980258e0 con 0x56338db767d0 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- 172.21.15.116:0/44781875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 msgr2=0x56338db8e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005fd0 tx=0x7f3d9801a000 comp rx=0 tx=0).stop 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- 172.21.15.116:0/44781875 shutdown_connections 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db767d0 0x56338db8e160 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6dbf0 0x56338db81fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 --2- 172.21.15.116:0/44781875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db6d2d0 0x56338db75f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- 172.21.15.116:0/44781875 >> 172.21.15.116:0/44781875 conn(0x56338db67a00 msgr2=0x56338db6dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- 172.21.15.116:0/44781875 shutdown_connections 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- 172.21.15.116:0/44781875 wait complete. 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.823+0000 7f3da8763740 1 -- start start 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 Processor -- start 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db79c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db767d0 0x56338db7a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01adff0 con 0x56338db6d2d0 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01ad920 con 0x56338db767d0 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01aab80 con 0x56338db6dbf0 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3da8763740 10 client.0 _ll_register_callbacks cb 0x7f3d900097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45498/0 (socket says 172.21.15.116:45498) 2024-03-19T02:51:11.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 -- 172.21.15.116:0/3385856005 learned_addr learned my addr 172.21.15.116:0/3385856005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9ffff640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db767d0 0x56338db7a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db79c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 -- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 msgr2=0x56338db79c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db79c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 -- 172.21.15.116:0/3385856005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db767d0 msgr2=0x56338db7a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db767d0 0x56338db7a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.824+0000 7f3d9f7fe640 1 -- 172.21.15.116:0/3385856005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56338db7a9d0 con 0x56338db6d2d0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3d9f7fe640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f3d900129d0 tx=0x7f3d90012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b870cf3d0ac617c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3d7effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90004230 con 0x56338db6d2d0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9598+0+0 (secure 0 0 0) 0x7f3d900043d0 con 0x56338db6d2d0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d98034070 con 0x56338db6d2d0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3d90030050 con 0x56338db6d2d0 2024-03-19T02:51:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3da8763740 10 client.22119 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.825+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56338dbec850 con 0x56338db6d2d0 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d98034070 con 0x56338db6d2d0 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f3d90022640 con 0x56338db6d2d0 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 10 client.22119 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 10 client.22119 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 20 client.22119 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338dbef060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 4 client.22119 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56338dadd9f0 con 0x56338db86680 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.826+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338dbef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.827+0000 7f3d7effd640 10 client.22119 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.827+0000 7f3d9effd640 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338dbef060 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1dbb47cf6de250b server_cookie=e49611b3de73b4fe in_seq=0 out_seq=0 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.827+0000 7f3d7effd640 1 -- 172.21.15.116:0/3385856005 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56338dadd9f0 con 0x56338db86680 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.827+0000 7f3d7effd640 10 client.22119 handle_command_reply: tid=0 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.827+0000 7f3da8763740 1 client.22119 shutdown 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 msgr2=0x56338db7e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f3d900129d0 tx=0x7f3d90012e90 comp rx=0 tx=0).stop 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 shutdown_connections 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56338db767d0 0x56338db7a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56338db6dbf0 0x56338db79c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x56338db86680 0x56338dbef060 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 --2- 172.21.15.116:0/3385856005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56338db6d2d0 0x56338db7e8f0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3d7effd640 0 client.22119 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.829+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 >> 172.21.15.116:0/3385856005 conn(0x56338db67a00 msgr2=0x56338db737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.830+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 shutdown_connections 2024-03-19T02:51:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.830+0000 7f3da8763740 1 -- 172.21.15.116:0/3385856005 wait complete. 2024-03-19T02:51:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.830+0000 7f3da8763740 20 client.22119 trim_cache size 0 max 16384 2024-03-19T02:51:11.834 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:11.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 msgr2=0x7f3d880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 0x7f3d880a4eb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009fd0 tx=0x7f3d98009500 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 msgr2=0x7f3da0198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009230 tx=0x7f3d9402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 shutdown_connections 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f3d880a29f0 0x7f3d880a4eb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da01066b0 0x7f3da019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da0101060 0x7f3da0198500 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 --2- 172.21.15.116:0/1548304993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3da01006a0 0x7f3da0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 >> 172.21.15.116:0/1548304993 conn(0x7f3da00778d0 msgr2=0x7f3da0068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 shutdown_connections 2024-03-19T02:51:11.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.832+0000 7f3da5a9a640 1 -- 172.21.15.116:0/1548304993 wait complete. 2024-03-19T02:51:11.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:51:11.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 -- 172.21.15.116:0/27073192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0100970 msgr2=0x7f7af0100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 --2- 172.21.15.116:0/27073192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0100970 0x7f7af0100df0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009900 tx=0x7f7ae002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 -- 172.21.15.116:0/27073192 shutdown_connections 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 --2- 172.21.15.116:0/27073192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0101330 0x7f7af010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 --2- 172.21.15.116:0/27073192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0100970 0x7f7af0100df0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 --2- 172.21.15.116:0/27073192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0105690 0x7f7af0105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 -- 172.21.15.116:0/27073192 >> 172.21.15.116:0/27073192 conn(0x7f7af0077580 msgr2=0x7f7af0077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.973+0000 7f7af7d22640 1 -- 172.21.15.116:0/27073192 shutdown_connections 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 -- 172.21.15.116:0/27073192 wait complete. 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 Processor -- start 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 -- start start 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0101330 0x7f7af01993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 0x7f7af01a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01109d0 con 0x7f7af0105690 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0110850 con 0x7f7af0101330 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.974+0000 7f7af7d22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0110b50 con 0x7f7af0100970 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af62bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 0x7f7af01a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46318/0 (socket says 172.21.15.116:46318) 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 -- 172.21.15.116:0/4037905944 learned_addr learned my addr 172.21.15.116:0/4037905944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af52bd640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0101330 0x7f7af01993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 -- 172.21.15.116:0/4037905944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0101330 msgr2=0x7f7af01993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0101330 0x7f7af01993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 -- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 msgr2=0x7f7af01a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 0x7f7af01a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 -- 172.21.15.116:0/4037905944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af01a4430 con 0x7f7af0100970 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af62bf640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 0x7f7af01a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af5abe640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e9f0 tx=0x7f7aec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7ae6ffd640 1 -- 172.21.15.116:0/4037905944 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00cd80 con 0x7f7af0100970 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af01a4600 con 0x7f7af0100970 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7ae6ffd640 1 -- 172.21.15.116:0/4037905944 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7aec0108e0 con 0x7f7af0100970 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7ae6ffd640 1 -- 172.21.15.116:0/4037905944 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec018920 con 0x7f7af0100970 2024-03-19T02:51:11.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.975+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01a4b20 con 0x7f7af0100970 2024-03-19T02:51:11.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 Processor -- start 2024-03-19T02:51:11.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 -- start start 2024-03-19T02:51:11.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7ae6ffd640 1 -- 172.21.15.116:0/4037905944 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7aec010a90 con 0x7f7af0100970 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d99896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d9989bf0 0x55d8d9992290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d8d98ec4b0 con 0x55d8d99892d0 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d8d9894d70 con 0x55d8d99927d0 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.979+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8d98e8550 con 0x55d8d9989bf0 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34048/0 (socket says 172.21.15.116:34048) 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 -- 172.21.15.116:0/119103391 learned_addr learned my addr 172.21.15.116:0/119103391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af62bf640 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d99896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7ae6ffd640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 0x7f7adc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af5abe640 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d9989bf0 0x55d8d9992290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 -- 172.21.15.116:0/119103391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d9989bf0 msgr2=0x55d8d9992290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d9989bf0 0x55d8d9992290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 -- 172.21.15.116:0/119103391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 msgr2=0x55d8d99896b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d99896b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af52bd640 1 -- 172.21.15.116:0/119103391 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8d98b7210 con 0x55d8d99927d0 2024-03-19T02:51:11.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7af62bf640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 0x7f7adc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.980+0000 7f7ae6ffd640 1 -- 172.21.15.116:0/4037905944 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7aec0cf4f0 con 0x7f7af0100970 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.981+0000 7f7af52bd640 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae0005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d93f10fc1245fa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.981+0000 7f7af62bf640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 0x7f7adc0a4e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae8004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.981+0000 7f7adbfff640 1 -- 172.21.15.116:0/119103391 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0045020 con 0x55d8d99927d0 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.981+0000 7f7adbfff640 1 -- 172.21.15.116:0/119103391 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ae002c780 con 0x55d8d99927d0 2024-03-19T02:51:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.981+0000 7f7adbfff640 1 -- 172.21.15.116:0/119103391 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00046c0 con 0x55d8d99927d0 2024-03-19T02:51:11.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- 172.21.15.116:0/119103391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 msgr2=0x55d8d9994bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae0005e10 comp rx=0 tx=0).stop 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- 172.21.15.116:0/119103391 shutdown_connections 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99927d0 0x55d8d9994bc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d9989bf0 0x55d8d9992290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/119103391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d99896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- 172.21.15.116:0/119103391 >> 172.21.15.116:0/119103391 conn(0x55d8d9983a00 msgr2=0x55d8d9983de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- 172.21.15.116:0/119103391 shutdown_connections 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- 172.21.15.116:0/119103391 wait complete. 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 -- start start 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.983+0000 7f7afa9ef740 1 Processor -- start 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d998f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99902f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d8d9995560 con 0x55d8d9989bf0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d8d98a8310 con 0x55d8d99892d0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8d99956e0 con 0x55d8d99927d0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7afa9ef740 10 client.0 _ll_register_callbacks cb 0x55d8d989c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45522/0 (socket says 172.21.15.116:45522) 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 -- 172.21.15.116:0/1234852072 learned_addr learned my addr 172.21.15.116:0/1234852072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af62bf640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99902f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af5abe640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d998f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 -- 172.21.15.116:0/1234852072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 msgr2=0x55d8d99902f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99902f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 -- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 msgr2=0x55d8d998f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d998f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 -- 172.21.15.116:0/1234852072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8d96003f0 con 0x55d8d9989bf0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7af52bd640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0036040 tx=0x7f7ae002cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9fddf740cfa342 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.984+0000 7f7ad97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0045070 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7ae00035f0 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae002c780 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7ae005b050 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 10 client.22131 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d8d96ca7d0 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ae002c780 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7ae00417b0 con 0x55d8d9989bf0 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 10 client.22131 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 10 client.22131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 20 client.22131 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d99a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 4 client.22131 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.985+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8d994a3e0 con 0x55d8d99a2a10 2024-03-19T02:51:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.986+0000 7f7af5abe640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d99a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.986+0000 7f7ad97fa640 10 client.22131 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.986+0000 7f7af5abe640 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d99a4e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4711cd5527392911 server_cookie=2883696774ed1b46 in_seq=0 out_seq=0 2024-03-19T02:51:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.987+0000 7f7ad97fa640 1 -- 172.21.15.116:0/1234852072 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8d994a3e0 con 0x55d8d99a2a10 2024-03-19T02:51:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.987+0000 7f7ad97fa640 10 client.22131 handle_command_reply: tid=0 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.987+0000 7f7afa9ef740 1 client.22131 shutdown 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 msgr2=0x55d8d998fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0036040 tx=0x7f7ae002cb60 comp rx=0 tx=0).stop 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 shutdown_connections 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d99a4e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99902f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7ad97fa640 0 client.22131 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d998fdb0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/1234852072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d998f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 >> 172.21.15.116:0/1234852072 conn(0x55d8d9983a00 msgr2=0x55d8d9985750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 shutdown_connections 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 1 -- 172.21.15.116:0/1234852072 wait complete. 2024-03-19T02:51:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:11.988+0000 7f7afa9ef740 20 client.22131 trim_cache size 0 max 16384 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.010+0000 7f7afa9ef740 1 Processor -- start 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 -- start start 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d9989bf0 0x55d8d999e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01aeb90 con 0x55d8d99892d0 2024-03-19T02:51:12.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01b06d0 con 0x55d8d9989bf0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01af260 con 0x55d8d99927d0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45536/0 (socket says 172.21.15.116:45536) 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 -- 172.21.15.116:0/2191774475 learned_addr learned my addr 172.21.15.116:0/2191774475 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af62bf640 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af52bd640 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d9989bf0 0x55d8d999e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 -- 172.21.15.116:0/2191774475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 msgr2=0x55d8d99aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 -- 172.21.15.116:0/2191774475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d9989bf0 msgr2=0x55d8d999e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d9989bf0 0x55d8d999e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 -- 172.21.15.116:0/2191774475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8d9959880 con 0x55d8d99892d0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.011+0000 7f7af5abe640 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c6a0 tx=0x7f7aec086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925e352d84740955 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.012+0000 7f7adb7fe640 1 -- 172.21.15.116:0/2191774475 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0a3f10 con 0x55d8d99892d0 2024-03-19T02:51:12.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.012+0000 7f7adb7fe640 1 -- 172.21.15.116:0/2191774475 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7aec099b20 con 0x55d8d99892d0 2024-03-19T02:51:12.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.012+0000 7f7adb7fe640 1 -- 172.21.15.116:0/2191774475 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0cf170 con 0x55d8d99892d0 2024-03-19T02:51:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- 172.21.15.116:0/2191774475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 msgr2=0x55d8d9991fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c6a0 tx=0x7f7aec086040 comp rx=0 tx=0).stop 2024-03-19T02:51:12.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- 172.21.15.116:0/2191774475 shutdown_connections 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d99aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d9989bf0 0x55d8d999e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/2191774475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d99892d0 0x55d8d9991fa0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- 172.21.15.116:0/2191774475 >> 172.21.15.116:0/2191774475 conn(0x55d8d9983a00 msgr2=0x55d8d9982310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- 172.21.15.116:0/2191774475 shutdown_connections 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- 172.21.15.116:0/2191774475 wait complete. 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- start start 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 Processor -- start 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d99962c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d9996d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01b0000 con 0x55d8d9989bf0 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01af930 con 0x55d8d99892d0 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01acb90 con 0x55d8d99927d0 2024-03-19T02:51:12.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.013+0000 7f7afa9ef740 10 client.0 _ll_register_callbacks cb 0x7f7ae0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45548/0 (socket says 172.21.15.116:45548) 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 -- 172.21.15.116:0/463465499 learned_addr learned my addr 172.21.15.116:0/463465499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af5abe640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d99962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af62bf640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d9996d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 -- 172.21.15.116:0/463465499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 msgr2=0x55d8d9996d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d9996d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 -- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 msgr2=0x55d8d99962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d99962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 -- 172.21.15.116:0/463465499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8d9a07b30 con 0x55d8d9989bf0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7af52bd640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f7ae002c630 tx=0x7f7ae002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ff1d66f5ca87dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7ad97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0045040 con 0x55d8d9989bf0 2024-03-19T02:51:12.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7ae0041680 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.014+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aec099b20 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7ae002c780 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 10 client.22137 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d8d9a08e30 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7aec099b20 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7ae0034440 con 0x55d8d9989bf0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 10 client.22137 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 10 client.22137 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 20 client.22137 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d9a0b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 4 client.22137 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8d98f99f0 con 0x55d8d99a2a10 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.015+0000 7f7af5abe640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d9a0b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7af5abe640 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d9a0b3a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e25de6c97c418d14 server_cookie=4a86c1367fd4104c in_seq=0 out_seq=0 2024-03-19T02:51:12.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7ad97fa640 10 client.22137 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7ad97fa640 1 -- 172.21.15.116:0/463465499 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8d98f99f0 con 0x55d8d99a2a10 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7ad97fa640 10 client.22137 handle_command_reply: tid=0 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 client.22137 shutdown 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 msgr2=0x55d8d9996800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f7ae002c630 tx=0x7f7ae002c680 comp rx=0 tx=0).stop 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 shutdown_connections 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55d8d99a2a10 0x55d8d9a0b3a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8d99927d0 0x55d8d9996d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7ad97fa640 0 client.22137 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d8d9989bf0 0x55d8d9996800 secure :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f7ae002c630 tx=0x7f7ae002c680 comp rx=0 tx=0).stop 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 --2- 172.21.15.116:0/463465499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d8d99892d0 0x55d8d99962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.016+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 >> 172.21.15.116:0/463465499 conn(0x55d8d9983a00 msgr2=0x55d8d9982ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 shutdown_connections 2024-03-19T02:51:12.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7afa9ef740 1 -- 172.21.15.116:0/463465499 wait complete. 2024-03-19T02:51:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7afa9ef740 20 client.22137 trim_cache size 0 max 16384 2024-03-19T02:51:12.023 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:12.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 msgr2=0x7f7adc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 0x7f7adc0a4e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae8004ab0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 msgr2=0x7f7af0198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.017+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e9f0 tx=0x7f7aec00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 shutdown_connections 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7adc0a2970 0x7f7adc0a4e30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af0105690 0x7f7af01a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0101330 0x7f7af01993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 --2- 172.21.15.116:0/4037905944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af0100970 0x7f7af0198e70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 >> 172.21.15.116:0/4037905944 conn(0x7f7af0077580 msgr2=0x7f7af00fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 shutdown_connections 2024-03-19T02:51:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.018+0000 7f7af7d22640 1 -- 172.21.15.116:0/4037905944 wait complete. 2024-03-19T02:51:12.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- 172.21.15.116:0/1108320262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f4101180 msgr2=0x7fb0f4101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/1108320262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f4101180 0x7fb0f4101600 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009900 tx=0x7fb0e802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- 172.21.15.116:0/1108320262 shutdown_connections 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/1108320262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101b40 0x7fb0f410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/1108320262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f4101180 0x7fb0f4101600 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/1108320262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f41068a0 0x7fb0f4106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- 172.21.15.116:0/1108320262 >> 172.21.15.116:0/1108320262 conn(0x7fb0f4075a20 msgr2=0x7fb0f4075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- 172.21.15.116:0/1108320262 shutdown_connections 2024-03-19T02:51:12.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- 172.21.15.116:0/1108320262 wait complete. 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 Processor -- start 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.165+0000 7fb0f99f4640 1 -- start start 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f4101b40 0x7fb0f410d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f41068a0 0x7fb0f4108e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410f8c0 con 0x7fb0f41068a0 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410f740 con 0x7fb0f4101b40 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f99f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410fa40 con 0x7fb0f4101180 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46368/0 (socket says 172.21.15.116:46368) 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f41068a0 0x7fb0f4108e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/183221444 learned_addr learned my addr 172.21.15.116:0/183221444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f4101b40 0x7fb0f410d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/183221444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f4101b40 msgr2=0x7fb0f410d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f4101b40 0x7fb0f410d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f41068a0 msgr2=0x7fb0f4108e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f41068a0 0x7fb0f4108e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.166+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/183221444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f4109620 con 0x7fb0f4101180 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400e9f0 tx=0x7fb0e400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f89f2640 1 -- 172.21.15.116:0/183221444 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e400cd80 con 0x7fb0f4101180 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f4109910 con 0x7fb0f4101180 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f89f2640 1 -- 172.21.15.116:0/183221444 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e40108e0 con 0x7fb0f4101180 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f89f2640 1 -- 172.21.15.116:0/183221444 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4018920 con 0x7fb0f4101180 2024-03-19T02:51:12.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.167+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f41ab010 con 0x7fb0f4101180 2024-03-19T02:51:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 Processor -- start 2024-03-19T02:51:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 -- start start 2024-03-19T02:51:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec34f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec351bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5634ec2a94b0 con 0x5634ec346bf0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5634ec251d70 con 0x5634ec34f7d0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634ec2a5550 con 0x5634ec3462d0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec351bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46374/0 (socket says 172.21.15.116:46374) 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f37fe640 1 -- 172.21.15.116:0/3313863765 learned_addr learned my addr 172.21.15.116:0/3313863765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f89f2640 1 -- 172.21.15.116:0/183221444 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb0e4010a90 con 0x7fb0f4101180 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.170+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec34f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3313863765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 msgr2=0x5634ec3466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3313863765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 msgr2=0x5634ec34f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec34f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3313863765 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ec274210 con 0x5634ec34f7d0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec351bc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb0e802c210 tx=0x7fb0e80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9104c1235d30ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0d9ffb640 1 -- 172.21.15.116:0/3313863765 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8004040 con 0x5634ec34f7d0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f89f2640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 0x7fb0dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0d9ffb640 1 -- 172.21.15.116:0/3313863765 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e802cd30 con 0x5634ec34f7d0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0d9ffb640 1 -- 172.21.15.116:0/3313863765 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8005540 con 0x5634ec34f7d0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f89f2640 1 -- 172.21.15.116:0/183221444 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fb0e40cf5c0 con 0x7fb0f4101180 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.171+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 0x7fb0dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3313863765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 msgr2=0x5634ec351bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 0x7fb0dc0a4e30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec011870 tx=0x7fb0ec009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec351bc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb0e802c210 tx=0x7fb0e80095e0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3313863765 shutdown_connections 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec351bc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec34f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3313863765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec3466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3313863765 >> 172.21.15.116:0/3313863765 conn(0x5634ec340a00 msgr2=0x5634ec340de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3313863765 shutdown_connections 2024-03-19T02:51:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3313863765 wait complete. 2024-03-19T02:51:12.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.172+0000 7fb0fc5de740 1 -- start start 2024-03-19T02:51:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 Processor -- start 2024-03-19T02:51:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec34c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec34f7d0 0x5634ec34d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5634ec352600 con 0x5634ec34f7d0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5634ec265310 con 0x5634ec3462d0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634ec352780 con 0x5634ec346bf0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0fc5de740 10 client.0 _ll_register_callbacks cb 0x5634ec2592d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec34c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec34c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34126/0 (socket says 172.21.15.116:34126) 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46388/0 (socket says 172.21.15.116:46388) 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3953194386 learned_addr learned my addr 172.21.15.116:0/3953194386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec34f7d0 0x5634ec34d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3953194386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 msgr2=0x5634ec34c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec34c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3953194386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec34f7d0 msgr2=0x5634ec34d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec34f7d0 0x5634ec34d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.173+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3953194386 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634ebfbd3f0 con 0x5634ec346bf0 2024-03-19T02:51:12.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8005c40 tx=0x7fb0e80054b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded69fff3ded736c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8004240 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0e802cd30 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fb0e80043e0 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb0e8005620 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 10 client.31846 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5634ec0877d0 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e802cd30 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fb0e8003710 con 0x5634ec346bf0 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 10 client.31846 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 10 client.31846 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 20 client.31846 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec35fca0 0x5634ec3620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 4 client.31846 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.174+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634ec3073e0 con 0x5634ec35fca0 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.175+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec35fca0 0x5634ec3620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.175+0000 7fb0d2ffd640 10 client.31846 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.175+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec35fca0 0x5634ec3620c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91a88d5b5b11934 server_cookie=cba3c96fad7591 in_seq=0 out_seq=0 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/3953194386 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5634ec3073e0 con 0x5634ec35fca0 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0d2ffd640 10 client.31846 handle_command_reply: tid=0 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 client.31846 shutdown 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 msgr2=0x5634ec34cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8005c40 tx=0x7fb0e80054b0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 shutdown_connections 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec35fca0 0x5634ec3620c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec34f7d0 0x5634ec34d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec346bf0 0x5634ec34cd60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3953194386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec34c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0d2ffd640 0 client.31846 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 >> 172.21.15.116:0/3953194386 conn(0x5634ec340a00 msgr2=0x5634ec342750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 shutdown_connections 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3953194386 wait complete. 2024-03-19T02:51:12.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.176+0000 7fb0fc5de740 20 client.31846 trim_cache size 0 max 16384 2024-03-19T02:51:12.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 Processor -- start 2024-03-19T02:51:12.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 -- start start 2024-03-19T02:51:12.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec35b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec34f7d0 0x5634ec3679d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b51d0 con 0x5634ec346bf0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b6d10 con 0x5634ec3462d0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b58a0 con 0x5634ec34f7d0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45576/0 (socket says 172.21.15.116:45576) 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3673711582 learned_addr learned my addr 172.21.15.116:0/3673711582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec35b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.192+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec34f7d0 0x5634ec3679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3673711582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec34f7d0 msgr2=0x5634ec3679d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec34f7d0 0x5634ec3679d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3673711582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 msgr2=0x5634ec35b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec35b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 -- 172.21.15.116:0/3673711582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ec3455f0 con 0x5634ec346bf0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8050490 tx=0x7fb0e8009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad6ba72037c94e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0d97fa640 1 -- 172.21.15.116:0/3673711582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e80037f0 con 0x5634ec346bf0 2024-03-19T02:51:12.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0d97fa640 1 -- 172.21.15.116:0/3673711582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e80041d0 con 0x5634ec346bf0 2024-03-19T02:51:12.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.193+0000 7fb0d97fa640 1 -- 172.21.15.116:0/3673711582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8004c70 con 0x5634ec346bf0 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3673711582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 msgr2=0x5634ec33fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8050490 tx=0x7fb0e8009660 comp rx=0 tx=0).stop 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3673711582 shutdown_connections 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec34f7d0 0x5634ec3679d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec346bf0 0x5634ec33fa20 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/3673711582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec3462d0 0x5634ec35b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3673711582 >> 172.21.15.116:0/3673711582 conn(0x5634ec340a00 msgr2=0x5634ec33f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3673711582 shutdown_connections 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- 172.21.15.116:0/3673711582 wait complete. 2024-03-19T02:51:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 -- start start 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.194+0000 7fb0fc5de740 1 Processor -- start 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec357bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec349050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b6640 con 0x5634ec349590 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b5f70 con 0x5634ec34f7d0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f41b31d0 con 0x5634ec3462d0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0fc5de740 10 client.0 _ll_register_callbacks cb 0x7fb0f410d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec357bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45580/0 (socket says 172.21.15.116:45580) 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 -- 172.21.15.116:0/600217338 learned_addr learned my addr 172.21.15.116:0/600217338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec349050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 -- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 msgr2=0x5634ec357bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec357bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 -- 172.21.15.116:0/600217338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 msgr2=0x5634ec349050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec349050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 -- 172.21.15.116:0/600217338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634ec3c5000 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec357bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f27fc640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec349050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0f37fe640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003c40 tx=0x7fb0ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841bade559fec63f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.195+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec004b20 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fb0ec017030 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0e80041d0 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fb0ec035050 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 10 client.22155 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5634ec3c62c0 con 0x5634ec349590 2024-03-19T02:51:12.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e80041d0 con 0x5634ec349590 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fb0ec028410 con 0x5634ec349590 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 10 client.22155 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 10 client.22155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 20 client.22155 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec3c6ac0 0x5634ec3c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 4 client.22155 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634ec2b69f0 con 0x5634ec3c6ac0 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.196+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec3c6ac0 0x5634ec3c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.197+0000 7fb0d2ffd640 10 client.22155 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.197+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec3c6ac0 0x5634ec3c8ee0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6239ad6be8c50f server_cookie=a44d96d6b2a0133a in_seq=0 out_seq=0 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.197+0000 7fb0d2ffd640 1 -- 172.21.15.116:0/600217338 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634ec2b69f0 con 0x5634ec3c6ac0 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.197+0000 7fb0d2ffd640 10 client.22155 handle_command_reply: tid=0 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.197+0000 7fb0fc5de740 1 client.22155 shutdown 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 msgr2=0x5634ec349a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec003c40 tx=0x7fb0ec004a40 comp rx=0 tx=0).stop 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 shutdown_connections 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x5634ec3c6ac0 0x5634ec3c8ee0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0d2ffd640 0 client.22155 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5634ec349590 0x5634ec349a10 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5634ec34f7d0 0x5634ec349050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 --2- 172.21.15.116:0/600217338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634ec3462d0 0x5634ec357bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 >> 172.21.15.116:0/600217338 conn(0x5634ec340a00 msgr2=0x5634ec33fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 shutdown_connections 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 1 -- 172.21.15.116:0/600217338 wait complete. 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0fc5de740 20 client.22155 trim_cache size 0 max 16384 2024-03-19T02:51:12.204 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 msgr2=0x7fb0dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 0x7fb0dc0a4e30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec011870 tx=0x7fb0ec009500 comp rx=0 tx=0).stop 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 msgr2=0x7fb0f410cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.198+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400e9f0 tx=0x7fb0e400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 shutdown_connections 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb0dc0a2970 0x7fb0dc0a4e30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f41068a0 0x7fb0f4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0f4101b40 0x7fb0f410d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 --2- 172.21.15.116:0/183221444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0f4101180 0x7fb0f410cc50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 >> 172.21.15.116:0/183221444 conn(0x7fb0f4075a20 msgr2=0x7fb0f40fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 shutdown_connections 2024-03-19T02:51:12.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.199+0000 7fb0f99f4640 1 -- 172.21.15.116:0/183221444 wait complete. 2024-03-19T02:51:12.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:51:12.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68569 5.76 MB/sec execute 514 sec latency 137.379 ms 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 -- 172.21.15.116:0/312599832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 msgr2=0x7faf60102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 --2- 172.21.15.116:0/312599832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60102e20 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7faf50009900 tx=0x7faf5002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 -- 172.21.15.116:0/312599832 shutdown_connections 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 --2- 172.21.15.116:0/312599832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 0x7faf6010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 --2- 172.21.15.116:0/312599832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 0x7faf6010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 --2- 172.21.15.116:0/312599832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60102e20 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 -- 172.21.15.116:0/312599832 >> 172.21.15.116:0/312599832 conn(0x7faf600778d0 msgr2=0x7faf60077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 -- 172.21.15.116:0/312599832 shutdown_connections 2024-03-19T02:51:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.344+0000 7faf64f53640 1 -- 172.21.15.116:0/312599832 wait complete. 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 Processor -- start 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 -- start start 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 0x7faf60100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 0x7faf60101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf60111a30 con 0x7faf60102a40 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf601118b0 con 0x7faf6010bd80 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf64f53640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf60111bb0 con 0x7faf60103360 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45594/0 (socket says 172.21.15.116:45594) 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 -- 172.21.15.116:0/44813565 learned_addr learned my addr 172.21.15.116:0/44813565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 0x7faf60100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 0x7faf60101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 -- 172.21.15.116:0/44813565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 msgr2=0x7faf60100bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 0x7faf60100bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 -- 172.21.15.116:0/44813565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 msgr2=0x7faf60101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 0x7faf60101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 -- 172.21.15.116:0/44813565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf601a6510 con 0x7faf60102a40 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.345+0000 7faf5e59c640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7faf50005940 tx=0x7faf5002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.346+0000 7faf4f7fe640 1 -- 172.21.15.116:0/44813565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50043020 con 0x7faf60102a40 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.346+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf601a67a0 con 0x7faf60102a40 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.346+0000 7faf4f7fe640 1 -- 172.21.15.116:0/44813565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf50004e00 con 0x7faf60102a40 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.346+0000 7faf4f7fe640 1 -- 172.21.15.116:0/44813565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50003970 con 0x7faf60102a40 2024-03-19T02:51:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.346+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf601a6cf0 con 0x7faf60102a40 2024-03-19T02:51:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.348+0000 7faf4f7fe640 1 -- 172.21.15.116:0/44813565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7faf50046020 con 0x7faf60102a40 2024-03-19T02:51:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 Processor -- start 2024-03-19T02:51:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 -- start start 2024-03-19T02:51:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6207e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf4f7fe640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 0x7faf480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b620877d0 0x561b62089bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b61fe14b0 con 0x561b6207e2d0 2024-03-19T02:51:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b61f89d70 con 0x561b6207ebf0 2024-03-19T02:51:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf67c1a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b61fdd550 con 0x561b620877d0 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf5e59c640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 0x7faf480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6207e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf5dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf5dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34178/0 (socket says 172.21.15.116:34178) 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf5dd9b640 1 -- 172.21.15.116:0/4007845763 learned_addr learned my addr 172.21.15.116:0/4007845763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.349+0000 7faf4f7fe640 1 -- 172.21.15.116:0/44813565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7faf500f10e0 con 0x7faf60102a40 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 -- 172.21.15.116:0/4007845763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b620877d0 msgr2=0x561b62089bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T02:51:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b620877d0 0x561b62089bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 -- 172.21.15.116:0/4007845763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 msgr2=0x561b6207e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6207e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 -- 172.21.15.116:0/4007845763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b61fac210 con 0x561b6207ebf0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6207e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5e59c640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 0x7faf480a4e30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faf500ee610 tx=0x7faf500057a0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faf54009c90 tx=0x7faf54011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5efc4ee291874ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf4cff9640 1 -- 172.21.15.116:0/4007845763 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5401aa50 con 0x561b6207ebf0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf4cff9640 1 -- 172.21.15.116:0/4007845763 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf54009e40 con 0x561b6207ebf0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.350+0000 7faf4cff9640 1 -- 172.21.15.116:0/4007845763 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54004b60 con 0x561b6207ebf0 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- 172.21.15.116:0/4007845763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 msgr2=0x561b62087290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faf54009c90 tx=0x7faf54011810 comp rx=0 tx=0).stop 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- 172.21.15.116:0/4007845763 shutdown_connections 2024-03-19T02:51:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b620877d0 0x561b62089bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62087290 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 --2- 172.21.15.116:0/4007845763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6207e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- 172.21.15.116:0/4007845763 >> 172.21.15.116:0/4007845763 conn(0x561b62078a00 msgr2=0x561b62078de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- 172.21.15.116:0/4007845763 shutdown_connections 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- 172.21.15.116:0/4007845763 wait complete. 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 -- start start 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.351+0000 7faf67c1a740 1 Processor -- start 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207e2d0 0x561b62084850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62084d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b6208a5c0 con 0x561b620877d0 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561b61f9d310 con 0x561b6207e2d0 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561b6208a740 con 0x561b6207ebf0 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf67c1a740 10 client.0 _ll_register_callbacks cb 0x561b61f912d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45610/0 (socket says 172.21.15.116:45610) 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 -- 172.21.15.116:0/3705555939 learned_addr learned my addr 172.21.15.116:0/3705555939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207e2d0 0x561b62084850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62084d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 -- 172.21.15.116:0/3705555939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 msgr2=0x561b62084d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62084d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 -- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207e2d0 msgr2=0x561b62084850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207e2d0 0x561b62084850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 -- 172.21.15.116:0/3705555939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b61cf53f0 con 0x561b620877d0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7faf58010f90 tx=0x7faf58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6904f01be24d7010 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.352+0000 7faf45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58019040 con 0x561b620877d0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7faf5800e3b0 con 0x561b620877d0 2024-03-19T02:51:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b6203f3e0 con 0x561b620877d0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7faf58020630 con 0x561b620877d0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 10 client.22167 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561b61dbf7d0 con 0x561b620877d0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561b6203f3e0 con 0x561b620877d0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7faf5800e7b0 con 0x561b620877d0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 10 client.22167 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 10 client.22167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 20 client.22167 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b62097c00 0x561b62099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 4 client.22167 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b61fee9f0 con 0x561b62097c00 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.353+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b62097c00 0x561b62099ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.354+0000 7faf45ffb640 10 client.22167 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.354+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b62097c00 0x561b62099ff0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316fc9227e3fd06 server_cookie=b04fbc44c4667cfd in_seq=0 out_seq=0 2024-03-19T02:51:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf45ffb640 1 -- 172.21.15.116:0/3705555939 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b61fee9f0 con 0x561b62097c00 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf45ffb640 10 client.22167 handle_command_reply: tid=0 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 client.22167 shutdown 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 msgr2=0x561b620852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7faf58010f90 tx=0x7faf58004a40 comp rx=0 tx=0).stop 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 shutdown_connections 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b62097c00 0x561b62099ff0 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b620852d0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62084d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf45ffb640 0 client.22167 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3705555939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207e2d0 0x561b62084850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 >> 172.21.15.116:0/3705555939 conn(0x561b62078a00 msgr2=0x561b6207a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 shutdown_connections 2024-03-19T02:51:12.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 1 -- 172.21.15.116:0/3705555939 wait complete. 2024-03-19T02:51:12.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.356+0000 7faf67c1a740 20 client.22167 trim_cache size 0 max 16384 2024-03-19T02:51:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.371+0000 7faf67c1a740 1 Processor -- start 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 -- start start 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62093360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b6207a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf601acaa0 con 0x561b620877d0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf601ae5e0 con 0x561b6207ebf0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf67c1a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf601ad170 con 0x561b6207e2d0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46444/0 (socket says 172.21.15.116:46444) 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 -- 172.21.15.116:0/1955448442 learned_addr learned my addr 172.21.15.116:0/1955448442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b6207a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62093360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 -- 172.21.15.116:0/1955448442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 msgr2=0x561b62093360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62093360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 -- 172.21.15.116:0/1955448442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 msgr2=0x561b6207a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b6207a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.372+0000 7faf5ed9d640 1 -- 172.21.15.116:0/1955448442 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b62079c30 con 0x561b6207e2d0 2024-03-19T02:51:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.373+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x561b6207e050 tx=0x7faf58004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7aac8f03c3494bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.373+0000 7faf47fff640 1 -- 172.21.15.116:0/1955448442 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58005bd0 con 0x561b6207e2d0 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.373+0000 7faf47fff640 1 -- 172.21.15.116:0/1955448442 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf58009480 con 0x561b6207e2d0 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.373+0000 7faf47fff640 1 -- 172.21.15.116:0/1955448442 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5800e3b0 con 0x561b6207e2d0 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 -- 172.21.15.116:0/1955448442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 msgr2=0x561b62086f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x561b6207e050 tx=0x7faf58004a40 comp rx=0 tx=0).stop 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 -- 172.21.15.116:0/1955448442 shutdown_connections 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b620877d0 0x561b6207a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b6207ebf0 0x561b62093360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 --2- 172.21.15.116:0/1955448442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207e2d0 0x561b62086f80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 -- 172.21.15.116:0/1955448442 >> 172.21.15.116:0/1955448442 conn(0x561b62078a00 msgr2=0x561b62082ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 -- 172.21.15.116:0/1955448442 shutdown_connections 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.374+0000 7faf67c1a740 1 -- 172.21.15.116:0/1955448442 wait complete. 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 -- start start 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 Processor -- start 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6208fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62081200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b620877d0 0x561b62081810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf601adf10 con 0x561b6207e2d0 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf601ad840 con 0x561b620877d0 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf601aaaa0 con 0x561b6207ebf0 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf67c1a740 10 client.0 _ll_register_callbacks cb 0x7faf58009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6208fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6208fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45620/0 (socket says 172.21.15.116:45620) 2024-03-19T02:51:12.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5ed9d640 1 -- 172.21.15.116:0/3788050537 learned_addr learned my addr 172.21.15.116:0/3788050537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62081200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b620877d0 0x561b62081810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 -- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b620877d0 msgr2=0x561b62081810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b620877d0 0x561b62081810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 -- 172.21.15.116:0/3788050537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 msgr2=0x561b6208fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6208fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.375+0000 7faf5dd9b640 1 -- 172.21.15.116:0/3788050537 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b62082010 con 0x561b6207ebf0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf5dd9b640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62081200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf54004650 tx=0x7faf54009890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f8bd04fea03eb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5403c070 con 0x561b6207ebf0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf58009480 con 0x561b6207ebf0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7faf54035a30 con 0x561b6207ebf0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7faf54048050 con 0x561b6207ebf0 2024-03-19T02:51:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf67c1a740 10 client.31867 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561b620fdf40 con 0x561b6207ebf0 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.376+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf58009480 con 0x561b6207ebf0 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7faf54041640 con 0x561b6207ebf0 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 10 client.31867 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 10 client.31867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 20 client.31867 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b620fe410 0x561b62100830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 4 client.31867 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b61fe8d40 con 0x561b620fe410 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b620fe410 0x561b62100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf45ffb640 10 client.31867 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.377+0000 7faf5ed9d640 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b620fe410 0x561b62100830 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca078921c15d27e7 server_cookie=437f732bbac7d8e8 in_seq=0 out_seq=0 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf45ffb640 1 -- 172.21.15.116:0/3788050537 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b61fe8d40 con 0x561b620fe410 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf45ffb640 10 client.31867 handle_command_reply: tid=0 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 client.31867 shutdown 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 msgr2=0x561b62081200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62081200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf54004650 tx=0x7faf54009890 comp rx=0 tx=0).stop 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 shutdown_connections 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x561b620fe410 0x561b62100830 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561b620877d0 0x561b62081810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf45ffb640 0 client.31867 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561b6207ebf0 0x561b62081200 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 --2- 172.21.15.116:0/3788050537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b6207e2d0 0x561b6208fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 >> 172.21.15.116:0/3788050537 conn(0x561b62078a00 msgr2=0x561b62082ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 shutdown_connections 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 1 -- 172.21.15.116:0/3788050537 wait complete. 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.382+0000 7faf67c1a740 20 client.31867 trim_cache size 0 max 16384 2024-03-19T02:51:12.385 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 msgr2=0x7faf480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 0x7faf480a4e30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faf500ee610 tx=0x7faf500057a0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 msgr2=0x7faf60100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7faf50005940 tx=0x7faf5002c690 comp rx=0 tx=0).stop 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 shutdown_connections 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7faf480a2970 0x7faf480a4e30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf6010bd80 0x7faf60101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf60103360 0x7faf60100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 --2- 172.21.15.116:0/44813565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60102a40 0x7faf60100690 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 >> 172.21.15.116:0/44813565 conn(0x7faf600778d0 msgr2=0x7faf600feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 shutdown_connections 2024-03-19T02:51:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:12.383+0000 7faf64f53640 1 -- 172.21.15.116:0/44813565 wait complete. 2024-03-19T02:51:12.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:12 smithi116 ceph-mon[32263]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-03-19T02:51:12.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:12 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1915628597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:12.396 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:12 smithi116 ceph-mon[32263]: osdmap e131: 12 total, 12 up, 12 in 2024-03-19T02:51:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:12 smithi162 ceph-mon[35761]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-03-19T02:51:12.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:12 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1915628597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:12.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:12 smithi162 ceph-mon[35761]: osdmap e131: 12 total, 12 up, 12 in 2024-03-19T02:51:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:12 smithi154 ceph-mon[35693]: pgmap v739: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-03-19T02:51:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:12 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1915628597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:12.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:12 smithi154 ceph-mon[35693]: osdmap e131: 12 total, 12 up, 12 in 2024-03-19T02:51:13.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68645 5.75 MB/sec execute 515 sec latency 189.079 ms 2024-03-19T02:51:13.398 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:51:13.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:13.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:51:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 -- 172.21.15.116:0/3151492807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 msgr2=0x7f4490100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 --2- 172.21.15.116:0/3151492807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f4490100e10 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f4480009900 tx=0x7f448002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 -- 172.21.15.116:0/3151492807 shutdown_connections 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 --2- 172.21.15.116:0/3151492807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4490101350 0x7f449010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 --2- 172.21.15.116:0/3151492807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f4490100e10 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 --2- 172.21.15.116:0/3151492807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44901056a0 0x7f4490105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 -- 172.21.15.116:0/3151492807 >> 172.21.15.116:0/3151492807 conn(0x7f4490077610 msgr2=0x7f4490077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 -- 172.21.15.116:0/3151492807 shutdown_connections 2024-03-19T02:51:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.532+0000 7f44972de640 1 -- 172.21.15.116:0/3151492807 wait complete. 2024-03-19T02:51:13.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 Processor -- start 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 -- start start 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4490101350 0x7f449019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44901056a0 0x7f449019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4490113cb0 con 0x7f4490100990 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4490113b30 con 0x7f4490101350 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f44972de640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4490113e30 con 0x7f44901056a0 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f449507a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f449507a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45648/0 (socket says 172.21.15.116:45648) 2024-03-19T02:51:13.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f449507a640 1 -- 172.21.15.116:0/2828996222 learned_addr learned my addr 172.21.15.116:0/2828996222 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f4494879640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4490101350 0x7f449019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.533+0000 7f449587b640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44901056a0 0x7f449019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 -- 172.21.15.116:0/2828996222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44901056a0 msgr2=0x7f449019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44901056a0 0x7f449019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 -- 172.21.15.116:0/2828996222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4490101350 msgr2=0x7f449019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4490101350 0x7f449019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 -- 172.21.15.116:0/2828996222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449019f120 con 0x7f4490100990 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f449507a640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f448c00d950 tx=0x7f448c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f44867fc640 1 -- 172.21.15.116:0/2828996222 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c017070 con 0x7f4490100990 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f44867fc640 1 -- 172.21.15.116:0/2828996222 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f448c009c30 con 0x7f4490100990 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f44867fc640 1 -- 172.21.15.116:0/2828996222 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c00f620 con 0x7f4490100990 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449019f410 con 0x7f4490100990 2024-03-19T02:51:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.534+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44901a6ca0 con 0x7f4490100990 2024-03-19T02:51:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f4499f9b740 1 Processor -- start 2024-03-19T02:51:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f44867fc640 1 -- 172.21.15.116:0/2828996222 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f448c0109a0 con 0x7f4490100990 2024-03-19T02:51:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f4499f9b740 1 -- start start 2024-03-19T02:51:13.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfc5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.539+0000 7f449507a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45660/0 (socket says 172.21.15.116:45660) 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 -- 172.21.15.116:0/1287276335 learned_addr learned my addr 172.21.15.116:0/1287276335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558fadf29160 con 0x558fadfc5b40 2024-03-19T02:51:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558faded1e10 con 0x558fadfc5190 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558fadf25200 con 0x558fadfd7090 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f44867fc640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 0x7f447c0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449587b640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfc5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4494879640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 -- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 msgr2=0x558fadfd9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 -- 172.21.15.116:0/1287276335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 msgr2=0x558fadfc5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfc5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 -- 172.21.15.116:0/1287276335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fadd057e0 con 0x558fadfc5b40 2024-03-19T02:51:13.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449587b640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfc5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f4494879640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449587b640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 0x7f447c0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.540+0000 7f449507a640 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f448c09b0a0 tx=0x7f448c022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ab36f9b4a08eab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.541+0000 7f44867fc640 1 -- 172.21.15.116:0/2828996222 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f448c0cee40 con 0x7f4490100990 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.541+0000 7f447b7fe640 1 -- 172.21.15.116:0/1287276335 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c020070 con 0x558fadfc5b40 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.541+0000 7f447b7fe640 1 -- 172.21.15.116:0/1287276335 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f448c009de0 con 0x558fadfc5b40 2024-03-19T02:51:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.541+0000 7f447b7fe640 1 -- 172.21.15.116:0/1287276335 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c09b110 con 0x558fadfc5b40 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 msgr2=0x558fadfd6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f449587b640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 0x7f447c0a4ee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4488005fd0 tx=0x7f4488004a40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f448c09b0a0 tx=0x7f448c022000 comp rx=0 tx=0).stop 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 shutdown_connections 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd6b50 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1287276335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfc5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 >> 172.21.15.116:0/1287276335 conn(0x558fadfbfab0 msgr2=0x558fadfbfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 shutdown_connections 2024-03-19T02:51:13.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- 172.21.15.116:0/1287276335 wait complete. 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 -- start start 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.542+0000 7f4499f9b740 1 Processor -- start 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfd4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558fadfd9d90 con 0x558fadfc5b40 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558fadee51a0 con 0x558fadfc5190 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558fadfd9f10 con 0x558fadfd7090 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4499f9b740 10 client.0 _ll_register_callbacks cb 0x558fadc3cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4494879640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4494879640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45676/0 (socket says 172.21.15.116:45676) 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f4494879640 1 -- 172.21.15.116:0/83424035 learned_addr learned my addr 172.21.15.116:0/83424035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f449507a640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfd4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.543+0000 7f449587b640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 -- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 msgr2=0x558fadfd4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfd4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 -- 172.21.15.116:0/83424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 msgr2=0x558fadfd4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 -- 172.21.15.116:0/83424035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fadd65f70 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4494879640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f449587b640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd4a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4488009f10 tx=0x7f4488004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15f0abce7f7cad59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488010be0 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f448c009de0 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f4488010d80 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f448801c460 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.544+0000 7f4499f9b740 10 client.31879 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558fadbfd170 con 0x558fadfd7090 2024-03-19T02:51:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f448c009de0 con 0x558fadfd7090 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f448801e7d0 con 0x558fadfd7090 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 10 client.31879 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 10 client.31879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 20 client.31879 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fadfe71f0 0x558fadfe95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 4 client.31879 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fadf366a0 con 0x558fadfe71f0 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.545+0000 7f449507a640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fadfe71f0 0x558fadfe95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.546+0000 7f4478ff9640 10 client.31879 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.546+0000 7f449507a640 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fadfe71f0 0x558fadfe95e0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e721059cb56e73f4 server_cookie=70f2254d0f16eb5b in_seq=0 out_seq=0 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4478ff9640 1 -- 172.21.15.116:0/83424035 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558fadf366a0 con 0x558fadfe71f0 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4478ff9640 10 client.31879 handle_command_reply: tid=0 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 client.31879 shutdown 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 msgr2=0x558fadfd4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd4a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4488009f10 tx=0x7f4488004ab0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 shutdown_connections 2024-03-19T02:51:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fadfe71f0 0x558fadfe95e0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfd7090 0x558fadfd4a90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfc5b40 0x558fadfd4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 --2- 172.21.15.116:0/83424035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5190 0x558fadfd4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 >> 172.21.15.116:0/83424035 conn(0x558fadfbfab0 msgr2=0x558fadfc1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4478ff9640 0 client.31879 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 shutdown_connections 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 1 -- 172.21.15.116:0/83424035 wait complete. 2024-03-19T02:51:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.548+0000 7f4499f9b740 20 client.31879 trim_cache size 0 max 16384 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.575+0000 7f4499f9b740 1 Processor -- start 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.575+0000 7f4499f9b740 1 -- start start 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.575+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.575+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfe2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfeeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4499f9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44901aca90 con 0x558fadfd7090 2024-03-19T02:51:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4499f9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44901ae5d0 con 0x558fadfc5b40 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4499f9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f44901ad160 con 0x558fadfc5190 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449587b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfeeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46512/0 (socket says 172.21.15.116:46512) 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 -- 172.21.15.116:0/1859336396 learned_addr learned my addr 172.21.15.116:0/1859336396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4494879640 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfe2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 -- 172.21.15.116:0/1859336396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 msgr2=0x558fadfe2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfe2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 -- 172.21.15.116:0/1859336396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 msgr2=0x558fadfeeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfeeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 -- 172.21.15.116:0/1859336396 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fadfbb160 con 0x558fadfc5190 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f4494879640 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfe2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f449507a640 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f448c097570 tx=0x7f448c023b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cfff4619c0c92cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f447affd640 1 -- 172.21.15.116:0/1859336396 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c020040 con 0x558fadfc5190 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f447affd640 1 -- 172.21.15.116:0/1859336396 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f448c009de0 con 0x558fadfc5190 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.576+0000 7f447affd640 1 -- 172.21.15.116:0/1859336396 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c023e90 con 0x558fadfc5190 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 -- 172.21.15.116:0/1859336396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 msgr2=0x558fadfd6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f448c097570 tx=0x7f448c023b30 comp rx=0 tx=0).stop 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 -- 172.21.15.116:0/1859336396 shutdown_connections 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfeeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfe2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 --2- 172.21.15.116:0/1859336396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd6740 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 -- 172.21.15.116:0/1859336396 >> 172.21.15.116:0/1859336396 conn(0x558fadfbfab0 msgr2=0x558fadfbe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 -- 172.21.15.116:0/1859336396 shutdown_connections 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.577+0000 7f4499f9b740 1 -- 172.21.15.116:0/1859336396 wait complete. 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 -- start start 2024-03-19T02:51:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 Processor -- start 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfd0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfd1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44901adf00 con 0x558fadfd7090 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44901ad830 con 0x558fadfc5b40 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f44901aaa90 con 0x558fadfc5190 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46518/0 (socket says 172.21.15.116:46518) 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 -- 172.21.15.116:0/3845080822 learned_addr learned my addr 172.21.15.116:0/3845080822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449587b640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfd1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4494879640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfd0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f4499f9b740 10 client.0 _ll_register_callbacks cb 0x7f4488009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 -- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 msgr2=0x558fadfd0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfd0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.578+0000 7f449507a640 1 -- 172.21.15.116:0/3845080822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 msgr2=0x558fadfd1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f449507a640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfd1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f449507a640 1 -- 172.21.15.116:0/3845080822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fae04c4e0 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f449587b640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfd1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f449507a640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f448c0a32d0 tx=0x7f448c00bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ce32996982f70a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c0b3060 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f448c009de0 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f448c097780 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f448c014070 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 10 client.31885 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558fae04d6e0 con 0x558fadfc5190 2024-03-19T02:51:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f448c009de0 con 0x558fadfc5190 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f448c0d2020 con 0x558fadfc5190 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 10 client.31885 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 10 client.31885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.579+0000 7f4499f9b740 20 client.31885 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fae04d910 0x558fae04fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4499f9b740 4 client.31885 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558fadf309f0 con 0x558fae04d910 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4494879640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fae04d910 0x558fae04fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4478ff9640 10 client.31885 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.580+0000 7f4494879640 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fae04d910 0x558fae04fd00 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a7b2c27af705f1 server_cookie=ff7f390e74fcc1e7 in_seq=0 out_seq=0 2024-03-19T02:51:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4478ff9640 1 -- 172.21.15.116:0/3845080822 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558fadf309f0 con 0x558fae04d910 2024-03-19T02:51:13.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4478ff9640 10 client.31885 handle_command_reply: tid=0 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 client.31885 shutdown 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 msgr2=0x558fadfd0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f448c0a32d0 tx=0x7f448c00bf60 comp rx=0 tx=0).stop 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 shutdown_connections 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558fae04d910 0x558fae04fd00 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fadfd7090 0x558fadfd1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558fadfc5b40 0x558fadfd0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 --2- 172.21.15.116:0/3845080822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fadfc5190 0x558fadfd0a90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4478ff9640 0 client.31885 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 >> 172.21.15.116:0/3845080822 conn(0x558fadfbfab0 msgr2=0x558fadfd3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 shutdown_connections 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 1 -- 172.21.15.116:0/3845080822 wait complete. 2024-03-19T02:51:13.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f4499f9b740 20 client.31885 trim_cache size 0 max 16384 2024-03-19T02:51:13.590 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:13.590 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:51:13.590 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "98dac90d-62da-4277-8f3a-4794d0f6cdee", 2024-03-19T02:51:13.590 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:51:13.590 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 msgr2=0x7f447c0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 0x7f447c0a4ee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4488005fd0 tx=0x7f4488004a40 comp rx=0 tx=0).stop 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 msgr2=0x7f449019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.583+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f448c00d950 tx=0x7f448c00de10 comp rx=0 tx=0).stop 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 shutdown_connections 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f447c0a2a20 0x7f447c0a4ee0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44901056a0 0x7f449019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4490101350 0x7f449019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 --2- 172.21.15.116:0/2828996222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490100990 0x7f449019a130 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 >> 172.21.15.116:0/2828996222 conn(0x7f4490077610 msgr2=0x7f44900fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 shutdown_connections 2024-03-19T02:51:13.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.584+0000 7f44972de640 1 -- 172.21.15.116:0/2828996222 wait complete. 2024-03-19T02:51:13.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:51:13.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:51:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- 172.21.15.116:0/4232254031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68481066a0 msgr2=0x7f6848106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 --2- 172.21.15.116:0/4232254031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68481066a0 0x7f6848106a80 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f6844009930 tx=0x7f684402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- 172.21.15.116:0/4232254031 shutdown_connections 2024-03-19T02:51:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 --2- 172.21.15.116:0/4232254031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6848101050 0x7f684810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 --2- 172.21.15.116:0/4232254031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 --2- 172.21.15.116:0/4232254031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68481066a0 0x7f6848106a80 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- 172.21.15.116:0/4232254031 >> 172.21.15.116:0/4232254031 conn(0x7f6848077820 msgr2=0x7f6848077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- 172.21.15.116:0/4232254031 shutdown_connections 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- 172.21.15.116:0/4232254031 wait complete. 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 Processor -- start 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.714+0000 7f6850377640 1 -- start start 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 0x7f6848198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68481066a0 0x7f684819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6848111a60 con 0x7f6848101050 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68481118e0 con 0x7f68481066a0 2024-03-19T02:51:13.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6848111be0 con 0x7f6848100690 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684e113640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 0x7f6848198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684e113640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46530/0 (socket says 172.21.15.116:46530) 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684e113640 1 -- 172.21.15.116:0/3040291151 learned_addr learned my addr 172.21.15.116:0/3040291151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684e914640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68481066a0 0x7f684819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 -- 172.21.15.116:0/3040291151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 msgr2=0x7f6848198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 -- 172.21.15.116:0/3040291151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68481066a0 msgr2=0x7f684819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68481066a0 0x7f684819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 -- 172.21.15.116:0/3040291151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684819d010 con 0x7f6848101050 2024-03-19T02:51:13.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684e113640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f684d912640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 0x7f6848198560 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f683800e9f0 tx=0x7f683800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f683f7fe640 1 -- 172.21.15.116:0/3040291151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683800ccc0 con 0x7f6848101050 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684819d300 con 0x7f6848101050 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f683f7fe640 1 -- 172.21.15.116:0/3040291151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6838005020 con 0x7f6848101050 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.715+0000 7f683f7fe640 1 -- 172.21.15.116:0/3040291151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838004a00 con 0x7f6848101050 2024-03-19T02:51:13.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.716+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68481a4b90 con 0x7f6848101050 2024-03-19T02:51:13.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.718+0000 7f683f7fe640 1 -- 172.21.15.116:0/3040291151 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6838004ba0 con 0x7f6848101050 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.718+0000 7f6852bfd740 1 Processor -- start 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 -- start start 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca618c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560ca60e4500 con 0x560ca6189f30 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560ca608cbe0 con 0x560ca6180a30 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f6852bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560ca60e05a0 con 0x560ca6181350 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f683f7fe640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 0x7f68340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e113640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e914640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e113640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46544/0 (socket says 172.21.15.116:46544) 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e914640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34286/0 (socket says 172.21.15.116:34286) 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e113640 1 -- 172.21.15.116:0/1657399319 learned_addr learned my addr 172.21.15.116:0/1657399319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684d912640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca618c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e914640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 0x7f68340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f683f7fe640 1 -- 172.21.15.116:0/3040291151 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f68380cf1d0 con 0x7f6848101050 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e914640 1 -- 172.21.15.116:0/1657399319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 msgr2=0x560ca61899f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.719+0000 7f684e914640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e914640 1 -- 172.21.15.116:0/1657399319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 msgr2=0x560ca618c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e914640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca618c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e914640 1 -- 172.21.15.116:0/1657399319 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca60ae000 con 0x560ca6180a30 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e113640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684d912640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca618c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:13.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e914640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 0x7f68340a4e30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f684000fc90 tx=0x7f6840018000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f684e914640 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6840011cb0 tx=0x7f684000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b74fde48dc5c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f683cbfe640 1 -- 172.21.15.116:0/1657399319 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684000ec50 con 0x560ca6180a30 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f683cbfe640 1 -- 172.21.15.116:0/1657399319 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6840009cc0 con 0x560ca6180a30 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.720+0000 7f683cbfe640 1 -- 172.21.15.116:0/1657399319 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68400057b0 con 0x560ca6180a30 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 -- 172.21.15.116:0/1657399319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 msgr2=0x560ca6180e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6840011cb0 tx=0x7f684000f790 comp rx=0 tx=0).stop 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 -- 172.21.15.116:0/1657399319 shutdown_connections 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca618c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6181350 0x560ca61899f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 --2- 172.21.15.116:0/1657399319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca6180e10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 -- 172.21.15.116:0/1657399319 >> 172.21.15.116:0/1657399319 conn(0x560ca617b160 msgr2=0x560ca617b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.721+0000 7f6852bfd740 1 -- 172.21.15.116:0/1657399319 shutdown_connections 2024-03-19T02:51:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 -- 172.21.15.116:0/1657399319 wait complete. 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 -- start start 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 Processor -- start 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca6186ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6187410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560ca618cc60 con 0x560ca6189f30 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560ca60a0160 con 0x560ca6181350 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560ca618cde0 con 0x560ca6180a30 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f6852bfd740 10 client.0 _ll_register_callbacks cb 0x560ca60dca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f684e914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f684e914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45742/0 (socket says 172.21.15.116:45742) 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f684e113640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca6186ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f684e914640 1 -- 172.21.15.116:0/2739839449 learned_addr learned my addr 172.21.15.116:0/2739839449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.722+0000 7f684d912640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6187410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 -- 172.21.15.116:0/2739839449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 msgr2=0x560ca6186ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca6186ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 -- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 msgr2=0x560ca6187410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6187410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 -- 172.21.15.116:0/2739839449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca5ec27d0 con 0x560ca6189f30 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f684e914640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f6840006010 tx=0x7f6840011840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d268627c3cab61f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840033020 con 0x560ca6189f30 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f6840011250 con 0x560ca6189f30 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6840009cc0 con 0x560ca6189f30 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f684003e050 con 0x560ca6189f30 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6852bfd740 10 client.22203 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:13.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.723+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560ca5df83f0 con 0x560ca6189f30 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6840009cc0 con 0x560ca6189f30 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f6840024030 con 0x560ca6189f30 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 10 client.22203 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 10 client.22203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 20 client.22203 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca619a0f0 0x560ca619c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 4 client.22203 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ca60f1a40 con 0x560ca619a0f0 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.724+0000 7f684e113640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca619a0f0 0x560ca619c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.725+0000 7f6831ffb640 10 client.22203 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.725+0000 7f684e113640 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca619a0f0 0x560ca619c4e0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e645b097b98650a1 server_cookie=db8adda585933406 in_seq=0 out_seq=0 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6831ffb640 1 -- 172.21.15.116:0/2739839449 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ca60f1a40 con 0x560ca619a0f0 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6831ffb640 10 client.22203 handle_command_reply: tid=0 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 client.22203 shutdown 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 msgr2=0x560ca6187950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f6840006010 tx=0x7f6840011840 comp rx=0 tx=0).stop 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 shutdown_connections 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca619a0f0 0x560ca619c4e0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6187950 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6187410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2739839449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca6186ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 >> 172.21.15.116:0/2739839449 conn(0x560ca617b160 msgr2=0x560ca617ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6831ffb640 0 client.22203 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 shutdown_connections 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 1 -- 172.21.15.116:0/2739839449 wait complete. 2024-03-19T02:51:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.733+0000 7f6852bfd740 20 client.22203 trim_cache size 0 max 16384 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 Processor -- start 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 -- start start 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca617a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6195a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68481aca50 con 0x560ca6189f30 2024-03-19T02:51:13.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68481ae590 con 0x560ca6181350 2024-03-19T02:51:13.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f6852bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68481ad120 con 0x560ca6180a30 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f684e914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f684e914640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45748/0 (socket says 172.21.15.116:45748) 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f684e914640 1 -- 172.21.15.116:0/2915190467 learned_addr learned my addr 172.21.15.116:0/2915190467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f684e113640 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca617a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.752+0000 7f684d912640 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 -- 172.21.15.116:0/2915190467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 msgr2=0x560ca617a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca617a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 -- 172.21.15.116:0/2915190467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 msgr2=0x560ca6195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 -- 172.21.15.116:0/2915190467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca5db80f0 con 0x560ca6189f30 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f684e914640 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f6840011cb0 tx=0x7f6840009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293ada8688d6abf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f6833fff640 1 -- 172.21.15.116:0/2915190467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840033070 con 0x560ca6189f30 2024-03-19T02:51:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f6833fff640 1 -- 172.21.15.116:0/2915190467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f684002ed00 con 0x560ca6189f30 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.753+0000 7f6833fff640 1 -- 172.21.15.116:0/2915190467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840004e60 con 0x560ca6189f30 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- 172.21.15.116:0/2915190467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 msgr2=0x560ca6186680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f6840011cb0 tx=0x7f6840009e50 comp rx=0 tx=0).stop 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- 172.21.15.116:0/2915190467 shutdown_connections 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6189f30 0x560ca6186680 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6181350 0x560ca6195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- 172.21.15.116:0/2915190467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6180a30 0x560ca617a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- 172.21.15.116:0/2915190467 >> 172.21.15.116:0/2915190467 conn(0x560ca617b160 msgr2=0x560ca617c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- 172.21.15.116:0/2915190467 shutdown_connections 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- 172.21.15.116:0/2915190467 wait complete. 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- start start 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 Processor -- start 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca618cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 0x560ca618d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6189f30 0x560ca618da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68481adec0 con 0x560ca6181350 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68481ad7f0 con 0x560ca6180a30 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f68481aaa50 con 0x560ca6189f30 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.754+0000 7f6852bfd740 10 client.0 _ll_register_callbacks cb 0x7f68380970b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684e113640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca618cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 0x560ca618d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684e113640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca618cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:34316/0 (socket says 172.21.15.116:34316) 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684e113640 1 -- 172.21.15.116:0/3791953258 learned_addr learned my addr 172.21.15.116:0/3791953258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684e914640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6189f30 0x560ca618da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 -- 172.21.15.116:0/3791953258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6189f30 msgr2=0x560ca618da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6189f30 0x560ca618da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 -- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 msgr2=0x560ca618cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca618cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 -- 172.21.15.116:0/3791953258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca61fe560 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f684d912640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 0x560ca618d530 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f68380cc540 tx=0x7f68380096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07618c6f0d3c0b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6831ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683801c040 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f68380051d0 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f684002ed00 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f683809b450 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6852bfd740 10 client.22209 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.755+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560ca61ff850 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f684002ed00 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f683809a520 con 0x560ca6181350 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 10 client.22209 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 10 client.22209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 20 client.22209 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca61fffa0 0x560ca62023c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 4 client.22209 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ca60ebd90 con 0x560ca61fffa0 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f684e113640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca61fffa0 0x560ca62023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f6831ffb640 10 client.22209 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.756+0000 7f684e113640 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca61fffa0 0x560ca62023c0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0b694fdc321939 server_cookie=2d92fdbf645709e7 in_seq=0 out_seq=0 2024-03-19T02:51:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6831ffb640 1 -- 172.21.15.116:0/3791953258 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560ca60ebd90 con 0x560ca61fffa0 2024-03-19T02:51:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6831ffb640 10 client.22209 handle_command_reply: tid=0 2024-03-19T02:51:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 client.22209 shutdown 2024-03-19T02:51:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 msgr2=0x560ca618d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 0x560ca618d530 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f68380cc540 tx=0x7f68380096e0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 shutdown_connections 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x560ca61fffa0 0x560ca62023c0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560ca6189f30 0x560ca618da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6831ffb640 0 client.22209 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560ca6181350 0x560ca618d530 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 --2- 172.21.15.116:0/3791953258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560ca6180a30 0x560ca618cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 >> 172.21.15.116:0/3791953258 conn(0x560ca617b160 msgr2=0x560ca617a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 shutdown_connections 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 1 -- 172.21.15.116:0/3791953258 wait complete. 2024-03-19T02:51:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.761+0000 7f6852bfd740 20 client.22209 trim_cache size 0 max 16384 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (588 inodes in the stack)", 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "98dac90d-62da-4277-8f3a-4794d0f6cdee": { 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "98dac90d-62da-4277-8f3a-4794d0f6cdee", 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:51:13.770 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:13.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 msgr2=0x7f68340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 0x7f68340a4e30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f684000fc90 tx=0x7f6840018000 comp rx=0 tx=0).stop 2024-03-19T02:51:13.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 msgr2=0x7f6848198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:13.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 0x7f6848198560 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f683800e9f0 tx=0x7f683800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 shutdown_connections 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f68340a2970 0x7f68340a4e30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68481066a0 0x7f684819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6848101050 0x7f6848198560 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 --2- 172.21.15.116:0/3040291151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6848100690 0x7f6848198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 >> 172.21.15.116:0/3040291151 conn(0x7f6848077820 msgr2=0x7f6848103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 shutdown_connections 2024-03-19T02:51:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:13.762+0000 7f6850377640 1 -- 172.21.15.116:0/3040291151 wait complete. 2024-03-19T02:51:13.776 INFO:tasks.cephfs.filesystem:scrub status for tag:98dac90d-62da-4277-8f3a-4794d0f6cdee - {'path': '/', 'tag': '98dac90d-62da-4277-8f3a-4794d0f6cdee', 'options': 'recursive,force'} 2024-03-19T02:51:14.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68730 5.75 MB/sec execute 516 sec latency 222.056 ms 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:14.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:14 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-03-19T02:51:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:51:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:14.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:14.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:14.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:14 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: pgmap v741: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.7 MiB/s wr, 60 op/s 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:14 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68824 5.75 MB/sec execute 517 sec latency 225.963 ms 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:15.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:15.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:15 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:15 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:15.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:15 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:16.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 68916 5.75 MB/sec execute 518 sec latency 173.783 ms 2024-03-19T02:51:16.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:16 smithi162 ceph-mon[35761]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:51:16.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:16 smithi154 ceph-mon[35693]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:51:16.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:16 smithi116 ceph-mon[32263]: pgmap v742: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-03-19T02:51:17.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69010 5.74 MB/sec execute 519 sec latency 183.316 ms 2024-03-19T02:51:18.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69085 5.73 MB/sec execute 520 sec latency 262.717 ms 2024-03-19T02:51:18.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:18 smithi162 ceph-mon[35761]: pgmap v743: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:18.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:18 smithi154 ceph-mon[35693]: pgmap v743: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:18.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:18 smithi116 ceph-mon[32263]: pgmap v743: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:19.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69169 5.73 MB/sec execute 521 sec latency 192.209 ms 2024-03-19T02:51:20.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69290 5.73 MB/sec execute 522 sec latency 178.877 ms 2024-03-19T02:51:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:20 smithi162 ceph-mon[35761]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:20 smithi154 ceph-mon[35693]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:20.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:20 smithi116 ceph-mon[32263]: pgmap v744: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 181 op/s 2024-03-19T02:51:21.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69364 5.72 MB/sec execute 523 sec latency 155.590 ms 2024-03-19T02:51:22.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69454 5.72 MB/sec execute 524 sec latency 191.918 ms 2024-03-19T02:51:22.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:22 smithi162 ceph-mon[35761]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 214 op/s 2024-03-19T02:51:22.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:22 smithi154 ceph-mon[35693]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 214 op/s 2024-03-19T02:51:22.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:22 smithi116 ceph-mon[32263]: pgmap v745: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 214 op/s 2024-03-19T02:51:23.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69542 5.71 MB/sec execute 525 sec latency 900.141 ms 2024-03-19T02:51:24.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69628 5.70 MB/sec execute 526 sec latency 1900.174 ms 2024-03-19T02:51:24.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:24 smithi162 ceph-mon[35761]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 193 op/s 2024-03-19T02:51:24.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:24 smithi154 ceph-mon[35693]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 193 op/s 2024-03-19T02:51:24.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:24 smithi116 ceph-mon[32263]: pgmap v746: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 193 op/s 2024-03-19T02:51:25.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69703 5.70 MB/sec execute 527 sec latency 2900.206 ms 2024-03-19T02:51:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:51:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:51:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:25 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:51:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:51:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:51:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:25 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:51:25.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:51:25.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:51:25.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:25 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:51:26.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69807 5.70 MB/sec execute 528 sec latency 3413.284 ms 2024-03-19T02:51:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:26 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:51:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:26 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:51:26.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:26 smithi162 ceph-mon[35761]: pgmap v747: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 259 op/s 2024-03-19T02:51:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:26 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:51:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:26 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:51:26.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:26 smithi154 ceph-mon[35693]: pgmap v747: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 259 op/s 2024-03-19T02:51:26.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:26 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:26 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:51:26.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:26 smithi116 ceph-mon[32263]: pgmap v747: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.9 MiB/s wr, 259 op/s 2024-03-19T02:51:27.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69891 5.70 MB/sec execute 529 sec latency 1092.281 ms 2024-03-19T02:51:28.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 69980 5.69 MB/sec execute 530 sec latency 2092.307 ms 2024-03-19T02:51:28.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:28 smithi162 ceph-mon[35761]: pgmap v748: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:28.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:28 smithi154 ceph-mon[35693]: pgmap v748: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:28 smithi116 ceph-mon[32263]: pgmap v748: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:29.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70072 5.70 MB/sec execute 531 sec latency 3092.361 ms 2024-03-19T02:51:30.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70163 5.71 MB/sec execute 532 sec latency 4092.410 ms 2024-03-19T02:51:30.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:30 smithi154 ceph-mon[35693]: pgmap v749: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:30.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:30 smithi116 ceph-mon[32263]: pgmap v749: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:30.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:30 smithi162 ceph-mon[35761]: pgmap v749: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 148 op/s 2024-03-19T02:51:31.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70252 5.70 MB/sec execute 533 sec latency 4605.952 ms 2024-03-19T02:51:32.339 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70346 5.69 MB/sec execute 534 sec latency 132.025 ms 2024-03-19T02:51:32.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:32 smithi154 ceph-mon[35693]: pgmap v750: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 211 op/s 2024-03-19T02:51:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:32 smithi116 ceph-mon[32263]: pgmap v750: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 211 op/s 2024-03-19T02:51:32.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:32 smithi162 ceph-mon[35761]: pgmap v750: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.2 MiB/s wr, 211 op/s 2024-03-19T02:51:33.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70447 5.68 MB/sec execute 535 sec latency 137.680 ms 2024-03-19T02:51:34.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70581 5.68 MB/sec execute 536 sec latency 142.736 ms 2024-03-19T02:51:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:34 smithi154 ceph-mon[35693]: pgmap v751: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-03-19T02:51:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:34 smithi116 ceph-mon[32263]: pgmap v751: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-03-19T02:51:34.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:34 smithi162 ceph-mon[35761]: pgmap v751: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 143 op/s 2024-03-19T02:51:35.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70689 5.68 MB/sec execute 537 sec latency 145.071 ms 2024-03-19T02:51:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:35 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:35 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:35.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:35 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70802 5.67 MB/sec execute 538 sec latency 173.061 ms 2024-03-19T02:51:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:36 smithi154 ceph-mon[35693]: pgmap v752: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 222 op/s 2024-03-19T02:51:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:36 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:51:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:36 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:51:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:36 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:51:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:36 smithi116 ceph-mon[32263]: pgmap v752: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 222 op/s 2024-03-19T02:51:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:36 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:51:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:36 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:51:36.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:36 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:51:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:36 smithi162 ceph-mon[35761]: pgmap v752: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 222 op/s 2024-03-19T02:51:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:36 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:51:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:36 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:51:36.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:36 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:51:37.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 70913 5.67 MB/sec execute 539 sec latency 153.887 ms 2024-03-19T02:51:38.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71010 5.66 MB/sec execute 540 sec latency 177.794 ms 2024-03-19T02:51:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:38 smithi154 ceph-mon[35693]: pgmap v753: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:38.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:38 smithi116 ceph-mon[32263]: pgmap v753: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:38.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:38 smithi162 ceph-mon[35761]: pgmap v753: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:39.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71157 5.67 MB/sec execute 541 sec latency 147.206 ms 2024-03-19T02:51:40.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71277 5.67 MB/sec execute 542 sec latency 144.850 ms 2024-03-19T02:51:40.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:40 smithi154 ceph-mon[35693]: pgmap v754: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:40.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:40 smithi116 ceph-mon[32263]: pgmap v754: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:40.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:40 smithi162 ceph-mon[35761]: pgmap v754: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 141 op/s 2024-03-19T02:51:41.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71379 5.68 MB/sec execute 543 sec latency 162.711 ms 2024-03-19T02:51:42.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71488 5.68 MB/sec execute 544 sec latency 563.815 ms 2024-03-19T02:51:42.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:42 smithi154 ceph-mon[35693]: pgmap v755: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 216 op/s 2024-03-19T02:51:42.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:42 smithi116 ceph-mon[32263]: pgmap v755: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 216 op/s 2024-03-19T02:51:42.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:42 smithi162 ceph-mon[35761]: pgmap v755: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 216 op/s 2024-03-19T02:51:43.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71604 5.68 MB/sec execute 545 sec latency 1563.841 ms 2024-03-19T02:51:43.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:51:43.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 -- 172.21.15.116:0/859265093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 msgr2=0x7ff8b4106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/859265093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b4106a80 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009930 tx=0x7ff8a402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 -- 172.21.15.116:0/859265093 shutdown_connections 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/859265093 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4101050 0x7ff8b410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/859265093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4100690 0x7ff8b4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/859265093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b4106a80 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 -- 172.21.15.116:0/859265093 >> 172.21.15.116:0/859265093 conn(0x7ff8b4077820 msgr2=0x7ff8b4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 -- 172.21.15.116:0/859265093 shutdown_connections 2024-03-19T02:51:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 -- 172.21.15.116:0/859265093 wait complete. 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.900+0000 7ff8b9c61640 1 Processor -- start 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 -- start start 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 0x7ff8b4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b4110960 con 0x7ff8b41066a0 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41107e0 con 0x7ff8b4101050 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b9c61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b4110ae0 con 0x7ff8b4100690 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 0x7ff8b4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57324/0 (socket says 172.21.15.116:57324) 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 0x7ff8b4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58866/0 (socket says 172.21.15.116:58866) 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 -- 172.21.15.116:0/1865595679 learned_addr learned my addr 172.21.15.116:0/1865595679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 -- 172.21.15.116:0/1865595679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 msgr2=0x7ff8b4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 0x7ff8b4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 -- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 msgr2=0x7ff8b419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 -- 172.21.15.116:0/1865595679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b419cfe0 con 0x7ff8b4101050 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.901+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800e9f0 tx=0x7ff8a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.902+0000 7ff8b0ff9640 1 -- 172.21.15.116:0/1865595679 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800cd80 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.902+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b419d2d0 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.902+0000 7ff8b0ff9640 1 -- 172.21.15.116:0/1865595679 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8a80108e0 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.902+0000 7ff8b0ff9640 1 -- 172.21.15.116:0/1865595679 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8018920 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.902+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b4102c70 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.904+0000 7ff8b0ff9640 1 -- 172.21.15.116:0/1865595679 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff8a8010a90 con 0x7ff8b4101050 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.904+0000 7ff8bc851740 1 Processor -- start 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.904+0000 7ff8bc851740 1 -- start start 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70dc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e2b7034500 con 0x55e2b70d0a30 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6fdcbe0 con 0x55e2b70d9f30 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8bc851740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2b70305a0 con 0x55e2b70d1350 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b0ff9640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 0x7ff89c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56420/0 (socket says 172.21.15.116:56420) 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 -- 172.21.15.116:0/761028439 learned_addr learned my addr 172.21.15.116:0/761028439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70dc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 0x7ff89c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 -- 172.21.15.116:0/761028439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 msgr2=0x55e2b70d99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 -- 172.21.15.116:0/761028439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 msgr2=0x55e2b70dc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70dc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 -- 172.21.15.116:0/761028439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b6ffe000 con 0x55e2b70d0a30 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70dc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b0ff9640 1 -- 172.21.15.116:0/1865595679 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7ff8a80cf4e0 con 0x7ff8b4101050 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.905+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac005c40 tx=0x7ff8ac013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cb5d599d8f8971 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff899ffb640 1 -- 172.21.15.116:0/761028439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac022020 con 0x55e2b70d0a30 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 0x7ff89c0a4eb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac023000 tx=0x7ff8ac021000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff899ffb640 1 -- 172.21.15.116:0/761028439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8ac0134e0 con 0x55e2b70d0a30 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff899ffb640 1 -- 172.21.15.116:0/761028439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac010ad0 con 0x55e2b70d0a30 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 -- 172.21.15.116:0/761028439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 msgr2=0x55e2b70d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac005c40 tx=0x7ff8ac013f50 comp rx=0 tx=0).stop 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 -- 172.21.15.116:0/761028439 shutdown_connections 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70dc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 --2- 172.21.15.116:0/761028439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d0a30 0x55e2b70d0e10 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 -- 172.21.15.116:0/761028439 >> 172.21.15.116:0/761028439 conn(0x55e2b70cb160 msgr2=0x55e2b70cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 -- 172.21.15.116:0/761028439 shutdown_connections 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.906+0000 7ff8bc851740 1 -- 172.21.15.116:0/761028439 wait complete. 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 -- start start 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 Processor -- start 2024-03-19T02:51:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 0x55e2b70d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e2b70dcd20 con 0x55e2b70d9f30 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2b6ff0160 con 0x55e2b70d1350 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e2b70dcea0 con 0x55e2b70d0a30 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8bc851740 10 client.0 _ll_register_callbacks cb 0x55e2b702ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 0x55e2b70d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 0x55e2b70d7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57342/0 (socket says 172.21.15.116:57342) 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58876/0 (socket says 172.21.15.116:58876) 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b2ffd640 1 -- 172.21.15.116:0/1108312793 learned_addr learned my addr 172.21.15.116:0/1108312793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.907+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 msgr2=0x55e2b70d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 0x55e2b70d7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1108312793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 msgr2=0x55e2b70d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1108312793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b6e127d0 con 0x55e2b70d0a30 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009230 tx=0x7ff8a4004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed259273c6d13546 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4005230 con 0x55e2b70d0a30 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2b7041a40 con 0x55e2b70d0a30 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7ff8a40053d0 con 0x55e2b70d0a30 2024-03-19T02:51:43.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff8a4054050 con 0x55e2b70d0a30 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8bc851740 10 client.31915 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.908+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e2b6d483f0 con 0x55e2b70d0a30 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e2b7041a40 con 0x55e2b70d0a30 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7ff8a404f030 con 0x55e2b70d0a30 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 10 client.31915 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 10 client.31915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 20 client.31915 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b70ea2c0 0x55e2b70ec710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 4 client.31915 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2b703bd90 con 0x55e2b70ea2c0 2024-03-19T02:51:43.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b70ea2c0 0x55e2b70ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b70ea2c0 0x55e2b70ec710 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be4e16e8784e691a server_cookie=9b21922a058228cf in_seq=0 out_seq=0 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.909+0000 7ff892ffd640 10 client.31915 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff892ffd640 1 -- 172.21.15.116:0/1108312793 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2b703bd90 con 0x55e2b70ea2c0 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff892ffd640 10 client.31915 handle_command_reply: tid=0 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 client.31915 shutdown 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 msgr2=0x55e2b70d6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009230 tx=0x7ff8a4004610 comp rx=0 tx=0).stop 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 shutdown_connections 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b70ea2c0 0x55e2b70ec710 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d1350 0x55e2b70d7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1108312793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70d6f30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff892ffd640 0 client.31915 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 >> 172.21.15.116:0/1108312793 conn(0x55e2b70cb160 msgr2=0x55e2b70cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 shutdown_connections 2024-03-19T02:51:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 1 -- 172.21.15.116:0/1108312793 wait complete. 2024-03-19T02:51:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.911+0000 7ff8bc851740 20 client.31915 trim_cache size 0 max 16384 2024-03-19T02:51:43.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 Processor -- start 2024-03-19T02:51:43.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 -- start start 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41acab0 con 0x55e2b70d9f30 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41ae5f0 con 0x55e2b70d0a30 2024-03-19T02:51:43.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8bc851740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41ad180 con 0x55e2b70d1350 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57354/0 (socket says 172.21.15.116:57354) 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56438/0 (socket says 172.21.15.116:56438) 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.932+0000 7ff8b37fe640 1 -- 172.21.15.116:0/3268135206 learned_addr learned my addr 172.21.15.116:0/3268135206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3268135206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 msgr2=0x55e2b70d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3268135206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 msgr2=0x55e2b70d9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3268135206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2b6d080f0 con 0x55e2b70d9f30 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac010470 tx=0x7ff8ac012350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf42b3b96b42b17f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8997fa640 1 -- 172.21.15.116:0/3268135206 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac022070 con 0x55e2b70d9f30 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8997fa640 1 -- 172.21.15.116:0/3268135206 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8ac00fde0 con 0x55e2b70d9f30 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.933+0000 7ff8997fa640 1 -- 172.21.15.116:0/3268135206 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac012d10 con 0x55e2b70d9f30 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- 172.21.15.116:0/3268135206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 msgr2=0x55e2b70e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac010470 tx=0x7ff8ac012350 comp rx=0 tx=0).stop 2024-03-19T02:51:43.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- 172.21.15.116:0/3268135206 shutdown_connections 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d9f30 0x55e2b70e5ac0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d1350 0x55e2b70d5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 --2- 172.21.15.116:0/3268135206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d0a30 0x55e2b70d9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- 172.21.15.116:0/3268135206 >> 172.21.15.116:0/3268135206 conn(0x55e2b70cb160 msgr2=0x55e2b70d7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- 172.21.15.116:0/3268135206 shutdown_connections 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- 172.21.15.116:0/3268135206 wait complete. 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.934+0000 7ff8bc851740 1 -- start start 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 Processor -- start 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 0x55e2b70de040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70de580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41adf20 con 0x55e2b70d1350 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41ad850 con 0x55e2b70d9f30 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41aaab0 con 0x55e2b70d0a30 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8bc851740 10 client.0 _ll_register_callbacks cb 0x7ff8a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 0x55e2b70de040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70de580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8b37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58896/0 (socket says 172.21.15.116:58896) 2024-03-19T02:51:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.935+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1611703810 learned_addr learned my addr 172.21.15.116:0/1611703810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 msgr2=0x55e2b70de580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70de580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1611703810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 msgr2=0x55e2b70de040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 0x55e2b70de040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 -- 172.21.15.116:0/1611703810 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2b714f760 con 0x55e2b70d0a30 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70de580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 0x55e2b70de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009230 tx=0x7ff8a402cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b79f0f10f8b2d774 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4003db0 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7ff8a4004fc0 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ac00fde0 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ff8a4066050 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8bc851740 10 client.31921 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e2b7150a70 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8ac00fde0 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.936+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7ff8a404f040 con 0x55e2b70d0a30 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 10 client.31921 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 10 client.31921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 20 client.31921 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b7151270 0x55e2b7153690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 4 client.31921 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2b6fb9540 con 0x55e2b7151270 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b7151270 0x55e2b7153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff892ffd640 10 client.31921 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.937+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b7151270 0x55e2b7153690 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b68f1501fd2508a server_cookie=1169c3af27b57119 in_seq=0 out_seq=0 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff892ffd640 1 -- 172.21.15.116:0/1611703810 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e2b6fb9540 con 0x55e2b7151270 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff892ffd640 10 client.31921 handle_command_reply: tid=0 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 client.31921 shutdown 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 msgr2=0x55e2b70ddb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009230 tx=0x7ff8a402cd20 comp rx=0 tx=0).stop 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 shutdown_connections 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x55e2b7151270 0x55e2b7153690 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2b70d9f30 0x55e2b70de580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2b70d1350 0x55e2b70de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 --2- 172.21.15.116:0/1611703810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e2b70d0a30 0x55e2b70ddb00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 >> 172.21.15.116:0/1611703810 conn(0x55e2b70cb160 msgr2=0x55e2b70c9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff892ffd640 0 client.31921 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 shutdown_connections 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 1 -- 172.21.15.116:0/1611703810 wait complete. 2024-03-19T02:51:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.939+0000 7ff8bc851740 20 client.31921 trim_cache size 0 max 16384 2024-03-19T02:51:43.958 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:43.958 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:51:43.958 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:51:43.958 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 msgr2=0x7ff89c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 0x7ff89c0a4eb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac023000 tx=0x7ff8ac021000 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 msgr2=0x7ff8b4198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800e9f0 tx=0x7ff8a800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 shutdown_connections 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff89c0a29f0 0x7ff89c0a4eb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b41066a0 0x7ff8b419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8b4101050 0x7ff8b4198530 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 --2- 172.21.15.116:0/1865595679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4100690 0x7ff8b4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 >> 172.21.15.116:0/1865595679 conn(0x7ff8b4077820 msgr2=0x7ff8b4068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 shutdown_connections 2024-03-19T02:51:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:43.940+0000 7ff8b9c61640 1 -- 172.21.15.116:0/1865595679 wait complete. 2024-03-19T02:51:43.961 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:51:43.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:51:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/4281742269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4100720 msgr2=0x7fe0f4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/4281742269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4100720 0x7fe0f4100ba0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009930 tx=0x7fe0e802bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/4281742269 shutdown_connections 2024-03-19T02:51:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/4281742269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f41010e0 0x7fe0f410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/4281742269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4100720 0x7fe0f4100ba0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/4281742269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f4106730 0x7fe0f4106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/4281742269 >> 172.21.15.116:0/4281742269 conn(0x7fe0f4077820 msgr2=0x7fe0f4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/4281742269 shutdown_connections 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/4281742269 wait complete. 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.077+0000 7fe0f9ae8640 1 Processor -- start 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 -- start start 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f4100720 0x7fe0f4196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 0x7fe0f41974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41109f0 con 0x7fe0f4106730 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4110870 con 0x7fe0f4100720 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f9ae8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4110b70 con 0x7fe0f41010e0 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 0x7fe0f41974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f4100720 0x7fe0f4196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 0x7fe0f41974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58924/0 (socket says 172.21.15.116:58924) 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56470/0 (socket says 172.21.15.116:56470) 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f2ffd640 1 -- 172.21.15.116:0/1511624966 learned_addr learned my addr 172.21.15.116:0/1511624966 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 -- 172.21.15.116:0/1511624966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 msgr2=0x7fe0f41974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 0x7fe0f41974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 -- 172.21.15.116:0/1511624966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f4100720 msgr2=0x7fe0f4196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f4100720 0x7fe0f4196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 -- 172.21.15.116:0/1511624966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f419bf80 con 0x7fe0f4106730 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.078+0000 7fe0f3fff640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00da10 tx=0x7fe0ec00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.079+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00bd70 con 0x7fe0f4106730 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.079+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f419c270 con 0x7fe0f4106730 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.079+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe0ec003e20 con 0x7fe0f4106730 2024-03-19T02:51:44.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.079+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec0106c0 con 0x7fe0f4106730 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.079+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f41a3b00 con 0x7fe0f4106730 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.080+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f4068610 con 0x7fe0f4106730 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.081+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe0ec003970 con 0x7fe0f4106730 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.081+0000 7fe0f0ff9640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 0x7fe0dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.081+0000 7fe0f37fe640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 0x7fe0dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.081+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fe0ec0cf5c0 con 0x7fe0f4106730 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.082+0000 7fe0f37fe640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 0x7fe0dc0a4e30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4005fd0 tx=0x7fe0e400b390 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.086+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0ec09bf40 con 0x7fe0f4106730 2024-03-19T02:51:44.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71698 5.69 MB/sec execute 546 sec latency 2563.868 ms 2024-03-19T02:51:44.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.367+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0f4108c90 con 0x7fe0f4106730 2024-03-19T02:51:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.368+0000 7fe0f0ff9640 1 -- 172.21.15.116:0/1511624966 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11950 (secure 0 0 0) 0x7fe0ec09b840 con 0x7fe0f4106730 2024-03-19T02:51:44.368 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:51:44.368 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:50:15.839334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:51:44.368 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 msgr2=0x7fe0dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 0x7fe0dc0a4e30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4005fd0 tx=0x7fe0e400b390 comp rx=0 tx=0).stop 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 msgr2=0x7fe0f419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00da10 tx=0x7fe0ec00ded0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 shutdown_connections 2024-03-19T02:51:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe0dc0a2970 0x7fe0dc0a4e30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0f4106730 0x7fe0f419b850 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0f41010e0 0x7fe0f41974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 --2- 172.21.15.116:0/1511624966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0f4100720 0x7fe0f4196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 >> 172.21.15.116:0/1511624966 conn(0x7fe0f4077820 msgr2=0x7fe0f4103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 shutdown_connections 2024-03-19T02:51:44.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.371+0000 7fe0f9ae8640 1 -- 172.21.15.116:0/1511624966 wait complete. 2024-03-19T02:51:44.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:51:44.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:44 smithi116 ceph-mon[32263]: pgmap v756: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-03-19T02:51:44.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:44 smithi154 ceph-mon[35693]: pgmap v756: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 -- 172.21.15.116:0/4074636572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 msgr2=0x7f396c105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 --2- 172.21.15.116:0/4074636572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c105a50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f395c009900 tx=0x7f395c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 -- 172.21.15.116:0/4074636572 shutdown_connections 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 --2- 172.21.15.116:0/4074636572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c1017f0 0x7f396c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 --2- 172.21.15.116:0/4074636572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c100e30 0x7f396c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 --2- 172.21.15.116:0/4074636572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c105a50 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 -- 172.21.15.116:0/4074636572 >> 172.21.15.116:0/4074636572 conn(0x7f396c076c50 msgr2=0x7f396c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 -- 172.21.15.116:0/4074636572 shutdown_connections 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.540+0000 7f3971910640 1 -- 172.21.15.116:0/4074636572 wait complete. 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 Processor -- start 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 -- start start 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c100e30 0x7f396c06e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f396c06b810 con 0x7f396c105670 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f396c06b690 con 0x7f396c1017f0 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f3971910640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f396c06b990 con 0x7f396c100e30 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c100e30 0x7f396c06e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57398/0 (socket says 172.21.15.116:57398) 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56476/0 (socket says 172.21.15.116:56476) 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.541+0000 7f396a7fc640 1 -- 172.21.15.116:0/649646154 learned_addr learned my addr 172.21.15.116:0/649646154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 -- 172.21.15.116:0/649646154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c100e30 msgr2=0x7f396c06e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c100e30 0x7f396c06e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 -- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 msgr2=0x7f396c072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 -- 172.21.15.116:0/649646154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c073180 con 0x7f396c1017f0 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396b7fe640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f396a7fc640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f396000e9f0 tx=0x7f396000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f397090e640 1 -- 172.21.15.116:0/649646154 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396000cd80 con 0x7f396c1017f0 2024-03-19T02:51:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c073470 con 0x7f396c1017f0 2024-03-19T02:51:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f397090e640 1 -- 172.21.15.116:0/649646154 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39600108e0 con 0x7f396c1017f0 2024-03-19T02:51:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f397090e640 1 -- 172.21.15.116:0/649646154 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960018920 con 0x7f396c1017f0 2024-03-19T02:51:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.542+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1ad2d0 con 0x7f396c1017f0 2024-03-19T02:51:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.544+0000 7f397090e640 1 -- 172.21.15.116:0/649646154 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f3960010a90 con 0x7f396c1017f0 2024-03-19T02:51:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 Processor -- start 2024-03-19T02:51:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 -- start start 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ed290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa1474b0 con 0x55e5aa1e42d0 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa0efd70 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f39745d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa143550 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1e46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56488/0 (socket says 172.21.15.116:56488) 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396b7fe640 1 -- 172.21.15.116:0/910383114 learned_addr learned my addr 172.21.15.116:0/910383114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f397090e640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 0x7f39540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396a7fc640 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396b7fe640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 0x7f39540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 -- 172.21.15.116:0/910383114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 msgr2=0x55e5aa1efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1efbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 -- 172.21.15.116:0/910383114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 msgr2=0x55e5aa1e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.545+0000 7f396affd640 1 -- 172.21.15.116:0/910383114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5aa112210 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f397090e640 1 -- 172.21.15.116:0/649646154 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f39600cf500 con 0x7f396c1017f0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f396affd640 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ed290 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f395c000c00 tx=0x7f395c005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f13f93a144f09e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f396b7fe640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 0x7f39540a4e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3964005c40 tx=0x7f3964004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f3951ffb640 1 -- 172.21.15.116:0/910383114 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c02c800 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f3951ffb640 1 -- 172.21.15.116:0/910383114 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f395c02c9a0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.546+0000 7f3951ffb640 1 -- 172.21.15.116:0/910383114 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0046c0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- 172.21.15.116:0/910383114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 msgr2=0x55e5aa1ed290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ed290 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f395c000c00 tx=0x7f395c005e40 comp rx=0 tx=0).stop 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- 172.21.15.116:0/910383114 shutdown_connections 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ed290 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 --2- 172.21.15.116:0/910383114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- 172.21.15.116:0/910383114 >> 172.21.15.116:0/910383114 conn(0x55e5aa1dea00 msgr2=0x55e5aa1dede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- 172.21.15.116:0/910383114 shutdown_connections 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- 172.21.15.116:0/910383114 wait complete. 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.547+0000 7f39745d9740 1 -- start start 2024-03-19T02:51:44.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 Processor -- start 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa1f4970 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa103310 con 0x55e5aa1e42d0 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e5aa1f4af0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f39745d9740 10 client.0 _ll_register_callbacks cb 0x55e5aa0f72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56490/0 (socket says 172.21.15.116:56490) 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 -- 172.21.15.116:0/1853770431 learned_addr learned my addr 172.21.15.116:0/1853770431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396a7fc640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396affd640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 -- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 msgr2=0x55e5aa1ead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 -- 172.21.15.116:0/1853770431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 msgr2=0x55e5aa1ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ea810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 -- 172.21.15.116:0/1853770431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5a9e5b3f0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396affd640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f396b7fe640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f3964010b60 tx=0x7f3964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4bccf8c1ce1146 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.548+0000 7f394affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39640208f0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f3964020a90 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f395c02c9a0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3964020c90 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 10 client.22245 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e5a9f257d0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f395c02c9a0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f396401f040 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 10 client.22245 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 10 client.22245 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 20 client.22245 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa201f20 0x55e5aa2043a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 4 client.22245 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5aa1a53e0 con 0x55e5aa201f20 2024-03-19T02:51:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.549+0000 7f396affd640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa201f20 0x55e5aa2043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.550+0000 7f396affd640 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa201f20 0x55e5aa2043a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f58858d7bee4046e server_cookie=e0b6769ad37310cb in_seq=0 out_seq=0 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.550+0000 7f394affd640 10 client.22245 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.550+0000 7f394affd640 1 -- 172.21.15.116:0/1853770431 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e5aa1a53e0 con 0x55e5aa201f20 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.550+0000 7f394affd640 10 client.22245 handle_command_reply: tid=0 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.550+0000 7f39745d9740 1 client.22245 shutdown 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 msgr2=0x55e5aa1eb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f3964010b60 tx=0x7f3964004a40 comp rx=0 tx=0).stop 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 shutdown_connections 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa201f20 0x55e5aa2043a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa1eb290 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f394affd640 0 client.22245 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 --2- 172.21.15.116:0/1853770431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 >> 172.21.15.116:0/1853770431 conn(0x55e5aa1dea00 msgr2=0x55e5aa1e0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 shutdown_connections 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 1 -- 172.21.15.116:0/1853770431 wait complete. 2024-03-19T02:51:44.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.551+0000 7f39745d9740 20 client.22245 trim_cache size 0 max 16384 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 Processor -- start 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 -- start start 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1fd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b5200 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b6d40 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f39745d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b58d0 con 0x55e5aa1e42d0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f396affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f396b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56504/0 (socket says 172.21.15.116:56504) 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f396a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.566+0000 7f396b7fe640 1 -- 172.21.15.116:0/3177678956 learned_addr learned my addr 172.21.15.116:0/3177678956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 -- 172.21.15.116:0/3177678956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 msgr2=0x55e5aa1ecf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ecf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 -- 172.21.15.116:0/3177678956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e4bf0 msgr2=0x55e5aa1fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 -- 172.21.15.116:0/3177678956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5aa1e4a20 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396affd640 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f396b7fe640 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3964010fd0 tx=0x7f3964009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61575ed3d8fb92e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f39517fa640 1 -- 172.21.15.116:0/3177678956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964010740 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f39517fa640 1 -- 172.21.15.116:0/3177678956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f396403f070 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.567+0000 7f39517fa640 1 -- 172.21.15.116:0/3177678956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964022910 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- 172.21.15.116:0/3177678956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 msgr2=0x55e5aa209c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f3964010fd0 tx=0x7f3964009480 comp rx=0 tx=0).stop 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- 172.21.15.116:0/3177678956 shutdown_connections 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa209c70 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 --2- 172.21.15.116:0/3177678956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- 172.21.15.116:0/3177678956 >> 172.21.15.116:0/3177678956 conn(0x55e5aa1dea00 msgr2=0x55e5aa1dd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- 172.21.15.116:0/3177678956 shutdown_connections 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- 172.21.15.116:0/3177678956 wait complete. 2024-03-19T02:51:44.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 -- start start 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.568+0000 7f39745d9740 1 Processor -- start 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1f5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa2672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b6670 con 0x55e5aa1e42d0 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b5fa0 con 0x55e5aa1ed7d0 2024-03-19T02:51:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1b3200 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f39745d9740 10 client.0 _ll_register_callbacks cb 0x7f3964013480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58978/0 (socket says 172.21.15.116:58978) 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 -- 172.21.15.116:0/3870371789 learned_addr learned my addr 172.21.15.116:0/3870371789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396b7fe640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa2672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 -- 172.21.15.116:0/3870371789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 msgr2=0x55e5aa2672a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa2672a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 -- 172.21.15.116:0/3870371789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 msgr2=0x55e5aa1f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396a7fc640 1 -- 172.21.15.116:0/3870371789 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5aa1f67c0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.569+0000 7f396affd640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f396a7fc640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3960097cd0 tx=0x7f3960010de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afc8bc5c086c4818 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396009f910 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f39600cf0b0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f396403f070 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f39600cf2d0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 10 client.31945 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e5aa2688e0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f396403f070 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f396009acc0 con 0x55e5aa1e4bf0 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 10 client.31945 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 10 client.31945 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 20 client.31945 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.570+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa269010 0x55e5aa26b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f39745d9740 4 client.31945 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5aa1549f0 con 0x55e5aa269010 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f396affd640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa269010 0x55e5aa26b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f394affd640 10 client.31945 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f396affd640 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa269010 0x55e5aa26b430 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=246ad1ad302ccba9 server_cookie=c8895ae2c1398b29 in_seq=0 out_seq=0 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f394affd640 1 -- 172.21.15.116:0/3870371789 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5aa1549f0 con 0x55e5aa269010 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f394affd640 10 client.31945 handle_command_reply: tid=0 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.571+0000 7f39745d9740 1 client.31945 shutdown 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 msgr2=0x55e5aa1f5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3960097cd0 tx=0x7f3960010de0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 shutdown_connections 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x55e5aa269010 0x55e5aa26b430 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5aa1ed7d0 0x55e5aa2672a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5aa1e4bf0 0x55e5aa1f5ec0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 --2- 172.21.15.116:0/3870371789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5aa1e42d0 0x55e5aa1f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f394affd640 0 client.31945 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 >> 172.21.15.116:0/3870371789 conn(0x55e5aa1dea00 msgr2=0x55e5aa1eb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 shutdown_connections 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 1 -- 172.21.15.116:0/3870371789 wait complete. 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f39745d9740 20 client.31945 trim_cache size 0 max 16384 2024-03-19T02:51:44.578 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 msgr2=0x7f39540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 0x7f39540a4e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3964005c40 tx=0x7f3964004ab0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 msgr2=0x7f396c06e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f396000e9f0 tx=0x7f396000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 shutdown_connections 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f39540a2970 0x7f39540a4e30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f396c105670 0x7f396c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f396c1017f0 0x7f396c06e610 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 --2- 172.21.15.116:0/649646154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f396c100e30 0x7f396c06e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.572+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 >> 172.21.15.116:0/649646154 conn(0x7f396c076c50 msgr2=0x7f396c100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.573+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 shutdown_connections 2024-03-19T02:51:44.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.573+0000 7f3971910640 1 -- 172.21.15.116:0/649646154 wait complete. 2024-03-19T02:51:44.584 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:51:44.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:44 smithi162 ceph-mon[35761]: pgmap v756: 129 pgs: 129 active+clean; 1.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.750+0000 7f615b39f640 1 -- 172.21.15.116:0/2209060335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 msgr2=0x7f615410b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.750+0000 7f615b39f640 1 --2- 172.21.15.116:0/2209060335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f615410b600 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6144009900 tx=0x7f614402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 -- 172.21.15.116:0/2209060335 shutdown_connections 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 --2- 172.21.15.116:0/2209060335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615410bbd0 0x7f615410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 --2- 172.21.15.116:0/2209060335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f615410b600 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 --2- 172.21.15.116:0/2209060335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6154102800 0x7f6154102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 -- 172.21.15.116:0/2209060335 >> 172.21.15.116:0/2209060335 conn(0x7f61540772d0 msgr2=0x7f61540fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 -- 172.21.15.116:0/2209060335 shutdown_connections 2024-03-19T02:51:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 -- 172.21.15.116:0/2209060335 wait complete. 2024-03-19T02:51:44.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.751+0000 7f615b39f640 1 Processor -- start 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 -- start start 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 0x7f6154197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615410bbd0 0x7f615419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6154111850 con 0x7f6154103120 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61541116d0 con 0x7f6154102800 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615b39f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61541119d0 con 0x7f615410bbd0 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56536/0 (socket says 172.21.15.116:56536) 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 -- 172.21.15.116:0/1427501942 learned_addr learned my addr 172.21.15.116:0/1427501942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615a39d640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 0x7f6154197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615ab9e640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615410bbd0 0x7f615419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 -- 172.21.15.116:0/1427501942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615410bbd0 msgr2=0x7f615419c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615410bbd0 0x7f615419c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 -- 172.21.15.116:0/1427501942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 msgr2=0x7f6154197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 0x7f6154197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f6159b9c640 1 -- 172.21.15.116:0/1427501942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615419cda0 con 0x7f6154103120 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.752+0000 7f615a39d640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 0x7f6154197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f6159b9c640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6144005940 tx=0x7f614402c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f614b7fe640 1 -- 172.21.15.116:0/1427501942 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144043020 con 0x7f6154103120 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615419d030 con 0x7f6154103120 2024-03-19T02:51:44.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f614b7fe640 1 -- 172.21.15.116:0/1427501942 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6144004e00 con 0x7f6154103120 2024-03-19T02:51:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f614b7fe640 1 -- 172.21.15.116:0/1427501942 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144003970 con 0x7f6154103120 2024-03-19T02:51:44.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.753+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61541a4920 con 0x7f6154103120 2024-03-19T02:51:44.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.756+0000 7f614b7fe640 1 -- 172.21.15.116:0/1427501942 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6144046020 con 0x7f6154103120 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.756+0000 7f615f1ea740 1 Processor -- start 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.756+0000 7f615f1ea740 1 -- start start 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc46cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc45a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc3c24b0 con 0x55f2fc46d500 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc36ad70 con 0x55f2fc46cb50 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615f1ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc3be550 con 0x55f2fc459cb0 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f614b7fe640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 0x7f61400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc45a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc45a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59000/0 (socket says 172.21.15.116:59000) 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56546/0 (socket says 172.21.15.116:56546) 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f6159b9c640 1 -- 172.21.15.116:0/1142766380 learned_addr learned my addr 172.21.15.116:0/1142766380 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615ab9e640 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc46cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615ab9e640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 0x7f61400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f614b7fe640 1 -- 172.21.15.116:0/1427501942 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f61440f1070 con 0x7f6154103120 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 -- 172.21.15.116:0/1142766380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 msgr2=0x55f2fc45a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc45a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 -- 172.21.15.116:0/1142766380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 msgr2=0x55f2fc46cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc46cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.757+0000 7f615a39d640 1 -- 172.21.15.116:0/1142766380 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2fc38d210 con 0x55f2fc46d500 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.758+0000 7f615a39d640 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6150009fd0 tx=0x7f61500118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef846f40014c68b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.758+0000 7f615ab9e640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 0x7f61400a4e30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f2fc468330 tx=0x7f614c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.758+0000 7f6148bfe640 1 -- 172.21.15.116:0/1142766380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150004310 con 0x55f2fc46d500 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.758+0000 7f6148bfe640 1 -- 172.21.15.116:0/1142766380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61500044b0 con 0x55f2fc46d500 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f6148bfe640 1 -- 172.21.15.116:0/1142766380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61500035f0 con 0x55f2fc46d500 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 -- 172.21.15.116:0/1142766380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 msgr2=0x55f2fc459690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6150009fd0 tx=0x7f61500118d0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 -- 172.21.15.116:0/1142766380 shutdown_connections 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc45a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc459690 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1142766380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc46cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 -- 172.21.15.116:0/1142766380 >> 172.21.15.116:0/1142766380 conn(0x55f2fc45c150 msgr2=0x55f2fc45c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 -- 172.21.15.116:0/1142766380 shutdown_connections 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.759+0000 7f615f1ea740 1 -- 172.21.15.116:0/1142766380 wait complete. 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 -- start start 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 Processor -- start 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc471430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc473fe0 con 0x55f2fc46cb50 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc37e310 con 0x55f2fc46d500 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f2fc474160 con 0x55f2fc459cb0 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615f1ea740 10 client.0 _ll_register_callbacks cb 0x55f2fc3722d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56548/0 (socket says 172.21.15.116:56548) 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f6159b9c640 1 -- 172.21.15.116:0/4194258645 learned_addr learned my addr 172.21.15.116:0/4194258645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615a39d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.760+0000 7f615ab9e640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc471430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 -- 172.21.15.116:0/4194258645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 msgr2=0x55f2fc471430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc471430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 -- 172.21.15.116:0/4194258645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 msgr2=0x55f2fc470ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 -- 172.21.15.116:0/4194258645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2fc0d63f0 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f6159b9c640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615a39d640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4709b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f615001b040 tx=0x7f6150004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44f451a9f46ae678 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f613dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150029070 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f6150003b00 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61500044b0 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6150035050 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615f1ea740 10 client.31957 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.761+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f2fc1a07d0 con 0x55f2fc459cb0 2024-03-19T02:51:44.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61500044b0 con 0x55f2fc459cb0 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f6150004be0 con 0x55f2fc459cb0 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 10 client.31957 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 10 client.31957 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 20 client.31957 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4813e0 0x55f2fc4837d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 4 client.31957 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2fc4203e0 con 0x55f2fc4813e0 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f6159b9c640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4813e0 0x55f2fc4837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f613dffb640 10 client.31957 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:44.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.762+0000 7f6159b9c640 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4813e0 0x55f2fc4837d0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=336f8e8f74335c0a server_cookie=cf14e2510bdc4351 in_seq=0 out_seq=0 2024-03-19T02:51:44.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f613dffb640 1 -- 172.21.15.116:0/4194258645 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2fc4203e0 con 0x55f2fc4813e0 2024-03-19T02:51:44.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f613dffb640 10 client.31957 handle_command_reply: tid=0 2024-03-19T02:51:44.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f615f1ea740 1 client.31957 shutdown 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 msgr2=0x55f2fc4709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4709b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f615001b040 tx=0x7f6150004970 comp rx=0 tx=0).stop 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.763+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 shutdown_connections 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc471430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4813e0 0x55f2fc4837d0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46cb50 0x55f2fc470ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 --2- 172.21.15.116:0/4194258645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4709b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f613dffb640 0 client.31957 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 >> 172.21.15.116:0/4194258645 conn(0x55f2fc45c150 msgr2=0x55f2fc45b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 shutdown_connections 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 1 -- 172.21.15.116:0/4194258645 wait complete. 2024-03-19T02:51:44.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.764+0000 7f615f1ea740 20 client.31957 trim_cache size 0 max 16384 2024-03-19T02:51:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 Processor -- start 2024-03-19T02:51:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 -- start start 2024-03-19T02:51:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc47ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc45af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61541ac770 con 0x55f2fc46d500 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61541ae2b0 con 0x55f2fc46cb50 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.778+0000 7f615f1ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61541ace40 con 0x55f2fc459cb0 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615ab9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc45af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc47ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59004/0 (socket says 172.21.15.116:59004) 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 -- 172.21.15.116:0/1923181852 learned_addr learned my addr 172.21.15.116:0/1923181852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 -- 172.21.15.116:0/1923181852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 msgr2=0x55f2fc47ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc47ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 -- 172.21.15.116:0/1923181852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 msgr2=0x55f2fc45af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc45af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 -- 172.21.15.116:0/1923181852 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2fc45ad90 con 0x55f2fc459cb0 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f615a39d640 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f615000a010 tx=0x7f6150005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b58d6987d7e3e0bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f613ffff640 1 -- 172.21.15.116:0/1923181852 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150029070 con 0x55f2fc459cb0 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f613ffff640 1 -- 172.21.15.116:0/1923181852 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6150003db0 con 0x55f2fc459cb0 2024-03-19T02:51:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.779+0000 7f613ffff640 1 -- 172.21.15.116:0/1923181852 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150000e60 con 0x55f2fc459cb0 2024-03-19T02:51:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- 172.21.15.116:0/1923181852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 msgr2=0x55f2fc4730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f615000a010 tx=0x7f6150005750 comp rx=0 tx=0).stop 2024-03-19T02:51:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- 172.21.15.116:0/1923181852 shutdown_connections 2024-03-19T02:51:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc46d500 0x55f2fc45af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46cb50 0x55f2fc47ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 --2- 172.21.15.116:0/1923181852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc459cb0 0x55f2fc4730e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- 172.21.15.116:0/1923181852 >> 172.21.15.116:0/1923181852 conn(0x55f2fc45c150 msgr2=0x55f2fc458d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- 172.21.15.116:0/1923181852 shutdown_connections 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- 172.21.15.116:0/1923181852 wait complete. 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 -- start start 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 Processor -- start 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.780+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc488500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61541adbe0 con 0x55f2fc459cb0 2024-03-19T02:51:44.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61541ad510 con 0x55f2fc46d500 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f61541aa770 con 0x55f2fc46cb50 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 10 client.0 _ll_register_callbacks cb 0x7f614c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59020/0 (socket says 172.21.15.116:59020) 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 -- 172.21.15.116:0/781786031 learned_addr learned my addr 172.21.15.116:0/781786031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615a39d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56564/0 (socket says 172.21.15.116:56564) 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615ab9e640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc488500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 -- 172.21.15.116:0/781786031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 msgr2=0x55f2fc488500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc488500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 -- 172.21.15.116:0/781786031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 msgr2=0x55f2fc479690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 -- 172.21.15.116:0/781786031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2fc488a40 con 0x55f2fc46cb50 2024-03-19T02:51:44.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615a39d640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f6159b9c640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f614402c820 tx=0x7f61440bd300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f27903da03f775de server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f613dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144004fb0 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f61440b9a70 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.781+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6150003db0 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f61440c1550 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 10 client.31963 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f2fc4e72f0 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6150003db0 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f61440c1170 con 0x55f2fc46cb50 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 10 client.31963 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 10 client.31963 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 20 client.31963 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4e7a20 0x55f2fc4e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 4 client.31963 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2fc3cf9f0 con 0x55f2fc4e7a20 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.782+0000 7f615a39d640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4e7a20 0x55f2fc4e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f613dffb640 10 client.31963 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615a39d640 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4e7a20 0x55f2fc4e9e40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=33645566b2b51cca server_cookie=b8b87420c8c13f35 in_seq=0 out_seq=0 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f613dffb640 1 -- 172.21.15.116:0/781786031 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2fc3cf9f0 con 0x55f2fc4e7a20 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f613dffb640 10 client.31963 handle_command_reply: tid=0 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 client.31963 shutdown 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 msgr2=0x55f2fc487fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f614402c820 tx=0x7f61440bd300 comp rx=0 tx=0).stop 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 shutdown_connections 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2fc46d500 0x55f2fc488500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f2fc46cb50 0x55f2fc487fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55f2fc4e7a20 0x55f2fc4e9e40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 --2- 172.21.15.116:0/781786031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f2fc459cb0 0x55f2fc479690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 >> 172.21.15.116:0/781786031 conn(0x55f2fc45c150 msgr2=0x55f2fc46f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f613dffb640 0 client.31963 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 shutdown_connections 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.783+0000 7f615f1ea740 1 -- 172.21.15.116:0/781786031 wait complete. 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615f1ea740 20 client.31963 trim_cache size 0 max 16384 2024-03-19T02:51:44.790 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 msgr2=0x7f61400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 0x7f61400a4e30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f2fc468330 tx=0x7f614c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 msgr2=0x7f61541982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6144005940 tx=0x7f614402c690 comp rx=0 tx=0).stop 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 shutdown_connections 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f61400a2970 0x7f61400a4e30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f615410bbd0 0x7f615419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6154103120 0x7f61541982f0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 --2- 172.21.15.116:0/1427501942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6154102800 0x7f6154197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.784+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 >> 172.21.15.116:0/1427501942 conn(0x7f61540772d0 msgr2=0x7f61540fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.785+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 shutdown_connections 2024-03-19T02:51:44.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.785+0000 7f615b39f640 1 -- 172.21.15.116:0/1427501942 wait complete. 2024-03-19T02:51:44.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 -- 172.21.15.116:0/1258863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 msgr2=0x7fd1e0107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 --2- 172.21.15.116:0/1258863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e0107a10 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009930 tx=0x7fd1d002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 -- 172.21.15.116:0/1258863662 shutdown_connections 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 --2- 172.21.15.116:0/1258863662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 0x7fd1e010e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 --2- 172.21.15.116:0/1258863662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e00fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 --2- 172.21.15.116:0/1258863662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e0107a10 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 -- 172.21.15.116:0/1258863662 >> 172.21.15.116:0/1258863662 conn(0x7fd1e0076aa0 msgr2=0x7fd1e00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 -- 172.21.15.116:0/1258863662 shutdown_connections 2024-03-19T02:51:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.920+0000 7fd1df59e640 1 -- 172.21.15.116:0/1258863662 wait complete. 2024-03-19T02:51:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 Processor -- start 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 -- start start 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 0x7fd1e01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01118f0 con 0x7fd1e0107630 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0111770 con 0x7fd1e01004d0 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1df59e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0111a70 con 0x7fd1e00ffb10 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1d7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 0x7fd1e01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59038/0 (socket says 172.21.15.116:59038) 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1de59c640 1 -- 172.21.15.116:0/840734427 learned_addr learned my addr 172.21.15.116:0/840734427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.921+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56586/0 (socket says 172.21.15.116:56586) 2024-03-19T02:51:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 -- 172.21.15.116:0/840734427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 msgr2=0x7fd1e01983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 0x7fd1e01983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 -- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 msgr2=0x7fd1e019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 -- 172.21.15.116:0/840734427 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e019ce50 con 0x7fd1e00ffb10 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1de59c640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009230 tx=0x7fd1d002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1d77fe640 1 -- 172.21.15.116:0/840734427 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0043020 con 0x7fd1e00ffb10 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1d77fe640 1 -- 172.21.15.116:0/840734427 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1d002ca70 con 0x7fd1e00ffb10 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e019d0e0 con 0x7fd1e00ffb10 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1d77fe640 1 -- 172.21.15.116:0/840734427 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0003a50 con 0x7fd1e00ffb10 2024-03-19T02:51:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.922+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e0100c20 con 0x7fd1e00ffb10 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.925+0000 7fd1d77fe640 1 -- 172.21.15.116:0/840734427 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd1d0046020 con 0x7fd1e00ffb10 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.925+0000 7fd1e75ec740 1 Processor -- start 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 -- start start 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1d77fe640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 0x7fd1cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x559501981470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x559501983da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595018db4b0 con 0x5595019819b0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559501883d70 con 0x559501978dd0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5595018d7550 con 0x5595019784b0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1d77fe640 1 -- 172.21.15.116:0/840734427 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fd1d00f1170 con 0x7fd1e00ffb10 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 0x7fd1cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x559501983da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59052/0 (socket says 172.21.15.116:59052) 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 -- 172.21.15.116:0/2649078535 learned_addr learned my addr 172.21.15.116:0/2649078535 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x559501981470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 -- 172.21.15.116:0/2649078535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 msgr2=0x559501981470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x559501981470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 -- 172.21.15.116:0/2649078535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 msgr2=0x559501983da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.926+0000 7fd1de59c640 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x559501983da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1de59c640 1 -- 172.21.15.116:0/2649078535 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595018a6210 con 0x5595019784b0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 0x7fd1cc0a4e30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009fd0 tx=0x7fd1d80116b0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1de59c640 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00b9e70 tx=0x7fd1d00b9870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=defc47478d69e808 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1d4ff9640 1 -- 172.21.15.116:0/2649078535 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00bd4a0 con 0x5595019784b0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1d4ff9640 1 -- 172.21.15.116:0/2649078535 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1d00c15f0 con 0x5595019784b0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.927+0000 7fd1d4ff9640 1 -- 172.21.15.116:0/2649078535 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00bb830 con 0x5595019784b0 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- 172.21.15.116:0/2649078535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 msgr2=0x559501978890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00b9e70 tx=0x7fd1d00b9870 comp rx=0 tx=0).stop 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- 172.21.15.116:0/2649078535 shutdown_connections 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x559501983da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x559501981470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/2649078535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x559501978890 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- 172.21.15.116:0/2649078535 >> 172.21.15.116:0/2649078535 conn(0x559501972be0 msgr2=0x559501972fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- 172.21.15.116:0/2649078535 shutdown_connections 2024-03-19T02:51:44.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- 172.21.15.116:0/2649078535 wait complete. 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- start start 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 Processor -- start 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.928+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559501984560 con 0x5595019784b0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559501897310 con 0x559501978dd0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5595019846e0 con 0x5595019819b0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 10 client.0 _ll_register_callbacks cb 0x55950188b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56604/0 (socket says 172.21.15.116:56604) 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 -- 172.21.15.116:0/3488676894 learned_addr learned my addr 172.21.15.116:0/3488676894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 -- 172.21.15.116:0/3488676894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 msgr2=0x55950197f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 -- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 msgr2=0x55950197ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 -- 172.21.15.116:0/3488676894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595015ef3f0 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002c780 tx=0x7fd1d0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927066b1d90d2735 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00bb540 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fd1d00d6260 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1d00c15f0 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd1d00bd4a0 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 10 client.22281 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.929+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5595016b97d0 con 0x5595019784b0 2024-03-19T02:51:44.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.930+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1d00c15f0 con 0x5595019784b0 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.930+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fd1d00bd700 con 0x5595019784b0 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.930+0000 7fd1e75ec740 10 client.22281 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.930+0000 7fd1e75ec740 10 client.22281 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1e75ec740 20 client.22281 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559501991920 0x559501993d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1e75ec740 4 client.22281 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595019393e0 con 0x559501991920 2024-03-19T02:51:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559501991920 0x559501993d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1c5ffb640 10 client.22281 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.931+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559501991920 0x559501993d80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a54315c3199fd7e server_cookie=3dbf41193799a082 in_seq=0 out_seq=0 2024-03-19T02:51:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3488676894 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5595019393e0 con 0x559501991920 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1c5ffb640 10 client.22281 handle_command_reply: tid=0 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 client.22281 shutdown 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 msgr2=0x55950197e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fd1d002c780 tx=0x7fd1d0005b50 comp rx=0 tx=0).stop 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 shutdown_connections 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x559501991920 0x559501993d80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3488676894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019784b0 0x55950197e8f0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1c5ffb640 0 client.22281 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 >> 172.21.15.116:0/3488676894 conn(0x559501972be0 msgr2=0x559501974930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 shutdown_connections 2024-03-19T02:51:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.932+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3488676894 wait complete. 2024-03-19T02:51:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.933+0000 7fd1e75ec740 20 client.22281 trim_cache size 0 max 16384 2024-03-19T02:51:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 Processor -- start 2024-03-19T02:51:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 -- start start 2024-03-19T02:51:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595019784b0 0x55950198d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01ac7a0 con 0x559501978dd0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01ae2e0 con 0x5595019784b0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01ace70 con 0x5595019819b0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56612/0 (socket says 172.21.15.116:56612) 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1d7fff640 1 -- 172.21.15.116:0/554956626 learned_addr learned my addr 172.21.15.116:0/554956626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595019784b0 0x55950198d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.950+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 -- 172.21.15.116:0/554956626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 msgr2=0x55950197f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 -- 172.21.15.116:0/554956626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595019784b0 msgr2=0x55950198d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595019784b0 0x55950198d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 -- 172.21.15.116:0/554956626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559501972a10 con 0x559501978dd0 2024-03-19T02:51:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x559501977f80 tx=0x7fd1c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cf4bf6e1cbb26c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1c7fff640 1 -- 172.21.15.116:0/554956626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80035f0 con 0x559501978dd0 2024-03-19T02:51:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1c7fff640 1 -- 172.21.15.116:0/554956626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1c8003c40 con 0x559501978dd0 2024-03-19T02:51:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.951+0000 7fd1c7fff640 1 -- 172.21.15.116:0/554956626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c801cb20 con 0x559501978dd0 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- 172.21.15.116:0/554956626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 msgr2=0x55950197da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x559501977f80 tx=0x7fd1c8005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- 172.21.15.116:0/554956626 shutdown_connections 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019819b0 0x55950197f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559501978dd0 0x55950197da50 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/554956626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595019784b0 0x55950198d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- 172.21.15.116:0/554956626 >> 172.21.15.116:0/554956626 conn(0x559501972be0 msgr2=0x559501971470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- 172.21.15.116:0/554956626 shutdown_connections 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- 172.21.15.116:0/554956626 wait complete. 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 -- start start 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.952+0000 7fd1e75ec740 1 Processor -- start 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x55950197bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01adc10 con 0x5595019819b0 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01ad540 con 0x559501978dd0 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e01aa7a0 con 0x5595019784b0 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1e75ec740 10 client.0 _ll_register_callbacks cb 0x7fd1d8011b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x55950197bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59088/0 (socket says 172.21.15.116:59088) 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 -- 172.21.15.116:0/3980924045 learned_addr learned my addr 172.21.15.116:0/3980924045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 -- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 msgr2=0x55950197b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 -- 172.21.15.116:0/3980924045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 msgr2=0x55950197bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x55950197bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 -- 172.21.15.116:0/3980924045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595019f6a90 con 0x5595019784b0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1ded9d640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x55950197bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1de59c640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00bd1a0 tx=0x7fd1d00bb7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38faa0f938d2f25e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.953+0000 7fd1c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00bdba0 con 0x5595019784b0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7fd1d00ee670 con 0x5595019784b0 2024-03-19T02:51:44.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c8003c40 con 0x5595019784b0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd1d00c15f0 con 0x5595019784b0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 10 client.31981 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5595019f6a90 con 0x5595019784b0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1c8003c40 con 0x5595019784b0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fd1d00c57b0 con 0x5595019784b0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 10 client.31981 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 10 client.31981 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 20 client.31981 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5595019f8060 0x5595019fa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 4 client.31981 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.954+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595018e89f0 con 0x5595019f8060 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.955+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5595019f8060 0x5595019fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.955+0000 7fd1c5ffb640 10 client.31981 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:44.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.955+0000 7fd1d7fff640 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5595019f8060 0x5595019fa480 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d58ce0a651b18f2b server_cookie=7f758917fd22b3ec in_seq=0 out_seq=0 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1c5ffb640 1 -- 172.21.15.116:0/3980924045 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595018e89f0 con 0x5595019f8060 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1c5ffb640 10 client.31981 handle_command_reply: tid=0 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 client.31981 shutdown 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 msgr2=0x55950197b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00bd1a0 tx=0x7fd1d00bb7a0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 shutdown_connections 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x5595019f8060 0x5595019fa480 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595019819b0 0x55950197bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559501978dd0 0x55950197b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 --2- 172.21.15.116:0/3980924045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595019784b0 0x55950197b330 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 >> 172.21.15.116:0/3980924045 conn(0x559501972be0 msgr2=0x559501971d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1c5ffb640 0 client.31981 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 shutdown_connections 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 1 -- 172.21.15.116:0/3980924045 wait complete. 2024-03-19T02:51:44.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.956+0000 7fd1e75ec740 20 client.31981 trim_cache size 0 max 16384 2024-03-19T02:51:44.962 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 msgr2=0x7fd1cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 0x7fd1cc0a4e30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009fd0 tx=0x7fd1d80116b0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 msgr2=0x7fd1e0197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009230 tx=0x7fd1d002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 shutdown_connections 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd1cc0a2970 0x7fd1cc0a4e30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e0107630 0x7fd1e019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1e01004d0 0x7fd1e01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 --2- 172.21.15.116:0/840734427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1e00ffb10 0x7fd1e0197e60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:44.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.957+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 >> 172.21.15.116:0/840734427 conn(0x7fd1e0076aa0 msgr2=0x7fd1e0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:44.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.958+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 shutdown_connections 2024-03-19T02:51:44.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:44.958+0000 7fd1df59e640 1 -- 172.21.15.116:0/840734427 wait complete. 2024-03-19T02:51:44.972 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:51:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 -- 172.21.15.116:0/4162413464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8100080 msgr2=0x7f7ef8100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/4162413464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8100080 0x7f7ef8100460 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800aa10 tx=0x7f7ee802cf70 comp rx=0 tx=0).stop 2024-03-19T02:51:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 -- 172.21.15.116:0/4162413464 shutdown_connections 2024-03-19T02:51:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/4162413464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef810cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/4162413464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100a30 0x7f7ef8104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/4162413464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8100080 0x7f7ef8100460 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 -- 172.21.15.116:0/4162413464 >> 172.21.15.116:0/4162413464 conn(0x7f7ef8076c50 msgr2=0x7f7ef80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 -- 172.21.15.116:0/4162413464 shutdown_connections 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.098+0000 7f7efd6a7640 1 -- 172.21.15.116:0/4162413464 wait complete. 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 Processor -- start 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 -- start start 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100080 0x7f7ef8195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8196630 0x7f7ef81a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810f830 con 0x7f7ef8196630 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810f6b0 con 0x7f7ef8100080 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7efd6a7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810f9b0 con 0x7f7ef8105150 2024-03-19T02:51:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59116/0 (socket says 172.21.15.116:59116) 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3237117871 learned_addr learned my addr 172.21.15.116:0/3237117871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100080 0x7f7ef8195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8196630 0x7f7ef81a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3237117871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100080 msgr2=0x7f7ef8195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.099+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100080 0x7f7ef8195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8196630 msgr2=0x7f7ef81a6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8196630 0x7f7ef81a6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3237117871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef81a6870 con 0x7f7ef8105150 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00e9c0 tx=0x7f7eec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7edffff640 1 -- 172.21.15.116:0/3237117871 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec00cd80 con 0x7f7ef8105150 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef81a6b60 con 0x7f7ef8105150 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7edffff640 1 -- 172.21.15.116:0/3237117871 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7eec0054d0 con 0x7f7ef8105150 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7edffff640 1 -- 172.21.15.116:0/3237117871 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec010430 con 0x7f7ef8105150 2024-03-19T02:51:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.100+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a7100 con 0x7f7ef8105150 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 Processor -- start 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 -- start start 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f994bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7edffff640 1 -- 172.21.15.116:0/3237117871 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7eec004fd0 con 0x7f7ef8105150 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b9f8ec4b0 con 0x562b9f9927d0 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b9f894d70 con 0x562b9f9892d0 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7f00368740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b9f8e8550 con 0x562b9f989bf0 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f994bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57562/0 (socket says 172.21.15.116:57562) 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef77fe640 1 -- 172.21.15.116:0/3139146699 learned_addr learned my addr 172.21.15.116:0/3139146699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.103+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59122/0 (socket says 172.21.15.116:59122) 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 -- 172.21.15.116:0/3139146699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 msgr2=0x562b9f992290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 -- 172.21.15.116:0/3139146699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 msgr2=0x562b9f994bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f994bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 -- 172.21.15.116:0/3139146699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b9f8b7210 con 0x562b9f9892d0 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009f50 tx=0x7f7ef00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e0a8fcdf6d68ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7edffff640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 0x7f7ee00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 0x7f7ee00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7edd7fa640 1 -- 172.21.15.116:0/3139146699 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0004d20 con 0x562b9f9892d0 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7edffff640 1 -- 172.21.15.116:0/3237117871 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7eec0cf600 con 0x7f7ef8105150 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7edd7fa640 1 -- 172.21.15.116:0/3139146699 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ef0004050 con 0x562b9f9892d0 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.104+0000 7f7edd7fa640 1 -- 172.21.15.116:0/3139146699 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0005280 con 0x562b9f9892d0 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- 172.21.15.116:0/3139146699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 msgr2=0x562b9f9896b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009f50 tx=0x7f7ef00118d0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 0x7f7ee00a4e60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8006010 tx=0x7f7ee8005b50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- 172.21.15.116:0/3139146699 shutdown_connections 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f994bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f992290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 --2- 172.21.15.116:0/3139146699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f9896b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- 172.21.15.116:0/3139146699 >> 172.21.15.116:0/3139146699 conn(0x562b9f983a00 msgr2=0x562b9f983de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- 172.21.15.116:0/3139146699 shutdown_connections 2024-03-19T02:51:45.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- 172.21.15.116:0/3139146699 wait complete. 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 -- start start 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.105+0000 7f7f00368740 1 Processor -- start 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f98f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f98fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562b9f9955c0 con 0x562b9f9927d0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562b9f8a8310 con 0x562b9f9892d0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b9f995740 con 0x562b9f989bf0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7f00368740 10 client.0 _ll_register_callbacks cb 0x562b9f89c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56648/0 (socket says 172.21.15.116:56648) 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f98f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 -- 172.21.15.116:0/1190333332 learned_addr learned my addr 172.21.15.116:0/1190333332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f98fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 -- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 msgr2=0x562b9f98fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f98fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 -- 172.21.15.116:0/1190333332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 msgr2=0x562b9f98f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f98f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 -- 172.21.15.116:0/1190333332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b9f6003f0 con 0x562b9f9927d0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7ef001b040 tx=0x7f7ef0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d777d2aa9c1df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.106+0000 7f7edaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0028020 con 0x562b9f9927d0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7ef0005050 con 0x562b9f9927d0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ef0004050 con 0x562b9f9927d0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7ef0004e10 con 0x562b9f9927d0 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 10 client.22299 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562b9f6ca7d0 con 0x562b9f9927d0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ef0004050 con 0x562b9f9927d0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7ef0024830 con 0x562b9f9927d0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 10 client.22299 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 10 client.22299 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 20 client.22299 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9f9a2960 0x562b9f9a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 4 client.22299 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b9f94a3e0 con 0x562b9f9a2960 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.107+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9f9a2960 0x562b9f9a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.108+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9f9a2960 0x562b9f9a4d50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a11d49be8f367f05 server_cookie=69772324736d8a7a in_seq=0 out_seq=0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.108+0000 7f7edaff5640 10 client.22299 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.108+0000 7f7edaff5640 1 -- 172.21.15.116:0/1190333332 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b9f94a3e0 con 0x562b9f9a2960 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.108+0000 7f7edaff5640 10 client.22299 handle_command_reply: tid=0 2024-03-19T02:51:45.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.108+0000 7f7f00368740 1 client.22299 shutdown 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 msgr2=0x562b9f990290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7ef001b040 tx=0x7f7ef0000f40 comp rx=0 tx=0).stop 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 shutdown_connections 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9f9a2960 0x562b9f9a4d50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9927d0 0x562b9f990290 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f989bf0 0x562b9f98fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 --2- 172.21.15.116:0/1190333332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9892d0 0x562b9f98f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 >> 172.21.15.116:0/1190333332 conn(0x562b9f983a00 msgr2=0x562b9f985750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7edaff5640 0 client.22299 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 shutdown_connections 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 1 -- 172.21.15.116:0/1190333332 wait complete. 2024-03-19T02:51:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.109+0000 7f7f00368740 20 client.22299 trim_cache size 0 max 16384 2024-03-19T02:51:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.124+0000 7f7f00368740 1 Processor -- start 2024-03-19T02:51:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.124+0000 7f7f00368740 1 -- start start 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.124+0000 7f7f00368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.124+0000 7f7f00368740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f989bf0 0x562b9f99e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.124+0000 7f7f00368740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9927d0 0x562b9f9aa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7f00368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b0e80 con 0x562b9f9892d0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7f00368740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b29c0 con 0x562b9f989bf0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7f00368740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b1550 con 0x562b9f9927d0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56650/0 (socket says 172.21.15.116:56650) 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/2292054065 learned_addr learned my addr 172.21.15.116:0/2292054065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9927d0 0x562b9f9aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f989bf0 0x562b9f99e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/2292054065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9927d0 msgr2=0x562b9f9aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9927d0 0x562b9f9aa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/2292054065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f989bf0 msgr2=0x562b9f99e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f989bf0 0x562b9f99e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/2292054065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b9f989a20 con 0x562b9f9892d0 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8035590 tx=0x7f7ee8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba7e70189c0419b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7edcff9640 1 -- 172.21.15.116:0/2292054065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8004840 con 0x562b9f9892d0 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7edcff9640 1 -- 172.21.15.116:0/2292054065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee8059070 con 0x562b9f9892d0 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.125+0000 7f7edcff9640 1 -- 172.21.15.116:0/2292054065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8004040 con 0x562b9f9892d0 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- 172.21.15.116:0/2292054065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 msgr2=0x562b9f991f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8035590 tx=0x7f7ee8043000 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- 172.21.15.116:0/2292054065 shutdown_connections 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9927d0 0x562b9f9aa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f989bf0 0x562b9f99e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 --2- 172.21.15.116:0/2292054065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f9892d0 0x562b9f991f40 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- 172.21.15.116:0/2292054065 >> 172.21.15.116:0/2292054065 conn(0x562b9f983a00 msgr2=0x562b9f982ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- 172.21.15.116:0/2292054065 shutdown_connections 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- 172.21.15.116:0/2292054065 wait complete. 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 -- start start 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.126+0000 7f7f00368740 1 Processor -- start 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 0x562b9f99ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 0x562b9f996510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b22f0 con 0x562b9f989bf0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81b1c20 con 0x562b9f9927d0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81aee80 con 0x562b9f9892d0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 10 client.0 _ll_register_callbacks cb 0x7f7eec0d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56664/0 (socket says 172.21.15.116:56664) 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 0x562b9f99ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3734750447 learned_addr learned my addr 172.21.15.116:0/3734750447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 0x562b9f996510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 msgr2=0x562b9f99ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 0x562b9f99ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3734750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 msgr2=0x562b9f996510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 0x562b9f996510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 -- 172.21.15.116:0/3734750447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b9f996d40 con 0x562b9f989bf0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef77fe640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 0x562b9f996510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 0x562b9f99ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7ef67fc640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f7eec098250 tx=0x7f7eec09b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5723181825aeef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7edaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec09bf80 con 0x562b9f989bf0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f7eec0860b0 con 0x562b9f989bf0 2024-03-19T02:51:45.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.127+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee8059070 con 0x562b9f989bf0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7eec09b880 con 0x562b9f989bf0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 10 client.22305 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x562b9fa08ad0 con 0x562b9f989bf0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee8059070 con 0x562b9f989bf0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7eec0a3b90 con 0x562b9f989bf0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 10 client.22305 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 10 client.22305 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 20 client.22305 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9fa09150 0x562b9fa0b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 4 client.22305 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b9f8f99f0 con 0x562b9fa09150 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9fa09150 0x562b9fa0b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7edaff5640 10 client.22305 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.128+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9fa09150 0x562b9fa0b570 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c790618391a5d7b4 server_cookie=c22cb1257f94bd01 in_seq=0 out_seq=0 2024-03-19T02:51:45.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7edaff5640 1 -- 172.21.15.116:0/3734750447 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b9f8f99f0 con 0x562b9fa09150 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7edaff5640 10 client.22305 handle_command_reply: tid=0 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 client.22305 shutdown 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 msgr2=0x562b9f995fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f7eec098250 tx=0x7f7eec09b710 comp rx=0 tx=0).stop 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 shutdown_connections 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x562b9fa09150 0x562b9fa0b570 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7edaff5640 0 client.22305 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562b9f9927d0 0x562b9f996510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b9f989bf0 0x562b9f995fd0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 --2- 172.21.15.116:0/3734750447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b9f9892d0 0x562b9f99ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 >> 172.21.15.116:0/3734750447 conn(0x562b9f983a00 msgr2=0x562b9f98f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 shutdown_connections 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 1 -- 172.21.15.116:0/3734750447 wait complete. 2024-03-19T02:51:45.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7f00368740 20 client.22305 trim_cache size 0 max 16384 2024-03-19T02:51:45.137 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 msgr2=0x7f7ee00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 0x7f7ee00a4e60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8006010 tx=0x7f7ee8005b50 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 msgr2=0x7f7ef81960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00e9c0 tx=0x7f7eec00ee80 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 shutdown_connections 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7ee00a29a0 0x7f7ee00a4e60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ef8196630 0x7f7ef81a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef8105150 0x7f7ef81960f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 --2- 172.21.15.116:0/3237117871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef8100080 0x7f7ef8195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.129+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 >> 172.21.15.116:0/3237117871 conn(0x7f7ef8076c50 msgr2=0x7f7ef8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.130+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 shutdown_connections 2024-03-19T02:51:45.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.130+0000 7f7efd6a7640 1 -- 172.21.15.116:0/3237117871 wait complete. 2024-03-19T02:51:45.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:51:45.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 -- 172.21.15.116:0/1395995125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 msgr2=0x7f807c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 --2- 172.21.15.116:0/1395995125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c102e20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f8070009900 tx=0x7f807002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 -- 172.21.15.116:0/1395995125 shutdown_connections 2024-03-19T02:51:45.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 --2- 172.21.15.116:0/1395995125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c10bd80 0x7f807c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 --2- 172.21.15.116:0/1395995125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c103360 0x7f807c10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 --2- 172.21.15.116:0/1395995125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c102e20 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 -- 172.21.15.116:0/1395995125 >> 172.21.15.116:0/1395995125 conn(0x7f807c0778d0 msgr2=0x7f807c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 -- 172.21.15.116:0/1395995125 shutdown_connections 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.271+0000 7f808300b640 1 -- 172.21.15.116:0/1395995125 wait complete. 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 Processor -- start 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 -- start start 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c103360 0x7f807c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 0x7f807c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f807c111a30 con 0x7f807c102a40 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1118b0 con 0x7f807c103360 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f808300b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f807c111bb0 con 0x7f807c10bd80 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56682/0 (socket says 172.21.15.116:56682) 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 -- 172.21.15.116:0/4084843998 learned_addr learned my addr 172.21.15.116:0/4084843998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f807bfff640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c103360 0x7f807c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f80815a8640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 0x7f807c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 -- 172.21.15.116:0/4084843998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 msgr2=0x7f807c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 0x7f807c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 -- 172.21.15.116:0/4084843998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c103360 msgr2=0x7f807c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c103360 0x7f807c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f8080da7640 1 -- 172.21.15.116:0/4084843998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c19cf60 con 0x7f807c102a40 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.272+0000 7f80815a8640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 0x7f807c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f8080da7640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f8070005940 tx=0x7f807002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f8079ffb640 1 -- 172.21.15.116:0/4084843998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070043020 con 0x7f807c102a40 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c19d1f0 con 0x7f807c102a40 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f8079ffb640 1 -- 172.21.15.116:0/4084843998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8070004e00 con 0x7f807c102a40 2024-03-19T02:51:45.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f8079ffb640 1 -- 172.21.15.116:0/4084843998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070003970 con 0x7f807c102a40 2024-03-19T02:51:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.273+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c1005c0 con 0x7f807c102a40 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8079ffb640 1 -- 172.21.15.116:0/4084843998 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8070046020 con 0x7f807c102a40 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 Processor -- start 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 -- start start 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a116b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a116b9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558a116134b0 con 0x558a116b0bf0 2024-03-19T02:51:45.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558a115bbd70 con 0x558a116b97d0 2024-03-19T02:51:45.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.276+0000 7f8085bf9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a1160f550 con 0x558a116b02d0 2024-03-19T02:51:45.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f80815a8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a116b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f80815a8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a116b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59170/0 (socket says 172.21.15.116:59170) 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f80815a8640 1 -- 172.21.15.116:0/3504507066 learned_addr learned my addr 172.21.15.116:0/3504507066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f8080da7640 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a116b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 -- 172.21.15.116:0/3504507066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 msgr2=0x558a116b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a116b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 -- 172.21.15.116:0/3504507066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 msgr2=0x558a116b9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a116b9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f807bfff640 1 -- 172.21.15.116:0/3504507066 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a115de210 con 0x558a116b97d0 2024-03-19T02:51:45.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.277+0000 7f8079ffb640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 0x7f806c0a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f807bfff640 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bbbc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8068009bf0 tx=0x7f8068011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2393bed6e38f138 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f80815a8640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 0x7f806c0a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f8079ffb640 1 -- 172.21.15.116:0/4084843998 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f80700f1140 con 0x7f807c102a40 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f8066ffd640 1 -- 172.21.15.116:0/3504507066 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068004050 con 0x558a116b97d0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f8066ffd640 1 -- 172.21.15.116:0/3504507066 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80680054d0 con 0x558a116b97d0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.278+0000 7f8066ffd640 1 -- 172.21.15.116:0/3504507066 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068004d10 con 0x558a116b97d0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 -- 172.21.15.116:0/3504507066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 msgr2=0x558a116bbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bbbc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8068009bf0 tx=0x7f8068011810 comp rx=0 tx=0).stop 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f80815a8640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 0x7f806c0a9240 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8074009fd0 tx=0x7f8074009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 -- 172.21.15.116:0/3504507066 shutdown_connections 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bbbc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a116b9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 --2- 172.21.15.116:0/3504507066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a116b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 -- 172.21.15.116:0/3504507066 >> 172.21.15.116:0/3504507066 conn(0x558a116aaa00 msgr2=0x558a116aade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 -- 172.21.15.116:0/3504507066 shutdown_connections 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.279+0000 7f8085bf9740 1 -- 172.21.15.116:0/3504507066 wait complete. 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 -- start start 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 Processor -- start 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116b7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558a116bc660 con 0x558a116b02d0 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558a115cf310 con 0x558a116b97d0 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558a116bc7e0 con 0x558a116b0bf0 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8085bf9740 10 client.0 _ll_register_callbacks cb 0x558a115c32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8080da7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8080da7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56710/0 (socket says 172.21.15.116:56710) 2024-03-19T02:51:45.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f8080da7640 1 -- 172.21.15.116:0/919213271 learned_addr learned my addr 172.21.15.116:0/919213271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f807bfff640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.280+0000 7f80815a8640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 -- 172.21.15.116:0/919213271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 msgr2=0x558a116b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 -- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 msgr2=0x558a116b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 -- 172.21.15.116:0/919213271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a113273f0 con 0x558a116b02d0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f807bfff640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f80815a8640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8080da7640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f8070006010 tx=0x7f80700b9790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a0da47683516af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f805bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80700bdb70 con 0x558a116b02d0 2024-03-19T02:51:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f8070004fb0 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80680054d0 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f80700bd470 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8085bf9740 10 client.22317 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558a113f17d0 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.281+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80680054d0 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f80700ee6d0 con 0x558a116b02d0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 10 client.22317 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 10 client.22317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 20 client.22317 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a116c9a20 0x558a116cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 4 client.22317 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a116713e0 con 0x558a116c9a20 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f807bfff640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a116c9a20 0x558a116cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f805bfff640 10 client.22317 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.282+0000 7f807bfff640 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a116c9a20 0x558a116cbe90 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2738f9f0fe5f3e32 server_cookie=d1bd993a263a6065 in_seq=0 out_seq=0 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f805bfff640 1 -- 172.21.15.116:0/919213271 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a116713e0 con 0x558a116c9a20 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f805bfff640 10 client.22317 handle_command_reply: tid=0 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 client.22317 shutdown 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 msgr2=0x558a116b6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f8070006010 tx=0x7f80700b9790 comp rx=0 tx=0).stop 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 shutdown_connections 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a116c9a20 0x558a116cbe90 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 --2- 172.21.15.116:0/919213271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116b6700 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 >> 172.21.15.116:0/919213271 conn(0x558a116aaa00 msgr2=0x558a116ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f805bfff640 0 client.22317 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 shutdown_connections 2024-03-19T02:51:45.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.284+0000 7f8085bf9740 1 -- 172.21.15.116:0/919213271 wait complete. 2024-03-19T02:51:45.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.285+0000 7f8085bf9740 20 client.22317 trim_cache size 0 max 16384 2024-03-19T02:51:45.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 Processor -- start 2024-03-19T02:51:45.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 -- start start 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1aca10 con 0x558a116b02d0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1ae550 con 0x558a116b97d0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8085bf9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1ad0e0 con 0x558a116b0bf0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f8080da7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f807bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f80815a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.306+0000 7f807bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59188/0 (socket says 172.21.15.116:59188) 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 -- 172.21.15.116:0/523668558 learned_addr learned my addr 172.21.15.116:0/523668558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 -- 172.21.15.116:0/523668558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 msgr2=0x558a116d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 -- 172.21.15.116:0/523668558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 msgr2=0x558a116c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 -- 172.21.15.116:0/523668558 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a116b0a20 con 0x558a116b0bf0 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f8080da7640 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f807bfff640 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8068011d10 tx=0x7f8068009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859de96e6e1b5a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f80667fc640 1 -- 172.21.15.116:0/523668558 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680046b0 con 0x558a116b0bf0 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f80667fc640 1 -- 172.21.15.116:0/523668558 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f806802e4d0 con 0x558a116b0bf0 2024-03-19T02:51:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.307+0000 7f80667fc640 1 -- 172.21.15.116:0/523668558 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068003bc0 con 0x558a116b0bf0 2024-03-19T02:51:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 -- 172.21.15.116:0/523668558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 msgr2=0x558a116b6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8068011d10 tx=0x7f8068009680 comp rx=0 tx=0).stop 2024-03-19T02:51:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 -- 172.21.15.116:0/523668558 shutdown_connections 2024-03-19T02:51:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b0bf0 0x558a116b6fa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 --2- 172.21.15.116:0/523668558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b02d0 0x558a116c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 -- 172.21.15.116:0/523668558 >> 172.21.15.116:0/523668558 conn(0x558a116aaa00 msgr2=0x558a116a9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 -- 172.21.15.116:0/523668558 shutdown_connections 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.308+0000 7f8085bf9740 1 -- 172.21.15.116:0/523668558 wait complete. 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 -- start start 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 Processor -- start 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1ade80 con 0x558a116b0bf0 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1ad7b0 con 0x558a116b97d0 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f807c1aaa10 con 0x558a116b02d0 2024-03-19T02:51:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8085bf9740 10 client.0 _ll_register_callbacks cb 0x7f80700057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f807bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8080da7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f807bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56718/0 (socket says 172.21.15.116:56718) 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f807bfff640 1 -- 172.21.15.116:0/1482946006 learned_addr learned my addr 172.21.15.116:0/1482946006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f8080da7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59200/0 (socket says 172.21.15.116:59200) 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.309+0000 7f80815a8640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 -- 172.21.15.116:0/1482946006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 msgr2=0x558a117280d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 -- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 msgr2=0x558a116bcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 -- 172.21.15.116:0/1482946006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a116bd280 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f8080da7640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f807bfff640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f8068000c00 tx=0x7f806802e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5991d0e4820cb206 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f805bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806803a020 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f806802e4d0 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f806802eab0 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f806802ecb0 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f8085bf9740 10 client.22323 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558a1172f870 con 0x558a116b0bf0 2024-03-19T02:51:45.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.310+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f806802e4d0 con 0x558a116b0bf0 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f8068003820 con 0x558a116b0bf0 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 10 client.22323 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 10 client.22323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 20 client.22323 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a1172fe00 0x558a11732220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 4 client.22323 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a116209f0 con 0x558a1172fe00 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8080da7640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a1172fe00 0x558a11732220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f805bfff640 10 client.22323 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.311+0000 7f8080da7640 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a1172fe00 0x558a11732220 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc4ce4d687575bf7 server_cookie=7731659f3b49c11d in_seq=0 out_seq=0 2024-03-19T02:51:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.313+0000 7f805bfff640 1 -- 172.21.15.116:0/1482946006 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a116209f0 con 0x558a1172fe00 2024-03-19T02:51:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.313+0000 7f805bfff640 10 client.22323 handle_command_reply: tid=0 2024-03-19T02:51:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.313+0000 7f8085bf9740 1 client.22323 shutdown 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.313+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 msgr2=0x558a11728610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.313+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f8068000c00 tx=0x7f806802e680 comp rx=0 tx=0).stop 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 shutdown_connections 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558a1172fe00 0x558a11732220 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558a116b97d0 0x558a116bcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a116b0bf0 0x558a11728610 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f805bfff640 0 client.22323 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 --2- 172.21.15.116:0/1482946006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558a116b02d0 0x558a117280d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 >> 172.21.15.116:0/1482946006 conn(0x558a116aaa00 msgr2=0x558a116abbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 shutdown_connections 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 1 -- 172.21.15.116:0/1482946006 wait complete. 2024-03-19T02:51:45.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.314+0000 7f8085bf9740 20 client.22323 trim_cache size 0 max 16384 2024-03-19T02:51:45.319 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:51:45.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 msgr2=0x7f806c0a9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 0x7f806c0a9240 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8074009fd0 tx=0x7f8074009500 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 msgr2=0x7f807c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f8070005940 tx=0x7f807002c690 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 shutdown_connections 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f806c0a6d80 0x7f806c0a9240 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f807c10bd80 0x7f807c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f807c103360 0x7f807c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 --2- 172.21.15.116:0/4084843998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f807c102a40 0x7f807c197f70 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 >> 172.21.15.116:0/4084843998 conn(0x7f807c0778d0 msgr2=0x7f807c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 shutdown_connections 2024-03-19T02:51:45.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:45.315+0000 7f808300b640 1 -- 172.21.15.116:0/4084843998 wait complete. 2024-03-19T02:51:45.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71849 5.70 MB/sec execute 547 sec latency 3563.906 ms 2024-03-19T02:51:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:45 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1511624966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1511624966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:45.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:45 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1511624966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:46.328 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:51:46.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:46.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:51:46.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 71978 5.71 MB/sec execute 548 sec latency 4080.886 ms 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- 172.21.15.116:0/1528942941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 msgr2=0x7f459010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 --2- 172.21.15.116:0/1528942941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f459010d560 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f4588009930 tx=0x7f458802bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- 172.21.15.116:0/1528942941 shutdown_connections 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 --2- 172.21.15.116:0/1528942941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f459010d560 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 --2- 172.21.15.116:0/1528942941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4590100690 0x7f4590100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 --2- 172.21.15.116:0/1528942941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45901066a0 0x7f4590106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- 172.21.15.116:0/1528942941 >> 172.21.15.116:0/1528942941 conn(0x7f4590077820 msgr2=0x7f4590077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- 172.21.15.116:0/1528942941 shutdown_connections 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- 172.21.15.116:0/1528942941 wait complete. 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 Processor -- start 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.449+0000 7f459704b640 1 -- start start 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45901066a0 0x7f459019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f459010f850 con 0x7f4590101050 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f459010f6d0 con 0x7f4590100690 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f459704b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f459010f9d0 con 0x7f45901066a0 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f458ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f45955e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45901066a0 0x7f459019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57654/0 (socket says 172.21.15.116:57654) 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f458ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56750/0 (socket says 172.21.15.116:56750) 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 -- 172.21.15.116:0/516401392 learned_addr learned my addr 172.21.15.116:0/516401392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 -- 172.21.15.116:0/516401392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45901066a0 msgr2=0x7f459019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45901066a0 0x7f459019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 -- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 msgr2=0x7f45901962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 -- 172.21.15.116:0/516401392 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459019ad80 con 0x7f4590100690 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f458ffff640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f4594de7640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4580007c80 tx=0x7f458000f500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.450+0000 7f458dffb640 1 -- 172.21.15.116:0/516401392 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580019020 con 0x7f4590100690 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.451+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459019b070 con 0x7f4590100690 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.451+0000 7f458dffb640 1 -- 172.21.15.116:0/516401392 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4580007e30 con 0x7f4590100690 2024-03-19T02:51:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.451+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45901a2900 con 0x7f4590100690 2024-03-19T02:51:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.451+0000 7f458dffb640 1 -- 172.21.15.116:0/516401392 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580014790 con 0x7f4590100690 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.453+0000 7f458dffb640 1 -- 172.21.15.116:0/516401392 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f458001c020 con 0x7f4590100690 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.453+0000 7f4599c33740 1 Processor -- start 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.453+0000 7f4599c33740 1 -- start start 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8a380 0x5612cad8a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 0x5612cad95c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612cacee160 con 0x5612cad8aca0 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5612cac96e10 con 0x5612cad8a380 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4599c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5612cacea200 con 0x5612cad93880 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4594de7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458dffb640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 0x7f457c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4594de7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56760/0 (socket says 172.21.15.116:56760) 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4594de7640 1 -- 172.21.15.116:0/1517111811 learned_addr learned my addr 172.21.15.116:0/1517111811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f45955e8640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8a380 0x5612cad8a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 0x5612cad95c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f45955e8640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 0x7f457c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 -- 172.21.15.116:0/1517111811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8a380 msgr2=0x5612cad8a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8a380 0x5612cad8a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458dffb640 1 -- 172.21.15.116:0/516401392 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f45800cf810 con 0x7f4590100690 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 -- 172.21.15.116:0/1517111811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 msgr2=0x5612cad93340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 -- 172.21.15.116:0/1517111811 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612caaca7e0 con 0x5612cad93880 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f4594de7640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.454+0000 7f458ffff640 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 0x5612cad95c70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f457800fce0 tx=0x7f4578005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b40d681ee83db5e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f45955e8640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 0x7f457c0a4e30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4588005940 tx=0x7f458802bc30 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4586ffd640 1 -- 172.21.15.116:0/1517111811 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457801e030 con 0x5612cad93880 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4586ffd640 1 -- 172.21.15.116:0/1517111811 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f457800f4d0 con 0x5612cad93880 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4586ffd640 1 -- 172.21.15.116:0/1517111811 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457801a530 con 0x5612cad93880 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 -- 172.21.15.116:0/1517111811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 msgr2=0x5612cad95c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 0x5612cad95c70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f457800fce0 tx=0x7f4578005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 -- 172.21.15.116:0/1517111811 shutdown_connections 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad93880 0x5612cad95c70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad93340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 --2- 172.21.15.116:0/1517111811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8a380 0x5612cad8a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.455+0000 7f4599c33740 1 -- 172.21.15.116:0/1517111811 >> 172.21.15.116:0/1517111811 conn(0x5612cad84ab0 msgr2=0x5612cad84e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- 172.21.15.116:0/1517111811 shutdown_connections 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- 172.21.15.116:0/1517111811 wait complete. 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- start start 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 Processor -- start 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad91200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612cad96670 con 0x5612cad93880 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5612cacaa1a0 con 0x5612cad8aca0 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5612cad967f0 con 0x5612cad8a380 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4599c33740 10 client.0 _ll_register_callbacks cb 0x5612caa01c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:46.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.456+0000 7f4594de7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57684/0 (socket says 172.21.15.116:57684) 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f4594de7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59238/0 (socket says 172.21.15.116:59238) 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 -- 172.21.15.116:0/3054721461 learned_addr learned my addr 172.21.15.116:0/3054721461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f45955e8640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad91200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 -- 172.21.15.116:0/3054721461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 msgr2=0x5612cad90780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 -- 172.21.15.116:0/3054721461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 msgr2=0x5612cad91200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad91200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 -- 172.21.15.116:0/3054721461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612cab2af70 con 0x5612cad8aca0 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f45955e8640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f4594de7640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f458ffff640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f457800ef70 tx=0x7f457800efa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e635b46008350d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f456ffff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578005250 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457800f4d0 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.457+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f45780053f0 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4578036050 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 10 client.32039 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5612ca9c2170 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f457800f4d0 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f457801a5c0 con 0x5612cad8aca0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 10 client.32039 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 10 client.32039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 20 client.32039 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cada39f0 0x5612cada5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 4 client.32039 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612cacfb6a0 con 0x5612cada39f0 2024-03-19T02:51:46.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.458+0000 7f4594de7640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cada39f0 0x5612cada5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.459+0000 7f4594de7640 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cada39f0 0x5612cada5de0 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d13822a836895d server_cookie=3a06141e620e2b22 in_seq=0 out_seq=0 2024-03-19T02:51:46.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.459+0000 7f456ffff640 10 client.32039 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f456ffff640 1 -- 172.21.15.116:0/3054721461 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612cacfb6a0 con 0x5612cada39f0 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f456ffff640 10 client.32039 handle_command_reply: tid=0 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 client.32039 shutdown 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 msgr2=0x5612cad90cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f457800ef70 tx=0x7f457800efa0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 shutdown_connections 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cada39f0 0x5612cada5de0 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f456ffff640 0 client.32039 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad90cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 --2- 172.21.15.116:0/3054721461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 >> 172.21.15.116:0/3054721461 conn(0x5612cad84ab0 msgr2=0x5612cad86800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 shutdown_connections 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 1 -- 172.21.15.116:0/3054721461 wait complete. 2024-03-19T02:51:46.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.461+0000 7f4599c33740 20 client.32039 trim_cache size 0 max 16384 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.484+0000 7f4599c33740 1 Processor -- start 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 -- start start 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad9f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad93880 0x5612cad839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45901acae0 con 0x5612cad8aca0 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45901ae620 con 0x5612cad93880 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4599c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f45901ad1b0 con 0x5612cad8a380 2024-03-19T02:51:46.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f458ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad9f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59242/0 (socket says 172.21.15.116:59242) 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 -- 172.21.15.116:0/3051376266 learned_addr learned my addr 172.21.15.116:0/3051376266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f45955e8640 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad93880 0x5612cad839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 -- 172.21.15.116:0/3051376266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad93880 msgr2=0x5612cad839b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad93880 0x5612cad839b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 -- 172.21.15.116:0/3051376266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 msgr2=0x5612cad9f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad9f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f4594de7640 1 -- 172.21.15.116:0/3051376266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612cad848e0 con 0x5612cad8a380 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.485+0000 7f458ffff640 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad9f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f4594de7640 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f458009ba20 tx=0x7f45800d1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5317eec14453173 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f45867fc640 1 -- 172.21.15.116:0/3051376266 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458009fbe0 con 0x5612cad8a380 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f45867fc640 1 -- 172.21.15.116:0/3051376266 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4580099d90 con 0x5612cad8a380 2024-03-19T02:51:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f45867fc640 1 -- 172.21.15.116:0/3051376266 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800cbcc0 con 0x5612cad8a380 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f4599c33740 1 -- 172.21.15.116:0/3051376266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 msgr2=0x5612cad92eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.486+0000 7f4599c33740 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f458009ba20 tx=0x7f45800d1040 comp rx=0 tx=0).stop 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 -- 172.21.15.116:0/3051376266 shutdown_connections 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad93880 0x5612cad839b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad8aca0 0x5612cad9f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 --2- 172.21.15.116:0/3051376266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad92eb0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 -- 172.21.15.116:0/3051376266 >> 172.21.15.116:0/3051376266 conn(0x5612cad84ab0 msgr2=0x5612cad832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 -- 172.21.15.116:0/3051376266 shutdown_connections 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 -- 172.21.15.116:0/3051376266 wait complete. 2024-03-19T02:51:46.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 -- start start 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 Processor -- start 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad8d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.487+0000 7f4599c33740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad8dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45901adf50 con 0x5612cad93880 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45901ad880 con 0x5612cad8aca0 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f45901aaae0 con 0x5612cad8a380 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 10 client.0 _ll_register_callbacks cb 0x7f4580097600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f458ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad8dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f458ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57694/0 (socket says 172.21.15.116:57694) 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4594de7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad8d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f458ffff640 1 -- 172.21.15.116:0/590831987 learned_addr learned my addr 172.21.15.116:0/590831987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 -- 172.21.15.116:0/590831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 msgr2=0x5612cad8d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad8d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 -- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 msgr2=0x5612cad8da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 -- 172.21.15.116:0/590831987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612cae08e30 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f458ffff640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f45955e8640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad8dfb0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f458802c170 tx=0x7f4588009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c5c45048a15060 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f456ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588054070 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f458803a040 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.488+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4580099d90 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f458802cca0 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 10 client.22341 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5612cae0a150 con 0x5612cad93880 2024-03-19T02:51:46.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4580099d90 con 0x5612cad93880 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f4588040040 con 0x5612cad93880 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 10 client.22341 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 10 client.22341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 20 client.22341 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cae0a8d0 0x5612cae0ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 4 client.22341 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5612cacf59f0 con 0x5612cae0a8d0 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.489+0000 7f4594de7640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cae0a8d0 0x5612cae0ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.490+0000 7f4594de7640 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cae0a8d0 0x5612cae0ccf0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca9ced7adee5f6e0 server_cookie=63088ad37f840977 in_seq=0 out_seq=0 2024-03-19T02:51:46.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.490+0000 7f456ffff640 10 client.22341 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f456ffff640 1 -- 172.21.15.116:0/590831987 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5612cacf59f0 con 0x5612cae0a8d0 2024-03-19T02:51:46.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f456ffff640 10 client.22341 handle_command_reply: tid=0 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 client.22341 shutdown 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 msgr2=0x5612cad8dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad8dfb0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f458802c170 tx=0x7f4588009260 comp rx=0 tx=0).stop 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 shutdown_connections 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5612cae0a8d0 0x5612cae0ccf0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612cad93880 0x5612cad8dfb0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5612cad8aca0 0x5612cad8da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 --2- 172.21.15.116:0/590831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5612cad8a380 0x5612cad8d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 >> 172.21.15.116:0/590831987 conn(0x5612cad84ab0 msgr2=0x5612cad83b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.491+0000 7f456ffff640 0 client.22341 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 shutdown_connections 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f4599c33740 1 -- 172.21.15.116:0/590831987 wait complete. 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f4599c33740 20 client.22341 trim_cache size 0 max 16384 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "a554ed5e-fb6b-4ac2-b722-40248e9c0b20", 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:51:46.497 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 msgr2=0x7f457c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 0x7f457c0a4e30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4588005940 tx=0x7f458802bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 msgr2=0x7f4590195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4580007c80 tx=0x7f458000f500 comp rx=0 tx=0).stop 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 shutdown_connections 2024-03-19T02:51:46.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f457c0a2970 0x7f457c0a4e30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45901066a0 0x7f459019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4590101050 0x7f45901962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 --2- 172.21.15.116:0/516401392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4590100690 0x7f4590195d90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 >> 172.21.15.116:0/516401392 conn(0x7f4590077820 msgr2=0x7f4590102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 shutdown_connections 2024-03-19T02:51:46.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.492+0000 7f459704b640 1 -- 172.21.15.116:0/516401392 wait complete. 2024-03-19T02:51:46.502 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:46 smithi116 ceph-mon[32263]: pgmap v757: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-03-19T02:51:46.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:51:46.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:51:46.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:46 smithi154 ceph-mon[35693]: pgmap v757: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 -- 172.21.15.116:0/1314559439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 msgr2=0x7f585410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 --2- 172.21.15.116:0/1314559439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f585410e660 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f584c009930 tx=0x7f584c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 -- 172.21.15.116:0/1314559439 shutdown_connections 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 --2- 172.21.15.116:0/1314559439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f585410e660 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 --2- 172.21.15.116:0/1314559439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5854068a00 0x7f5854068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 --2- 172.21.15.116:0/1314559439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58541077a0 0x7f5854107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.661+0000 7f5858cde640 1 -- 172.21.15.116:0/1314559439 >> 172.21.15.116:0/1314559439 conn(0x7f5854077820 msgr2=0x7f5854077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 -- 172.21.15.116:0/1314559439 shutdown_connections 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 -- 172.21.15.116:0/1314559439 wait complete. 2024-03-19T02:51:46.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 Processor -- start 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 -- start start 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5854068a00 0x7f5854197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.662+0000 7f5858cde640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58541077a0 0x7f585419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5858cde640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5854111a60 con 0x7f58540693c0 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5858cde640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58541118e0 con 0x7f5854068a00 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5858cde640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5854111be0 con 0x7f58541077a0 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56802/0 (socket says 172.21.15.116:56802) 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 -- 172.21.15.116:0/1968136975 learned_addr learned my addr 172.21.15.116:0/1968136975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f585259c640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5854068a00 0x7f5854197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5852d9d640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58541077a0 0x7f585419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 -- 172.21.15.116:0/1968136975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58541077a0 msgr2=0x7f585419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58541077a0 0x7f585419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 -- 172.21.15.116:0/1968136975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5854068a00 msgr2=0x7f5854197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5854068a00 0x7f5854197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 -- 172.21.15.116:0/1968136975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585419cd70 con 0x7f58540693c0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5851d9b640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9f0 tx=0x7f584800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f58437fe640 1 -- 172.21.15.116:0/1968136975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584800cd80 con 0x7f58540693c0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f58437fe640 1 -- 172.21.15.116:0/1968136975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58480054d0 con 0x7f58540693c0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585419d060 con 0x7f58540693c0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f58437fe640 1 -- 172.21.15.116:0/1968136975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848010490 con 0x7f58540693c0 2024-03-19T02:51:46.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.663+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58541a48f0 con 0x7f58540693c0 2024-03-19T02:51:46.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f58437fe640 1 -- 172.21.15.116:0/1968136975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5848005020 con 0x7f58540693c0 2024-03-19T02:51:46.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 Processor -- start 2024-03-19T02:51:46.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 -- start start 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4e92e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e9bf30 0x5632c4e9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f58437fe640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 0x7f583c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632c4df6500 con 0x5632c4e93350 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5632c4d9ebe0 con 0x5632c4e9bf30 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.666+0000 7f585b99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632c4df25a0 con 0x5632c4e92a30 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5852d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4e92e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56806/0 (socket says 172.21.15.116:56806) 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 -- 172.21.15.116:0/3839868074 learned_addr learned my addr 172.21.15.116:0/3839868074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f58437fe640 1 -- 172.21.15.116:0/1968136975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f58480cf5b0 con 0x7f58540693c0 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5851d9b640 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e9bf30 0x5632c4e9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5852d9d640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 0x7f583c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 -- 172.21.15.116:0/3839868074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 msgr2=0x5632c4e92e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4e92e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 -- 172.21.15.116:0/3839868074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e9bf30 msgr2=0x5632c4e9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e9bf30 0x5632c4e9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 -- 172.21.15.116:0/3839868074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c4dc0000 con 0x5632c4e93350 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f585259c640 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f5844009fd0 tx=0x7f58440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687adaae916c355b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5852d9d640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 0x7f583c0a4e30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f584c005940 tx=0x7f584c0058d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5840ff9640 1 -- 172.21.15.116:0/3839868074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584401e070 con 0x5632c4e93350 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5840ff9640 1 -- 172.21.15.116:0/3839868074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5844003c30 con 0x5632c4e93350 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.667+0000 7f5840ff9640 1 -- 172.21.15.116:0/3839868074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400f940 con 0x5632c4e93350 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 -- 172.21.15.116:0/3839868074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 msgr2=0x5632c4e9b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f5844009fd0 tx=0x7f58440118d0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 -- 172.21.15.116:0/3839868074 shutdown_connections 2024-03-19T02:51:46.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e9bf30 0x5632c4e9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e9b9f0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 --2- 172.21.15.116:0/3839868074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4e92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 -- 172.21.15.116:0/3839868074 >> 172.21.15.116:0/3839868074 conn(0x5632c4e8d160 msgr2=0x5632c4e8d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 -- 172.21.15.116:0/3839868074 shutdown_connections 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.668+0000 7f585b99b740 1 -- 172.21.15.116:0/3839868074 wait complete. 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 -- start start 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 Processor -- start 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4e98fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e9bf30 0x5632c4e99a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632c4e9ed20 con 0x5632c4e93350 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5632c4db2160 con 0x5632c4e92a30 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632c4e9eea0 con 0x5632c4e9bf30 2024-03-19T02:51:46.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585b99b740 10 client.0 _ll_register_callbacks cb 0x5632c4deea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56820/0 (socket says 172.21.15.116:56820) 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 -- 172.21.15.116:0/2520990821 learned_addr learned my addr 172.21.15.116:0/2520990821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5852d9d640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e9bf30 0x5632c4e99a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f585259c640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4e98fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 -- 172.21.15.116:0/2520990821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e9bf30 msgr2=0x5632c4e99a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e9bf30 0x5632c4e99a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 -- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 msgr2=0x5632c4e98fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4e98fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 -- 172.21.15.116:0/2520990821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c4bd47d0 con 0x5632c4e93350 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5851d9b640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f584809b530 tx=0x7f584809b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13144dbfe7a80eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.669+0000 7f5839ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584809bf30 con 0x5632c4e93350 2024-03-19T02:51:46.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f58480ccba0 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5844003c30 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f584809b830 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 10 client.22353 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5632c4b0a3f0 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5844003c30 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f58480cf180 con 0x5632c4e93350 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 10 client.22353 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 10 client.22353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 20 client.22353 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4eac2c0 0x5632c4eae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 4 client.22353 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.670+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632c4e03a40 con 0x5632c4eac2c0 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.671+0000 7f585259c640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4eac2c0 0x5632c4eae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.671+0000 7f5839ffb640 10 client.22353 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.671+0000 7f585259c640 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4eac2c0 0x5632c4eae6f0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420cd9c92b2a1973 server_cookie=f953fb718bf59531 in_seq=0 out_seq=0 2024-03-19T02:51:46.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:46 smithi162 ceph-mon[35761]: pgmap v757: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 207 op/s 2024-03-19T02:51:46.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f5839ffb640 1 -- 172.21.15.116:0/2520990821 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632c4e03a40 con 0x5632c4eac2c0 2024-03-19T02:51:46.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f5839ffb640 10 client.22353 handle_command_reply: tid=0 2024-03-19T02:51:46.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 client.22353 shutdown 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 msgr2=0x5632c4e99500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f584809b530 tx=0x7f584809b250 comp rx=0 tx=0).stop 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 shutdown_connections 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4eac2c0 0x5632c4eae6f0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e9bf30 0x5632c4e99a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e99500 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 --2- 172.21.15.116:0/2520990821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4e98fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 >> 172.21.15.116:0/2520990821 conn(0x5632c4e8d160 msgr2=0x5632c4e8eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f5839ffb640 0 client.22353 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 shutdown_connections 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 1 -- 172.21.15.116:0/2520990821 wait complete. 2024-03-19T02:51:46.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.679+0000 7f585b99b740 20 client.22353 trim_cache size 0 max 16384 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.698+0000 7f585b99b740 1 Processor -- start 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.698+0000 7f585b99b740 1 -- start start 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.698+0000 7f585b99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.698+0000 7f585b99b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4ea7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585b99b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e8ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585b99b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58541ae9e0 con 0x5632c4e93350 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585b99b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58541b0520 con 0x5632c4e92a30 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585b99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58541af0b0 con 0x5632c4e8d160 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e8ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59300/0 (socket says 172.21.15.116:59300) 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585259c640 1 -- 172.21.15.116:0/3141212775 learned_addr learned my addr 172.21.15.116:0/3141212775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5851d9b640 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4ea7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 -- 172.21.15.116:0/3141212775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 msgr2=0x5632c4e9b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 -- 172.21.15.116:0/3141212775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 msgr2=0x5632c4ea7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4ea7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 -- 172.21.15.116:0/3141212775 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c4aca0f0 con 0x5632c4e93350 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f585259c640 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f5852d9d640 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e8ecc0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f584c009850 tx=0x7f584c005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c5907dba0eb2fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f583bfff640 1 -- 172.21.15.116:0/3141212775 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c005250 con 0x5632c4e93350 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f583bfff640 1 -- 172.21.15.116:0/3141212775 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f584c048040 con 0x5632c4e93350 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.699+0000 7f583bfff640 1 -- 172.21.15.116:0/3141212775 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c003850 con 0x5632c4e93350 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- 172.21.15.116:0/3141212775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 msgr2=0x5632c4e8ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e8ecc0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f584c009850 tx=0x7f584c005df0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- 172.21.15.116:0/3141212775 shutdown_connections 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4e8ecc0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e92a30 0x5632c4ea7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 --2- 172.21.15.116:0/3141212775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e8d160 0x5632c4e9b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- 172.21.15.116:0/3141212775 >> 172.21.15.116:0/3141212775 conn(0x5632c4e97c00 msgr2=0x5632c4e8e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- 172.21.15.116:0/3141212775 shutdown_connections 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- 172.21.15.116:0/3141212775 wait complete. 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.700+0000 7f585b99b740 1 -- start start 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 Processor -- start 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 0x5632c4ea0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4ea06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58541afe50 con 0x5632c4e93350 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58541af780 con 0x5632c4e8d160 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f58541ac9e0 con 0x5632c4e92a30 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 10 client.0 _ll_register_callbacks cb 0x7f5848097ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56844/0 (socket says 172.21.15.116:56844) 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 -- 172.21.15.116:0/866289500 learned_addr learned my addr 172.21.15.116:0/866289500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585259c640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 0x5632c4ea0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5851d9b640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4ea06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 -- 172.21.15.116:0/866289500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 msgr2=0x5632c4ea06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4ea06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 -- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 msgr2=0x5632c4ea0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 0x5632c4ea0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 -- 172.21.15.116:0/866289500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c4f11b80 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585259c640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 0x5632c4ea0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5852d9d640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f584c02c170 tx=0x7f584c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688436a9de327e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5839ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c05c070 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9510+0+0 (secure 0 0 0) 0x7f584c0035f0 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.701+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f584c048040 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f584c068050 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 10 client.22359 fetch_fsmap learned FSMap version 42 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5632c4f12e90 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f584c048040 con 0x5632c4e93350 2024-03-19T02:51:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f584c04d040 con 0x5632c4e93350 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 10 client.22359 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 10 client.22359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 20 client.22359 populate_metadata read hostname 'smithi116' 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4e9bf30 0x5632c4f155f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 4 client.22359 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632c4dfdd90 con 0x5632c4e9bf30 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585259c640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4e9bf30 0x5632c4f155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f5839ffb640 10 client.22359 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.702+0000 7f585259c640 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4e9bf30 0x5632c4f155f0 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=add60cdf487894f6 server_cookie=2399df3987b59af0 in_seq=0 out_seq=0 2024-03-19T02:51:46.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f5839ffb640 1 -- 172.21.15.116:0/866289500 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5632c4dfdd90 con 0x5632c4e9bf30 2024-03-19T02:51:46.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f5839ffb640 10 client.22359 handle_command_reply: tid=0 2024-03-19T02:51:46.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 client.22359 shutdown 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 msgr2=0x5632c4f115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f584c02c170 tx=0x7f584c004970 comp rx=0 tx=0).stop 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 shutdown_connections 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5632c4e9bf30 0x5632c4f155f0 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632c4e93350 0x5632c4f115f0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632c4e92a30 0x5632c4ea06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 --2- 172.21.15.116:0/866289500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5632c4e8d160 0x5632c4ea0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f5839ffb640 0 client.22359 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 >> 172.21.15.116:0/866289500 conn(0x5632c4e97c00 msgr2=0x5632c4e98f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 shutdown_connections 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 1 -- 172.21.15.116:0/866289500 wait complete. 2024-03-19T02:51:46.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.705+0000 7f585b99b740 20 client.22359 trim_cache size 0 max 16384 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (544 inodes in the stack)", 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "a554ed5e-fb6b-4ac2-b722-40248e9c0b20": { 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "a554ed5e-fb6b-4ac2-b722-40248e9c0b20", 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:51:46.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:51:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 msgr2=0x7f583c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 0x7f583c0a4e30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f584c005940 tx=0x7f584c0058d0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 msgr2=0x7f58541982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9f0 tx=0x7f584800edf0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 shutdown_connections 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f583c0a2970 0x7f583c0a4e30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58541077a0 0x7f585419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58540693c0 0x7f58541982c0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 --2- 172.21.15.116:0/1968136975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5854068a00 0x7f5854197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 >> 172.21.15.116:0/1968136975 conn(0x7f5854077820 msgr2=0x7f58540fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 shutdown_connections 2024-03-19T02:51:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:51:46.706+0000 7f5858cde640 1 -- 172.21.15.116:0/1968136975 wait complete. 2024-03-19T02:51:46.718 INFO:tasks.cephfs.filesystem:scrub status for tag:a554ed5e-fb6b-4ac2-b722-40248e9c0b20 - {'path': '/', 'tag': 'a554ed5e-fb6b-4ac2-b722-40248e9c0b20', 'options': 'recursive,force'} 2024-03-19T02:51:47.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72095 5.71 MB/sec execute 549 sec latency 250.958 ms 2024-03-19T02:51:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:51:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:47.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:47 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:47 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:47.669 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:47 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72142 5.70 MB/sec execute 550 sec latency 126.257 ms 2024-03-19T02:51:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:48 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:51:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:48 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:48 smithi154 ceph-mon[35693]: pgmap v758: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:48 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:48 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:51:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:48 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:48 smithi116 ceph-mon[32263]: pgmap v758: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:48 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:48 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:51:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:48 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:48 smithi162 ceph-mon[35761]: pgmap v758: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:48.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:48 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:51:49.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72216 5.70 MB/sec execute 551 sec latency 145.966 ms 2024-03-19T02:51:50.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72296 5.70 MB/sec execute 552 sec latency 148.549 ms 2024-03-19T02:51:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:50 smithi154 ceph-mon[35693]: pgmap v759: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:50.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:50 smithi116 ceph-mon[32263]: pgmap v759: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:50.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:50 smithi162 ceph-mon[35761]: pgmap v759: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 129 op/s 2024-03-19T02:51:51.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72363 5.70 MB/sec execute 553 sec latency 134.671 ms 2024-03-19T02:51:52.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72454 5.70 MB/sec execute 554 sec latency 137.042 ms 2024-03-19T02:51:52.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:52 smithi154 ceph-mon[35693]: pgmap v760: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-03-19T02:51:52.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:52 smithi116 ceph-mon[32263]: pgmap v760: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-03-19T02:51:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:52 smithi162 ceph-mon[35761]: pgmap v760: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 231 op/s 2024-03-19T02:51:53.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72575 5.71 MB/sec execute 555 sec latency 153.389 ms 2024-03-19T02:51:54.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72728 5.72 MB/sec execute 556 sec latency 147.521 ms 2024-03-19T02:51:54.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:54 smithi154 ceph-mon[35693]: pgmap v761: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 156 op/s 2024-03-19T02:51:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:54 smithi116 ceph-mon[32263]: pgmap v761: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 156 op/s 2024-03-19T02:51:54.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:54 smithi162 ceph-mon[35761]: pgmap v761: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 156 op/s 2024-03-19T02:51:55.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72828 5.72 MB/sec execute 557 sec latency 178.798 ms 2024-03-19T02:51:56.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72898 5.72 MB/sec execute 558 sec latency 163.798 ms 2024-03-19T02:51:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:56 smithi154 ceph-mon[35693]: pgmap v762: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2024-03-19T02:51:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:56 smithi116 ceph-mon[32263]: pgmap v762: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2024-03-19T02:51:56.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:56 smithi162 ceph-mon[35761]: pgmap v762: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.0 MiB/s wr, 217 op/s 2024-03-19T02:51:57.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 72986 5.72 MB/sec execute 559 sec latency 412.009 ms 2024-03-19T02:51:58.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73053 5.71 MB/sec execute 560 sec latency 1412.037 ms 2024-03-19T02:51:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:51:58 smithi154 ceph-mon[35693]: pgmap v763: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:51:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:51:58 smithi116 ceph-mon[32263]: pgmap v763: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:51:58.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:51:58 smithi162 ceph-mon[35761]: pgmap v763: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:51:59.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73112 5.70 MB/sec execute 561 sec latency 2412.062 ms 2024-03-19T02:52:00.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73170 5.69 MB/sec execute 562 sec latency 3412.094 ms 2024-03-19T02:52:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:00 smithi154 ceph-mon[35693]: pgmap v764: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:52:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:00.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:00.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:00.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:00 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:00 smithi116 ceph-mon[32263]: pgmap v764: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:52:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:00.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:00 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:00.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:00 smithi162 ceph-mon[35761]: pgmap v764: 129 pgs: 129 active+clean; 1.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 162 op/s 2024-03-19T02:52:00.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:00.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:00.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:00.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:00 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:01.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73198 5.68 MB/sec execute 563 sec latency 3921.759 ms 2024-03-19T02:52:01.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:01 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:01 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:01.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:01 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:02.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73265 5.68 MB/sec execute 564 sec latency 961.878 ms 2024-03-19T02:52:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:02 smithi154 ceph-mon[35693]: pgmap v765: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 211 op/s 2024-03-19T02:52:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:02 smithi116 ceph-mon[32263]: pgmap v765: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 211 op/s 2024-03-19T02:52:02.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:02 smithi162 ceph-mon[35761]: pgmap v765: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 211 op/s 2024-03-19T02:52:03.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73346 5.67 MB/sec execute 565 sec latency 1961.904 ms 2024-03-19T02:52:04.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73432 5.67 MB/sec execute 566 sec latency 2961.938 ms 2024-03-19T02:52:04.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:04 smithi154 ceph-mon[35693]: pgmap v766: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 108 op/s 2024-03-19T02:52:04.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:04 smithi116 ceph-mon[32263]: pgmap v766: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 108 op/s 2024-03-19T02:52:04.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:04 smithi162 ceph-mon[35761]: pgmap v766: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 108 op/s 2024-03-19T02:52:05.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73514 5.66 MB/sec execute 567 sec latency 3961.978 ms 2024-03-19T02:52:06.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73602 5.66 MB/sec execute 568 sec latency 4471.126 ms 2024-03-19T02:52:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:06 smithi154 ceph-mon[35693]: pgmap v767: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 191 op/s 2024-03-19T02:52:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:06 smithi154 ceph-mon[35693]: osdmap e132: 12 total, 12 up, 12 in 2024-03-19T02:52:06.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:06 smithi116 ceph-mon[32263]: pgmap v767: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 191 op/s 2024-03-19T02:52:06.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:06 smithi116 ceph-mon[32263]: osdmap e132: 12 total, 12 up, 12 in 2024-03-19T02:52:06.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:06 smithi162 ceph-mon[35761]: pgmap v767: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 191 op/s 2024-03-19T02:52:06.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:06 smithi162 ceph-mon[35761]: osdmap e132: 12 total, 12 up, 12 in 2024-03-19T02:52:07.340 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73699 5.66 MB/sec execute 569 sec latency 223.862 ms 2024-03-19T02:52:08.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73798 5.66 MB/sec execute 570 sec latency 147.439 ms 2024-03-19T02:52:08.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:08 smithi154 ceph-mon[35693]: pgmap v769: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:08 smithi116 ceph-mon[32263]: pgmap v769: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:08.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:08 smithi162 ceph-mon[35761]: pgmap v769: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:09.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 73902 5.66 MB/sec execute 571 sec latency 146.859 ms 2024-03-19T02:52:10.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74002 5.65 MB/sec execute 572 sec latency 190.929 ms 2024-03-19T02:52:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:10 smithi154 ceph-mon[35693]: pgmap v770: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:10.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:10 smithi116 ceph-mon[32263]: pgmap v770: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:10.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:10 smithi162 ceph-mon[35761]: pgmap v770: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 157 op/s 2024-03-19T02:52:11.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74132 5.67 MB/sec execute 573 sec latency 146.808 ms 2024-03-19T02:52:12.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74244 5.67 MB/sec execute 574 sec latency 152.649 ms 2024-03-19T02:52:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:12 smithi154 ceph-mon[35693]: pgmap v771: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 192 op/s 2024-03-19T02:52:13.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:12 smithi154 ceph-mon[35693]: osdmap e133: 12 total, 12 up, 12 in 2024-03-19T02:52:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:12 smithi116 ceph-mon[32263]: pgmap v771: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 192 op/s 2024-03-19T02:52:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:12 smithi116 ceph-mon[32263]: osdmap e133: 12 total, 12 up, 12 in 2024-03-19T02:52:13.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74351 5.66 MB/sec execute 575 sec latency 142.405 ms 2024-03-19T02:52:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:12 smithi162 ceph-mon[35761]: pgmap v771: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 192 op/s 2024-03-19T02:52:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:12 smithi162 ceph-mon[35761]: osdmap e133: 12 total, 12 up, 12 in 2024-03-19T02:52:14.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74464 5.66 MB/sec execute 576 sec latency 139.264 ms 2024-03-19T02:52:15.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:14 smithi154 ceph-mon[35693]: pgmap v773: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 116 op/s 2024-03-19T02:52:15.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:14 smithi116 ceph-mon[32263]: pgmap v773: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 116 op/s 2024-03-19T02:52:15.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74592 5.66 MB/sec execute 577 sec latency 156.667 ms 2024-03-19T02:52:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:14 smithi162 ceph-mon[35761]: pgmap v773: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 116 op/s 2024-03-19T02:52:16.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74694 5.65 MB/sec execute 578 sec latency 132.317 ms 2024-03-19T02:52:16.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:52:16.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 -- 172.21.15.116:0/3933826538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d0101080 msgr2=0x7f31d010d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 --2- 172.21.15.116:0/3933826538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d0101080 0x7f31d010d540 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009ca0 tx=0x7f31c802c180 comp rx=0 tx=0).stop 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 -- 172.21.15.116:0/3933826538 shutdown_connections 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 --2- 172.21.15.116:0/3933826538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d0101080 0x7f31d010d540 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 --2- 172.21.15.116:0/3933826538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d01006e0 0x7f31d0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 --2- 172.21.15.116:0/3933826538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 0x7f31d01059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 -- 172.21.15.116:0/3933826538 >> 172.21.15.116:0/3933826538 conn(0x7f31d00777b0 msgr2=0x7f31d0077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 -- 172.21.15.116:0/3933826538 shutdown_connections 2024-03-19T02:52:16.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.850+0000 7f31d7eff640 1 -- 172.21.15.116:0/3933826538 wait complete. 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 Processor -- start 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 -- start start 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 0x7f31d01992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 0x7f31d01a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0110960 con 0x7f31d01006e0 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01107e0 con 0x7f31d0101080 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d7eff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31d0110ae0 con 0x7f31d01055c0 2024-03-19T02:52:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d5c9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d5c9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40496/0 (socket says 172.21.15.116:40496) 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d549a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 0x7f31d01992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d5c9b640 1 -- 172.21.15.116:0/1537020357 learned_addr learned my addr 172.21.15.116:0/1537020357 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.851+0000 7f31d649c640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 0x7f31d01a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 -- 172.21.15.116:0/1537020357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 msgr2=0x7f31d01a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 0x7f31d01a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 -- 172.21.15.116:0/1537020357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 msgr2=0x7f31d01992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 0x7f31d01992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 -- 172.21.15.116:0/1537020357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d01a4340 con 0x7f31d01006e0 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d549a640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 0x7f31d01992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d5c9b640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e9f0 tx=0x7f31cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31c6ffd640 1 -- 172.21.15.116:0/1537020357 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00cd80 con 0x7f31d01006e0 2024-03-19T02:52:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d01a4510 con 0x7f31d01006e0 2024-03-19T02:52:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31c6ffd640 1 -- 172.21.15.116:0/1537020357 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31cc0054d0 con 0x7f31d01006e0 2024-03-19T02:52:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31c6ffd640 1 -- 172.21.15.116:0/1537020357 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc010490 con 0x7f31d01006e0 2024-03-19T02:52:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.852+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d01a4a10 con 0x7f31d01006e0 2024-03-19T02:52:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31c6ffd640 1 -- 172.21.15.116:0/1537020357 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f31cc005020 con 0x7f31d01006e0 2024-03-19T02:52:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 Processor -- start 2024-03-19T02:52:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 -- start start 2024-03-19T02:52:16.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dabe19500 con 0x558dabec3340 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558dabdc1be0 con 0x558dabeb56e0 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.855+0000 7f31dabc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558dabe155a0 con 0x558dabeb6000 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d5c9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59534/0 (socket says 172.21.15.116:59534) 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 -- 172.21.15.116:0/2216496114 learned_addr learned my addr 172.21.15.116:0/2216496114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d549a640 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 -- 172.21.15.116:0/2216496114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 msgr2=0x558dabec2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 -- 172.21.15.116:0/2216496114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 msgr2=0x558dabec5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 -- 172.21.15.116:0/2216496114 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dabde3000 con 0x558dabeb56e0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d549a640 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31c6ffd640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 0x7f31bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d5c9b640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 0x7f31bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.856+0000 7f31d649c640 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009ca0 tx=0x7f31c8009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=927f840d19713492 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.857+0000 7f31c6ffd640 1 -- 172.21.15.116:0/1537020357 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f31cc0cf500 con 0x7f31d01006e0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.857+0000 7f31bbfff640 1 -- 172.21.15.116:0/2216496114 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8043070 con 0x558dabeb56e0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.857+0000 7f31bbfff640 1 -- 172.21.15.116:0/2216496114 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31c8003e20 con 0x558dabeb56e0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.857+0000 7f31bbfff640 1 -- 172.21.15.116:0/2216496114 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8039040 con 0x558dabeb56e0 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- 172.21.15.116:0/2216496114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 msgr2=0x558dabeb5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009ca0 tx=0x7f31c8009900 comp rx=0 tx=0).stop 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- 172.21.15.116:0/2216496114 shutdown_connections 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 --2- 172.21.15.116:0/2216496114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabeb5ac0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- 172.21.15.116:0/2216496114 >> 172.21.15.116:0/2216496114 conn(0x558dabeb0160 msgr2=0x558dabeb0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31d5c9b640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 0x7f31bc0a4e30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0cc920 tx=0x7f31cc086040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- 172.21.15.116:0/2216496114 shutdown_connections 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- 172.21.15.116:0/2216496114 wait complete. 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 -- start start 2024-03-19T02:52:16.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.858+0000 7f31dabc7740 1 Processor -- start 2024-03-19T02:52:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 0x558dabebca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabec3340 0x558dabebcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dabeb6b70 con 0x558dabeb56e0 2024-03-19T02:52:16.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558dabdd5160 con 0x558dabeb6000 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558dabeb6cf0 con 0x558dabec3340 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31dabc7740 10 client.0 _ll_register_callbacks cb 0x558dabe11a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40504/0 (socket says 172.21.15.116:40504) 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31d649c640 1 -- 172.21.15.116:0/1781073909 learned_addr learned my addr 172.21.15.116:0/1781073909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31d549a640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 0x558dabebca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.859+0000 7f31d649c640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabec3340 0x558dabebcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 -- 172.21.15.116:0/1781073909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabec3340 msgr2=0x558dabebcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabec3340 0x558dabebcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 -- 172.21.15.116:0/1781073909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 msgr2=0x558dabebc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 -- 172.21.15.116:0/1781073909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dabbf77d0 con 0x558dabeb6000 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d649c640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:16.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31d549a640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 0x558dabebca20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f31d006a6b0 tx=0x7f31c000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=390a08fd1f810f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c00120d0 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f31c0004030 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31c8003e20 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f31c0012270 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.860+0000 7f31dabc7740 10 client.32075 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558dabb2d3f0 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31c8003e20 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f31c001c460 con 0x558dabeb6000 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 10 client.32075 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 10 client.32075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 20 client.32075 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabecf320 0x558dabed1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 4 client.32075 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:16.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558dabe26a40 con 0x558dabecf320 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.861+0000 7f31d649c640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabecf320 0x558dabed1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.862+0000 7f31b97fa640 10 client.32075 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.862+0000 7f31d649c640 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabecf320 0x558dabed1710 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cacc18b30fa61d40 server_cookie=ed21840bb245d332 in_seq=0 out_seq=0 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31b97fa640 1 -- 172.21.15.116:0/1781073909 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558dabe26a40 con 0x558dabecf320 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31b97fa640 10 client.32075 handle_command_reply: tid=0 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 client.32075 shutdown 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 msgr2=0x558dabebca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 0x558dabebca20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f31d006a6b0 tx=0x7f31c000f790 comp rx=0 tx=0).stop 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 shutdown_connections 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabecf320 0x558dabed1710 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabec3340 0x558dabebcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb6000 0x558dabebca20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 --2- 172.21.15.116:0/1781073909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabebc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 >> 172.21.15.116:0/1781073909 conn(0x558dabeb0160 msgr2=0x558dabeb1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31b97fa640 0 client.32075 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 shutdown_connections 2024-03-19T02:52:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 1 -- 172.21.15.116:0/1781073909 wait complete. 2024-03-19T02:52:16.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.864+0000 7f31dabc7740 20 client.32075 trim_cache size 0 max 16384 2024-03-19T02:52:16.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 Processor -- start 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 -- start start 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabeaf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabec3340 0x558dabed6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01ac980 con 0x558dabeb56e0 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01ae4c0 con 0x558dabec3340 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31dabc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01ad050 con 0x558dabeb6000 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40506/0 (socket says 172.21.15.116:40506) 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31d649c640 1 -- 172.21.15.116:0/659404334 learned_addr learned my addr 172.21.15.116:0/659404334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.888+0000 7f31d549a640 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabeaf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabec3340 0x558dabed6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 -- 172.21.15.116:0/659404334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 msgr2=0x558dabeaf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabeaf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 -- 172.21.15.116:0/659404334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabec3340 msgr2=0x558dabed6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabec3340 0x558dabed6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 -- 172.21.15.116:0/659404334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dabeb5e30 con 0x558dabeb56e0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31d649c640 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f31c8004af0 tx=0x7f31c8004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd378153593c900a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31bb7fe640 1 -- 172.21.15.116:0/659404334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8039040 con 0x558dabeb56e0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31bb7fe640 1 -- 172.21.15.116:0/659404334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31c8056800 con 0x558dabeb56e0 2024-03-19T02:52:16.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.889+0000 7f31bb7fe640 1 -- 172.21.15.116:0/659404334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8043040 con 0x558dabeb56e0 2024-03-19T02:52:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- 172.21.15.116:0/659404334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 msgr2=0x558dabec2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f31c8004af0 tx=0x7f31c8004560 comp rx=0 tx=0).stop 2024-03-19T02:52:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- 172.21.15.116:0/659404334 shutdown_connections 2024-03-19T02:52:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabec3340 0x558dabed6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabeaf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 --2- 172.21.15.116:0/659404334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabeb56e0 0x558dabec2b00 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- 172.21.15.116:0/659404334 >> 172.21.15.116:0/659404334 conn(0x558dabeb0160 msgr2=0x558dabeae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- 172.21.15.116:0/659404334 shutdown_connections 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- 172.21.15.116:0/659404334 wait complete. 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 -- start start 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.890+0000 7f31dabc7740 1 Processor -- start 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabecbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01addf0 con 0x558dabec3340 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01ad720 con 0x558dabeb56e0 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31d01aa980 con 0x558dabeb6000 2024-03-19T02:52:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31dabc7740 10 client.0 _ll_register_callbacks cb 0x7f31c0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d649c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40512/0 (socket says 172.21.15.116:40512) 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d649c640 1 -- 172.21.15.116:0/3683448502 learned_addr learned my addr 172.21.15.116:0/3683448502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d649c640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabecbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 -- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 msgr2=0x558dabecbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabecbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 -- 172.21.15.116:0/3683448502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 msgr2=0x558dabec74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 -- 172.21.15.116:0/3683448502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dabaed0f0 con 0x558dabeb6000 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d649c640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31d549a640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec6f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f31c001ca70 tx=0x7f31c0012140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b499b03dd99f725d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.891+0000 7f31b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:52:16.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c001cd00 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558dabe20d90 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f31c0005040 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f31c0005240 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 10 client.32071 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558dabf354b0 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x558dabe20d90 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f31c0031030 con 0x558dabeb6000 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 10 client.32071 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 10 client.32071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 20 client.32071 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabf35c50 0x558dabf38070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 4 client.32071 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558dabd9e540 con 0x558dabf35c50 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.892+0000 7f31d649c640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabf35c50 0x558dabf38070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.893+0000 7f31b97fa640 10 client.32071 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.893+0000 7f31d649c640 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabf35c50 0x558dabf38070 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6c46e095946d7b server_cookie=c178c792bb0cfeca in_seq=0 out_seq=0 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31b97fa640 1 -- 172.21.15.116:0/3683448502 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558dabd9e540 con 0x558dabf35c50 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31b97fa640 10 client.32071 handle_command_reply: tid=0 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 client.32071 shutdown 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 msgr2=0x558dabec6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec6f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f31c001ca70 tx=0x7f31c0012140 comp rx=0 tx=0).stop 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 shutdown_connections 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x558dabf35c50 0x558dabf38070 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dabec3340 0x558dabec74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558dabeb6000 0x558dabec6f60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31b97fa640 0 client.32071 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 --2- 172.21.15.116:0/3683448502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558dabeb56e0 0x558dabecbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 >> 172.21.15.116:0/3683448502 conn(0x558dabeb0160 msgr2=0x558dabeaf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 shutdown_connections 2024-03-19T02:52:16.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 1 -- 172.21.15.116:0/3683448502 wait complete. 2024-03-19T02:52:16.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31dabc7740 20 client.32071 trim_cache size 0 max 16384 2024-03-19T02:52:16.902 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:16.902 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:52:16.902 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:52:16.902 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 msgr2=0x7f31bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 0x7f31bc0a4e30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0cc920 tx=0x7f31cc086040 comp rx=0 tx=0).stop 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 msgr2=0x7f31d0198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e9f0 tx=0x7f31cc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 shutdown_connections 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f31bc0a2970 0x7f31bc0a4e30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31d01055c0 0x7f31d01a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d0101080 0x7f31d01992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 --2- 172.21.15.116:0/1537020357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31d01006e0 0x7f31d0198d80 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 >> 172.21.15.116:0/1537020357 conn(0x7f31d00777b0 msgr2=0x7f31d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.895+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 shutdown_connections 2024-03-19T02:52:16.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:16.896+0000 7f31d7eff640 1 -- 172.21.15.116:0/1537020357 wait complete. 2024-03-19T02:52:16.907 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:52:16.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:52:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:16 smithi116 ceph-mon[32263]: pgmap v774: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 195 op/s 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 -- 172.21.15.116:0/2900712533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 msgr2=0x7f5bec106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2900712533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec106b10 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009930 tx=0x7f5bdc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 -- 172.21.15.116:0/2900712533 shutdown_connections 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2900712533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2900712533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2900712533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec106b10 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 -- 172.21.15.116:0/2900712533 >> 172.21.15.116:0/2900712533 conn(0x7f5bec077820 msgr2=0x7f5bec077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 -- 172.21.15.116:0/2900712533 shutdown_connections 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.046+0000 7f5bf297e640 1 -- 172.21.15.116:0/2900712533 wait complete. 2024-03-19T02:52:17.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 Processor -- start 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 -- start start 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec1038b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec1109f0 con 0x7f5bec106730 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec110870 con 0x7f5bec1010e0 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bf297e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec110b70 con 0x7f5bec100720 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5beb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5bebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42516/0 (socket says 172.21.15.116:42516) 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 -- 172.21.15.116:0/2153820446 learned_addr learned my addr 172.21.15.116:0/2153820446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.047+0000 7f5beb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:59576/0 (socket says 172.21.15.116:59576) 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bf0f1b640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec1038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 -- 172.21.15.116:0/2153820446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 msgr2=0x7f5bec103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 -- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 msgr2=0x7f5bec1038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec1038b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 -- 172.21.15.116:0/2153820446 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec1a6500 con 0x7f5bec100720 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5beb7fe640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bebfff640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009230 tx=0x7f5bdc02c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc043020 con 0x7f5bec100720 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bec1a6790 con 0x7f5bec100720 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bdc02ca70 con 0x7f5bec100720 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003a50 con 0x7f5bec100720 2024-03-19T02:52:17.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.048+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bec1a6d30 con 0x7f5bec100720 2024-03-19T02:52:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.050+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bec068610 con 0x7f5bec100720 2024-03-19T02:52:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.051+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5bdc046020 con 0x7f5bec100720 2024-03-19T02:52:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.052+0000 7f5be97fa640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 0x7f5bd40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.052+0000 7f5beb7fe640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 0x7f5bd40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.052+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f5bdc0f1170 con 0x7f5bec100720 2024-03-19T02:52:17.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.053+0000 7f5beb7fe640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 0x7f5bd40a4e30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5bec1048c0 tx=0x7f5be0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.058+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bdc04d020 con 0x7f5bec100720 2024-03-19T02:52:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:16 smithi154 ceph-mon[35693]: pgmap v774: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 195 op/s 2024-03-19T02:52:17.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74803 5.65 MB/sec execute 579 sec latency 322.157 ms 2024-03-19T02:52:17.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.341+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bec109ca0 con 0x7f5bec100720 2024-03-19T02:52:17.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.342+0000 7f5be97fa640 1 -- 172.21.15.116:0/2153820446 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11950 (secure 0 0 0) 0x7f5bdc0bdba0 con 0x7f5bec100720 2024-03-19T02:52:17.343 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:52:17.344 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:50:15.839334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24494,24451,24484,24470,24463]},"id":1}]} 2024-03-19T02:52:17.344 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 msgr2=0x7f5bd40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 0x7f5bd40a4e30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5bec1048c0 tx=0x7f5be0005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 msgr2=0x7f5bec102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009230 tx=0x7f5bdc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 shutdown_connections 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5bd40a2970 0x7f5bd40a4e30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bec106730 0x7f5bec1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bec1010e0 0x7f5bec103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 --2- 172.21.15.116:0/2153820446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bec100720 0x7f5bec102e30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 >> 172.21.15.116:0/2153820446 conn(0x7f5bec077820 msgr2=0x7f5bec068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.344+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 shutdown_connections 2024-03-19T02:52:17.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.345+0000 7f5bf297e640 1 -- 172.21.15.116:0/2153820446 wait complete. 2024-03-19T02:52:17.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:52:17.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:16 smithi162 ceph-mon[35761]: pgmap v774: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 195 op/s 2024-03-19T02:52:17.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 -- 172.21.15.116:0/1400627715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20401005c0 msgr2=0x7f20401009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 --2- 172.21.15.116:0/1400627715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20401005c0 0x7f20401009a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2030009900 tx=0x7f203002bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 -- 172.21.15.116:0/1400627715 shutdown_connections 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 --2- 172.21.15.116:0/1400627715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204010e170 0x7f2040110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 --2- 172.21.15.116:0/1400627715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2040100ee0 0x7f204010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 --2- 172.21.15.116:0/1400627715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20401005c0 0x7f20401009a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 -- 172.21.15.116:0/1400627715 >> 172.21.15.116:0/1400627715 conn(0x7f20400778d0 msgr2=0x7f2040077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 -- 172.21.15.116:0/1400627715 shutdown_connections 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.498+0000 7f204567e640 1 -- 172.21.15.116:0/1400627715 wait complete. 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.499+0000 7f204567e640 1 Processor -- start 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.499+0000 7f204567e640 1 -- start start 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.499+0000 7f204567e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.499+0000 7f204567e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2040100ee0 0x7f2040102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.499+0000 7f204567e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 0x7f2040103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f204567e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2040113c20 con 0x7f204010e170 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f204567e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2040113aa0 con 0x7f2040100ee0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f204567e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2040113da0 con 0x7f20401005c0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42538/0 (socket says 172.21.15.116:42538) 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 -- 172.21.15.116:0/2340416940 learned_addr learned my addr 172.21.15.116:0/2340416940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f20367fc640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2040100ee0 0x7f2040102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203f7fe640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 0x7f2040103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 -- 172.21.15.116:0/2340416940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2040100ee0 msgr2=0x7f2040102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2040100ee0 0x7f2040102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 -- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 msgr2=0x7f2040103320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 0x7f2040103320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 -- 172.21.15.116:0/2340416940 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20401aa8b0 con 0x7f20401005c0 2024-03-19T02:52:17.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203f7fe640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 0x7f2040103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.500+0000 7f203effd640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2030005940 tx=0x7f203002c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.501+0000 7f203cff9640 1 -- 172.21.15.116:0/2340416940 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030043020 con 0x7f20401005c0 2024-03-19T02:52:17.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.501+0000 7f203cff9640 1 -- 172.21.15.116:0/2340416940 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20300042d0 con 0x7f20401005c0 2024-03-19T02:52:17.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.501+0000 7f203cff9640 1 -- 172.21.15.116:0/2340416940 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030003ab0 con 0x7f20401005c0 2024-03-19T02:52:17.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.501+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20401aab40 con 0x7f20401005c0 2024-03-19T02:52:17.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.501+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20401ab040 con 0x7f20401005c0 2024-03-19T02:52:17.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.503+0000 7f203cff9640 1 -- 172.21.15.116:0/2340416940 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2030046020 con 0x7f20401005c0 2024-03-19T02:52:17.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f203cff9640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 0x7f202c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 Processor -- start 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f203cff9640 1 -- 172.21.15.116:0/2340416940 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f20300f1170 con 0x7f20401005c0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 -- start start 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f20367fc640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 0x7f202c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc495290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc497bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dcc3ef4b0 con 0x557dcc4957d0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557dcc397d70 con 0x557dcc48cbf0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f2048263740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557dcc3eb550 con 0x557dcc48c2d0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.504+0000 7f203f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc497bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42542/0 (socket says 172.21.15.116:42542) 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 -- 172.21.15.116:0/1345890321 learned_addr learned my addr 172.21.15.116:0/1345890321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f20367fc640 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc495290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f20367fc640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 0x7f202c0a4eb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2040104060 tx=0x7f2028005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 -- 172.21.15.116:0/1345890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 msgr2=0x557dcc495290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc495290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 -- 172.21.15.116:0/1345890321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 msgr2=0x557dcc497bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc497bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 -- 172.21.15.116:0/1345890321 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dcc3ba210 con 0x557dcc48c2d0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203f7fe640 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc497bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f203effd640 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f20300c5830 tx=0x7f20300bd230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac635804aac47cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f20357fa640 1 -- 172.21.15.116:0/1345890321 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20300d6070 con 0x557dcc48c2d0 2024-03-19T02:52:17.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f20357fa640 1 -- 172.21.15.116:0/1345890321 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20300bd5c0 con 0x557dcc48c2d0 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.505+0000 7f20357fa640 1 -- 172.21.15.116:0/1345890321 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20300053b0 con 0x557dcc48c2d0 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 -- 172.21.15.116:0/1345890321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 msgr2=0x557dcc48c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f20300c5830 tx=0x7f20300bd230 comp rx=0 tx=0).stop 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 -- 172.21.15.116:0/1345890321 shutdown_connections 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc497bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc495290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 --2- 172.21.15.116:0/1345890321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48c6b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 -- 172.21.15.116:0/1345890321 >> 172.21.15.116:0/1345890321 conn(0x557dcc486a00 msgr2=0x557dcc486de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 -- 172.21.15.116:0/1345890321 shutdown_connections 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.506+0000 7f2048263740 1 -- 172.21.15.116:0/1345890321 wait complete. 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 -- start start 2024-03-19T02:52:17.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 Processor -- start 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48c2d0 0x557dcc492860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dcc498380 con 0x557dcc48cbf0 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557dcc3ab310 con 0x557dcc48c2d0 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557dcc498500 con 0x557dcc4957d0 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f20367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f20367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40544/0 (socket says 172.21.15.116:40544) 2024-03-19T02:52:17.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f20367fc640 1 -- 172.21.15.116:0/1546631291 learned_addr learned my addr 172.21.15.116:0/1546631291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f2048263740 10 client.0 _ll_register_callbacks cb 0x557dcc39f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.507+0000 7f203f7fe640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f203effd640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48c2d0 0x557dcc492860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 -- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 msgr2=0x557dcc4932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4932e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 -- 172.21.15.116:0/1546631291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48c2d0 msgr2=0x557dcc492860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48c2d0 0x557dcc492860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 -- 172.21.15.116:0/1546631291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dcc1033f0 con 0x557dcc48cbf0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20367fc640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f202801cae0 tx=0x7f2028004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed91d46e4bdfb944 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20267fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202801c320 con 0x557dcc48cbf0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f202800f980 con 0x557dcc48cbf0 2024-03-19T02:52:17.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20300bd5c0 con 0x557dcc48cbf0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2028030050 con 0x557dcc48cbf0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f2048263740 10 client.22395 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.508+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x557dcc1cd7d0 con 0x557dcc48cbf0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20300bd5c0 con 0x557dcc48cbf0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f2028004930 con 0x557dcc48cbf0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 10 client.22395 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 10 client.22395 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 20 client.22395 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc4a59b0 0x557dcc4a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 4 client.22395 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557dcc44d3e0 con 0x557dcc4a59b0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.509+0000 7f203effd640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc4a59b0 0x557dcc4a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.510+0000 7f20267fc640 10 client.22395 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.510+0000 7f203effd640 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc4a59b0 0x557dcc4a7e20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fb0b60bd63d8819 server_cookie=d4e20268b093d835 in_seq=0 out_seq=0 2024-03-19T02:52:17.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f20267fc640 1 -- 172.21.15.116:0/1546631291 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557dcc44d3e0 con 0x557dcc4a59b0 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f20267fc640 10 client.22395 handle_command_reply: tid=0 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 client.22395 shutdown 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 msgr2=0x557dcc492da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f202801cae0 tx=0x7f2028004730 comp rx=0 tx=0).stop 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 shutdown_connections 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc4a59b0 0x557dcc4a7e20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f20267fc640 0 client.22395 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48cbf0 0x557dcc492da0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 --2- 172.21.15.116:0/1546631291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48c2d0 0x557dcc492860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 >> 172.21.15.116:0/1546631291 conn(0x557dcc486a00 msgr2=0x557dcc488750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 shutdown_connections 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 1 -- 172.21.15.116:0/1546631291 wait complete. 2024-03-19T02:52:17.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.511+0000 7f2048263740 20 client.22395 trim_cache size 0 max 16384 2024-03-19T02:52:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 Processor -- start 2024-03-19T02:52:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 -- start start 2024-03-19T02:52:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc4a1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.532+0000 7f2048263740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b52b0 con 0x557dcc48c2d0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f2048263740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b6df0 con 0x557dcc48cbf0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f2048263740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b5980 con 0x557dcc4957d0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40548/0 (socket says 172.21.15.116:40548) 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 -- 172.21.15.116:0/2571332230 learned_addr learned my addr 172.21.15.116:0/2571332230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f20367fc640 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc4a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203f7fe640 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 -- 172.21.15.116:0/2571332230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 msgr2=0x557dcc4ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 -- 172.21.15.116:0/2571332230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 msgr2=0x557dcc4a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc4a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 -- 172.21.15.116:0/2571332230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dcc48d690 con 0x557dcc48c2d0 2024-03-19T02:52:17.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f203effd640 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f20300bd550 tx=0x7f20300095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b555d1c5554467e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f2034ff9640 1 -- 172.21.15.116:0/2571332230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20300d6040 con 0x557dcc48c2d0 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f2034ff9640 1 -- 172.21.15.116:0/2571332230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20300bd5c0 con 0x557dcc48c2d0 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.533+0000 7f2034ff9640 1 -- 172.21.15.116:0/2571332230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20300ee890 con 0x557dcc48c2d0 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- 172.21.15.116:0/2571332230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 msgr2=0x557dcc494f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f20300bd550 tx=0x7f20300095e0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- 172.21.15.116:0/2571332230 shutdown_connections 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc4957d0 0x557dcc4ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc4a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 --2- 172.21.15.116:0/2571332230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc48c2d0 0x557dcc494f90 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- 172.21.15.116:0/2571332230 >> 172.21.15.116:0/2571332230 conn(0x557dcc486a00 msgr2=0x557dcc486de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- 172.21.15.116:0/2571332230 shutdown_connections 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- 172.21.15.116:0/2571332230 wait complete. 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.534+0000 7f2048263740 1 -- start start 2024-03-19T02:52:17.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 Processor -- start 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc48ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b6720 con 0x557dcc4957d0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b6050 con 0x557dcc48cbf0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20401b32b0 con 0x557dcc48c2d0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f2048263740 10 client.0 _ll_register_callbacks cb 0x7f202801c9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40564/0 (socket says 172.21.15.116:40564) 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 -- 172.21.15.116:0/3012485755 learned_addr learned my addr 172.21.15.116:0/3012485755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f20367fc640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc48ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 -- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 msgr2=0x557dcc48fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 -- 172.21.15.116:0/3012485755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 msgr2=0x557dcc48ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc48ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203f7fe640 1 -- 172.21.15.116:0/3012485755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dcc50b2a0 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.535+0000 7f203effd640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f203f7fe640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f203800ffc0 tx=0x7f20380128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eff298ba8051854 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038022020 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f2038004d20 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20300bd5c0 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f203802a050 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f2048263740 10 client.22401 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x557dcc50c4f0 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20300bd5c0 con 0x557dcc4957d0 2024-03-19T02:52:17.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f20380104c0 con 0x557dcc4957d0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f2048263740 10 client.22401 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.536+0000 7f2048263740 10 client.22401 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 20 client.22401 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc50cca0 0x557dcc50f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 4 client.22401 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557dcc3fc9f0 con 0x557dcc50cca0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f203effd640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc50cca0 0x557dcc50f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f20267fc640 10 client.22401 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f203effd640 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc50cca0 0x557dcc50f0c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7244e46748ccb9e0 server_cookie=aab2f3a46a6a5756 in_seq=0 out_seq=0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f20267fc640 1 -- 172.21.15.116:0/3012485755 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557dcc3fc9f0 con 0x557dcc50cca0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f20267fc640 10 client.22401 handle_command_reply: tid=0 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 1 client.22401 shutdown 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 msgr2=0x557dcc50ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.537+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f203800ffc0 tx=0x7f20380128e0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 shutdown_connections 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x557dcc50cca0 0x557dcc50f0c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dcc4957d0 0x557dcc50ad60 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f20267fc640 0 client.22401 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557dcc48cbf0 0x557dcc48ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 --2- 172.21.15.116:0/3012485755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557dcc48c2d0 0x557dcc48fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 >> 172.21.15.116:0/3012485755 conn(0x557dcc486a00 msgr2=0x557dcc485310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 shutdown_connections 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 1 -- 172.21.15.116:0/3012485755 wait complete. 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f2048263740 20 client.22401 trim_cache size 0 max 16384 2024-03-19T02:52:17.545 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 msgr2=0x7f202c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 0x7f202c0a4eb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2040104060 tx=0x7f2028005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 msgr2=0x7f20401028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2030005940 tx=0x7f203002c5f0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 shutdown_connections 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f202c0a29f0 0x7f202c0a4eb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f204010e170 0x7f2040103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2040100ee0 0x7f2040102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 --2- 172.21.15.116:0/2340416940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20401005c0 0x7f20401028a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 >> 172.21.15.116:0/2340416940 conn(0x7f20400778d0 msgr2=0x7f204010f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 shutdown_connections 2024-03-19T02:52:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.538+0000 7f204567e640 1 -- 172.21.15.116:0/2340416940 wait complete. 2024-03-19T02:52:17.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:52:17.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 -- 172.21.15.116:0/1935046666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508102a60 msgr2=0x7f1508102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 --2- 172.21.15.116:0/1935046666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508102a60 0x7f1508102e40 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f1504009900 tx=0x7f150402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 -- 172.21.15.116:0/1935046666 shutdown_connections 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 --2- 172.21.15.116:0/1935046666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f150810bda0 0x7f150810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 --2- 172.21.15.116:0/1935046666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508103380 0x7f150810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 --2- 172.21.15.116:0/1935046666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508102a60 0x7f1508102e40 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 -- 172.21.15.116:0/1935046666 >> 172.21.15.116:0/1935046666 conn(0x7f15080778d0 msgr2=0x7f1508077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 -- 172.21.15.116:0/1935046666 shutdown_connections 2024-03-19T02:52:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.661+0000 7f15103b2640 1 -- 172.21.15.116:0/1935046666 wait complete. 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 Processor -- start 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 -- start start 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 0x7f150819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 0x7f150819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1508111a30 con 0x7f150810bda0 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15081118b0 con 0x7f1508103380 2024-03-19T02:52:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f15103b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1508111bb0 con 0x7f1508102a60 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150e94f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150d94d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 0x7f150819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150e14e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 0x7f150819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150d94d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 0x7f150819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37870/0 (socket says 172.21.15.116:37870) 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150e14e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 0x7f150819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54614/0 (socket says 172.21.15.116:54614) 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150e94f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44788/0 (socket says 172.21.15.116:44788) 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.662+0000 7f150d94d640 1 -- 172.21.15.116:0/30374452 learned_addr learned my addr 172.21.15.116:0/30374452 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 -- 172.21.15.116:0/30374452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 msgr2=0x7f150819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 0x7f150819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 -- 172.21.15.116:0/30374452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 msgr2=0x7f150819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 0x7f150819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 -- 172.21.15.116:0/30374452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150819f130 con 0x7f150810bda0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f150e94f640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f1500009fd0 tx=0x7f150000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f14ff7fe640 1 -- 172.21.15.116:0/30374452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150000ce20 con 0x7f150810bda0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f14ff7fe640 1 -- 172.21.15.116:0/30374452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1500005020 con 0x7f150810bda0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150819f420 con 0x7f150810bda0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f14ff7fe640 1 -- 172.21.15.116:0/30374452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500010690 con 0x7f150810bda0 2024-03-19T02:52:17.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.663+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15081a6cb0 con 0x7f150810bda0 2024-03-19T02:52:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.665+0000 7f14ff7fe640 1 -- 172.21.15.116:0/30374452 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f15000051d0 con 0x7f150810bda0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 Processor -- start 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 -- start start 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f14ff7fe640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 0x7f14f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559655dc24b0 con 0x559655e5fbf0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559655d6ad70 con 0x559655e5f2d0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f1512f9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559655dbe550 con 0x559655e687d0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150d94d640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 0x7f14f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150e14e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150d94d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f14ff7fe640 1 -- 172.21.15.116:0/30374452 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f15000cf880 con 0x7f150810bda0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150d94d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44794/0 (socket says 172.21.15.116:44794) 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150e14e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37874/0 (socket says 172.21.15.116:37874) 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150d94d640 1 -- 172.21.15.116:0/2631648007 learned_addr learned my addr 172.21.15.116:0/2631648007 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.666+0000 7f150e94f640 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 -- 172.21.15.116:0/2631648007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 msgr2=0x559655e6abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 -- 172.21.15.116:0/2631648007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 msgr2=0x559655e68290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 -- 172.21.15.116:0/2631648007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559655d8d210 con 0x559655e5f2d0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150d94d640 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150d94d640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 0x7f14f40a4e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f14f8011a70 tx=0x7f14f8009ca0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f150e14e640 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1504000c00 tx=0x7f15040095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2c0bb4dabfa64 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f14fcbfe640 1 -- 172.21.15.116:0/2631648007 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504004040 con 0x559655e5f2d0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f14fcbfe640 1 -- 172.21.15.116:0/2631648007 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15040054f0 con 0x559655e5f2d0 2024-03-19T02:52:17.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.667+0000 7f14fcbfe640 1 -- 172.21.15.116:0/2631648007 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150402cd80 con 0x559655e5f2d0 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 -- 172.21.15.116:0/2631648007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 msgr2=0x559655e5f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1504000c00 tx=0x7f15040095f0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 -- 172.21.15.116:0/2631648007 shutdown_connections 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655e68290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 --2- 172.21.15.116:0/2631648007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655e5f6b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 -- 172.21.15.116:0/2631648007 >> 172.21.15.116:0/2631648007 conn(0x559655e59a00 msgr2=0x559655e59de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 -- 172.21.15.116:0/2631648007 shutdown_connections 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.668+0000 7f1512f9f740 1 -- 172.21.15.116:0/2631648007 wait complete. 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 -- start start 2024-03-19T02:52:17.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 Processor -- start 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e65d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e66290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559655e6b560 con 0x559655e5f2d0 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559655d7e310 con 0x559655e687d0 2024-03-19T02:52:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559655e6b6e0 con 0x559655e5fbf0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f1512f9f740 10 client.0 _ll_register_callbacks cb 0x559655d722d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44808/0 (socket says 172.21.15.116:44808) 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150d94d640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e65d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 -- 172.21.15.116:0/3697841030 learned_addr learned my addr 172.21.15.116:0/3697841030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e94f640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e66290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 -- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 msgr2=0x559655e65d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e65d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 -- 172.21.15.116:0/3697841030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 msgr2=0x559655e66290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e66290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e14e640 1 -- 172.21.15.116:0/3697841030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559655ad63f0 con 0x559655e5f2d0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150d94d640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e65d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.669+0000 7f150e94f640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e66290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f150e14e640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f150402ca60 tx=0x7f15040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970b2a66dfef5d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15040044e0 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15040054f0 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f1504003a00 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1504003c00 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 10 client.22413 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559655ba07d0 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15040054f0 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f1504034cc0 con 0x559655e5f2d0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 10 client.22413 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 10 client.22413 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 20 client.22413 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655e78980 0x559655e7ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 4 client.22413 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.670+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559655e203e0 con 0x559655e78980 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.671+0000 7f150d94d640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655e78980 0x559655e7ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.671+0000 7f14f1ffb640 10 client.22413 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.671+0000 7f150d94d640 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655e78980 0x559655e7ad70 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa68b48cd6aa908 server_cookie=f469b7993f866bb7 in_seq=0 out_seq=0 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f14f1ffb640 1 -- 172.21.15.116:0/3697841030 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559655e203e0 con 0x559655e78980 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f14f1ffb640 10 client.22413 handle_command_reply: tid=0 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 client.22413 shutdown 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 msgr2=0x559655e65810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f150402ca60 tx=0x7f15040042f0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 shutdown_connections 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e66290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e65d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655e78980 0x559655e7ad70 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 --2- 172.21.15.116:0/3697841030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e65810 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f14f1ffb640 0 client.22413 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 >> 172.21.15.116:0/3697841030 conn(0x559655e59a00 msgr2=0x559655e5b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 shutdown_connections 2024-03-19T02:52:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 1 -- 172.21.15.116:0/3697841030 wait complete. 2024-03-19T02:52:17.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.672+0000 7f1512f9f740 20 client.22413 trim_cache size 0 max 16384 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 Processor -- start 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 -- start start 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e5b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e5b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15081b0d00 con 0x559655e5f2d0 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15081b2840 con 0x559655e687d0 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f1512f9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f15081b13d0 con 0x559655e5fbf0 2024-03-19T02:52:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f150e14e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f150e94f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e5b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f150e14e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44824/0 (socket says 172.21.15.116:44824) 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f150e14e640 1 -- 172.21.15.116:0/4030273288 learned_addr learned my addr 172.21.15.116:0/4030273288 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.687+0000 7f150d94d640 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e5b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 -- 172.21.15.116:0/4030273288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 msgr2=0x559655e5b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e5b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 -- 172.21.15.116:0/4030273288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 msgr2=0x559655e803d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 -- 172.21.15.116:0/4030273288 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559655e6abc0 con 0x559655e687d0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e14e640 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f150e94f640 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e5b880 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f15000ccbf0 tx=0x7f150004a7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae09ae7543150c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f14f3fff640 1 -- 172.21.15.116:0/4030273288 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150009bb00 con 0x559655e687d0 2024-03-19T02:52:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f14f3fff640 1 -- 172.21.15.116:0/4030273288 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150009fc50 con 0x559655e687d0 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.688+0000 7f14f3fff640 1 -- 172.21.15.116:0/4030273288 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000982d0 con 0x559655e687d0 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- 172.21.15.116:0/4030273288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 msgr2=0x559655e5b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e5b880 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f15000ccbf0 tx=0x7f150004a7c0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- 172.21.15.116:0/4030273288 shutdown_connections 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e687d0 0x559655e5b880 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e5fbf0 0x559655e5b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 --2- 172.21.15.116:0/4030273288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5f2d0 0x559655e803d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- 172.21.15.116:0/4030273288 >> 172.21.15.116:0/4030273288 conn(0x559655e59a00 msgr2=0x559655e5ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- 172.21.15.116:0/4030273288 shutdown_connections 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- 172.21.15.116:0/4030273288 wait complete. 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 -- start start 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 Processor -- start 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.689+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655ed6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15081b2170 con 0x559655e5fbf0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15081b1aa0 con 0x559655e5f2d0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f15081aed00 con 0x559655e687d0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 10 client.0 _ll_register_callbacks cb 0x7f14f80115c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150e14e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655ed6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44828/0 (socket says 172.21.15.116:44828) 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150e14e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655ed6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37904/0 (socket says 172.21.15.116:37904) 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 -- 172.21.15.116:0/688786597 learned_addr learned my addr 172.21.15.116:0/688786597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150e94f640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 -- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 msgr2=0x559655e6db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 -- 172.21.15.116:0/688786597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 msgr2=0x559655ed6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655ed6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 -- 172.21.15.116:0/688786597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559655e6e240 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f150d94d640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009fd0 tx=0x7f14f8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60affcc8013b153c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f14f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f8029a30 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f14f8029bd0 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.690+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f150009fc50 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f14f803a050 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 10 client.22419 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559655ede640 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150009fc50 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f14f80051b0 con 0x559655e5fbf0 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 10 client.22419 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 10 client.22419 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 20 client.22419 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655ededa0 0x559655ee11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 4 client.22419 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559655dcf9f0 con 0x559655ededa0 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.691+0000 7f150e14e640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655ededa0 0x559655ee11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f14f1ffb640 10 client.22419 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f150e14e640 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655ededa0 0x559655ee11c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85679ed3163c3660 server_cookie=90527cabeda66d4d in_seq=0 out_seq=0 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f14f1ffb640 1 -- 172.21.15.116:0/688786597 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559655dcf9f0 con 0x559655ededa0 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f14f1ffb640 10 client.22419 handle_command_reply: tid=0 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 client.22419 shutdown 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 msgr2=0x559655ed7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009fd0 tx=0x7f14f8005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 shutdown_connections 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559655e687d0 0x559655e6db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x559655ededa0 0x559655ee11c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559655e5fbf0 0x559655ed7350 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 --2- 172.21.15.116:0/688786597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559655e5f2d0 0x559655ed6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f14f1ffb640 0 client.22419 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 >> 172.21.15.116:0/688786597 conn(0x559655e59a00 msgr2=0x559655e5b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 shutdown_connections 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 1 -- 172.21.15.116:0/688786597 wait complete. 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.692+0000 7f1512f9f740 20 client.22419 trim_cache size 0 max 16384 2024-03-19T02:52:17.700 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 msgr2=0x7f14f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 0x7f14f40a4e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f14f8011a70 tx=0x7f14f8009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 msgr2=0x7f150819ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f1500009fd0 tx=0x7f150000ef10 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 shutdown_connections 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f14f40a2970 0x7f14f40a4e30 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f150810bda0 0x7f150819ea00 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1508103380 0x7f150819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 --2- 172.21.15.116:0/30374452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1508102a60 0x7f150819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 >> 172.21.15.116:0/30374452 conn(0x7f15080778d0 msgr2=0x7f1508107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 shutdown_connections 2024-03-19T02:52:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.693+0000 7f15103b2640 1 -- 172.21.15.116:0/30374452 wait complete. 2024-03-19T02:52:17.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:52:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 -- 172.21.15.116:0/3222521911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 msgr2=0x7fefc0107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 --2- 172.21.15.116:0/3222521911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc0107b80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009930 tx=0x7fefb002bc20 comp rx=0 tx=0).stop 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 -- 172.21.15.116:0/3222521911 shutdown_connections 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 --2- 172.21.15.116:0/3222521911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 --2- 172.21.15.116:0/3222521911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 0x7fefc0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 --2- 172.21.15.116:0/3222521911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc0107b80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.827+0000 7fefc6259640 1 -- 172.21.15.116:0/3222521911 >> 172.21.15.116:0/3222521911 conn(0x7fefc00fd2c0 msgr2=0x7fefc00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- 172.21.15.116:0/3222521911 shutdown_connections 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- 172.21.15.116:0/3222521911 wait complete. 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 Processor -- start 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- start start 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 0x7fefc0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111a60 con 0x7fefc0102150 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01118e0 con 0x7fefc0101790 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.828+0000 7fefc6259640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111be0 con 0x7fefc01077a0 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44844/0 (socket says 172.21.15.116:44844) 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefb7fff640 1 -- 172.21.15.116:0/3333908108 learned_addr learned my addr 172.21.15.116:0/3333908108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 0x7fefc0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 -- 172.21.15.116:0/3333908108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 msgr2=0x7fefc0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 0x7fefc0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 -- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 msgr2=0x7fefc0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 -- 172.21.15.116:0/3333908108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc019d030 con 0x7fefc01077a0 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefb7fff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbffff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc019c900 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fefb800ec50 tx=0x7fefb800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbd7fa640 1 -- 172.21.15.116:0/3333908108 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8003b70 con 0x7fefc01077a0 2024-03-19T02:52:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbd7fa640 1 -- 172.21.15.116:0/3333908108 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefb8003d10 con 0x7fefc01077a0 2024-03-19T02:52:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc019d320 con 0x7fefc01077a0 2024-03-19T02:52:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefbd7fa640 1 -- 172.21.15.116:0/3333908108 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8010760 con 0x7fefc01077a0 2024-03-19T02:52:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.829+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc01a4bb0 con 0x7fefc01077a0 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.832+0000 7fefbd7fa640 1 -- 172.21.15.116:0/3333908108 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fefb8004020 con 0x7fefc01077a0 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.832+0000 7fefc8e40740 1 Processor -- start 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 -- start start 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459ef36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459efc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a459e564b0 con 0x55a459efc7d0 2024-03-19T02:52:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a459dfed70 con 0x55a459ef32d0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefc8e40740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a459e52550 con 0x55a459ef3bf0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44848/0 (socket says 172.21.15.116:44848) 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefb7fff640 1 -- 172.21.15.116:0/326316243 learned_addr learned my addr 172.21.15.116:0/326316243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefb7fff640 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459ef36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459efc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 -- 172.21.15.116:0/326316243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 msgr2=0x55a459ef36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459ef36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbd7fa640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 0x7fefac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 -- 172.21.15.116:0/326316243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 msgr2=0x55a459efebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.833+0000 7fefbf7fe640 1 -- 172.21.15.116:0/326316243 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a459e21210 con 0x55a459ef3bf0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefbffff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 0x7fefac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459efc290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009230 tx=0x7fefb0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f8d328abfc7720 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefbd7fa640 1 -- 172.21.15.116:0/3333908108 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fefb80cf6e0 con 0x7fefc01077a0 2024-03-19T02:52:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefb5ffb640 1 -- 172.21.15.116:0/326316243 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0045020 con 0x55a459ef3bf0 2024-03-19T02:52:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefb5ffb640 1 -- 172.21.15.116:0/326316243 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefb002c7d0 con 0x55a459ef3bf0 2024-03-19T02:52:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.834+0000 7fefb5ffb640 1 -- 172.21.15.116:0/326316243 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0005360 con 0x55a459ef3bf0 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- 172.21.15.116:0/326316243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 msgr2=0x55a459efc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefbffff640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 0x7fefac0a4eb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fefb8097dc0 tx=0x7fefb801d000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459efc290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009230 tx=0x7fefb0009560 comp rx=0 tx=0).stop 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- 172.21.15.116:0/326316243 shutdown_connections 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459efc290 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 --2- 172.21.15.116:0/326316243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459ef36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- 172.21.15.116:0/326316243 >> 172.21.15.116:0/326316243 conn(0x55a459eeda00 msgr2=0x55a459eedde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- 172.21.15.116:0/326316243 shutdown_connections 2024-03-19T02:52:17.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- 172.21.15.116:0/326316243 wait complete. 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.835+0000 7fefc8e40740 1 -- start start 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 Processor -- start 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef3bf0 0x55a459ef9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a459eff560 con 0x55a459efc7d0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a459e12310 con 0x55a459ef3bf0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a459eff6e0 con 0x55a459ef32d0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefc8e40740 10 client.0 _ll_register_callbacks cb 0x55a459e062d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54688/0 (socket says 172.21.15.116:54688) 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 -- 172.21.15.116:0/2376445284 learned_addr learned my addr 172.21.15.116:0/2376445284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef3bf0 0x55a459ef9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 -- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef3bf0 msgr2=0x55a459ef9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef3bf0 0x55a459ef9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 -- 172.21.15.116:0/2376445284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 msgr2=0x55a459efa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.836+0000 7fefb7fff640 1 -- 172.21.15.116:0/2376445284 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a459b6a3f0 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fefa8013e10 tx=0x7fefa8013590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac0a29d6ee6492e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fef9effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8010430 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fefa80105d0 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb002c7d0 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fefa802a050 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fefc8e40740 10 client.32125 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a459c347d0 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.837+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefb002c7d0 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fefa8020500 con 0x55a459ef32d0 2024-03-19T02:52:17.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 10 client.32125 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 10 client.32125 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 20 client.32125 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f0b4d0 0x55a459f0b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 4 client.32125 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a459eb43e0 con 0x55a459f0b4d0 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f0b4d0 0x55a459f0b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fef9effd640 10 client.32125 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.838+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f0b4d0 0x55a459f0b8f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cce069440ed4a38 server_cookie=4aac03831b86600f in_seq=0 out_seq=0 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fef9effd640 1 -- 172.21.15.116:0/2376445284 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a459eb43e0 con 0x55a459f0b4d0 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fef9effd640 10 client.32125 handle_command_reply: tid=0 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 client.32125 shutdown 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 msgr2=0x55a459ef9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fefa8013e10 tx=0x7fefa8013590 comp rx=0 tx=0).stop 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 shutdown_connections 2024-03-19T02:52:17.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f0b4d0 0x55a459f0b8f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef3bf0 0x55a459ef9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2376445284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef32d0 0x55a459ef9800 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fef9effd640 0 client.32125 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 >> 172.21.15.116:0/2376445284 conn(0x55a459eeda00 msgr2=0x55a459eef750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 shutdown_connections 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 1 -- 172.21.15.116:0/2376445284 wait complete. 2024-03-19T02:52:17.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.839+0000 7fefc8e40740 20 client.32125 trim_cache size 0 max 16384 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.854+0000 7fefc8e40740 1 Processor -- start 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.854+0000 7fefc8e40740 1 -- start start 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.854+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459efbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.854+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459ef87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefc8e40740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01aca20 con 0x55a459efc7d0 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefc8e40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01ae560 con 0x55a459ef32d0 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefc8e40740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01ad0f0 con 0x55a459ef3bf0 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefbf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459ef87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44864/0 (socket says 172.21.15.116:44864) 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 -- 172.21.15.116:0/2590831623 learned_addr learned my addr 172.21.15.116:0/2590831623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459efbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 -- 172.21.15.116:0/2590831623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 msgr2=0x55a459ef87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459ef87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 -- 172.21.15.116:0/2590831623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 msgr2=0x55a459efbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459efbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 -- 172.21.15.116:0/2590831623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a459ef3a20 con 0x55a459efc7d0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459ef87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fefa8020fd0 tx=0x7fefa80139b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a372f46b12501cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb57fa640 1 -- 172.21.15.116:0/2590831623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8005d30 con 0x55a459efc7d0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb57fa640 1 -- 172.21.15.116:0/2590831623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefa8009e20 con 0x55a459efc7d0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.855+0000 7fefb57fa640 1 -- 172.21.15.116:0/2590831623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8010820 con 0x55a459efc7d0 2024-03-19T02:52:17.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- 172.21.15.116:0/2590831623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 msgr2=0x55a459efa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fefa8020fd0 tx=0x7fefa80139b0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- 172.21.15.116:0/2590831623 shutdown_connections 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459efc7d0 0x55a459efa0a0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459ef3bf0 0x55a459ef87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2590831623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459efbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- 172.21.15.116:0/2590831623 >> 172.21.15.116:0/2590831623 conn(0x55a459eeda00 msgr2=0x55a459ef80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- 172.21.15.116:0/2590831623 shutdown_connections 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- 172.21.15.116:0/2590831623 wait complete. 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.856+0000 7fefc8e40740 1 -- start start 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 Processor -- start 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459f1dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459efc7d0 0x55a459f18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01ade90 con 0x55a459ef3bf0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01ad7c0 con 0x55a459ef32d0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01aaa20 con 0x55a459efc7d0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefc8e40740 10 client.0 _ll_register_callbacks cb 0x7fefa8013c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44874/0 (socket says 172.21.15.116:44874) 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 -- 172.21.15.116:0/2588046966 learned_addr learned my addr 172.21.15.116:0/2588046966 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459f1dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459efc7d0 0x55a459f18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 -- 172.21.15.116:0/2588046966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459efc7d0 msgr2=0x55a459f18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459efc7d0 0x55a459f18760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 -- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 msgr2=0x55a459f1dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459f1dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 -- 172.21.15.116:0/2588046966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a459f18f70 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fefbf7fe640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x55a459ef2da0 tx=0x7fefb002c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0798b49d573223 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fef9effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.857+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0045040 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fefb0004240 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa8009e20 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fefb005c050 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 10 client.22437 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a459f7b250 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefa8009e20 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fefb00046c0 con 0x55a459ef3bf0 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 10 client.22437 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 10 client.22437 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:52:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 20 client.22437 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f7b650 0x55a459f7da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 4 client.22437 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a459e639f0 con 0x55a459f7b650 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.858+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f7b650 0x55a459f7da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fef9effd640 10 client.22437 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefb7fff640 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f7b650 0x55a459f7da40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=605b1edb7b22a762 server_cookie=7aab812038a8baf8 in_seq=0 out_seq=0 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fef9effd640 1 -- 172.21.15.116:0/2588046966 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a459e639f0 con 0x55a459f7b650 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fef9effd640 10 client.22437 handle_command_reply: tid=0 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 client.22437 shutdown 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 msgr2=0x55a459f18220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x55a459ef2da0 tx=0x7fefb002c1e0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 shutdown_connections 2024-03-19T02:52:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55a459f7b650 0x55a459f7da40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a459efc7d0 0x55a459f18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fef9effd640 0 client.22437 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a459ef3bf0 0x55a459f18220 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 --2- 172.21.15.116:0/2588046966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a459ef32d0 0x55a459f1dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 >> 172.21.15.116:0/2588046966 conn(0x55a459eeda00 msgr2=0x55a459f081c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 shutdown_connections 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 1 -- 172.21.15.116:0/2588046966 wait complete. 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.859+0000 7fefc8e40740 20 client.22437 trim_cache size 0 max 16384 2024-03-19T02:52:17.868 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 msgr2=0x7fefac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 0x7fefac0a4eb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fefb8097dc0 tx=0x7fefb801d000 comp rx=0 tx=0).stop 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 msgr2=0x7fefc019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc019c900 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fefb800ec50 tx=0x7fefb800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 shutdown_connections 2024-03-19T02:52:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fefac0a29f0 0x7fefac0a4eb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc01077a0 0x7fefc019c900 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefc0102150 0x7fefc0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 --2- 172.21.15.116:0/3333908108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefc0101790 0x7fefc0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 >> 172.21.15.116:0/3333908108 conn(0x7fefc00fd2c0 msgr2=0x7fefc00fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 shutdown_connections 2024-03-19T02:52:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:17.860+0000 7fefc6259640 1 -- 172.21.15.116:0/3333908108 wait complete. 2024-03-19T02:52:17.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:52:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 -- 172.21.15.116:0/1523915962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 msgr2=0x7fca9810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/1523915962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca9810d570 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fca90009cd0 tx=0x7fca9002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 -- 172.21.15.116:0/1523915962 shutdown_connections 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/1523915962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca9810d570 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/1523915962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/1523915962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 0x7fca98106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 -- 172.21.15.116:0/1523915962 >> 172.21.15.116:0/1523915962 conn(0x7fca980778d0 msgr2=0x7fca98077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 -- 172.21.15.116:0/1523915962 shutdown_connections 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.004+0000 7fca9ce6e640 1 -- 172.21.15.116:0/1523915962 wait complete. 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 Processor -- start 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 -- start start 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca98199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 0x7fca981a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca98110970 con 0x7fca98101060 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca981107f0 con 0x7fca981006a0 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9ce6e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca98110af0 con 0x7fca981066b0 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37976/0 (socket says 172.21.15.116:37976) 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9659c640 1 -- 172.21.15.116:0/377799834 learned_addr learned my addr 172.21.15.116:0/377799834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca98199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca96d9d640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 0x7fca981a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 -- 172.21.15.116:0/377799834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 msgr2=0x7fca981a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 0x7fca981a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 -- 172.21.15.116:0/377799834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 msgr2=0x7fca98198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca95d9b640 1 -- 172.21.15.116:0/377799834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca98196c00 con 0x7fca98101060 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.005+0000 7fca9659c640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca95d9b640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca98199350 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e9f0 tx=0x7fca8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca877fe640 1 -- 172.21.15.116:0/377799834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00ccc0 con 0x7fca98101060 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca98196ef0 con 0x7fca98101060 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca98197490 con 0x7fca98101060 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca877fe640 1 -- 172.21.15.116:0/377799834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca8c0108e0 con 0x7fca98101060 2024-03-19T02:52:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.006+0000 7fca877fe640 1 -- 172.21.15.116:0/377799834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c026060 con 0x7fca98101060 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.008+0000 7fca877fe640 1 -- 172.21.15.116:0/377799834 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fca8c010430 con 0x7fca98101060 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 Processor -- start 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 -- start start 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c72290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c74bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f17bcc4b0 con 0x557f17c727d0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557f17b74d70 con 0x557f17c692d0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9fa56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557f17bc8550 con 0x557f17c69bf0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca877fe640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 0x7fca800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca95d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c74bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c72290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c72290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54730/0 (socket says 172.21.15.116:54730) 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:37982/0 (socket says 172.21.15.116:37982) 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca9659c640 1 -- 172.21.15.116:0/396979351 learned_addr learned my addr 172.21.15.116:0/396979351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 0x7fca800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca877fe640 1 -- 172.21.15.116:0/377799834 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fca8c0cf4b0 con 0x7fca98101060 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 -- 172.21.15.116:0/396979351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 msgr2=0x557f17c72290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c72290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 -- 172.21.15.116:0/396979351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 msgr2=0x557f17c74bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c74bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca96d9d640 1 -- 172.21.15.116:0/396979351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f17b97210 con 0x557f17c692d0 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.009+0000 7fca95d9b640 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c74bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.010+0000 7fca96d9d640 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca9002c630 tx=0x7fca90032700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510eb8d1f6d63c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.010+0000 7fca96d9d640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 0x7fca800a4e30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fca90037040 tx=0x7fca90042000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.091+0000 7fca84ff9640 1 -- 172.21.15.116:0/396979351 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90051070 con 0x557f17c692d0 2024-03-19T02:52:18.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.091+0000 7fca84ff9640 1 -- 172.21.15.116:0/396979351 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca90004040 con 0x557f17c692d0 2024-03-19T02:52:18.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.091+0000 7fca84ff9640 1 -- 172.21.15.116:0/396979351 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90039040 con 0x557f17c692d0 2024-03-19T02:52:18.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- 172.21.15.116:0/396979351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 msgr2=0x557f17c696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca9002c630 tx=0x7fca90032700 comp rx=0 tx=0).stop 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- 172.21.15.116:0/396979351 shutdown_connections 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c74bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c72290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 --2- 172.21.15.116:0/396979351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c696b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- 172.21.15.116:0/396979351 >> 172.21.15.116:0/396979351 conn(0x557f17c63a00 msgr2=0x557f17c63de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- 172.21.15.116:0/396979351 shutdown_connections 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- 172.21.15.116:0/396979351 wait complete. 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 -- start start 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.092+0000 7fca9fa56740 1 Processor -- start 2024-03-19T02:52:18.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c6f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c727d0 0x557f17c702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f17c75560 con 0x557f17c69bf0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557f17b88310 con 0x557f17c692d0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557f17c756e0 con 0x557f17c727d0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 10 client.0 _ll_register_callbacks cb 0x557f17b7c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44914/0 (socket says 172.21.15.116:44914) 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 -- 172.21.15.116:0/2892747921 learned_addr learned my addr 172.21.15.116:0/2892747921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9659c640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c6f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca96d9d640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c727d0 0x557f17c702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 -- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c727d0 msgr2=0x557f17c702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c727d0 0x557f17c702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 -- 172.21.15.116:0/2892747921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 msgr2=0x557f17c6f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c6f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 -- 172.21.15.116:0/2892747921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f178e03f0 con 0x557f17c69bf0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9659c640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c6f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca95d9b640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0981b0 tx=0x7fca8c009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d17fc934f2f2eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c09f880 con 0x557f17c69bf0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fca8c0cf170 con 0x557f17c69bf0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.093+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca90004040 con 0x557f17c69bf0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fca8c0a3ed0 con 0x557f17c69bf0 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 10 client.22449 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:18.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x557f179aa7d0 con 0x557f17c69bf0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca90004040 con 0x557f17c69bf0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fca8c0975c0 con 0x557f17c69bf0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 10 client.22449 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 10 client.22449 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 20 client.22449 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17c82ac0 0x557f17c84f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 4 client.22449 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f17c2a3e0 con 0x557f17c82ac0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.094+0000 7fca9659c640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17c82ac0 0x557f17c84f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca7dffb640 10 client.22449 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9659c640 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17c82ac0 0x557f17c84f20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3d20fbbb8c92a1c server_cookie=7046201c56b61c8 in_seq=0 out_seq=0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca7dffb640 1 -- 172.21.15.116:0/2892747921 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f17c2a3e0 con 0x557f17c82ac0 2024-03-19T02:52:18.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca7dffb640 10 client.22449 handle_command_reply: tid=0 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 client.22449 shutdown 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 msgr2=0x557f17c6fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0981b0 tx=0x7fca8c009700 comp rx=0 tx=0).stop 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 shutdown_connections 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17c82ac0 0x557f17c84f20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c727d0 0x557f17c702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c69bf0 0x557f17c6fd80 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 --2- 172.21.15.116:0/2892747921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c6f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.095+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 >> 172.21.15.116:0/2892747921 conn(0x557f17c63a00 msgr2=0x557f17c65750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.096+0000 7fca7dffb640 0 client.22449 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.096+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 shutdown_connections 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.096+0000 7fca9fa56740 1 -- 172.21.15.116:0/2892747921 wait complete. 2024-03-19T02:52:18.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.096+0000 7fca9fa56740 20 client.22449 trim_cache size 0 max 16384 2024-03-19T02:52:18.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 Processor -- start 2024-03-19T02:52:18.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 -- start start 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c7e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c727d0 0x557f17c8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800ff10 con 0x557f17c692d0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca78011a50 con 0x557f17c727d0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9fa56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca780105e0 con 0x557f17c69bf0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44928/0 (socket says 172.21.15.116:44928) 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 -- 172.21.15.116:0/1479064168 learned_addr learned my addr 172.21.15.116:0/1479064168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca95d9b640 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c7e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca96d9d640 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c727d0 0x557f17c8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 -- 172.21.15.116:0/1479064168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 msgr2=0x557f17c7e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c7e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 -- 172.21.15.116:0/1479064168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c727d0 msgr2=0x557f17c8a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c727d0 0x557f17c8a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.110+0000 7fca9659c640 1 -- 172.21.15.116:0/1479064168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f17c74ef0 con 0x557f17c692d0 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.111+0000 7fca9659c640 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fca88009620 tx=0x7fca88005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66df4199fb4f7e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.111+0000 7fca7ffff640 1 -- 172.21.15.116:0/1479064168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca88005020 con 0x557f17c692d0 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.111+0000 7fca7ffff640 1 -- 172.21.15.116:0/1479064168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca88026030 con 0x557f17c692d0 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.111+0000 7fca7ffff640 1 -- 172.21.15.116:0/1479064168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca880220f0 con 0x557f17c692d0 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- 172.21.15.116:0/1479064168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 msgr2=0x557f17c71f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fca88009620 tx=0x7fca88005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- 172.21.15.116:0/1479064168 shutdown_connections 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c727d0 0x557f17c8a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c7e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1479064168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c692d0 0x557f17c71f70 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- 172.21.15.116:0/1479064168 >> 172.21.15.116:0/1479064168 conn(0x557f17c63a00 msgr2=0x557f17c6abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- 172.21.15.116:0/1479064168 shutdown_connections 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- 172.21.15.116:0/1479064168 wait complete. 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- start start 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 Processor -- start 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c7a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c6be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca78011380 con 0x557f17c727d0 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca78010cb0 con 0x557f17c692d0 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800df10 con 0x557f17c69bf0 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca9fa56740 10 client.0 _ll_register_callbacks cb 0x7fca8c097410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:18.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca96d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca96d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44930/0 (socket says 172.21.15.116:44930) 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca96d9d640 1 -- 172.21.15.116:0/1878380156 learned_addr learned my addr 172.21.15.116:0/1878380156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.112+0000 7fca95d9b640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c6be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 -- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 msgr2=0x557f17c6be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c6be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 -- 172.21.15.116:0/1878380156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 msgr2=0x557f17c7a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c7a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 -- 172.21.15.116:0/1878380156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f17c6c8e0 con 0x557f17c727d0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca96d9d640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fca9002c630 tx=0x7fca90004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7359ec955c10a248 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90051040 con 0x557f17c727d0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fca90004040 con 0x557f17c727d0 2024-03-19T02:52:18.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca88026030 con 0x557f17c727d0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fca90063050 con 0x557f17c727d0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca9fa56740 10 client.22455 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x557f17ce8b80 con 0x557f17c727d0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.113+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca88026030 con 0x557f17c727d0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fca9004b030 con 0x557f17c727d0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 10 client.22455 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 10 client.22455 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 20 client.22455 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17ce8ed0 0x557f17ceb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 4 client.22455 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f17bd99f0 con 0x557f17ce8ed0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9659c640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17ce8ed0 0x557f17ceb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca7dffb640 10 client.22455 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9659c640 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17ce8ed0 0x557f17ceb2e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1af52018ba486f88 server_cookie=f94bc6e9d6b50f7e in_seq=0 out_seq=0 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca7dffb640 1 -- 172.21.15.116:0/1878380156 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f17bd99f0 con 0x557f17ce8ed0 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca7dffb640 10 client.22455 handle_command_reply: tid=0 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.114+0000 7fca9fa56740 1 client.22455 shutdown 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 msgr2=0x557f17c6c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fca9002c630 tx=0x7fca90004f90 comp rx=0 tx=0).stop 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 shutdown_connections 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x557f17ce8ed0 0x557f17ceb2e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f17c727d0 0x557f17c6c3a0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f17c69bf0 0x557f17c6be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca7dffb640 0 client.22455 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 --2- 172.21.15.116:0/1878380156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f17c692d0 0x557f17c7a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 >> 172.21.15.116:0/1878380156 conn(0x557f17c63a00 msgr2=0x557f17c6e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 shutdown_connections 2024-03-19T02:52:18.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 1 -- 172.21.15.116:0/1878380156 wait complete. 2024-03-19T02:52:18.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9fa56740 20 client.22455 trim_cache size 0 max 16384 2024-03-19T02:52:18.123 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 msgr2=0x7fca800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 0x7fca800a4e30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fca90037040 tx=0x7fca90042000 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 msgr2=0x7fca98199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca98199350 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e9f0 tx=0x7fca8c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 shutdown_connections 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fca800a2970 0x7fca800a4e30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca981066b0 0x7fca981a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca98101060 0x7fca98199350 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 --2- 172.21.15.116:0/377799834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca981006a0 0x7fca98198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 >> 172.21.15.116:0/377799834 conn(0x7fca980778d0 msgr2=0x7fca98102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 shutdown_connections 2024-03-19T02:52:18.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.115+0000 7fca9ce6e640 1 -- 172.21.15.116:0/377799834 wait complete. 2024-03-19T02:52:18.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:52:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.252+0000 7f6a47ad1640 1 -- 172.21.15.116:0/3623247890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 msgr2=0x7f6a40101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.252+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/3623247890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a40101c10 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009930 tx=0x7f6a3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:17 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2153820446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.256+0000 7f6a47ad1640 1 -- 172.21.15.116:0/3623247890 shutdown_connections 2024-03-19T02:52:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.256+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/3623247890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.256+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/3623247890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a40101c10 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.256+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/3623247890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 0x7f6a40107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.256+0000 7f6a47ad1640 1 -- 172.21.15.116:0/3623247890 >> 172.21.15.116:0/3623247890 conn(0x7f6a400fd2c0 msgr2=0x7f6a400ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.259+0000 7f6a47ad1640 1 -- 172.21.15.116:0/3623247890 shutdown_connections 2024-03-19T02:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.259+0000 7f6a47ad1640 1 -- 172.21.15.116:0/3623247890 wait complete. 2024-03-19T02:52:18.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:17 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2153820446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:18.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.260+0000 7f6a47ad1640 1 Processor -- start 2024-03-19T02:52:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a47ad1640 1 -- start start 2024-03-19T02:52:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a47ad1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a4019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a47ad1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a4019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a4019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44950/0 (socket says 172.21.15.116:44950) 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a4586d640 1 -- 172.21.15.116:0/105459375 learned_addr learned my addr 172.21.15.116:0/105459375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.262+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 0x7f6a4019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40111a60 con 0x7f6a40101790 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401118e0 con 0x7f6a401077a0 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40111be0 con 0x7f6a40102150 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4606e640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 0x7f6a4019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 -- 172.21.15.116:0/105459375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 msgr2=0x7f6a4019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 0x7f6a4019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 -- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 msgr2=0x7f6a4019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a4019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.263+0000 7f6a4506c640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4019f1b0 con 0x7f6a40102150 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a4506c640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4019a700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c170 tx=0x7f6a3002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a36ffd640 1 -- 172.21.15.116:0/105459375 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30043020 con 0x7f6a40102150 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4019f440 con 0x7f6a40102150 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a36ffd640 1 -- 172.21.15.116:0/105459375 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3002ca70 con 0x7f6a40102150 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a36ffd640 1 -- 172.21.15.116:0/105459375 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003970 con 0x7f6a40102150 2024-03-19T02:52:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.264+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401a6d30 con 0x7f6a40102150 2024-03-19T02:52:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.267+0000 7f6a36ffd640 1 -- 172.21.15.116:0/105459375 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6a30046020 con 0x7f6a40102150 2024-03-19T02:52:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.267+0000 7f6a4a798740 1 Processor -- start 2024-03-19T02:52:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.267+0000 7f6a4a798740 1 -- start start 2024-03-19T02:52:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.267+0000 7f6a36ffd640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 0x7f6a2c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.267+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa6929290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4a798740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563aa68834b0 con 0x563aa69297d0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4a798740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563aa682bd70 con 0x563aa6920bf0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4a798740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563aa687f550 con 0x563aa69202d0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a36ffd640 1 -- 172.21.15.116:0/105459375 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f6a300f0fc0 con 0x7f6a40102150 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4606e640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 0x7f6a2c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54792/0 (socket says 172.21.15.116:54792) 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 -- 172.21.15.116:0/1899236272 learned_addr learned my addr 172.21.15.116:0/1899236272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4606e640 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4506c640 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa6929290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 -- 172.21.15.116:0/1899236272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 msgr2=0x563aa6929290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa6929290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 -- 172.21.15.116:0/1899236272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 msgr2=0x563aa692bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4586d640 1 -- 172.21.15.116:0/1899236272 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aa684e210 con 0x563aa69202d0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.268+0000 7f6a4606e640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 0x7f6a2c0a4eb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005fd0 tx=0x7f6a38013750 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.269+0000 7f6a4586d640 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009d40 tx=0x7f6a3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dccbc37a17fd2e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.269+0000 7f6a2bfff640 1 -- 172.21.15.116:0/1899236272 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c009e30 con 0x563aa69202d0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.269+0000 7f6a2bfff640 1 -- 172.21.15.116:0/1899236272 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3c005020 con 0x563aa69202d0 2024-03-19T02:52:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.269+0000 7f6a2bfff640 1 -- 172.21.15.116:0/1899236272 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c004b80 con 0x563aa69202d0 2024-03-19T02:52:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- 172.21.15.116:0/1899236272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 msgr2=0x563aa69206b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009d40 tx=0x7f6a3c011810 comp rx=0 tx=0).stop 2024-03-19T02:52:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- 172.21.15.116:0/1899236272 shutdown_connections 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa6929290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- 172.21.15.116:0/1899236272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69202d0 0x563aa69206b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- 172.21.15.116:0/1899236272 >> 172.21.15.116:0/1899236272 conn(0x563aa691aa00 msgr2=0x563aa691ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- 172.21.15.116:0/1899236272 shutdown_connections 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- 172.21.15.116:0/1899236272 wait complete. 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- start start 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 Processor -- start 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa6926c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 0x563aa6927190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563aa692c5c0 con 0x563aa69202d0 2024-03-19T02:52:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.270+0000 7f6a4a798740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563aa683f310 con 0x563aa69297d0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4a798740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563aa692c740 con 0x563aa6920bf0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44962/0 (socket says 172.21.15.116:44962) 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 -- 172.21.15.116:0/4117574890 learned_addr learned my addr 172.21.15.116:0/4117574890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4a798740 10 client.0 _ll_register_callbacks cb 0x563aa68332d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4506c640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa6926c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4606e640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 0x563aa6927190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 -- 172.21.15.116:0/4117574890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 msgr2=0x563aa6926c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa6926c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 -- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 msgr2=0x563aa6927190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 0x563aa6927190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 -- 172.21.15.116:0/4117574890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aa65973f0 con 0x563aa69202d0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4606e640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 0x563aa6927190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c004630 tx=0x7f6a3c024bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1319aeee29f0b00a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a297fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c032060 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a3c005020 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.271+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f6a3c016040 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6a3c009e30 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 10 client.22467 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x563aa66617d0 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3c005020 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f6a3c02d670 con 0x563aa69202d0 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 10 client.22467 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 10 client.22467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 20 client.22467 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa6939c50 0x563aa693c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 4 client.22467 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aa68e13e0 con 0x563aa6939c50 2024-03-19T02:52:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.272+0000 7f6a4506c640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa6939c50 0x563aa693c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.273+0000 7f6a4506c640 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa6939c50 0x563aa693c0e0 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5dccb66edeb3819 server_cookie=f4f7af068e2cb8ca in_seq=0 out_seq=0 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.273+0000 7f6a297fa640 10 client.22467 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a297fa640 1 -- 172.21.15.116:0/4117574890 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563aa68e13e0 con 0x563aa6939c50 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a297fa640 10 client.22467 handle_command_reply: tid=0 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 client.22467 shutdown 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 msgr2=0x563aa6926710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c004630 tx=0x7f6a3c024bb0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 shutdown_connections 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa6939c50 0x563aa693c0e0 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69297d0 0x563aa6927190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa6926c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4117574890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa6926710 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a297fa640 0 client.22467 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 >> 172.21.15.116:0/4117574890 conn(0x563aa691aa00 msgr2=0x563aa691c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 shutdown_connections 2024-03-19T02:52:18.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 1 -- 172.21.15.116:0/4117574890 wait complete. 2024-03-19T02:52:18.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.274+0000 7f6a4a798740 20 client.22467 trim_cache size 0 max 16384 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 Processor -- start 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 -- start start 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa69419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69297d0 0x563aa69319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2400ff60 con 0x563aa69202d0 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a24011aa0 con 0x563aa6920bf0 2024-03-19T02:52:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4a798740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a24010630 con 0x563aa69297d0 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4606e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69297d0 0x563aa69319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44974/0 (socket says 172.21.15.116:44974) 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.295+0000 7f6a4586d640 1 -- 172.21.15.116:0/758844776 learned_addr learned my addr 172.21.15.116:0/758844776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4506c640 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa69419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 -- 172.21.15.116:0/758844776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69297d0 msgr2=0x563aa69319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69297d0 0x563aa69319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 -- 172.21.15.116:0/758844776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 msgr2=0x563aa69419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa69419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 -- 172.21.15.116:0/758844776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aa69215a0 con 0x563aa69202d0 2024-03-19T02:52:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a4586d640 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c011850 tx=0x7f6a3c0095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecd676a434f3bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a2b7fe640 1 -- 172.21.15.116:0/758844776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c032040 con 0x563aa69202d0 2024-03-19T02:52:18.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a2b7fe640 1 -- 172.21.15.116:0/758844776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3c009860 con 0x563aa69202d0 2024-03-19T02:52:18.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.296+0000 7f6a2b7fe640 1 -- 172.21.15.116:0/758844776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c019bb0 con 0x563aa69202d0 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 -- 172.21.15.116:0/758844776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 msgr2=0x563aa69353c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c011850 tx=0x7f6a3c0095b0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 -- 172.21.15.116:0/758844776 shutdown_connections 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa69297d0 0x563aa69319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa6920bf0 0x563aa69419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 --2- 172.21.15.116:0/758844776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69202d0 0x563aa69353c0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.297+0000 7f6a4a798740 1 -- 172.21.15.116:0/758844776 >> 172.21.15.116:0/758844776 conn(0x563aa691aa00 msgr2=0x563aa691c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- 172.21.15.116:0/758844776 shutdown_connections 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- 172.21.15.116:0/758844776 wait complete. 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- start start 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 Processor -- start 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa692efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a240113d0 con 0x563aa69297d0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a24010d00 con 0x563aa69202d0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2400df60 con 0x563aa6920bf0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4a798740 10 client.0 _ll_register_callbacks cb 0x7f6a38013620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4586d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38070/0 (socket says 172.21.15.116:38070) 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.298+0000 7f6a4586d640 1 -- 172.21.15.116:0/4148650402 learned_addr learned my addr 172.21.15.116:0/4148650402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4606e640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 -- 172.21.15.116:0/4148650402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 msgr2=0x563aa692efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa692efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 -- 172.21.15.116:0/4148650402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 msgr2=0x563aa692f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 -- 172.21.15.116:0/4148650402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aa692faf0 con 0x563aa69202d0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4586d640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c000c00 tx=0x7f6a3c02d600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ef8c342bfd52262 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a297fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c015070 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.299+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a3c009860 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f6a3c019930 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6a3c04b050 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a4a798740 10 client.32174 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x563aa69b0e40 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3c009860 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f6a3c003860 con 0x563aa69202d0 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a4a798740 10 client.32174 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a4a798740 10 client.32174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.300+0000 7f6a4a798740 20 client.32174 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa69b1520 0x563aa69b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a4a798740 4 client.32174 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:18.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563aa68909f0 con 0x563aa69b1520 2024-03-19T02:52:18.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a4506c640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa69b1520 0x563aa69b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a4506c640 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa69b1520 0x563aa69b3940 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed88b9c62099cb05 server_cookie=ea3cb957c8691c32 in_seq=0 out_seq=0 2024-03-19T02:52:18.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.301+0000 7f6a297fa640 10 client.32174 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a297fa640 1 -- 172.21.15.116:0/4148650402 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563aa68909f0 con 0x563aa69b1520 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a297fa640 10 client.32174 handle_command_reply: tid=0 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 client.32174 shutdown 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 msgr2=0x563aa6931a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c000c00 tx=0x7f6a3c02d600 comp rx=0 tx=0).stop 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 shutdown_connections 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x563aa69b1520 0x563aa69b3940 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563aa69297d0 0x563aa692f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563aa6920bf0 0x563aa692efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a297fa640 0 client.32174 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 --2- 172.21.15.116:0/4148650402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563aa69202d0 0x563aa6931a30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 >> 172.21.15.116:0/4148650402 conn(0x563aa691aa00 msgr2=0x563aa691bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 shutdown_connections 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.303+0000 7f6a4a798740 1 -- 172.21.15.116:0/4148650402 wait complete. 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a4a798740 20 client.32174 trim_cache size 0 max 16384 2024-03-19T02:52:18.308 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 msgr2=0x7f6a2c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 0x7f6a2c0a4eb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005fd0 tx=0x7f6a38013750 comp rx=0 tx=0).stop 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 msgr2=0x7f6a4019a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4019a700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c170 tx=0x7f6a3002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 shutdown_connections 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f6a2c0a29f0 0x7f6a2c0a4eb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a401077a0 0x7f6a4019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a40102150 0x7f6a4019a700 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 --2- 172.21.15.116:0/105459375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a40101790 0x7f6a4019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 >> 172.21.15.116:0/105459375 conn(0x7f6a400fd2c0 msgr2=0x7f6a4010cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 shutdown_connections 2024-03-19T02:52:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:18.304+0000 7f6a47ad1640 1 -- 172.21.15.116:0/105459375 wait complete. 2024-03-19T02:52:18.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74903 5.64 MB/sec execute 580 sec latency 1322.187 ms 2024-03-19T02:52:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:17 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2153820446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:18 smithi154 ceph-mon[35693]: pgmap v775: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:18 smithi116 ceph-mon[32263]: pgmap v775: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:19.317 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:52:19.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:52:19.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:52:19.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 74989 5.64 MB/sec execute 581 sec latency 2322.236 ms 2024-03-19T02:52:19.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:18 smithi162 ceph-mon[35761]: pgmap v775: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:19.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- 172.21.15.116:0/327601690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 msgr2=0x7fc42810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/327601690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc42810d560 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fc420009cd0 tx=0x7fc42002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- 172.21.15.116:0/327601690 shutdown_connections 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/327601690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc42810d560 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/327601690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/327601690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 0x7fc428106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- 172.21.15.116:0/327601690 >> 172.21.15.116:0/327601690 conn(0x7fc428077820 msgr2=0x7fc428077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- 172.21.15.116:0/327601690 shutdown_connections 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- 172.21.15.116:0/327601690 wait complete. 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 Processor -- start 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 -- start start 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.442+0000 7fc42f5c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 0x7fc4281a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc428110960 con 0x7fc428101050 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281107e0 con 0x7fc428100690 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc428110ae0 con 0x7fc4281066a0 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45002/0 (socket says 172.21.15.116:45002) 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42d362640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 -- 172.21.15.116:0/617022288 learned_addr learned my addr 172.21.15.116:0/617022288 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42db63640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 0x7fc4281a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 -- 172.21.15.116:0/617022288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 msgr2=0x7fc4281a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 0x7fc4281a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 -- 172.21.15.116:0/617022288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 msgr2=0x7fc428198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 -- 172.21.15.116:0/617022288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4281a4430 con 0x7fc428101050 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42d362640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42cb61640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fc41800e9f0 tx=0x7fc41800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc41e7fc640 1 -- 172.21.15.116:0/617022288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41800cd80 con 0x7fc428101050 2024-03-19T02:52:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc41e7fc640 1 -- 172.21.15.116:0/617022288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4180108e0 con 0x7fc428101050 2024-03-19T02:52:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4281a4600 con 0x7fc428101050 2024-03-19T02:52:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc41e7fc640 1 -- 172.21.15.116:0/617022288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418018920 con 0x7fc428101050 2024-03-19T02:52:19.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.443+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4281a4b50 con 0x7fc428101050 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc41e7fc640 1 -- 172.21.15.116:0/617022288 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc418010a90 con 0x7fc428101050 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 Processor -- start 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 -- start start 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518517d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x565185188c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651850e1160 con 0x56518517dca0 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565185089e10 con 0x565185186880 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc4321a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5651850dd200 con 0x56518517d380 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.446+0000 7fc41e7fc640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 0x7fc4140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42cb61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x565185188c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45016/0 (socket says 172.21.15.116:45016) 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 -- 172.21.15.116:0/3963020347 learned_addr learned my addr 172.21.15.116:0/3963020347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42db63640 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518517d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42db63640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 0x7fc4140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 -- 172.21.15.116:0/3963020347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 msgr2=0x56518517d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518517d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 -- 172.21.15.116:0/3963020347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 msgr2=0x565185188c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x565185188c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc41e7fc640 1 -- 172.21.15.116:0/617022288 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc4180cf4c0 con 0x7fc428101050 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 -- 172.21.15.116:0/3963020347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565184ebd7e0 con 0x56518517dca0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42d362640 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc4240114c0 tx=0x7fc424011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd11df804c8d325e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc42db63640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 0x7fc4140a4e30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc420005cc0 tx=0x7fc420005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc4137fe640 1 -- 172.21.15.116:0/3963020347 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424004020 con 0x56518517dca0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc4137fe640 1 -- 172.21.15.116:0/3963020347 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4240041c0 con 0x56518517dca0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.447+0000 7fc4137fe640 1 -- 172.21.15.116:0/3963020347 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400f820 con 0x56518517dca0 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- 172.21.15.116:0/3963020347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 msgr2=0x565185186340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fc4240114c0 tx=0x7fc424011980 comp rx=0 tx=0).stop 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- 172.21.15.116:0/3963020347 shutdown_connections 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x565185188c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x565185186340 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3963020347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518517d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- 172.21.15.116:0/3963020347 >> 172.21.15.116:0/3963020347 conn(0x565185177ab0 msgr2=0x565185177e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- 172.21.15.116:0/3963020347 shutdown_connections 2024-03-19T02:52:19.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- 172.21.15.116:0/3963020347 wait complete. 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 -- start start 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.448+0000 7fc4321a8740 1 Processor -- start 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517dca0 0x565185183d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851842d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565185189670 con 0x56518517d380 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56518509d1a0 con 0x565185186880 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5651851897f0 con 0x56518517dca0 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 10 client.0 _ll_register_callbacks cb 0x565184df4c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45020/0 (socket says 172.21.15.116:45020) 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 -- 172.21.15.116:0/751695700 learned_addr learned my addr 172.21.15.116:0/751695700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42cb61640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517dca0 0x565185183d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42db63640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851842d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 -- 172.21.15.116:0/751695700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517dca0 msgr2=0x565185183d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517dca0 0x565185183d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 -- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 msgr2=0x5651851842d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851842d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 -- 172.21.15.116:0/751695700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565184f1df70 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc42d362640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc42401b040 tx=0x7fc424009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadc32f3d1bcb575 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc410ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42401d040 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc424004370 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.449+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4240041c0 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc42402f050 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 10 client.22485 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x565184db5170 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4240041c0 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc424020070 con 0x56518517d380 2024-03-19T02:52:19.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 10 client.22485 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 10 client.22485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 20 client.22485 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x565185196b40 0x565185198f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 4 client.22485 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651850ee6a0 con 0x565185196b40 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.450+0000 7fc42cb61640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x565185196b40 0x565185198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.451+0000 7fc410ff9640 10 client.22485 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.451+0000 7fc42cb61640 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x565185196b40 0x565185198f30 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b35d5f36a0b92e6d server_cookie=ec7c45f473bc1ae3 in_seq=0 out_seq=0 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc410ff9640 1 -- 172.21.15.116:0/751695700 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651850ee6a0 con 0x565185196b40 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc410ff9640 10 client.22485 handle_command_reply: tid=0 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 client.22485 shutdown 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 msgr2=0x565185183850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fc42401b040 tx=0x7fc424009de0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 shutdown_connections 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x565185196b40 0x565185198f30 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851842d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517dca0 0x565185183d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 --2- 172.21.15.116:0/751695700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185183850 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc410ff9640 0 client.22485 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 >> 172.21.15.116:0/751695700 conn(0x565185177ab0 msgr2=0x565185179800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 shutdown_connections 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 1 -- 172.21.15.116:0/751695700 wait complete. 2024-03-19T02:52:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.452+0000 7fc4321a8740 20 client.22485 trim_cache size 0 max 16384 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 Processor -- start 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 -- start start 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56518517dca0 0x565185192410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281aed30 con 0x56518517d380 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281b0870 con 0x56518517dca0 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.475+0000 7fc4321a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281af400 con 0x565185186880 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42db63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45022/0 (socket says 172.21.15.116:45022) 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42db63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54848/0 (socket says 172.21.15.116:54848) 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 -- 172.21.15.116:0/2270940567 learned_addr learned my addr 172.21.15.116:0/2270940567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42cb61640 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56518517dca0 0x565185192410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 -- 172.21.15.116:0/2270940567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 msgr2=0x56518519e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 -- 172.21.15.116:0/2270940567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56518517dca0 msgr2=0x565185192410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56518517dca0 0x565185192410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 -- 172.21.15.116:0/2270940567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651851778e0 con 0x56518517d380 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42db63640 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc42d362640 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc424004f30 tx=0x7fc424004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bad7754a43716be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.476+0000 7fc412ffd640 1 -- 172.21.15.116:0/2270940567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42401d040 con 0x56518517d380 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc412ffd640 1 -- 172.21.15.116:0/2270940567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc424015030 con 0x56518517d380 2024-03-19T02:52:19.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc412ffd640 1 -- 172.21.15.116:0/2270940567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240043d0 con 0x56518517d380 2024-03-19T02:52:19.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 -- 172.21.15.116:0/2270940567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 msgr2=0x565185185f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fc424004f30 tx=0x7fc424004f60 comp rx=0 tx=0).stop 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 -- 172.21.15.116:0/2270940567 shutdown_connections 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x565185186880 0x56518519e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56518517dca0 0x565185192410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.477+0000 7fc4321a8740 1 --2- 172.21.15.116:0/2270940567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517d380 0x565185185f80 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- 172.21.15.116:0/2270940567 >> 172.21.15.116:0/2270940567 conn(0x565185177ab0 msgr2=0x5651851840a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- 172.21.15.116:0/2270940567 shutdown_connections 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- 172.21.15.116:0/2270940567 wait complete. 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- start start 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 Processor -- start 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518518aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x56518518b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281b01a0 con 0x56518517dca0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281afad0 con 0x565185186880 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4281acd30 con 0x56518517d380 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.478+0000 7fc4321a8740 10 client.0 _ll_register_callbacks cb 0x7fc4180c8040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42d362640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518518aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42d362640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518518aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54864/0 (socket says 172.21.15.116:54864) 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42d362640 1 -- 172.21.15.116:0/3273987411 learned_addr learned my addr 172.21.15.116:0/3273987411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x56518518b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42db63640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 -- 172.21.15.116:0/3273987411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 msgr2=0x56518518aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518518aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 -- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 msgr2=0x5651851fbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851fbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 -- 172.21.15.116:0/3273987411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651851fc300 con 0x56518517dca0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc42cb61640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x56518518b020 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fc4180cc8a0 tx=0x7fc4180d1a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164baf102c7664cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc410ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41809b890 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc418097f00 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.479+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc424015030 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc41809f930 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 10 client.22491 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5651851fd680 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc424015030 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc41809ace0 con 0x56518517dca0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 10 client.22491 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 10 client.22491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 20 client.22491 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5651851fddb0 0x5651852001d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 4 client.22491 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5651850e89f0 con 0x5651851fddb0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.480+0000 7fc42d362640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5651851fddb0 0x5651852001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.481+0000 7fc410ff9640 10 client.22491 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.481+0000 7fc42d362640 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5651851fddb0 0x5651852001d0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1cb99b3f4850221 server_cookie=2cc1d669a0c0a834 in_seq=0 out_seq=0 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.483+0000 7fc410ff9640 1 -- 172.21.15.116:0/3273987411 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5651850e89f0 con 0x5651851fddb0 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.483+0000 7fc410ff9640 10 client.22491 handle_command_reply: tid=0 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.483+0000 7fc4321a8740 1 client.22491 shutdown 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 msgr2=0x56518518b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x56518518b020 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fc4180cc8a0 tx=0x7fc4180d1a40 comp rx=0 tx=0).stop 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 shutdown_connections 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5651851fddb0 0x5651852001d0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565185186880 0x5651851fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56518517dca0 0x56518518b020 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 --2- 172.21.15.116:0/3273987411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56518517d380 0x56518518aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc410ff9640 0 client.22491 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 >> 172.21.15.116:0/3273987411 conn(0x565185177ab0 msgr2=0x5651851837f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 shutdown_connections 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 1 -- 172.21.15.116:0/3273987411 wait complete. 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc4321a8740 20 client.22491 trim_cache size 0 max 16384 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "9a62d5c6-8bb9-4181-b008-f5f35f7867a1", 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-19T02:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 msgr2=0x7fc4140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 0x7fc4140a4e30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc420005cc0 tx=0x7fc420005c50 comp rx=0 tx=0).stop 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 msgr2=0x7fc4281993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fc41800e9f0 tx=0x7fc41800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 shutdown_connections 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc4140a2970 0x7fc4140a4e30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4281066a0 0x7fc4281a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc428101050 0x7fc4281993b0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 --2- 172.21.15.116:0/617022288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc428100690 0x7fc428198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 >> 172.21.15.116:0/617022288 conn(0x7fc428077820 msgr2=0x7fc428102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.484+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 shutdown_connections 2024-03-19T02:52:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.485+0000 7fc42f5c6640 1 -- 172.21.15.116:0/617022288 wait complete. 2024-03-19T02:52:19.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:52:19.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:52:19.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 -- 172.21.15.116:0/2883807051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc738101060 msgr2=0x7fc73810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 --2- 172.21.15.116:0/2883807051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc738101060 0x7fc73810d570 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fc730009cd0 tx=0x7fc73002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 -- 172.21.15.116:0/2883807051 shutdown_connections 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 --2- 172.21.15.116:0/2883807051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc738101060 0x7fc73810d570 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 --2- 172.21.15.116:0/2883807051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 --2- 172.21.15.116:0/2883807051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7381066b0 0x7fc738106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 -- 172.21.15.116:0/2883807051 >> 172.21.15.116:0/2883807051 conn(0x7fc738076aa0 msgr2=0x7fc738076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 -- 172.21.15.116:0/2883807051 shutdown_connections 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.652+0000 7fc73e019640 1 -- 172.21.15.116:0/2883807051 wait complete. 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 Processor -- start 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 -- start start 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc738101060 0x7fc738197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc738110930 con 0x7fc7381066b0 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381107b0 con 0x7fc7381006a0 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc73e019640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc738110ab0 con 0x7fc738101060 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc737fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc738101060 0x7fc738197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38162/0 (socket says 172.21.15.116:38162) 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc737fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45056/0 (socket says 172.21.15.116:45056) 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.653+0000 7fc7377fe640 1 -- 172.21.15.116:0/876072349 learned_addr learned my addr 172.21.15.116:0/876072349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 -- 172.21.15.116:0/876072349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc738101060 msgr2=0x7fc738197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc738101060 0x7fc738197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 -- 172.21.15.116:0/876072349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 msgr2=0x7fc738196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 -- 172.21.15.116:0/876072349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73819bec0 con 0x7fc7381066b0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc7377fe640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc737fff640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc73002c630 tx=0x7fc730035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc734ff9640 1 -- 172.21.15.116:0/876072349 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730039040 con 0x7fc7381066b0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73819c150 con 0x7fc7381066b0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc734ff9640 1 -- 172.21.15.116:0/876072349 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc730044070 con 0x7fc7381066b0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc734ff9640 1 -- 172.21.15.116:0/876072349 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730004330 con 0x7fc7381066b0 2024-03-19T02:52:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.654+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7381a3a40 con 0x7fc7381066b0 2024-03-19T02:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc734ff9640 1 -- 172.21.15.116:0/876072349 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc7300044d0 con 0x7fc7381066b0 2024-03-19T02:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 Processor -- start 2024-03-19T02:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 -- start start 2024-03-19T02:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef700320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564cef658500 con 0x564cef6f5350 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564cef600be0 con 0x564cef6f4a30 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc740c0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564cef6545a0 con 0x564cef6fdf30 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc734ff9640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 0x7fc7200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45058/0 (socket says 172.21.15.116:45058) 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc7377fe640 1 -- 172.21.15.116:0/245721542 learned_addr learned my addr 172.21.15.116:0/245721542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc736ffd640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc737fff640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 0x7fc7200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.657+0000 7fc736ffd640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef700320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc734ff9640 1 -- 172.21.15.116:0/876072349 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc7300f1710 con 0x7fc7381066b0 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 -- 172.21.15.116:0/245721542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 msgr2=0x564cef700320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef700320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 -- 172.21.15.116:0/245721542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 msgr2=0x564cef6fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 -- 172.21.15.116:0/245721542 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cef622000 con 0x564cef6f4a30 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc7377fe640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc737fff640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 0x7fc7200a4e30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc7300bd7a0 tx=0x7fc73002c0f0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc736ffd640 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f4e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc72c005d00 tx=0x7fc72c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80f16b8b4fd386fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc71dffb640 1 -- 172.21.15.116:0/245721542 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004020 con 0x564cef6f4a30 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc71dffb640 1 -- 172.21.15.116:0/245721542 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc72c0054d0 con 0x564cef6f4a30 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.658+0000 7fc71dffb640 1 -- 172.21.15.116:0/245721542 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004a00 con 0x564cef6f4a30 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- 172.21.15.116:0/245721542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 msgr2=0x564cef6f4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f4e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc72c005d00 tx=0x7fc72c004730 comp rx=0 tx=0).stop 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- 172.21.15.116:0/245721542 shutdown_connections 2024-03-19T02:52:19.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef700320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f5350 0x564cef6fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 --2- 172.21.15.116:0/245721542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f4e10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- 172.21.15.116:0/245721542 >> 172.21.15.116:0/245721542 conn(0x564cef6ef160 msgr2=0x564cef6ef540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- 172.21.15.116:0/245721542 shutdown_connections 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- 172.21.15.116:0/245721542 wait complete. 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.659+0000 7fc740c0a740 1 -- start start 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 Processor -- start 2024-03-19T02:52:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6fb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564cef700cc0 con 0x564cef6fdf30 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564cef614160 con 0x564cef6f4a30 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564cef700e40 con 0x564cef6f5350 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc740c0a740 10 client.0 _ll_register_callbacks cb 0x564cef650a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc7377fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6fb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45072/0 (socket says 172.21.15.116:45072) 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 -- 172.21.15.116:0/718753928 learned_addr learned my addr 172.21.15.116:0/718753928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 -- 172.21.15.116:0/718753928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 msgr2=0x564cef6fb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6fb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 -- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 msgr2=0x564cef6faef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6faef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 -- 172.21.15.116:0/718753928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cef4367d0 con 0x564cef6fdf30 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc7377fe640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6fb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc736ffd640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00b470 tx=0x7fc72c004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd66a6f88720e0ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.660+0000 7fc716ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c010740 con 0x564cef6fdf30 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc72c0108e0 con 0x564cef6fdf30 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72c0054d0 con 0x564cef6fdf30 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc72c038050 con 0x564cef6fdf30 2024-03-19T02:52:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 10 client.22503 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564cef36c3f0 con 0x564cef6fdf30 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc72c0054d0 con 0x564cef6fdf30 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc72c028040 con 0x564cef6fdf30 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 10 client.22503 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 10 client.22503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.661+0000 7fc740c0a740 20 client.22503 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef710590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc740c0a740 4 client.22503 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cef665a40 con 0x564cef70e1a0 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc7377fe640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef710590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc716ffd640 10 client.22503 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.662+0000 7fc7377fe640 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef710590 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab1e87f0966619bc server_cookie=f9976feb240faebb in_seq=0 out_seq=0 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc716ffd640 1 -- 172.21.15.116:0/718753928 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cef665a40 con 0x564cef70e1a0 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc716ffd640 10 client.22503 handle_command_reply: tid=0 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 client.22503 shutdown 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 msgr2=0x564cef6fb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00b470 tx=0x7fc72c004530 comp rx=0 tx=0).stop 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 shutdown_connections 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef710590 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6fb970 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc716ffd640 0 client.22503 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6fb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 --2- 172.21.15.116:0/718753928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6faef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 >> 172.21.15.116:0/718753928 conn(0x564cef6ef160 msgr2=0x564cef6f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 shutdown_connections 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 1 -- 172.21.15.116:0/718753928 wait complete. 2024-03-19T02:52:19.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.665+0000 7fc740c0a740 20 client.22503 trim_cache size 0 max 16384 2024-03-19T02:52:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 Processor -- start 2024-03-19T02:52:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 -- start start 2024-03-19T02:52:19.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 0x564cef6f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef6f6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381aca80 con 0x564cef6f4a30 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381ae5c0 con 0x564cef6f5350 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc740c0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381ad150 con 0x564cef6fdf30 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.689+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45078/0 (socket says 172.21.15.116:45078) 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 -- 172.21.15.116:0/3294606516 learned_addr learned my addr 172.21.15.116:0/3294606516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc7377fe640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 0x564cef6f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef6f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 -- 172.21.15.116:0/3294606516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 msgr2=0x564cef6f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef6f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 -- 172.21.15.116:0/3294606516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 msgr2=0x564cef6f9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 0x564cef6f9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 -- 172.21.15.116:0/3294606516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cef32c0f0 con 0x564cef6f4a30 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc7377fe640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 0x564cef6f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc736ffd640 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fc72c003e70 tx=0x7fc72c00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f2a496585cfe8e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc71d7fa640 1 -- 172.21.15.116:0/3294606516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c003f70 con 0x564cef6f4a30 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc71d7fa640 1 -- 172.21.15.116:0/3294606516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc72c003aa0 con 0x564cef6f4a30 2024-03-19T02:52:19.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.690+0000 7fc71d7fa640 1 -- 172.21.15.116:0/3294606516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c005650 con 0x564cef6f4a30 2024-03-19T02:52:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- 172.21.15.116:0/3294606516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 msgr2=0x564cef6fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fc72c003e70 tx=0x7fc72c00bbb0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- 172.21.15.116:0/3294606516 shutdown_connections 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6fdf30 0x564cef6f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f5350 0x564cef6f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 --2- 172.21.15.116:0/3294606516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6f4a30 0x564cef6fd620 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- 172.21.15.116:0/3294606516 >> 172.21.15.116:0/3294606516 conn(0x564cef6ef160 msgr2=0x564cef6fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- 172.21.15.116:0/3294606516 shutdown_connections 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- 172.21.15.116:0/3294606516 wait complete. 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 -- start start 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.691+0000 7fc740c0a740 1 Processor -- start 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6f8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6f8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381adef0 con 0x564cef6fdf30 2024-03-19T02:52:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381ad820 con 0x564cef6f4a30 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7381aaa80 con 0x564cef6f5350 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc740c0a740 10 client.0 _ll_register_callbacks cb 0x7fc72800bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:38194/0 (socket says 172.21.15.116:38194) 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 -- 172.21.15.116:0/1013575073 learned_addr learned my addr 172.21.15.116:0/1013575073 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc7377fe640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 -- 172.21.15.116:0/1013575073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 msgr2=0x564cef6f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 -- 172.21.15.116:0/1013575073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 msgr2=0x564cef6f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.692+0000 7fc736ffd640 1 -- 172.21.15.116:0/1013575073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cef773390 con 0x564cef6f4a30 2024-03-19T02:52:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc736ffd640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc72c005d00 tx=0x7fc72c00b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf0ca634b2edd5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc716ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004460 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72c003aa0 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fc72c0035f0 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fc72c0053d0 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc740c0a740 10 client.32210 fetch_fsmap learned FSMap version 42 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564cef774640 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.693+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc72c003aa0 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fc72c017050 con 0x564cef6f4a30 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 10 client.32210 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 10 client.32210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 20 client.32210 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef776c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 4 client.32210 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564cef65fd90 con 0x564cef70e1a0 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc7377fe640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef776c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc7377fe640 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef776c10 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484696f903aa5fb6 server_cookie=32728acab2227514 in_seq=0 out_seq=0 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.694+0000 7fc716ffd640 10 client.32210 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc716ffd640 1 -- 172.21.15.116:0/1013575073 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x564cef65fd90 con 0x564cef70e1a0 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc716ffd640 10 client.32210 handle_command_reply: tid=0 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 client.32210 shutdown 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 msgr2=0x564cef6f7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc72c005d00 tx=0x7fc72c00b710 comp rx=0 tx=0).stop 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 shutdown_connections 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x564cef70e1a0 0x564cef776c10 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cef6fdf30 0x564cef6f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564cef6f5350 0x564cef6f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 --2- 172.21.15.116:0/1013575073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564cef6f4a30 0x564cef6f7be0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc716ffd640 0 client.32210 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 >> 172.21.15.116:0/1013575073 conn(0x564cef6ef160 msgr2=0x564cef6ed950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 shutdown_connections 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 1 -- 172.21.15.116:0/1013575073 wait complete. 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.696+0000 7fc740c0a740 20 client.32210 trim_cache size 0 max 16384 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (447 inodes in the stack)", 2024-03-19T02:52:19.702 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: "9a62d5c6-8bb9-4181-b008-f5f35f7867a1": { 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "9a62d5c6-8bb9-4181-b008-f5f35f7867a1", 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:19.703 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 msgr2=0x7fc7200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 0x7fc7200a4e30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc7300bd7a0 tx=0x7fc73002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 msgr2=0x7fc73819b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fc73002c630 tx=0x7fc730035d60 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 shutdown_connections 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc7200a2970 0x7fc7200a4e30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7381066b0 0x7fc73819b790 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc738101060 0x7fc738197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 --2- 172.21.15.116:0/876072349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7381006a0 0x7fc738196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 >> 172.21.15.116:0/876072349 conn(0x7fc738076aa0 msgr2=0x7fc7380fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 shutdown_connections 2024-03-19T02:52:19.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:19.697+0000 7fc73e019640 1 -- 172.21.15.116:0/876072349 wait complete. 2024-03-19T02:52:19.709 INFO:tasks.cephfs.filesystem:scrub status for tag:9a62d5c6-8bb9-4181-b008-f5f35f7867a1 - {'path': '/', 'tag': '9a62d5c6-8bb9-4181-b008-f5f35f7867a1', 'options': 'recursive,force'} 2024-03-19T02:52:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:19 smithi154 ceph-mon[35693]: scrub queued for path: / 2024-03-19T02:52:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:20.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:19 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:20.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:19 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:19 smithi116 ceph-mon[32263]: scrub queued for path: / 2024-03-19T02:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:19 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:19 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75076 5.63 MB/sec execute 582 sec latency 2513.888 ms 2024-03-19T02:52:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:19 smithi162 ceph-mon[35761]: scrub queued for path: / 2024-03-19T02:52:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:19 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:20.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:19 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.992 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: pgmap v776: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:20.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:20.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:20.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:20.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: active paths [/] 2024-03-19T02:52:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:20 smithi154 ceph-mon[35693]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: pgmap v776: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: active paths [/] 2024-03-19T02:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:20 smithi116 ceph-mon[32263]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75183 5.64 MB/sec execute 583 sec latency 796.069 ms 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: pgmap v776: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 192 op/s 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:21.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:20 smithi162 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-19T02:52:22.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75284 5.64 MB/sec execute 584 sec latency 144.525 ms 2024-03-19T02:52:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:22 smithi154 ceph-mon[35693]: pgmap v777: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-19T02:52:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:22 smithi116 ceph-mon[32263]: pgmap v777: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-19T02:52:23.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75385 5.64 MB/sec execute 585 sec latency 142.316 ms 2024-03-19T02:52:23.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:22 smithi162 ceph-mon[35761]: pgmap v777: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-19T02:52:24.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75507 5.64 MB/sec execute 586 sec latency 143.432 ms 2024-03-19T02:52:25.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:24 smithi154 ceph-mon[35693]: pgmap v778: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-19T02:52:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:24 smithi116 ceph-mon[32263]: pgmap v778: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-19T02:52:25.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75622 5.64 MB/sec execute 587 sec latency 172.529 ms 2024-03-19T02:52:25.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:24 smithi162 ceph-mon[35761]: pgmap v778: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-19T02:52:26.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75757 5.66 MB/sec execute 588 sec latency 156.086 ms 2024-03-19T02:52:27.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:26 smithi154 ceph-mon[35693]: pgmap v779: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 265 op/s 2024-03-19T02:52:27.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:26 smithi116 ceph-mon[32263]: pgmap v779: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 265 op/s 2024-03-19T02:52:27.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 75909 5.67 MB/sec execute 589 sec latency 149.896 ms 2024-03-19T02:52:27.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:26 smithi162 ceph-mon[35761]: pgmap v779: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 265 op/s 2024-03-19T02:52:28.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76054 5.67 MB/sec execute 590 sec latency 172.014 ms 2024-03-19T02:52:29.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:28 smithi154 ceph-mon[35693]: pgmap v780: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:28 smithi116 ceph-mon[32263]: pgmap v780: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:29.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76223 5.68 MB/sec execute 591 sec latency 132.914 ms 2024-03-19T02:52:29.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:28 smithi162 ceph-mon[35761]: pgmap v780: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:30.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:30 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:30.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:30 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:30.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76284 5.68 MB/sec execute 592 sec latency 215.115 ms 2024-03-19T02:52:30.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:30.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:30 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:31 smithi154 ceph-mon[35693]: pgmap v781: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:31.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:31.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:31 smithi154 ceph-mon[35693]: scrub summary: idle 2024-03-19T02:52:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:31 smithi116 ceph-mon[32263]: pgmap v781: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:31.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:31 smithi116 ceph-mon[32263]: scrub summary: idle 2024-03-19T02:52:31.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76350 5.68 MB/sec execute 593 sec latency 146.114 ms 2024-03-19T02:52:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:31 smithi162 ceph-mon[35761]: pgmap v781: 129 pgs: 129 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 182 op/s 2024-03-19T02:52:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:31.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:31 smithi162 ceph-mon[35761]: scrub summary: idle 2024-03-19T02:52:32.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76399 5.67 MB/sec execute 594 sec latency 700.488 ms 2024-03-19T02:52:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:33 smithi116 ceph-mon[32263]: pgmap v782: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-03-19T02:52:33.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76429 5.67 MB/sec execute 595 sec latency 1700.514 ms 2024-03-19T02:52:33.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:33 smithi162 ceph-mon[35761]: pgmap v782: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-03-19T02:52:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:33 smithi154 ceph-mon[35693]: pgmap v782: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 239 op/s 2024-03-19T02:52:34.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76480 5.66 MB/sec execute 596 sec latency 2700.539 ms 2024-03-19T02:52:35.341 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76588 5.68 MB/sec execute 597 sec latency 3700.567 ms 2024-03-19T02:52:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:35 smithi162 ceph-mon[35761]: pgmap v783: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-03-19T02:52:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:35 smithi154 ceph-mon[35693]: pgmap v783: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-03-19T02:52:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:35 smithi116 ceph-mon[32263]: pgmap v783: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 134 op/s 2024-03-19T02:52:36.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76670 5.68 MB/sec execute 598 sec latency 4213.831 ms 2024-03-19T02:52:36.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:36 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:52:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:36 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:52:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:36 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:52:37.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 76725 5.67 MB/sec execute 599 sec latency 955.683 ms 2024-03-19T02:52:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:37 smithi162 ceph-mon[35761]: pgmap v784: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 180 op/s 2024-03-19T02:52:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:37 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:37 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:37 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:52:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:37 smithi154 ceph-mon[35693]: pgmap v784: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 180 op/s 2024-03-19T02:52:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:37 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:37 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:37 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:52:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:37 smithi116 ceph-mon[32263]: pgmap v784: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 180 op/s 2024-03-19T02:52:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:37 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:37 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:37.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:37 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:52:38.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 cleanup 600 sec 2024-03-19T02:52:39.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 cleanup 601 sec 2024-03-19T02:52:39.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:39 smithi162 ceph-mon[35761]: pgmap v785: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:39.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:39 smithi154 ceph-mon[35693]: pgmap v785: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:39.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:39 smithi116 ceph-mon[32263]: pgmap v785: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:40.342 INFO:tasks.workunit.client.0.smithi116.stdout: 10 cleanup 602 sec 2024-03-19T02:52:41.342 INFO:tasks.workunit.client.0.smithi116.stdout: 3 cleanup 603 sec 2024-03-19T02:52:41.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:41 smithi162 ceph-mon[35761]: pgmap v786: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:41.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:41 smithi154 ceph-mon[35693]: pgmap v786: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:41.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:41 smithi116 ceph-mon[32263]: pgmap v786: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 102 op/s 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: 0 cleanup 604 sec 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: Operation Count AvgLat MaxLat 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: ---------------------------------------- 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: NTCreateX 108410 10.993 4364.050 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: Close 79893 0.082 67.778 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: Rename 4629 50.367 4759.551 2024-03-19T02:52:41.910 INFO:tasks.workunit.client.0.smithi116.stdout: Unlink 21660 8.712 142.535 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Qpathinfo 98327 7.787 721.472 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Qfileinfo 17257 0.141 15.819 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Qfsinfo 18054 0.446 84.460 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Sfileinfo 8881 1.106 84.853 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Find 38084 91.534 1049.875 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: WriteX 53833 0.350 59.176 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: ReadX 171269 0.005 1.398 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: LockX 356 4.610 54.456 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: UnlockX 356 4.557 53.161 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: Flush 7604 11.282 134.883 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-19T02:52:41.911 INFO:tasks.workunit.client.0.smithi116.stdout:Throughput 5.66791 MB/sec 10 clients 10 procs max_latency=4759.561 ms 2024-03-19T02:52:41.912 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:52:41.912 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:52:43.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:43 smithi162 ceph-mon[35761]: pgmap v787: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.5 MiB/s wr, 139 op/s 2024-03-19T02:52:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:43 smithi154 ceph-mon[35693]: pgmap v787: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.5 MiB/s wr, 139 op/s 2024-03-19T02:52:43.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:43 smithi116 ceph-mon[32263]: pgmap v787: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.5 MiB/s wr, 139 op/s 2024-03-19T02:52:43.675 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-19T02:52:43.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-19T02:52:44.042 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:44.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T02:52:44.077 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T02:52:44.077 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-19T02:52:44.089 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-19T02:52:44.089 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:52:44.089 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:52:44.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-19T02:52:44.164 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-19T02:52:44.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1099287493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48101330 msgr2=0x7f0b4810d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1099287493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48101330 0x7f0b4810d500 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009ca0 tx=0x7f0b3802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1099287493 shutdown_connections 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1099287493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48101330 0x7f0b4810d500 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1099287493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48100970 0x7f0b48100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1099287493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48105680 0x7f0b48105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1099287493 >> 172.21.15.116:0/1099287493 conn(0x7f0b480774b0 msgr2=0x7f0b480778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1099287493 shutdown_connections 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.340+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1099287493 wait complete. 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 Processor -- start 2024-03-19T02:52:44.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 -- start start 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 0x7f0b48197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48101330 0x7f0b48198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48111a40 con 0x7f0b48105680 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0b481118c0 con 0x7f0b48100970 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4fc63640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0b48111bc0 con 0x7f0b48101330 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35732/0 (socket says 172.21.15.116:35732) 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4d9ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 0x7f0b48197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 -- 172.21.15.116:0/1668985889 learned_addr learned my addr 172.21.15.116:0/1668985889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4d1fe640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48101330 0x7f0b48198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 -- 172.21.15.116:0/1668985889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48101330 msgr2=0x7f0b48198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48101330 0x7f0b48198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 -- 172.21.15.116:0/1668985889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 msgr2=0x7f0b48197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 0x7f0b48197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 -- 172.21.15.116:0/1668985889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4819ce30 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4d9ff640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 0x7f0b48197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.341+0000 7f0b4e200640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f0b3802c700 tx=0x7f0b38035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.342+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38039040 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.342+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4819d0c0 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.342+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b38044070 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.342+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b380042a0 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.342+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b481a49b0 con 0x7f0b48105680 2024-03-19T02:52:44.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.343+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4804f580 con 0x7f0b48105680 2024-03-19T02:52:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.344+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0b38003970 con 0x7f0b48105680 2024-03-19T02:52:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.344+0000 7f0b46ffd640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 0x7f0b300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.344+0000 7f0b4d9ff640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 0x7f0b300a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.345+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f0b380f14f0 con 0x7f0b48105680 2024-03-19T02:52:44.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.345+0000 7f0b4d9ff640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 0x7f0b300a4ed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009fc0 tx=0x7f0b40005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.349+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b380bdf80 con 0x7f0b48105680 2024-03-19T02:52:44.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:44 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:44.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:44 smithi162 ceph-mon[35761]: pgmap v788: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 82 op/s 2024-03-19T02:52:44.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:44 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:44.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:44 smithi154 ceph-mon[35693]: pgmap v788: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 82 op/s 2024-03-19T02:52:44.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:44 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:44.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:44 smithi116 ceph-mon[32263]: pgmap v788: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 82 op/s 2024-03-19T02:52:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.545+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f0b48061c60 con 0x7f0b300a2a10 2024-03-19T02:52:44.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.556+0000 7f0b46ffd640 1 -- 172.21.15.116:0/1668985889 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f0b48061c60 con 0x7f0b300a2a10 2024-03-19T02:52:44.557 INFO:teuthology.orchestra.run.smithi116.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-19T00:00:00", "created": "2024-03-19T02:27:49", "first": "2024-03-19T02:28:00", "last": "2024-03-19T02:52:00", "last_pruned": "2024-03-19T02:52:00", "created_count": 25, "pruned_count": 19, "active": true} 2024-03-19T02:52:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 msgr2=0x7f0b300a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 0x7f0b300a4ed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009fc0 tx=0x7f0b40005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 msgr2=0x7f0b4819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f0b3802c700 tx=0x7f0b38035d60 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 shutdown_connections 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0b300a2a10 0x7f0b300a4ed0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b48105680 0x7f0b4819c700 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0b48101330 0x7f0b48198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 --2- 172.21.15.116:0/1668985889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0b48100970 0x7f0b48197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 >> 172.21.15.116:0/1668985889 conn(0x7f0b480774b0 msgr2=0x7f0b480fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 shutdown_connections 2024-03-19T02:52:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.558+0000 7f0b4fc63640 1 -- 172.21.15.116:0/1668985889 wait complete. 2024-03-19T02:52:44.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/766470937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac1039d0 msgr2=0x7fd3ac103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/766470937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac103e50 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009930 tx=0x7fd3a002bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/766470937 shutdown_connections 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/766470937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac104390 0x7fd3ac1108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/766470937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac103e50 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/766470937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 0x7fd3ac109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.673+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/766470937 >> 172.21.15.116:0/766470937 conn(0x7fd3ac0ff4a0 msgr2=0x7fd3ac1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/766470937 shutdown_connections 2024-03-19T02:52:44.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/766470937 wait complete. 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 Processor -- start 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- start start 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 0x7fd3ac19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac113ca0 con 0x7fd3ac104390 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac113b20 con 0x7fd3ac1099e0 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.674+0000 7fd3b0e4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac113e20 con 0x7fd3ac1039d0 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3aa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35740/0 (socket says 172.21.15.116:35740) 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 -- 172.21.15.116:0/3852163407 learned_addr learned my addr 172.21.15.116:0/3852163407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:44.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3aad9d640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 0x7fd3ac19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 -- 172.21.15.116:0/3852163407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac1039d0 msgr2=0x7fd3ac19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 -- 172.21.15.116:0/3852163407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 msgr2=0x7fd3ac19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 0x7fd3ac19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 -- 172.21.15.116:0/3852163407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac19f190 con 0x7fd3ac104390 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3a9d9b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009230 tx=0x7fd3a002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0043020 con 0x7fd3ac104390 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac19f420 con 0x7fd3ac104390 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3a002ca70 con 0x7fd3ac104390 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac1a6d10 con 0x7fd3ac104390 2024-03-19T02:52:44.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.675+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00039f0 con 0x7fd3ac104390 2024-03-19T02:52:44.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.676+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac04f580 con 0x7fd3ac104390 2024-03-19T02:52:44.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.677+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd3a0046020 con 0x7fd3ac104390 2024-03-19T02:52:44.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.678+0000 7fd39b7fe640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 0x7fd3940a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.678+0000 7fd3aa59c640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 0x7fd3940a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.678+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fd3a00f14b0 con 0x7fd3ac104390 2024-03-19T02:52:44.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.679+0000 7fd3aa59c640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 0x7fd3940a4ed0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005fd0 tx=0x7fd39c009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.682+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3a004d020 con 0x7fd3ac104390 2024-03-19T02:52:44.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.884+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 --> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd3ac061c60 con 0x7fd3940a2a10 2024-03-19T02:52:44.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.894+0000 7fd39b7fe640 1 -- 172.21.15.116:0/3852163407 <== mgr.25556 v2:172.21.15.116:6800/2695100500 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd3ac061c60 con 0x7fd3940a2a10 2024-03-19T02:52:44.894 INFO:teuthology.orchestra.run.smithi116.stdout:/ 1m 6m3h 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 msgr2=0x7fd3940a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 0x7fd3940a4ed0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005fd0 tx=0x7fd39c009500 comp rx=0 tx=0).stop 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 msgr2=0x7fd3ac19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009230 tx=0x7fd3a002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 shutdown_connections 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3940a2a10 0x7fd3940a4ed0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3ac1099e0 0x7fd3ac19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3ac104390 0x7fd3ac19a6e0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 --2- 172.21.15.116:0/3852163407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd3ac1039d0 0x7fd3ac19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 >> 172.21.15.116:0/3852163407 conn(0x7fd3ac0ff4a0 msgr2=0x7fd3ac0ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 shutdown_connections 2024-03-19T02:52:44.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:44.896+0000 7fd3b0e4b640 1 -- 172.21.15.116:0/3852163407 wait complete. 2024-03-19T02:52:44.907 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-19T02:52:44.917 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-19T02:52:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:45 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:45 smithi116 ceph-mon[32263]: from='client.22515 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:45 smithi116 ceph-mon[32263]: from='client.22521 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:45 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:45 smithi162 ceph-mon[35761]: from='client.22515 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:45 smithi162 ceph-mon[35761]: from='client.22521 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:45 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:45 smithi154 ceph-mon[35693]: from='client.22515 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:45 smithi154 ceph-mon[35693]: from='client.22521 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:46.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:46 smithi162 ceph-mon[35761]: pgmap v789: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-03-19T02:52:46.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:46 smithi154 ceph-mon[35693]: pgmap v789: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-03-19T02:52:46.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:46 smithi116 ceph-mon[32263]: pgmap v789: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.8 MiB/s wr, 232 op/s 2024-03-19T02:52:47.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:47 smithi162 ceph-mon[35761]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:47.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:47 smithi154 ceph-mon[35693]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:47.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:47 smithi116 ceph-mon[32263]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T02:52:48.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:48 smithi162 ceph-mon[35761]: pgmap v790: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:48 smithi154 ceph-mon[35693]: pgmap v790: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:48 smithi116 ceph-mon[32263]: pgmap v790: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:49.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:52:49.710 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.826+0000 7fcdde5b3640 1 -- 172.21.15.116:0/4129985016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 msgr2=0x7fcdd8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.826+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/4129985016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd8106a80 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009930 tx=0x7fcdc802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 -- 172.21.15.116:0/4129985016 shutdown_connections 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/4129985016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8101050 0x7fcdd810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/4129985016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8100690 0x7fcdd8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/4129985016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd8106a80 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 -- 172.21.15.116:0/4129985016 >> 172.21.15.116:0/4129985016 conn(0x7fcdd8077820 msgr2=0x7fcdd8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 -- 172.21.15.116:0/4129985016 shutdown_connections 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 -- 172.21.15.116:0/4129985016 wait complete. 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 Processor -- start 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.827+0000 7fcdde5b3640 1 -- start start 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8100690 0x7fcdd8198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd81a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8110960 con 0x7fcdd81066a0 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81107e0 con 0x7fcdd8100690 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8110ae0 con 0x7fcdd8101050 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcddcb50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd81a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8100690 0x7fcdd8198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39474/0 (socket says 172.21.15.116:39474) 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1257683040 learned_addr learned my addr 172.21.15.116:0/1257683040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1257683040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8100690 msgr2=0x7fcdd8198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8100690 0x7fcdd8198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 msgr2=0x7fcdd81a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd81a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1257683040 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd81a43e0 con 0x7fcdd8101050 2024-03-19T02:52:49.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcddcb50640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd81a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00e9f0 tx=0x7fcdcc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd57fa640 1 -- 172.21.15.116:0/1257683040 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00cd80 con 0x7fcdd8101050 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd81a45b0 con 0x7fcdd8101050 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.828+0000 7fcdd57fa640 1 -- 172.21.15.116:0/1257683040 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc0108e0 con 0x7fcdd8101050 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.829+0000 7fcdd57fa640 1 -- 172.21.15.116:0/1257683040 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc018920 con 0x7fcdd8101050 2024-03-19T02:52:49.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.829+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd81a4b50 con 0x7fcdd8101050 2024-03-19T02:52:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcdd57fa640 1 -- 172.21.15.116:0/1257683040 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcdcc010a90 con 0x7fcdd8101050 2024-03-19T02:52:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 Processor -- start 2024-03-19T02:52:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 -- start start 2024-03-19T02:52:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcdd57fa640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 0x7fcdc00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45f0f30 0x556cb45f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cb454b500 con 0x556cb45f0f30 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556cb44f3be0 con 0x556cb45e7a30 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.831+0000 7fcde1197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556cb45475a0 con 0x556cb45e8350 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcddcb50640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 0x7fcdc00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39488/0 (socket says 172.21.15.116:39488) 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45e7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57216/0 (socket says 172.21.15.116:57216) 2024-03-19T02:52:49.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1437080996 learned_addr learned my addr 172.21.15.116:0/1437080996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcddcb50640 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45f0f30 0x556cb45f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd57fa640 1 -- 172.21.15.116:0/1257683040 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fcdcc0cf520 con 0x7fcdd8101050 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1437080996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 msgr2=0x556cb45e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45e7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1437080996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45f0f30 msgr2=0x556cb45f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45f0f30 0x556cb45f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1437080996 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb4515000 con 0x556cb45e8350 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcddcb50640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 0x7fcdc00a4e30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009fd0 tx=0x7fcdd00116b0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c6a0 tx=0x7fcdcc09b1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db06ac13b9fee7b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdbe7fc640 1 -- 172.21.15.116:0/1437080996 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0ae3c0 con 0x556cb45e8350 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdbe7fc640 1 -- 172.21.15.116:0/1437080996 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc0adfb0 con 0x556cb45e8350 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.832+0000 7fcdbe7fc640 1 -- 172.21.15.116:0/1437080996 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc010de0 con 0x556cb45e8350 2024-03-19T02:52:49.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- 172.21.15.116:0/1437080996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 msgr2=0x556cb45f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c6a0 tx=0x7fcdcc09b1f0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- 172.21.15.116:0/1437080996 shutdown_connections 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45f0f30 0x556cb45f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb45f09f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 --2- 172.21.15.116:0/1437080996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- 172.21.15.116:0/1437080996 >> 172.21.15.116:0/1437080996 conn(0x556cb45e2160 msgr2=0x556cb45e2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- 172.21.15.116:0/1437080996 shutdown_connections 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- 172.21.15.116:0/1437080996 wait complete. 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.833+0000 7fcde1197740 1 -- start start 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 Processor -- start 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45ede20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb45ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cb45f3cc0 con 0x556cb45e8350 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556cb4507160 con 0x556cb45e7a30 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556cb45f3e40 con 0x556cb45f0f30 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcde1197740 10 client.0 _ll_register_callbacks cb 0x556cb4543a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40448/0 (socket says 172.21.15.116:40448) 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3583848328 learned_addr learned my addr 172.21.15.116:0/3583848328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcddcb50640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb45ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3583848328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 msgr2=0x556cb45ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb45ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 msgr2=0x556cb45ede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45ede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.834+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3583848328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb43297d0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc09b590 tx=0x7fcdcc09b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da44812a25e9862a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0ac020 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fcdcc097f00 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc0adfb0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc0ae1e0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 10 client.22533 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556cb425f3f0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdcc0adfb0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fcdcc0cf1b0 con 0x556cb45e8350 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 10 client.22533 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 10 client.22533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:49.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 20 client.22533 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb4600f90 0x556cb4603380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 4 client.22533 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cb4558a40 con 0x556cb4600f90 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.835+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb4600f90 0x556cb4603380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.836+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb4600f90 0x556cb4603380 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf3fd1e990e1541 server_cookie=16fb0bfa9bb2ff22 in_seq=0 out_seq=0 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.836+0000 7fcdb77fe640 10 client.22533 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3583848328 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556cb4558a40 con 0x556cb4600f90 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcdb77fe640 10 client.22533 handle_command_reply: tid=0 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 client.22533 shutdown 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 msgr2=0x556cb45ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc09b590 tx=0x7fcdcc09b890 comp rx=0 tx=0).stop 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 shutdown_connections 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb4600f90 0x556cb4603380 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb45ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb45ee360 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 --2- 172.21.15.116:0/3583848328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb45ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcdb77fe640 0 client.22533 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 >> 172.21.15.116:0/3583848328 conn(0x556cb45e2160 msgr2=0x556cb45e3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 shutdown_connections 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 1 -- 172.21.15.116:0/3583848328 wait complete. 2024-03-19T02:52:49.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.837+0000 7fcde1197740 20 client.22533 trim_cache size 0 max 16384 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 Processor -- start 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 -- start start 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb4606790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 0x556cb45fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81aed80 con 0x556cb45e7a30 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81b0950 con 0x556cb45f0f30 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcde1197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81af480 con 0x556cb45e8350 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40464/0 (socket says 172.21.15.116:40464) 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 -- 172.21.15.116:0/2756104244 learned_addr learned my addr 172.21.15.116:0/2756104244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb4606790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcddcb50640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 0x556cb45fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 -- 172.21.15.116:0/2756104244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 msgr2=0x556cb4606790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb4606790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 -- 172.21.15.116:0/2756104244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 msgr2=0x556cb45fc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 0x556cb45fc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 -- 172.21.15.116:0/2756104244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb421f0f0 con 0x556cb45e7a30 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcddcb50640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 0x556cb45fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005ab0 tx=0x7fcdc8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd815752171596fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdbdffb640 1 -- 172.21.15.116:0/2756104244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8049050 con 0x556cb45e7a30 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdbdffb640 1 -- 172.21.15.116:0/2756104244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdc804bab0 con 0x556cb45e7a30 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.855+0000 7fcdbdffb640 1 -- 172.21.15.116:0/2756104244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc804b2f0 con 0x556cb45e7a30 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- 172.21.15.116:0/2756104244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 msgr2=0x556cb45f0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005ab0 tx=0x7fcdc8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- 172.21.15.116:0/2756104244 shutdown_connections 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45f0f30 0x556cb45fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45e8350 0x556cb4606790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 --2- 172.21.15.116:0/2756104244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e7a30 0x556cb45f0550 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- 172.21.15.116:0/2756104244 >> 172.21.15.116:0/2756104244 conn(0x556cb45e2160 msgr2=0x556cb45fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- 172.21.15.116:0/2756104244 shutdown_connections 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- 172.21.15.116:0/2756104244 wait complete. 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.856+0000 7fcde1197740 1 -- start start 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 Processor -- start 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb465f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb4663fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81b0280 con 0x556cb45e8350 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81afb80 con 0x556cb45e7a30 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81acd80 con 0x556cb45f0f30 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcde1197740 10 client.0 _ll_register_callbacks cb 0x7fcdcc09b3d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:49.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40480/0 (socket says 172.21.15.116:40480) 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1649617049 learned_addr learned my addr 172.21.15.116:0/1649617049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb465f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcddcb50640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb4663fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1649617049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 msgr2=0x556cb4663fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb4663fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 msgr2=0x556cb465f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb465f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 -- 172.21.15.116:0/1649617049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb46645e0 con 0x556cb45e8350 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb465f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc097cf0 tx=0x7fcdcc098130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d00bd5d431efb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.857+0000 7fcdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:49.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0ac030 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fcdcc0afbc0 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdc804bab0 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc099c50 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 10 client.22539 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556cb466b960 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdc804bab0 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fcdcc09ace0 con 0x556cb45e8350 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 10 client.22539 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 10 client.22539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 20 client.22539 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb466bf10 0x556cb466e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 4 client.22539 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556cb4552d90 con 0x556cb466bf10 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.858+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb466bf10 0x556cb466e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcdb77fe640 10 client.22539 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb466bf10 0x556cb466e330 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e759e619e0d663b server_cookie=e455cdd969274813 in_seq=0 out_seq=0 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcdb77fe640 1 -- 172.21.15.116:0/1649617049 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556cb4552d90 con 0x556cb466bf10 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcdb77fe640 10 client.22539 handle_command_reply: tid=0 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 client.22539 shutdown 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 msgr2=0x556cb465f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc097cf0 tx=0x7fcdcc098130 comp rx=0 tx=0).stop 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 shutdown_connections 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x556cb466bf10 0x556cb466e330 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556cb45f0f30 0x556cb4663fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cb45e8350 0x556cb465f610 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 --2- 172.21.15.116:0/1649617049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556cb45e7a30 0x556cb465f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcdb77fe640 0 client.22539 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 >> 172.21.15.116:0/1649617049 conn(0x556cb45e2160 msgr2=0x556cb45fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 shutdown_connections 2024-03-19T02:52:49.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 1 -- 172.21.15.116:0/1649617049 wait complete. 2024-03-19T02:52:49.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.859+0000 7fcde1197740 20 client.22539 trim_cache size 0 max 16384 2024-03-19T02:52:49.869 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:49.869 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-19T02:52:49.869 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-19T02:52:49.869 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:49.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 msgr2=0x7fcdc00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 0x7fcdc00a4e30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009fd0 tx=0x7fcdd00116b0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 msgr2=0x7fcdd8199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00e9f0 tx=0x7fcdcc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 shutdown_connections 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcdc00a2970 0x7fcdc00a4e30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd81066a0 0x7fcdd81a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdd8101050 0x7fcdd8199360 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 --2- 172.21.15.116:0/1257683040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdd8100690 0x7fcdd8198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 >> 172.21.15.116:0/1257683040 conn(0x7fcdd8077820 msgr2=0x7fcdd8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 shutdown_connections 2024-03-19T02:52:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.860+0000 7fcdde5b3640 1 -- 172.21.15.116:0/1257683040 wait complete. 2024-03-19T02:52:49.872 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:52:49.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:52:49.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 -- 172.21.15.116:0/756135394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d50102a90 msgr2=0x7f5d50102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 --2- 172.21.15.116:0/756135394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d50102a90 0x7f5d50102e70 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009900 tx=0x7f5d4802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 -- 172.21.15.116:0/756135394 shutdown_connections 2024-03-19T02:52:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 --2- 172.21.15.116:0/756135394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d5010be60 0x7f5d5010e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 --2- 172.21.15.116:0/756135394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d501033b0 0x7f5d5010b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 --2- 172.21.15.116:0/756135394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d50102a90 0x7f5d50102e70 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 -- 172.21.15.116:0/756135394 >> 172.21.15.116:0/756135394 conn(0x7f5d50077820 msgr2=0x7f5d50077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 -- 172.21.15.116:0/756135394 shutdown_connections 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.973+0000 7f5d552fe640 1 -- 172.21.15.116:0/756135394 wait complete. 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 Processor -- start 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 -- start start 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50102a90 0x7f5d501980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5010be60 0x7f5d5019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50111b10 con 0x7f5d501033b0 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50111990 con 0x7f5d5010be60 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d552fe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50111c90 con 0x7f5d50102a90 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.974+0000 7f5d4e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40508/0 (socket says 172.21.15.116:40508) 2024-03-19T02:52:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50102a90 0x7f5d501980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 -- 172.21.15.116:0/4250801201 learned_addr learned my addr 172.21.15.116:0/4250801201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4f7fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5010be60 0x7f5d5019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 -- 172.21.15.116:0/4250801201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50102a90 msgr2=0x7f5d501980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50102a90 0x7f5d501980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 -- 172.21.15.116:0/4250801201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5010be60 msgr2=0x7f5d5019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5010be60 0x7f5d5019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 -- 172.21.15.116:0/4250801201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5019d0a0 con 0x7f5d501033b0 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d4e7fc640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400ebc0 tx=0x7f5d4400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44010630 con 0x7f5d501033b0 2024-03-19T02:52:49.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d44010c80 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44018b30 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5019d390 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.975+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d501a4c20 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.976+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d50068610 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.977+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5d44018cd0 con 0x7f5d501033b0 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.978+0000 7f5d37fff640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 0x7f5d380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.978+0000 7f5d4effd640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 0x7f5d380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.978+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f5d440cf7e0 con 0x7f5d501033b0 2024-03-19T02:52:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.978+0000 7f5d4effd640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 0x7f5d380a4e30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5d48006010 tx=0x7f5d48005e40 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:49.982+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d4409bb10 con 0x7f5d501033b0 2024-03-19T02:52:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.223+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d50106f50 con 0x7f5d501033b0 2024-03-19T02:52:50.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.224+0000 7f5d37fff640 1 -- 172.21.15.116:0/4250801201 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11954 (secure 0 0 0) 0x7f5d4409fbb0 con 0x7f5d501033b0 2024-03-19T02:52:50.225 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:52:50.225 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:52:45.840939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:52:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 45 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.225+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 msgr2=0x7f5d380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.225+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 0x7f5d380a4e30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5d48006010 tx=0x7f5d48005e40 comp rx=0 tx=0).stop 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 msgr2=0x7f5d501985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400ebc0 tx=0x7f5d4400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 shutdown_connections 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5d380a2970 0x7f5d380a4e30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5010be60 0x7f5d5019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d501033b0 0x7f5d501985f0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 --2- 172.21.15.116:0/4250801201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50102a90 0x7f5d501980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 >> 172.21.15.116:0/4250801201 conn(0x7f5d50077820 msgr2=0x7f5d500feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 shutdown_connections 2024-03-19T02:52:50.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.226+0000 7f5d552fe640 1 -- 172.21.15.116:0/4250801201 wait complete. 2024-03-19T02:52:50.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2423045562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf381006a0 msgr2=0x7fdf38100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2423045562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf381006a0 0x7fdf38100b20 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009930 tx=0x7fdf2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2423045562 shutdown_connections 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2423045562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf38101060 0x7fdf3810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2423045562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf381006a0 0x7fdf38100b20 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2423045562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381066b0 0x7fdf38106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2423045562 >> 172.21.15.116:0/2423045562 conn(0x7fdf380778d0 msgr2=0x7fdf38077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2423045562 shutdown_connections 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.341+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2423045562 wait complete. 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 Processor -- start 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 -- start start 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381006a0 0x7fdf38198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf381066b0 0x7fdf381a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf38110970 con 0x7fdf38101060 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381107f0 con 0x7fdf381006a0 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3eeb8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf38110af0 con 0x7fdf381066b0 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381006a0 0x7fdf38198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40524/0 (socket says 172.21.15.116:40524) 2024-03-19T02:52:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf37fff640 1 -- 172.21.15.116:0/2221836900 learned_addr learned my addr 172.21.15.116:0/2221836900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf3d455640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf381066b0 0x7fdf381a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf37fff640 1 -- 172.21.15.116:0/2221836900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf381066b0 msgr2=0x7fdf381a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.342+0000 7fdf37fff640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf381066b0 0x7fdf381a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf37fff640 1 -- 172.21.15.116:0/2221836900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381006a0 msgr2=0x7fdf38198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf37fff640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381006a0 0x7fdf38198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf37fff640 1 -- 172.21.15.116:0/2221836900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf381a43f0 con 0x7fdf38101060 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf37fff640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009230 tx=0x7fdf2802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf35ffb640 1 -- 172.21.15.116:0/2221836900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28043020 con 0x7fdf38101060 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf35ffb640 1 -- 172.21.15.116:0/2221836900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf2802ca70 con 0x7fdf38101060 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf381a45c0 con 0x7fdf38101060 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf35ffb640 1 -- 172.21.15.116:0/2221836900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28003a50 con 0x7fdf38101060 2024-03-19T02:52:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.343+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf381a4a80 con 0x7fdf38101060 2024-03-19T02:52:50.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf35ffb640 1 -- 172.21.15.116:0/2221836900 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fdf28046020 con 0x7fdf38101060 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 Processor -- start 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 -- start start 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392bf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b392224b0 con 0x558b392bf2d0 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b391cad70 con 0x558b392bfbf0 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf41b78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b3921e550 con 0x558b392c87d0 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf35ffb640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 0x7fdf240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf37fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf3d455640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392bf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57280/0 (socket says 172.21.15.116:57280) 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.346+0000 7fdf3cc54640 1 -- 172.21.15.116:0/35381926 learned_addr learned my addr 172.21.15.116:0/35381926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3d455640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 0x7fdf240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf35ffb640 1 -- 172.21.15.116:0/2221836900 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fdf280f1110 con 0x7fdf38101060 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 -- 172.21.15.116:0/35381926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 msgr2=0x558b392cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 -- 172.21.15.116:0/35381926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 msgr2=0x558b392bf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392bf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 -- 172.21.15.116:0/35381926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b391ed210 con 0x558b392bfbf0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009c70 tx=0x7fdf20011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3bd47f7f08dcd4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf3d455640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 0x7fdf240a4e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf30005c40 tx=0x7fdf30009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf2effd640 1 -- 172.21.15.116:0/35381926 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20004230 con 0x558b392bfbf0 2024-03-19T02:52:50.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf2effd640 1 -- 172.21.15.116:0/35381926 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf20009e20 con 0x558b392bfbf0 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.347+0000 7fdf2effd640 1 -- 172.21.15.116:0/35381926 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20019640 con 0x558b392bfbf0 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- 172.21.15.116:0/35381926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 msgr2=0x558b392c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009c70 tx=0x7fdf20011810 comp rx=0 tx=0).stop 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- 172.21.15.116:0/35381926 shutdown_connections 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392c8290 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 --2- 172.21.15.116:0/35381926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392bf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- 172.21.15.116:0/35381926 >> 172.21.15.116:0/35381926 conn(0x558b392b9a00 msgr2=0x558b392b9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- 172.21.15.116:0/35381926 shutdown_connections 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- 172.21.15.116:0/35381926 wait complete. 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 -- start start 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.348+0000 7fdf41b78740 1 Processor -- start 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392bf2d0 0x558b392c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392c87d0 0x558b392c6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b392cb5c0 con 0x558b392bfbf0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b391de310 con 0x558b392c87d0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b392cb740 con 0x558b392bf2d0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 10 client.0 _ll_register_callbacks cb 0x558b391d22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392bf2d0 0x558b392c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40546/0 (socket says 172.21.15.116:40546) 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 -- 172.21.15.116:0/3805859941 learned_addr learned my addr 172.21.15.116:0/3805859941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf3d455640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392c87d0 0x558b392c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 -- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392bf2d0 msgr2=0x558b392c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392bf2d0 0x558b392c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 -- 172.21.15.116:0/3805859941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392c87d0 msgr2=0x558b392c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392c87d0 0x558b392c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 -- 172.21.15.116:0/3805859941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b38f363f0 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf37fff640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf2802c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b500e6cfcbf4e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf17fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf280bdba0 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fdf280053d0 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.349+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf20009e20 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdf280bd4a0 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 10 client.22557 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558b390007d0 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf20009e20 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fdf280bd700 con 0x558b392bfbf0 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 10 client.22557 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 10 client.22557 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:52:50.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 20 client.22557 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b392d8af0 0x558b392daf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 4 client.22557 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b392803e0 con 0x558b392d8af0 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b392d8af0 0x558b392daf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf17fff640 10 client.22557 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.350+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b392d8af0 0x558b392daf50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a15394f66769be3 server_cookie=bd468ed9d182af0 in_seq=0 out_seq=0 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf17fff640 1 -- 172.21.15.116:0/3805859941 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b392803e0 con 0x558b392d8af0 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf17fff640 10 client.22557 handle_command_reply: tid=0 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 client.22557 shutdown 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 msgr2=0x558b392c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf2802c740 comp rx=0 tx=0).stop 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 shutdown_connections 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b392d8af0 0x558b392daf50 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392c87d0 0x558b392c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bfbf0 0x558b392c5d00 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf17fff640 0 client.22557 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 --2- 172.21.15.116:0/3805859941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392bf2d0 0x558b392c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 >> 172.21.15.116:0/3805859941 conn(0x558b392b9a00 msgr2=0x558b392bb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 shutdown_connections 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 1 -- 172.21.15.116:0/3805859941 wait complete. 2024-03-19T02:52:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.351+0000 7fdf41b78740 20 client.22557 trim_cache size 0 max 16384 2024-03-19T02:52:50.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.364+0000 7fdf41b78740 1 Processor -- start 2024-03-19T02:52:50.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.364+0000 7fdf41b78740 1 -- start start 2024-03-19T02:52:50.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381aecd0 con 0x558b392bf2d0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381b08a0 con 0x558b392bfbf0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf41b78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381af3d0 con 0x558b392c87d0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40552/0 (socket says 172.21.15.116:40552) 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 -- 172.21.15.116:0/4039870473 learned_addr learned my addr 172.21.15.116:0/4039870473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf37fff640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3d455640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 -- 172.21.15.116:0/4039870473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 msgr2=0x558b392bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 -- 172.21.15.116:0/4039870473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 msgr2=0x558b392d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 -- 172.21.15.116:0/4039870473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b392b3fb0 con 0x558b392bf2d0 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf37fff640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdf200058c0 tx=0x7fdf20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f8f0964b15f5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf2e7fc640 1 -- 172.21.15.116:0/4039870473 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf20009d60 con 0x558b392bf2d0 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf2e7fc640 1 -- 172.21.15.116:0/4039870473 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf20029070 con 0x558b392bf2d0 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.365+0000 7fdf2e7fc640 1 -- 172.21.15.116:0/4039870473 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2001a440 con 0x558b392bf2d0 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- 172.21.15.116:0/4039870473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 msgr2=0x558b392c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fdf200058c0 tx=0x7fdf20000bc0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- 172.21.15.116:0/4039870473 shutdown_connections 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 --2- 172.21.15.116:0/4039870473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392c7ef0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- 172.21.15.116:0/4039870473 >> 172.21.15.116:0/4039870473 conn(0x558b392b9a00 msgr2=0x558b392bade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- 172.21.15.116:0/4039870473 shutdown_connections 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- 172.21.15.116:0/4039870473 wait complete. 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 -- start start 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.366+0000 7fdf41b78740 1 Processor -- start 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381b01d0 con 0x558b392bf2d0 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381afad0 con 0x558b392bfbf0 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381accd0 con 0x558b392c87d0 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 10 client.0 _ll_register_callbacks cb 0x7fdf280057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40554/0 (socket says 172.21.15.116:40554) 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 -- 172.21.15.116:0/714089219 learned_addr learned my addr 172.21.15.116:0/714089219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf37fff640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3d455640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 -- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 msgr2=0x558b392ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 -- 172.21.15.116:0/714089219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 msgr2=0x558b392cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 -- 172.21.15.116:0/714089219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3933e0f0 con 0x558b392bf2d0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf3cc54640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009bc0 tx=0x7fdf20003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c709f347a2dcfd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf17fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2003a060 con 0x558b392bf2d0 2024-03-19T02:52:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fdf200050c0 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.367+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf20029070 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdf20041050 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 10 client.22563 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558b3933f3f0 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf20029070 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7fdf20018070 con 0x558b392bf2d0 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 10 client.22563 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 10 client.22563 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 20 client.22563 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b3933f940 0x558b39341d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 4 client.22563 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 --> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b3922f9f0 con 0x558b3933f940 2024-03-19T02:52:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf37fff640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b3933f940 0x558b39341d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf37fff640 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b3933f940 0x558b39341d60 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a83da35fac6dd1de server_cookie=889f22136c722bd2 in_seq=0 out_seq=0 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.368+0000 7fdf17fff640 10 client.22563 ms_handle_connect on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf17fff640 1 -- 172.21.15.116:0/714089219 <== mds.4 v2:172.21.15.162:6834/831287214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b3922f9f0 con 0x558b3933f940 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf17fff640 10 client.22563 handle_command_reply: tid=0 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 client.22563 shutdown 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 msgr2=0x558b392cc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009bc0 tx=0x7fdf20003e40 comp rx=0 tx=0).stop 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 shutdown_connections 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:6834/831287214,v1:172.21.15.162:6835/831287214] conn(0x558b3933f940 0x558b39341d60 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b392c87d0 0x558b392ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf17fff640 0 client.22563 ms_handle_reset on v2:172.21.15.162:6834/831287214 2024-03-19T02:52:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b392bfbf0 0x558b392cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 --2- 172.21.15.116:0/714089219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b392bf2d0 0x558b392cc530 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 >> 172.21.15.116:0/714089219 conn(0x558b392b9a00 msgr2=0x558b392b8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 shutdown_connections 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 1 -- 172.21.15.116:0/714089219 wait complete. 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf41b78740 20 client.22563 trim_cache size 0 max 16384 2024-03-19T02:52:50.379 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 msgr2=0x7fdf240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 0x7fdf240a4e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdf30005c40 tx=0x7fdf30009500 comp rx=0 tx=0).stop 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 msgr2=0x7fdf38199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009230 tx=0x7fdf2802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 shutdown_connections 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fdf240a2970 0x7fdf240a4e30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf381066b0 0x7fdf381a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf38101060 0x7fdf38199370 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 --2- 172.21.15.116:0/2221836900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf381006a0 0x7fdf38198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.369+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 >> 172.21.15.116:0/2221836900 conn(0x7fdf380778d0 msgr2=0x7fdf38102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.370+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 shutdown_connections 2024-03-19T02:52:50.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.370+0000 7fdf3eeb8640 1 -- 172.21.15.116:0/2221836900 wait complete. 2024-03-19T02:52:50.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-19T02:52:50.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:50 smithi116 ceph-mon[32263]: pgmap v791: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:50.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:50 smithi162 ceph-mon[35761]: pgmap v791: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 -- 172.21.15.116:0/1780153697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 msgr2=0x7f5024106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 --2- 172.21.15.116:0/1780153697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f5024106be0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f5014009930 tx=0x7f501402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 -- 172.21.15.116:0/1780153697 shutdown_connections 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 --2- 172.21.15.116:0/1780153697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 0x7f502410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 --2- 172.21.15.116:0/1780153697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f50241001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 --2- 172.21.15.116:0/1780153697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f5024106be0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 -- 172.21.15.116:0/1780153697 >> 172.21.15.116:0/1780153697 conn(0x7f5024076c40 msgr2=0x7f50240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 -- 172.21.15.116:0/1780153697 shutdown_connections 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.484+0000 7f50289cd640 1 -- 172.21.15.116:0/1780153697 wait complete. 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 Processor -- start 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 -- start start 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f5024198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 0x7f5024198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f502419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5024111bc0 con 0x7f5024106800 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5024111a40 con 0x7f50240ffd20 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f50289cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5024111d40 con 0x7f50241006e0 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f502419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f5024198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f501bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 0x7f5024198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f502419c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40568/0 (socket says 172.21.15.116:40568) 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f5022d9d640 1 -- 172.21.15.116:0/954094187 learned_addr learned my addr 172.21.15.116:0/954094187 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 -- 172.21.15.116:0/954094187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 msgr2=0x7f5024198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 0x7f5024198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 -- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 msgr2=0x7f502419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f502419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.485+0000 7f502259c640 1 -- 172.21.15.116:0/954094187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502419d0f0 con 0x7f50240ffd20 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f502259c640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f5024198100 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5014009230 tx=0x7f50140045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f501b7fe640 1 -- 172.21.15.116:0/954094187 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014043020 con 0x7f50240ffd20 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502419d380 con 0x7f50240ffd20 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f501b7fe640 1 -- 172.21.15.116:0/954094187 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f501402ca70 con 0x7f50240ffd20 2024-03-19T02:52:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f501b7fe640 1 -- 172.21.15.116:0/954094187 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50140039b0 con 0x7f50240ffd20 2024-03-19T02:52:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.486+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50241a4c70 con 0x7f50240ffd20 2024-03-19T02:52:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f501b7fe640 1 -- 172.21.15.116:0/954094187 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5014046020 con 0x7f50240ffd20 2024-03-19T02:52:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 Processor -- start 2024-03-19T02:52:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 -- start start 2024-03-19T02:52:50.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf462e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcf45914b0 con 0x55fcf462e2d0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcf4539d70 con 0x55fcf46377d0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.488+0000 7f502b5b9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fcf458d550 con 0x55fcf462ebf0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf462e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501b7fe640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 0x7f50100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f502259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f502259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39612/0 (socket says 172.21.15.116:39612) 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57340/0 (socket says 172.21.15.116:57340) 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f502259c640 1 -- 172.21.15.116:0/2276716229 learned_addr learned my addr 172.21.15.116:0/2276716229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5022d9d640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 0x7f50100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501b7fe640 1 -- 172.21.15.116:0/954094187 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f50140f10c0 con 0x7f50240ffd20 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 -- 172.21.15.116:0/2276716229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 msgr2=0x55fcf4637290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 -- 172.21.15.116:0/2276716229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 msgr2=0x55fcf462e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf462e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 -- 172.21.15.116:0/2276716229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf455c210 con 0x55fcf46377d0 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f502259c640 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5022d9d640 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf462e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f501bfff640 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f500c009bf0 tx=0x7f500c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8be8a2e67cc87a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5022d9d640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 0x7f50100a4e30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f501c0094d0 tx=0x7f501c004ab0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5018ff9640 1 -- 172.21.15.116:0/2276716229 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c004050 con 0x55fcf46377d0 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5018ff9640 1 -- 172.21.15.116:0/2276716229 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f500c0054d0 con 0x55fcf46377d0 2024-03-19T02:52:50.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.489+0000 7f5018ff9640 1 -- 172.21.15.116:0/2276716229 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c004c40 con 0x55fcf46377d0 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- 172.21.15.116:0/2276716229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 msgr2=0x55fcf4639bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f500c009bf0 tx=0x7f500c011810 comp rx=0 tx=0).stop 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- 172.21.15.116:0/2276716229 shutdown_connections 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4639bc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462ebf0 0x55fcf4637290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 --2- 172.21.15.116:0/2276716229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf462e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- 172.21.15.116:0/2276716229 >> 172.21.15.116:0/2276716229 conn(0x55fcf4628a00 msgr2=0x55fcf4628de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- 172.21.15.116:0/2276716229 shutdown_connections 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- 172.21.15.116:0/2276716229 wait complete. 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.490+0000 7f502b5b9740 1 -- start start 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 Processor -- start 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462e2d0 0x55fcf46347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4635220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcf463a5c0 con 0x55fcf462ebf0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcf454d310 con 0x55fcf46377d0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fcf463a740 con 0x55fcf462e2d0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502b5b9740 10 client.0 _ll_register_callbacks cb 0x55fcf45412d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40598/0 (socket says 172.21.15.116:40598) 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4635220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 -- 172.21.15.116:0/438414865 learned_addr learned my addr 172.21.15.116:0/438414865 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f502259c640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462e2d0 0x55fcf46347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 -- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462e2d0 msgr2=0x55fcf46347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462e2d0 0x55fcf46347a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 -- 172.21.15.116:0/438414865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 msgr2=0x55fcf4635220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4635220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 -- 172.21.15.116:0/438414865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcf42a53f0 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f501bfff640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f500c01b040 tx=0x7f500c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d8b62ef7df5db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f5009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.491+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c029070 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f500c003a00 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f500c0054d0 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f500c035050 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 10 client.22575 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55fcf436f7d0 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f500c0054d0 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f500c004fb0 con 0x55fcf462ebf0 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 10 client.22575 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 10 client.22575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 20 client.22575 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46479e0 0x55fcf4649dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 4 client.22575 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcf45ef3e0 con 0x55fcf46479e0 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.492+0000 7f502259c640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46479e0 0x55fcf4649dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f5009ffb640 10 client.22575 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502259c640 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46479e0 0x55fcf4649dd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6267232fb8d32769 server_cookie=60331116f0831eec in_seq=0 out_seq=0 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f5009ffb640 1 -- 172.21.15.116:0/438414865 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcf45ef3e0 con 0x55fcf46479e0 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f5009ffb640 10 client.22575 handle_command_reply: tid=0 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 client.22575 shutdown 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 msgr2=0x55fcf4634ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f500c01b040 tx=0x7f500c009fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 shutdown_connections 2024-03-19T02:52:50.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf46377d0 0x55fcf4635220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46479e0 0x55fcf4649dd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4634ce0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 --2- 172.21.15.116:0/438414865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf462e2d0 0x55fcf46347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f5009ffb640 0 client.22575 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.493+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 >> 172.21.15.116:0/438414865 conn(0x55fcf4628a00 msgr2=0x55fcf462a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.494+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 shutdown_connections 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.494+0000 7f502b5b9740 1 -- 172.21.15.116:0/438414865 wait complete. 2024-03-19T02:52:50.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.494+0000 7f502b5b9740 20 client.22575 trim_cache size 0 max 16384 2024-03-19T02:52:50.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:50 smithi154 ceph-mon[35693]: pgmap v791: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 186 op/s 2024-03-19T02:52:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 Processor -- start 2024-03-19T02:52:50.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 -- start start 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 0x55fcf4635040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf464f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50241aed20 con 0x55fcf462e2d0 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50241b08f0 con 0x55fcf462ebf0 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502b5b9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50241af420 con 0x55fcf46377d0 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40600/0 (socket says 172.21.15.116:40600) 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 -- 172.21.15.116:0/1744199863 learned_addr learned my addr 172.21.15.116:0/1744199863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f501bfff640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 0x55fcf4635040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f5022d9d640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf464f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 -- 172.21.15.116:0/1744199863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 msgr2=0x55fcf464f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf464f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 -- 172.21.15.116:0/1744199863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 msgr2=0x55fcf4635040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 0x55fcf4635040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 -- 172.21.15.116:0/1744199863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf462ea20 con 0x55fcf462e2d0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f501bfff640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 0x55fcf4635040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f502259c640 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f50140dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d543e7c3653d7442 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f500bfff640 1 -- 172.21.15.116:0/1744199863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50140bd3f0 con 0x55fcf462e2d0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.507+0000 7f500bfff640 1 -- 172.21.15.116:0/1744199863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50140bdaf0 con 0x55fcf462e2d0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f500bfff640 1 -- 172.21.15.116:0/1744199863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50140b9a60 con 0x55fcf462e2d0 2024-03-19T02:52:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- 172.21.15.116:0/1744199863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 msgr2=0x55fcf4636ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f50140dd040 comp rx=0 tx=0).stop 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- 172.21.15.116:0/1744199863 shutdown_connections 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf464f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462ebf0 0x55fcf4635040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 --2- 172.21.15.116:0/1744199863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462e2d0 0x55fcf4636ed0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- 172.21.15.116:0/1744199863 >> 172.21.15.116:0/1744199863 conn(0x55fcf4628a00 msgr2=0x55fcf4629c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- 172.21.15.116:0/1744199863 shutdown_connections 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- 172.21.15.116:0/1744199863 wait complete. 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 -- start start 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.508+0000 7f502b5b9740 1 Processor -- start 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 0x55fcf463fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf4631590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50241b0220 con 0x55fcf462ebf0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50241afb20 con 0x55fcf462e2d0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50241acd20 con 0x55fcf46377d0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 10 client.0 _ll_register_callbacks cb 0x7f501c013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f501bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 0x55fcf463fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f501bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40616/0 (socket says 172.21.15.116:40616) 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f501bfff640 1 -- 172.21.15.116:0/3600516632 learned_addr learned my addr 172.21.15.116:0/3600516632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f5022d9d640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf4631590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 -- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 msgr2=0x55fcf4631590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf4631590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 -- 172.21.15.116:0/3600516632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 msgr2=0x55fcf4631050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 -- 172.21.15.116:0/3600516632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcf4631dc0 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f501bfff640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502259c640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 0x55fcf463fb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f5014009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd318045053133f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f5009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50140d5070 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50140bdaf0 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.509+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f50140c1160 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f50140c1540 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 10 client.32282 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55fcf46adb20 con 0x55fcf462e2d0 2024-03-19T02:52:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50140bdaf0 con 0x55fcf462e2d0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f50140bd650 con 0x55fcf462e2d0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 10 client.32282 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 10 client.32282 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 20 client.32282 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46ae1b0 0x55fcf46b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 4 client.32282 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 --> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcf459e9f0 con 0x55fcf46ae1b0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.510+0000 7f501bfff640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46ae1b0 0x55fcf46b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f5009ffb640 10 client.32282 ms_handle_connect on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f501bfff640 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46ae1b0 0x55fcf46b05d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c4255c8a512ab793 server_cookie=3c55a1ecc29dd6b2 in_seq=0 out_seq=0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f5009ffb640 1 -- 172.21.15.116:0/3600516632 <== mds.3 v2:172.21.15.162:6836/3347737639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcf459e9f0 con 0x55fcf46ae1b0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f5009ffb640 10 client.32282 handle_command_reply: tid=0 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 client.32282 shutdown 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 msgr2=0x55fcf463fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 0x55fcf463fb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5014006010 tx=0x7f5014009260 comp rx=0 tx=0).stop 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 shutdown_connections 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fcf46377d0 0x55fcf4631590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.162:6836/3347737639,v1:172.21.15.162:6837/3347737639] conn(0x55fcf46ae1b0 0x55fcf46b05d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcf462ebf0 0x55fcf4631050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f5009ffb640 0 client.32282 ms_handle_reset on v2:172.21.15.162:6836/3347737639 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 --2- 172.21.15.116:0/3600516632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcf462e2d0 0x55fcf463fb80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 >> 172.21.15.116:0/3600516632 conn(0x55fcf4628a00 msgr2=0x55fcf462a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 shutdown_connections 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 1 -- 172.21.15.116:0/3600516632 wait complete. 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f502b5b9740 20 client.32282 trim_cache size 0 max 16384 2024-03-19T02:52:50.521 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 msgr2=0x7f50100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.511+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 0x7f50100a4e30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f501c0094d0 tx=0x7f501c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 msgr2=0x7f5024198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f5024198100 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5014009230 tx=0x7f50140045e0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 shutdown_connections 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f50100a2970 0x7f50100a4e30 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5024106800 0x7f502419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50241006e0 0x7f5024198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 --2- 172.21.15.116:0/954094187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50240ffd20 0x7f5024198100 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 >> 172.21.15.116:0/954094187 conn(0x7f5024076c40 msgr2=0x7f50241049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 shutdown_connections 2024-03-19T02:52:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.512+0000 7f50289cd640 1 -- 172.21.15.116:0/954094187 wait complete. 2024-03-19T02:52:50.524 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1459036280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb41006a0 msgr2=0x7efeb4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1459036280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb41006a0 0x7efeb4100b20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009930 tx=0x7efe9c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1459036280 shutdown_connections 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1459036280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb4101060 0x7efeb410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1459036280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb41006a0 0x7efeb4100b20 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1459036280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41066b0 0x7efeb4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1459036280 >> 172.21.15.116:0/1459036280 conn(0x7efeb40778d0 msgr2=0x7efeb4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1459036280 shutdown_connections 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1459036280 wait complete. 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 Processor -- start 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.669+0000 7efeb9d3b640 1 -- start start 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41006a0 0x7efeb4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb41066b0 0x7efeb419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4111a70 con 0x7efeb4101060 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41118f0 con 0x7efeb41006a0 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4111bf0 con 0x7efeb41066b0 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40622/0 (socket says 172.21.15.116:40622) 2024-03-19T02:52:50.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 -- 172.21.15.116:0/1906083923 learned_addr learned my addr 172.21.15.116:0/1906083923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb37fe640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41006a0 0x7efeb4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb41066b0 0x7efeb419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 -- 172.21.15.116:0/1906083923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb41066b0 msgr2=0x7efeb419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb41066b0 0x7efeb419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 -- 172.21.15.116:0/1906083923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41006a0 msgr2=0x7efeb4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41006a0 0x7efeb4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 -- 172.21.15.116:0/1906083923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb419cf80 con 0x7efeb4101060 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb2ffd640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009230 tx=0x7efe9c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb0ff9640 1 -- 172.21.15.116:0/1906083923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c043020 con 0x7efeb4101060 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb0ff9640 1 -- 172.21.15.116:0/1906083923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe9c02ca70 con 0x7efeb4101060 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb419d210 con 0x7efeb4101060 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb0ff9640 1 -- 172.21.15.116:0/1906083923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c003a50 con 0x7efeb4101060 2024-03-19T02:52:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.670+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb41a4b00 con 0x7efeb4101060 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.672+0000 7efeb0ff9640 1 -- 172.21.15.116:0/1906083923 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7efe9c046020 con 0x7efeb4101060 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 Processor -- start 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 -- start start 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd7399f2d0 0x55dd7399f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd739a87d0 0x55dd739aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd739024b0 con 0x55dd7399f2d0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd738aad70 con 0x55dd7399fbf0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efebca02740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd738fe550 con 0x55dd739a87d0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb0ff9640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 0x7efea00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd739a87d0 0x55dd739aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57406/0 (socket says 172.21.15.116:57406) 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd7399f2d0 0x55dd7399f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 -- 172.21.15.116:0/938231774 learned_addr learned my addr 172.21.15.116:0/938231774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 0x7efea00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 -- 172.21.15.116:0/938231774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd739a87d0 msgr2=0x55dd739aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd739a87d0 0x55dd739aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 -- 172.21.15.116:0/938231774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd7399f2d0 msgr2=0x55dd7399f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd7399f2d0 0x55dd7399f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb37fe640 1 -- 172.21.15.116:0/938231774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd738cd210 con 0x55dd7399fbf0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.673+0000 7efeb0ff9640 1 -- 172.21.15.116:0/1906083923 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7efe9c0f1130 con 0x7efeb4101060 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 0x7efea00a4e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efeac005c40 tx=0x7efeac009bb0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efeb37fe640 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efea4009c70 tx=0x7efea4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1248130aaa9434f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efea9ffb640 1 -- 172.21.15.116:0/938231774 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4004230 con 0x55dd7399fbf0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efea9ffb640 1 -- 172.21.15.116:0/938231774 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efea4009e20 con 0x55dd7399fbf0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efea9ffb640 1 -- 172.21.15.116:0/938231774 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4019640 con 0x55dd7399fbf0 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 -- 172.21.15.116:0/938231774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 msgr2=0x55dd739a8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efea4009c70 tx=0x7efea4011810 comp rx=0 tx=0).stop 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 -- 172.21.15.116:0/938231774 shutdown_connections 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd739a87d0 0x55dd739aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a8290 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 --2- 172.21.15.116:0/938231774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd7399f2d0 0x55dd7399f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 -- 172.21.15.116:0/938231774 >> 172.21.15.116:0/938231774 conn(0x55dd73999a00 msgr2=0x55dd73999de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.674+0000 7efebca02740 1 -- 172.21.15.116:0/938231774 shutdown_connections 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 -- 172.21.15.116:0/938231774 wait complete. 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 -- start start 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 Processor -- start 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd739ab5c0 con 0x55dd739a87d0 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dd738be310 con 0x55dd7399fbf0 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dd739ab740 con 0x55dd7399f2d0 2024-03-19T02:52:50.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efebca02740 10 client.0 _ll_register_callbacks cb 0x55dd738b22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efeb37fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40642/0 (socket says 172.21.15.116:40642) 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efeb3fff640 1 -- 172.21.15.116:0/2906094418 learned_addr learned my addr 172.21.15.116:0/2906094418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.675+0000 7efeb2ffd640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 -- 172.21.15.116:0/2906094418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 msgr2=0x55dd739a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 -- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 msgr2=0x55dd739a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 -- 172.21.15.116:0/2906094418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd736163f0 con 0x55dd739a87d0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7efeac010010 tx=0x7efeac005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ef19cc4f53e580 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac020870 con 0x55dd739a87d0 2024-03-19T02:52:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7efeac020a10 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea4009e20 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7efeac020c10 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efebca02740 10 client.22593 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55dd736e07d0 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efea4009e20 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7efeac027070 con 0x55dd739a87d0 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efebca02740 10 client.22593 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.676+0000 7efebca02740 10 client.22593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efebca02740 20 client.22593 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd739b8b50 0x55dd739bafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efebca02740 4 client.22593 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd739603e0 con 0x55dd739b8b50 2024-03-19T02:52:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efeb37fe640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd739b8b50 0x55dd739bafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efe92ffd640 10 client.22593 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.677+0000 7efeb37fe640 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd739b8b50 0x55dd739bafd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d2c8fd5bc7c2b94c server_cookie=2d9a25eb0ff8f64 in_seq=0 out_seq=0 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efe92ffd640 1 -- 172.21.15.116:0/2906094418 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd739603e0 con 0x55dd739b8b50 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efe92ffd640 10 client.22593 handle_command_reply: tid=0 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 client.22593 shutdown 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 msgr2=0x55dd739a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7efeac010010 tx=0x7efeac005bd0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 shutdown_connections 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd739b8b50 0x55dd739bafd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739a6240 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efe92ffd640 0 client.22593 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 --2- 172.21.15.116:0/2906094418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 >> 172.21.15.116:0/2906094418 conn(0x55dd73999a00 msgr2=0x55dd7399b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 shutdown_connections 2024-03-19T02:52:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 1 -- 172.21.15.116:0/2906094418 wait complete. 2024-03-19T02:52:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.678+0000 7efebca02740 20 client.22593 trim_cache size 0 max 16384 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 Processor -- start 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 -- start start 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41aec10 con 0x55dd739a87d0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41b07e0 con 0x55dd7399fbf0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.691+0000 7efebca02740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41af310 con 0x55dd7399f2d0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40648/0 (socket says 172.21.15.116:40648) 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 -- 172.21.15.116:0/1160032693 learned_addr learned my addr 172.21.15.116:0/1160032693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb37fe640 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb2ffd640 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 -- 172.21.15.116:0/1160032693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 msgr2=0x55dd739a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 -- 172.21.15.116:0/1160032693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 msgr2=0x55dd739b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 -- 172.21.15.116:0/1160032693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd739952c0 con 0x55dd739a87d0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efeb3fff640 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7efeac00fd40 tx=0x7efeac00fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453200e996ed0d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efea97fa640 1 -- 172.21.15.116:0/1160032693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac004cb0 con 0x55dd739a87d0 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efea97fa640 1 -- 172.21.15.116:0/1160032693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efeac009480 con 0x55dd739a87d0 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.692+0000 7efea97fa640 1 -- 172.21.15.116:0/1160032693 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac029ac0 con 0x55dd739a87d0 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- 172.21.15.116:0/1160032693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 msgr2=0x55dd7399b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7efeac00fd40 tx=0x7efeac00fd70 comp rx=0 tx=0).stop 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- 172.21.15.116:0/1160032693 shutdown_connections 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd7399b4f0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399fbf0 0x55dd739b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- 172.21.15.116:0/1160032693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399f2d0 0x55dd739a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- 172.21.15.116:0/1160032693 >> 172.21.15.116:0/1160032693 conn(0x55dd73999a00 msgr2=0x55dd7399ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- 172.21.15.116:0/1160032693 shutdown_connections 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- 172.21.15.116:0/1160032693 wait complete. 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- start start 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 Processor -- start 2024-03-19T02:52:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399f2d0 0x55dd739ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399fbf0 0x55dd739acb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41b0110 con 0x55dd739a87d0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41afa10 con 0x55dd7399f2d0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.693+0000 7efebca02740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41acc10 con 0x55dd7399fbf0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efebca02740 10 client.0 _ll_register_callbacks cb 0x7efeac011550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40660/0 (socket says 172.21.15.116:40660) 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399fbf0 0x55dd739acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 -- 172.21.15.116:0/2566208271 learned_addr learned my addr 172.21.15.116:0/2566208271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb37fe640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399f2d0 0x55dd739ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 -- 172.21.15.116:0/2566208271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399fbf0 msgr2=0x55dd739acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399fbf0 0x55dd739acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 -- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399f2d0 msgr2=0x55dd739ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399f2d0 0x55dd739ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 -- 172.21.15.116:0/2566208271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd73a1e170 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efeb3fff640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7efeac00fc40 tx=0x7efeac004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8481a9dca79a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efe92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac046060 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7efeac0297c0 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeac009480 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7efeac003c80 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efebca02740 10 client.22599 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55dd73a1f470 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.694+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efeac009480 con 0x55dd739a87d0 2024-03-19T02:52:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7efeac027040 con 0x55dd739a87d0 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 10 client.22599 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 10 client.22599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 20 client.22599 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd73a1f920 0x55dd73a21d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 4 client.22599 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 --> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd7390f9f0 con 0x55dd73a1f920 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efeb37fe640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd73a1f920 0x55dd73a21d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efe92ffd640 10 client.22599 ms_handle_connect on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efeb37fe640 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd73a1f920 0x55dd73a21d40 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d201974384e06a5 server_cookie=d7a2c747210a5d33 in_seq=0 out_seq=0 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efe92ffd640 1 -- 172.21.15.116:0/2566208271 <== mds.2 v2:172.21.15.154:6836/455932973 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd7390f9f0 con 0x55dd73a1f920 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efe92ffd640 10 client.22599 handle_command_reply: tid=0 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.695+0000 7efebca02740 1 client.22599 shutdown 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 msgr2=0x55dd739ad040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7efeac00fc40 tx=0x7efeac004c70 comp rx=0 tx=0).stop 2024-03-19T02:52:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 shutdown_connections 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:6836/455932973,v1:172.21.15.154:6837/455932973] conn(0x55dd73a1f920 0x55dd73a21d40 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd739a87d0 0x55dd739ad040 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efe92ffd640 0 client.22599 ms_handle_reset on v2:172.21.15.154:6836/455932973 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dd7399fbf0 0x55dd739acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 --2- 172.21.15.116:0/2566208271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dd7399f2d0 0x55dd739ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 >> 172.21.15.116:0/2566208271 conn(0x55dd73999a00 msgr2=0x55dd739a6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 shutdown_connections 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 1 -- 172.21.15.116:0/2566208271 wait complete. 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efebca02740 20 client.22599 trim_cache size 0 max 16384 2024-03-19T02:52:50.704 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 msgr2=0x7efea00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 0x7efea00a4e30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efeac005c40 tx=0x7efeac009bb0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 msgr2=0x7efeb41984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7efe9c009230 tx=0x7efe9c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 shutdown_connections 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7efea00a2970 0x7efea00a4e30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efeb41066b0 0x7efeb419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efeb4101060 0x7efeb41984d0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 --2- 172.21.15.116:0/1906083923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb41006a0 0x7efeb4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 >> 172.21.15.116:0/1906083923 conn(0x7efeb40778d0 msgr2=0x7efeb4102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 shutdown_connections 2024-03-19T02:52:50.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.696+0000 7efeb9d3b640 1 -- 172.21.15.116:0/1906083923 wait complete. 2024-03-19T02:52:50.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.811+0000 7f08cd627640 1 -- 172.21.15.116:0/4211302124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 msgr2=0x7f08c810b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.811+0000 7f08cd627640 1 --2- 172.21.15.116:0/4211302124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c810b840 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f08bc009900 tx=0x7f08bc02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.811+0000 7f08cd627640 1 -- 172.21.15.116:0/4211302124 shutdown_connections 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.811+0000 7f08cd627640 1 --2- 172.21.15.116:0/4211302124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 0x7f08c810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 --2- 172.21.15.116:0/4211302124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c810b840 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 --2- 172.21.15.116:0/4211302124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 0x7f08c8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- 172.21.15.116:0/4211302124 >> 172.21.15.116:0/4211302124 conn(0x7f08c80778d0 msgr2=0x7f08c8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- 172.21.15.116:0/4211302124 shutdown_connections 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- 172.21.15.116:0/4211302124 wait complete. 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 Processor -- start 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- start start 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 0x7f08c8100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 0x7f08c8101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8111a30 con 0x7f08c8103360 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81118b0 con 0x7f08c8102a40 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.812+0000 7f08cd627640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8111bb0 con 0x7f08c810bd80 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40670/0 (socket says 172.21.15.116:40670) 2024-03-19T02:52:50.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 -- 172.21.15.116:0/2000526918 learned_addr learned my addr 172.21.15.116:0/2000526918 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 0x7f08c8100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c77fe640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 0x7f08c8101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 -- 172.21.15.116:0/2000526918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 msgr2=0x7f08c8101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 0x7f08c8101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 -- 172.21.15.116:0/2000526918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 msgr2=0x7f08c8100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 0x7f08c8100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 -- 172.21.15.116:0/2000526918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c81a6450 con 0x7f08c8103360 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f08bc005940 tx=0x7f08bc02c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08affff640 1 -- 172.21.15.116:0/2000526918 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc043020 con 0x7f08c8103360 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08affff640 1 -- 172.21.15.116:0/2000526918 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08bc004e00 con 0x7f08c8103360 2024-03-19T02:52:50.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c81a66e0 con 0x7f08c8103360 2024-03-19T02:52:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08affff640 1 -- 172.21.15.116:0/2000526918 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc003970 con 0x7f08c8103360 2024-03-19T02:52:50.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.813+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c81a6c30 con 0x7f08c8103360 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.815+0000 7f08affff640 1 -- 172.21.15.116:0/2000526918 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f08bc046020 con 0x7f08c8103360 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 Processor -- start 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 -- start start 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 0x55e502f566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f61bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e502eb94b0 con 0x55e502f562d0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e502e61d70 con 0x55e502f5f7d0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08d020e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e502eb5550 con 0x55e502f56bf0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08affff640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 0x7f08b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 0x55e502f566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39742/0 (socket says 172.21.15.116:39742) 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f61bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 -- 172.21.15.116:0/571276512 learned_addr learned my addr 172.21.15.116:0/571276512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c77fe640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 0x7f08b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 -- 172.21.15.116:0/571276512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 msgr2=0x55e502f61bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08affff640 1 -- 172.21.15.116:0/2000526918 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f08bc0f1090 con 0x7f08c8103360 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f61bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 -- 172.21.15.116:0/571276512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 msgr2=0x55e502f566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 0x55e502f566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c6ffd640 1 -- 172.21.15.116:0/571276512 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e502e84210 con 0x55e502f56bf0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.816+0000 7f08c77fe640 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 0x55e502f566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08c77fe640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 0x7f08b00a4e30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005f60 tx=0x7f08c0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f08b80114f0 tx=0x7f08b80119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6150d85a804e6d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08ad7fa640 1 -- 172.21.15.116:0/571276512 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b801e070 con 0x55e502f56bf0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08ad7fa640 1 -- 172.21.15.116:0/571276512 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b800fad0 con 0x55e502f56bf0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08ad7fa640 1 -- 172.21.15.116:0/571276512 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8019ab0 con 0x55e502f56bf0 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08d020e740 1 -- 172.21.15.116:0/571276512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 msgr2=0x55e502f5f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08d020e740 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f08b80114f0 tx=0x7f08b80119b0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08d020e740 1 -- 172.21.15.116:0/571276512 shutdown_connections 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08d020e740 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f61bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.817+0000 7f08d020e740 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f5f290 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 --2- 172.21.15.116:0/571276512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f562d0 0x55e502f566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- 172.21.15.116:0/571276512 >> 172.21.15.116:0/571276512 conn(0x55e502f50a00 msgr2=0x55e502f50de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- 172.21.15.116:0/571276512 shutdown_connections 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- 172.21.15.116:0/571276512 wait complete. 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- start start 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 Processor -- start 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f5d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e502f625c0 con 0x55e502f56bf0 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e502e75310 con 0x55e502f5f7d0 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e502f62740 con 0x55e502f562d0 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08d020e740 10 client.0 _ll_register_callbacks cb 0x55e502e692d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39752/0 (socket says 172.21.15.116:39752) 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40680/0 (socket says 172.21.15.116:40680) 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c6ffd640 1 -- 172.21.15.116:0/2339351828 learned_addr learned my addr 172.21.15.116:0/2339351828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.818+0000 7f08c77fe640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f5d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 -- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 msgr2=0x55e502f5c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 -- 172.21.15.116:0/2339351828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 msgr2=0x55e502f5d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f5d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 -- 172.21.15.116:0/2339351828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e502bcd3f0 con 0x55e502f56bf0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0ee4e0 tx=0x7f08bc0bcf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d908777c919aae99 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0d6060 con 0x55e502f56bf0 2024-03-19T02:52:50.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f08bc0b9a90 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b800fad0 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f08bc004fb0 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08d020e740 10 client.22611 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e502c977d0 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08b800fad0 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.819+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f08bc003b10 con 0x55e502f56bf0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 10 client.22611 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 10 client.22611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 20 client.22611 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502f6fad0 0x55e502f71f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 4 client.22611 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e502f173e0 con 0x55e502f6fad0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502f6fad0 0x55e502f71f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08aaff5640 10 client.22611 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.820+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502f6fad0 0x55e502f71f90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87098a606a93b774 server_cookie=3c779b31aefe33c7 in_seq=0 out_seq=0 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08aaff5640 1 -- 172.21.15.116:0/2339351828 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e502f173e0 con 0x55e502f6fad0 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08aaff5640 10 client.22611 handle_command_reply: tid=0 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 client.22611 shutdown 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 msgr2=0x55e502f5cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0ee4e0 tx=0x7f08bc0bcf60 comp rx=0 tx=0).stop 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 shutdown_connections 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502f6fad0 0x55e502f71f90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f5d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f5cd50 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 --2- 172.21.15.116:0/2339351828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08aaff5640 0 client.22611 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:50.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 >> 172.21.15.116:0/2339351828 conn(0x55e502f50a00 msgr2=0x55e502f52750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 shutdown_connections 2024-03-19T02:52:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 1 -- 172.21.15.116:0/2339351828 wait complete. 2024-03-19T02:52:50.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.821+0000 7f08d020e740 20 client.22611 trim_cache size 0 max 16384 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 Processor -- start 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 -- start start 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f4fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b0d20 con 0x55e502f56bf0 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b28f0 con 0x55e502f5f7d0 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08d020e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b1420 con 0x55e502f562d0 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40686/0 (socket says 172.21.15.116:40686) 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c67fc640 1 -- 172.21.15.116:0/2983714044 learned_addr learned my addr 172.21.15.116:0/2983714044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c77fe640 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f4fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.834+0000 7f08c67fc640 1 -- 172.21.15.116:0/2983714044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 msgr2=0x55e502f5ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08c67fc640 1 -- 172.21.15.116:0/2983714044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 msgr2=0x55e502f4fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f4fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08c67fc640 1 -- 172.21.15.116:0/2983714044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e502f5b7c0 con 0x55e502f56bf0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08c67fc640 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0bd3b0 tx=0x7f08bc0ee2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bbf12db4c108e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08acff9640 1 -- 172.21.15.116:0/2983714044 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc0bcd80 con 0x55e502f56bf0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08acff9640 1 -- 172.21.15.116:0/2983714044 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08bc0c5bc0 con 0x55e502f56bf0 2024-03-19T02:52:50.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08acff9640 1 -- 172.21.15.116:0/2983714044 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc003b10 con 0x55e502f56bf0 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 -- 172.21.15.116:0/2983714044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 msgr2=0x55e502f6b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0bd3b0 tx=0x7f08bc0ee2f0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 -- 172.21.15.116:0/2983714044 shutdown_connections 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f5f7d0 0x55e502f4fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f56bf0 0x55e502f6b390 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 --2- 172.21.15.116:0/2983714044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f562d0 0x55e502f5ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 -- 172.21.15.116:0/2983714044 >> 172.21.15.116:0/2983714044 conn(0x55e502f50a00 msgr2=0x55e502f50de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 -- 172.21.15.116:0/2983714044 shutdown_connections 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.835+0000 7f08d020e740 1 -- 172.21.15.116:0/2983714044 wait complete. 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 -- start start 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 Processor -- start 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f562d0 0x55e502f635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f63b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b2220 con 0x55e502f5f7d0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81b1b20 con 0x55e502f562d0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81aed20 con 0x55e502f56bf0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08d020e740 10 client.0 _ll_register_callbacks cb 0x7f08bc009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40696/0 (socket says 172.21.15.116:40696) 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 -- 172.21.15.116:0/1094406994 learned_addr learned my addr 172.21.15.116:0/1094406994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c67fc640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f63b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f562d0 0x55e502f635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 -- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 msgr2=0x55e502f63b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f63b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 -- 172.21.15.116:0/1094406994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f562d0 msgr2=0x55e502f635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f562d0 0x55e502f635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.836+0000 7f08c77fe640 1 -- 172.21.15.116:0/1094406994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e502f643f0 con 0x55e502f5f7d0 2024-03-19T02:52:50.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08c77fe640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f08c0011740 tx=0x7f08c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b11e777dca1aae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c000fb30 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f08c000fcd0 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08bc0c5bc0 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f08c0036050 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 10 client.22617 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e502fd6490 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08bc0c5bc0 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f08c0004a10 con 0x55e502f5f7d0 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 10 client.22617 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 10 client.22617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 20 client.22617 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502fd6c70 0x55e502fd9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 4 client.22617 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 --> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e502ec69f0 con 0x55e502fd6c70 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.837+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502fd6c70 0x55e502fd9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08aaff5640 10 client.22617 ms_handle_connect on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08c6ffd640 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502fd6c70 0x55e502fd9090 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=943708f581809b0a server_cookie=d5a84bcc6172fa43 in_seq=0 out_seq=0 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08aaff5640 1 -- 172.21.15.116:0/1094406994 <== mds.1 v2:172.21.15.162:6838/1427360219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e502ec69f0 con 0x55e502fd6c70 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08aaff5640 10 client.22617 handle_command_reply: tid=0 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 client.22617 shutdown 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 msgr2=0x55e502fd4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f08c0011740 tx=0x7f08c00046c0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 shutdown_connections 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:6838/1427360219,v1:172.21.15.162:6839/1427360219] conn(0x55e502fd6c70 0x55e502fd9090 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e502f5f7d0 0x55e502fd4e80 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e502f56bf0 0x55e502f63b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08aaff5640 0 client.22617 ms_handle_reset on v2:172.21.15.162:6838/1427360219 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 --2- 172.21.15.116:0/1094406994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e502f562d0 0x55e502f635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 >> 172.21.15.116:0/1094406994 conn(0x55e502f50a00 msgr2=0x55e502f5d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 shutdown_connections 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 1 -- 172.21.15.116:0/1094406994 wait complete. 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.838+0000 7f08d020e740 20 client.22617 trim_cache size 0 max 16384 2024-03-19T02:52:50.847 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 msgr2=0x7f08b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 0x7f08b00a4e30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005f60 tx=0x7f08c0005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 msgr2=0x7f08c8100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f08bc005940 tx=0x7f08bc02c690 comp rx=0 tx=0).stop 2024-03-19T02:52:50.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 shutdown_connections 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f08b00a2970 0x7f08b00a4e30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08c810bd80 0x7f08c8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08c8103360 0x7f08c8100bd0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 --2- 172.21.15.116:0/2000526918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08c8102a40 0x7f08c8100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 >> 172.21.15.116:0/2000526918 conn(0x7f08c80778d0 msgr2=0x7f08c80fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 shutdown_connections 2024-03-19T02:52:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.839+0000 7f08cd627640 1 -- 172.21.15.116:0/2000526918 wait complete. 2024-03-19T02:52:50.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:52:50.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 -- 172.21.15.116:0/862623283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 msgr2=0x7f8e6810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 --2- 172.21.15.116:0/862623283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e6810d4e0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009900 tx=0x7f8e6002bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:50.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 -- 172.21.15.116:0/862623283 shutdown_connections 2024-03-19T02:52:50.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 --2- 172.21.15.116:0/862623283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e6810d4e0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 --2- 172.21.15.116:0/862623283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68100970 0x7f8e68100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 --2- 172.21.15.116:0/862623283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68105660 0x7f8e68105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 -- 172.21.15.116:0/862623283 >> 172.21.15.116:0/862623283 conn(0x7f8e680774c0 msgr2=0x7f8e680778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 -- 172.21.15.116:0/862623283 shutdown_connections 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.955+0000 7f8e6d426640 1 -- 172.21.15.116:0/862623283 wait complete. 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 Processor -- start 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 -- start start 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68100970 0x7f8e68198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68105660 0x7f8e681a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68110930 con 0x7f8e68101330 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681107b0 con 0x7f8e68105660 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e6d426640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68110ab0 con 0x7f8e68100970 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40718/0 (socket says 172.21.15.116:40718) 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 -- 172.21.15.116:0/1849591460 learned_addr learned my addr 172.21.15.116:0/1849591460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e66ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68100970 0x7f8e68198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e677fe640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68105660 0x7f8e681a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 -- 172.21.15.116:0/1849591460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68100970 msgr2=0x7f8e68198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68100970 0x7f8e68198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 -- 172.21.15.116:0/1849591460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68105660 msgr2=0x7f8e681a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68105660 0x7f8e681a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 -- 172.21.15.116:0/1849591460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e681a4390 con 0x7f8e68101330 2024-03-19T02:52:50.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.956+0000 7f8e667fc640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e9f0 tx=0x7f8e5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.957+0000 7f8e5ffff640 1 -- 172.21.15.116:0/1849591460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000cd80 con 0x7f8e68101330 2024-03-19T02:52:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.957+0000 7f8e5ffff640 1 -- 172.21.15.116:0/1849591460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e500108e0 con 0x7f8e68101330 2024-03-19T02:52:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.957+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e681a4560 con 0x7f8e68101330 2024-03-19T02:52:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.957+0000 7f8e5ffff640 1 -- 172.21.15.116:0/1849591460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50018920 con 0x7f8e68101330 2024-03-19T02:52:50.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.957+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e681a4a80 con 0x7f8e68101330 2024-03-19T02:52:50.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e5ffff640 1 -- 172.21.15.116:0/1849591460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8e50010a90 con 0x7f8e68101330 2024-03-19T02:52:50.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 Processor -- start 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 -- start start 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 0x5650573a9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a97d0 0x5650573abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650573034b0 con 0x5650573a0bf0 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650572abd70 con 0x5650573a02d0 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e7000c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5650572ff550 con 0x5650573a97d0 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.959+0000 7f8e5ffff640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 0x5650573a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 0x5650573a9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40724/0 (socket says 172.21.15.116:40724) 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e66ffd640 1 -- 172.21.15.116:0/3519788414 learned_addr learned my addr 172.21.15.116:0/3519788414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e667fc640 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a97d0 0x5650573abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e5ffff640 1 -- 172.21.15.116:0/1849591460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f8e500cf4a0 con 0x7f8e68101330 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 -- 172.21.15.116:0/3519788414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a97d0 msgr2=0x5650573abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a97d0 0x5650573abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 -- 172.21.15.116:0/3519788414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 msgr2=0x5650573a9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 0x5650573a9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 -- 172.21.15.116:0/3519788414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650572ce210 con 0x5650573a02d0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a06b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009230 tx=0x7f8e600042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c661143905f6da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e677fe640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 0x7f8e540a4e30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8e60005ec0 tx=0x7f8e60048000 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/3519788414 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60050070 con 0x5650573a02d0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/3519788414 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e60003aa0 con 0x5650573a02d0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.960+0000 7f8e5d7fa640 1 -- 172.21.15.116:0/3519788414 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60043950 con 0x5650573a02d0 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- 172.21.15.116:0/3519788414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 msgr2=0x5650573a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a06b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009230 tx=0x7f8e600042f0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- 172.21.15.116:0/3519788414 shutdown_connections 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a97d0 0x5650573abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a0bf0 0x5650573a9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 --2- 172.21.15.116:0/3519788414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a06b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- 172.21.15.116:0/3519788414 >> 172.21.15.116:0/3519788414 conn(0x56505739aa00 msgr2=0x56505739ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- 172.21.15.116:0/3519788414 shutdown_connections 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- 172.21.15.116:0/3519788414 wait complete. 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.961+0000 7f8e7000c740 1 -- start start 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 Processor -- start 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650573ac560 con 0x5650573a97d0 2024-03-19T02:52:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650572bf310 con 0x5650573a02d0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5650573ac6e0 con 0x5650573a0bf0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 10 client.0 _ll_register_callbacks cb 0x5650572b32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e667fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40740/0 (socket says 172.21.15.116:40740) 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 -- 172.21.15.116:0/134779909 learned_addr learned my addr 172.21.15.116:0/134779909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 -- 172.21.15.116:0/134779909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 msgr2=0x5650573a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 -- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 msgr2=0x5650573a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 -- 172.21.15.116:0/134779909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650570173f0 con 0x5650573a97d0 2024-03-19T02:52:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e677fe640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009430 tx=0x7f8e60009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259baf978a18359 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60058060 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f8e60005050 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.962+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e60003aa0 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8e6005f050 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 10 client.22629 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5650570e17d0 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e60003aa0 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f8e6004a070 con 0x5650573a97d0 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 10 client.22629 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 10 client.22629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 20 client.22629 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650573b9a60 0x5650573bbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 4 client.22629 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650573613e0 con 0x5650573b9a60 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.963+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650573b9a60 0x5650573bbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e4e7fc640 10 client.22629 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650573b9a60 0x5650573bbe50 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4d8ba89ce29b02 server_cookie=39a28103d0ff1593 in_seq=0 out_seq=0 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/134779909 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650573613e0 con 0x5650573b9a60 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e4e7fc640 10 client.22629 handle_command_reply: tid=0 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 client.22629 shutdown 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 msgr2=0x5650573a7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009430 tx=0x7f8e60009260 comp rx=0 tx=0).stop 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 shutdown_connections 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650573b9a60 0x5650573bbe50 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a7350 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e4e7fc640 0 client.22629 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 --2- 172.21.15.116:0/134779909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 >> 172.21.15.116:0/134779909 conn(0x56505739aa00 msgr2=0x56505739c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 shutdown_connections 2024-03-19T02:52:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.964+0000 7f8e7000c740 1 -- 172.21.15.116:0/134779909 wait complete. 2024-03-19T02:52:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.965+0000 7f8e7000c740 20 client.22629 trim_cache size 0 max 16384 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.978+0000 7f8e7000c740 1 Processor -- start 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.978+0000 7f8e7000c740 1 -- start start 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681aeb90 con 0x5650573a97d0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681b0760 con 0x5650573a0bf0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e7000c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681af290 con 0x5650573a02d0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39806/0 (socket says 172.21.15.116:39806) 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.116:57514/0 (socket says 172.21.15.116:57514) 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40746/0 (socket says 172.21.15.116:40746) 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 -- 172.21.15.116:0/4266808954 learned_addr learned my addr 172.21.15.116:0/4266808954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 -- 172.21.15.116:0/4266808954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 msgr2=0x5650573b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 -- 172.21.15.116:0/4266808954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 msgr2=0x5650573a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e66ffd640 1 -- 172.21.15.116:0/4266808954 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650573abf30 con 0x5650573a02d0 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e667fc640 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.979+0000 7f8e677fe640 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8e58005fd0 tx=0x7f8e58003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82db8def2d1aa5db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e5cff9640 1 -- 172.21.15.116:0/4266808954 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801e030 con 0x5650573a02d0 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e5cff9640 1 -- 172.21.15.116:0/4266808954 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5801e680 con 0x5650573a02d0 2024-03-19T02:52:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e5cff9640 1 -- 172.21.15.116:0/4266808954 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e580047e0 con 0x5650573a02d0 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e7000c740 1 -- 172.21.15.116:0/4266808954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 msgr2=0x5650573a9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e7000c740 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8e58005fd0 tx=0x7f8e58003e00 comp rx=0 tx=0).stop 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.980+0000 7f8e7000c740 1 -- 172.21.15.116:0/4266808954 shutdown_connections 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x5650573a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a0bf0 0x5650573b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- 172.21.15.116:0/4266808954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a02d0 0x5650573a9000 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- 172.21.15.116:0/4266808954 >> 172.21.15.116:0/4266808954 conn(0x56505739aa00 msgr2=0x565057399ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- 172.21.15.116:0/4266808954 shutdown_connections 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- 172.21.15.116:0/4266808954 wait complete. 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- start start 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 Processor -- start 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681b0090 con 0x5650573a97d0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681af990 con 0x5650573a02d0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681acb90 con 0x5650573a0bf0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e7000c740 10 client.0 _ll_register_callbacks cb 0x7f8e6004e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40752/0 (socket says 172.21.15.116:40752) 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e677fe640 1 -- 172.21.15.116:0/2703073660 learned_addr learned my addr 172.21.15.116:0/2703073660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e667fc640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.981+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 -- 172.21.15.116:0/2703073660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 msgr2=0x5650573a4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 -- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 msgr2=0x5650573a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:50.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 -- 172.21.15.116:0/2703073660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505741efc0 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e677fe640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009900 tx=0x7f8e600049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a29915375626b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60068060 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f8e600435a0 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e5801e680 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8e6006f050 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 10 client.22635 fetch_fsmap learned FSMap version 45 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5650574201c0 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5801e680 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8378+0+0 (secure 0 0 0) 0x7f8e6004a070 con 0x5650573a97d0 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 10 client.22635 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T02:52:50.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 10 client.22635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 20 client.22635 populate_metadata read hostname 'smithi116' 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650574208b0 0x565057422cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 4 client.22635 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.982+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 --> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650573109f0 con 0x5650574208b0 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650574208b0 0x565057422cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e4e7fc640 10 client.22635 ms_handle_connect on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e66ffd640 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650574208b0 0x565057422cd0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28bd6f7536cbc3e8 server_cookie=f8e4e28d121e476 in_seq=0 out_seq=0 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e4e7fc640 1 -- 172.21.15.116:0/2703073660 <== mds.0 v2:172.21.15.154:6838/4009274564 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650573109f0 con 0x5650574208b0 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e4e7fc640 10 client.22635 handle_command_reply: tid=0 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 client.22635 shutdown 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 msgr2=0x56505741ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009900 tx=0x7f8e600049c0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 shutdown_connections 2024-03-19T02:52:50.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:6838/4009274564,v1:172.21.15.154:6839/4009274564] conn(0x5650574208b0 0x565057422cd0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650573a97d0 0x56505741ea80 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650573a0bf0 0x5650573a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e4e7fc640 0 client.22635 ms_handle_reset on v2:172.21.15.154:6838/4009274564 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 --2- 172.21.15.116:0/2703073660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650573a02d0 0x5650573a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 >> 172.21.15.116:0/2703073660 conn(0x56505739aa00 msgr2=0x565057399310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 shutdown_connections 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 1 -- 172.21.15.116:0/2703073660 wait complete. 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.983+0000 7f8e7000c740 20 client.22635 trim_cache size 0 max 16384 2024-03-19T02:52:50.992 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-19T02:52:50.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 msgr2=0x7f8e540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 0x7f8e540a4e30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8e60005ec0 tx=0x7f8e60048000 comp rx=0 tx=0).stop 2024-03-19T02:52:50.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 msgr2=0x7f8e68199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e9f0 tx=0x7f8e5000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 shutdown_connections 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e68105660 0x7f8e681a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e68101330 0x7f8e68199310 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 --2- 172.21.15.116:0/1849591460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e68100970 0x7f8e68198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 >> 172.21.15.116:0/1849591460 conn(0x7f8e680774c0 msgr2=0x7f8e680fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 shutdown_connections 2024-03-19T02:52:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:50.984+0000 7f8e6d426640 1 -- 172.21.15.116:0/1849591460 wait complete. 2024-03-19T02:52:51.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:51 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4250801201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:51.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:51 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4250801201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:51.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:51 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4250801201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:51.996 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-19T02:52:51.997 INFO:tasks.fwd_scrub:done joining 2024-03-19T02:52:51.997 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-19T02:52:52.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:52:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.121+0000 7fe965f16640 1 -- 172.21.15.116:0/2013654542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe960106ba0 msgr2=0x7fe96010e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.121+0000 7fe965f16640 1 --2- 172.21.15.116:0/2013654542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe960106ba0 0x7fe96010e720 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fe958009ca0 tx=0x7fe95802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 -- 172.21.15.116:0/2013654542 shutdown_connections 2024-03-19T02:52:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 --2- 172.21.15.116:0/2013654542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe960106ba0 0x7fe96010e720 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 --2- 172.21.15.116:0/2013654542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe960100940 0x7fe960106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 --2- 172.21.15.116:0/2013654542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9600fff90 0x7fe960100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 -- 172.21.15.116:0/2013654542 >> 172.21.15.116:0/2013654542 conn(0x7fe960076ce0 msgr2=0x7fe9600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 -- 172.21.15.116:0/2013654542 shutdown_connections 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 -- 172.21.15.116:0/2013654542 wait complete. 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 Processor -- start 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.122+0000 7fe965f16640 1 -- start start 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9600fff90 0x7fe960197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe960106ba0 0x7fe960198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe960111ab0 con 0x7fe96019c8b0 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe960111930 con 0x7fe9600fff90 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe960111c30 con 0x7fe960106ba0 2024-03-19T02:52:52.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9600fff90 0x7fe960197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40778/0 (socket says 172.21.15.116:40778) 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe960106ba0 0x7fe960198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 -- 172.21.15.116:0/3571957305 learned_addr learned my addr 172.21.15.116:0/3571957305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 -- 172.21.15.116:0/3571957305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe960106ba0 msgr2=0x7fe960198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe960106ba0 0x7fe960198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 -- 172.21.15.116:0/3571957305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9600fff90 msgr2=0x7fe960197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9600fff90 0x7fe960197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 -- 172.21.15.116:0/3571957305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96019d3a0 con 0x7fe96019c8b0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95ffff640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe95802c700 tx=0x7fe958035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe958039040 con 0x7fe96019c8b0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe958044070 con 0x7fe96019c8b0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9580042a0 con 0x7fe96019c8b0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe96019d630 con 0x7fe96019c8b0 2024-03-19T02:52:52.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.123+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9601a4df0 con 0x7fe96019c8b0 2024-03-19T02:52:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.124+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe960068a60 con 0x7fe96019c8b0 2024-03-19T02:52:52.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.125+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fe958003970 con 0x7fe96019c8b0 2024-03-19T02:52:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.126+0000 7fe95cff9640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 0x7fe9480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.126+0000 7fe95f7fe640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 0x7fe9480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.126+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7fe9580f1520 con 0x7fe96019c8b0 2024-03-19T02:52:52.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.127+0000 7fe95f7fe640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 0x7fe9480a4e30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe950009bd0 tx=0x7fe950005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.130+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9580bdfd0 con 0x7fe96019c8b0 2024-03-19T02:52:52.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.313+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe960100940 con 0x7fe96019c8b0 2024-03-19T02:52:52.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.314+0000 7fe95cff9640 1 -- 172.21.15.116:0/3571957305 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+25403 (secure 0 0 0) 0x7fe9580bd8d0 con 0x7fe96019c8b0 2024-03-19T02:52:52.316 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:52:52.316 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":133,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:52:11.968737+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":45,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":45,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2648298519":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/953095802":"2024-03-20T02:27:48.704049+0000","172.21.15.116:6801/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/2284944336":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3807729141":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/1256325072":"2024-03-20T02:27:48.704049+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":32,"length":1},{"begin":45,"length":1}]},{"pool":4,"snaps":[{"begin":32,"length":1},{"begin":45,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 msgr2=0x7fe9480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 0x7fe9480a4e30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe950009bd0 tx=0x7fe950005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 msgr2=0x7fe96019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fe95802c700 tx=0x7fe958035d60 comp rx=0 tx=0).stop 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 shutdown_connections 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fe9480a2970 0x7fe9480a4e30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe96019c8b0 0x7fe96019cd60 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe960106ba0 0x7fe960198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 --2- 172.21.15.116:0/3571957305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9600fff90 0x7fe960197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.316+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 >> 172.21.15.116:0/3571957305 conn(0x7fe960076ce0 msgr2=0x7fe9600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.317+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 shutdown_connections 2024-03-19T02:52:52.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.317+0000 7fe965f16640 1 -- 172.21.15.116:0/3571957305 wait complete. 2024-03-19T02:52:52.328 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:52 smithi116 ceph-mon[32263]: pgmap v792: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-03-19T02:52:52.329 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 45, 'snap_epoch': 132, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 45, 'snap_epoch': 132, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:52:52.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:52:52.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:52 smithi162 ceph-mon[35761]: pgmap v792: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-03-19T02:52:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 -- 172.21.15.116:0/3899433943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 msgr2=0x7f2e14106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3899433943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e14106a90 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009930 tx=0x7f2e0402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 -- 172.21.15.116:0/3899433943 shutdown_connections 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3899433943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100fd0 0x7f2e1410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3899433943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100610 0x7f2e14100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3899433943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e14106a90 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 -- 172.21.15.116:0/3899433943 >> 172.21.15.116:0/3899433943 conn(0x7f2e140fc120 msgr2=0x7f2e140fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 -- 172.21.15.116:0/3899433943 shutdown_connections 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.433+0000 7f2e13fff640 1 -- 172.21.15.116:0/3899433943 wait complete. 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 Processor -- start 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 -- start start 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100610 0x7f2e14198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100fd0 0x7f2e14199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e141108e0 con 0x7f2e141066b0 2024-03-19T02:52:52.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14110760 con 0x7f2e14100610 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e13fff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14110a60 con 0x7f2e14100fd0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40794/0 (socket says 172.21.15.116:40794) 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e0a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100fd0 0x7f2e14199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 -- 172.21.15.116:0/3317183640 learned_addr learned my addr 172.21.15.116:0/3317183640 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e12ffd640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100610 0x7f2e14198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 -- 172.21.15.116:0/3317183640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100fd0 msgr2=0x7f2e14199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100fd0 0x7f2e14199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 -- 172.21.15.116:0/3317183640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100610 msgr2=0x7f2e14198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100610 0x7f2e14198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.434+0000 7f2e137fe640 1 -- 172.21.15.116:0/3317183640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e141a43c0 con 0x7f2e141066b0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e137fe640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00eb10 tx=0x7f2e0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c004b20 con 0x7f2e141066b0 2024-03-19T02:52:52.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e0c004cc0 con 0x7f2e141066b0 2024-03-19T02:52:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c010690 con 0x7f2e141066b0 2024-03-19T02:52:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e141a4590 con 0x7f2e141066b0 2024-03-19T02:52:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.435+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141a4a60 con 0x7f2e141066b0 2024-03-19T02:52:52.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.436+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e14068980 con 0x7f2e141066b0 2024-03-19T02:52:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.437+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f2e0c005000 con 0x7f2e141066b0 2024-03-19T02:52:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.438+0000 7f2e10ff9640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 0x7f2e000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.438+0000 7f2e12ffd640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 0x7f2e000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.438+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f2e0c0cf960 con 0x7f2e141066b0 2024-03-19T02:52:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.438+0000 7f2e12ffd640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 0x7f2e000a4e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2e04000c00 tx=0x7f2e04005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.441+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e0c053740 con 0x7f2e141066b0 2024-03-19T02:52:52.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:52 smithi154 ceph-mon[35693]: pgmap v792: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-03-19T02:52:52.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.615+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2e14108b80 con 0x7f2e141066b0 2024-03-19T02:52:52.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.616+0000 7f2e10ff9640 1 -- 172.21.15.116:0/3317183640 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) ==== 93+0+10 (secure 0 0 0) 0x7f2e0c09c2e0 con 0x7f2e141066b0 2024-03-19T02:52:52.617 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:52:52.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 msgr2=0x7f2e000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 0x7f2e000a4e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2e04000c00 tx=0x7f2e04005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 msgr2=0x7f2e141a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00eb10 tx=0x7f2e0c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 shutdown_connections 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f2e000a2970 0x7f2e000a4e30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e141066b0 0x7f2e141a3e80 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e14100fd0 0x7f2e14199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 --2- 172.21.15.116:0/3317183640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e14100610 0x7f2e14198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 >> 172.21.15.116:0/3317183640 conn(0x7f2e140fc120 msgr2=0x7f2e140fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 shutdown_connections 2024-03-19T02:52:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.618+0000 7f2e13fff640 1 -- 172.21.15.116:0/3317183640 wait complete. 2024-03-19T02:52:52.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:52:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- 172.21.15.116:0/2836447532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860810bcc0 msgr2=0x7f860810e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 --2- 172.21.15.116:0/2836447532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860810bcc0 0x7f860810e110 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f8600009900 tx=0x7f860002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- 172.21.15.116:0/2836447532 shutdown_connections 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 --2- 172.21.15.116:0/2836447532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860810bcc0 0x7f860810e110 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 --2- 172.21.15.116:0/2836447532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8608103210 0x7f860810b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 --2- 172.21.15.116:0/2836447532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 0x7f8608102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- 172.21.15.116:0/2836447532 >> 172.21.15.116:0/2836447532 conn(0x7f86080fc070 msgr2=0x7f86080fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- 172.21.15.116:0/2836447532 shutdown_connections 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- 172.21.15.116:0/2836447532 wait complete. 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 Processor -- start 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 -- start start 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.735+0000 7f860e79f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 0x7f8608197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860810bcc0 0x7f860819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8608111910 con 0x7f8608103210 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8608111790 con 0x7f86081028f0 2024-03-19T02:52:52.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8608111a90 con 0x7f860810bcc0 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40822/0 (socket says 172.21.15.116:40822) 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 -- 172.21.15.116:0/1562707335 learned_addr learned my addr 172.21.15.116:0/1562707335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860df9e640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860810bcc0 0x7f860819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860d79d640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 0x7f8608197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 -- 172.21.15.116:0/1562707335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860810bcc0 msgr2=0x7f860819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860810bcc0 0x7f860819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 -- 172.21.15.116:0/1562707335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 msgr2=0x7f8608197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 0x7f8608197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 -- 172.21.15.116:0/1562707335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860819ce10 con 0x7f8608103210 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860cf9c640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e9f0 tx=0x7f85f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f800cd80 con 0x7f8608103210 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860819d100 con 0x7f8608103210 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85f80054d0 con 0x7f8608103210 2024-03-19T02:52:52.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86081a4990 con 0x7f8608103210 2024-03-19T02:52:52.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.736+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8018630 con 0x7f8608103210 2024-03-19T02:52:52.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.737+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8608068530 con 0x7f8608103210 2024-03-19T02:52:52.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.739+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f85f8004e60 con 0x7f8608103210 2024-03-19T02:52:52.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.739+0000 7f85fe7fc640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 0x7f85f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.740+0000 7f860d79d640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 0x7f85f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.740+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f85f80cf630 con 0x7f8608103210 2024-03-19T02:52:52.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.740+0000 7f860d79d640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 0x7f85f40a4e30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8604005fd0 tx=0x7f8604009500 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.744+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85f809bfb0 con 0x7f8608103210 2024-03-19T02:52:52.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.918+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8608106de0 con 0x7f8608103210 2024-03-19T02:52:52.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.919+0000 7f85fe7fc640 1 -- 172.21.15.116:0/1562707335 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v133) ==== 104+0+11 (secure 0 0 0) 0x7f85f809b8b0 con 0x7f8608103210 2024-03-19T02:52:52.920 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 msgr2=0x7f85f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 0x7f85f40a4e30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8604005fd0 tx=0x7f8604009500 comp rx=0 tx=0).stop 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 msgr2=0x7f86081983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f85f800e9f0 tx=0x7f85f800edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 shutdown_connections 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f85f40a2970 0x7f85f40a4e30 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f860810bcc0 0x7f860819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8608103210 0x7f86081983b0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 --2- 172.21.15.116:0/1562707335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86081028f0 0x7f8608197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 >> 172.21.15.116:0/1562707335 conn(0x7f86080fc070 msgr2=0x7f86080fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 shutdown_connections 2024-03-19T02:52:52.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:52.921+0000 7f860e79f640 1 -- 172.21.15.116:0/1562707335 wait complete. 2024-03-19T02:52:52.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:52:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- 172.21.15.116:0/236594777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f581066a0 msgr2=0x7f7f58106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/236594777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f581066a0 0x7f7f58106a80 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009930 tx=0x7f7f4802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- 172.21.15.116:0/236594777 shutdown_connections 2024-03-19T02:52:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/236594777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f58101050 0x7f7f5810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/236594777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 0x7f7f58100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/236594777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f581066a0 0x7f7f58106a80 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- 172.21.15.116:0/236594777 >> 172.21.15.116:0/236594777 conn(0x7f7f58077820 msgr2=0x7f7f58077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- 172.21.15.116:0/236594777 shutdown_connections 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- 172.21.15.116:0/236594777 wait complete. 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 Processor -- start 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.038+0000 7f7f5dc27640 1 -- start start 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 0x7f7f58197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f581066a0 0x7f7f5819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58111a60 con 0x7f7f58101050 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581118e0 con 0x7f7f58100690 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58111be0 con 0x7f7f581066a0 2024-03-19T02:52:53.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40844/0 (socket says 172.21.15.116:40844) 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 -- 172.21.15.116:0/4089451267 learned_addr learned my addr 172.21.15.116:0/4089451267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 0x7f7f58197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f57fff640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f581066a0 0x7f7f5819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 -- 172.21.15.116:0/4089451267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f581066a0 msgr2=0x7f7f5819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f581066a0 0x7f7f5819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 -- 172.21.15.116:0/4089451267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 msgr2=0x7f7f58197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 0x7f7f58197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 -- 172.21.15.116:0/4089451267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5819cfe0 con 0x7f7f58101050 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f56ffd640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00ea60 tx=0x7f7f4c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00cde0 con 0x7f7f58101050 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f4c0035f0 con 0x7f7f58101050 2024-03-19T02:52:53.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5819d2d0 con 0x7f7f58101050 2024-03-19T02:52:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c010690 con 0x7f7f58101050 2024-03-19T02:52:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.039+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581a4b60 con 0x7f7f58101050 2024-03-19T02:52:53.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.041+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f58068610 con 0x7f7f58101050 2024-03-19T02:52:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.041+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7f4c0037a0 con 0x7f7f58101050 2024-03-19T02:52:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.042+0000 7f7f54ff9640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 0x7f7f400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.042+0000 7f7f577fe640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 0x7f7f400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.042+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f7f4c0cf480 con 0x7f7f58101050 2024-03-19T02:52:53.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.042+0000 7f7f577fe640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 0x7f7f400a4e30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7f48005ab0 tx=0x7f7f48005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.046+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f4c09be00 con 0x7f7f58101050 2024-03-19T02:52:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.226+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7f58108c60 con 0x7f7f58101050 2024-03-19T02:52:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.226+0000 7f7f54ff9640 1 -- 172.21.15.116:0/4089451267 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v133) ==== 100+0+11 (secure 0 0 0) 0x7f7f4c09b700 con 0x7f7f58101050 2024-03-19T02:52:53.227 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:52:53.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 msgr2=0x7f7f400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 0x7f7f400a4e30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7f48005ab0 tx=0x7f7f48005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 msgr2=0x7f7f58198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00ea60 tx=0x7f7f4c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 shutdown_connections 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f7f400a2970 0x7f7f400a4e30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f581066a0 0x7f7f5819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f58101050 0x7f7f58198530 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 --2- 172.21.15.116:0/4089451267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f58100690 0x7f7f58197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 >> 172.21.15.116:0/4089451267 conn(0x7f7f58077820 msgr2=0x7f7f58102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 shutdown_connections 2024-03-19T02:52:53.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.228+0000 7f7f5dc27640 1 -- 172.21.15.116:0/4089451267 wait complete. 2024-03-19T02:52:53.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:53 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3571957305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:53.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:53 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3317183640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:53 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1562707335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/114380213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a8810be30 msgr2=0x7f5a8810e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/114380213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a8810be30 0x7f5a8810e260 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009ca0 tx=0x7f5a8002c180 comp rx=0 tx=0).stop 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/114380213 shutdown_connections 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/114380213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a8810be30 0x7f5a8810e260 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/114380213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a88103380 0x7f5a8810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/114380213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88102a60 0x7f5a88102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/114380213 >> 172.21.15.116:0/114380213 conn(0x7f5a88076aa0 msgr2=0x7f5a88076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/114380213 shutdown_connections 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.344+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/114380213 wait complete. 2024-03-19T02:52:53.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 Processor -- start 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 -- start start 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 0x7f5a88198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 0x7f5a8819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88111a70 con 0x7f5a88102a60 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881118f0 con 0x7f5a8810be30 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8e8bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88111bf0 con 0x7f5a88103380 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40856/0 (socket says 172.21.15.116:40856) 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a877fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 0x7f5a88198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 -- 172.21.15.116:0/922957471 learned_addr learned my addr 172.21.15.116:0/922957471 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8ce5c640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 0x7f5a8819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 -- 172.21.15.116:0/922957471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 msgr2=0x7f5a88198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 0x7f5a88198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 -- 172.21.15.116:0/922957471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 msgr2=0x7f5a8819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 0x7f5a8819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a87fff640 1 -- 172.21.15.116:0/922957471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8819cf90 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a877fe640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 0x7f5a88198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.345+0000 7f5a8ce5c640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 0x7f5a8819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a87fff640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800e9f0 tx=0x7f5a7800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7800cd80 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8819d280 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a780054d0 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a78010490 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.346+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881a4b10 con 0x7f5a88102a60 2024-03-19T02:52:53.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.347+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a8806a630 con 0x7f5a88102a60 2024-03-19T02:52:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.348+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5a78005000 con 0x7f5a88102a60 2024-03-19T02:52:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.348+0000 7f5a857fa640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 0x7f5a700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.349+0000 7f5a877fe640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 0x7f5a700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.349+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7f5a780cf7d0 con 0x7f5a88102a60 2024-03-19T02:52:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.349+0000 7f5a877fe640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 0x7f5a700a4e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a8810dca0 tx=0x7f5a7c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.352+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a78053740 con 0x7f5a88102a60 2024-03-19T02:52:53.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:53 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3571957305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:53.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:53 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3317183640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:53 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1562707335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:53 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3571957305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:53.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:53 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3317183640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:53 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1562707335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:53.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.526+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5a881992f0 con 0x7f5a88102a60 2024-03-19T02:52:53.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.527+0000 7f5a857fa640 1 -- 172.21.15.116:0/922957471 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v133) ==== 103+0+11 (secure 0 0 0) 0x7f5a780052f0 con 0x7f5a88102a60 2024-03-19T02:52:53.528 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:52:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.528+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 msgr2=0x7f5a700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.528+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 0x7f5a700a4e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5a8810dca0 tx=0x7f5a7c005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.528+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 msgr2=0x7f5a88197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.528+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800e9f0 tx=0x7f5a7800edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 shutdown_connections 2024-03-19T02:52:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a700a2970 0x7f5a700a4e30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a8810be30 0x7f5a8819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a88103380 0x7f5a88198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 --2- 172.21.15.116:0/922957471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88102a60 0x7f5a88197ff0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 >> 172.21.15.116:0/922957471 conn(0x7f5a88076aa0 msgr2=0x7f5a880fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 shutdown_connections 2024-03-19T02:52:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.529+0000 7f5a8e8bf640 1 -- 172.21.15.116:0/922957471 wait complete. 2024-03-19T02:52:53.541 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-19T02:52:53.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 -- 172.21.15.116:0/1683930733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1010a0 msgr2=0x7ff30c10d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1683930733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1010a0 0x7ff30c10d5b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff304009be0 tx=0x7ff30402bef0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 -- 172.21.15.116:0/1683930733 shutdown_connections 2024-03-19T02:52:53.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1683930733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1010a0 0x7ff30c10d5b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1683930733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1006e0 0x7ff30c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1683930733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 -- 172.21.15.116:0/1683930733 >> 172.21.15.116:0/1683930733 conn(0x7ff30c0777c0 msgr2=0x7ff30c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 -- 172.21.15.116:0/1683930733 shutdown_connections 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.642+0000 7ff311ef1640 1 -- 172.21.15.116:0/1683930733 wait complete. 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 Processor -- start 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 -- start start 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 0x7ff30c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1010a0 0x7ff30c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c1109b0 con 0x7ff30c1066f0 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c110830 con 0x7ff30c1006e0 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff311ef1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c110b30 con 0x7ff30c1010a0 2024-03-19T02:52:53.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40870/0 (socket says 172.21.15.116:40870) 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1010a0 0x7ff30c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 0x7ff30c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 -- 172.21.15.116:0/1660748366 learned_addr learned my addr 172.21.15.116:0/1660748366 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 -- 172.21.15.116:0/1660748366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1010a0 msgr2=0x7ff30c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1010a0 0x7ff30c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 -- 172.21.15.116:0/1660748366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 msgr2=0x7ff30c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 0x7ff30c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30bfff640 1 -- 172.21.15.116:0/1660748366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c19cf70 con 0x7ff30c1066f0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.643+0000 7ff30b7fe640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 0x7ff30c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff30bfff640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7ff304036040 tx=0x7ff304033a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304043020 con 0x7ff30c1066f0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30c19d200 con 0x7ff30c1066f0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff304033e60 con 0x7ff30c1066f0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304034430 con 0x7ff30c1066f0 2024-03-19T02:52:53.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.644+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff30c1a4af0 con 0x7ff30c1066f0 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.645+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff30c068a60 con 0x7ff30c1066f0 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.646+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7ff304046020 con 0x7ff30c1066f0 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.646+0000 7ff308ff9640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 0x7ff2f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.646+0000 7ff30b7fe640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 0x7ff2f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.646+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9510+0+0 (secure 0 0 0) 0x7ff3040f1730 con 0x7ff30c1066f0 2024-03-19T02:52:53.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.647+0000 7ff30b7fe640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 0x7ff2f40a4e30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009f40 tx=0x7ff2fc005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:53.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.651+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff30404d030 con 0x7ff30c1066f0 2024-03-19T02:52:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.873+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7ff30c109cb0 con 0x7ff30c1066f0 2024-03-19T02:52:53.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.881+0000 7ff308ff9640 1 -- 172.21.15.116:0/1660748366 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740368 (secure 0 0 0) 0x7ff3040be1c0 con 0x7ff30c1066f0 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 26, 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "active_gid": 25556, 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "active_name": "x", 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addrs": { 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6800", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2695100500 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6801", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2695100500 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addr": "172.21.15.116:6801/2695100500", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "active_change": "2024-03-19T02:27:48.704366+0000", 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "active_mgr_features": 4540701547738038271, 2024-03-19T02:52:53.887 INFO:teuthology.orchestra.run.smithi116.stdout: "available": true, 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "standbys": [ 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 25483, 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.892 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T02:52:53.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:53.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:53.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T02:52:53.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T02:52:53.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T02:52:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T02:52:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T02:52:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T02:52:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T02:52:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T02:52:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T02:52:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T02:52:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T02:52:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T02:52:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T02:52:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T02:52:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.962 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T02:52:53.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T02:52:53.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T02:52:53.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.971 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.974 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T02:52:53.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.978 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T02:52:53.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T02:52:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T02:52:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:53.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T02:52:53.989 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.992 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T02:52:53.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T02:52:53.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:53.998 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:53.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.000 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.013 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.029 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.034 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.038 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T02:52:54.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.039 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.040 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.041 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.042 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.043 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.046 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.047 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.048 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.049 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.050 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.053 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.054 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.055 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.056 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.059 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.060 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.061 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.062 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.064 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.065 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.066 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.068 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T02:52:54.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.071 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.072 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.075 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T02:52:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T02:52:54.078 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.079 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T02:52:54.081 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.083 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.086 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.089 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T02:52:54.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.094 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T02:52:54.097 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.098 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.100 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.104 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.105 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.106 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.107 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.109 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.112 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.114 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.115 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T02:52:54.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.118 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.119 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.122 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.124 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.125 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T02:52:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T02:52:54.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.128 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.129 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.130 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.131 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.134 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.138 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.140 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.141 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.150 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.151 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T02:52:54.153 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.154 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.164 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.166 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.172 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T02:52:54.174 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.175 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.177 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.178 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T02:52:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.185 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.195 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.198 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T02:52:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T02:52:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.204 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.208 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.211 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.216 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.222 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.225 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.226 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.227 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.232 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.234 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.235 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.236 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T02:52:54.237 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.238 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.239 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.240 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.241 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.242 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.243 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.244 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.246 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.250 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.252 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.253 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T02:52:54.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T02:52:54.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.257 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.259 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.266 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.267 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.268 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.269 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.271 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.272 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.278 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.279 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.280 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.282 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T02:52:54.289 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T02:52:54.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.291 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:54.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.293 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T02:52:54.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.296 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.300 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.301 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T02:52:54.302 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.303 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.305 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.307 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.308 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.309 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.310 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.314 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.315 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.316 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.318 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.319 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.320 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.324 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.325 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.327 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.330 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.331 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 25559, 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "y", 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-19T02:52:54.333 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.335 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.336 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.337 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.338 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.340 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T02:52:54.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.342 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.344 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.347 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T02:52:54.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.350 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T02:52:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T02:52:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.357 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.358 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.361 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.362 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.364 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.365 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.366 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.367 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.368 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.369 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.370 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.371 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.372 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.373 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T02:52:54.374 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.376 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.377 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T02:52:54.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T02:52:54.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T02:52:54.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T02:52:54.381 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.383 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.384 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.385 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.386 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T02:52:54.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T02:52:54.388 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.389 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.390 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.391 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T02:52:54.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.393 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.394 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.395 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.397 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.398 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.399 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T02:52:54.400 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T02:52:54.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T02:52:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.403 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.404 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.405 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.406 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.413 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.414 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.415 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T02:52:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T02:52:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.421 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.422 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T02:52:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T02:52:54.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.425 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T02:52:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.433 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.434 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.435 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.436 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T02:52:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T02:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T02:52:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T02:52:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T02:52:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T02:52:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.472 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T02:52:54.474 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.479 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.484 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T02:52:54.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.486 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.487 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.490 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.491 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.492 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.493 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.497 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T02:52:54.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T02:52:54.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T02:52:54.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T02:52:54.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T02:52:54.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T02:52:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.512 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.513 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.516 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T02:52:54.518 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T02:52:54.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.523 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T02:52:54.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.527 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.528 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.531 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T02:52:54.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T02:52:54.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T02:52:54.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T02:52:54.537 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.538 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.539 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T02:52:54.541 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.543 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.546 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.548 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.549 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.549 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.549 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.549 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.550 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T02:52:54.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.555 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T02:52:54.556 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.558 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T02:52:54.560 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.572 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T02:52:54.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T02:52:54.575 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T02:52:54.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T02:52:54.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.593 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.594 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T02:52:54.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T02:52:54.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T02:52:54.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.610 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.611 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.612 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.613 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.614 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.615 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.616 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.617 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.620 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.625 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.627 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.629 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.636 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.640 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.641 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.644 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.645 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.646 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T02:52:54.648 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.649 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T02:52:54.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:54.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.658 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.659 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.667 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.668 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.671 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.674 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T02:52:54.679 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T02:52:54.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T02:52:54.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.692 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.695 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.702 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.707 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.708 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T02:52:54.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.711 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.718 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.720 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.728 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.733 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.736 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.738 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.754 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.757 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T02:52:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.767 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.770 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.771 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.774 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.777 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.778 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.780 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.782 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T02:52:54.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T02:52:54.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T02:52:54.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T02:52:54.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T02:52:54.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.792 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.794 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.798 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "modules": [ 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard", 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "iostat", 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "nfs", 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "restful", 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: "snap_schedule" 2024-03-19T02:52:54.801 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.802 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.803 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.805 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.806 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.816 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.817 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:54.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.820 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T02:52:54.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.822 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T02:52:54.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T02:52:54.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.830 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.832 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:54.834 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.841 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T02:52:54.842 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T02:52:54.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T02:52:54.848 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T02:52:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T02:52:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.860 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.865 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.869 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.872 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T02:52:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T02:52:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T02:52:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T02:52:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T02:52:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T02:52:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T02:52:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.895 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.900 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T02:52:54.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T02:52:54.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T02:52:54.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T02:52:54.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T02:52:54.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.938 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.942 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.954 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.960 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T02:52:54.965 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:54.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.970 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.972 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.974 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T02:52:54.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T02:52:54.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T02:52:54.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.978 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.985 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:54.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:54.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T02:52:54.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.995 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:54.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:54.999 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.001 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T02:52:55.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T02:52:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T02:52:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T02:52:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T02:52:55.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.034 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.035 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.044 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.046 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.047 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.048 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.049 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.052 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.063 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.071 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.072 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T02:52:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T02:52:55.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:55.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.084 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.086 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T02:52:55.089 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.090 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.091 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.092 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.093 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.094 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.095 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.096 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.098 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:55.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:55.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:55.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T02:52:55.102 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T02:52:55.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.104 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.106 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.109 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.111 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.114 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.116 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.117 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.118 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.119 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.124 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.886+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 msgr2=0x7ff2f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.886+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 0x7ff2f40a4e30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009f40 tx=0x7ff2fc005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.886+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 msgr2=0x7ff30c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.886+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7ff304036040 tx=0x7ff304033a60 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 shutdown_connections 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7ff2f40a2970 0x7ff2f40a4e30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30c1066f0 0x7ff30c19c840 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff30c1010a0 0x7ff30c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 --2- 172.21.15.116:0/1660748366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c1006e0 0x7ff30c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 >> 172.21.15.116:0/1660748366 conn(0x7ff30c0777c0 msgr2=0x7ff30c06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 shutdown_connections 2024-03-19T02:52:55.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:52:53.887+0000 7ff311ef1640 1 -- 172.21.15.116:0/1660748366 wait complete. 2024-03-19T02:52:55.126 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.126 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.129 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.131 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.134 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.135 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T02:52:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.137 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.138 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.140 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.141 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.144 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.151 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T02:52:55.155 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.158 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.160 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.163 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.164 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.166 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.167 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.168 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.169 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.170 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.172 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.173 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T02:52:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:55.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.178 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T02:52:55.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.197 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.198 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.201 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T02:52:55.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.203 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.204 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.206 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.212 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.213 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.216 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T02:52:55.218 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T02:52:55.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.222 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T02:52:55.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T02:52:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T02:52:55.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.228 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "services": { 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard": "https://172.21.15.116:8443/" 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "always_on_modules": { 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus": [ 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T02:52:55.230 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific": [ 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T02:52:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy": [ 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T02:52:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "reef": [ 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T02:52:55.233 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "squid": [ 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T02:52:55.234 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "last_failure_osd_epoch": 95, 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "active_clients": [ 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2677381812 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "libcephsqlite", 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2456460187 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 762249954 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 47520365 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 136643018 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T02:52:55.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 302062357 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T02:52:55.239 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:55.246 DEBUG:tasks.check_counter:Getting stats from a 2024-03-19T02:52:55.246 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-19T02:52:55.246 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:54 smithi116 ceph-mon[32263]: pgmap v793: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-03-19T02:52:55.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:54 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4089451267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:54 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/922957471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:54 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1660748366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:54 smithi154 ceph-mon[35693]: pgmap v793: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:54 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4089451267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:54 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/922957471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:54 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1660748366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:54 smithi162 ceph-mon[35761]: pgmap v793: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:54 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4089451267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:54 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/922957471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:55.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:54 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1660748366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T02:52:55.494 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:56.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:56 smithi116 ceph-mon[32263]: pgmap v794: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-03-19T02:52:56.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:56 smithi162 ceph-mon[35761]: pgmap v794: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-03-19T02:52:56.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:56 smithi154 ceph-mon[35693]: pgmap v794: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-03-19T02:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2160, 2024-03-19T02:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2131, 2024-03-19T02:52:57.555 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 2367536, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 804756, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.194418731, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.066439254, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.075097273, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.050441191, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2130, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.057937715, 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000027200 2024-03-19T02:52:57.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1817168, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 283872 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1693, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2031, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 4616270, 2024-03-19T02:52:57.557 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 1301822, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 8, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.133823672, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.046819644, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.031619117, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.043591619, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2028, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.048294183, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000023813 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.558 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 764016 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1869, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1869, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 4859627, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 593519, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-19T02:52:57.559 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.123763626, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.039100377, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.035920119, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.038763564, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1868, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.044690664, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000023924 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.560 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8744, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.184087572, 2024-03-19T02:52:57.561 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000021053 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-19T02:52:57.562 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-19T02:52:57.563 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 462, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 462, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 462, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 1, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-19T02:52:57.564 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-19T02:52:57.565 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:52:57.566 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-19T02:52:57.567 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:52:57.568 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-19T02:52:57.569 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 7974, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 11, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-19T02:52:57.570 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 12764924, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 12750265, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 7974 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 12, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 775, 2024-03-19T02:52:57.571 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 763, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 10, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 34, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 24, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 462, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 816, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 354, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 54076, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.572 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.574 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.576 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.577 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.578 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.579 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.580 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 2, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-19T02:52:57.584 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 3801, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:52:57.585 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:52:57.586 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 199041, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 59, 2024-03-19T02:52:57.587 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 9640, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 201, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 392144, 2024-03-19T02:52:57.588 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2010, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 5194, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 5194, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5194, 2024-03-19T02:52:57.589 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.527958330, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000486707 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5194, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 5195 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 5194, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 5194, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-19T02:52:57.590 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 3161, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 2031, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-19T02:52:57.591 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-19T02:52:57.592 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-19T02:52:57.593 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 133, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 43, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 6, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 8, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 2, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:52:57.594 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:52:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 5718, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 11299187, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 5718, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5718, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 11299187, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 5194, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 8570625, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5194, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 8570625, 2024-03-19T02:52:57.597 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:52:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 5194, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 5194, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5194, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 5194, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:52:57.599 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x5641711966e0": { 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:52:57.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:52:57.601 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:52:58.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:52:58 smithi116 ceph-mon[32263]: pgmap v795: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:52:58.305 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-19T02:52:58.305 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-19T02:52:58.305 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-19T02:52:58.306 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:52:58.306 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:52:58.306 DEBUG:tasks.check_counter:Getting stats from d 2024-03-19T02:52:58.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-19T02:52:58.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:52:58 smithi162 ceph-mon[35761]: pgmap v795: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:52:58.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:52:58 smithi154 ceph-mon[35693]: pgmap v795: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:52:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:53:00.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:00 smithi116 ceph-mon[32263]: pgmap v796: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-19T02:53:00.326 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.001161865, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000178588, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:00.327 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 461, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 450, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1643424, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 278672, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.058805179, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.020624636, 2024-03-19T02:53:00.328 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.038156930, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.008032142, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 449, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.010876208, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000024223 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1643424, 2024-03-19T02:53:00.329 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 278672 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 7, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 347, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 3472, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 294784, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.020456362, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.015037058, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002522897, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T02:53:00.330 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 345, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.012625830, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000036596 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 294784 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:00.332 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:00.333 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:00.334 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 148714, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 30, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:00.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 464, 2024-03-19T02:53:00.336 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1578330, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 464, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 464, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1578330, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:00.337 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:53:00.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:00 smithi162 ceph-mon[35761]: pgmap v796: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:53:00.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:00 smithi154 ceph-mon[35693]: pgmap v796: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 75 KiB/s wr, 22 op/s 2024-03-19T02:53:01.098 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-19T02:53:01.099 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-19T02:53:01.099 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-19T02:53:01.099 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-19T02:53:01.099 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-19T02:53:01.099 DEBUG:tasks.check_counter:Getting stats from g 2024-03-19T02:53:01.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-19T02:53:01.341 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:53:02.419 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:02 smithi162 ceph-mon[35761]: pgmap v797: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-03-19T02:53:02.443 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:02 smithi116 ceph-mon[32263]: pgmap v797: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-03-19T02:53:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:02 smithi154 ceph-mon[35693]: pgmap v797: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 27 op/s 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1649, 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1649, 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 555657, 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 523559, 2024-03-19T02:53:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.141592600, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.052591960, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.034421394, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.042681050, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1648, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.047900050, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000029065 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 3994, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 3965, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 9946420, 2024-03-19T02:53:03.042 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 1388669, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.253306026, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.071256611, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.101504532, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.079070471, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3964, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.087255278, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000022011 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.043 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1817712, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 285152 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 6, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 345, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1828, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 852661, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 9, 2024-03-19T02:53:03.044 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.019553745, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.013778003, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002881094, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000055049, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 342, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.013006339, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000038030 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.045 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 851696 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T02:53:03.046 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8658, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.175865543, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000020312 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:03.048 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 333, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 333, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 333, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 1, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-19T02:53:03.049 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-19T02:53:03.050 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:03.051 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.052 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:03.053 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-19T02:53:03.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 7004, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 8, 2024-03-19T02:53:03.055 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 11440986, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 11433695, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 7004 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.056 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 10, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 624, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 614, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 8, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 29, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 21, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 333, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 659, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 326, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-19T02:53:03.057 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 54020, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.059 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.061 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.062 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-19T02:53:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.063 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.065 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.066 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-19T02:53:03.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.069 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.070 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 1, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 3, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 2, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 3, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 8393, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.071 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:03.072 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 198257, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 57, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 201, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 291928, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 1467, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 5118, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-19T02:53:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 5118, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5118, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.321961670, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000453685 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5118, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 5119 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 5118, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 5118, 2024-03-19T02:53:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 3173, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 1943, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-19T02:53:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-19T02:53:03.079 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-19T02:53:03.080 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 133, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 43, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 6, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 8, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 2, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-19T02:53:03.081 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-19T02:53:03.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 5645, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 9966082, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 5645, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:53:03.083 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5645, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 9966082, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:53:03.084 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 5118, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 7248597, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5118, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 7248597, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:03.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 5118, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 5118, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 5118, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 5118, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.086 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T02:53:03.087 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x5583208be6e0": { 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:03.089 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:53:03.872 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-19T02:53:03.872 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-19T02:53:03.872 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-19T02:53:03.872 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:03.873 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:03.873 DEBUG:tasks.check_counter:Getting stats from j 2024-03-19T02:53:03.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-19T02:53:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:53:04.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:04 smithi162 ceph-mon[35761]: pgmap v798: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-03-19T02:53:04.482 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:04 smithi116 ceph-mon[32263]: pgmap v798: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-03-19T02:53:04.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:04 smithi154 ceph-mon[35693]: pgmap v798: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 44 KiB/s wr, 15 op/s 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 463, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 452, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1643968, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 279952, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-19T02:53:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.056834153, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.018527263, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.037924907, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.007725695, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 451, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.010936377, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000024249 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.877 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1643968, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 279952 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 7, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 349, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 3472, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 295136, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.019729519, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.014289617, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002622625, 2024-03-19T02:53:05.878 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 347, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.015996723, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000046100 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 295136 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.879 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.000307347, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000223926, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:05.880 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.881 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:05.882 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:05.883 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 148714, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 30, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:05.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T02:53:05.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 466, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1578590, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 466, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T02:53:05.886 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 466, 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1578590, 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T02:53:05.887 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T02:53:06.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:06 smithi162 ceph-mon[35761]: pgmap v799: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 36 op/s 2024-03-19T02:53:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:06 smithi154 ceph-mon[35693]: pgmap v799: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 36 op/s 2024-03-19T02:53:06.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:06 smithi116 ceph-mon[32263]: pgmap v799: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 89 KiB/s wr, 36 op/s 2024-03-19T02:53:06.567 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-19T02:53:06.567 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-19T02:53:06.567 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-19T02:53:06.567 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-19T02:53:06.568 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-19T02:53:06.568 DEBUG:tasks.check_counter:Getting stats from b 2024-03-19T02:53:06.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-19T02:53:07.187 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:53:08.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:08 smithi154 ceph-mon[35693]: pgmap v800: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:08.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:08 smithi162 ceph-mon[35761]: pgmap v800: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:08 smithi116 ceph-mon[32263]: pgmap v800: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:10.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:10 smithi154 ceph-mon[35693]: pgmap v801: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:10.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:10 smithi116 ceph-mon[32263]: pgmap v801: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 2034405, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 2251793, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 605467174, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 2060667368, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 23, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 91.903527581, 2024-03-19T02:53:10.543 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 39.146856312, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 62.208802434, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 3.027830854, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2251785, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 52.173397603, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000023169 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 4093034, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.558626683, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000136 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 45680, 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 673760 2024-03-19T02:53:10.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 118544, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 161659, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 32877124, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 529267123, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 12, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 10.447920171, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 4.166212764, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 3.959456299, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 2.467319466, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 161662, 2024-03-19T02:53:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 6.271399247, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000038793 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 178350, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.030071292, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000168 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 852416, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 416704 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 93892, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 142245, 2024-03-19T02:53:10.546 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 23586953, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 427142083, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 376, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 8.600066449, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 3.896829670, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 2.986113913, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 1.645914466, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 142245, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 5.128543592, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000036054 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 154068, 2024-03-19T02:53:10.547 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.027530725, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000178 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1926720, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 361840 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 112180, 2024-03-19T02:53:10.548 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 193.627239081, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001726040 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 63833, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 18.858977821, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000295442 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-19T02:53:10.549 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 1796024, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 1788207, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1788207, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 5931.831457987, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003317195 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 7817, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 5, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 2, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 375, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 21, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 4, 2024-03-19T02:53:10.550 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 9065, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 3481, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 5573, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 11, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 4005, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 120, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 3, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 3, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 5, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 3413741, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 3402021, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 1234, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:10.551 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 8, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 7, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 110, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 20, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 737, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 17300, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 4, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 234, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-19T02:53:10.552 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 17300, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 2897, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 46, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 2851, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 14403, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 8, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 117416, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 161521, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 171033, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 41990, 2024-03-19T02:53:10.553 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 1442, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 121298 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 37, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 93, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 103, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 33194, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 33091, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:10.554 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 46, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 7, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 25, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.555 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 205128, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 153171, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 147481, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 57647, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 5690, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 383, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 255, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 243, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 140, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 5, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 12, 2024-03-19T02:53:10.556 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 876685567, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 1439823905, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 55274, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 172.090425228, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003113406 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 7631, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 40879, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 33248, 2024-03-19T02:53:10.557 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 282, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 331, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 49, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 9065, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 45563, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 36498, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 3, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 33300, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 33297, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 114792, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 231804 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-19T02:53:10.558 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 1891415, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 3823, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 1796024, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 3732, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 9111, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 33171, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 133.413039661, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004021978 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1596730, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 4760.009006972, 2024-03-19T02:53:10.559 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002981098 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9901, 2024-03-19T02:53:10.560 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 81.689326578, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.008250613 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-19T02:53:10.561 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 26, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.020123225, 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000773970 2024-03-19T02:53:10.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 117, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.683141659, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005838817 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 26, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.705700589, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.104065407 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.564 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 19776, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 83.675832890, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004231180 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 86921, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 650.941571134, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.007488887 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7195, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 33.316421128, 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004630496 2024-03-19T02:53:10.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 114, 2024-03-19T02:53:10.566 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.087512957, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.009539587 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 20, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.346748503, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.017337425 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-03-19T02:53:10.567 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.001343919, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000671959 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1108, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3.744727446, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003379717 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.568 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.025401091, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002822343 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.569 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 33091, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 180.171560235, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005444729 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.570 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 7, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 5, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 97737, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 48868, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 1523, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:10.571 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:10.572 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:10.573 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 1144475, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 314, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:10.574 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 201, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 23790312, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 106557, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-19T02:53:10.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 150097, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 1446710794, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 37, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 150097, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 150097, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 585.767808786, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003902595 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-19T02:53:10.576 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 150097, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 155329 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 150097, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 20215, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 129882, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 87, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 1174, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 2862, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 58090, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 1492, 2024-03-19T02:53:10.577 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 20, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 67253, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 17353, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 1827, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:10.578 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-19T02:53:10.579 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 134, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 49, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 12, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-19T02:53:10.580 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 3638, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 14, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 1519 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.581 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 66182, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 33091, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 2246837, 2024-03-19T02:53:10.582 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 459579004, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 2246837, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 2246837, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 459579004, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:10.583 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 150097, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 1450352801, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 150097, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1450352801, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 150097, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 150097, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 150097, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 150097, 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 33091, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 3642095, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 33091, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:10.586 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 2816, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 3642095, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x560cc60d4500": { 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 136, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 205128, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1435629601, 2024-03-19T02:53:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 205128, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 55274, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1435629465, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:10.588 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-19T02:53:10.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:10 smithi162 ceph-mon[35761]: pgmap v801: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 47 KiB/s wr, 25 op/s 2024-03-19T02:53:11.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:11 smithi154 ceph-mon[35693]: osdmap e134: 12 total, 12 up, 12 in 2024-03-19T02:53:11.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:11 smithi116 ceph-mon[32263]: osdmap e134: 12 total, 12 up, 12 in 2024-03-19T02:53:11.630 INFO:tasks.check_counter:Daemon mds.b mds.exported=7 2024-03-19T02:53:11.631 INFO:tasks.check_counter:Daemon mds.b mds.imported=20 2024-03-19T02:53:11.631 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-19T02:53:11.631 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=26 2024-03-19T02:53:11.631 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=20 2024-03-19T02:53:11.631 DEBUG:tasks.check_counter:Getting stats from e 2024-03-19T02:53:11.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-19T02:53:11.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:11 smithi162 ceph-mon[35761]: osdmap e134: 12 total, 12 up, 12 in 2024-03-19T02:53:11.975 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:53:13.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:12 smithi154 ceph-mon[35693]: pgmap v803: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:12 smithi116 ceph-mon[32263]: pgmap v803: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:13.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:12 smithi162 ceph-mon[35761]: pgmap v803: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 3950, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 3921, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 409678633, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1376256, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 7, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 3, 2024-03-19T02:53:14.589 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.503475981, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.098065167, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.309135538, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.085975721, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3920, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.096336801, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000024575 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1820816, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 289328 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 2287, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 2631, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 555419242, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1582186, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 10, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.443675548, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.053590224, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.327511407, 2024-03-19T02:53:14.591 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.047765032, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2628, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.061965404, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000023578 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 855808 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.592 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 2243, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 2243, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 474544028, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 712457, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 6, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.419416318, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.036822310, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.323150682, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.046419551, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2242, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.047506016, 2024-03-19T02:53:14.593 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000021189 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-19T02:53:14.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 12523, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.343026550, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000027391 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.595 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 0, 2024-03-19T02:53:14.596 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 42154, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 19173, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 22981, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 14764, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 2, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-03-19T02:53:14.597 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 4, 2024-03-19T02:53:14.598 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 234, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 0, 2024-03-19T02:53:14.599 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 0 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-19T02:53:14.600 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:14.601 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 205127, 2024-03-19T02:53:14.602 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 140, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 18, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 876685567, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 1439823769, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 1439823709, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.603 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 205127 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 7627, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 40824, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 33197, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 278, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 317, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 39, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 42154, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 42264, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 110, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 0, 2024-03-19T02:53:14.604 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 109884, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 215420 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.609 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.610 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.611 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.612 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.614 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.615 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.616 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 7, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 5, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 4, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 0, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 0, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 10147, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:14.617 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:14.618 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:14.619 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 241010, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 83, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:14.620 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9880, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 47213152, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 212171, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:14.621 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 7941, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 26, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 4, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 7941, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7941, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 7.490805764, 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000943307 2024-03-19T02:53:14.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7941, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 7942 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 7941, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 7941, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 0, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 4538, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 0, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 3400, 2024-03-19T02:53:14.623 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 2, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:14.624 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-19T02:53:14.625 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 134, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 44, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 11, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 11, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 0, 2024-03-19T02:53:14.626 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 2, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 0 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 0, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T02:53:14.627 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 8476, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1438848882, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 8476, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:14.628 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 8476, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1438848882, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:14.629 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 7941, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 1435652303, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 7941, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1435652303, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:14.630 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 7941, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 7941, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 7941, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 7941, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-19T02:53:14.632 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x55c81bfa66e0": { 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:14.633 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:14.634 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-19T02:53:15.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:14 smithi154 ceph-mon[35693]: pgmap v804: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:15.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:14 smithi116 ceph-mon[32263]: pgmap v804: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:15.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:14 smithi162 ceph-mon[35761]: pgmap v804: 129 pgs: 129 active+clean; 1.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-03-19T02:53:15.549 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-19T02:53:15.549 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-19T02:53:15.549 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-19T02:53:15.550 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:15.550 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:15.550 DEBUG:tasks.check_counter:Getting stats from h 2024-03-19T02:53:15.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-19T02:53:15.866 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:53:17.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:16 smithi154 ceph-mon[35693]: pgmap v805: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T02:53:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:16 smithi116 ceph-mon[32263]: pgmap v805: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T02:53:17.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:16 smithi162 ceph-mon[35761]: pgmap v805: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T02:53:17.842 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 376663, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 421446, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 113714781, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 353526830, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 62, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 20.357927212, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 8.544050561, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 13.988864734, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.385132490, 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:17.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 421445, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 8.941612289, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000021216 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 765434, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.127799493, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000166 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1855360, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 338240 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:17.844 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 23971, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 25893, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 6877091, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 98741523, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 56, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 1.584931725, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.578424449, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.630783762, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.369978882, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 25890, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.946592764, 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000036562 2024-03-19T02:53:17.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 21725, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.003721503, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000171 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 775888 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 70226, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 51106, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 28995579, 2024-03-19T02:53:17.846 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 82831032, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 9, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 4.142297098, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 1.824317397, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 2.470287631, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.374871581, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 51105, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.286406299, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000025171 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 95906, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.015378468, 2024-03-19T02:53:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000160 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 6160, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 16160 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 34348, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 30.102780852, 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000876405 2024-03-19T02:53:17.848 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 18486, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.542173759, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000083423 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 327898, 2024-03-19T02:53:17.849 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 325148, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 325148, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 274.936539692, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000845573 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 2750, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 20, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 1, 2024-03-19T02:53:17.850 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 952, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 64, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 832, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 56, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 766, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 249, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 1, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 1, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 4, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 625067, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 616214, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 450, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 73, 2024-03-19T02:53:17.851 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 3, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 85, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 11, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 213, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 1552, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 4, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 234, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-19T02:53:17.852 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 1552, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 220, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 220, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 1332, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 23297, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 32169, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:17.853 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 34352, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 9707, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 979, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 25381 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 2, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 30, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 20, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 86, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 34, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 10, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:17.854 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 7241, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 7231, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 3, 2024-03-19T02:53:17.855 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 1, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 46581, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 46581, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 74, 2024-03-19T02:53:17.856 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 74, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 5, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 249285930, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 16148, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 42.111008868, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002607815 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.857 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 808, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 8194, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 7386, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 45, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 83, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 38, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 952, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 9386, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 8434, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 1, 2024-03-19T02:53:17.858 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 7277, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 7276, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 69372, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 223612 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 350877, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 3061, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 327898, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 941, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 8015, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7206, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 11.776881640, 2024-03-19T02:53:17.859 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001634316 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 287940, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 91.654164569, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000318309 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-19T02:53:17.860 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1660, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 4.001931119, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002410801 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-19T02:53:17.861 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-03-19T02:53:17.863 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.004656642, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002328321 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-19T02:53:17.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3583, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3.258511384, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000909436 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 16056, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 41.887911544, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002608863 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1240, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 102.326018727, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.082520982 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.865 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.085833554, 2024-03-19T02:53:17.866 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.008583355 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 220, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.097276071, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000442163 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-19T02:53:17.868 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.869 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7231, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 19.843354442, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002744206 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-19T02:53:17.870 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 3, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 27351, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 13675, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 1527, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:17.871 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:17.872 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-19T02:53:17.873 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 397655, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 149, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:17.874 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9880, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 2670955, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 11418, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.875 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 37916, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 246573588, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 29, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 37916, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 37916, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 118.081938201, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003114303 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 37916, 2024-03-19T02:53:17.876 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 39877 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 37916, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 3260, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 34656, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 10, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 1708, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 17856, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 1031, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-19T02:53:17.877 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 1, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 14532, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 1552, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 20, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:17.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-19T02:53:17.879 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 135, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 45, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 12, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 2162, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 1005 2024-03-19T02:53:17.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-19T02:53:17.881 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 14462, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 7231, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 470856, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 107173961, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 470856, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:17.882 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 470856, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 107173961, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:17.883 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 37916, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 247365463, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 37916, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 247365463, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:17.884 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 37916, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 37916, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 37916, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 37916, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.885 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 7231, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 791875, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 7231, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:17.886 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1708, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 791875, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x556ba7798500": { 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 46581, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 245091626, 2024-03-19T02:53:17.887 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 46581, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 16148, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 245091626, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:17.888 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-19T02:53:18.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:18 smithi154 ceph-mon[35693]: osdmap e135: 12 total, 12 up, 12 in 2024-03-19T02:53:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:17 smithi116 ceph-mon[32263]: osdmap e135: 12 total, 12 up, 12 in 2024-03-19T02:53:18.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:18 smithi162 ceph-mon[35761]: osdmap e135: 12 total, 12 up, 12 in 2024-03-19T02:53:18.548 INFO:tasks.check_counter:Daemon mds.h mds.exported=3 2024-03-19T02:53:18.549 INFO:tasks.check_counter:Daemon mds.h mds.imported=11 2024-03-19T02:53:18.549 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-19T02:53:18.549 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:18.549 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:18.549 DEBUG:tasks.check_counter:Getting stats from k 2024-03-19T02:53:18.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-19T02:53:18.827 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:53:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:19 smithi154 ceph-mon[35693]: pgmap v807: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-19T02:53:19.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:19 smithi116 ceph-mon[32263]: pgmap v807: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-19T02:53:19.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:19 smithi162 ceph-mon[35761]: pgmap v807: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 383 B/s wr, 13 op/s 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 1636, 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1638, 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 96901984, 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 520312, 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 9, 2024-03-19T02:53:20.738 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.166554068, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.033461010, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.089718876, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.034408492, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1636, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.038233506, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000023370 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.739 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 1548, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1894, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 59117246, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1263049, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 8, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.154274916, 2024-03-19T02:53:20.740 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.036583817, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.071199597, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.035991444, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1892, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.044946800, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000023756 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-19T02:53:20.741 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 772688 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 3392, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 3363, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 92583280, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1200112, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 5, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.312941530, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.076698799, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.153523096, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.078059056, 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:20.742 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3362, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.080842990, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000024046 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1822256, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 291904 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-19T02:53:20.743 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10293, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.204735479, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000019890 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 8183, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 67, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 8116, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 8088, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 1, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 0, 2024-03-19T02:53:20.746 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 0, 2024-03-19T02:53:20.747 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:20.748 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 0 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 0, 2024-03-19T02:53:20.749 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:20.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 46581, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 74, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 5, 2024-03-19T02:53:20.751 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 249285930, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 249285869, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 46581 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 806, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 8151, 2024-03-19T02:53:20.752 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 7345, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 35, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 54, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 19, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 8183, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 8285, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 102, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 0, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 0, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 0, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 65012, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 215420 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 0, 2024-03-19T02:53:20.753 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.754 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.755 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.758 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-19T02:53:20.759 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.760 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.762 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-19T02:53:20.763 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-19T02:53:20.764 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 3, 2024-03-19T02:53:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 3, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 4770, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:20.767 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:20.768 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 233697, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 80, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9880, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:20.769 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 7707187, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 37274, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 6034, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 21, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 3, 2024-03-19T02:53:20.770 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 6034, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 6034, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3.521908551, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000583677 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 6034, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 6035 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 6034, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 6034, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 0, 2024-03-19T02:53:20.771 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 3220, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 2812, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 1, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-19T02:53:20.773 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-19T02:53:20.774 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 135, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 45, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 10, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 11, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 1, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 0, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 2, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 0 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-19T02:53:20.775 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 0, 2024-03-19T02:53:20.776 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 6572, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 247980684, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 6572, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 6572, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 247980684, 2024-03-19T02:53:20.777 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:20.778 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 6034, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 245110060, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 6034, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 245110060, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:20.779 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 6034, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 6034, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 6034, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 6034, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-19T02:53:20.781 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x5556e34c66e0": { 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:20.782 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 0, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 0, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-19T02:53:20.783 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-19T02:53:21.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:21 smithi154 ceph-mon[35693]: pgmap v808: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-19T02:53:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:21 smithi116 ceph-mon[32263]: pgmap v808: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-19T02:53:21.417 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:21 smithi162 ceph-mon[35761]: pgmap v808: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-19T02:53:21.477 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-19T02:53:21.477 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-19T02:53:21.477 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-19T02:53:21.477 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:21.477 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:21.477 DEBUG:tasks.check_counter:Getting stats from c 2024-03-19T02:53:21.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-19T02:53:21.789 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:53:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:23 smithi154 ceph-mon[35693]: pgmap v809: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:23.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:23 smithi116 ceph-mon[32263]: pgmap v809: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:23.284 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:23 smithi162 ceph-mon[35761]: pgmap v809: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 10056, 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 11836, 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 4725703, 2024-03-19T02:53:23.698 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 13962805, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 16, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.738125303, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.286703523, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.440763674, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.075707544, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 11835, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.300406923, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000025382 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 15335, 2024-03-19T02:53:23.699 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.003065451, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000199 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1843808, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 320944 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1001, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1436, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 248885, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1723687, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 56, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:23.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.097212103, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.035373247, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.031766483, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.022037926, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1434, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.063895533, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000044557 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 183, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000030878, 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000168 2024-03-19T02:53:23.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 8400, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 806080 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 56501, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 30882, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 30276995, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 11903624, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 56, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 2.527282336, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.954362287, 2024-03-19T02:53:23.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 2.038624384, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.038159774, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 30880, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.733590074, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023756 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 84296, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.012772636, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000151 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:23.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4070, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.550223763, 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000380890 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2341, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.100386848, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000042882 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 5062, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 4704, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4704, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 10.775263295, 2024-03-19T02:53:23.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002290659 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 358, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 20, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 1, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 15, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 1, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 14, 2024-03-19T02:53:23.706 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 14, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 392, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 2, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 2, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 3, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 8447, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 4408, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 354, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 14, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 9, 2024-03-19T02:53:23.707 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 338, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 2, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 21, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 556, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 556, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:23.708 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 556, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 333, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 1314, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 1668, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 1049, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 232, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 1393 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.709 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 29, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 20, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 25, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 453, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 453, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:23.710 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 9, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 1, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T02:53:23.711 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 7974, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 7974, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 11, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 11, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T02:53:23.712 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 12764924, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1621, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.221502351, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002604258 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 20, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 796, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 776, 2024-03-19T02:53:23.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 18, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 73, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 55, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 15, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 1281, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 1266, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 2, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 459, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 457, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 59912, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.714 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 7774, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 4, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 5062, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 36, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 921, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 432, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3.756372727, 2024-03-19T02:53:23.715 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.008695307 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.001326279, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000331569 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.716 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 574, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.190501098, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000331883 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.717 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.718 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.719 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.720 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 396, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.290742489, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000734198 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2827, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.552351687, 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000549116 2024-03-19T02:53:23.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 18, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.182875090, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.010159727 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:23.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:23.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.725 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 453, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.801093925, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.010598441 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:23.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 2, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 2, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 676, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 338, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 1534, 2024-03-19T02:53:23.727 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:23.728 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:23.729 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 291001, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 113, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:23.730 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 201, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 166205, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 503, 2024-03-19T02:53:23.731 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 4485, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 8873558, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 4485, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4485, 2024-03-19T02:53:23.732 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 11.663767856, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002600617 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4485, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 5056 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 4485, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 892, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 3593, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-03-19T02:53:23.733 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 336, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 1957, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 274, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 1, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 920, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 556, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-03-19T02:53:23.734 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T02:53:23.735 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 135, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 45, 2024-03-19T02:53:23.736 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 12, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 17, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 5, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 703, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 279 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T02:53:23.737 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 5, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 906, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 453, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:23.738 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 67554, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 29137081, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 67554, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 67554, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 29137081, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 4485, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 8924287, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 4485, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 8924287, 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:23.740 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 4485, 2024-03-19T02:53:23.741 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 4485, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 4485, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 4485, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:23.742 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 453, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 50729, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 453, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 336, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 50729, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:23.743 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55bb53ffc500": { 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 7974, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 8570620, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 7974, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1621, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 8570620, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:23.744 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:23.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:23.745 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T02:53:24.546 INFO:tasks.check_counter:Daemon mds.c mds.exported=9 2024-03-19T02:53:24.546 INFO:tasks.check_counter:Daemon mds.c mds.imported=2 2024-03-19T02:53:24.546 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-19T02:53:24.547 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:24.547 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:24.547 DEBUG:tasks.check_counter:Getting stats from f 2024-03-19T02:53:24.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-19T02:53:24.875 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:53:25.165 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:25 smithi162 ceph-mon[35761]: pgmap v810: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:25.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:25 smithi116 ceph-mon[32263]: pgmap v810: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:25 smithi154 ceph-mon[35693]: pgmap v810: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-19T02:53:26.750 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T02:53:26.750 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:26.750 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 55356, 2024-03-19T02:53:26.750 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 30406, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 29484771, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 12056633, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 56, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 2.485802590, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.944151972, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 1.955575438, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.056453788, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 30404, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.707559029, 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023271 2024-03-19T02:53:26.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 80618, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.012856020, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000159 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 641104 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 6781, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 8108, 2024-03-19T02:53:26.752 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 2085716, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 8662877, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 64, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.459897981, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.179844802, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.331133958, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.012546190, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8105, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.204911000, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000025282 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 13431, 2024-03-19T02:53:26.753 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.002219390, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000165 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 135280 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1940, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1756, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 2249363, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 3861854, 2024-03-19T02:53:26.754 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 10, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.180913492, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.070632120, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.079737930, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.038425413, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1756, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.082842675, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000047176 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 345, 2024-03-19T02:53:26.755 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000052741, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000152 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1840928, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 315568 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3243, 2024-03-19T02:53:26.756 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.312253797, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000404641 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1804, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.071583216, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000039680 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 4572, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 4209, 2024-03-19T02:53:26.757 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4209, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 8.389142900, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001993143 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 363, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 20, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 1, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 11, 2024-03-19T02:53:26.758 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 1, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 10, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 10, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 357, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 2, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 7123, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 3981, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 363, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 13, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:26.759 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 8, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 314, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 1, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 20, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 511, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-03-19T02:53:26.760 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 511, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 511, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 253, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 1032, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 1438, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 878, 2024-03-19T02:53:26.761 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 177, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 1012 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 32, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 22, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 25, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 326, 2024-03-19T02:53:26.762 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 326, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 8, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 1, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T02:53:26.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 7004, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 7004, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 8, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 8, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T02:53:26.764 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 11440986, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1283, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.250062679, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.003312597 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 16, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 648, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 632, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 13, 2024-03-19T02:53:26.765 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 71, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 58, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 11, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 1076, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 1065, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 327, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 327, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 59676, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 6524, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 2, 2024-03-19T02:53:26.766 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 4572, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 26, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 663, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 306, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2.781696700, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.009090512 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.767 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 500, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.191123144, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000382246 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.768 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.769 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T02:53:26.770 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T02:53:26.771 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 378, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.278785481, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000737527 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T02:53:26.772 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2681, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.479938550, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000552009 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 18, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.158354963, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.008797497 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:26.773 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.775 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.776 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 326, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3.499244062, 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.010733877 2024-03-19T02:53:26.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 3, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 2, 2024-03-19T02:53:26.778 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 620, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 620, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 1550, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:26.779 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:26.780 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T02:53:26.781 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 285713, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 118, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:26.782 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 201, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 144784, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 391, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T02:53:26.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 3553, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 7516456, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 29, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 3553, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3553, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 10.059799657, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002831353 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3553, 2024-03-19T02:53:26.784 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3988 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 3553, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 770, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 2783, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 259, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 1542, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 241, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T02:53:26.785 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 1, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 665, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 511, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T02:53:26.786 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T02:53:26.787 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 135, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 45, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 12, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 17, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 5, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 532, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-03-19T02:53:26.788 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 207 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:26.789 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 5, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 652, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 326, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 64073, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 28019556, 2024-03-19T02:53:26.790 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 64073, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 64073, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 28019556, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:26.791 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3553, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 7553238, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3553, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 7553238, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3553, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3553, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3553, 2024-03-19T02:53:26.793 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3553, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 326, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 36782, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:26.794 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 326, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 259, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 36782, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x556832c7e500": { 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 7004, 2024-03-19T02:53:26.795 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 7246682, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 7004, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1283, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 7246682, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:26.796 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T02:53:27.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:27 smithi162 ceph-mon[35761]: pgmap v811: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-19T02:53:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:27 smithi154 ceph-mon[35693]: pgmap v811: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-19T02:53:27.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:27 smithi116 ceph-mon[32263]: pgmap v811: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-19T02:53:27.594 INFO:tasks.check_counter:Daemon mds.f mds.exported=8 2024-03-19T02:53:27.594 INFO:tasks.check_counter:Daemon mds.f mds.imported=1 2024-03-19T02:53:27.594 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-19T02:53:27.594 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:27.594 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:27.594 DEBUG:tasks.check_counter:Getting stats from i 2024-03-19T02:53:27.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-19T02:53:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:53:29.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:29 smithi162 ceph-mon[35761]: pgmap v812: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:53:29.307 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:29 smithi154 ceph-mon[35693]: pgmap v812: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:53:29.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:29 smithi116 ceph-mon[32263]: pgmap v812: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 50183, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 27196, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 25276677, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 15253417, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 62, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:29.700 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 2.276974327, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.864963146, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 1.739144008, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.090411038, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 27198, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.676298101, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000024865 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 70348, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.011373428, 2024-03-19T02:53:29.701 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000161 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 9643, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 12559, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 3004162, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 11082272, 2024-03-19T02:53:29.702 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 58, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 7, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.625783201, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.256739465, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.417767991, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.018450136, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 12552, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.288251025, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000022964 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:29.703 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 20169, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.003111128, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000154 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 136720 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 13439, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 12294, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 6604695, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 4864518, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 10, 2024-03-19T02:53:29.704 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.727156590, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.279118290, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.508612375, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.042938700, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 12294, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.286130838, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023274 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 21284, 2024-03-19T02:53:29.705 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.002924067, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000137 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1858832, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 959904 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T02:53:29.706 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5579, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.986667096, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000356097 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2427, 2024-03-19T02:53:29.707 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.098321250, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000040511 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 6448, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 6048, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 6048, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 15.207023299, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002514388 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.708 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 400, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 20, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 1, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 13, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 1, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 12, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 12, 2024-03-19T02:53:29.709 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 278, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 2, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 2, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 3, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 12815, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 8324, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 398, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 40, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T02:53:29.710 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 8, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 155, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 1, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 11, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 250, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-03-19T02:53:29.711 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 250, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 250, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 266, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 983, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 2517, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:29.712 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 2152, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 356, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 221, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 2145 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 30, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 20, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 51, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 60, 2024-03-19T02:53:29.713 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 539, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 539, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T02:53:29.714 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 8, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 1, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T02:53:29.715 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 7610, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 7610, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 10, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 10, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T02:53:29.716 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 13301238, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2497, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 7.098628554, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002842862 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 18, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 767, 2024-03-19T02:53:29.717 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 749, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 16, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 72, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 56, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 13, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 1037, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 1024, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 2, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 575, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 573, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 61440, 2024-03-19T02:53:29.718 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 10749, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 560, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 6448, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 55, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 1851, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 529, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.706612789, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.008897188 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.719 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 93, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.181680000, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001953548 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.720 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1368, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.696233364, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000508942 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T02:53:29.721 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.722 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T02:53:29.723 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.724 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.724 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T02:53:29.724 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1030, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.721345443, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000700335 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2300, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.283030006, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000557839 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.726 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 189, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2.346041841, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.012412919 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.727 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:29.729 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.730 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 539, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 5.272079856, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.009781224 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 3, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 2, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 818, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 818, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 1554, 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:29.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:29.734 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:29.735 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:29.736 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 322119, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 121, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:29.737 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 201, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 147624, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 470, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T02:53:29.738 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 5227, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 9420979, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 5227, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5227, 2024-03-19T02:53:29.739 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 14.226716399, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002721774 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5227, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 5780 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 5227, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 582, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 4645, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T02:53:29.740 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 332, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 2829, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 272, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 1, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 1092, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:29.741 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 250, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:29.742 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T02:53:29.743 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 135, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 45, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 12, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 17, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 5, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 659, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-03-19T02:53:29.744 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 315 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T02:53:29.745 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 5, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 1078, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 539, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:29.746 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 73261, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 28257633, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 73261, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 73261, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 28257633, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.747 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 5227, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 9480330, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 5227, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 9480330, 2024-03-19T02:53:29.748 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:29.749 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 5227, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 5227, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 5227, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 5227, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 539, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 59351, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 539, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:29.751 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 332, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 59351, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55fee43f6500": { 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:29.752 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 7610, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 9106934, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 7610, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2497, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 9106934, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:29.753 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:29.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:29.754 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T02:53:30.458 INFO:tasks.check_counter:Daemon mds.i mds.exported=8 2024-03-19T02:53:30.459 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-03-19T02:53:30.459 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-19T02:53:30.459 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:30.459 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:30.459 DEBUG:tasks.check_counter:Getting stats from l 2024-03-19T02:53:30.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-19T02:53:30.726 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:53:31.142 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:31 smithi162 ceph-mon[35761]: pgmap v813: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:31.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:31 smithi154 ceph-mon[35693]: pgmap v813: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:31.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:31 smithi116 ceph-mon[32263]: pgmap v813: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 2212, 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2183, 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 2385200, 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 825838, 2024-03-19T02:53:32.553 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 5, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 1, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.196197436, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.060770535, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.081351825, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.053666209, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2182, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.062076807, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000028449 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.554 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1823072, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 293824 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 90, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 438, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 23954, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 890232, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 8, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-19T02:53:32.555 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.021579684, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.012228853, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.003873086, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.002427060, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 436, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.017050944, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000039107 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 863776 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3471, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 3473, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 9993584, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1103785, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 5, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.198433733, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.056868364, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.054084503, 2024-03-19T02:53:32.557 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.076660166, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3471, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.073526137, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000021182 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T02:53:32.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8862, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.188509950, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000021271 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T02:53:32.559 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 548, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 548, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 548, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-03-19T02:53:32.561 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 1, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-19T02:53:32.562 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T02:53:32.563 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.564 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T02:53:32.565 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.566 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 7610, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 10, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T02:53:32.567 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 13301238, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 13293674, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 7610 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 12, 2024-03-19T02:53:32.568 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 732, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 720, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 10, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 31, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 21, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 548, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 715, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 167, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 55760, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215420 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.569 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.570 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T02:53:32.571 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.572 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.573 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T02:53:32.574 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.574 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.574 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.574 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.576 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.577 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.578 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 3, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 2, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 2, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 10184, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T02:53:32.582 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T02:53:32.583 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T02:53:32.584 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 191025, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 56, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T02:53:32.585 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9640, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 201, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 452688, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 2354, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T02:53:32.586 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 5227, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 21, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 3, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 5227, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5227, 2024-03-19T02:53:32.587 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2.722946037, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000520938 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5227, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 5228 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 5227, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 5227, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 3231, 2024-03-19T02:53:32.588 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 1994, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 1, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-03-19T02:53:32.589 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T02:53:32.590 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 135, 2024-03-19T02:53:32.591 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 45, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 5, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 6, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 1, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T02:53:32.592 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T02:53:32.593 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 5769, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 11852983, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 5769, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 5769, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 11852983, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 5227, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 9111894, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 5227, 2024-03-19T02:53:32.595 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 9111894, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:32.596 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 5227, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 5227, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 5227, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 5227, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:32.597 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T02:53:32.598 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x5638d1f206e0": { 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T02:53:32.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T02:53:32.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T02:53:32.600 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T02:53:33.288 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:33 smithi162 ceph-mon[35761]: pgmap v814: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-19T02:53:33.290 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-19T02:53:33.290 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-19T02:53:33.290 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-19T02:53:33.291 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-19T02:53:33.291 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T02:53:33.291 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-19T02:53:33.312 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-19T02:53:33.312 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:53:33.312 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:53:33.417 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:53:33.418 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:53:33.478 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:33.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:33 smithi154 ceph-mon[35693]: pgmap v814: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-19T02:53:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:33 smithi116 ceph-mon[32263]: pgmap v814: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-19T02:53:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- 172.21.15.116:0/838163277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 msgr2=0x7f641c106b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 --2- 172.21.15.116:0/838163277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c106b40 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f6414009cd0 tx=0x7f641402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- 172.21.15.116:0/838163277 shutdown_connections 2024-03-19T02:53:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 --2- 172.21.15.116:0/838163277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c101160 0x7f641c10d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 --2- 172.21.15.116:0/838163277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c1007a0 0x7f641c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 --2- 172.21.15.116:0/838163277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c106b40 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- 172.21.15.116:0/838163277 >> 172.21.15.116:0/838163277 conn(0x7f641c077880 msgr2=0x7f641c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- 172.21.15.116:0/838163277 shutdown_connections 2024-03-19T02:53:33.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- 172.21.15.116:0/838163277 wait complete. 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 Processor -- start 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.625+0000 7f6423e14640 1 -- start start 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 0x7f641c10e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 0x7f641c10e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f641c111b20 con 0x7f641c106760 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1119a0 con 0x7f641c101160 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f641c111ca0 con 0x7f641c1007a0 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6421bb0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 0x7f641c10e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44122/0 (socket says 172.21.15.116:44122) 2024-03-19T02:53:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 -- 172.21.15.116:0/984447820 learned_addr learned my addr 172.21.15.116:0/984447820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64213af640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 0x7f641c10e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 -- 172.21.15.116:0/984447820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 msgr2=0x7f641c10e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 0x7f641c10e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 -- 172.21.15.116:0/984447820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 msgr2=0x7f641c10e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 0x7f641c10e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 -- 172.21.15.116:0/984447820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c1a6580 con 0x7f641c106760 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64213af640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 0x7f641c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6421bb0640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 0x7f641c10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f64223b1640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f641800e970 tx=0x7f641800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641800ccc0 con 0x7f641c106760 2024-03-19T02:53:33.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f641800ce60 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641c1a6870 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418010430 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.626+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641c1a6dc0 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.627+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641c068a60 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.628+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f6418004f90 con 0x7f641c106760 2024-03-19T02:53:33.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.629+0000 7f640effd640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 0x7f64080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.629+0000 7f6421bb0640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 0x7f64080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.629+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f64180cf5b0 con 0x7f641c106760 2024-03-19T02:53:33.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.629+0000 7f6421bb0640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 0x7f64080a4e30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6414004af0 tx=0x7f6414005c50 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:33.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.634+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f641809bf30 con 0x7f641c106760 2024-03-19T02:53:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.811+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f641c108cc0 con 0x7f641c106760 2024-03-19T02:53:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.812+0000 7f640effd640 1 -- 172.21.15.116:0/984447820 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+25403 (secure 0 0 0) 0x7f641809b830 con 0x7f641c106760 2024-03-19T02:53:33.814 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:53:33.814 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":135,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:53:16.985543+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2648298519":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/953095802":"2024-03-20T02:27:48.704049+0000","172.21.15.116:6801/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/2284944336":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3807729141":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/1256325072":"2024-03-20T02:27:48.704049+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 msgr2=0x7f64080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 0x7f64080a4e30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6414004af0 tx=0x7f6414005c50 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 msgr2=0x7f641c10eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f641800e970 tx=0x7f641800ed70 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 shutdown_connections 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f64080a2970 0x7f64080a4e30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c106760 0x7f641c10eb30 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f641c101160 0x7f641c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 --2- 172.21.15.116:0/984447820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f641c1007a0 0x7f641c10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 >> 172.21.15.116:0/984447820 conn(0x7f641c077880 msgr2=0x7f641c103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 shutdown_connections 2024-03-19T02:53:33.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.814+0000 7f6423e14640 1 -- 172.21.15.116:0/984447820 wait complete. 2024-03-19T02:53:33.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:33.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 -- 172.21.15.116:0/3338896605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1006d0 msgr2=0x7f0f0c100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 --2- 172.21.15.116:0/3338896605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c100b50 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009930 tx=0x7f0ef802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 -- 172.21.15.116:0/3338896605 shutdown_connections 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 --2- 172.21.15.116:0/3338896605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c101090 0x7f0f0c10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 --2- 172.21.15.116:0/3338896605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c100b50 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 --2- 172.21.15.116:0/3338896605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 -- 172.21.15.116:0/3338896605 >> 172.21.15.116:0/3338896605 conn(0x7f0f0c076aa0 msgr2=0x7f0f0c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 -- 172.21.15.116:0/3338896605 shutdown_connections 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 -- 172.21.15.116:0/3338896605 wait complete. 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.931+0000 7f0f12d69640 1 Processor -- start 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 -- start start 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c101090 0x7f0f0c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c111af0 con 0x7f0f0c1066e0 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c111970 con 0x7f0f0c1006d0 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c111c70 con 0x7f0f0c101090 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f10b05640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44136/0 (socket says 172.21.15.116:44136) 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 -- 172.21.15.116:0/1043069212 learned_addr learned my addr 172.21.15.116:0/1043069212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f0bfff640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c101090 0x7f0f0c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 -- 172.21.15.116:0/1043069212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c101090 msgr2=0x7f0f0c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c101090 0x7f0f0c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 -- 172.21.15.116:0/1043069212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 msgr2=0x7f0f0c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:33.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 -- 172.21.15.116:0/1043069212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c19f240 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f10b05640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f11306640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400eb10 tx=0x7f0f0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04004b20 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c19f530 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f04004cc0 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.932+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04010690 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.933+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1a6dc0 con 0x7f0f0c1066e0 2024-03-19T02:53:33.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.934+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c101e10 con 0x7f0f0c1066e0 2024-03-19T02:53:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.935+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f0f04005020 con 0x7f0f0c1066e0 2024-03-19T02:53:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.935+0000 7f0f09ffb640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 0x7f0efc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.935+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f0f040cf7c0 con 0x7f0f0c1066e0 2024-03-19T02:53:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.935+0000 7f0f10b05640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 0x7f0efc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:33.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.936+0000 7f0f10b05640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 0x7f0efc0a4e30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009f40 tx=0x7f0f00005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:33.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:33.939+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f0409c140 con 0x7f0f0c1066e0 2024-03-19T02:53:34.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.116+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0f0c19b5a0 con 0x7f0f0c1066e0 2024-03-19T02:53:34.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.116+0000 7f0f09ffb640 1 -- 172.21.15.116:0/1043069212 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) ==== 93+0+10 (secure 0 0 0) 0x7f0f0409ba40 con 0x7f0f0c1066e0 2024-03-19T02:53:34.117 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:53:34.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 msgr2=0x7f0efc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 0x7f0efc0a4e30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009f40 tx=0x7f0f00005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 msgr2=0x7f0f0c19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400eb10 tx=0x7f0f0400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 shutdown_connections 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f0efc0a2970 0x7f0efc0a4e30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0c1066e0 0x7f0f0c19eb10 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c101090 0x7f0f0c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 --2- 172.21.15.116:0/1043069212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c1006d0 0x7f0f0c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 >> 172.21.15.116:0/1043069212 conn(0x7f0f0c076aa0 msgr2=0x7f0f0c10bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 shutdown_connections 2024-03-19T02:53:34.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.118+0000 7f0f12d69640 1 -- 172.21.15.116:0/1043069212 wait complete. 2024-03-19T02:53:34.129 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:34 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/984447820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:34.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- 172.21.15.116:0/593248934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0693e0 msgr2=0x7f786c10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 --2- 172.21.15.116:0/593248934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0693e0 0x7f786c10e680 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f7864009cd0 tx=0x7f786402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- 172.21.15.116:0/593248934 shutdown_connections 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 --2- 172.21.15.116:0/593248934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0693e0 0x7f786c10e680 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 --2- 172.21.15.116:0/593248934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 0x7f786c068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 --2- 172.21.15.116:0/593248934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c107870 0x7f786c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- 172.21.15.116:0/593248934 >> 172.21.15.116:0/593248934 conn(0x7f786c077820 msgr2=0x7f786c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- 172.21.15.116:0/593248934 shutdown_connections 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- 172.21.15.116:0/593248934 wait complete. 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 Processor -- start 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.233+0000 7f787231b640 1 -- start start 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 0x7f786c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c0693e0 0x7f786c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f786c111a70 con 0x7f786c107870 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f786c1118f0 con 0x7f786c068a20 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f786c111bf0 con 0x7f786c0693e0 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44140/0 (socket says 172.21.15.116:44140) 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 -- 172.21.15.116:0/201994068 learned_addr learned my addr 172.21.15.116:0/201994068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f786bfff640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 0x7f786c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f786b7fe640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c0693e0 0x7f786c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 -- 172.21.15.116:0/201994068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c0693e0 msgr2=0x7f786c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c0693e0 0x7f786c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 -- 172.21.15.116:0/201994068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 msgr2=0x7f786c19a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 0x7f786c19a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 -- 172.21.15.116:0/201994068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c19f140 con 0x7f786c107870 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78708b8640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f7864004af0 tx=0x7f7864035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864039040 con 0x7f786c107870 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c19f3d0 con 0x7f786c107870 2024-03-19T02:53:34.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7864044070 con 0x7f786c107870 2024-03-19T02:53:34.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78640042a0 con 0x7f786c107870 2024-03-19T02:53:34.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.234+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1a6cc0 con 0x7f786c107870 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.235+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c19b4a0 con 0x7f786c107870 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.237+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f7864003970 con 0x7f786c107870 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.238+0000 7f78697fa640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 0x7f78580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.238+0000 7f786bfff640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 0x7f78580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.238+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f78640f1b50 con 0x7f786c107870 2024-03-19T02:53:34.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.238+0000 7f786bfff640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 0x7f78580a4e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7854009bd0 tx=0x7f7854005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.241+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78640be5e0 con 0x7f786c107870 2024-03-19T02:53:34.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.415+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f786c1a6ef0 con 0x7f786c107870 2024-03-19T02:53:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.416+0000 7f78697fa640 1 -- 172.21.15.116:0/201994068 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v135) ==== 104+0+11 (secure 0 0 0) 0x7f78640bdee0 con 0x7f786c107870 2024-03-19T02:53:34.417 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:53:34.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:34 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/984447820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:34.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 msgr2=0x7f78580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 0x7f78580a4e30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7854009bd0 tx=0x7f7854005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:34.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 msgr2=0x7f786c19ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f7864004af0 tx=0x7f7864035ce0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 shutdown_connections 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f78580a2970 0x7f78580a4e30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c107870 0x7f786c19ea10 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f786c0693e0 0x7f786c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 --2- 172.21.15.116:0/201994068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f786c068a20 0x7f786c19a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 >> 172.21.15.116:0/201994068 conn(0x7f786c077820 msgr2=0x7f786c10cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 shutdown_connections 2024-03-19T02:53:34.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.418+0000 7f787231b640 1 -- 172.21.15.116:0/201994068 wait complete. 2024-03-19T02:53:34.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:34.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:34 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/984447820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 -- 172.21.15.116:0/1106012438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1066a0 msgr2=0x7f5b2c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 --2- 172.21.15.116:0/1106012438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c106a80 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009930 tx=0x7f5b1c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 -- 172.21.15.116:0/1106012438 shutdown_connections 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 --2- 172.21.15.116:0/1106012438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c101050 0x7f5b2c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 --2- 172.21.15.116:0/1106012438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c100690 0x7f5b2c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.534+0000 7f5b32aac640 1 --2- 172.21.15.116:0/1106012438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c106a80 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- 172.21.15.116:0/1106012438 >> 172.21.15.116:0/1106012438 conn(0x7f5b2c077820 msgr2=0x7f5b2c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- 172.21.15.116:0/1106012438 shutdown_connections 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- 172.21.15.116:0/1106012438 wait complete. 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 Processor -- start 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- start start 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c101050 0x7f5b2c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c110960 con 0x7f5b2c100690 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c1107e0 con 0x7f5b2c101050 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b32aac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c110ae0 con 0x7f5b2c1066a0 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b30848640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b30848640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44148/0 (socket says 172.21.15.116:44148) 2024-03-19T02:53:34.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b30848640 1 -- 172.21.15.116:0/22872834 learned_addr learned my addr 172.21.15.116:0/22872834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b2bfff640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c101050 0x7f5b2c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.535+0000 7f5b31049640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 -- 172.21.15.116:0/22872834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1066a0 msgr2=0x7f5b2c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 -- 172.21.15.116:0/22872834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c101050 msgr2=0x7f5b2c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c101050 0x7f5b2c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 -- 172.21.15.116:0/22872834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c1a64a0 con 0x7f5b2c100690 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b30848640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009230 tx=0x7f5b1c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c043020 con 0x7f5b2c100690 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c1a6730 con 0x7f5b2c100690 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b1c02ca70 con 0x7f5b2c100690 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c003a80 con 0x7f5b2c100690 2024-03-19T02:53:34.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.536+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c1a6c80 con 0x7f5b2c100690 2024-03-19T02:53:34.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.537+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c068610 con 0x7f5b2c100690 2024-03-19T02:53:34.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.538+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5b1c046020 con 0x7f5b2c100690 2024-03-19T02:53:34.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.539+0000 7f5b29ffb640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 0x7f5b140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.539+0000 7f5b2bfff640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 0x7f5b140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.539+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f5b1c0f14e0 con 0x7f5b2c100690 2024-03-19T02:53:34.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.539+0000 7f5b2bfff640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 0x7f5b140a4e30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c198d50 tx=0x7f5b20005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.542+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b1c04d020 con 0x7f5b2c100690 2024-03-19T02:53:34.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.720+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5b2c02d1f0 con 0x7f5b2c100690 2024-03-19T02:53:34.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.720+0000 7f5b29ffb640 1 -- 172.21.15.116:0/22872834 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v135) ==== 100+0+11 (secure 0 0 0) 0x7f5b1c02cc20 con 0x7f5b2c100690 2024-03-19T02:53:34.722 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 msgr2=0x7f5b140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 0x7f5b140a4e30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c198d50 tx=0x7f5b20005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 msgr2=0x7f5b2c196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009230 tx=0x7f5b1c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 shutdown_connections 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5b140a2970 0x7f5b140a4e30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b2c1066a0 0x7f5b2c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b2c101050 0x7f5b2c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 --2- 172.21.15.116:0/22872834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b2c100690 0x7f5b2c196bf0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 >> 172.21.15.116:0/22872834 conn(0x7f5b2c077820 msgr2=0x7f5b2c102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 shutdown_connections 2024-03-19T02:53:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.722+0000 7f5b32aac640 1 -- 172.21.15.116:0/22872834 wait complete. 2024-03-19T02:53:34.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 -- 172.21.15.116:0/3762370623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4106750 msgr2=0x7fd2a4106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/3762370623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4106750 0x7fd2a4106b30 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009930 tx=0x7fd2a002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 -- 172.21.15.116:0/3762370623 shutdown_connections 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/3762370623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4101100 0x7fd2a410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/3762370623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4100740 0x7fd2a4100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/3762370623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4106750 0x7fd2a4106b30 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 -- 172.21.15.116:0/3762370623 >> 172.21.15.116:0/3762370623 conn(0x7fd2a4077880 msgr2=0x7fd2a4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 -- 172.21.15.116:0/3762370623 shutdown_connections 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.838+0000 7fd2abbc6640 1 -- 172.21.15.116:0/3762370623 wait complete. 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 Processor -- start 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 -- start start 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4101100 0x7fd2a41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4106750 0x7fd2a41a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4110a10 con 0x7fd2a4100740 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4110890 con 0x7fd2a4106750 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4110b90 con 0x7fd2a4101100 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44166/0 (socket says 172.21.15.116:44166) 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 -- 172.21.15.116:0/1019975405 learned_addr learned my addr 172.21.15.116:0/1019975405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9161640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4101100 0x7fd2a41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2aa163640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4106750 0x7fd2a41a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 -- 172.21.15.116:0/1019975405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4101100 msgr2=0x7fd2a41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4101100 0x7fd2a41993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 -- 172.21.15.116:0/1019975405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4106750 msgr2=0x7fd2a41a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:34.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4106750 0x7fd2a41a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 -- 172.21.15.116:0/1019975405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a41a4450 con 0x7fd2a4100740 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2a9962640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009230 tx=0x7fd2a002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0043020 con 0x7fd2a4100740 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2a002ca70 con 0x7fd2a4100740 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0003a50 con 0x7fd2a4100740 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.839+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a41a4620 con 0x7fd2a4100740 2024-03-19T02:53:34.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.840+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41a4b60 con 0x7fd2a4100740 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.840+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a4101e80 con 0x7fd2a4100740 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.841+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd2a0046020 con 0x7fd2a4100740 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.842+0000 7fd29affd640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 0x7fd2900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.842+0000 7fd2a9161640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 0x7fd2900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.842+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7fd2a00f14b0 con 0x7fd2a4100740 2024-03-19T02:53:34.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.842+0000 7fd2a9161640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 0x7fd2900a4eb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd2a406a9a0 tx=0x7fd294005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:34.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:34.846+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2a004d020 con 0x7fd2a4100740 2024-03-19T02:53:35.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.022+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd2a4108cb0 con 0x7fd2a4100740 2024-03-19T02:53:35.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.022+0000 7fd29affd640 1 -- 172.21.15.116:0/1019975405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v135) ==== 103+0+11 (secure 0 0 0) 0x7fd2a00bdf40 con 0x7fd2a4100740 2024-03-19T02:53:35.024 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:53:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 msgr2=0x7fd2900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 0x7fd2900a4eb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd2a406a9a0 tx=0x7fd294005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 msgr2=0x7fd2a4198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009230 tx=0x7fd2a002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 shutdown_connections 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd2900a29f0 0x7fd2900a4eb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a4106750 0x7fd2a41a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2a4101100 0x7fd2a41993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 --2- 172.21.15.116:0/1019975405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a4100740 0x7fd2a4198e90 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 >> 172.21.15.116:0/1019975405 conn(0x7fd2a4077880 msgr2=0x7fd2a41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 shutdown_connections 2024-03-19T02:53:35.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.024+0000 7fd2abbc6640 1 -- 172.21.15.116:0/1019975405 wait complete. 2024-03-19T02:53:35.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:53:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 -- 172.21.15.116:0/3272006762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9941006b0 msgr2=0x7fa994100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 --2- 172.21.15.116:0/3272006762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9941006b0 0x7fa994100b30 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009930 tx=0x7fa97c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 -- 172.21.15.116:0/3272006762 shutdown_connections 2024-03-19T02:53:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 --2- 172.21.15.116:0/3272006762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa994101070 0x7fa99410d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 --2- 172.21.15.116:0/3272006762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9941006b0 0x7fa994100b30 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 --2- 172.21.15.116:0/3272006762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941066c0 0x7fa994106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 -- 172.21.15.116:0/3272006762 >> 172.21.15.116:0/3272006762 conn(0x7fa9940777f0 msgr2=0x7fa994077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 -- 172.21.15.116:0/3272006762 shutdown_connections 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.138+0000 7fa998a9a640 1 -- 172.21.15.116:0/3272006762 wait complete. 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 Processor -- start 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 -- start start 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941006b0 0x7fa994196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9941066c0 0x7fa9941976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110980 con 0x7fa994101070 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110800 con 0x7fa9941066c0 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa998a9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110b00 con 0x7fa9941006b0 2024-03-19T02:53:35.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44170/0 (socket says 172.21.15.116:44170) 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa99259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941006b0 0x7fa994196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 -- 172.21.15.116:0/685353515 learned_addr learned my addr 172.21.15.116:0/685353515 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa992d9d640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9941066c0 0x7fa9941976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 -- 172.21.15.116:0/685353515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941006b0 msgr2=0x7fa994196c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941006b0 0x7fa994196c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 -- 172.21.15.116:0/685353515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9941066c0 msgr2=0x7fa9941976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9941066c0 0x7fa9941976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 -- 172.21.15.116:0/685353515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9941a64e0 con 0x7fa994101070 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.139+0000 7fa98bfff640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009230 tx=0x7fa97c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.140+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c043020 con 0x7fa994101070 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.140+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9941a6770 con 0x7fa994101070 2024-03-19T02:53:35.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.140+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa97c02ca70 con 0x7fa994101070 2024-03-19T02:53:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.140+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c003a50 con 0x7fa994101070 2024-03-19T02:53:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.140+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a6d10 con 0x7fa994101070 2024-03-19T02:53:35.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.141+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa994101df0 con 0x7fa994101070 2024-03-19T02:53:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.142+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fa97c046020 con 0x7fa994101070 2024-03-19T02:53:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.142+0000 7fa98b7fe640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 0x7fa9800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.142+0000 7fa99259c640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 0x7fa9800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.142+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7fa97c0f1520 con 0x7fa994101070 2024-03-19T02:53:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.143+0000 7fa99259c640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 0x7fa9800a4e30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa984009b50 tx=0x7fa984005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.146+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa97c04d020 con 0x7fa994101070 2024-03-19T02:53:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.332+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa99406a420 con 0x7fa994101070 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.332+0000 7fa98b7fe640 1 -- 172.21.15.116:0/685353515 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v135) ==== 81+0+1016 (secure 0 0 0) 0x7fa97c0bdfb0 con 0x7fa994101070 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3573130182 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3953610795 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:35.334 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2583747917 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/996139070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/707840070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2020074950 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1993666663 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2648298519 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/953095802 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3573130182 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2284944336 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3807729141 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1256325072 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:listed 17 entries 2024-03-19T02:53:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 msgr2=0x7fa9800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 0x7fa9800a4e30 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa984009b50 tx=0x7fa984005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 msgr2=0x7fa9941971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009230 tx=0x7fa97c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 shutdown_connections 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fa9800a2970 0x7fa9800a4e30 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9941066c0 0x7fa9941976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994101070 0x7fa9941971b0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 --2- 172.21.15.116:0/685353515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa9941006b0 0x7fa994196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.334+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 >> 172.21.15.116:0/685353515 conn(0x7fa9940777f0 msgr2=0x7fa994103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.335+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 shutdown_connections 2024-03-19T02:53:35.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.335+0000 7fa998a9a640 1 -- 172.21.15.116:0/685353515 wait complete. 2024-03-19T02:53:35.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:35 smithi116 ceph-mon[32263]: pgmap v815: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:35.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1043069212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/201994068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/22872834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:35 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1019975405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.347 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T02:53:35.347 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:53:35.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:35 smithi162 ceph-mon[35761]: pgmap v815: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1043069212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/201994068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/22872834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:35 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1019975405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- 172.21.15.116:0/1235323235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 msgr2=0x7fb518101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 --2- 172.21.15.116:0/1235323235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518101c40 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fb508009930 tx=0x7fb50802bc20 comp rx=0 tx=0).stop 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- 172.21.15.116:0/1235323235 shutdown_connections 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 --2- 172.21.15.116:0/1235323235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 0x7fb51810e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 --2- 172.21.15.116:0/1235323235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518101c40 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 --2- 172.21.15.116:0/1235323235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 0x7fb518107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- 172.21.15.116:0/1235323235 >> 172.21.15.116:0/1235323235 conn(0x7fb5180fd2b0 msgr2=0x7fb5180ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- 172.21.15.116:0/1235323235 shutdown_connections 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- 172.21.15.116:0/1235323235 wait complete. 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 Processor -- start 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.503+0000 7fb51df3b640 1 -- start start 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 0x7fb518198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 0x7fb51819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb518111a90 con 0x7fb5181017c0 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb518111910 con 0x7fb518102180 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb518111c10 con 0x7fb5181077d0 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44176/0 (socket says 172.21.15.116:44176) 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 -- 172.21.15.116:0/3627629616 learned_addr learned my addr 172.21.15.116:0/3627629616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:35.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb517fff640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 0x7fb51819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb516ffd640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 0x7fb518198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 -- 172.21.15.116:0/3627629616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 msgr2=0x7fb51819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 0x7fb51819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 -- 172.21.15.116:0/3627629616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 msgr2=0x7fb518198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 0x7fb518198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 -- 172.21.15.116:0/3627629616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51819d010 con 0x7fb5181017c0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb5177fe640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e9f0 tx=0x7fb50c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c00cd80 con 0x7fb5181017c0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb50c0054d0 con 0x7fb5181017c0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51819d300 con 0x7fb5181017c0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c010490 con 0x7fb5181017c0 2024-03-19T02:53:35.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.504+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5181a4b90 con 0x7fb5181017c0 2024-03-19T02:53:35.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:35 smithi154 ceph-mon[35693]: pgmap v815: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:35.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1043069212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/201994068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/22872834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:35 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1019975405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.505+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb518068610 con 0x7fb5181017c0 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.507+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb50c005000 con 0x7fb5181017c0 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.507+0000 7fb514ff9640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 0x7fb5000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.507+0000 7fb516ffd640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 0x7fb5000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.507+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7fb50c0cf750 con 0x7fb5181017c0 2024-03-19T02:53:35.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.508+0000 7fb516ffd640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 0x7fb5000a4e30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb508005ab0 tx=0x7fb508005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.511+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb50c053740 con 0x7fb5181017c0 2024-03-19T02:53:35.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.686+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb518199370 con 0x7fb5181017c0 2024-03-19T02:53:35.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.688+0000 7fb514ff9640 1 -- 172.21.15.116:0/3627629616 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+25403 (secure 0 0 0) 0x7fb50c0052f0 con 0x7fb5181017c0 2024-03-19T02:53:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:53:35.690 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":135,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:53:16.985543+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2648298519":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/953095802":"2024-03-20T02:27:48.704049+0000","172.21.15.116:6801/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/2284944336":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3807729141":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/1256325072":"2024-03-20T02:27:48.704049+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 msgr2=0x7fb5000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 0x7fb5000a4e30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb508005ab0 tx=0x7fb508005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 msgr2=0x7fb518198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e9f0 tx=0x7fb50c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 shutdown_connections 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb5000a2970 0x7fb5000a4e30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5181077d0 0x7fb51819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb518102180 0x7fb518198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 --2- 172.21.15.116:0/3627629616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5181017c0 0x7fb518198020 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 >> 172.21.15.116:0/3627629616 conn(0x7fb5180fd2b0 msgr2=0x7fb5180fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 shutdown_connections 2024-03-19T02:53:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.690+0000 7fb51df3b640 1 -- 172.21.15.116:0/3627629616 wait complete. 2024-03-19T02:53:35.703 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:20:31.845523+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:25:06.506384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:25:07.182320+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:25:10.435819+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 134, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:35.703 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 -- 172.21.15.116:0/660872585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 msgr2=0x7f8aec10e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 --2- 172.21.15.116:0/660872585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec10e6a0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4009cd0 tx=0x7f8ae402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 -- 172.21.15.116:0/660872585 shutdown_connections 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 --2- 172.21.15.116:0/660872585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec10e6a0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 --2- 172.21.15.116:0/660872585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1017d0 0x7f8aec101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 --2- 172.21.15.116:0/660872585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1077e0 0x7f8aec107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 -- 172.21.15.116:0/660872585 >> 172.21.15.116:0/660872585 conn(0x7f8aec0fd260 msgr2=0x7f8aec0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 -- 172.21.15.116:0/660872585 shutdown_connections 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.808+0000 7f8af30fe640 1 -- 172.21.15.116:0/660872585 wait complete. 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 Processor -- start 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 -- start start 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1017d0 0x7f8aec197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1077e0 0x7f8aec19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8aec111aa0 con 0x7f8aec102190 2024-03-19T02:53:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8aec111920 con 0x7f8aec1077e0 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af30fe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8aec111c20 con 0x7f8aec1017d0 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44208/0 (socket says 172.21.15.116:44208) 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 -- 172.21.15.116:0/1590776164 learned_addr learned my addr 172.21.15.116:0/1590776164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af0e9a640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1017d0 0x7f8aec197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8af169b640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1077e0 0x7f8aec19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 -- 172.21.15.116:0/1590776164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1017d0 msgr2=0x7f8aec197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1017d0 0x7f8aec197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 -- 172.21.15.116:0/1590776164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1077e0 msgr2=0x7f8aec19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1077e0 0x7f8aec19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 -- 172.21.15.116:0/1590776164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aec19cfd0 con 0x7f8aec102190 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8aebfff640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800e9f0 tx=0x7f8ad800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800cd80 con 0x7f8aec102190 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.809+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ad80108e0 con 0x7f8aec102190 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.810+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8018920 con 0x7f8aec102190 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.810+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aec19d2c0 con 0x7f8aec102190 2024-03-19T02:53:35.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.810+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aec1a4b50 con 0x7f8aec102190 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.811+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aec068a60 con 0x7f8aec102190 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.812+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8ad8010a90 con 0x7f8aec102190 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.812+0000 7f8ae9ffb640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 0x7f8adc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.813+0000 7f8af0e9a640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 0x7f8adc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.813+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f8ad80cf4c0 con 0x7f8aec102190 2024-03-19T02:53:35.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.813+0000 7f8af0e9a640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 0x7f8adc0a4eb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009b50 tx=0x7f8ae0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:35.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.817+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ad809b7f0 con 0x7f8aec102190 2024-03-19T02:53:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.992+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8aec02d1f0 con 0x7f8aec102190 2024-03-19T02:53:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.993+0000 7f8ae9ffb640 1 -- 172.21.15.116:0/1590776164 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) ==== 93+0+10 (secure 0 0 0) 0x7f8ad809f890 con 0x7f8aec102190 2024-03-19T02:53:35.994 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T02:53:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 msgr2=0x7f8adc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 0x7f8adc0a4eb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009b50 tx=0x7f8ae0005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 msgr2=0x7f8aec198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800e9f0 tx=0x7f8ad800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 shutdown_connections 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8adc0a29f0 0x7f8adc0a4eb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aec1077e0 0x7f8aec19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aec102190 0x7f8aec198520 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 --2- 172.21.15.116:0/1590776164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aec1017d0 0x7f8aec197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 >> 172.21.15.116:0/1590776164 conn(0x7f8aec0fd260 msgr2=0x7f8aec0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 shutdown_connections 2024-03-19T02:53:35.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:35.995+0000 7f8af30fe640 1 -- 172.21.15.116:0/1590776164 wait complete. 2024-03-19T02:53:36.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 -- 172.21.15.116:0/3109450420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd01005b0 msgr2=0x7f8fd0100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/3109450420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd01005b0 0x7f8fd0100a30 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009930 tx=0x7f8fc402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 -- 172.21.15.116:0/3109450420 shutdown_connections 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/3109450420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0100f70 0x7f8fd010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/3109450420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd01005b0 0x7f8fd0100a30 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/3109450420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd0106760 0x7f8fd0106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 -- 172.21.15.116:0/3109450420 >> 172.21.15.116:0/3109450420 conn(0x7f8fd00fc080 msgr2=0x7f8fd00fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 -- 172.21.15.116:0/3109450420 shutdown_connections 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.111+0000 7f8fd59ff640 1 -- 172.21.15.116:0/3109450420 wait complete. 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 Processor -- start 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 -- start start 2024-03-19T02:53:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd01005b0 0x7f8fd0196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0106760 0x7f8fd019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0110950 con 0x7f8fd0100f70 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd01107d0 con 0x7f8fd01005b0 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fd59ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0110ad0 con 0x7f8fd0106760 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fce7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fce7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44216/0 (socket says 172.21.15.116:44216) 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fce7fc640 1 -- 172.21.15.116:0/829610576 learned_addr learned my addr 172.21.15.116:0/829610576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fceffd640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd01005b0 0x7f8fd0196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.112+0000 7f8fcf7fe640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0106760 0x7f8fd019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 -- 172.21.15.116:0/829610576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0106760 msgr2=0x7f8fd019b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0106760 0x7f8fd019b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 -- 172.21.15.116:0/829610576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd01005b0 msgr2=0x7f8fd0196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd01005b0 0x7f8fd0196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 -- 172.21.15.116:0/829610576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd019be60 con 0x7f8fd0100f70 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fce7fc640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009230 tx=0x7f8fc402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4043020 con 0x7f8fd0100f70 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc402ca70 con 0x7f8fd0100f70 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd019c0f0 con 0x7f8fd0100f70 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003a50 con 0x7f8fd0100f70 2024-03-19T02:53:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.113+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd0069df0 con 0x7f8fd0100f70 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.114+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd0101cf0 con 0x7f8fd0100f70 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.115+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8fc4046020 con 0x7f8fd0100f70 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.116+0000 7f8fd49fd640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 0x7f8fb80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.116+0000 7f8fceffd640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 0x7f8fb80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.116+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f8fc40f1110 con 0x7f8fd0100f70 2024-03-19T02:53:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.116+0000 7f8fceffd640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 0x7f8fb80a4e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00095f0 tx=0x7f8fc0010040 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.120+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fc404d020 con 0x7f8fd0100f70 2024-03-19T02:53:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/685353515' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3627629616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:36.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:36 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1590776164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.296+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8fd01981c0 con 0x7f8fd0100f70 2024-03-19T02:53:36.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.296+0000 7f8fd49fd640 1 -- 172.21.15.116:0/829610576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v135) ==== 104+0+11 (secure 0 0 0) 0x7f8fc40bdba0 con 0x7f8fd0100f70 2024-03-19T02:53:36.298 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 msgr2=0x7f8fb80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 0x7f8fb80a4e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00095f0 tx=0x7f8fc0010040 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 msgr2=0x7f8fd01973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009230 tx=0x7f8fc402c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 shutdown_connections 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8fb80a2970 0x7f8fb80a4e30 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd0106760 0x7f8fd019b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fd0100f70 0x7f8fd01973b0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 --2- 172.21.15.116:0/829610576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd01005b0 0x7f8fd0196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.298+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 >> 172.21.15.116:0/829610576 conn(0x7f8fd00fc080 msgr2=0x7f8fd00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.299+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 shutdown_connections 2024-03-19T02:53:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.299+0000 7f8fd59ff640 1 -- 172.21.15.116:0/829610576 wait complete. 2024-03-19T02:53:36.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/210087436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac101050 msgr2=0x7f5aac10d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/210087436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac101050 0x7f5aac10d560 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009ca0 tx=0x7f5aa402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/210087436 shutdown_connections 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/210087436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac101050 0x7f5aac10d560 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/210087436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac100690 0x7f5aac100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/210087436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac1066a0 0x7f5aac106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/210087436 >> 172.21.15.116:0/210087436 conn(0x7f5aac077820 msgr2=0x7f5aac077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/210087436 shutdown_connections 2024-03-19T02:53:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.415+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/210087436 wait complete. 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 Processor -- start 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 -- start start 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 0x7f5aac197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac101050 0x7f5aac1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac111a60 con 0x7f5aac1066a0 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac1118e0 con 0x7f5aac101050 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5ab0bb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac111be0 con 0x7f5aac100690 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aaa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 0x7f5aac197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aaa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 0x7f5aac197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41540/0 (socket says 172.21.15.116:41540) 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44226/0 (socket says 172.21.15.116:44226) 2024-03-19T02:53:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aaa59c640 1 -- 172.21.15.116:0/503104943 learned_addr learned my addr 172.21.15.116:0/503104943 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.416+0000 7f5aa9d9b640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac101050 0x7f5aac1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 -- 172.21.15.116:0/503104943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 msgr2=0x7f5aac197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 0x7f5aac197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 -- 172.21.15.116:0/503104943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac101050 msgr2=0x7f5aac1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac101050 0x7f5aac1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 -- 172.21.15.116:0/503104943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac19cf50 con 0x7f5aac1066a0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5aaad9d640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f5aa402c700 tx=0x7f5aa4035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4039040 con 0x7f5aac1066a0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa4044070 con 0x7f5aac1066a0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40042a0 con 0x7f5aac1066a0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac19d1e0 con 0x7f5aac1066a0 2024-03-19T02:53:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.417+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac1a4ad0 con 0x7f5aac1066a0 2024-03-19T02:53:36.422 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/685353515' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:36.422 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3627629616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:36.422 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:36 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1590776164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.418+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac101dd0 con 0x7f5aac1066a0 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.419+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f5aa4004440 con 0x7f5aac1066a0 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.420+0000 7f5a9b7fe640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 0x7f5a940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.420+0000 7f5aaa59c640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 0x7f5a940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.420+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f5aa40f14a0 con 0x7f5aac1066a0 2024-03-19T02:53:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.420+0000 7f5aaa59c640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 0x7f5a940a4e30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009c10 tx=0x7f5a9c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.424+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5aa40bdfd0 con 0x7f5aac1066a0 2024-03-19T02:53:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/685353515' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:36.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3627629616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:36.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:36 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1590776164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.602+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5aac108c00 con 0x7f5aac1066a0 2024-03-19T02:53:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.602+0000 7f5a9b7fe640 1 -- 172.21.15.116:0/503104943 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v135) ==== 100+0+11 (secure 0 0 0) 0x7f5aa40bd8d0 con 0x7f5aac1066a0 2024-03-19T02:53:36.604 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 msgr2=0x7f5a940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 0x7f5a940a4e30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009c10 tx=0x7f5a9c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 msgr2=0x7f5aac19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f5aa402c700 tx=0x7f5aa4035d60 comp rx=0 tx=0).stop 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 shutdown_connections 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f5a940a2970 0x7f5a940a4e30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aac1066a0 0x7f5aac19c820 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5aac101050 0x7f5aac1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 --2- 172.21.15.116:0/503104943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5aac100690 0x7f5aac197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 >> 172.21.15.116:0/503104943 conn(0x7f5aac077820 msgr2=0x7f5aac102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 shutdown_connections 2024-03-19T02:53:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.605+0000 7f5ab0bb9640 1 -- 172.21.15.116:0/503104943 wait complete. 2024-03-19T02:53:36.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:36.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 -- 172.21.15.116:0/3036217959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc1077a0 msgr2=0x7f8ccc107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3036217959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc107b80 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009930 tx=0x7f8cc802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 -- 172.21.15.116:0/3036217959 shutdown_connections 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3036217959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3036217959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 0x7f8ccc068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3036217959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc107b80 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.721+0000 7f8cd4532640 1 -- 172.21.15.116:0/3036217959 >> 172.21.15.116:0/3036217959 conn(0x7f8ccc077820 msgr2=0x7f8ccc077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- 172.21.15.116:0/3036217959 shutdown_connections 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- 172.21.15.116:0/3036217959 wait complete. 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 Processor -- start 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- start start 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 0x7f8ccc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc111a60 con 0x7f8ccc0693c0 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc1118e0 con 0x7f8ccc1077a0 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd4532640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc111be0 con 0x7f8ccc068a00 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd1acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd1acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44236/0 (socket says 172.21.15.116:44236) 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd1acd640 1 -- 172.21.15.116:0/3153094872 learned_addr learned my addr 172.21.15.116:0/3153094872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd2acf640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.722+0000 7f8cd22ce640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 0x7f8ccc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 -- 172.21.15.116:0/3153094872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 msgr2=0x7f8ccc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 0x7f8ccc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 -- 172.21.15.116:0/3153094872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc1077a0 msgr2=0x7f8ccc19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 -- 172.21.15.116:0/3153094872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc19d020 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd1acd640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00e9f0 tx=0x7f8cbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00cd80 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cbc0054d0 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ccc19d310 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc010490 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.723+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ccc1a4ba0 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.724+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ccc068610 con 0x7f8ccc0693c0 2024-03-19T02:53:36.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.725+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f8cbc005020 con 0x7f8ccc0693c0 2024-03-19T02:53:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.726+0000 7f8cc37fe640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 0x7f8cb80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.726+0000 7f8cd22ce640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 0x7f8cb80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.726+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f8cbc0cf5b0 con 0x7f8ccc0693c0 2024-03-19T02:53:36.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.726+0000 7f8cd22ce640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 0x7f8cb80a4e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005ab0 tx=0x7f8cc8005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:36.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.730+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8cbc09bf30 con 0x7f8ccc0693c0 2024-03-19T02:53:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.917+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8ccc02d1f0 con 0x7f8ccc0693c0 2024-03-19T02:53:36.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.918+0000 7f8cc37fe640 1 -- 172.21.15.116:0/3153094872 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v135) ==== 103+0+11 (secure 0 0 0) 0x7f8cbc09b830 con 0x7f8ccc0693c0 2024-03-19T02:53:36.919 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T02:53:36.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 msgr2=0x7f8cb80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 0x7f8cb80a4e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005ab0 tx=0x7f8cc8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 msgr2=0x7f8ccc198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00e9f0 tx=0x7f8cbc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 shutdown_connections 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f8cb80a2970 0x7f8cb80a4e30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ccc1077a0 0x7f8ccc19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ccc0693c0 0x7f8ccc198570 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 --2- 172.21.15.116:0/3153094872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc068a00 0x7f8ccc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 >> 172.21.15.116:0/3153094872 conn(0x7f8ccc077820 msgr2=0x7f8ccc0fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 shutdown_connections 2024-03-19T02:53:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:36.920+0000 7f8cd4532640 1 -- 172.21.15.116:0/3153094872 wait complete. 2024-03-19T02:53:36.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 -- 172.21.15.116:0/2262825326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 msgr2=0x7f2534106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2262825326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f2534106a80 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2524009930 tx=0x7f252402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 -- 172.21.15.116:0/2262825326 shutdown_connections 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2262825326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534101050 0x7f253410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2262825326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534100690 0x7f2534100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2262825326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f2534106a80 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 -- 172.21.15.116:0/2262825326 >> 172.21.15.116:0/2262825326 conn(0x7f2534077820 msgr2=0x7f2534077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 -- 172.21.15.116:0/2262825326 shutdown_connections 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.041+0000 7f2539f3b640 1 -- 172.21.15.116:0/2262825326 wait complete. 2024-03-19T02:53:37.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 Processor -- start 2024-03-19T02:53:37.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 -- start start 2024-03-19T02:53:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534100690 0x7f2534196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534101050 0x7f2534197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:37.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2534110970 con 0x7f25341066a0 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25341107f0 con 0x7f2534101050 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2539f3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2534110af0 con 0x7f2534100690 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44246/0 (socket says 172.21.15.116:44246) 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f25337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534100690 0x7f2534196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2533fff640 1 -- 172.21.15.116:0/2698302751 learned_addr learned my addr 172.21.15.116:0/2698302751 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.042+0000 7f2532ffd640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534101050 0x7f2534197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 -- 172.21.15.116:0/2698302751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534100690 msgr2=0x7f2534196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534100690 0x7f2534196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 -- 172.21.15.116:0/2698302751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534101050 msgr2=0x7f2534197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534101050 0x7f2534197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 -- 172.21.15.116:0/2698302751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253419bec0 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2533fff640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f252c00eb10 tx=0x7f252c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c004b20 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f252c004cc0 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253419c1b0 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c010690 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.043+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25341a3a40 con 0x7f25341066a0 2024-03-19T02:53:37.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.044+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2534068610 con 0x7f25341066a0 2024-03-19T02:53:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.045+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f252c005020 con 0x7f25341066a0 2024-03-19T02:53:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.046+0000 7f2530ff9640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 0x7f251c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.046+0000 7f25337fe640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 0x7f251c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.046+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f252c0cf790 con 0x7f25341066a0 2024-03-19T02:53:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.047+0000 7f25337fe640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 0x7f251c0a4e30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2524009900 tx=0x7f2524005a20 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:37.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.050+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f252c09c110 con 0x7f25341066a0 2024-03-19T02:53:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.254+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2534108c60 con 0x7f25341066a0 2024-03-19T02:53:37.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.254+0000 7f2530ff9640 1 -- 172.21.15.116:0/2698302751 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v135) ==== 81+0+1016 (secure 0 0 0) 0x7f252c09ba10 con 0x7f25341066a0 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3573130182 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3953610795 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/647665947 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2583747917 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/996139070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/707840070 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2020074950 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1136807551 2024-03-20T02:17:41.784026+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1993666663 2024-03-20T02:18:07.512230+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2648298519 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/953095802 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3573130182 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2284944336 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3807729141 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1256325072 2024-03-20T02:27:48.704049+0000 2024-03-19T02:53:37.257 INFO:teuthology.orchestra.run.smithi116.stderr:listed 17 entries 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 msgr2=0x7f251c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 0x7f251c0a4e30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2524009900 tx=0x7f2524005a20 comp rx=0 tx=0).stop 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 msgr2=0x7f253419b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f252c00eb10 tx=0x7f252c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 shutdown_connections 2024-03-19T02:53:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f251c0a2970 0x7f251c0a4e30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25341066a0 0x7f253419b790 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2534101050 0x7f2534197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 --2- 172.21.15.116:0/2698302751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2534100690 0x7f2534196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 >> 172.21.15.116:0/2698302751 conn(0x7f2534077820 msgr2=0x7f2534102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 shutdown_connections 2024-03-19T02:53:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:37.257+0000 7f2539f3b640 1 -- 172.21.15.116:0/2698302751 wait complete. 2024-03-19T02:53:37.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:37 smithi116 ceph-mon[32263]: pgmap v816: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:53:37.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/829610576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/503104943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:37 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:37.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:37 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3153094872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.269 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-19T02:53:37.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:37.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:37.344 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:53:37.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[84163]: fuse finished with error 0 and tester_r 0 2024-03-19T02:53:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:37 smithi162 ceph-mon[35761]: pgmap v816: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:53:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/829610576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/503104943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:37 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:37.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:37 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3153094872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:37 smithi154 ceph-mon[35693]: pgmap v816: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-19T02:53:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/829610576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/503104943' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:37 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:37.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:37 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3153094872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:38.305 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:38 smithi116 ceph-mon[32263]: pgmap v817: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:38.305 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:38 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2698302751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:38.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:38 smithi162 ceph-mon[35761]: pgmap v817: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:38.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:38 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2698302751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:38 smithi154 ceph-mon[35693]: pgmap v817: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:38 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2698302751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:53:40.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:40 smithi162 ceph-mon[35761]: pgmap v818: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:40.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:40 smithi154 ceph-mon[35693]: pgmap v818: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:40.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:40 smithi116 ceph-mon[32263]: pgmap v818: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: pgmap v819: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:42.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:42 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: pgmap v819: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-19T02:53:43.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:43.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:43.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:42 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: pgmap v819: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:42 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' 2024-03-19T02:53:43.445 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T02:53:43.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:43.446 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T02:53:43.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:43.489 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T02:53:43.587 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T02:53:43.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:43.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:43.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:43.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-19T02:53:43.730 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-19T02:53:43.742 INFO:tasks.cephadm:Teardown begin 2024-03-19T02:53:43.742 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T02:53:43.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 -- 172.21.15.116:0/3591014308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4102170 msgr2=0x7f96e410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 --2- 172.21.15.116:0/3591014308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4102170 0x7f96e410e680 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009cd0 tx=0x7f96dc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 -- 172.21.15.116:0/3591014308 shutdown_connections 2024-03-19T02:53:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 --2- 172.21.15.116:0/3591014308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4102170 0x7f96e410e680 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 --2- 172.21.15.116:0/3591014308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41017b0 0x7f96e4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 --2- 172.21.15.116:0/3591014308 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e41077c0 0x7f96e4107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 -- 172.21.15.116:0/3591014308 >> 172.21.15.116:0/3591014308 conn(0x7f96e4078c20 msgr2=0x7f96e4079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 -- 172.21.15.116:0/3591014308 shutdown_connections 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.850+0000 7f96ec061640 1 -- 172.21.15.116:0/3591014308 wait complete. 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 Processor -- start 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 -- start start 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e4102170 0x7f96e41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41077c0 0x7f96e419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4111a80 con 0x7f96e41017b0 2024-03-19T02:53:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4111900 con 0x7f96e41077c0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ec061640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4111c00 con 0x7f96e4102170 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96e9dfd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96e9dfd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50078/0 (socket says 172.21.15.116:50078) 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96e9dfd640 1 -- 172.21.15.116:0/1341833208 learned_addr learned my addr 172.21.15.116:0/1341833208 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96e95fc640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e4102170 0x7f96e41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.851+0000 7f96ea5fe640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41077c0 0x7f96e419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 -- 172.21.15.116:0/1341833208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e4102170 msgr2=0x7f96e41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e4102170 0x7f96e41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 -- 172.21.15.116:0/1341833208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41077c0 msgr2=0x7f96e419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41077c0 0x7f96e419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 -- 172.21.15.116:0/1341833208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e419cf90 con 0x7f96e41017b0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96e9dfd640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9f0 tx=0x7f96e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e000cd80 con 0x7f96e41017b0 2024-03-19T02:53:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96e00054d0 con 0x7f96e41017b0 2024-03-19T02:53:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e419d280 con 0x7f96e41017b0 2024-03-19T02:53:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0010490 con 0x7f96e41017b0 2024-03-19T02:53:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.852+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e41a4b10 con 0x7f96e41017b0 2024-03-19T02:53:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.853+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e4067d80 con 0x7f96e41017b0 2024-03-19T02:53:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.854+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f96e0005000 con 0x7f96e41017b0 2024-03-19T02:53:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.855+0000 7f96daffd640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 0x7f96d00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.855+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f96e00cf6d0 con 0x7f96e41017b0 2024-03-19T02:53:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.855+0000 7f96e95fc640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 0x7f96d00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:43.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.855+0000 7f96e95fc640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 0x7f96d00a4eb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96e4103ab0 tx=0x7f96d4005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:43.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:43.858+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96e0053740 con 0x7f96e41017b0 2024-03-19T02:53:44.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.071+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96e41992f0 con 0x7f96e41017b0 2024-03-19T02:53:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.072+0000 7f96daffd640 1 -- 172.21.15.116:0/1341833208 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11954 (secure 0 0 0) 0x7f96e00052f0 con 0x7f96e41017b0 2024-03-19T02:53:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:53:44.072 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:52:45.840939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:53:44.072 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 45 2024-03-19T02:53:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 msgr2=0x7f96d00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 0x7f96d00a4eb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f96e4103ab0 tx=0x7f96d4005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 msgr2=0x7f96e4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9f0 tx=0x7f96e000edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 shutdown_connections 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f96d00a29f0 0x7f96d00a4eb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96e41077c0 0x7f96e419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f96e4102170 0x7f96e41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 --2- 172.21.15.116:0/1341833208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e41017b0 0x7f96e4197fa0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 >> 172.21.15.116:0/1341833208 conn(0x7f96e4078c20 msgr2=0x7f96e4104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 shutdown_connections 2024-03-19T02:53:44.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.074+0000 7f96ec061640 1 -- 172.21.15.116:0/1341833208 wait complete. 2024-03-19T02:53:44.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:44.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 -- 172.21.15.116:0/56341598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101066f0 msgr2=0x7f0310106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 --2- 172.21.15.116:0/56341598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101066f0 0x7f0310106ad0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f0304009930 tx=0x7f030402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:44.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 -- 172.21.15.116:0/56341598 shutdown_connections 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 --2- 172.21.15.116:0/56341598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101010a0 0x7f031010d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 --2- 172.21.15.116:0/56341598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 0x7f0310100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 --2- 172.21.15.116:0/56341598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101066f0 0x7f0310106ad0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 -- 172.21.15.116:0/56341598 >> 172.21.15.116:0/56341598 conn(0x7f03100777c0 msgr2=0x7f0310077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 -- 172.21.15.116:0/56341598 shutdown_connections 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.189+0000 7f0314d32640 1 -- 172.21.15.116:0/56341598 wait complete. 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 Processor -- start 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 -- start start 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 0x7f0310197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101066f0 0x7f031019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111ab0 con 0x7f03101010a0 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111930 con 0x7f03101006e0 2024-03-19T02:53:44.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f0314d32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0310111c30 con 0x7f03101066f0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50096/0 (socket says 172.21.15.116:50096) 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 -- 172.21.15.116:0/2837348980 learned_addr learned my addr 172.21.15.116:0/2837348980 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030e59c640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 0x7f0310197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030ed9d640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101066f0 0x7f031019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 -- 172.21.15.116:0/2837348980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101066f0 msgr2=0x7f031019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101066f0 0x7f031019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 -- 172.21.15.116:0/2837348980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 msgr2=0x7f0310197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 0x7f0310197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 -- 172.21.15.116:0/2837348980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031019cfe0 con 0x7f03101010a0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.190+0000 7f030dd9b640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f030000ea60 tx=0x7f030000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.191+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030000cde0 con 0x7f03101010a0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.191+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f03000035f0 con 0x7f03101010a0 2024-03-19T02:53:44.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.191+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0300010690 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.191+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031019d2d0 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.191+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03101a4b60 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.192+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0310068a60 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.193+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f03000037a0 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.194+0000 7f02ff7fe640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 0x7f02f80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.194+0000 7f030e59c640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 0x7f02f80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.194+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f03000cf710 con 0x7f03101010a0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.194+0000 7f030e59c640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 0x7f02f80a4e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0304005ab0 tx=0x7f0304005fd0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.197+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0300053740 con 0x7f03101010a0 2024-03-19T02:53:44.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.413+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0310199340 con 0x7f03101010a0 2024-03-19T02:53:44.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.413+0000 7f02ff7fe640 1 -- 172.21.15.116:0/2837348980 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11954 (secure 0 0 0) 0x7f0310199340 con 0x7f03101010a0 2024-03-19T02:53:44.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:53:44.414 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1165879311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1165879311},{"type":"v1","addr":"172.21.15.116:6835","nonce":1165879311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/261800782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":261800782},{"type":"v1","addr":"172.21.15.116:6837","nonce":261800782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:25:08.015174+0000","modified":"2024-03-19T02:52:45.840939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24484,"mds_2":24470,"mds_3":24463,"mds_4":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1891911264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1891911264},{"type":"v1","addr":"172.21.15.116:6839","nonce":1891911264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/3941624950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3941624950},{"type":"v1","addr":"172.21.15.116:6841","nonce":3941624950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6833/1010748955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1010748955},{"type":"v1","addr":"172.21.15.154:6833","nonce":1010748955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/750284800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":750284800},{"type":"v1","addr":"172.21.15.162:6833","nonce":750284800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6835/1464153745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1464153745},{"type":"v1","addr":"172.21.15.154:6835","nonce":1464153745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.162:6835/831287214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":831287214},{"type":"v1","addr":"172.21.15.162:6835","nonce":831287214}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.162:6837/3347737639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":3347737639},{"type":"v1","addr":"172.21.15.162:6837","nonce":3347737639}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.154:6837/455932973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":455932973},{"type":"v1","addr":"172.21.15.154:6837","nonce":455932973}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6839/1427360219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1427360219},{"type":"v1","addr":"172.21.15.162:6839","nonce":1427360219}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/4009274564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":4009274564},{"type":"v1","addr":"172.21.15.154:6839","nonce":4009274564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24470,24463,24484,24451,24494]},"id":1}]} 2024-03-19T02:53:44.414 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 45 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 msgr2=0x7f02f80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 0x7f02f80a4e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0304005ab0 tx=0x7f0304005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 msgr2=0x7f0310198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f030000ea60 tx=0x7f030000ef20 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 shutdown_connections 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f02f80a2970 0x7f02f80a4e30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03101066f0 0x7f031019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03101010a0 0x7f0310198530 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.415+0000 7f0314d32640 1 --2- 172.21.15.116:0/2837348980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03101006e0 0x7f0310197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.416+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 >> 172.21.15.116:0/2837348980 conn(0x7f03100777c0 msgr2=0x7f03100fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.416+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 shutdown_connections 2024-03-19T02:53:44.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.416+0000 7f0314d32640 1 -- 172.21.15.116:0/2837348980 wait complete. 2024-03-19T02:53:44.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 -- 172.21.15.116:0/3887362440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10be80 msgr2=0x7fcf6c10e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3887362440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10be80 0x7fcf6c10e2d0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fcf64009ca0 tx=0x7fcf6402c180 comp rx=0 tx=0).stop 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 -- 172.21.15.116:0/3887362440 shutdown_connections 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3887362440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c10be80 0x7fcf6c10e2d0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3887362440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3887362440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 -- 172.21.15.116:0/3887362440 >> 172.21.15.116:0/3887362440 conn(0x7fcf6c077880 msgr2=0x7fcf6c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 -- 172.21.15.116:0/3887362440 shutdown_connections 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.529+0000 7fcf726c6640 1 -- 172.21.15.116:0/3887362440 wait complete. 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 Processor -- start 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 -- start start 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c10be80 0x7fcf6c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c111ae0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c111960 con 0x7fcf6c10be80 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf726c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c111c60 con 0x7fcf6c102ab0 2024-03-19T02:53:44.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50122/0 (socket says 172.21.15.116:50122) 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 -- 172.21.15.116:0/3122140326 learned_addr learned my addr 172.21.15.116:0/3122140326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6bfff640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf70c63640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c10be80 0x7fcf6c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 -- 172.21.15.116:0/3122140326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 msgr2=0x7fcf6c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 -- 172.21.15.116:0/3122140326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c10be80 msgr2=0x7fcf6c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c10be80 0x7fcf6c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6b7fe640 1 -- 172.21.15.116:0/3122140326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6c19cfd0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.530+0000 7fcf6bfff640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:44.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf6b7fe640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000e9f0 tx=0x7fcf6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000cd80 con 0x7fcf6c1033d0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6c19d2c0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf600108e0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60018920 con 0x7fcf6c1033d0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.531+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6c1a4b50 con 0x7fcf6c1033d0 2024-03-19T02:53:44.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.532+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6c068a60 con 0x7fcf6c1033d0 2024-03-19T02:53:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.533+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fcf60010a90 con 0x7fcf6c1033d0 2024-03-19T02:53:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.533+0000 7fcf697fa640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 0x7fcf540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.534+0000 7fcf6bfff640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 0x7fcf540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.534+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7fcf600cf4f0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.534+0000 7fcf6bfff640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 0x7fcf540a4e30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0095f0 tx=0x7fcf5c005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.538+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf6009b820 con 0x7fcf6c1033d0 2024-03-19T02:53:44.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.715+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf6c106fd0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.717+0000 7fcf697fa640 1 -- 172.21.15.116:0/3122140326 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+25403 (secure 0 0 0) 0x7fcf6009f8c0 con 0x7fcf6c1033d0 2024-03-19T02:53:44.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T02:53:44.718 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":135,"fsid":"a1da9bbc-e596-11ee-95c9-87774f69a715","created":"2024-03-19T02:17:06.753416+0000","modified":"2024-03-19T02:53:16.985543+0000","last_up_change":"2024-03-19T02:24:20.810191+0000","last_in_change":"2024-03-19T02:24:00.571493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:20:31.845523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:25:06.506384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:25:07.182320+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:25:10.435819+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":134,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"228d73fd-9b05-45b9-b2fb-93b859800b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6803","nonce":3168976854}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6805","nonce":3168976854}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6809","nonce":3168976854}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3168976854},{"type":"v1","addr":"172.21.15.116:6807","nonce":3168976854}]},"public_addr":"172.21.15.116:6803/3168976854","cluster_addr":"172.21.15.116:6805/3168976854","heartbeat_back_addr":"172.21.15.116:6809/3168976854","heartbeat_front_addr":"172.21.15.116:6807/3168976854","state":["exists","up"]},{"osd":1,"uuid":"fd497f40-8e3f-473a-8a69-450ad65aa6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6801","nonce":3251901849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6803","nonce":3251901849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6807","nonce":3251901849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3251901849},{"type":"v1","addr":"172.21.15.154:6805","nonce":3251901849}]},"public_addr":"172.21.15.154:6801/3251901849","cluster_addr":"172.21.15.154:6803/3251901849","heartbeat_back_addr":"172.21.15.154:6807/3251901849","heartbeat_front_addr":"172.21.15.154:6805/3251901849","state":["exists","up"]},{"osd":2,"uuid":"89b45a09-6328-44d9-84ba-890c9225881f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6801","nonce":2685600713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6803","nonce":2685600713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6807","nonce":2685600713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2685600713},{"type":"v1","addr":"172.21.15.162:6805","nonce":2685600713}]},"public_addr":"172.21.15.162:6801/2685600713","cluster_addr":"172.21.15.162:6803/2685600713","heartbeat_back_addr":"172.21.15.162:6807/2685600713","heartbeat_front_addr":"172.21.15.162:6805/2685600713","state":["exists","up"]},{"osd":3,"uuid":"5d6c5e73-72bf-40e9-95b8-8473f091ea90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6811","nonce":2748385188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6813","nonce":2748385188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6817","nonce":2748385188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2748385188},{"type":"v1","addr":"172.21.15.116:6815","nonce":2748385188}]},"public_addr":"172.21.15.116:6811/2748385188","cluster_addr":"172.21.15.116:6813/2748385188","heartbeat_back_addr":"172.21.15.116:6817/2748385188","heartbeat_front_addr":"172.21.15.116:6815/2748385188","state":["exists","up"]},{"osd":4,"uuid":"4bf282e6-a231-437d-af9b-f8b7a11d67c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6809","nonce":719513760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6811","nonce":719513760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6815","nonce":719513760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":719513760},{"type":"v1","addr":"172.21.15.154:6813","nonce":719513760}]},"public_addr":"172.21.15.154:6809/719513760","cluster_addr":"172.21.15.154:6811/719513760","heartbeat_back_addr":"172.21.15.154:6815/719513760","heartbeat_front_addr":"172.21.15.154:6813/719513760","state":["exists","up"]},{"osd":5,"uuid":"4f44ae22-a0e6-44f2-b6d3-c9bcc2f7b639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6809","nonce":1110895518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6811","nonce":1110895518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6815","nonce":1110895518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1110895518},{"type":"v1","addr":"172.21.15.162:6813","nonce":1110895518}]},"public_addr":"172.21.15.162:6809/1110895518","cluster_addr":"172.21.15.162:6811/1110895518","heartbeat_back_addr":"172.21.15.162:6815/1110895518","heartbeat_front_addr":"172.21.15.162:6813/1110895518","state":["exists","up"]},{"osd":6,"uuid":"e9047802-fa47-4d12-98d7-b7a23641af81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6819","nonce":119271604}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6821","nonce":119271604}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6825","nonce":119271604}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":119271604},{"type":"v1","addr":"172.21.15.116:6823","nonce":119271604}]},"public_addr":"172.21.15.116:6819/119271604","cluster_addr":"172.21.15.116:6821/119271604","heartbeat_back_addr":"172.21.15.116:6825/119271604","heartbeat_front_addr":"172.21.15.116:6823/119271604","state":["exists","up"]},{"osd":7,"uuid":"c2ab68c8-d137-4f5c-a8e1-e539b0819897","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6817","nonce":2515714011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6819","nonce":2515714011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6823","nonce":2515714011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2515714011},{"type":"v1","addr":"172.21.15.154:6821","nonce":2515714011}]},"public_addr":"172.21.15.154:6817/2515714011","cluster_addr":"172.21.15.154:6819/2515714011","heartbeat_back_addr":"172.21.15.154:6823/2515714011","heartbeat_front_addr":"172.21.15.154:6821/2515714011","state":["exists","up"]},{"osd":8,"uuid":"ba54ef24-b96d-4423-b606-6b742a134ca7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6817","nonce":2609002747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6819","nonce":2609002747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6823","nonce":2609002747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2609002747},{"type":"v1","addr":"172.21.15.162:6821","nonce":2609002747}]},"public_addr":"172.21.15.162:6817/2609002747","cluster_addr":"172.21.15.162:6819/2609002747","heartbeat_back_addr":"172.21.15.162:6823/2609002747","heartbeat_front_addr":"172.21.15.162:6821/2609002747","state":["exists","up"]},{"osd":9,"uuid":"3ff4d2a4-7e1e-4039-b2bf-8ae924f5a873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6827","nonce":3348335330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6829","nonce":3348335330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6833","nonce":3348335330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3348335330},{"type":"v1","addr":"172.21.15.116:6831","nonce":3348335330}]},"public_addr":"172.21.15.116:6827/3348335330","cluster_addr":"172.21.15.116:6829/3348335330","heartbeat_back_addr":"172.21.15.116:6833/3348335330","heartbeat_front_addr":"172.21.15.116:6831/3348335330","state":["exists","up"]},{"osd":10,"uuid":"b644847f-a734-4600-9861-c8574c3f0622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6825","nonce":1964548863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6827","nonce":1964548863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6831","nonce":1964548863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1964548863},{"type":"v1","addr":"172.21.15.154:6829","nonce":1964548863}]},"public_addr":"172.21.15.154:6825/1964548863","cluster_addr":"172.21.15.154:6827/1964548863","heartbeat_back_addr":"172.21.15.154:6831/1964548863","heartbeat_front_addr":"172.21.15.154:6829/1964548863","state":["exists","up"]},{"osd":11,"uuid":"883bfd81-b55d-441a-9a0c-2f5faef7f4a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6825","nonce":3174333504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6827","nonce":3174333504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6831","nonce":3174333504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":3174333504},{"type":"v1","addr":"172.21.15.162:6829","nonce":3174333504}]},"public_addr":"172.21.15.162:6825/3174333504","cluster_addr":"172.21.15.162:6827/3174333504","heartbeat_back_addr":"172.21.15.162:6831/3174333504","heartbeat_front_addr":"172.21.15.162:6829/3174333504","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:19:40.675451+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:04.676115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:28.130272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:20:53.233592+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:17.860928+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:21:43.234465+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:08.925845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:22:34.280282+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:00.523302+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:26.873825+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:23:52.500237+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:24:19.450295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6800/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:6800/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3953610795":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/647665947":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2583747917":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/996139070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/707840070":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2020074950":"2024-03-20T02:17:41.784026+0000","172.21.15.116:6801/1136807551":"2024-03-20T02:17:41.784026+0000","172.21.15.116:0/1993666663":"2024-03-20T02:18:07.512230+0000","172.21.15.116:0/2648298519":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/953095802":"2024-03-20T02:27:48.704049+0000","172.21.15.116:6801/3573130182":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/2284944336":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/3807729141":"2024-03-20T02:27:48.704049+0000","172.21.15.116:0/1256325072":"2024-03-20T02:27:48.704049+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 msgr2=0x7fcf540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 0x7fcf540a4e30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0095f0 tx=0x7fcf5c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 msgr2=0x7fcf6c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000e9f0 tx=0x7fcf6000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 shutdown_connections 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fcf540a2970 0x7fcf540a4e30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf6c10be80 0x7fcf6c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf6c1033d0 0x7fcf6c198520 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 --2- 172.21.15.116:0/3122140326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf6c102ab0 0x7fcf6c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 >> 172.21.15.116:0/3122140326 conn(0x7fcf6c077880 msgr2=0x7fcf6c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 shutdown_connections 2024-03-19T02:53:44.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.719+0000 7fcf726c6640 1 -- 172.21.15.116:0/3122140326 wait complete. 2024-03-19T02:53:44.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:44 smithi116 ceph-mon[32263]: pgmap v820: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-19T02:53:44.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1341833208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:44.729 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:44 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/2837348980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:44.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T02:53:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 -- 172.21.15.116:0/413999663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1077c0 msgr2=0x7f98fc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 --2- 172.21.15.116:0/413999663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1077c0 0x7f98fc107ba0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009930 tx=0x7f98ec02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 -- 172.21.15.116:0/413999663 shutdown_connections 2024-03-19T02:53:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 --2- 172.21.15.116:0/413999663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 0x7f98fc10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 --2- 172.21.15.116:0/413999663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1017b0 0x7f98fc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 --2- 172.21.15.116:0/413999663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1077c0 0x7f98fc107ba0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 -- 172.21.15.116:0/413999663 >> 172.21.15.116:0/413999663 conn(0x7f98fc078c20 msgr2=0x7f98fc079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 -- 172.21.15.116:0/413999663 shutdown_connections 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.834+0000 7f9901bb9640 1 -- 172.21.15.116:0/413999663 wait complete. 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 Processor -- start 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 -- start start 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 0x7f98fc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1077c0 0x7f98fc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc111a80 con 0x7f98fc1017b0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc111900 con 0x7f98fc102170 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f9901bb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc111c00 con 0x7f98fc1077c0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50148/0 (socket says 172.21.15.116:50148) 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 -- 172.21.15.116:0/3046005143 learned_addr learned my addr 172.21.15.116:0/3046005143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fbfff640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1077c0 0x7f98fc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98faffd640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 0x7f98fc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 -- 172.21.15.116:0/3046005143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1077c0 msgr2=0x7f98fc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1077c0 0x7f98fc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 -- 172.21.15.116:0/3046005143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 msgr2=0x7f98fc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 0x7f98fc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.835+0000 7f98fb7fe640 1 -- 172.21.15.116:0/3046005143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc19cff0 con 0x7f98fc1017b0 2024-03-19T02:53:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f98fb7fe640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009230 tx=0x7f98ec02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec043020 con 0x7f98fc1017b0 2024-03-19T02:53:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98ec02ca70 con 0x7f98fc1017b0 2024-03-19T02:53:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98fc19d280 con 0x7f98fc1017b0 2024-03-19T02:53:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec003a50 con 0x7f98fc1017b0 2024-03-19T02:53:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.836+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98fc1a4b70 con 0x7f98fc1017b0 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.837+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98fc068a60 con 0x7f98fc1017b0 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.838+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f98ec046020 con 0x7f98fc1017b0 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.838+0000 7f98f8ff9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 0x7f98e40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.839+0000 7f98faffd640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 0x7f98e40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.839+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9510+0+0 (secure 0 0 0) 0x7f98ec0f1120 con 0x7f98fc1017b0 2024-03-19T02:53:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.839+0000 7f98faffd640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 0x7f98e40a4e30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f98fc103b00 tx=0x7f98f0005f90 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:44.843+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98ec04d020 con 0x7f98fc1017b0 2024-03-19T02:53:44.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:44 smithi162 ceph-mon[35761]: pgmap v820: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-19T02:53:44.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1341833208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:44.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:44 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/2837348980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:44 smithi154 ceph-mon[35693]: pgmap v820: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-19T02:53:45.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1341833208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:45.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:44 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/2837348980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:45.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.061+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f98fc109d20 con 0x7f98fc1017b0 2024-03-19T02:53:45.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.065+0000 7f98f8ff9640 1 -- 172.21.15.116:0/3046005143 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7f98ec0bdcd0 con 0x7f98fc1017b0 2024-03-19T02:53:45.065 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.067+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 msgr2=0x7f98e40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.067+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 0x7f98e40a4e30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f98fc103b00 tx=0x7f98f0005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.067+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 msgr2=0x7f98fc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.067+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009230 tx=0x7f98ec02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 shutdown_connections 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f98e40a2970 0x7f98e40a4e30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98fc1077c0 0x7f98fc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98fc102170 0x7f98fc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 --2- 172.21.15.116:0/3046005143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98fc1017b0 0x7f98fc198000 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 >> 172.21.15.116:0/3046005143 conn(0x7f98fc078c20 msgr2=0x7f98fc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:45.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 shutdown_connections 2024-03-19T02:53:45.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.068+0000 7f9901bb9640 1 -- 172.21.15.116:0/3046005143 wait complete. 2024-03-19T02:53:45.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T02:53:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 -- 172.21.15.116:0/178127039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb484102a60 msgr2=0x7fb484102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 --2- 172.21.15.116:0/178127039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb484102a60 0x7fb484102e40 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb474009900 tx=0x7fb47402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 -- 172.21.15.116:0/178127039 shutdown_connections 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 --2- 172.21.15.116:0/178127039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb48410be30 0x7fb48410e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 --2- 172.21.15.116:0/178127039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb48410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 --2- 172.21.15.116:0/178127039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb484102a60 0x7fb484102e40 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 -- 172.21.15.116:0/178127039 >> 172.21.15.116:0/178127039 conn(0x7fb484076aa0 msgr2=0x7fb484076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 -- 172.21.15.116:0/178127039 shutdown_connections 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.190+0000 7fb489f53640 1 -- 172.21.15.116:0/178127039 wait complete. 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 Processor -- start 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 -- start start 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 0x7fb484198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb484198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:45.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb484111a70 con 0x7fb48410be30 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4841118f0 con 0x7fb484102a60 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb489f53640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb484111bf0 con 0x7fb484103380 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb483fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb483fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50210/0 (socket says 172.21.15.116:50210) 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb483fff640 1 -- 172.21.15.116:0/4117189243 learned_addr learned my addr 172.21.15.116:0/4117189243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.191+0000 7fb482ffd640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb484198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb4837fe640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 0x7fb484198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 -- 172.21.15.116:0/4117189243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 msgr2=0x7fb484198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb484198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 -- 172.21.15.116:0/4117189243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 msgr2=0x7fb484198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 0x7fb484198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 -- 172.21.15.116:0/4117189243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48419cff0 con 0x7fb48410be30 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb482ffd640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb484198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:45.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb4837fe640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 0x7fb484198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb483fff640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00eb40 tx=0x7fb47c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c004b20 con 0x7fb48410be30 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb47c004cc0 con 0x7fb48410be30 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48419d2e0 con 0x7fb48410be30 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c0107b0 con 0x7fb48410be30 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.192+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4841a4b70 con 0x7fb48410be30 2024-03-19T02:53:45.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.193+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb48406a610 con 0x7fb48410be30 2024-03-19T02:53:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.194+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fb47c005020 con 0x7fb48410be30 2024-03-19T02:53:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.195+0000 7fb480ff9640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 0x7fb4700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.195+0000 7fb4837fe640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 0x7fb4700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.195+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9852+0+0 (secure 0 0 0) 0x7fb47c0cfbe0 con 0x7fb48410be30 2024-03-19T02:53:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.195+0000 7fb4837fe640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 0x7fb4700a4e30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb474036040 tx=0x7fb474000bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.199+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb47c09c410 con 0x7fb48410be30 2024-03-19T02:53:45.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:45.412+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb484199350 con 0x7fb48410be30 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mds[66695]: -- 172.21.15.162:0/1847350339 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5644a6e71a40 con 0x5644a6e45000 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: starting mds.i at 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mds[65757]: -- 172.21.15.162:0/4289159223 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566d02cba40 con 0x5566d029f000 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: starting mds.l at 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mds[66376]: -- 172.21.15.162:0/40457563 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561a6a8e1a40 con 0x561a6a8b4c00 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: starting mds.c at 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.419 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.420 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mds[66068]: -- 172.21.15.162:0/1503828205 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f455d37a40 con 0x55f455d0b000 2024-03-19T02:53:45.420 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: starting mds.f at 2024-03-19T02:53:45.506 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mds[66139]: -- 172.21.15.154:0/1832083592 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec15063a40 con 0x55ec15036c00 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: starting mds.b at 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mds[65207]: -- 172.21.15.154:0/2468015172 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56161d645a40 con 0x56161d619000 2024-03-19T02:53:45.507 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: starting mds.k at 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mds[65524]: -- 172.21.15.154:0/396214229 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c40d81fa40 con 0x55c40d7f3000 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: starting mds.e at 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mds[65832]: -- 172.21.15.154:0/2063931322 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5653ccefba40 con 0x5653ccece800 2024-03-19T02:53:45.508 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: starting mds.h at 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a[32259]: 2024-03-19T02:53:45.061+0000 7f6503133700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mds[71465]: -- 172.21.15.116:0/789716390 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563672d71a40 con 0x563672d44c00 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: starting mds.g at 2024-03-19T02:53:45.509 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:45.510 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:45.510 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mds[71824]: -- 172.21.15.116:0/3745311174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55954a0eba40 con 0x55954a0bf000 2024-03-19T02:53:45.510 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: starting mds.a at 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3122140326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: osdmap e136: 12 total, 12 up, 12 in 2024-03-19T02:53:45.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T02:53:45.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:45 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3122140326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: osdmap e136: 12 total, 12 up, 12 in 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T02:53:46.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:45 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3122140326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3046005143' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: osdmap e136: 12 total, 12 up, 12 in 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T02:53:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:45 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:53:46.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.070+0000 7fb480ff9640 1 -- 172.21.15.116:0/4117189243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7fb47c09bd10 con 0x7fb48410be30 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 msgr2=0x7fb4700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 0x7fb4700a4e30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb474036040 tx=0x7fb474000bc0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 msgr2=0x7fb48419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00eb40 tx=0x7fb47c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 shutdown_connections 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fb4700a2970 0x7fb4700a4e30 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb48410be30 0x7fb48419c8c0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb484103380 0x7fb484198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 --2- 172.21.15.116:0/4117189243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb484102a60 0x7fb484198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.072+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 >> 172.21.15.116:0/4117189243 conn(0x7fb484076aa0 msgr2=0x7fb4840fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:46.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.073+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 shutdown_connections 2024-03-19T02:53:46.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.073+0000 7fb489f53640 1 -- 172.21.15.116:0/4117189243 wait complete. 2024-03-19T02:53:46.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 -- 172.21.15.116:0/3752867478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c103380 msgr2=0x7fd38c10b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 --2- 172.21.15.116:0/3752867478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c103380 0x7fd38c10b860 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fd380009900 tx=0x7fd38002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 -- 172.21.15.116:0/3752867478 shutdown_connections 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 --2- 172.21.15.116:0/3752867478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 --2- 172.21.15.116:0/3752867478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c103380 0x7fd38c10b860 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 --2- 172.21.15.116:0/3752867478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd38c102a60 0x7fd38c102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 -- 172.21.15.116:0/3752867478 >> 172.21.15.116:0/3752867478 conn(0x7fd38c076aa0 msgr2=0x7fd38c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 -- 172.21.15.116:0/3752867478 shutdown_connections 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.194+0000 7fd39223f640 1 -- 172.21.15.116:0/3752867478 wait complete. 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 Processor -- start 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 -- start start 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd38c103380 0x7fd38c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c111a70 con 0x7fd38c102a60 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c1118f0 con 0x7fd38c10be30 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd39223f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c111bf0 con 0x7fd38c103380 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50224/0 (socket says 172.21.15.116:50224) 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38b7fe640 1 -- 172.21.15.116:0/3870169070 learned_addr learned my addr 172.21.15.116:0/3870169070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38bfff640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38b7fe640 1 -- 172.21.15.116:0/3870169070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd38c103380 msgr2=0x7fd38c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.195+0000 7fd38b7fe640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd38c103380 0x7fd38c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd38b7fe640 1 -- 172.21.15.116:0/3870169070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 msgr2=0x7fd38c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd38b7fe640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd38b7fe640 1 -- 172.21.15.116:0/3870169070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c19cfb0 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd38bfff640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd38b7fe640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00d770 tx=0x7fd37c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c017070 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd37c0054d0 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c19d2a0 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c004cf0 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.196+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1a4b30 con 0x7fd38c102a60 2024-03-19T02:53:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.197+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c06a610 con 0x7fd38c102a60 2024-03-19T02:53:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.198+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fd37c004e90 con 0x7fd38c102a60 2024-03-19T02:53:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.198+0000 7fd388ff9640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.198+0000 7fd38affd640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:46.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.199+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9852+0+0 (secure 0 0 0) 0x7fd37c0ceeb0 con 0x7fd38c102a60 2024-03-19T02:53:46.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.199+0000 7fd38affd640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 0x7fd3740a4e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd380005770 tx=0x7fd380000bc0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.203+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd37c09b6e0 con 0x7fd38c102a60 2024-03-19T02:53:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:46.378+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd38c106f50 con 0x7fd38c102a60 2024-03-19T02:53:46.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: pgmap v822: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:53:46.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T02:53:46.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T02:53:46.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: Cluster is now healthy 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6833/2989745904,v1:172.21.15.162:6835/2989745904] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6838/1279874954,v1:172.21.15.162:6839/1279874954] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6832/1279622799,v1:172.21.15.162:6834/1279622799] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.162:6836/210856088,v1:172.21.15.162:6837/210856088] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6832/2130685281,v1:172.21.15.154:6834/2130685281] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6837/776001335,v1:172.21.15.154:6839/776001335] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6839/2852946776,v1:172.21.15.116:6841/2852946776] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.116:6838/1091352520,v1:172.21.15.116:6840/1091352520] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6833/1905270078,v1:172.21.15.154:6835/1905270078] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: mds.? [v2:172.21.15.154:6836/3242131779,v1:172.21.15.154:6838/3242131779] up:boot 2024-03-19T02:53:46.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: fsmap 12 up:standby 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:46.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:46 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: pgmap v822: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: Cluster is now healthy 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6833/2989745904,v1:172.21.15.162:6835/2989745904] up:boot 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6838/1279874954,v1:172.21.15.162:6839/1279874954] up:boot 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6832/1279622799,v1:172.21.15.162:6834/1279622799] up:boot 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.162:6836/210856088,v1:172.21.15.162:6837/210856088] up:boot 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6832/2130685281,v1:172.21.15.154:6834/2130685281] up:boot 2024-03-19T02:53:47.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6837/776001335,v1:172.21.15.154:6839/776001335] up:boot 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6839/2852946776,v1:172.21.15.116:6841/2852946776] up:boot 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.116:6838/1091352520,v1:172.21.15.116:6840/1091352520] up:boot 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6833/1905270078,v1:172.21.15.154:6835/1905270078] up:boot 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: mds.? [v2:172.21.15.154:6836/3242131779,v1:172.21.15.154:6838/3242131779] up:boot 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: fsmap 12 up:standby 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:47.008 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:46 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: pgmap v822: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: Cluster is now healthy 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/4117189243' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6833/2989745904,v1:172.21.15.162:6835/2989745904] up:boot 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6838/1279874954,v1:172.21.15.162:6839/1279874954] up:boot 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6832/1279622799,v1:172.21.15.162:6834/1279622799] up:boot 2024-03-19T02:53:47.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.162:6836/210856088,v1:172.21.15.162:6837/210856088] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6832/2130685281,v1:172.21.15.154:6834/2130685281] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6837/776001335,v1:172.21.15.154:6839/776001335] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6839/2852946776,v1:172.21.15.116:6841/2852946776] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.116:6838/1091352520,v1:172.21.15.116:6840/1091352520] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6833/1905270078,v1:172.21.15.154:6835/1905270078] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: mds.? [v2:172.21.15.154:6836/3242131779,v1:172.21.15.154:6838/3242131779] up:boot 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: fsmap 12 up:standby 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:47.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:47.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:47.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:47.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='mgr.25556 172.21.15.116:0/2658190000' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:47.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:46 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:47.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.071+0000 7fd388ff9640 1 -- 172.21.15.116:0/3870169070 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) ==== 181+0+0 (secure 0 0 0) 0x7fd37c09afe0 con 0x7fd38c102a60 2024-03-19T02:53:47.072 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2024-03-19T02:53:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 msgr2=0x7fd3740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 0x7fd3740a4e30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd380005770 tx=0x7fd380000bc0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 msgr2=0x7fd38c198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00d770 tx=0x7fd37c00dc30 comp rx=0 tx=0).stop 2024-03-19T02:53:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 shutdown_connections 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fd3740a2970 0x7fd3740a4e30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd38c10be30 0x7fd38c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd38c103380 0x7fd38c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 --2- 172.21.15.116:0/3870169070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c102a60 0x7fd38c198010 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 >> 172.21.15.116:0/3870169070 conn(0x7fd38c076aa0 msgr2=0x7fd38c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.075+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 shutdown_connections 2024-03-19T02:53:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.076+0000 7fd39223f640 1 -- 172.21.15.116:0/3870169070 wait complete. 2024-03-19T02:53:47.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T02:53:47.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1805214848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f00693e0 msgr2=0x7fc3f010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:47.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1805214848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f00693e0 0x7fc3f010e680 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009cd0 tx=0x7fc3e802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1805214848 shutdown_connections 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1805214848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f00693e0 0x7fc3f010e680 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1805214848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f0068a20 0x7fc3f0068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1805214848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 0x7fc3f0107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1805214848 >> 172.21.15.116:0/1805214848 conn(0x7fc3f0077820 msgr2=0x7fc3f0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1805214848 shutdown_connections 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.208+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1805214848 wait complete. 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 Processor -- start 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 -- start start 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:47.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f00693e0 0x7fc3f0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 0x7fc3f019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0111a30 con 0x7fc3f0068a20 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f01118b0 con 0x7fc3f00693e0 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f78f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0111bb0 con 0x7fc3f0107870 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50242/0 (socket says 172.21.15.116:50242) 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 -- 172.21.15.116:0/1165309228 learned_addr learned my addr 172.21.15.116:0/1165309228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f4e8c640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f00693e0 0x7fc3f0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f5e8e640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 0x7fc3f019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 -- 172.21.15.116:0/1165309228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 msgr2=0x7fc3f019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:47.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 0x7fc3f019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 -- 172.21.15.116:0/1165309228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f00693e0 msgr2=0x7fc3f0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f00693e0 0x7fc3f0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 -- 172.21.15.116:0/1165309228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f019cf40 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.209+0000 7fc3f568d640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00d950 tx=0x7fc3ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.210+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec017030 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.210+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc3ec004020 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.210+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f019d230 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.210+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0047a0 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.210+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f01a4ac0 con 0x7fc3f0068a20 2024-03-19T02:53:47.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.211+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f0068610 con 0x7fc3f0068a20 2024-03-19T02:53:47.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.212+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7fc3ec0041d0 con 0x7fc3f0068a20 2024-03-19T02:53:47.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.212+0000 7fc3e67fc640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 0x7fc3dc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:47.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.213+0000 7fc3f4e8c640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 0x7fc3dc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:47.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.213+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9356+0+0 (secure 0 0 0) 0x7fc3ec0cf180 con 0x7fc3f0068a20 2024-03-19T02:53:47.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.213+0000 7fc3f4e8c640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 0x7fc3dc0a4eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00fd960 tx=0x7fc3e000a2d0 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:47.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.217+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc3ec09bc20 con 0x7fc3f0068a20 2024-03-19T02:53:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:47.396+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc3f0109da0 con 0x7fc3f0068a20 2024-03-19T02:53:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.079+0000 7fc3e67fc640 1 -- 172.21.15.116:0/1165309228 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) ==== 169+0+0 (secure 0 0 0) 0x7fc3ec09b520 con 0x7fc3f0068a20 2024-03-19T02:53:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2024-03-19T02:53:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 msgr2=0x7fc3dc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 0x7fc3dc0a4eb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00fd960 tx=0x7fc3e000a2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 msgr2=0x7fc3f0197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00d950 tx=0x7fc3ec00de10 comp rx=0 tx=0).stop 2024-03-19T02:53:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 shutdown_connections 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7fc3dc0a29f0 0x7fc3dc0a4eb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3f0107870 0x7fc3f019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3f00693e0 0x7fc3f0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 --2- 172.21.15.116:0/1165309228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f0068a20 0x7fc3f0197f50 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 >> 172.21.15.116:0/1165309228 conn(0x7fc3f0077820 msgr2=0x7fc3f00fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 shutdown_connections 2024-03-19T02:53:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.081+0000 7fc3f78f1640 1 -- 172.21.15.116:0/1165309228 wait complete. 2024-03-19T02:53:48.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-19T02:53:48.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- 172.21.15.116:0/2610146664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 msgr2=0x7f00a010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:48.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/2610146664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a010d560 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f009c009930 tx=0x7f009c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:48.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- 172.21.15.116:0/2610146664 shutdown_connections 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/2610146664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a010d560 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/2610146664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 0x7f00a0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/2610146664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00a01066a0 0x7f00a0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- 172.21.15.116:0/2610146664 >> 172.21.15.116:0/2610146664 conn(0x7f00a0077820 msgr2=0x7f00a0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- 172.21.15.116:0/2610146664 shutdown_connections 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- 172.21.15.116:0/2610146664 wait complete. 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 Processor -- start 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 -- start start 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 0x7f00a0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.209+0000 7f00a7c9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00a01066a0 0x7f00a019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:48.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0111a60 con 0x7f00a0101050 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01118e0 con 0x7f00a0100690 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0111be0 con 0x7f00a01066a0 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47356/0 (socket says 172.21.15.116:47356) 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 -- 172.21.15.116:0/965372561 learned_addr learned my addr 172.21.15.116:0/965372561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 -- 172.21.15.116:0/965372561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00a01066a0 msgr2=0x7f00a019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00a01066a0 0x7f00a019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 -- 172.21.15.116:0/965372561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 msgr2=0x7f00a0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5a36640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 0x7f00a0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 0x7f00a0197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 -- 172.21.15.116:0/965372561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a019cf70 con 0x7f00a0101050 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a5235640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f009000d950 tx=0x7f009000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:48.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090017070 con 0x7f00a0101050 2024-03-19T02:53:48.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0090009c30 con 0x7f00a0101050 2024-03-19T02:53:48.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a019d260 con 0x7f00a0101050 2024-03-19T02:53:48.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009000f620 con 0x7f00a0101050 2024-03-19T02:53:48.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.210+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a01a4af0 con 0x7f00a0101050 2024-03-19T02:53:48.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.211+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a0068610 con 0x7f00a0101050 2024-03-19T02:53:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.212+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 137422+0+0 (secure 0 0 0) 0x7f00900109a0 con 0x7f00a0101050 2024-03-19T02:53:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.213+0000 7f0096ffd640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 0x7f008c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.213+0000 7f00a5a36640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 0x7f008c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.213+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 8797+0+0 (secure 0 0 0) 0x7f00900cec80 con 0x7f00a0101050 2024-03-19T02:53:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.213+0000 7f00a5a36640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 0x7f008c0a4e30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0098005fd0 tx=0x7f0098009200 comp rx=0 tx=0).ready entity=mgr.25556 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:48.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.217+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0090009de0 con 0x7f00a0101050 2024-03-19T02:53:48.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:48.395+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f00a002d1f0 con 0x7f00a0101050 2024-03-19T02:53:48.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:48 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:48.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:48 smithi162 ceph-mon[35761]: osdmap e137: 12 total, 12 up, 12 in 2024-03-19T02:53:48.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:48 smithi162 ceph-mon[35761]: pgmap v824: 97 pgs: 97 active+clean; 296 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:48.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:48 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:48 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:48.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:48 smithi154 ceph-mon[35693]: osdmap e137: 12 total, 12 up, 12 in 2024-03-19T02:53:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:48 smithi154 ceph-mon[35693]: pgmap v824: 97 pgs: 97 active+clean; 296 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:48.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:48 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:48 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/3870169070' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:48 smithi116 ceph-mon[32263]: osdmap e137: 12 total, 12 up, 12 in 2024-03-19T02:53:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:48 smithi116 ceph-mon[32263]: pgmap v824: 97 pgs: 97 active+clean; 296 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:48.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:48 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.080+0000 7f0096ffd640 1 -- 172.21.15.116:0/965372561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v139) ==== 178+0+0 (secure 0 0 0) 0x7f009009b8d0 con 0x7f00a0101050 2024-03-19T02:53:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' removed 2024-03-19T02:53:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 msgr2=0x7f008c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 0x7f008c0a4e30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0098005fd0 tx=0x7f0098009200 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 msgr2=0x7f00a01984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f009000d950 tx=0x7f009000de10 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 shutdown_connections 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:6800/2695100500,v1:172.21.15.116:6801/2695100500] conn(0x7f008c0a2970 0x7f008c0a4e30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00a01066a0 0x7f00a019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.082+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00a0101050 0x7f00a01984c0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.083+0000 7f00a7c9a640 1 --2- 172.21.15.116:0/965372561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00a0100690 0x7f00a0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.083+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 >> 172.21.15.116:0/965372561 conn(0x7f00a0077820 msgr2=0x7f00a0102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.083+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 shutdown_connections 2024-03-19T02:53:49.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T02:53:49.083+0000 7f00a7c9a640 1 -- 172.21.15.116:0/965372561 wait complete. 2024-03-19T02:53:49.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:49.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:49.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:49.233 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-19T02:53:49.233 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T02:53:49.253 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T02:53:49.278 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T02:53:49.294 INFO:tasks.cephadm:Stopping all daemons... 2024-03-19T02:53:49.294 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-19T02:53:49.294 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a 2024-03-19T02:53:49.352 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:49.353 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 ceph-mon[32263]: osdmap e138: 12 total, 12 up, 12 in 2024-03-19T02:53:49.353 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 ceph-mon[32263]: from='client.? 172.21.15.116:0/965372561' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:49.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:49 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:49.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:49 smithi162 ceph-mon[35761]: osdmap e138: 12 total, 12 up, 12 in 2024-03-19T02:53:49.418 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:49 smithi162 ceph-mon[35761]: from='client.? 172.21.15.116:0/965372561' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:49 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/1165309228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T02:53:49.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:49 smithi154 ceph-mon[35693]: osdmap e138: 12 total, 12 up, 12 in 2024-03-19T02:53:49.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:49 smithi154 ceph-mon[35693]: from='client.? 172.21.15.116:0/965372561' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T02:53:49.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 systemd[1]: Stopping Ceph mon.a for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:50.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a[32259]: 2024-03-19T02:53:49.657+0000 7f650893e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:53:50.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a[32259]: 2024-03-19T02:53:49.657+0000 7f650893e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-19T02:53:50.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:49 smithi116 podman[123074]: 2024-03-19 02:53:49.697415102 +0000 UTC m=+0.173036306 container died e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:53:50.384 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:50 smithi116 podman[123074]: 2024-03-19 02:53:50.115589513 +0000 UTC m=+0.591210729 container cleanup e99971b5917117bf1249190800d94d22170655b0ff90f1547b138393b3a1a8b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-19T02:53:50.384 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 02:53:50 smithi116 bash[123074]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-a 2024-03-19T02:53:50.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.a.service' 2024-03-19T02:53:50.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:50.541 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-19T02:53:50.541 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-19T02:53:50.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.b 2024-03-19T02:53:50.989 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:50 smithi154 systemd[1]: Stopping Ceph mon.b for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-b[35688]: 2024-03-19T02:53:50.988+0000 7f39fcc9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:53:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-b[35688]: 2024-03-19T02:53:50.988+0000 7f39fcc9e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-19T02:53:51.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:51 smithi154 podman[76441]: 2024-03-19 02:53:51.100233383 +0000 UTC m=+0.230009752 container died f2d9dcebfc1f0e0c76ececb66a0e9d49a611923d60953d8fd372d3ddb0fd792b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-19T02:53:51.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:51 smithi154 podman[76441]: 2024-03-19 02:53:51.505979498 +0000 UTC m=+0.635755866 container cleanup f2d9dcebfc1f0e0c76ececb66a0e9d49a611923d60953d8fd372d3ddb0fd792b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0) 2024-03-19T02:53:51.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 19 02:53:51 smithi154 bash[76441]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-b 2024-03-19T02:53:51.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.b.service' 2024-03-19T02:53:51.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:51.964 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-19T02:53:51.965 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-19T02:53:51.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.c 2024-03-19T02:53:52.369 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:52 smithi162 systemd[1]: Stopping Ceph mon.c for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c[35757]: 2024-03-19T02:53:52.368+0000 7f6de1632700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:53:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c[35757]: 2024-03-19T02:53:52.368+0000 7f6de1632700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-19T02:53:52.668 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:52 smithi162 podman[74873]: 2024-03-19 02:53:52.61412705 +0000 UTC m=+0.447558470 container died ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-19T02:53:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:53 smithi162 podman[74873]: 2024-03-19 02:53:53.002542462 +0000 UTC m=+0.835973881 container cleanup ee7ec7f8e7aad57ecd83b1b8a3e20279f77836ef863d9a63b1b7ac0e4db735a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-19T02:53:53.168 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 02:53:53 smithi162 bash[74873]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mon-c 2024-03-19T02:53:53.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mon.c.service' 2024-03-19T02:53:53.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:53.480 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-19T02:53:53.480 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-19T02:53:53.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x 2024-03-19T02:53:53.759 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:53 smithi116 systemd[1]: Stopping Ceph mgr.x for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:54.122 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:53:53.786+0000 7fdda2466700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:53:54.122 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x[32492]: 2024-03-19T02:53:53.786+0000 7fdda2466700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-19T02:53:54.122 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:53 smithi116 podman[123195]: 2024-03-19 02:53:53.818677011 +0000 UTC m=+0.159163931 container died 32a03d90c7164f57e886350c7a08cbcde2a63939ca49a5d5fc9f5c655133c3ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-19T02:53:54.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:54 smithi116 podman[123195]: 2024-03-19 02:53:54.20508511 +0000 UTC m=+0.545572036 container cleanup 32a03d90c7164f57e886350c7a08cbcde2a63939ca49a5d5fc9f5c655133c3ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-19T02:53:54.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 02:53:54 smithi116 bash[123195]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-x 2024-03-19T02:53:54.626 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-19T02:53:54.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:54.664 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-19T02:53:54.664 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-19T02:53:54.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.y 2024-03-19T02:53:55.006 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:53:54 smithi154 systemd[1]: Stopping Ceph mgr.y for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:55.256 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:53:55 smithi154 podman[76562]: 2024-03-19 02:53:55.005784447 +0000 UTC m=+0.151958306 container died c129d349b125d44f10fc7f6ce8b463ee8b87c629625d04af28a034c9f08c4074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-19T02:53:55.613 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:53:55 smithi154 podman[76562]: 2024-03-19 02:53:55.352574461 +0000 UTC m=+0.498748316 container cleanup c129d349b125d44f10fc7f6ce8b463ee8b87c629625d04af28a034c9f08c4074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-19T02:53:55.613 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 19 02:53:55 smithi154 bash[76562]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-y 2024-03-19T02:53:55.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-19T02:53:55.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:55.851 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-19T02:53:55.852 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-19T02:53:55.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.z 2024-03-19T02:53:56.168 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:53:55 smithi162 systemd[1]: Stopping Ceph mgr.z for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:56.484 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:53:56 smithi162 podman[74996]: 2024-03-19 02:53:56.17848616 +0000 UTC m=+0.138141841 container died bd5ac1cbffd9c0364405785c70ab3831ac04a181e17efc63867dae268f18a681 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:53:56.827 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:53:56 smithi162 podman[74996]: 2024-03-19 02:53:56.575310823 +0000 UTC m=+0.534966497 container cleanup bd5ac1cbffd9c0364405785c70ab3831ac04a181e17efc63867dae268f18a681 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-19T02:53:56.827 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 02:53:56 smithi162 bash[74996]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mgr-z 2024-03-19T02:53:56.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-19T02:53:57.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:53:57.032 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-19T02:53:57.032 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-19T02:53:57.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.0 2024-03-19T02:53:57.353 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:53:57 smithi116 systemd[1]: Stopping Ceph osd.0 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:53:57.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:53:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0[44508]: 2024-03-19T02:53:57.351+0000 7f1b9e402700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:53:57.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:53:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0[44508]: 2024-03-19T02:53:57.351+0000 7f1b9e402700 -1 osd.0 139 *** Got signal Terminated *** 2024-03-19T02:53:57.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:53:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0[44508]: 2024-03-19T02:53:57.351+0000 7f1b9e402700 -1 osd.0 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:03.037 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:02 smithi116 podman[123316]: 2024-03-19 02:54:02.777915553 +0000 UTC m=+5.544770629 container died ff044e02929c30e06cddf60ad664ab82489f54b560ec2bb13e3c2342eb366e1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-19T02:54:03.454 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:03 smithi116 podman[123316]: 2024-03-19 02:54:03.152734477 +0000 UTC m=+5.919589552 container cleanup ff044e02929c30e06cddf60ad664ab82489f54b560ec2bb13e3c2342eb366e1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:03.454 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:03 smithi116 bash[123316]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0 2024-03-19T02:54:03.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:03 smithi116 podman[123333]: 2024-03-19 02:54:03.454210073 +0000 UTC m=+0.674369425 container remove ff044e02929c30e06cddf60ad664ab82489f54b560ec2bb13e3c2342eb366e1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-19T02:54:04.059 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:03 smithi116 podman[123423]: 2024-03-19 02:54:03.670046922 +0000 UTC m=+0.017802260 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:04.059 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:03 smithi116 podman[123423]: 2024-03-19 02:54:03.796752654 +0000 UTC m=+0.144508021 container create b7f936729e612ae1d5eb1d251070983a9746a7e06754af9bcb47762487d34003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T02:54:04.355 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:04 smithi116 podman[123423]: 2024-03-19 02:54:04.058667295 +0000 UTC m=+0.406422636 container init b7f936729e612ae1d5eb1d251070983a9746a7e06754af9bcb47762487d34003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-19T02:54:04.355 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:04 smithi116 podman[123423]: 2024-03-19 02:54:04.062110754 +0000 UTC m=+0.409866081 container start b7f936729e612ae1d5eb1d251070983a9746a7e06754af9bcb47762487d34003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:54:04.355 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:04 smithi116 podman[123423]: 2024-03-19 02:54:04.158346619 +0000 UTC m=+0.506101941 container attach b7f936729e612ae1d5eb1d251070983a9746a7e06754af9bcb47762487d34003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T02:54:04.736 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 02:54:04 smithi116 podman[123423]: 2024-03-19 02:54:04.355107406 +0000 UTC m=+0.702862723 container died b7f936729e612ae1d5eb1d251070983a9746a7e06754af9bcb47762487d34003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:54:05.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.0.service' 2024-03-19T02:54:05.094 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:05.095 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-19T02:54:05.095 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-19T02:54:05.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.1 2024-03-19T02:54:05.450 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:05 smithi154 systemd[1]: Stopping Ceph osd.1 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:05.756 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:54:05.448+0000 7f98bc360700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:05.756 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:54:05.448+0000 7f98bc360700 -1 osd.1 139 *** Got signal Terminated *** 2024-03-19T02:54:05.756 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1[40280]: 2024-03-19T02:54:05.448+0000 7f98bc360700 -1 osd.1 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:11.193 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:10 smithi154 podman[76698]: 2024-03-19 02:54:10.892093049 +0000 UTC m=+5.569240841 container died 71cb2d9bb61e0b2df64fa9a1e3b43b90297e10a28ff279581230b85102083a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T02:54:11.486 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:11 smithi154 podman[76698]: 2024-03-19 02:54:11.257790738 +0000 UTC m=+5.934938537 container cleanup 71cb2d9bb61e0b2df64fa9a1e3b43b90297e10a28ff279581230b85102083a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-19T02:54:11.487 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:11 smithi154 bash[76698]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1 2024-03-19T02:54:11.756 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:11 smithi154 podman[76718]: 2024-03-19 02:54:11.486601277 +0000 UTC m=+0.589282336 container remove 71cb2d9bb61e0b2df64fa9a1e3b43b90297e10a28ff279581230b85102083a39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-19T02:54:12.240 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:11 smithi154 podman[76809]: 2024-03-19 02:54:11.825593547 +0000 UTC m=+0.018358820 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:12.240 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:11 smithi154 podman[76809]: 2024-03-19 02:54:11.987621067 +0000 UTC m=+0.180386325 container create 8dec71ef929e62f0b0525a2134478ccff2e41a7b6a41a6978c916cfdcdf15773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:12.240 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:12 smithi154 podman[76809]: 2024-03-19 02:54:12.240092622 +0000 UTC m=+0.432857888 container init 8dec71ef929e62f0b0525a2134478ccff2e41a7b6a41a6978c916cfdcdf15773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-19T02:54:12.506 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:12 smithi154 podman[76809]: 2024-03-19 02:54:12.24376311 +0000 UTC m=+0.436528367 container start 8dec71ef929e62f0b0525a2134478ccff2e41a7b6a41a6978c916cfdcdf15773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:12.506 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:12 smithi154 podman[76809]: 2024-03-19 02:54:12.381966418 +0000 UTC m=+0.574731686 container attach 8dec71ef929e62f0b0525a2134478ccff2e41a7b6a41a6978c916cfdcdf15773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T02:54:14.219 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 19 02:54:13 smithi154 podman[76809]: 2024-03-19 02:54:13.843401835 +0000 UTC m=+2.036167093 container died 8dec71ef929e62f0b0525a2134478ccff2e41a7b6a41a6978c916cfdcdf15773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:54:14.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.1.service' 2024-03-19T02:54:14.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:14.629 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-19T02:54:14.629 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-19T02:54:14.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.2 2024-03-19T02:54:14.918 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:14 smithi162 systemd[1]: Stopping Ceph osd.2 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:15.418 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2[40891]: 2024-03-19T02:54:14.952+0000 7fcc65f64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:15.418 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2[40891]: 2024-03-19T02:54:14.952+0000 7fcc65f64700 -1 osd.2 139 *** Got signal Terminated *** 2024-03-19T02:54:15.418 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2[40891]: 2024-03-19T02:54:14.952+0000 7fcc65f64700 -1 osd.2 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:20.754 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:20 smithi162 podman[75134]: 2024-03-19 02:54:20.457154283 +0000 UTC m=+5.631764777 container died e47de26fe76ac456bd98f1affaddeef2a6c69419eb5fca50357ebd7d152fe7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:54:21.113 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:20 smithi162 podman[75134]: 2024-03-19 02:54:20.845143591 +0000 UTC m=+6.019754074 container cleanup e47de26fe76ac456bd98f1affaddeef2a6c69419eb5fca50357ebd7d152fe7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-19T02:54:21.113 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:20 smithi162 bash[75134]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2 2024-03-19T02:54:21.418 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75152]: 2024-03-19 02:54:21.113236554 +0000 UTC m=+0.658239880 container remove e47de26fe76ac456bd98f1affaddeef2a6c69419eb5fca50357ebd7d152fe7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-19T02:54:21.760 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75246]: 2024-03-19 02:54:21.359378616 +0000 UTC m=+0.017597837 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:21.760 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75246]: 2024-03-19 02:54:21.485667424 +0000 UTC m=+0.143886668 container create b9f54378d6fa605c8babe3f761245cc32b7a39e29b7d1d222a7e506637ab54b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD) 2024-03-19T02:54:21.760 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75246]: 2024-03-19 02:54:21.759299132 +0000 UTC m=+0.417518345 container init b9f54378d6fa605c8babe3f761245cc32b7a39e29b7d1d222a7e506637ab54b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-19T02:54:22.048 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75246]: 2024-03-19 02:54:21.762482259 +0000 UTC m=+0.420701468 container start b9f54378d6fa605c8babe3f761245cc32b7a39e29b7d1d222a7e506637ab54b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:22.048 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:21 smithi162 podman[75246]: 2024-03-19 02:54:21.859356529 +0000 UTC m=+0.517575744 container attach b9f54378d6fa605c8babe3f761245cc32b7a39e29b7d1d222a7e506637ab54b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-19T02:54:22.323 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 02:54:22 smithi162 podman[75246]: 2024-03-19 02:54:22.04787378 +0000 UTC m=+0.706093002 container died b9f54378d6fa605c8babe3f761245cc32b7a39e29b7d1d222a7e506637ab54b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T02:54:22.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.2.service' 2024-03-19T02:54:22.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:22.734 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-19T02:54:22.734 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-19T02:54:22.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.3 2024-03-19T02:54:23.065 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:22 smithi116 systemd[1]: Stopping Ceph osd.3 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:23.331 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:23 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3[51460]: 2024-03-19T02:54:23.064+0000 7f53a50f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:23.331 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:23 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3[51460]: 2024-03-19T02:54:23.064+0000 7f53a50f9700 -1 osd.3 139 *** Got signal Terminated *** 2024-03-19T02:54:23.331 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:23 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3[51460]: 2024-03-19T02:54:23.064+0000 7f53a50f9700 -1 osd.3 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:24.257 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:23.905+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:24.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:23 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:23.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:25.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:24 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:24.855+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:25.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:24 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:24.774+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:26.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:25 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:25.843+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:26.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:25 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:25.793+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:26.600 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:26.275+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:26.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:26.599+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:27.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:26 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:26.824+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:27.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:26 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:26.838+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:27.574 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:27.315+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:27.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:27.574+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:28.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:27 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:27.777+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:28.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:27 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:27.874+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:28.607 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:28.337+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:28.867 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:28 smithi116 podman[123530]: 2024-03-19 02:54:28.572790628 +0000 UTC m=+5.644711441 container died c3e7c627252409bfe6f9169ade03e7ec55a80bbf1d2a544855333c78c434e04a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-19T02:54:28.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:28.606+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:29.242 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:28 smithi116 podman[123530]: 2024-03-19 02:54:28.981534274 +0000 UTC m=+6.053455092 container cleanup c3e7c627252409bfe6f9169ade03e7ec55a80bbf1d2a544855333c78c434e04a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-19T02:54:29.242 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:28 smithi116 bash[123530]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3 2024-03-19T02:54:29.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:28 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:28.866+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:29.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:28 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:28.764+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:29.509 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:29 smithi116 podman[123560]: 2024-03-19 02:54:29.241346262 +0000 UTC m=+0.671650817 container remove c3e7c627252409bfe6f9169ade03e7ec55a80bbf1d2a544855333c78c434e04a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-19T02:54:29.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:29.614+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:29.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:29.378+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:29.912 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:29 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:29.844+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:29.912 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:29 smithi116 podman[123654]: 2024-03-19 02:54:29.55579069 +0000 UTC m=+0.017393689 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:29.912 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:29 smithi116 podman[123654]: 2024-03-19 02:54:29.683958022 +0000 UTC m=+0.145561011 container create a1cfa60eabc6624003195437964f0944d51f500168686c05f18c9443a2413661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:29.912 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:29 smithi116 podman[123654]: 2024-03-19 02:54:29.91207688 +0000 UTC m=+0.373679881 container init a1cfa60eabc6624003195437964f0944d51f500168686c05f18c9443a2413661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:54:30.205 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:29 smithi116 podman[123654]: 2024-03-19 02:54:29.915740994 +0000 UTC m=+0.377343983 container start a1cfa60eabc6624003195437964f0944d51f500168686c05f18c9443a2413661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:30.206 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:30 smithi116 podman[123654]: 2024-03-19 02:54:30.054166682 +0000 UTC m=+0.515769673 container attach a1cfa60eabc6624003195437964f0944d51f500168686c05f18c9443a2413661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:54:30.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:29 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:29.795+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:30.509 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 02:54:30 smithi116 podman[123654]: 2024-03-19 02:54:30.205672771 +0000 UTC m=+0.667275762 container died a1cfa60eabc6624003195437964f0944d51f500168686c05f18c9443a2413661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-19T02:54:30.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:30.614+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:30.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:30.384+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:30.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.3.service' 2024-03-19T02:54:30.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:30.848 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-19T02:54:30.849 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-19T02:54:30.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.4 2024-03-19T02:54:31.172 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:30 smithi154 systemd[1]: Stopping Ceph osd.4 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:31.172 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:30 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:30.763+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:31.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:30 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:30.888+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:31.509 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:31 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:31.171+0000 7ff612c81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:31.509 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:31 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:31.171+0000 7ff612c81700 -1 osd.4 139 *** Got signal Terminated *** 2024-03-19T02:54:31.509 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:31 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:31.171+0000 7ff612c81700 -1 osd.4 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:31.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:31.577+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:31.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:31.377+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:31.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:31.377+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:32.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:31 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:31.732+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:32.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:31 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:31.864+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:32.506 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:32 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:32.112+0000 7ff60b26d700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.738179+0000 front 2024-03-19T02:54:09.738154+0000 (oldest deadline 2024-03-19T02:54:31.438361+0000) 2024-03-19T02:54:32.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:32.624+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:32.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:32.624+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:32.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:32.375+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:32.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:32.375+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:33.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:32 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:32.762+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:33.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:32 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:32.762+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:33.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:32 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:32.540+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:33.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:32 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:32.913+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:33.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:32 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:32.913+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:33.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:33.144+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:33.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:33.370+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:33.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:33.370+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:33.506 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:33 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:33.063+0000 7ff60b26d700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.738179+0000 front 2024-03-19T02:54:09.738154+0000 (oldest deadline 2024-03-19T02:54:31.438361+0000) 2024-03-19T02:54:33.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:33.656+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:33.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:33 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:33.656+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:34.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:33 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:33.570+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:34.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:33 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:33.777+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:34.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:33 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:33.777+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:34.250 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:33 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:33.956+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:34.250 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:33 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:33.956+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:34.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:34 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:34.122+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:34.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:34 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:34.370+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:34.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:34 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:34.370+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:34.506 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:34 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:34.061+0000 7ff60b26d700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.738179+0000 front 2024-03-19T02:54:09.738154+0000 (oldest deadline 2024-03-19T02:54:31.438361+0000) 2024-03-19T02:54:34.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:34.249+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:35.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:34 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:34.610+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:35.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:34 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:34.790+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:35.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:34 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:34.790+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:35.118 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:34 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:34.703+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:35.118 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:34 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:34.703+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:35.255 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:34.931+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:35.255 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:34 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:34.931+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:35.391 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:35 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:35.117+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:35.506 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:35 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:35.069+0000 7ff60b26d700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.738179+0000 front 2024-03-19T02:54:09.738154+0000 (oldest deadline 2024-03-19T02:54:31.438361+0000) 2024-03-19T02:54:35.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:35.254+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:35.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:35 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:35.391+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:35.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:35 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:35.391+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:36.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:35 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:35.642+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:36.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:35 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:35.828+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:36.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:35 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:35.829+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:36.111 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:35 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:35.700+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:36.111 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:35 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:35.700+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:36.221 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:35.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:36.222 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:35.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:36.411 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:36 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:36.110+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:36.506 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4[46921]: 2024-03-19T02:54:36.117+0000 7ff60b26d700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.738179+0000 front 2024-03-19T02:54:09.738154+0000 (oldest deadline 2024-03-19T02:54:31.438361+0000) 2024-03-19T02:54:36.508 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:36.220+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:36.663 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:36 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:36.410+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:36.664 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:36 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:36.410+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:36.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:36 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:36.662+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:36.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:36 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:36.662+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:36.980 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:36.677+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:36.980 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:36 smithi154 podman[76915]: 2024-03-19 02:54:36.672117459 +0000 UTC m=+5.626663547 container died bbf5d75fa2ac1b2213122c73ddcce8d97ba308aa2d9d6cb62a40da96753c29df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-19T02:54:36.980 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:36.836+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:36.980 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:36 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:36.836+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:37.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:36.935+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:37.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:36 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:36.935+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:37.256 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[76915]: 2024-03-19 02:54:37.080190737 +0000 UTC m=+6.034736834 container cleanup bbf5d75fa2ac1b2213122c73ddcce8d97ba308aa2d9d6cb62a40da96753c29df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:54:37.256 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 bash[76915]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4 2024-03-19T02:54:37.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:37 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:37.134+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:37.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:37.241+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:37.682 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[76934]: 2024-03-19 02:54:37.372966046 +0000 UTC m=+0.660092731 container remove bbf5d75fa2ac1b2213122c73ddcce8d97ba308aa2d9d6cb62a40da96753c29df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-19T02:54:37.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:37 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:37.436+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:37.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:37 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:37.436+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:37.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:37 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:37.684+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:37.918 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:37 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:37.684+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:37.960 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:37 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:37.722+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:37.961 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:37 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:37.838+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:37.961 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:37 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:37.838+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:37.961 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[77025]: 2024-03-19 02:54:37.682020906 +0000 UTC m=+0.108949018 container create 895d4d624781ad14ada63480b4dadcfcabdda1682491c68b09bf5ca385ff15b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:37.961 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[77025]: 2024-03-19 02:54:37.591904689 +0000 UTC m=+0.018832820 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:37.961 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[77025]: 2024-03-19 02:54:37.960431824 +0000 UTC m=+0.387359950 container init 895d4d624781ad14ada63480b4dadcfcabdda1682491c68b09bf5ca385ff15b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-19T02:54:38.257 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:37 smithi154 podman[77025]: 2024-03-19 02:54:37.963631694 +0000 UTC m=+0.390559811 container start 895d4d624781ad14ada63480b4dadcfcabdda1682491c68b09bf5ca385ff15b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:38.257 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:38 smithi154 podman[77025]: 2024-03-19 02:54:38.026876361 +0000 UTC m=+0.453804492 container attach 895d4d624781ad14ada63480b4dadcfcabdda1682491c68b09bf5ca385ff15b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T02:54:38.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:37.934+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:38.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:37 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:37.934+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:38.395 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:38 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:38.093+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:38.571 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 19 02:54:38 smithi154 podman[77025]: 2024-03-19 02:54:38.259269678 +0000 UTC m=+0.686197782 container died 895d4d624781ad14ada63480b4dadcfcabdda1682491c68b09bf5ca385ff15b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T02:54:38.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:38 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:38.395+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:38.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:38 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:38.395+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:38.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:38.290+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:38.864 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:38 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:38.766+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:38.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.4.service' 2024-03-19T02:54:38.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:38.929 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-19T02:54:38.929 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-19T02:54:38.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.5 2024-03-19T02:54:38.960 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:38 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:38.687+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:38.960 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:38 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:38.687+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:39.241 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:38.900+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:39.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:38.900+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:39.249 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:38 smithi162 systemd[1]: Stopping Ceph osd.5 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:39.250 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:39.125+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:39.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:38 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:38.885+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:39.256 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:38 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:38.885+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:39.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:39.241+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:39.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:39.249+0000 7f56718c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:39.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:39.249+0000 7f56718c5700 -1 osd.5 139 *** Got signal Terminated *** 2024-03-19T02:54:39.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:39.249+0000 7f56718c5700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:39.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:39.423+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:39.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:39.423+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:40.116 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:39.676+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:40.116 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:39 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:39.676+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:40.257 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:39.765+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:40.257 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:39.906+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:40.257 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:39 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:39.906+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:40.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:39.926+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:40.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:39 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:39.926+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:40.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:40.115+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:40.693 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:40.430+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:40.693 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:40.430+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:40.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:40 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:40.275+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:41.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:40 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:40.720+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:41.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:40 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:40.894+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:41.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:40 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:40.894+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:41.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:40 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:40.894+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:41.158 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:40.693+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:41.159 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:40.693+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:41.159 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:40 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:40.693+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.958404+0000 front 2024-03-19T02:54:18.958426+0000 (oldest deadline 2024-03-19T02:54:40.658612+0000) 2024-03-19T02:54:41.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:40 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:40.922+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:41.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:40 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:40.922+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:41.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:41.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:41.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:41.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:41.727 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:41.445+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:41.727 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:41.445+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:41.727 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:41.445+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:41.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:41.282+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:42.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:41.718+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:42.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:41.927+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:42.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:41.927+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:42.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:41 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:41.927+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:42.161 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:41.726+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:42.162 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:41.726+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:42.162 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:41 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:41.726+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.958404+0000 front 2024-03-19T02:54:18.958426+0000 (oldest deadline 2024-03-19T02:54:40.658612+0000) 2024-03-19T02:54:42.251 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:41.908+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:42.252 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:41.908+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:42.252 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:41 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:41.908+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:42.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:42.160+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:42.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:42.160+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:42.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:42.251+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:42.708 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:42.430+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:42.708 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:42.430+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:42.709 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:42.430+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:43.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:42.747+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:43.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:42.954+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:43.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:42.954+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:43.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:42 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:42.954+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:43.168 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:42.707+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:43.168 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:42.707+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:43.168 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:42 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:42.707+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.958404+0000 front 2024-03-19T02:54:18.958426+0000 (oldest deadline 2024-03-19T02:54:40.658612+0000) 2024-03-19T02:54:43.202 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:42.909+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:43.202 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:42.909+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:43.202 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:42.909+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:43.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:43.201+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:43.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:43.201+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:43.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:43.170+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:43.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:43.171+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:43.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:43.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:43.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:43.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:43.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:43.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:44.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:43.743+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:44.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:43.915+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:44.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:43.915+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:44.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:43 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:43.915+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:44.122 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:43.703+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.154678+0000 front 2024-03-19T02:53:59.154653+0000 (oldest deadline 2024-03-19T02:54:26.155286+0000) 2024-03-19T02:54:44.122 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:43.703+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.155776+0000 front 2024-03-19T02:54:06.155748+0000 (oldest deadline 2024-03-19T02:54:32.055844+0000) 2024-03-19T02:54:44.122 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:43 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5[47704]: 2024-03-19T02:54:43.703+0000 7f566deb9700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.958404+0000 front 2024-03-19T02:54:18.958426+0000 (oldest deadline 2024-03-19T02:54:40.658612+0000) 2024-03-19T02:54:44.165 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:43.860+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:44.165 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:43.860+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:44.165 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:43 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:43.860+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:44.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:44.121+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:44.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:44.121+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:44.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:44.354+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:44.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:44.354+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:44.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:44 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:44.354+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:44.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:44.165+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:44.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:44.165+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:45.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:44.721+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:45.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:44.893+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:45.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:44.893+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:45.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:44 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:44.893+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:45.108 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:44 smithi162 podman[75378]: 2024-03-19 02:54:44.848961612 +0000 UTC m=+5.724654309 container died 640d9be4a59ee7af31f419c0d446cf14bf234bf43fbc4d0ec5d62abc55809309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-19T02:54:45.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:45.135+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:45.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:45.135+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:45.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:44.888+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:45.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:44.888+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:45.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:44 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:44.888+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:45.418 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:45 smithi162 podman[75378]: 2024-03-19 02:54:45.19866377 +0000 UTC m=+6.074356469 container cleanup 640d9be4a59ee7af31f419c0d446cf14bf234bf43fbc4d0ec5d62abc55809309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T02:54:45.418 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:45 smithi162 bash[75378]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5 2024-03-19T02:54:45.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:45.170+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:45.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:45.170+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:45.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:45.353+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:45.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:45.353+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:45.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:45 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:45.353+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:45.814 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:45 smithi162 podman[75399]: 2024-03-19 02:54:45.525157547 +0000 UTC m=+0.670310020 container remove 640d9be4a59ee7af31f419c0d446cf14bf234bf43fbc4d0ec5d62abc55809309 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:54:46.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:45.731+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:46.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:45.731+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:46.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:45.850+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:46.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:45.850+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:46.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:45 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:45.850+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:46.080 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:45 smithi162 podman[75492]: 2024-03-19 02:54:45.716845801 +0000 UTC m=+0.017528922 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:46.080 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:45 smithi162 podman[75492]: 2024-03-19 02:54:45.826239226 +0000 UTC m=+0.126922348 container create 63f4ddc2f14eaa887839d9afed53d8b752d55e97dbfaa8d4420e6dbe52526899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True) 2024-03-19T02:54:46.080 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:46 smithi162 podman[75492]: 2024-03-19 02:54:46.079644809 +0000 UTC m=+0.380327950 container init 63f4ddc2f14eaa887839d9afed53d8b752d55e97dbfaa8d4420e6dbe52526899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:54:46.260 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:46.100+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:46.260 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:46.100+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:46.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:45.932+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:46.261 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:45.932+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:46.261 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:45 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:45.932+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:46.367 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:46.150+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:46.368 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:46.150+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:46.368 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:46 smithi162 podman[75492]: 2024-03-19 02:54:46.08391504 +0000 UTC m=+0.384598172 container start 63f4ddc2f14eaa887839d9afed53d8b752d55e97dbfaa8d4420e6dbe52526899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-19T02:54:46.368 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:46 smithi162 podman[75492]: 2024-03-19 02:54:46.162526644 +0000 UTC m=+0.463209773 container attach 63f4ddc2f14eaa887839d9afed53d8b752d55e97dbfaa8d4420e6dbe52526899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T02:54:46.668 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 02:54:46 smithi162 podman[75492]: 2024-03-19 02:54:46.379667335 +0000 UTC m=+0.680350457 container died 63f4ddc2f14eaa887839d9afed53d8b752d55e97dbfaa8d4420e6dbe52526899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T02:54:46.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:46.366+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:46.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:46.366+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:46.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:46 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:46.366+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:47.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:46.701+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:47.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:46.701+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:47.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:46.826+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:47.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:46.826+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:47.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:46 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:46.826+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:47.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.5.service' 2024-03-19T02:54:47.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:47.135 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-19T02:54:47.135 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-19T02:54:47.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.6 2024-03-19T02:54:47.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:46.983+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:47.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:46.983+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:47.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:46.983+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:47.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:47.088+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:47.260 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:47.088+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:47.260 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 systemd[1]: Stopping Ceph osd.6 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:47.369 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:47.174+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:47.369 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:47.174+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:47.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:47.368+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:47.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:47.368+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:47.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:47 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:47.368+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:47.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:47.444+0000 7f0c5a6d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:47.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:47.444+0000 7f0c5a6d8700 -1 osd.6 139 *** Got signal Terminated *** 2024-03-19T02:54:47.759 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:47.444+0000 7f0c5a6d8700 -1 osd.6 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:48.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:47.690+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:48.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:47.690+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:48.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:47.863+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:48.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:47.863+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:48.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:47 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:47.863+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:48.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:48.055+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:48.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:48.055+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:48.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:47.953+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:48.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:47.953+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:48.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:47 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:47.953+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:48.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:48.395+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:48.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:48.395+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:48.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:48.395+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:48.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:48.161+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:48.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:48 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:48.161+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:49.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:48.688+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:49.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:48.688+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:49.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:48.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:49.008 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:48.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:49.008 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:48 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:48.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:49.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:49.031+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:49.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:49.031+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:49.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:48.924+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:49.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:48.924+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:49.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:48 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:48.924+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:49.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:49.149+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:49.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:49.149+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:49.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:49.424+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:49.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:49.424+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:49.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:49 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:49.424+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:50.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:49.654+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:50.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:49.654+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:50.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:49.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:50.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:49.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:50.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:49 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:49.864+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:50.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:50 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:50.061+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:50.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:50 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:50.061+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:50.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:49.875+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:50.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:49.875+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:50.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:49 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:49.875+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:50.403 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:50.143+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:50.403 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:50.143+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:50.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:50.402+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:50.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:50.402+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:50.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:50 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:50.402+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:51.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:50.605+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:51.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:50.605+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:51.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:50.605+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:51.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:50.822+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:51.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:50.822+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:51.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:50 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:50.822+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:51.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:50 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:50.837+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:51.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:50 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:50.837+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:51.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:50 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:50.837+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:51.355 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:51 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:51.097+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:51.356 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:51 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:51.097+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:51.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:51.105+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:51.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:51.105+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:51.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:51.105+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.896103+0000 front 2024-03-19T02:54:26.896126+0000 (oldest deadline 2024-03-19T02:54:50.996400+0000) 2024-03-19T02:54:51.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:51 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:51.355+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:51.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:51 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:51.355+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:51.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:51 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:51.355+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:52.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:51.636+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:52.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:51.636+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:52.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:51.636+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:52.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:51.791+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:52.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:51.791+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:52.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:51.791+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:52.136 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:51.835+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:52.136 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:51.835+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:52.136 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:51.835+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:52.136 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:51 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:51.835+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:52.388 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:52.058+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:52.388 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:52.058+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:52.388 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:52.058+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:52.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:52.134+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.291319+0000 front 2024-03-19T02:54:10.291363+0000 (oldest deadline 2024-03-19T02:54:33.791429+0000) 2024-03-19T02:54:52.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:52.134+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:17.693755+0000 front 2024-03-19T02:54:17.693771+0000 (oldest deadline 2024-03-19T02:54:42.993848+0000) 2024-03-19T02:54:52.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6[57990]: 2024-03-19T02:54:52.134+0000 7f0c52cc4700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.896103+0000 front 2024-03-19T02:54:26.896126+0000 (oldest deadline 2024-03-19T02:54:50.996400+0000) 2024-03-19T02:54:52.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:52.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:52.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:52.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:52.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:52 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:52.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:53.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:52.659+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:52.659+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:52.659+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:52.749+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:52.749+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:52.749+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:53.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:52 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:52.749+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:53.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 podman[123761]: 2024-03-19 02:54:53.037840317 +0000 UTC m=+5.708235540 container died 888e55a5cf28a624dd1b59912089639f2d40a5f7dead8a008ad0bf81353e17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-19T02:54:53.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:52.829+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:53.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:52.829+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:53.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:52.829+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:53.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:52 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:52.829+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:53.389 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:53.026+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:53.389 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:53.026+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:53.389 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:53.026+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:53.554 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 podman[123761]: 2024-03-19 02:54:53.34382766 +0000 UTC m=+6.014222881 container cleanup 888e55a5cf28a624dd1b59912089639f2d40a5f7dead8a008ad0bf81353e17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-19T02:54:53.554 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 bash[123761]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6 2024-03-19T02:54:53.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:53.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:53.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:53.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:53.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:53 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:53.387+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:53.811 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 podman[123778]: 2024-03-19 02:54:53.553626756 +0000 UTC m=+0.514910953 container remove 888e55a5cf28a624dd1b59912089639f2d40a5f7dead8a008ad0bf81353e17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T02:54:54.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:53.633+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:54.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:53.633+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:54.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:53.633+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:54.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:53.745+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:54.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:53.745+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:54.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:53.745+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:54.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:53 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:53.745+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:54.071 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 podman[123868]: 2024-03-19 02:54:53.819716601 +0000 UTC m=+0.017614691 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:54:54.072 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:53 smithi116 podman[123868]: 2024-03-19 02:54:53.954292413 +0000 UTC m=+0.152190474 container create b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-19T02:54:54.072 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:53.809+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:54.072 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:53.809+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:54.072 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:53.809+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:54.072 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:53 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:53.809+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:54.342 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:54.068+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:54.342 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:54.068+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:54.342 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:54.068+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:54.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:54 smithi116 podman[123868]: 2024-03-19 02:54:54.215714367 +0000 UTC m=+0.413612449 container init b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:54:54.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:54 smithi116 podman[123868]: 2024-03-19 02:54:54.219089164 +0000 UTC m=+0.416987233 container start b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:54:54.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:54 smithi116 podman[123868]: 2024-03-19 02:54:54.298606959 +0000 UTC m=+0.496505025 container attach b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-19T02:54:54.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:54.341+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:54.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:54.341+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:54.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:54 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:54.341+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:54.852 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:54 smithi116 podman[123868]: 2024-03-19 02:54:54.51327934 +0000 UTC m=+0.711177409 container died b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T02:54:55.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:54.638+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:55.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:54.638+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:55.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:54.638+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:55.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:54.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:55.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:54.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:55.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:54.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:55.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:54.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:55.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.6.service' 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:55 smithi116 podman[123894]: 2024-03-19 02:54:55.060986025 +0000 UTC m=+0.537671647 container remove b0f07db2901255971e0a9eb74e908d7e778ab8a868ffb90799c37872fd81c025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:55 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:55 smithi116 systemd[1]: Stopped Ceph osd.6 for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 02:54:55 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.6.service: Consumed 3min 13.844s CPU time. 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:54.850+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:54.850+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:54.850+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:55.107 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:54 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:54.850+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:55.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:54:55.123 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-19T02:54:55.123 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-19T02:54:55.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.7 2024-03-19T02:54:55.302 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:55.045+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:55.302 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:55.045+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:55.302 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:55.045+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:55.450 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 systemd[1]: Stopping Ceph osd.7 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:54:55.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:55.300+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:55.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:55.300+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:55.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:55 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:55.300+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:55.721 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.448+0000 7f6c7686b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:54:55.721 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.448+0000 7f6c7686b700 -1 osd.7 139 *** Got signal Terminated *** 2024-03-19T02:54:55.722 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.448+0000 7f6c7686b700 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:54:55.722 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.596+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:55.722 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.596+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:55.722 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:55.596+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:56.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:55.719+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:56.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:55.719+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:56.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:55.719+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:56.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:55 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:55.719+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:56.249 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:55 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:55.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:56.249 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:55 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:55.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:56.249 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:55 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:55.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:56.249 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:55 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:55.894+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:56.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:56.093+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:56.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:56.093+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:56.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:56.093+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:56.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:56.313+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:56.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:56.313+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:56.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:56 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:56.313+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:56.612+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:56.612+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:56.612+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:56.757+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:56.757+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:56.757+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:57.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:56 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:56.757+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:57.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:56 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:56.846+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:57.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:56 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:56.846+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:57.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:56 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:56.846+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:57.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:56 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:56.846+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:57.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:57.098+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:57.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:57.098+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:57.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:57.098+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:57.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:57.295+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:57.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:57.295+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:57.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:57.295+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:57.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:57 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:57.295+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:57.564+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:57.564+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:57.564+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:57.721+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:57.721+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:57.721+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:58.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:57 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:57.721+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:58.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:57.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:58.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:57.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:58.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:57.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:58.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:57.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:58.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:57 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:57.805+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:54:58.345 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:58.051+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:58.345 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:58.051+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:58.345 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:58.052+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:58.345 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:58.052+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:54:58.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:58.342+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:58.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:58.342+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:58.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:58.342+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:58.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:58 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:58.342+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:54:59.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:58.528+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:54:59.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:58.528+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:58.528+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:58.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:58.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:58.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:58.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:54:59.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:58.752+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:54:59.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:58.790+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:54:59.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:58.790+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:54:59.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:58.790+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:54:59.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:58.790+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:54:59.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:58 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:58.790+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:59.088+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:59.088+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:59.088+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:54:59.088+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:59.326+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:54:59.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:59.326+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:54:59.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:59.326+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:54:59.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:54:59 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:54:59.326+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:59.512+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.127724+0000 front 2024-03-19T02:54:10.127757+0000 (oldest deadline 2024-03-19T02:54:32.328603+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:59.512+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.230996+0000 front 2024-03-19T02:54:19.231034+0000 (oldest deadline 2024-03-19T02:54:45.031767+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7[53894]: 2024-03-19T02:54:59.512+0000 7f6c6ee57700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.032505+0000 front 2024-03-19T02:54:25.032365+0000 (oldest deadline 2024-03-19T02:54:49.732315+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:59.731+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:59.731+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:59.731+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:00.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:59.731+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:00.008 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:54:59 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:54:59.731+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:00.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:59 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:59.773+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:00.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:59 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:59.773+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:00.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:59 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:59.773+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:00.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:59 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:59.773+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:00.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:54:59 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:54:59.773+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:00.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:00.123+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:00.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:00.123+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:00.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:00.123+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:00.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:00.123+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:00.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:00.317+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:00.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:00.317+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:00.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:00.317+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:00.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:00 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:00.317+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:01.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:00 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:00.723+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:01.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:00 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:00.723+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:01.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:00 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:00.723+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:01.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:00 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:00.723+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:01.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:00 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:00.723+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:01.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:00.817+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:01.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:00.817+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:01.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:00.817+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:01.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:00.817+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:01.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:00 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:00.817+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:01.301 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:01 smithi154 podman[77132]: 2024-03-19 02:55:01.03913262 +0000 UTC m=+5.724904450 container died ea1e4126f04d3d11aa7f7cf2ccc93ef613ad069efe5c5ff669a8c028af3f3d71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:01.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:01.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:01.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:01.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:01.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:01.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:01.418 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:01.157+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:01.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:01.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:01.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:01.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:01.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:01.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:01.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:01 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:01.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:01.667 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:01 smithi154 podman[77132]: 2024-03-19 02:55:01.390795025 +0000 UTC m=+6.076566853 container cleanup ea1e4126f04d3d11aa7f7cf2ccc93ef613ad069efe5c5ff669a8c028af3f3d71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T02:55:01.667 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:01 smithi154 bash[77132]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7 2024-03-19T02:55:02.006 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:01 smithi154 podman[77149]: 2024-03-19 02:55:01.666968174 +0000 UTC m=+0.622091774 container remove ea1e4126f04d3d11aa7f7cf2ccc93ef613ad069efe5c5ff669a8c028af3f3d71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:55:02.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:01 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:01.746+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:02.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:01 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:01.746+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:02.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:01 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:01.746+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:02.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:01 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:01.746+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:02.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:01 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:01.746+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:02.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:01.814+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:02.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:01.814+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:02.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:01.814+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:02.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:01.814+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:02.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:01 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:01.814+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:02.270 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:02.009413811 +0000 UTC m=+0.111158368 container create 6476b5cf3ec8aa79049fb75f5b8c80470a035836104d8a0f1d77791d5a25c64a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-19T02:55:02.270 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:01.917333677 +0000 UTC m=+0.019078182 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:55:02.567 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:02.270219555 +0000 UTC m=+0.371964063 container init 6476b5cf3ec8aa79049fb75f5b8c80470a035836104d8a0f1d77791d5a25c64a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T02:55:02.568 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:02.273654185 +0000 UTC m=+0.375398688 container start 6476b5cf3ec8aa79049fb75f5b8c80470a035836104d8a0f1d77791d5a25c64a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:55:02.568 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:02.345319237 +0000 UTC m=+0.447063742 container attach 6476b5cf3ec8aa79049fb75f5b8c80470a035836104d8a0f1d77791d5a25c64a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-19T02:55:02.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:02.203+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:02.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:02.203+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:02.203+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:02.203+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:02.309+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:02.309+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:02.309+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:02.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:02 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:02.309+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:02.848 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 19 02:55:02 smithi154 podman[77240]: 2024-03-19 02:55:02.567862933 +0000 UTC m=+0.669607436 container died 6476b5cf3ec8aa79049fb75f5b8c80470a035836104d8a0f1d77791d5a25c64a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-19T02:55:02.848 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:02.750+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:02.849 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:02.750+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:02.849 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:02.750+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:02.849 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:02.750+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:02.849 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:02.750+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:03.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.7.service' 2024-03-19T02:55:03.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:03.182 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-19T02:55:03.182 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-19T02:55:03.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.8 2024-03-19T02:55:03.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:02.775+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:03.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:02.775+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:03.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:02.775+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:03.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:02.775+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:03.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:02 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:02.775+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:03.294+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:03.294+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:03.294+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:03.294+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.218+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:03.486 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.218+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.218+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.218+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 systemd[1]: Stopping Ceph osd.8 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.483+0000 7f198d5e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.483+0000 7f198d5e7700 -1 osd.8 139 *** Got signal Terminated *** 2024-03-19T02:55:03.487 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:03 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:03.483+0000 7f198d5e7700 -1 osd.8 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:55:04.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:03.718+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:04.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:03.718+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:04.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:03.718+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:04.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:03.718+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:04.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:03 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:03.718+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:04.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:03.769+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:04.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:03.769+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:04.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:03.769+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:04.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:03.769+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:04.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:03 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:03.769+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:04.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:04.219+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:04.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:04.219+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:04.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:04.219+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:04.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:04.219+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:04.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:04.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:04.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:04.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:04.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:04.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:04.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:04 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:04.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:05.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:04.697+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:05.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:04.697+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:05.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:04.697+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:05.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:04.697+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:05.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:04 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:04.697+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:05.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:04.777+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:05.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:04.777+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:05.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:04.777+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:05.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:04.777+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:05.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:04 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:04.777+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:05.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:05.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:05.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:05.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:05.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:05.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:05.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:05.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:05.668 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:05.222+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:05.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:05.222+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:05.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:05.222+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:05.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:05 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:05.222+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:06.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:05.655+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:06.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:05.655+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:06.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:05.655+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:06.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:05.655+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:06.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:05 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:05.655+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:06.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:05.735+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:06.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:05.735+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:06.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:05.735+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:06.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:05.735+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:06.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:05 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:05.735+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:06.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:06.292+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:06.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:06.292+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:06.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:06.292+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:06.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:06.292+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:06.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:06.227+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:06.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:06.227+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:06.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:06.227+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:06.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:06.227+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:06.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:06.227+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.633362+0000 front 2024-03-19T02:54:41.633298+0000 (oldest deadline 2024-03-19T02:55:05.733448+0000) 2024-03-19T02:55:07.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:06 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:06.675+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:07.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:06 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:06.675+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:07.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:06 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:06.675+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:07.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:06 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:06.675+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:07.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:06 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:06.675+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:07.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:06.776+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:07.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:06.776+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:07.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:06.776+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:07.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:06.776+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:07.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:06 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:06.776+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:07.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:07.302+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:07.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:07.302+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:07.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:07.302+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:07.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:07.302+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:07.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:07.232+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:07.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:07.232+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:07.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:07.232+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:07.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:07.232+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:07.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:07 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:07.232+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.633362+0000 front 2024-03-19T02:54:41.633298+0000 (oldest deadline 2024-03-19T02:55:05.733448+0000) 2024-03-19T02:55:08.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:08.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:08.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:08.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:08.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:08.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:07 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:07.639+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:08.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:07.788+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:08.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:07.788+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:08.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:07.788+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:08.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:07.788+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:08.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:07 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:07.788+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:08.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:08.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:08.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:08.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:08.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:08.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:08.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:08.306+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:08.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:08.214+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:10.028751+0000 front 2024-03-19T02:54:10.028620+0000 (oldest deadline 2024-03-19T02:54:32.328781+0000) 2024-03-19T02:55:08.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:08.214+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.829883+0000 front 2024-03-19T02:54:15.829914+0000 (oldest deadline 2024-03-19T02:54:40.530111+0000) 2024-03-19T02:55:08.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:08.214+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:25.831342+0000 front 2024-03-19T02:54:25.831362+0000 (oldest deadline 2024-03-19T02:54:51.131389+0000) 2024-03-19T02:55:08.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:08.214+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.432424+0000 front 2024-03-19T02:54:33.432495+0000 (oldest deadline 2024-03-19T02:54:57.532458+0000) 2024-03-19T02:55:08.669 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:08 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8[54248]: 2024-03-19T02:55:08.214+0000 7f1989bdb700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.633362+0000 front 2024-03-19T02:54:41.633298+0000 (oldest deadline 2024-03-19T02:55:05.733448+0000) 2024-03-19T02:55:09.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:09.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:09.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:09.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:09.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:09.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:08 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:08.593+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:09.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:09.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:09.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:09.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:09.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:09.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:08 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:08.766+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:09.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 podman[75598]: 2024-03-19 02:55:09.00091147 +0000 UTC m=+5.620423511 container died c653484c3c0c4f75bdf3e6563e239548fc637becafcafb48d126844362168880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:55:09.638 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 podman[75598]: 2024-03-19 02:55:09.38437234 +0000 UTC m=+6.003884370 container cleanup c653484c3c0c4f75bdf3e6563e239548fc637becafcafb48d126844362168880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0) 2024-03-19T02:55:09.638 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 bash[75598]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8 2024-03-19T02:55:09.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:09 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:09.298+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:09.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:09 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:09.298+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:09.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:09 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:09.298+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:09.639 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:09 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:09.298+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:09.918 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 podman[75625]: 2024-03-19 02:55:09.635993307 +0000 UTC m=+0.634923083 container remove c653484c3c0c4f75bdf3e6563e239548fc637becafcafb48d126844362168880 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-19T02:55:10.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:10.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:10.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:10.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:10.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:10.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:09 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:09.609+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:10.250 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 podman[75722]: 2024-03-19 02:55:09.872645308 +0000 UTC m=+0.018355806 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:55:10.250 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:09 smithi162 podman[75722]: 2024-03-19 02:55:09.986826166 +0000 UTC m=+0.132536659 container create bcc56ba9844f6319ca83390805183aa6f6adfcc6de2a1926bd8214390659d60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:10.250 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:10 smithi162 podman[75722]: 2024-03-19 02:55:10.247608612 +0000 UTC m=+0.393319106 container init bcc56ba9844f6319ca83390805183aa6f6adfcc6de2a1926bd8214390659d60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:10.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:09 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:09.767+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:10 smithi162 podman[75722]: 2024-03-19 02:55:10.251022262 +0000 UTC m=+0.396732756 container start bcc56ba9844f6319ca83390805183aa6f6adfcc6de2a1926bd8214390659d60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:10 smithi162 podman[75722]: 2024-03-19 02:55:10.331276287 +0000 UTC m=+0.476986798 container attach bcc56ba9844f6319ca83390805183aa6f6adfcc6de2a1926bd8214390659d60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:10.257+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:10.257+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:10.257+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:10.543 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:10.257+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:10.826 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 02:55:10 smithi162 podman[75722]: 2024-03-19 02:55:10.54219167 +0000 UTC m=+0.687902167 container died bcc56ba9844f6319ca83390805183aa6f6adfcc6de2a1926bd8214390659d60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.623+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.623+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.623+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.623+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:11.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:10 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:10.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:11.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.8.service' 2024-03-19T02:55:11.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:11.211 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-19T02:55:11.211 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-19T02:55:11.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.9 2024-03-19T02:55:11.241 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:11.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:11.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:11.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:11.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:11.242 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:10 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:10.803+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:11.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:11 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:11.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:11.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:11 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:11.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:11.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:11 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:11.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:11.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:11 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:11.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:11.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:11 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:11.277+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:11.506 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 systemd[1]: Stopping Ceph osd.9 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:11.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.504+0000 7fcb0244d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:11.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.504+0000 7fcb0244d700 -1 osd.9 139 *** Got signal Terminated *** 2024-03-19T02:55:11.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.504+0000 7fcb0244d700 -1 osd.9 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:55:12.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:12.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:12.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:12.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:12.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:12.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:11 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:11.620+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:12.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:12.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:12.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:12.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:12.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:12.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:11 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:11.808+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:12.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:12 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:12.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:12.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:12 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:12.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:12.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:12 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:12.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:12.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:12 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:12.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:12.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:12 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:12.272+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:13.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:13.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:13.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:13.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:13.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:13.008 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:12 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:12.626+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:13.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:13.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:13.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:13.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:13.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:13.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:12 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:12.770+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:13.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:13 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:13.255+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:13.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:13 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:13.255+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:13.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:13 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:13.255+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:13.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:13 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:13.255+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:13.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:13 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:13.255+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:14.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:14.113 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:14.113 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:14.113 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:14.114 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:14.114 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:13 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:13.635+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:14.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:14.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:14.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:14.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:14.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:14.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:13 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:13.760+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:14.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:14.226+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:14.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:14.226+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:14.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:14.226+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:14.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:14.226+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:14.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:14.226+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:15.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:14 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:14.678+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:15.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:15.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:15.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:15.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:15.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:15.260 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:14 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:14.801+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:15.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:15 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:15.230+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:15.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:15 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:15.230+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:15.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:15 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:15.230+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:15.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:15 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:15.230+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:15.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:15 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:15.230+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:16.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:16.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:15 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:15.654+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:02.087160+0000 front 2024-03-19T02:54:02.087122+0000 (oldest deadline 2024-03-19T02:54:23.187259+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:07.789258+0000 front 2024-03-19T02:54:07.789229+0000 (oldest deadline 2024-03-19T02:54:31.889321+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:18.092668+0000 front 2024-03-19T02:54:18.092543+0000 (oldest deadline 2024-03-19T02:54:41.592619+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:27.493577+0000 front 2024-03-19T02:54:27.493629+0000 (oldest deadline 2024-03-19T02:54:51.593668+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:33.195267+0000 front 2024-03-19T02:54:33.195328+0000 (oldest deadline 2024-03-19T02:54:57.295299+0000) 2024-03-19T02:55:16.259 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:15 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9[65006]: 2024-03-19T02:55:15.826+0000 7fcafaa39700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.197444+0000 front 2024-03-19T02:54:44.197404+0000 (oldest deadline 2024-03-19T02:55:08.297293+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:16.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:16 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:16.196+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:17.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:17.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:16 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:16.624+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:17.231 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:16 smithi116 podman[123986]: 2024-03-19 02:55:16.953712234 +0000 UTC m=+5.548923528 container died a8f7a4612add56d6625f5f8ea8a41c58fd72aa372db9adfa3e0415bca49825a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:17.509 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:17 smithi116 podman[123986]: 2024-03-19 02:55:17.29619537 +0000 UTC m=+5.891406666 container cleanup a8f7a4612add56d6625f5f8ea8a41c58fd72aa372db9adfa3e0415bca49825a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T02:55:17.509 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:17 smithi116 bash[123986]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9 2024-03-19T02:55:17.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:17.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:17.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:17.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:17.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:17.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:17 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:17.176+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:17.970 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:17 smithi116 podman[124003]: 2024-03-19 02:55:17.522625358 +0000 UTC m=+0.565590795 container remove a8f7a4612add56d6625f5f8ea8a41c58fd72aa372db9adfa3e0415bca49825a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:18.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:18.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:17 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:17.653+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:18.243 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:17 smithi116 podman[124092]: 2024-03-19 02:55:17.872060247 +0000 UTC m=+0.017642332 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:55:18.244 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:17 smithi116 podman[124092]: 2024-03-19 02:55:17.987396797 +0000 UTC m=+0.132978913 container create f6bb67328e04217f765294a65514f5c0b054c953229bb7b974e20dbaaffbc398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:55:18.244 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:18 smithi116 podman[124092]: 2024-03-19 02:55:18.243308132 +0000 UTC m=+0.388890214 container init f6bb67328e04217f765294a65514f5c0b054c953229bb7b974e20dbaaffbc398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T02:55:18.509 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:18 smithi116 podman[124092]: 2024-03-19 02:55:18.246843248 +0000 UTC m=+0.392425325 container start f6bb67328e04217f765294a65514f5c0b054c953229bb7b974e20dbaaffbc398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:55:18.509 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:18 smithi116 podman[124092]: 2024-03-19 02:55:18.33065007 +0000 UTC m=+0.476232150 container attach f6bb67328e04217f765294a65514f5c0b054c953229bb7b974e20dbaaffbc398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T02:55:18.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:18.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:18.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:18.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:18.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:18.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:18.206+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:18.829 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 02:55:18 smithi116 podman[124092]: 2024-03-19 02:55:18.538204508 +0000 UTC m=+0.683786579 container died f6bb67328e04217f765294a65514f5c0b054c953229bb7b974e20dbaaffbc398 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T02:55:19.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:19.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:18 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:18.637+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:19.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.9.service' 2024-03-19T02:55:19.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:19.183 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-19T02:55:19.184 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-19T02:55:19.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.10 2024-03-19T02:55:19.507 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 systemd[1]: Stopping Ceph osd.10 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:19.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:19 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:19.169+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:20.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.533+0000 7fb8858de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.533+0000 7fb8858de700 -1 osd.10 139 *** Got signal Terminated *** 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.533+0000 7fb8858de700 -1 osd.10 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:20.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:19 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:19.617+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:20.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:20 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:20.142+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:21.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:21.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:20 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:20.665+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:21.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:21.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:21.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:21.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:21.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:21.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:21 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:21.156+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:22.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:22.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:21 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:21.631+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:22.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:22.669 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:22 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:22.170+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:23.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:23.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:22 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:22.656+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:23.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:23.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:23 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:23.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:24.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:53:59.108319+0000 front 2024-03-19T02:53:59.108360+0000 (oldest deadline 2024-03-19T02:54:23.808467+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:09.010342+0000 front 2024-03-19T02:54:09.010572+0000 (oldest deadline 2024-03-19T02:54:32.510397+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:19.512333+0000 front 2024-03-19T02:54:19.512141+0000 (oldest deadline 2024-03-19T02:54:40.612022+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T02:54:26.413556+0000 front 2024-03-19T02:54:26.413591+0000 (oldest deadline 2024-03-19T02:54:52.313620+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:32.314135+0000 front 2024-03-19T02:54:32.314114+0000 (oldest deadline 2024-03-19T02:54:58.214267+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:44.015904+0000 front 2024-03-19T02:54:44.016007+0000 (oldest deadline 2024-03-19T02:55:07.516099+0000) 2024-03-19T02:55:24.007 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:23 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10[60503]: 2024-03-19T02:55:23.645+0000 7fb881ed2700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:52.117753+0000 front 2024-03-19T02:54:52.117731+0000 (oldest deadline 2024-03-19T02:55:14.417587+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:24.668 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:24 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:24.186+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:25.394 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:25 smithi154 podman[77357]: 2024-03-19 02:55:25.109850877 +0000 UTC m=+5.728714544 container died fee7f806bcef480ed184382c2d18d3041ec9ec2b600f90fcbe45f0c4b90f99a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:25.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:25 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:25.161+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:25.752 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:25 smithi154 podman[77357]: 2024-03-19 02:55:25.492447611 +0000 UTC m=+6.111311283 container cleanup fee7f806bcef480ed184382c2d18d3041ec9ec2b600f90fcbe45f0c4b90f99a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T02:55:25.752 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:25 smithi154 bash[77357]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10 2024-03-19T02:55:26.006 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:25 smithi154 podman[77374]: 2024-03-19 02:55:25.751790669 +0000 UTC m=+0.640381871 container remove fee7f806bcef480ed184382c2d18d3041ec9ec2b600f90fcbe45f0c4b90f99a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:55:26.307 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:25.949484219 +0000 UTC m=+0.017801081 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:55:26.307 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:26.060837961 +0000 UTC m=+0.129154804 container create 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-19T02:55:26.307 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:26.306959812 +0000 UTC m=+0.375276673 container init 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:26.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:26 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:26.154+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:26.605 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:26.310051315 +0000 UTC m=+0.378368168 container start 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:55:26.605 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:26.388110512 +0000 UTC m=+0.456427366 container attach 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:26.874 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:26 smithi154 podman[77460]: 2024-03-19 02:55:26.605530822 +0000 UTC m=+0.673847670 container died 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:27.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.10.service' 2024-03-19T02:55:27.149 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:27 smithi154 podman[77486]: 2024-03-19 02:55:27.107924385 +0000 UTC m=+0.492460069 container remove 1793c8c194f11e25b74e4585c67c76049730760a3a913771ba68b94f2b4e0302 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:55:27.150 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:27 smithi154 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-19T02:55:27.150 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:27 smithi154 systemd[1]: Stopped Ceph osd.10 for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:55:27.150 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 19 02:55:27 smithi154 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.10.service: Consumed 2min 25.562s CPU time. 2024-03-19T02:55:27.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:27.164 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-19T02:55:27.164 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-19T02:55:27.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.11 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:27.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:27.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.150+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:27.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 systemd[1]: Stopping Ceph osd.11 for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:27.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.478+0000 7f40de4c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:27.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.478+0000 7f40de4c1700 -1 osd.11 139 *** Got signal Terminated *** 2024-03-19T02:55:27.918 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:27 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:27.478+0000 7f40de4c1700 -1 osd.11 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T02:55:28.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:28.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:28.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:28.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:28.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:28.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:28.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:28 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:28.103+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:29.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:29.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:29 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:29.085+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:30.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:30.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:30.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:30.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:30.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:30.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:30.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:30 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:30.070+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:31.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:31.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:31.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:31 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:31.111+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-03-19T02:55:07.084080+0000 front 2024-03-19T02:55:07.084106+0000 (oldest deadline 2024-03-19T02:55:30.584102+0000) 2024-03-19T02:55:32.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T02:54:01.975804+0000 front 2024-03-19T02:54:01.975820+0000 (oldest deadline 2024-03-19T02:54:26.075786+0000) 2024-03-19T02:55:32.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-19T02:54:06.076309+0000 front 2024-03-19T02:54:06.076387+0000 (oldest deadline 2024-03-19T02:54:31.376250+0000) 2024-03-19T02:55:32.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T02:54:15.477180+0000 front 2024-03-19T02:54:15.477268+0000 (oldest deadline 2024-03-19T02:54:40.777260+0000) 2024-03-19T02:55:32.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-19T02:54:30.778665+0000 front 2024-03-19T02:54:30.778862+0000 (oldest deadline 2024-03-19T02:54:56.678719+0000) 2024-03-19T02:55:32.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T02:54:41.280303+0000 front 2024-03-19T02:54:41.280388+0000 (oldest deadline 2024-03-19T02:55:10.681191+0000) 2024-03-19T02:55:32.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T02:54:50.681796+0000 front 2024-03-19T02:54:50.681812+0000 (oldest deadline 2024-03-19T02:55:15.381836+0000) 2024-03-19T02:55:32.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-19T02:55:00.083237+0000 front 2024-03-19T02:55:00.083157+0000 (oldest deadline 2024-03-19T02:55:21.183063+0000) 2024-03-19T02:55:32.419 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11[61036]: 2024-03-19T02:55:32.069+0000 7f40daab5700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-03-19T02:55:07.084080+0000 front 2024-03-19T02:55:07.084106+0000 (oldest deadline 2024-03-19T02:55:30.584102+0000) 2024-03-19T02:55:33.265 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:32 smithi162 podman[75828]: 2024-03-19 02:55:32.995079111 +0000 UTC m=+5.640373235 container died a7c180972453af153e9dd8ddb658650d065a9ac2450f580586784e360eb48007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:33.589 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:33 smithi162 podman[75828]: 2024-03-19 02:55:33.346052454 +0000 UTC m=+5.991346570 container cleanup a7c180972453af153e9dd8ddb658650d065a9ac2450f580586784e360eb48007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:33.589 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:33 smithi162 bash[75828]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11 2024-03-19T02:55:33.902 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:33 smithi162 podman[75845]: 2024-03-19 02:55:33.58890342 +0000 UTC m=+0.597352904 container remove a7c180972453af153e9dd8ddb658650d065a9ac2450f580586784e360eb48007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-19T02:55:34.154 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:33 smithi162 podman[75935]: 2024-03-19 02:55:33.804487471 +0000 UTC m=+0.017447236 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:55:34.154 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:33 smithi162 podman[75935]: 2024-03-19 02:55:33.956466295 +0000 UTC m=+0.169426052 container create a1749a70e5e8671c5e0e00529c57759128f3cd1cf730fe723d8106888fd93a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T02:55:34.154 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:34 smithi162 podman[75935]: 2024-03-19 02:55:34.15052049 +0000 UTC m=+0.363480257 container init a1749a70e5e8671c5e0e00529c57759128f3cd1cf730fe723d8106888fd93a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T02:55:34.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:34 smithi162 podman[75935]: 2024-03-19 02:55:34.153637997 +0000 UTC m=+0.366597755 container start a1749a70e5e8671c5e0e00529c57759128f3cd1cf730fe723d8106888fd93a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:55:34.418 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:34 smithi162 podman[75935]: 2024-03-19 02:55:34.250548545 +0000 UTC m=+0.463508320 container attach a1749a70e5e8671c5e0e00529c57759128f3cd1cf730fe723d8106888fd93a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-19T02:55:34.745 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 02:55:34 smithi162 podman[75935]: 2024-03-19 02:55:34.44486808 +0000 UTC m=+0.657827846 container died a1749a70e5e8671c5e0e00529c57759128f3cd1cf730fe723d8106888fd93a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:35.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@osd.11.service' 2024-03-19T02:55:35.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:35.060 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-19T02:55:35.060 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-19T02:55:35.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.a 2024-03-19T02:55:35.366 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:35 smithi116 systemd[1]: Stopping Ceph mds.a for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:35.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: 2024-03-19T02:55:35.364+0000 7f57f35b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:35.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:35 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a[71820]: 2024-03-19T02:55:35.364+0000 7f57f35b5700 -1 mds.a *** got signal Terminated *** 2024-03-19T02:55:37.463 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:37 smithi116 podman[124221]: 2024-03-19 02:55:37.183783063 +0000 UTC m=+1.924065512 container died 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T02:55:37.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:37 smithi116 podman[124221]: 2024-03-19 02:55:37.565049625 +0000 UTC m=+2.305332078 container cleanup 1e618512a368d9e3b3c69bb7d54032b924e57238f8948c7693799ca9ea362676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T02:55:37.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 02:55:37 smithi116 bash[124221]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-a 2024-03-19T02:55:37.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.a.service' 2024-03-19T02:55:37.958 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:37.959 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-19T02:55:37.959 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-19T02:55:37.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.d 2024-03-19T02:55:38.259 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:38 smithi116 systemd[1]: Stopping Ceph mds.d for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:38.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d[70859]: 2024-03-19T02:55:38.292+0000 7f26a7d31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:38.759 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:38 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d[70859]: 2024-03-19T02:55:38.292+0000 7f26a7d31700 -1 mds.d *** got signal Terminated *** 2024-03-19T02:55:41.910 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:41 smithi116 podman[124340]: 2024-03-19 02:55:41.570538593 +0000 UTC m=+3.391490705 container died f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-19T02:55:41.910 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:41 smithi116 podman[124340]: 2024-03-19 02:55:41.91012942 +0000 UTC m=+3.731081531 container cleanup f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T02:55:42.170 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:41 smithi116 bash[124340]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d 2024-03-19T02:55:42.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.d.service' 2024-03-19T02:55:42.432 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:42 smithi116 podman[124354]: 2024-03-19 02:55:42.169889935 +0000 UTC m=+0.593693274 container remove f620a04f2f707877fbe4e2bb96cb6e6d22197335c8bf8f15b745d8a72c986f01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-19T02:55:42.432 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:42 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.d.service: Deactivated successfully. 2024-03-19T02:55:42.432 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:42 smithi116 systemd[1]: Stopped Ceph mds.d for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:55:42.432 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 02:55:42 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.d.service: Consumed 1.140s CPU time. 2024-03-19T02:55:42.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:42.444 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-19T02:55:42.444 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-19T02:55:42.444 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.g 2024-03-19T02:55:42.759 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:42 smithi116 systemd[1]: Stopping Ceph mds.g for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:43.259 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: 2024-03-19T02:55:42.797+0000 7f0bc9331700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:43.259 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:42 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g[71461]: 2024-03-19T02:55:42.797+0000 7f0bc9331700 -1 mds.g *** got signal Terminated *** 2024-03-19T02:55:45.423 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 podman[124459]: 2024-03-19 02:55:45.163588679 +0000 UTC m=+2.491680404 container died 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-19T02:55:45.732 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 podman[124459]: 2024-03-19 02:55:45.497078404 +0000 UTC m=+2.825170127 container cleanup 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-19T02:55:45.732 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 bash[124459]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g 2024-03-19T02:55:45.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.g.service' 2024-03-19T02:55:45.991 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 podman[124473]: 2024-03-19 02:55:45.731733947 +0000 UTC m=+0.562771013 container remove 688e51c6203081d8260507ad1461c36796e3d9149aebc1d935b107440e39a3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T02:55:45.991 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.g.service: Deactivated successfully. 2024-03-19T02:55:45.991 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 systemd[1]: Stopped Ceph mds.g for a1da9bbc-e596-11ee-95c9-87774f69a715. 2024-03-19T02:55:45.991 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 02:55:45 smithi116 systemd[1]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.g.service: Consumed 3.616s CPU time. 2024-03-19T02:55:46.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:46.002 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-19T02:55:46.003 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-19T02:55:46.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.j 2024-03-19T02:55:46.259 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:46 smithi116 systemd[1]: Stopping Ceph mds.j for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:46.759 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j[70473]: 2024-03-19T02:55:46.317+0000 7f41d8d54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:46.759 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:46 smithi116 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j[70473]: 2024-03-19T02:55:46.317+0000 7f41d8d54700 -1 mds.j *** got signal Terminated *** 2024-03-19T02:55:50.744 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:50 smithi116 podman[124580]: 2024-03-19 02:55:50.342792839 +0000 UTC m=+4.127175822 container died 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-19T02:55:51.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:50 smithi116 podman[124580]: 2024-03-19 02:55:50.826126632 +0000 UTC m=+4.610509621 container cleanup 9b06d97a41fe8ac767a959adda27740e94f1feecbb6cf4304205feb09a5a25d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T02:55:51.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 02:55:50 smithi116 bash[124580]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-j 2024-03-19T02:55:51.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.j.service' 2024-03-19T02:55:51.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:51.282 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-19T02:55:51.282 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-19T02:55:51.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.b 2024-03-19T02:55:51.592 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:51 smithi154 systemd[1]: Stopping Ceph mds.b for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:51.850 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:55:51.591+0000 7f0999eec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:51.850 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:51 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b[66135]: 2024-03-19T02:55:51.591+0000 7f0999eec700 -1 mds.b *** got signal Terminated *** 2024-03-19T02:55:53.459 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:53 smithi154 podman[77567]: 2024-03-19 02:55:53.188537633 +0000 UTC m=+1.707689002 container died 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:55:53.756 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:53 smithi154 podman[77567]: 2024-03-19 02:55:53.549506357 +0000 UTC m=+2.068657741 container cleanup 3f46e3f611951cdea2f924209d75a4ef3d201df4d5855aa6d282492420a2fd7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:55:53.757 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 19 02:55:53 smithi154 bash[77567]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-b 2024-03-19T02:55:53.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.b.service' 2024-03-19T02:55:53.993 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:53.994 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-19T02:55:53.994 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-19T02:55:53.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.e 2024-03-19T02:55:54.256 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:54 smithi154 systemd[1]: Stopping Ceph mds.e for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:54.756 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: 2024-03-19T02:55:54.361+0000 7f35d47d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:54.756 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:54 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e[65520]: 2024-03-19T02:55:54.361+0000 7f35d47d1700 -1 mds.e *** got signal Terminated *** 2024-03-19T02:55:57.454 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:57 smithi154 podman[77700]: 2024-03-19 02:55:57.178868602 +0000 UTC m=+2.957257338 container died 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T02:55:57.756 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:57 smithi154 podman[77700]: 2024-03-19 02:55:57.570237482 +0000 UTC m=+3.348626250 container cleanup 5b9c314a90993ecacc4601bc031fd55bee95eea41538539cd3be471e47a5bc95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T02:55:57.757 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 19 02:55:57 smithi154 bash[77700]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-e 2024-03-19T02:55:57.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.e.service' 2024-03-19T02:55:57.941 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:55:57.941 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-19T02:55:57.941 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-19T02:55:57.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.h 2024-03-19T02:55:58.256 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:55:58 smithi154 systemd[1]: Stopping Ceph mds.h for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:55:58.756 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:55:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: 2024-03-19T02:55:58.264+0000 7fe7ed6a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:55:58.756 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:55:58 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h[65828]: 2024-03-19T02:55:58.264+0000 7fe7ed6a1700 -1 mds.h *** got signal Terminated *** 2024-03-19T02:56:01.460 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:56:01 smithi154 podman[77820]: 2024-03-19 02:56:01.174458222 +0000 UTC m=+3.018855006 container died fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T02:56:01.756 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:56:01 smithi154 podman[77820]: 2024-03-19 02:56:01.531699875 +0000 UTC m=+3.376096673 container cleanup fbebced225e9cd538a973839c8f6e947e228fcc5d59255741284e9c720793c63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-19T02:56:01.756 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 19 02:56:01 smithi154 bash[77820]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-h 2024-03-19T02:56:01.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.h.service' 2024-03-19T02:56:01.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:01.937 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-19T02:56:01.938 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-19T02:56:01.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.k 2024-03-19T02:56:02.256 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:02 smithi154 systemd[1]: Stopping Ceph mds.k for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:56:02.756 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: 2024-03-19T02:56:02.259+0000 7fa261123700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:56:02.757 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:02 smithi154 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k[65203]: 2024-03-19T02:56:02.259+0000 7fa261123700 -1 mds.k *** got signal Terminated *** 2024-03-19T02:56:05.444 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:05 smithi154 podman[77941]: 2024-03-19 02:56:05.192691152 +0000 UTC m=+3.029569853 container died 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:56:05.711 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:05 smithi154 podman[77941]: 2024-03-19 02:56:05.535516161 +0000 UTC m=+3.372394856 container cleanup 3aef6bc0f01cbd28fb566e3b3bb852a60b6482605c33360139539686f81e76df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T02:56:05.711 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 19 02:56:05 smithi154 bash[77941]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-k 2024-03-19T02:56:05.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.k.service' 2024-03-19T02:56:05.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:05.964 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-19T02:56:05.964 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-19T02:56:05.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.c 2024-03-19T02:56:06.262 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:06 smithi162 systemd[1]: Stopping Ceph mds.c for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:56:06.668 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: 2024-03-19T02:56:06.260+0000 7f2f44d90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:56:06.668 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:06 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c[66372]: 2024-03-19T02:56:06.260+0000 7f2f44d90700 -1 mds.c *** got signal Terminated *** 2024-03-19T02:56:09.418 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:09 smithi162 podman[76053]: 2024-03-19 02:56:09.139449532 +0000 UTC m=+2.977192866 container died 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T02:56:09.771 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:09 smithi162 podman[76053]: 2024-03-19 02:56:09.494221459 +0000 UTC m=+3.331964792 container cleanup 004f362370e8a947c6a18b5a6f1e90ef9d1e76c892f081552b51cc8f7e939020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:56:09.771 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 02:56:09 smithi162 bash[76053]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-c 2024-03-19T02:56:09.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.c.service' 2024-03-19T02:56:09.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:09.948 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-19T02:56:09.949 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-19T02:56:09.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.f 2024-03-19T02:56:10.418 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:10 smithi162 systemd[1]: Stopping Ceph mds.f for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:56:10.418 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: 2024-03-19T02:56:10.273+0000 7f9fd5846700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:56:10.418 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:10 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f[66064]: 2024-03-19T02:56:10.273+0000 7f9fd5846700 -1 mds.f *** got signal Terminated *** 2024-03-19T02:56:13.407 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:13 smithi162 podman[76185]: 2024-03-19 02:56:13.145050848 +0000 UTC m=+2.974770749 container died 3c3ca9b06120bddef210931bc6539ed9673c40bccb5be14ed803d768cae8a851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-19T02:56:13.668 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:13 smithi162 podman[76185]: 2024-03-19 02:56:13.514515965 +0000 UTC m=+3.344235869 container cleanup 3c3ca9b06120bddef210931bc6539ed9673c40bccb5be14ed803d768cae8a851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T02:56:13.668 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 02:56:13 smithi162 bash[76185]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-f 2024-03-19T02:56:13.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.f.service' 2024-03-19T02:56:13.978 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:13.979 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-19T02:56:13.979 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-19T02:56:13.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.i 2024-03-19T02:56:14.302 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:14 smithi162 systemd[1]: Stopping Ceph mds.i for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:56:14.668 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: 2024-03-19T02:56:14.301+0000 7f6e6c159700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:56:14.668 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:14 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i[66691]: 2024-03-19T02:56:14.301+0000 7f6e6c159700 -1 mds.i *** got signal Terminated *** 2024-03-19T02:56:17.418 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:17 smithi162 podman[76305]: 2024-03-19 02:56:17.152776596 +0000 UTC m=+2.960645772 container died e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-19T02:56:17.802 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:17 smithi162 podman[76305]: 2024-03-19 02:56:17.518050133 +0000 UTC m=+3.325919315 container cleanup e4de0e85a9bc68f94e940ecd42f1da6d672f2baaab9d8577e0f126026e546c3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-19T02:56:17.803 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 02:56:17 smithi162 bash[76305]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-i 2024-03-19T02:56:17.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.i.service' 2024-03-19T02:56:17.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:17.974 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-19T02:56:17.974 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-19T02:56:17.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.l 2024-03-19T02:56:18.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:18 smithi162 systemd[1]: Stopping Ceph mds.l for a1da9bbc-e596-11ee-95c9-87774f69a715... 2024-03-19T02:56:18.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: 2024-03-19T02:56:18.280+0000 7fc2f923e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:56:18.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:18 smithi162 ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l[65753]: 2024-03-19T02:56:18.280+0000 7fc2f923e700 -1 mds.l *** got signal Terminated *** 2024-03-19T02:56:21.418 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:21 smithi162 podman[76427]: 2024-03-19 02:56:21.137162446 +0000 UTC m=+2.938289111 container died 11fb4d09207d92d965a77a490ae3d8207e0b2154c5f654fa4ee5f11cdd9b8884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T02:56:21.889 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:21 smithi162 podman[76427]: 2024-03-19 02:56:21.488662379 +0000 UTC m=+3.289789049 container cleanup 11fb4d09207d92d965a77a490ae3d8207e0b2154c5f654fa4ee5f11cdd9b8884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-19T02:56:21.890 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:21 smithi162 bash[76427]: ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l 2024-03-19T02:56:21.890 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 02:56:21 smithi162 podman[76441]: 2024-03-19 02:56:21.656314222 +0000 UTC m=+0.512520056 container remove 11fb4d09207d92d965a77a490ae3d8207e0b2154c5f654fa4ee5f11cdd9b8884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a1da9bbc-e596-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T02:56:21.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1da9bbc-e596-11ee-95c9-87774f69a715@mds.l.service' 2024-03-19T02:56:21.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T02:56:21.943 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-19T02:56:21.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T02:56:22.134 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:56:25.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T02:56:25.358 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:56:28.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T02:56:28.894 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T02:56:31.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:56:31.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:56:31.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:56:31.324 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-19T02:56:31.326 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi116/crash 2024-03-19T02:56:31.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash -- . 2024-03-19T02:56:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T02:56:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-19T02:56:31.377 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi154/crash 2024-03-19T02:56:31.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash -- . 2024-03-19T02:56:31.437 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T02:56:31.437 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-03-19T02:56:31.439 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi162/crash 2024-03-19T02:56:31.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash -- . 2024-03-19T02:56:31.475 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T02:56:31.475 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-03-19T02:56:31.476 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-19T02:56:31.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T02:56:31.565 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-19T02:19:06.291993+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:56:31.565 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-19T02:56:31.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[SEC\]' /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T02:56:31.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]' /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T02:56:31.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[WRN\]' /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T02:56:31.749 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-19T02:19:06.291993+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:56:31.750 INFO:tasks.cephadm:Compressing logs... 2024-03-19T02:56:31.750 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T02:56:31.793 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T02:56:31.795 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T02:56:31.822 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T02:56:31.823 INFO:teuthology.orchestra.run.smithi162.stderr:: No such file or directory 2024-03-19T02:56:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-19T02:56:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T02:56:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:gzip/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.l.log: -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-19T02:56:31.824 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.11.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.5% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log 2024-03-19T02:56:31.828 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T02:56:31.829 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-19T02:56:31.835 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log: 87.8% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T02:56:31.835 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-19T02:56:31.836 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.2.log: 92.6% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-19T02:56:31.839 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T02:56:31.839 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2024-03-19T02:56:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-19T02:56:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-19T02:56:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-19T02:56:31.851 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-19T02:56:31.853 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T02:56:31.853 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-19T02:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T02:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-19T02:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-19T02:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T02:56:31.857 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-19T02:56:31.860 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-19T02:56:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/ceph-client.admin.log: 87.8% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T02:56:31.887 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T02:56:31.890 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T02:56:31.959 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T02:56:31.968 INFO:teuthology.orchestra.run.smithi116.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-19T02:56:32.012 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.b.log: 84.4% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T02:56:32.024 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-19T02:56:32.154 INFO:teuthology.orchestra.run.smithi154.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T02:56:32.168 INFO:teuthology.orchestra.run.smithi116.stderr: 94.1% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-19T02:56:32.194 INFO:teuthology.orchestra.run.smithi116.stderr: 94.7% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-19T02:56:32.224 INFO:teuthology.orchestra.run.smithi116.stderr: 84.5% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T02:56:32.233 INFO:teuthology.orchestra.run.smithi116.stderr: 90.1% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-19T02:56:32.300 INFO:teuthology.orchestra.run.smithi162.stderr: 94.5% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-19T02:56:32.354 INFO:teuthology.orchestra.run.smithi154.stderr: 90.5% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T02:56:32.534 INFO:teuthology.orchestra.run.smithi154.stderr: 92.6% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-19T02:56:32.582 INFO:teuthology.orchestra.run.smithi154.stderr: 90.9% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T02:56:32.827 INFO:teuthology.orchestra.run.smithi162.stderr: 92.1% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-19T02:56:33.684 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T02:56:37.596 INFO:teuthology.orchestra.run.smithi162.stderr: 94.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-19T02:56:37.975 INFO:teuthology.orchestra.run.smithi162.stderr: 94.2% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-19T02:56:38.614 INFO:teuthology.orchestra.run.smithi162.stderr: 94.2% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-19T02:56:46.888 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-19T02:56:53.069 INFO:teuthology.orchestra.run.smithi154.stderr: 92.0% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-19T02:56:54.232 INFO:teuthology.orchestra.run.smithi162.stderr: 91.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-19T02:57:01.910 INFO:teuthology.orchestra.run.smithi162.stderr: 91.1% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-19T02:57:04.113 INFO:teuthology.orchestra.run.smithi162.stderr: 91.2% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-19T02:57:10.472 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-19T02:57:10.474 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-19T02:57:10.474 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m38.664s 2024-03-19T02:57:10.474 INFO:teuthology.orchestra.run.smithi162.stderr:user 2m18.415s 2024-03-19T02:57:10.474 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m4.340s 2024-03-19T02:57:18.126 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-19T02:59:00.171 INFO:teuthology.orchestra.run.smithi154.stderr: 95.0% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-19T03:01:15.827 INFO:teuthology.orchestra.run.smithi116.stderr: 91.5% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-19T03:01:17.631 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-19T03:01:25.886 INFO:teuthology.orchestra.run.smithi154.stderr: 94.8% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-19T03:01:41.145 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-19T03:02:16.692 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-19T03:02:23.062 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-19T03:02:23.070 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-19T03:02:43.591 INFO:teuthology.orchestra.run.smithi154.stderr: 91.4% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-19T03:02:47.139 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-19T03:07:16.294 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-19T03:07:16.313 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-19T03:07:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:real 10m44.507s 2024-03-19T03:07:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:user 6m38.777s 2024-03-19T03:07:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m17.939s 2024-03-19T03:13:17.233 INFO:teuthology.orchestra.run.smithi154.stderr: 95.0% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-19T03:20:55.508 INFO:teuthology.orchestra.run.smithi154.stderr: 94.9% -- replaced with /var/log/ceph/a1da9bbc-e596-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-19T03:20:55.526 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-03-19T03:20:55.527 INFO:teuthology.orchestra.run.smithi154.stderr:real 24m23.716s 2024-03-19T03:20:55.527 INFO:teuthology.orchestra.run.smithi154.stderr:user 11m24.715s 2024-03-19T03:20:55.527 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m32.310s 2024-03-19T03:20:55.527 INFO:tasks.cephadm:Archiving logs... 2024-03-19T03:20:55.528 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi116/log 2024-03-19T03:20:55.529 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:22:01.102 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi154/log 2024-03-19T03:22:01.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:23:38.149 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi162/log 2024-03-19T03:23:38.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:24:00.424 INFO:tasks.cephadm:Removing cluster... 2024-03-19T03:24:00.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force 2024-03-19T03:24:01.130 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T03:24:01.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force 2024-03-19T03:24:02.446 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T03:24:03.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1da9bbc-e596-11ee-95c9-87774f69a715 --force 2024-03-19T03:24:03.409 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: a1da9bbc-e596-11ee-95c9-87774f69a715 2024-03-19T03:24:03.862 INFO:tasks.cephadm:Removing cephadm ... 2024-03-19T03:24:03.862 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:24:03.905 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:24:03.947 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:24:03.963 INFO:tasks.cephadm:Teardown complete 2024-03-19T03:24:03.964 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T03:24:03.976 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T03:24:03.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:24:03.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:24:03.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:24:04.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T03:24:04.114 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T03:24:04.115 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:24:04.117 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:24:04.120 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:24:04.133 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:24:04.133 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +3742us[+3742us] +/- 123ms 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 7 377 121 -1257us[-1286us] +/- 63ms 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -367us[ -367us] +/- 72ms 2024-03-19T03:24:04.138 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:24:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:24:04.171 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:24:04.172 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-19T03:24:04.172 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +3643us[+3640us] +/- 123ms 2024-03-19T03:24:04.172 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -1422us[-1424us] +/- 64ms 2024-03-19T03:24:04.172 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -479us[ -482us] +/- 72ms 2024-03-19T03:24:04.172 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:24:04.204 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:24:04.205 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-19T03:24:04.205 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +3632us[+3632us] +/- 123ms 2024-03-19T03:24:04.205 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -1433us[-1429us] +/- 64ms 2024-03-19T03:24:04.205 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -478us[ -478us] +/- 72ms 2024-03-19T03:24:04.205 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:24:04.206 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T03:24:04.219 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T03:24:04.219 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T03:24:04.230 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:24:04.418 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:24:04.573 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:24:04.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:24:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:04.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:24:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:04.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:24:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:04.743 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T03:24:04.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T03:24:04.768 INFO:teuthology.task.internal:Duration was 5498.749700 seconds 2024-03-19T03:24:04.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T03:24:04.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T03:24:04.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:24:04.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:24:04.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:24:04.829 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:24:04.848 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:24:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:24:05.088 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T03:24:05.089 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T03:24:05.089 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:24:05.114 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-03-19T03:24:05.114 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:24:05.142 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-03-19T03:24:05.142 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:24:05.169 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T03:24:05.170 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:24:05.172 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:24:05.184 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:24:05.238 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T03:24:05.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:24:05.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:24:05.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:24:05.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T03:24:05.677 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T03:24:05.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:24:05.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:24:05.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:24:05.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T03:24:05.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.809 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-19T03:24:05.810 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-03-19T03:24:05.826 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-03-19T03:24:05.852 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:05.892 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:05.906 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:24:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:24:05.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T03:24:05.931 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T03:24:05.933 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi116 2024-03-19T03:24:05.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:24:05.981 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi154 2024-03-19T03:24:05.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:24:06.021 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359/remote/smithi162 2024-03-19T03:24:06.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:24:06.061 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T03:24:06.061 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:24:06.064 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:24:06.066 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:24:06.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T03:24:06.124 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T03:24:06.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T03:24:06.135 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T03:24:06.136 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:24:06.138 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:24:06.140 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:24:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:24 /home/ubuntu/cephtest 2024-03-19T03:24:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:24 /home/ubuntu/cephtest 2024-03-19T03:24:06.164 INFO:teuthology.orchestra.run.smithi162.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:24 /home/ubuntu/cephtest 2024-03-19T03:24:06.165 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T03:24:06.177 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T03:24:06.206 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T03:24:06.231 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-19T03:24:06.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:24:06.257 DEBUG:teuthology.nuke:shortname: smithi154 2024-03-19T03:24:06.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:24:06.262 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.551268', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExK3MeTqY5yegk6BPu1UkymIULSfF7y5Y9GoblRO2xb0mDBqNcTgXalfPHOOnIecfH+PaBRfUf0C6WhuSd2nlU='} 2024-03-19T03:24:06.278 DEBUG:teuthology.nuke:shortname: smithi162 2024-03-19T03:24:06.278 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:24:06.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.549751', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP76y+R/X891su6OJ6zDW3SuJGUFRxKzshM3Fq0BBiNykMchTCBKJlLTkePomavQiunKAn85o8KIA+9LoynM3Hk='} 2024-03-19T03:24:06.295 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 01:34:46.552466', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/1YuAsd4LGvbQGnzXfYEHIBR/xRsWmSfb2mOPTy3yIwEoyaGqIdEsSSMZubRw2zeS0sWMB6E8Cb0OJ/JxIGUU='} 2024-03-19T03:24:06.313 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-19T03:24:06.313 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:24:06.332 INFO:teuthology.orchestra.console.smithi154:Power off 2024-03-19T03:24:06.332 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:24:06.342 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-19T03:24:06.343 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:06.358 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-03-19T03:24:06.358 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:06.373 INFO:teuthology.orchestra.console.smithi162:Power off 2024-03-19T03:24:06.373 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:24:06.381 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-19T03:24:06.382 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-19T03:24:06.396 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-03-19T03:24:06.396 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:06.419 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-19T03:24:10.382 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:10.390 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:10.420 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:10.516 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-19T03:24:10.517 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-19T03:24:10.617 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-19T03:24:14.519 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:14.526 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:14.618 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:14.628 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-19T03:24:14.636 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-19T03:24:14.729 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-19T03:24:14.729 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-19T03:24:18.638 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:18.730 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:24:18.749 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-03-19T03:24:18.841 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-03-19T03:24:18.850 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-03-19T03:24:18.942 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-03-19T03:24:19.047 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 5498.749699831009 failure_reason: '"2024-03-19T02:19:06.291993+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-19T03:24:19.047 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T03:24:19.142 INFO:teuthology.run:FAIL